Image classification, training method, device, equipment and medium for image classification model

By acquiring the association hierarchy information of the target image and establishing the association matrix, the problem of low image classification accuracy in existing technologies is solved, achieving higher classification accuracy and reliability.

CN114429567BActive Publication Date: 2024-11-22TENCENT TECHNOLOGY (SHENZHEN) CO LTD
View PDF 9 Cites 0 Cited by

Patent Information

Application Number
CN202210097757.1
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-01-27
Publication Date
2024-11-22
Estimated Expiration
2042-01-27

AI Technical Summary

Technical Problem

In existing technologies, the accuracy of image classification is relatively low, mainly because it only considers the information of the image itself and ignores the correlation between different classification levels.

Method used

By acquiring the association hierarchy information of the target image and combining it with the association information of the first classification hierarchy, the classification result of the target image at the first classification hierarchy is obtained. The image classification model is then used for training, and an association matrix is ​​established to model the association relationship between different classification hierarchies.

Benefits of technology

This improves the accuracy and reliability of image classification by considering the correlation information between the image and the associated hierarchy, thus enriching the information content of the classification results and enhancing their reliability.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114429567B_ABST
    Figure CN114429567B_ABST
Patent Text Reader

Abstract

The present application discloses an image classification, a training method, an apparatus, a device and a medium for an image classification model, and belongs to the field of artificial intelligence technology. The image classification method comprises: obtaining a target image; based on the target image, obtaining the association information between the first classification level and the association level; based on the classification result and the association information corresponding to the target image at the association level, obtaining the classification result corresponding to the target image at the first classification level; wherein the association level is a classification level that satisfies the association condition with the first classification level among at least two classification levels. In this way, in the process of obtaining the classification results corresponding to the target image at the first classification level, not only the image itself is considered, but also the association information between the first classification level and the association level is considered. The information considered is relatively rich, which is conducive to improving the reliability of the classification results, and thus improving the accuracy of image classification.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The embodiments of the present application relate to the field of artificial intelligence technology, and in particular to an image classification, image classification model training method, device, equipment and medium. Background Art

[0002] With the development of artificial intelligence technology, there are more and more ways to classify images, such as structured classification of images. In the process of structured classification of images, it is necessary to obtain classification results corresponding to the images at at least two classification levels.

[0003] In the related art, the classification results corresponding to the image at each classification level are directly obtained based on the image. In this way, the classification result acquisition process only considers the image itself, the information considered is relatively limited, the reliability of the classification result is poor, and the accuracy of image classification is low. Summary of the invention

[0004] The embodiments of the present application provide an image classification, image classification model training method, device, equipment and medium, which can be used to improve the accuracy of image classification. The technical solution is as follows:

[0005] On the one hand, an embodiment of the present application provides an image classification method, the method comprising:

[0006] Get the target image;

[0007] Based on the target image, acquiring association information between the first classification level and the association level;

[0008] Based on the classification result corresponding to the target image at the association level and the association information, obtaining the classification result corresponding to the target image at the first classification level;

[0009] The first classification level is any classification level among at least two classification levels that meets the selection condition, and the associated level is a classification level among the at least two classification levels that meets the association condition with the first classification level.

[0010] A method for training an image classification model is also provided, the method comprising:

[0011] Obtaining a sample image and classification labels corresponding to the sample image at at least two classification levels;

[0012] Calling the image classification model to obtain the classification results corresponding to the sample image at the at least two classification levels, respectively, wherein the classification result corresponding to the sample image at the first classification level is obtained based on the classification result corresponding to the sample image at the association level and the association information between the first classification level and the association level;

[0013] Based on the classification results corresponding to the sample image at the at least two classification levels and the classification labels corresponding to the sample image at the at least two classification levels, a target loss is obtained; and the image classification model is trained using the target loss to obtain a target image classification model;

[0014] The first classification level is any classification level among the at least two classification levels that meets the selection condition, and the associated level is a classification level among the at least two classification levels that meets the association condition with the first classification level.

[0015] In another aspect, an image classification device is provided, the device comprising:

[0016] A first acquisition unit, used for acquiring a target image;

[0017] A second acquisition unit is configured to acquire association information between the first classification level and the association level based on the target image; and acquire a classification result corresponding to the target image at the first classification level based on the classification result corresponding to the target image at the association level and the association information;

[0018] The first classification level is any classification level among at least two classification levels that meets the selection condition, and the associated level is a classification level among the at least two classification levels that meets the association condition with the first classification level.

[0019] In one possible implementation, the second acquisition unit is used to obtain, based on the target image, the matching probability of the target image and each candidate category corresponding to the first classification level, on the premise that the target image belongs to any candidate category corresponding to the association level; obtain the association sub-information between the first classification level and any candidate category based on the matching probability of the target image and each candidate category corresponding to the first classification level; and obtain the association information between the first classification level and the association level based on the association sub-information between the first classification level and each candidate category corresponding to the association level.

[0020] In one possible implementation, the second acquisition unit is used to normalize the matching probability between the target image and each candidate category corresponding to the first classification level to obtain the normalized matching probability between the target image and each candidate category corresponding to the first classification level; based on the normalized matching probability between the target image and each candidate category corresponding to the first classification level, obtain the associated sub-information between the first classification level and any candidate category.

[0021] In one possible implementation, the second acquisition unit is used to perform feature extraction on the target image to obtain image features corresponding to the target image at the first classification level; based on the image features corresponding to the target image at the first classification level, obtain the matching probability of the target image with each candidate category corresponding to the first classification level under the premise that the target image belongs to any of the candidate categories.

[0022] In one possible implementation, different classification levels have different granularities, and the classification level that satisfies the association condition with the first classification level is a classification level that is located a reference number of digits before the first classification level in the order of granularity from fine to coarse.

[0023] In one possible implementation, the reference number is one, the classification level that meets the selection condition is the classification level other than the classification level with the finest granularity among the at least two classification levels, and the second acquisition unit is further used to sequentially acquire the classification results corresponding to the target image at the at least two classification levels in the order of the granularity of the at least two classification levels from fine to coarse.

[0024] In a possible implementation, the second acquisition unit is further used to classify the image features corresponding to the target image at the finest classification level to obtain the classification result corresponding to the target image at the finest classification level.

[0025] In one possible implementation, the second acquisition unit is used to call the target image classification model to obtain the association information between the first classification level and the association level based on the target image; and call the target image classification model to obtain the classification result corresponding to the target image at the first classification level based on the classification result corresponding to the target image at the association level and the association information.

[0026] In one possible implementation, the target image is an image for biological classification, and the at least two classification levels are at least two biological classification levels; or, the target image is an image for medical classification, and the at least two classification levels are at least two medical classification levels.

[0027] A training device for an image classification model is also provided, the device comprising:

[0028] A first acquisition unit, used to acquire a sample image and classification labels corresponding to the sample image at at least two classification levels;

[0029] A second acquisition unit is used to call the image classification model to obtain the classification results corresponding to the sample image at the at least two classification levels, wherein the classification result corresponding to the sample image at the first classification level is obtained based on the classification result corresponding to the sample image at the association level and the association information between the first classification level and the association level;

[0030] A third acquisition unit, configured to acquire a target loss based on the classification results corresponding to the sample image at the at least two classification levels and the classification labels corresponding to the sample image at the at least two classification levels;

[0031] A training unit, used to train the image classification model using the target loss to obtain a target image classification model;

[0032] The first classification level is any classification level among the at least two classification levels that meets the selection condition, and the associated level is a classification level among the at least two classification levels that meets the association condition with the first classification level.

[0033] In one possible implementation, the third acquisition unit is used to obtain a first loss based on the classification results corresponding to the sample image at the at least two classification levels and the classification labels corresponding to the sample image at the at least two classification levels; obtain a second loss corresponding to the first classification level based on association information between the first classification level and the association level; and obtain the target loss based on the first loss and the second losses corresponding to each classification level that meets the selection conditions.

[0034] In a possible implementation, the classification labels corresponding to the sample images at at least two classification levels are extracted from a label tree; the device further includes:

[0035] A construction unit is used to construct the label tree based on the arrangement order of the granularity of the at least two classification levels and the classification labels corresponding to each labeled image under the at least two classification levels, wherein each labeled image includes the sample image.

[0036] On the other hand, a computer device is provided, comprising a processor and a memory, wherein the memory stores at least one computer program, and the at least one computer program is loaded and executed by the processor so that the computer device implements any of the above-mentioned image classification methods or image classification model training methods.

[0037] On the other hand, a computer-readable storage medium is also provided, in which at least one computer program is stored. The at least one computer program is loaded and executed by a processor so that a computer implements any of the above-mentioned image classification methods or image classification model training methods.

[0038] On the other hand, a computer program product is also provided, which includes a computer program or computer instructions, and the computer program or the computer instructions are loaded and executed by a processor so that a computer implements any of the above-mentioned image classification methods or image classification model training methods.

[0039] The technical solution provided by the embodiments of the present application brings at least the following beneficial effects:

[0040] The technical solution provided in the embodiment of the present application not only takes into account the image itself, but also takes into account the associated information between the first classification level and the associated level in the process of obtaining the classification results corresponding to the target image at the first classification level. The considered information is relatively rich, which is conducive to improving the reliability of the classification results, and thus improving the accuracy of image classification. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0042] Figure 1 It is a schematic diagram of an implementation environment provided by an embodiment of the present application;

[0043] Figure 2 is a flow chart of an image classification method provided in an embodiment of the present application;

[0044] Figure 3 is a schematic diagram of a process of obtaining classification results corresponding to a target image at at least two classification levels provided by an embodiment of the present application;

[0045] Figure 4 is a schematic diagram of a visual presentation result provided by an embodiment of the present application;

[0046] Figure 5 It is a flowchart of a training method of an image classification model provided in an embodiment of the present application;

[0047] Figure 6 is a schematic diagram of a tag tree provided in an embodiment of the present application;

[0048] Figure 7 is a schematic diagram of an image classification device provided in an embodiment of the present application;

[0049] Figure 8 is a schematic diagram of a training device for an image classification model provided in an embodiment of the present application;

[0050] Fig. 9 It is a structural diagram of a server provided in an embodiment of the present application;

[0051] Fig.10 It is a structural diagram of a terminal provided in an embodiment of the present application. DETAILED DESCRIPTION

[0052] In order to make the objectives, technical solutions and advantages of the present application clearer, the implementation methods of the present application will be further described in detail below with reference to the accompanying drawings.

[0053] In an exemplary embodiment, the image classification method and the training method of the image classification model provided in the embodiments of the present application can be applied to various scenarios, including but not limited to cloud technology, artificial intelligence, smart transportation, assisted driving, etc.

[0054] Artificial Intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology of computer science. Artificial intelligence attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines, so that machines have the functions of perception, reasoning and decision-making.

[0055] Artificial intelligence technology is a comprehensive discipline that covers a wide range of fields, including both hardware-level and software-level technologies. Basic artificial intelligence technologies generally include sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operating / interactive systems, mechatronics and other technologies. Artificial intelligence software technologies mainly include computer vision technology, speech processing technology, natural language processing technology, as well as machine learning / deep learning, autonomous driving, smart transportation and other major directions.

[0056] Computer Vision (CV) is a science that studies how to make machines "see". To put it more specifically, it refers to machine vision such as using cameras and computers to replace human eyes to identify and measure targets, and further perform image processing to make computer processing into images that are more suitable for human eye observation or transmission to instruments for detection. As a scientific discipline, computer vision studies related theories and technologies, and attempts to establish an artificial intelligence system that can obtain information from images or multi-dimensional data. Computer vision technology usually includes image processing (such as image classification, image segmentation, etc.), image recognition, image semantic understanding, image retrieval, OCR (Optical Character Recognition), video processing, video semantic understanding, video content / behavior recognition, three-dimensional object reconstruction, 3D (Three-Dimension) technology, virtual reality, augmented reality, map construction, autonomous driving, smart transportation and other technologies.

[0057] Machine Learning (ML) is a multi-disciplinary subject that involves probability theory, statistics, approximation theory, convex analysis, algorithm complexity theory and other disciplines. It specializes in studying how computers simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to make computers intelligent. Its applications are spread across all areas of artificial intelligence. Machine learning and deep learning usually include artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and teaching learning.

[0058] With the research and advancement of artificial intelligence technology, artificial intelligence technology has been studied and applied in many fields, such as common smart homes, smart wearable devices, virtual assistants, smart speakers, smart marketing, unmanned driving, automatic driving, drones, robots, smart medical care, smart customer service, Internet of Vehicles, automatic driving, smart transportation, etc. I believe that with the development of technology, artificial intelligence technology will be applied in more fields and play an increasingly important role.

[0059] Figure 1 The schematic diagram of the implementation environment provided by the embodiment of the present application is shown. The implementation environment includes: a terminal 11 and a server 12.

[0060] The image classification method provided in the embodiment of the present application can be executed by the terminal 11, or by the server 12, or by both the terminal 11 and the server 12, and the embodiment of the present application does not limit this. In the case where the image classification method provided in the embodiment of the present application is jointly executed by the terminal 11 and the server 12, the server 12 undertakes the main computing work, and the terminal 11 undertakes the secondary computing work; or, the server 12 undertakes the secondary computing work, and the terminal 11 undertakes the main computing work; or, the server 12 and the terminal 11 adopt a distributed computing architecture for collaborative computing.

[0061] The training method of the image classification model provided in the embodiment of the present application can be executed by the terminal 11, can be executed by the server 12, or can be jointly executed by the terminal 11 and the server 12, and the embodiment of the present application does not limit this. In the case where the training method of the image classification model provided in the embodiment of the present application is jointly executed by the terminal 11 and the server 12, the server 12 undertakes the main computing work, and the terminal 11 undertakes the secondary computing work; or, the server 12 undertakes the secondary computing work, and the terminal 11 undertakes the main computing work; or, the server 12 and the terminal 11 adopt a distributed computing architecture for collaborative computing.

[0062] It should be noted that the execution device of the image classification method and the execution device of the image classification model training method may be the same or different, and the embodiments of the present application do not limit this.

[0063] In a possible implementation, the terminal 11 can be any electronic product that can interact with the user through one or more methods such as keyboard, touchpad, touch screen, remote control, voice interaction or handwriting device, such as PC (Personal Computer), mobile phone, smart phone, PDA (Personal Digital Assistant), wearable device, PPC (Pocket PC), tablet computer, smart car machine, smart TV, smart speaker, smart voice interaction device, smart home appliance, car terminal, etc. The server 12 can be a server, or a server cluster composed of multiple servers, or a cloud computing service center. The terminal 11 establishes a communication connection with the server 12 through a wired or wireless network.

[0064] Those skilled in the art should understand that the above-mentioned terminal 11 and server 12 are only examples, and other existing or future terminals or servers that are applicable to the present application should also be included in the protection scope of the present application and are included here by reference.

[0065] Based on the above Figure 1In the implementation environment shown, the present application embodiment provides an image classification method, which is executed by a computer device, which may be a terminal 11 or a server 12, and the present application embodiment does not limit this. Figure 2 As shown, the image classification method provided in the embodiment of the present application includes the following steps 201 and 202.

[0066] In step 201, a target image is acquired.

[0067] The target image refers to any image to be classified. In the embodiment of the present application, the purpose of classifying the target image is to obtain the classification results corresponding to the target image at at least two classification levels. Exemplarily, the process of obtaining the classification results corresponding to the target image at at least two classification levels can also be called the process of performing structured classification on the target image. In other words, the target image is an image that can be structured classified.

[0068] Structured classification is an image classification method, which is a basic task in computer vision. Structured classification aims to determine the category to which an image belongs at different classification levels. For example, according to biological classification, each bird belongs to a category in each of the seven classification levels: kingdom, phylum, class, order, family, genus, and species. Structured classification will classify the image of a certain bird at each of the seven classification levels, and obtain the classification results corresponding to the image at each of the seven classification levels, and then determine the category to which the image belongs at each of the seven classification levels. Structured classification can take into account the structured relationship between classification levels, so as to better learn the correlation between the categories corresponding to the classification levels.

[0069] At least two classification levels refer to classification levels for reference in structured classification of target images. The embodiments of the present application do not limit the setting of at least two classification levels, and can be flexibly set according to the type of target image, or flexibly adjusted according to actual application scenarios. Different classification levels have different granularities. A coarser classification level is used for coarser classification of target images, and a finer classification level is used for finer classification of target images.

[0070] Each classification level corresponds to at least one candidate category, and the at least one candidate category corresponding to any classification level refers to the category belonging to any classification level in a structured classification scenario. The candidate categories corresponding to at least two classification levels are set based on experience or flexibly adjusted according to the application scenario, which is not limited in the embodiments of the present application. Exemplarily, the number of candidate categories corresponding to the coarse-grained classification level is not greater than the number of candidate categories corresponding to the fine-grained classification level. Exemplarily, in the order of granularity from fine to coarse, each candidate category corresponding to the finer-grained classification level in two adjacent classification levels is a subdivision category under a candidate category corresponding to the coarser-grained classification level.

[0071] In an exemplary embodiment, the target image is an image for biological classification. In this case, the at least two classification levels are at least two biological classification levels. Exemplarily, the number of biological classification levels is 7, and the 7 biological classification levels are: species, genus, family, order, class, phylum, kingdom in order from fine to coarse granularity. Of course, the at least two biological classification levels can also be other situations, which are not limited in the embodiments of the present application.

[0072] In an exemplary embodiment, the target image is an image for medical classification. In this case, the at least two classification levels are at least two medical classification levels. Exemplarily, the number of medical classification levels is 2, and the two medical classification levels are, in order from fine to coarse granularity: a small medical disease category, a large medical disease category. Of course, the at least two medical classification levels can also be other situations, which are not limited in the embodiments of the present application.

[0073] It should be noted that the above-mentioned target image being an image for biological classification and the target image being an image for medical classification are merely illustrative examples, and the embodiments of the present application are not limited thereto. In some embodiments, the target image may also be other types of images.

[0074] The embodiment of the present application does not limit the method of acquiring the target image. For example, the target image can be acquired by a computer device. For example, the target image can be sent to the computer device by an image acquisition device. For example, the target image can be extracted from an image library by the computer device. For example, the target image can be crawled from the network by the computer device, etc.

[0075] In step 202, based on the target image, the association information between the first classification level and the association level is obtained; based on the classification result corresponding to the target image at the association level and the association information, the classification result corresponding to the target image at the first classification level is obtained.

[0076] The first classification level is any classification level among at least two classification levels that meets the selection condition, and the associated level is a classification level among at least two classification levels that meets the association condition with the first classification level.

[0077] After acquiring the target image, the target image is structured classified to obtain classification results corresponding to the target image at at least two classification levels. In an embodiment of the present application, in the process of performing structured classification on the target image, the process of obtaining the classification result corresponding to the target image at the first classification level includes: based on the target image, obtaining the association information between the first classification level and the association level; based on the classification result corresponding to the target image at the association level and the association information, obtaining the classification result corresponding to the target image at the first classification level. In other words, the process of obtaining the classification result corresponding to the target image at the first classification level not only considers the target image itself, but also considers the association information between the classification levels. The information considered is relatively rich, which is conducive to improving the reliability of the classification results, and thus improving the accuracy of image classification.

[0078] The first classification level is any classification level that satisfies the selection condition among at least two classification levels, and the classification level that satisfies the selection condition refers to the classification level that needs to obtain the classification result corresponding to the target image by considering the association information between the association level. The selection condition is set based on experience or flexibly adjusted according to the application scenario, and the embodiments of the present application do not limit this.

[0079] In an exemplary embodiment, the classification levels satisfying the selection condition refer to all classification levels with associated levels. In an exemplary embodiment, the classification levels satisfying the selection condition refer to one or more designated classification levels among all classification levels with associated levels.

[0080] The associated level of the first classification level is a classification level that satisfies the association condition with the first classification level among at least two classification levels. The association condition is set based on experience or flexibly adjusted according to application scenarios, and the embodiments of the present application do not limit this.

[0081] In an exemplary embodiment, different classification levels have different granularities, and the classification level that satisfies the association condition with the first classification level may refer to the classification level that is located before the first classification level by a reference number of digits in the order of granularity from fine to coarse. In other words, the association level of the first classification level is the classification level that is located before the first classification level by a reference number of digits in the order of granularity from fine to coarse. In this case, the granularity of the association level of the first classification level is finer than that of the first classification level.

[0082] The reference number is set based on experience or flexibly adjusted according to the application scenario, and the embodiments of the present application do not limit this. For example, the reference number is one, or the reference number is two, or the reference number is three, etc. The embodiments of the present application are described by taking the reference number as one as an example. The principle of the image classification method when the reference number is other numbers is similar to the principle of the image classification method when the reference number is one.

[0083] For the case where the reference number is one, the associated level of the first classification level is the classification level that is located before the first classification level in the order of granularity from fine to coarse, that is, the associated level of the first classification level is the classification level of finer granularity that is adjacent to the first classification level in the order of granularity from fine to coarse.

[0084] In an exemplary embodiment, taking the case where the reference number is one and the classification levels satisfying the selection condition are all the classification levels with associated levels as an example, since each classification level except the classification level with the finest granularity has an associated level, the classification level satisfying the selection condition is the classification level except the classification level with the finest granularity among the at least two classification levels. In other words, the first classification level is any classification level except the classification level with the finest granularity among the at least two classification levels.

[0085] In an exemplary embodiment, the classification result corresponding to the target image at any classification level includes the matching probability of the target image and each candidate category corresponding to the classification level. The embodiment of the present application does not limit the representation form of the classification result corresponding to the target image at any classification level. Exemplarily, the classification result corresponding to the target image at any classification level is represented by a matrix or a vector.

[0086] It should be noted that the implementation process of step 202 can be implemented by running a pre-edited computer program, or by calling a target image classification model. The embodiment of the present application takes calling a target image classification model to implement step 202 as an example. That is, calling the target image classification model to obtain the association information between the first classification level and the association level based on the target image; calling the target image classification model to obtain the classification result corresponding to the target image at the first classification level based on the classification result and association information corresponding to the target image at the association level.

[0087] The target image classification model is a trained image classification model, and the target image classification model can obtain the classification results corresponding to the target image at at least two classification levels. After the target image is input into the target image classification model, the target image model can obtain the classification results corresponding to the target image at at least two classification levels. The process of obtaining the classification results corresponding to the target image at at least two classification levels is an internal processing process of the target image classification model, which is related to the structure of the target image classification model. The embodiment of the present application does not limit the structure of the target image classification model, as long as the process of obtaining the classification result corresponding to the target image at the first classification level is the process described in step 202.

[0088] Exemplarily, when the associated level of the first classification level is a classification level that is located before the first classification level in the order of granularity from fine to coarse, and the first classification level is any classification level other than the classification level with the finest granularity among at least two classification levels, the target image classification model may include a classification model and a transfer model. Exemplarily, the classification model is used to obtain the classification result corresponding to the target image at the classification level with the finest granularity, the number of transfer models is the same as the number of other classification levels except the classification level with the finest granularity, and the transfer model corresponds one-to-one to other classification levels except the classification level with the finest granularity. Any transfer model is used to predict the association information between the classification level corresponding to any transfer model and the associated level, and determine the classification result of the target image at the classification level corresponding to any transfer model based on the association information and the classification result corresponding to the target image at the association level.

[0089] Exemplarily, the classification model is also used to extract image features of the target image. Exemplarily, the classification model can be any model that can extract image features and output classification results. Exemplarily, the classification model can be a ResNet (Residual Neural Network) model.

[0090] The target image classification model is obtained by training the image classification model. The process of training the target image classification model is detailed in Figure 5 The embodiments shown will not be described in detail here.

[0091] Exemplarily, the association information between the first classification level and the association level is used to indicate information on which the classification result corresponding to the target image at the first classification level is obtained based on the classification result corresponding to the target image at the association level. The association information between the first classification level and the association level is association information determined under the influence of the target image. It should be noted that the association information between the first classification level and the association level determined under the influence of different images may be different, but the association relationship between the candidate category corresponding to the first classification level and the candidate category corresponding to the association level indicated by the determined association information is similar.

[0092] In a possible implementation, based on the target image, the process of obtaining association information between the first classification level and the association level includes the following steps 1 to 3.

[0093] Step 1: Based on the target image, obtain the matching probability between the target image and each candidate category corresponding to the first classification level under the premise that the target image belongs to any candidate category corresponding to the association level.

[0094] On the premise that the target image belongs to any candidate category corresponding to the association level, the matching probability between the target image and each candidate category corresponding to the first classification level is a conditional probability obtained on the basis of the target image, which can indicate the association between each candidate category corresponding to the first classification level and any candidate category corresponding to the association level, that is, the association between the first classification level and any candidate category corresponding to the association level.

[0095] Exemplarily, based on the target image, the process of obtaining the matching probability between the target image and each candidate category corresponding to the first classification level under the premise that the target image belongs to any candidate category corresponding to the association level can be: extracting features from the target image to obtain the image features corresponding to the target image at the first classification level; based on the image features corresponding to the target image at the first classification level, obtaining the matching probability between the target image and each candidate category corresponding to the first classification level under the premise that the target image belongs to any candidate category corresponding to the association level. In this way, the image features used to obtain the association information between different classification levels and association levels are different.

[0096] Exemplarily, the process of performing feature extraction on a target image to obtain image features corresponding to the target image at a first classification level includes: performing feature extraction on the target image to obtain image features corresponding to the target image at each classification level; and extracting image features corresponding to the target image at the first classification level from the image features corresponding to the target image at each classification level.

[0097] Exemplarily, the image features corresponding to the target image at a coarser classification level can be regarded as the coarser features of the target image, and the image features corresponding to the target image at a finer classification level can be regarded as the finer features of the target image. The coarser features of the target image can provide data support for the coarser classification of the target image, and the finer features of the target image can provide data support for the finer classification of the target image.

[0098] Exemplarily, in the case where step 202 is implemented by calling the target image classification model, the process of extracting features from the target image is also the processing process of the target image classification model. That is, the target image classification model is called to extract features from the target image to obtain image features corresponding to the target image at at least two classification levels. After the target image is input into the target image classification model, the target image classification model can extract features from the target image to obtain image features corresponding to the target image at at least two classification levels. The image features corresponding to the target image at any classification level are the information used to obtain the classification results corresponding to the target image at any classification level.

[0099] After obtaining the image features corresponding to the target image at the first classification level, based on the image features corresponding to the target image at the first classification level, obtain the matching probability of the target image with each candidate category corresponding to the first classification level under the premise that the target image belongs to any candidate category corresponding to the associated level.

[0100] Exemplarily, for the case where step 202 is implemented by calling the target image classification model, the above process of obtaining the matching probability of the target image with each candidate category corresponding to the first classification level under the premise that the target image belongs to any candidate category corresponding to the associated level based on the image features corresponding to the target image at the first classification level is implemented by calling the target image classification model.

[0101] It should be noted that the above-described process of obtaining the matching probability of the target image with each candidate category corresponding to the first classification level based on the image features corresponding to the target image at the first classification level, under the premise that the target image belongs to any candidate category corresponding to the associated level, is only an exemplary description of the process of obtaining the matching probability of the target image with each candidate category corresponding to the first classification level based on the target image, under the premise that the target image belongs to any candidate category corresponding to the associated level, and the embodiments of the present application are not limited to this.

[0102] In an exemplary embodiment, based on the target image, the process of obtaining the matching probability between the target image and each candidate category corresponding to the first classification level under the premise that the target image belongs to any candidate category corresponding to the association level can also be: extracting features from the target image to obtain the general image features of the target image; based on the general image features, obtaining the matching probability between the target image and each candidate category corresponding to the first classification level under the premise that the target image belongs to any candidate category corresponding to the association level. In this way, the process of obtaining the association information between different classification levels and association levels is based on the general image features.

[0103] Step 2: Based on the matching probability between the target image and each candidate category corresponding to the first classification level, obtain the associated sub-information between the first classification level and any candidate category.

[0104] The matching probability of the target image with each candidate category corresponding to the first classification level in step 2 refers to the matching probability of the target image with each candidate category corresponding to the first classification level under the premise that the target image belongs to any candidate category corresponding to the association level. Since the matching probability of the target image with each candidate category corresponding to the first classification level can indicate the association between the first classification level and any candidate category corresponding to the association level, after obtaining the matching probability of the target image with each candidate category corresponding to the first classification level, the association sub-information between the first classification level and any candidate category is obtained based on the matching probability of the target image with each candidate category corresponding to the first classification level.

[0105] Exemplarily, based on the matching probabilities between the target image and each candidate category corresponding to the first classification level, the method of obtaining the associated sub-information between the first classification level and any candidate category can be: directly based on the matching probabilities between the target image and each candidate category corresponding to the first classification level, obtaining the associated sub-information between the first classification level and any candidate category. Exemplarily, a matrix or vector consisting of the matching probabilities between the target image and each candidate category corresponding to the first classification level is used as the associated sub-information between the first classification level and any candidate category.

[0106] Exemplarily, based on the matching probability between the target image and each candidate category corresponding to the first classification level, the method for obtaining the associated sub-information between the first classification level and any candidate category can also be: normalizing the matching probability between the target image and each candidate category corresponding to the first classification level to obtain the normalized matching probability between the target image and each candidate category corresponding to the first classification level; based on the normalized matching probability between the target image and each candidate category corresponding to the first classification level, obtaining the associated sub-information between the first classification level and any candidate category.

[0107] Exemplarily, the normalization process is implemented using an activation function, for example, the activation function is softmax (an activation function). The matching probabilities of the target image and each candidate category corresponding to the first classification level are normalized to ensure that the sum of the normalized matching probabilities of the target image and each candidate category corresponding to the first classification level is 1, thereby improving the reliability of the obtained associated sub-information.

[0108] Exemplarily, based on the normalized matching probabilities between the target image and each candidate category corresponding to the first classification level, a method for obtaining associated sub-information between the first classification level and any candidate category can be: using a matrix or vector consisting of the normalized matching probabilities between the target image and each candidate category corresponding to the first classification level as the associated sub-information between the first classification level and any candidate category.

[0109] Exemplarily, under the premise that the target image belongs to any candidate category corresponding to the association level, the normalized matching probability of the target image and each candidate category corresponding to the first classification level can constitute a column vector, and by applying a softmax activation function in the column direction, the column vector can meet the normalization condition. Exemplarily, a fine-grained category may belong to any coarse-grained category during modeling, and this method can ensure that the sum of the probabilities of a fine-grained category belonging to each coarse-grained category is 1.

[0110] Step 3: Based on the associated sub-information between the first classification level and each candidate category corresponding to the associated level, the associated information between the first classification level and the associated level is obtained.

[0111] According to step 1 and step 2, the associated sub-information between the first classification level and each candidate category corresponding to the associated level can be obtained. The number of associated sub-information is the same as the number of candidate categories corresponding to the associated level.

[0112] After obtaining the associated sub-information between each candidate category corresponding to the first classification level and the associated level, based on the associated sub-information between each candidate category corresponding to the first classification level and the associated level, the associated information between the first classification level and the associated level is obtained. Exemplarily, the associated sub-information between each candidate category corresponding to the first classification level and the associated level is a column matrix or a column vector, and the matrix or vector composed of the associated sub-information between each candidate category corresponding to the first classification level and the associated level is used as the associated information between the first classification level and the associated level.

[0113] Since the association information between the first classification level and the association level is obtained based on the association sub-information between the first classification level and the respective candidate categories corresponding to the association level, the classification result corresponding to the target image at the association level includes the matching probability between the target image and the respective candidate categories corresponding to the association level. Therefore, the classification result corresponding to the target image at the first classification level can be determined based on the classification result corresponding to the target image at the association level and the association information between the first classification level and the association level.

[0114] Exemplarily, the association information between the first classification level and the association level can be represented by a matrix, and the association information represented by the matrix can be called an association matrix. Exemplarily, the association matrix can also be called a transfer matrix. The number of elements in the association matrix is ​​the product of the number of candidate categories corresponding to the first classification level and the number of candidate categories corresponding to the association level.

[0115] For example, assuming that the number of at least two classification levels is K (K is an integer not less than 2), the number of candidate categories corresponding to the K classification levels arranged in order from fine to coarse granularity is (C1, C2, ... C K ). Assuming that the first classification level is the classification level at the kth (k is an integer not less than 2 and not greater than K) position in the order of granularity from fine to coarse, and the reference number is one, then the associated level of the first classification level is the classification level at the (k-1)th position in the order of granularity from fine to coarse, and the number of candidate categories corresponding to the first classification level is C k , the number of candidate categories corresponding to the associated level is C k-1 , then the association matrix between the first classification level and the association level can be a matrix with a size of C k *C k-1 The matrix, that is, C k Row C k-1 The correlation matrix can be expressed using Formula 1:

[0116]

[0117] Where x represents the target image; represents the association matrix between the first classification level and the association level; express The matrix element located in the i-th row and j-th column in C, where i is not less than 1 and not greater than C k An integer, j is not less than 1 and not greater than C k-1 An integer. The expression of is shown in Formula 2:

[0118]

[0119] Among them, p(sk =i|s k-1 = j; x) indicates that the target image x belongs to the associated level s k-1 Under the premise of the corresponding j-th candidate category, the target image x and the first classification level s k The matching probability (or normalized matching probability) of the corresponding i-th candidate category.

[0120] For example, for a structured classification task, assuming that the number of classification levels is K, the K classification levels are represented as (s 1 ,s 2 , ..., s K ), each classification level corresponds to (C1, C2, ...C K ) candidate categories. Assuming that the classification level gradually becomes coarse-grained when increasing from 1 to K, structured classification is actually to learn and model the probability distribution shown in Formula 3:

[0121] p(s 1 ,s 2 , ..., s K |x) (Formula 3)

[0122] Among them, p(s k |x) represents the target image x in the kth classification level s k The probability distribution under the k-th classification level s k The corresponding classification results are shown below.

[0123] In the embodiment of the present application, for the case where the associated level of the first classification level is a classification level that is located before the first classification level in the order of granularity from fine to coarse, the classification result corresponding to the target image at the kth classification level is modeled as being statistically dependent on the fine-grained classification result corresponding to the target image at the (k-1)th classification level and the target image x, so the classification result corresponding to the target image x at the kth classification level can be expressed using the following formula 4:

[0124]

[0125] Among them, p(s k-1 |x) indicates that the target image x is at the (k-1)th classification level s k-1 The probability distribution under the (k-1) classification level s k-1 The corresponding classification results are as follows; Describes the transition probability from the (k-1)th classification level to the kth classification level, that is, defines the probability of the transition process from fine-grained to coarse-grained. Formula 4 can be understood as a structured annotation method. The classification result corresponding to the target image x at the kth classification level is predicted based on the classification result corresponding to the target image x at the (k-1)th classification level and the information of the target image x itself.

[0126] The above formula 4 defines a learning process from fine-grained to coarse-grained, gradually learning coarse-grained predictions based on fine-grained predictions, and the core of it is the transition probability In the present embodiment, the transition probability It is expressed as the correlation matrix shown in Formula 1.

[0127] For example, assuming that the association information between the first classification level and the association level is the association matrix represented by Formula 1, the classification result corresponding to the target image at the association level is obtained by Indicates that the classification result of the target image at the first classification level It can be calculated based on formula 5:

[0128]

[0129] in, and are matrices, The size is C k *C k-1 , The size is C k-1 *1, The size is C k *1.

[0130] Exemplarily, when the associated level of the first classification level is the classification level that is located before the first classification level in the order of granularity from fine to coarse, and the first classification level is any classification level except the classification level with the finest granularity among at least two classification levels, since the classification result corresponding to the target image at the coarse-grained classification level is obtained based on the classification result corresponding to the target image at the fine-grained classification level, the classification results corresponding to the target image at at least two classification levels are obtained in sequence according to the order of granularity from fine to coarse of the at least two classification levels. In other words, the classification results corresponding to the target image at at least two classification levels are obtained in sequence according to the order of granularity from fine to coarse of the at least two classification levels.

[0131] In the process of sequentially obtaining the classification results corresponding to the target image at at least two classification levels in the order of granularity from fine to coarse, after obtaining the classification result corresponding to the target image at the classification level with the finest granularity, the classification levels located at the 2nd, 3rd, ..., Kth in the order of granularity from fine to coarse are sequentially used as the first classification level, and the classification results corresponding to the target image at the 2nd, 3rd, ..., Kth classification levels are sequentially obtained in accordance with the method of step 202. After obtaining the classification result corresponding to the target image at the classification level located at the Kth, the classification results corresponding to the target image at at least two classification levels are obtained. Wherein, K is the number of classification levels, and K is an integer not less than 2.

[0132] Exemplarily, the process of obtaining the classification result corresponding to the target image at the finest granularity classification level includes: classifying the image features corresponding to the target image at the finest granularity classification level to obtain the classification result corresponding to the target image at the finest granularity classification level. Since there is no associated level of the finest granularity classification level in at least two classification levels, the classification result corresponding to the target image at the finest granularity classification level can be obtained directly by classifying the image features corresponding to the target image at the finest granularity classification level. Exemplarily, the classification result corresponding to the target image at the finest granularity classification level includes the matching probability of the target image and each candidate category corresponding to the finest granularity classification level.

[0133] It should be noted that the process of classifying the image features corresponding to the target image at the finest classification level and obtaining the classification result corresponding to the target image at the finest classification level can be regarded as not considering any associated information, or can be regarded as considering the associated information between the finest classification level and itself. The associated information between the finest classification level and itself is a unit matrix, and the size of the unit matrix is ​​C1*C1.

[0134] Exemplarily, in the case where step 202 is implemented by calling a target image classification model, the above process of classifying the image features corresponding to the target image at the finest classification level and obtaining the classification results corresponding to the target image at the finest classification level is implemented by calling a target image classification model.

[0135] In some embodiments, the common features of the target image may be classified to obtain the classification result corresponding to the target image at the finest classification level.

[0136] For example, it is assumed that the number of classification levels is 3, and the 3 classification levels are classification level 1, classification level 2, and classification level 3 in the order of fine to coarse granularity. Then, in the embodiment of the present application, the classification level with the finest granularity is classification level 1, and the other classification levels except the classification level with the finest granularity are classification level 2 and classification level 3. The associated level of classification level 2 is classification level 1, and the associated level of classification level 3 is classification level 2.

[0137] In the process of structured classification of the target image, the classification result corresponding to the target image at classification level 1, the classification result corresponding to the target image at classification level 2, and the classification result corresponding to the target image at classification level 3 are obtained in sequence. Among them, the classification result corresponding to the target image at classification level 1 is obtained by classifying the image features corresponding to the target image at classification level 1; the classification result corresponding to the target image at classification level 2 is determined based on the classification result corresponding to the target image at classification level 1 and the association information between classification level 2 and classification level 1; the classification result corresponding to the target image at classification level 3 is determined based on the classification result corresponding to the target image at classification level 2 and the association information between classification level 3 and classification level 2.

[0138] In an exemplary embodiment, when the associated level of the first classification level is a classification level that is located before the first classification level in the order of granularity from fine to coarse, and the first classification level is any classification level except the classification level with the finest granularity among the at least two classification levels, the process of obtaining the classification results corresponding to the target image at the at least two classification levels is as follows: Figure 3 As shown. Figure 3 In the example, there are three classification levels, which are classification level 1, classification level 2, and classification level 3 in the order of fine to coarse granularity. That is, classification level 1 is the classification level with the finest granularity. The target image x is input into the target image classification model, and the target image classification model extracts features of the target image x to obtain image features 301 corresponding to the target image x at classification level 1, image features 302 corresponding to the target image x at classification level 2, and image features 303 corresponding to the target image x at classification level 3.

[0139] The image features 301 corresponding to the target image x at classification level 1 are classified, and based on the obtained results and the association information 304 between the classification level 1 and itself, the classification result 305 corresponding to the target image x at classification level 1 is obtained. The number of candidate categories corresponding to the classification level 1 is 8, and the association information 304 between the classification level 1 and itself is a matrix of size 8*8. The classification result 305 corresponding to the target image x at classification level 1 includes the matching probability of the target image x and the 8 candidate categories corresponding to the classification level 1.

[0140] Based on the image feature 302 corresponding to the target image x at classification level 2, the association information 306 between classification level 2 and classification level 1 is obtained, and based on the association information 306 between classification level 2 and classification level 1 and the classification result 305 corresponding to the target image x at classification level 1, the classification result 307 corresponding to the target image x at classification level 2 is determined. The number of candidate categories corresponding to classification level 2 is 4, and the association information 306 between classification level 2 and classification level 1 is a matrix with a size of 4*8. The classification result 307 corresponding to the target image x at classification level 2 includes the matching probability of the target image x and the 4 candidate categories corresponding to classification level 2.

[0141] Based on the image feature 303 corresponding to the target image x at classification level 3, the association information 308 between the classification level 3 and the classification level 2 is obtained, and based on the association information 308 between the classification level 3 and the classification level 2 and the classification result 307 corresponding to the target image x at classification level 2, the classification result 309 corresponding to the target image x at classification level 3 is determined. The number of candidate categories corresponding to the classification level 3 is 2, and the association information 308 between the classification level 3 and the classification level 2 is a matrix with a size of 2*4. The classification result 309 corresponding to the target image x at classification level 3 includes the matching probability of the target image x and the two candidate categories corresponding to the classification level 3.

[0142] according to Figure 3 It can be seen that each candidate category corresponding to classification level 1 is a sub-category of a candidate category corresponding to classification level 2, and each candidate category corresponding to classification level 2 is a sub-category of a candidate category corresponding to classification level 3.

[0143] It should be noted that the above-described process of obtaining the classification results corresponding to the target image at at least two classification levels in sequence from fine to coarse granularity of at least two classification levels is only an exemplary description of the process of structured classification of the target image, and the embodiments of the present application are not limited to this.

[0144] In an exemplary embodiment, taking the reference number as two, that is, the associated level of the first classification level is the classification level located at the first two levels of the first classification level in the order of granularity from fine to coarse, and the first classification level is any classification level with an associated level as an example, in the process of structural classification of the target image, assuming that K classification levels are arranged in order from fine to coarse granularity, after obtaining the classification result corresponding to the target image at the (k-2)th classification level, the classification result corresponding to the target image at the kth classification level can be obtained based on the classification result corresponding to the target image at the (k-2)th classification level and the association information between the kth classification level and the (k-2)th classification level. The process of obtaining the classification result corresponding to the target image at the kth classification level can be performed before the process of obtaining the classification result corresponding to the target image at the (k-1)th classification level, or it can be performed afterwards, or it can be performed in parallel, etc., wherein k is an integer not less than 3 and not greater than K, and K is an integer not less than 3.

[0145] In an exemplary embodiment, after obtaining the classification results corresponding to the target image at at least two classification levels, the method further includes: determining the target categories to which the target image belongs at at least two classification levels based on the classification results corresponding to the target image at at least two classification levels. The target category to which the target image belongs at any classification level can be regarded as the category to which the target image belongs at any classification level determined in the process of structured classification.

[0146] The target category to which the target image belongs at any classification level is determined based on the classification result corresponding to the target image at the classification level. Exemplarily, the classification result corresponding to the target image at the classification level includes the matching probability between the target image and each candidate category corresponding to the classification level, and the target category to which the target image belongs at any classification level is determined based on the candidate category with the highest matching probability with the target image among each candidate category corresponding to the classification level.

[0147] Exemplarily, if the number of candidate categories with the highest probability of matching the target image among the candidate categories corresponding to any classification level is one, then the one candidate category is used as the target category to which the target image belongs under the classification level. Exemplarily, if the number of candidate categories with the highest probability of matching the target image among the candidate categories corresponding to any classification level is multiple, then any one candidate category is selected from the multiple candidate categories as the target category to which the target image belongs under the classification level.

[0148] In an exemplary embodiment, the target categories to which the target image belongs at at least two classification levels can be determined by referring to the method of obtaining the target category to which the target image belongs at any classification level. In an exemplary embodiment, after determining the target categories to which the target image belongs at at least two classification levels, the target categories to which the target image belongs at at least two target classification levels can be stored in correspondence with the target image for direct retrieval and use.

[0149] In an exemplary embodiment, after obtaining the classification results corresponding to the target image at at least two classification levels, the activation areas corresponding to the at least two classification levels can be visualized according to the classification results corresponding to the target image at at least two classification levels. The activation area corresponding to any classification level is used to indicate the image area in the target image that is focused on in the process of obtaining the classification results corresponding to the target image at any classification level. By visually presenting the activation areas corresponding to at least two classification levels, the interpretability of the judgment of the classification model of the target image can be increased.

[0150] Exemplarily, the process of visually presenting the activation areas corresponding to at least two classification levels can be implemented by using CAM (Class Activation Mapping), Grad-CAM (Gradient-weighted ClassActivation Mapping), and the like.

[0151] For example, the results of the visualization are as follows: Figure 4 As shown. Figure 4 It can be seen that the activation areas corresponding to different classification levels (species, family, and order in order from fine to coarse granularity) have a hierarchical relationship. Targets of different major categories often have the same activation areas at the coarse-grained classification level, while the activation areas of concern at the fine-grained classification level often change. This activation area containing structured discriminant information is conducive to better analysis of the judgment area learned by the model, thereby increasing the interpretability of the model's judgment.

[0152] In an embodiment of the present application, since the correlation information between different classification levels is taken into consideration, when determining the category to which the target image belongs, even if the determined category is not an accurate category, the determined category will be a category that is semantically related to the accurate category to a large extent, thereby reducing the severity of the model prediction error.

[0153] The image classification method provided in the embodiment of the present application, in the process of obtaining the classification results corresponding to the target image at the first classification level, not only considers the image itself, but also considers the association information between the first classification level and the association level. The considered information is relatively rich, which is conducive to improving the reliability of the classification results, and thus improving the accuracy of image classification.

[0154] Based on the above Figure 1 In the implementation environment shown, the present application embodiment provides a training method for an image classification model, and the training method for the image classification model is executed by a computer device, and the computer device can be a terminal 11 or a server 12, which is not limited in the present application embodiment. Figure 5 As shown, the training method of the image classification model provided in the embodiment of the present application includes the following steps 501 to 503.

[0155] In step 501, a sample image and classification labels corresponding to the sample image at at least two classification levels are obtained.

[0156] A sample image refers to an image based on which the parameters of the image classification model are updated once. The number of sample images is one or more, and the present application embodiment does not limit this. A sample image refers to an image with a structured classification label. The present application embodiment does not limit the method for obtaining the sample image. Exemplarily, the sample image is extracted from a structured image library or a structured image data set. Exemplarily, the sample image is crawled from the Internet, etc.

[0157] The structured labels of sample images refer to the classification labels corresponding to the sample images at at least two classification levels. The classification labels corresponding to the sample images at any classification level are used to indicate the real category to which the sample images belong at any classification level. The classification labels corresponding to the sample images at at least two classification levels are used to provide supervision information for the training process of the image classification model.

[0158] Exemplarily, the classification labels corresponding to the sample images at at least two classification levels are obtained by the labeling personnel based on their experience. In an exemplary embodiment, the classification labels corresponding to the sample images at at least two classification levels are stored corresponding to the sample images, and the classification labels corresponding to the sample images at at least two classification levels can be obtained when the sample images are obtained.

[0159] It should be noted that at least two classification levels involved in the embodiments of the present application are Figure 2 In the embodiment shown, at least two classification levels are the same. The candidate categories corresponding to at least two classification levels in the embodiment of the present application are the same as Figure 2 The candidate categories corresponding to at least two classification levels involved in the illustrated embodiment are also the same, so as to ensure the classification effect of the image classification model.

[0160] In an exemplary embodiment, the classification labels corresponding to the sample images at at least two classification levels are extracted from the label tree. The label tree uses a tree structure to store the classification labels corresponding to each labeled image at at least two classification levels to save storage space and improve extraction efficiency.

[0161] Exemplarily, before extracting the classification labels corresponding to the sample images at at least two classification levels from the label tree, it is necessary to first construct the label tree. Exemplarily, the process of constructing the label tree includes: constructing the label tree based on the arrangement order of the granularity of at least two classification levels and the classification labels corresponding to each labeled image at at least two classification levels. Wherein, each labeled image includes the sample image.

[0162] A labeled image refers to an image that has been labeled with classification labels corresponding to at least two classification levels. Exemplarily, the root node in the label tree is used to store the classification labels corresponding to each labeled image at the coarsest classification level, and the leaf nodes in the label tree are used to store the identification information of each labeled image. Exemplarily, the identification information of the labeled image may refer to the labeled image itself, or may refer to the storage address of the labeled image, etc., which is not limited in the embodiments of the present application. The nodes of each layer between the root node and the leaf node in the label tree are respectively used to store the classification labels corresponding to each labeled image at the classification level from the second coarsest classification level to the finest classification level.

[0163] Exemplarily, based on the arrangement order of the granularity of at least two classification levels, and the classification labels corresponding to each annotated image at at least two classification levels, the process of constructing the label tree is achieved by traversing each annotated image. The principle of traversing each annotated image is the same. The embodiment of the present application takes the process of traversing any annotated image as an example to illustrate.

[0164] Exemplarily, taking the case where the number of at least two classification levels is K (K is an integer not less than 2), and the granularity of the K classification levels is arranged in order from coarse to fine, the process of traversing any labeled image includes: searching the existing root nodes of the label tree for whether there is a root node for storing the classification label corresponding to any labeled image at the first classification level (that is, the coarsest classification level).

[0165] If there is no root node for storing the classification label corresponding to any labeled image at the first classification level, then create a root node for storing the classification label corresponding to any labeled image at the first classification level, and then create a child node of the root node for storing the classification label corresponding to any labeled image at the second classification level, and then create a child node of the child node for storing the classification label corresponding to any labeled image at the third classification level, and so on. After creating a child node for storing the classification label corresponding to any labeled image at the Kth classification level (that is, the finest classification level), create a child node of the child node for storing the identification information of any labeled image (the child node is a leaf node of the label tree).

[0166] If there is a root node for storing the classification label corresponding to any tagged image at the first classification level, then find out whether there is a child node for storing the classification label corresponding to any tagged image at the second classification level in the child nodes of the root node. If there is no child node for storing the classification label corresponding to any tagged image at the second classification level, then create a child node of the root node for storing the classification label corresponding to any tagged image at the second classification level. For the subsequent process, refer to the relevant process in the case where there is no root node for storing the classification label corresponding to any tagged image at the first classification level, which will not be repeated here.

[0167] If there is a child node for storing the classification label corresponding to any tagged image at the second classification level, then find out whether there is a child node for storing the classification label corresponding to any tagged image at the third classification level in the child node of the child node. By analogy with the above-mentioned process, until it is determined that the classification labels corresponding to any tagged image at each classification level and the identification information of any tagged image are stored in the label tree, the process of traversing any tagged image is completed.

[0168] After the process of traversing each labeled image is completed, a label tree is obtained. In an exemplary embodiment, after the label tree is constructed, the label tree can also be updated according to the classification labels corresponding to the new labeled images at at least two classification levels.

[0169] For example, taking at least two classification levels in the order of fine to coarse granularity as species, family, and order, the label tree is as follows: Figure 6 shown. Figure 6 The label tree shown is used to store the classification labels corresponding to the four annotated images related to birds at the three classification levels of species, family, and order. Figure 6 In the label tree shown in FIG. 1 , the classification labels corresponding to each of the four bird-related annotated images at the three classification levels of species, family, and order can be extracted. Figure 6 In the label tree shown, it is possible to extract that the corresponding classification labels of the first image from left to right at the three classification levels of species, family, and order are "Rock Wren", "Pteridae", and "Passerines", respectively.

[0170] In step 502, the image classification model is called to obtain classification results corresponding to the sample image at at least two classification levels. The classification result corresponding to the sample image at the first classification level is obtained based on the classification result corresponding to the sample image at the associated level and the association information between the first classification level and the associated level.

[0171] The first classification level is any classification level among at least two classification levels that meets the selection condition, and the associated level is a classification level among at least two classification levels that meets the association condition with the first classification level.

[0172] After obtaining the sample image, the image classification model is called to obtain the classification results corresponding to the sample image at at least two classification levels. The classification result corresponding to the sample image at any classification level includes the matching probability of the sample image and each candidate category corresponding to the classification level.

[0173] In an exemplary embodiment, different classification levels have different granularities, and the associated level is a classification level that is located a reference number of digits before the first classification level in the order of granularity from fine to coarse.

[0174] In an exemplary embodiment, the reference number is one, and the classification level that meets the selection condition is a classification level other than the classification level with the finest granularity among at least two classification levels. That is, the first classification level is any classification level other than the classification level with the finest granularity among at least two classification levels.

[0175] Exemplarily, the process of calling the image classification model to obtain the classification results corresponding to the sample image at at least two classification levels is: calling the image classification model in the order of the granularity of at least two classification levels from fine to coarse, and obtaining the classification results corresponding to the sample image at at least two classification levels in sequence.

[0176] In an exemplary embodiment, the process of calling an image classification model to obtain classification results corresponding to a sample image at at least two classification levels includes: calling the image classification model to perform feature extraction on the sample image to obtain image features corresponding to the sample image at at least two classification levels; based on the image features corresponding to the sample image at at least two classification levels, obtaining classification results corresponding to the sample image at at least two classification levels.

[0177] In an exemplary embodiment, the process of calling an image classification model to obtain classification results corresponding to a sample image at at least two classification levels includes: calling the image classification model to perform feature extraction on the sample image to obtain common image features of the sample image; based on the common image features, obtaining classification results corresponding to the sample image at at least two classification levels.

[0178] Exemplarily, the process of obtaining the classification result corresponding to the sample image at the finest classification level includes: classifying the image features corresponding to the sample image at the finest classification level to obtain the classification result corresponding to the sample image at the finest classification level.

[0179] Exemplarily, the process of obtaining the classification result corresponding to the sample image at the first classification level includes: obtaining the association information between the first classification level and the association level based on the sample image; and determining the classification result corresponding to the sample image at the first classification level based on the classification result corresponding to the sample image at the association level and the association information between the first classification level and the association level. In other words, the association information between the first classification level and the association level is obtained based on the image features corresponding to the sample image at the first classification level.

[0180] Exemplarily, based on the sample image, the process of obtaining association information between the first classification level and the associated level includes: based on the sample image, obtaining the matching probability between the sample image and each candidate category corresponding to the first classification level, under the premise that the sample image belongs to any candidate category corresponding to the associated level; based on the matching probability between the sample image and each candidate category corresponding to the first classification level, obtaining association sub-information between the first classification level and any candidate category; based on the association sub-information between the first classification level and each candidate category corresponding to the associated level, obtaining association information between the first classification level and the associated level.

[0181] Exemplarily, based on the sample image, the process of obtaining the matching probability of the sample image with each candidate category corresponding to the first classification level under the premise that the sample image belongs to any candidate category corresponding to the associated level includes: performing feature extraction on the sample image to obtain image features corresponding to the sample image at the first classification level; based on the image features corresponding to the sample image at the first classification level, obtaining the matching probability of the sample image with each candidate category corresponding to the first classification level under the premise that the sample image belongs to any candidate category corresponding to the associated level.

[0182] Exemplarily, based on the matching probabilities between the sample image and each candidate category corresponding to the first classification level, the process of obtaining associated sub-information between the first classification level and any candidate category includes: normalizing the matching probabilities between the sample image and each candidate category corresponding to the first classification level to obtain the normalized matching probabilities between the sample image and each candidate category corresponding to the first classification level; and obtaining associated sub-information between the first classification level and any candidate category based on the normalized matching probabilities between the sample image and each candidate category corresponding to the first classification level.

[0183] The implementation of the relevant process in step 502 is shown in Figure 2 Step 202 in the illustrated embodiment will not be described in detail here.

[0184] It should be noted that, in the case where there are multiple sample images, the implementation process of step 502 is: calling the image classification model to obtain the classification results corresponding to each sample image at at least two classification levels.

[0185] In step 503, based on the classification results corresponding to the sample images at at least two classification levels and the classification labels corresponding to the sample images at at least two classification levels, a target loss is obtained; the image classification model is trained using the target loss to obtain a target image classification model.

[0186] After obtaining the classification results corresponding to the sample images at at least two classification levels, a target loss is obtained based on the classification results corresponding to the sample images at at least two classification levels and the classification labels corresponding to the sample images at at least two classification levels. The target loss is the loss based on which the parameters of the image classification model are updated.

[0187] In one possible implementation, based on the classification results corresponding to the sample images at at least two classification levels and the classification labels corresponding to the sample images at at least two classification levels, the process of obtaining the target loss includes: obtaining a first loss based on the classification results corresponding to the sample images at at least two classification levels and the classification labels corresponding to the sample images at at least two classification levels; obtaining a target loss based on the first loss.

[0188] The first loss is used to measure the difference between the classification results corresponding to the sample image at at least two classification levels and the classification labels corresponding to the sample image at at least two classification levels. Exemplarily, the smaller the first loss is, the closer the classification results corresponding to the sample image at at least two classification levels predicted by the image classification model are to the classification labels corresponding to the sample image at at least two classification levels, which means that the classification effect of the image classification model is better. In other words, the optimization goal of the first loss is to minimize it.

[0189] Exemplarily, based on the classification results corresponding to the sample image at at least two classification levels and the classification labels corresponding to the sample image at at least two classification levels, the process of obtaining the first loss includes: obtaining the first sub-loss corresponding to any classification level based on the classification results corresponding to the sample image at any classification level and the classification label corresponding to the sample image at the any classification level; obtaining the first loss based on the first sub-losses corresponding to each classification level.

[0190] Exemplarily, the first sub-loss corresponding to any classification level refers to the cross entropy loss between the classification result corresponding to the sample image at any classification level and the classification label corresponding to the sample image at any classification level, or the mean square error loss, etc.

[0191] Exemplarily, for the case where the number of sample images is one, based on the first sub-losses corresponding to each classification level, the method of obtaining the first loss includes: taking the sum (or weighted sum) of the first sub-losses corresponding to each classification level obtained on the basis of the one sample image as the first loss.

[0192] Exemplarily, in the case where there are multiple sample images, based on the first sub-losses corresponding to each classification level, the method of obtaining the first loss includes: adding the sum (or weighted sum) of the first sub-losses corresponding to each classification level obtained on the basis of any sample image to the classification loss corresponding to the any sample image; and taking the sum (or weighted sum) of the classification losses corresponding to each sample image as the first loss.

[0193] Exemplarily, taking the number of sample images as N (N is an integer not less than 1) as an example, the first loss is calculated based on Formula 6:

[0194]

[0195] Among them, L cE represents the first loss; N represents the number of sample images, that is, the number of sample images based on which the parameters of the image classification model are updated once, which can be called batch size; K (K is an integer not less than 2) represents the number of classification levels; Indicates the classification result corresponding to the nth sample image at the kth classification level; represents the classification label corresponding to the nth sample image at the kth classification level; GE(,) represents the cross entropy loss function. Where n is an integer not less than 1 and not greater than N, and k is an integer not less than 1 and not greater than K.

[0196] For example, it is assumed that the K classification levels are arranged in order from fine to coarse granularity, and the classification levels that meet the selection conditions are the classification levels except the classification level with the finest granularity. Since the classification results corresponding to the sample images at other classification levels except the classification level with the finest granularity can be determined based on the classification results corresponding to the sample images at the associated levels of other classification levels and the association information between the other classification levels and the associated levels, Formula 6 can also be expressed using the following Formula 7:

[0197]

[0198] in, Indicates the classification result corresponding to the nth sample image at the finest classification level; Indicates the classification label corresponding to the nth sample image at the finest classification level; represents the association information between the k-th classification level obtained based on the n-th sample image and the associated level of the k-th classification level; Indicates the classification result corresponding to the nth sample image at the (k-1)th classification level; Indicates the classification result corresponding to the nth sample image at the kth classification level; Indicates the classification label corresponding to the nth sample image at the kth classification level. Where n is an integer not less than 1 and not greater than N, and k is an integer not less than 2 and not greater than K. For the meaning of other parameters, see Formula 6.

[0199] After obtaining the first loss, the target loss is obtained based on the first loss. In an exemplary embodiment, the method of obtaining the target loss based on the first loss includes: using the first loss as the target loss. In this way, the efficiency of obtaining the target loss is high.

[0200] In an exemplary embodiment, based on the first loss, the method of obtaining the target loss includes: determining the second loss corresponding to the first classification level based on the association information between the first classification level and the association level; and obtaining the target loss based on the first loss and the second losses corresponding to each classification level that meets the selection conditions.

[0201] The second loss corresponding to the first classification level is used to regularize the association information between the first classification level and the association level. Association information can model the relationship between different classification levels, but the relationship between different candidate categories corresponding to the same classification level has not been clearly modeled. Therefore, the association information can be constrained to establish a connection between the candidate categories corresponding to the same classification level. In an embodiment of the present application, by regularizing the association information, the image classification model is allowed to directly learn the association between the candidate categories corresponding to the same classification level, rather than pre-defining the relationship between the candidate categories corresponding to the same classification level, which is more effective.

[0202] Exemplarily, based on the association information between the first classification level and the association level, the process of determining the second loss corresponding to the first classification level includes: taking the information entropy brought by the association information between the first classification level and the association level as the second loss corresponding to the first classification level.

[0203] Exemplarily, taking the case where the first classification level is the kth classification level (k is an integer not less than 2 and not greater than K), and the association information between the first classification level and the association level is represented by a matrix (that is, the association information is an association matrix), the second loss corresponding to the first classification level is calculated based on Formula 8:

[0204]

[0205] Among them, l k represents the second loss corresponding to the k-th classification level; C k-1 represents the number of columns of the association matrix between the kth classification level and the association level obtained based on the sample image x; It represents the column vector consisting of the j-th matrix elements in the association matrix between the k-th classification level and the association level obtained based on the sample image x; <,> represents the dot product operation of two vectors.

[0206] Exemplarily, referring to the method of obtaining the second loss corresponding to the first classification level, the second loss corresponding to each classification level that meets the selection condition can be obtained. Then, based on the first loss and the second loss corresponding to each classification level that meets the selection condition, the target loss is obtained.

[0207] Exemplarily, based on the first loss and the second loss corresponding to each classification level that meets the selection conditions, the process of obtaining the target loss is: based on the second loss corresponding to each classification level that meets the selection conditions, obtaining the third loss; based on the first loss and the third loss, obtaining the target loss.

[0208] Exemplarily, for the case where the number of sample images is one, based on the second losses corresponding to each classification level that meets the selection conditions, the process of obtaining the third loss includes: taking the sum (or weighted sum) of the second losses corresponding to each classification level that meets the selection conditions obtained based on the one sample image as the third loss.

[0209] Exemplarily, in the case where there are multiple sample images, based on the second losses corresponding to each classification level that meets the selection conditions, the process of obtaining the third loss includes: taking the sum (or weighted sum) of the second losses corresponding to each classification level that meets the selection conditions obtained on the basis of any sample image as the level loss corresponding to the any sample image; taking the sum (or weighted sum) of the level losses corresponding to each sample image as the third loss.

[0210] Exemplarily, taking the case where each classification level that meets the selection condition is each classification level except the finest-grained classification level and the number of sample images is N (N is an integer not less than 1) as an example, the third loss is calculated based on Formula 9:

[0211]

[0212] Among them, L Conf Indicates the third loss; Represents a column vector consisting of the jth column matrix elements in the association matrix between the kth classification level and the association level obtained based on the nth sample image. It should be noted that, for the case where the third loss is calculated based on Formula 9, the larger the third loss, the larger the information entropy brought by the association information, that is, the better the regularization constraint effect on the association information, that is, the optimization goal of the third loss is to maximize.

[0213] The third loss calculated based on Formula 9 can regularize the associated information, thereby preventing the associated information from degenerating into a trivial solution, and can also alleviate the problem of overconfident judgment caused by the first loss calculated based on the cross entropy loss function.

[0214] After obtaining the third loss, the target loss is obtained based on the third loss and the first loss. The embodiment of the present application does not limit the implementation method of obtaining the target loss based on the third loss and the first loss. Exemplarily, the optimization goal of the target loss is minimization.

[0215] For example, in the case where the first loss is calculated based on Formula 7 and the third loss is calculated based on Formula 9, the target loss is calculated based on Formula 10:

[0216] L=L CE +λL Conf (Formula 10)

[0217] Where L represents the target loss; L CE Indicates the first loss; L Conf represents the third loss; λ represents a hyperparameter used to balance the first loss and the third loss. Exemplarily, since the optimization goal of the third loss calculated based on Formula 9 is maximization, λ is a negative number to ensure that the optimization goal of the obtained target loss is minimization. Exemplarily, the first loss can also be called cross entropy loss, and the second loss can also be called confusion regularization loss.

[0218] After obtaining the target loss, the image classification model is trained using the target loss to obtain the target image classification model. Exemplarily, the process of training the image classification model using the target loss to obtain the target image classification model includes: using the target loss to reversely update the parameters of the image classification model once to obtain an updated image classification model; determining whether the updated image classification model meets the training termination condition; if the updated image classification model meets the training termination condition, using the updated image classification model as the target image classification model.

[0219] If the updated image classification model does not meet the training termination conditions, a new target loss is obtained based on steps 501 to 503, and the parameters of the updated image classification model are reversely updated using the new target loss, and so on, until an image classification model that meets the training termination conditions is obtained, and the image classification model that meets the training termination conditions is used as the target image classification model.

[0220] The training termination condition is set based on experience or flexibly adjusted according to the application scenario, and the embodiments of the present application do not limit this. In an exemplary embodiment, the image classification model satisfies the training termination condition, including but not limited to any of the following: the number of parameter updates (also referred to as the number of iterations) that have been executed when acquiring the image classification model reaches the maximum number of iterations, the target loss based on which the image classification model is acquired converges, and the target loss based on which the image classification model is acquired is less than the loss threshold. The maximum number of iterations and the loss threshold are set based on experience or flexibly adjusted according to the application scenario, and the embodiments of the present application do not limit this.

[0221] Exemplarily, taking the structure of an image classification model including a classification model and a transfer model as an example, the process of training the image classification model is implemented according to the following algorithm.

[0222] Input: Training data D, batch size N, max iteration T / / Input training data set D, batch size N, maximum number of iterations T

[0223] Output: Classification Network parametersωand Transition Network parametersθ / / Output classification model parameters ω and transition model parameters θ

[0224] 1: Initialize the parameters ω and θ. / / Initialize parameters ω and θ

[0225] 2: for t = 1 to T do / / for the case where the number of iterations is not greater than T

[0226] 3: / / Extract N training data from the training data set D, each training data includes a sample image x n And the label y corresponding to the sample image n

[0227] where / / Label y corresponding to the sample image n Including the classification labels corresponding to the sample images at K classification levels

[0228] 4:Forward to compute the prediction scores for the finest-grained classification, and compute the prediction scores of other coarser-grained according to Eq.(5). / / Get the classification results of the sample image at the finest-grained classification level, and get the classification results of the sample image at other classification levels except the finest-grained classification level based on formula 5

[0229] 5:Updateωandθaccording to Eq.(10). / / Use the target loss calculated based on formula 10 to update parameters ω and θ

[0230] 6:end for / / End

[0231] Return:optimal parametersω*andθ* / / Return the optimized parametersω* andθ*

[0232] The image classification model composed of the classification model with the above-mentioned optimized parameter ω* and the transfer model with the above-mentioned optimized parameter θ* is the trained target image classification model.

[0233] After obtaining the target image classification model, the target image classification model can be called to obtain the classification results corresponding to the target image to be structured classified at at least two classification levels. For the implementation process, see Figure 2 The embodiments shown will not be described in detail here.

[0234] In the embodiment of the present application, a deep learning structured classification method based on an association matrix (or transfer matrix) is proposed. The image classification model of the method includes two parts, the first part is a classification model, and the second part is a transfer model. First, the classification model is used to classify at the most fine-grained classification level, and then the association matrix between the next classification level and the association level is gradually learned according to the structure of the classification level. The classification result corresponding to the image at the previous classification level and the association matrix are combined to obtain the classification result corresponding to the image at the next classification level. The association matrix can clearly model the association relationship between different classification levels, so that the model can better guide the learning of data sets with structured labels. In addition, the embodiment of the present application proposes a regular constraint term of confusion loss to the association matrix, so that the model can further learn the relationship between candidate categories corresponding to the same classification level.

[0235] The method provided in the embodiment of the present application can be applied to the classification of various data sets with structured categories, such as various natural image data sets for biological classification, medical image data sets for medical classification, etc. Based on the method provided in the embodiment of the present application, the structured information in the data set can be fully utilized, and the categories attributable at different classification levels can be determined more accurately, while reducing the severity of prediction errors.

[0236] In the training method of the image classification model provided in the embodiment of the present application, the classification result corresponding to the sample image at the first classification level is determined based on the classification result corresponding to the sample image at the associated level of the first classification level and the association information between the first classification level and the associated level. That is to say, in the process of obtaining the classification result, not only the sample image itself but also the association information between the classification levels is considered. The considered information is relatively rich, which is conducive to improving the reliability of the classification result, thereby improving the training effect of the image classification model.

[0237] Exemplarily, the image classification method provided in the embodiments of the present application can be applied in the medical field. In the medical field, the target image refers to an image used for medical classification, and the image used for medical classification can also be called a medical image; at least two classification levels refer to at least two medical classification levels.

[0238] In this case, the image classification method includes: obtaining a medical image; based on the medical image, obtaining association information between the first medical classification level and the associated level; based on the classification results corresponding to the medical image at the associated level and the associated information, obtaining the classification results corresponding to the medical image at the first medical classification level.

[0239] The first medical classification level is any medical classification level among at least two medical classification levels that meets the selection condition, and the associated level is a medical classification level among at least two medical classification levels that meets the association condition with the first medical classification level.

[0240] The implementation process of the above image classification method can be found in Figure 2 The embodiments shown will not be described in detail here.

[0241] Exemplarily, the image classification method provided in the embodiments of the present application can be applied in the field of biology. In the field of biology, the target image refers to an image used for biological classification, and the image used for biological classification can also be called a biological image; at least two classification levels refer to at least two biological classification levels.

[0242] In this case, the image classification method includes: obtaining a biological image; based on the biological image, obtaining association information between a first biological classification level and an association level; based on the classification result corresponding to the biological image at the association level and the association information, obtaining a classification result corresponding to the biological image at the first biological classification level.

[0243] The first biological classification level is any biological classification level among at least two biological classification levels that meets the selection condition, and the associated level is a biological classification level among at least two biological classification levels that meets the association condition with the first biological classification level.

[0244] The implementation process of the above image classification method can be found in Figure 2 The embodiments shown will not be described in detail here.

[0245] See also Figure 7 , an embodiment of the present application provides an image classification device, the device comprising:

[0246] A first acquisition unit 701 is used to acquire a target image;

[0247] The second acquisition unit 702 is used to acquire the association information between the first classification level and the association level based on the target image; based on the classification result corresponding to the target image at the association level and the association information, acquire the classification result corresponding to the target image at the first classification level;

[0248] The first classification level is any classification level among at least two classification levels that meets the selection condition, and the associated level is a classification level among at least two classification levels that meets the association condition with the first classification level.

[0249] In one possible implementation, the second acquisition unit 702 is used to obtain, based on the target image, the matching probability of the target image and each candidate category corresponding to the first classification level, under the premise that the target image belongs to any candidate category corresponding to the association level; obtain the associated sub-information between the first classification level and any candidate category based on the matching probability of the target image and each candidate category corresponding to the first classification level; obtain the associated information between the first classification level and the association level based on the associated sub-information between the first classification level and each candidate category corresponding to the association level.

[0250] In one possible implementation, the second acquisition unit 702 is used to normalize the matching probability between the target image and each candidate category corresponding to the first classification level to obtain the normalized matching probability between the target image and each candidate category corresponding to the first classification level; based on the normalized matching probability between the target image and each candidate category corresponding to the first classification level, obtain the associated sub-information between the first classification level and any candidate category.

[0251] In one possible implementation, the second acquisition unit 702 is used to perform feature extraction on the target image to obtain image features corresponding to the target image at the first classification level; based on the image features corresponding to the target image at the first classification level, obtain the matching probability of the target image with each candidate category corresponding to the first classification level under the premise that the target image belongs to any candidate category.

[0252] In a possible implementation, different classification levels have different granularities, and the classification level that satisfies the association condition with the first classification level is the classification level that is located a reference number of digits before the first classification level in the order of granularity from fine to coarse.

[0253] In one possible implementation, the reference number is one, and the classification level that meets the selection conditions is a classification level other than the classification level with the finest granularity among at least two classification levels. The second acquisition unit 702 is also used to obtain the classification results corresponding to the target image at at least two classification levels in order from fine to coarse granularity of the at least two classification levels.

[0254] In a possible implementation, the second acquisition unit 702 is further configured to classify image features corresponding to the target image at the finest classification level to obtain a classification result corresponding to the target image at the finest classification level.

[0255] In one possible implementation, the second acquisition unit 702 is used to call the target image classification model to obtain the association information between the first classification level and the association level based on the target image; and call the target image classification model to obtain the classification result corresponding to the target image at the first classification level based on the classification result and association information corresponding to the target image at the association level.

[0256] In one possible implementation, the target image is an image for biological classification, and the at least two classification levels are at least two biological classification levels; or, the target image is an image for medical classification, and the at least two classification levels are at least two medical classification levels.

[0257] The image classification device provided in the embodiment of the present application, in the process of obtaining the classification results corresponding to the target image at the first classification level, not only considers the image itself, but also considers the association information between the first classification level and the association level. The considered information is relatively rich, which is conducive to improving the reliability of the classification results, and thus improving the accuracy of image classification.

[0258] See also Figure 8 , an embodiment of the present application provides a training device for an image classification model, the device comprising:

[0259] A first acquisition unit 801 is used to acquire a sample image and classification labels corresponding to the sample image at at least two classification levels;

[0260] A second acquisition unit 802 is used to call the image classification model to obtain classification results corresponding to the sample image at at least two classification levels, wherein the classification result corresponding to the sample image at the first classification level is obtained based on the classification result corresponding to the sample image at the association level and the association information between the first classification level and the association level;

[0261] A third acquisition unit 803 is used to acquire a target loss based on the classification results corresponding to the sample image at at least two classification levels and the classification labels corresponding to the sample image at at least two classification levels;

[0262] A training unit 804 is used to train the image classification model using the target loss to obtain a target image classification model;

[0263] The first classification level is any classification level among at least two classification levels that meets the selection condition, and the associated level is a classification level among at least two classification levels that meets the association condition with the first classification level.

[0264] In one possible implementation, the third acquisition unit 803 is used to obtain a first loss based on the classification results corresponding to the sample image at at least two classification levels and the classification labels corresponding to the sample image at at least two classification levels; obtain a second loss corresponding to the first classification level based on the association information between the first classification level and the association level; and obtain a target loss based on the first loss and the second losses corresponding to each classification level that meets the selection conditions.

[0265] In a possible implementation, classification labels corresponding to the sample images at at least two classification levels are extracted from a label tree; the device further includes:

[0266] The construction unit is used to construct a label tree based on the arrangement order of the granularity of at least two classification levels and the classification labels corresponding to each labeled image at at least two classification levels, and each labeled image includes a sample image.

[0267] In the training device for the image classification model provided by the embodiment of the present application, the classification result corresponding to the sample image at the first classification level is determined based on the classification result corresponding to the sample image at the associated level of the first classification level and the association information between the first classification level and the associated level. That is to say, in the process of obtaining the classification result, not only the sample image itself but also the association information between the classification levels is considered. The considered information is relatively rich, which is conducive to improving the reliability of the classification result, thereby improving the training effect of the image classification model.

[0268] It should be noted that the device provided in the above embodiment only uses the division of the above functional units as an example to implement its functions. In actual applications, the above functions can be assigned to different functional units as needed, that is, the internal structure of the device is divided into different functional units to complete all or part of the functions described above. In addition, the device and method embodiments provided in the above embodiment belong to the same concept, and the specific implementation process is detailed in the method embodiment, which will not be repeated here.

[0269] In an exemplary embodiment, a computer device is also provided, the computer device includes a processor and a memory, and at least one computer program is stored in the memory. The at least one computer program is loaded and executed by one or more processors, so that the computer device implements any of the above-mentioned image classification methods or image classification model training methods. The computer device can be a server or a terminal. Next, the structures of the server and the terminal are introduced respectively.

[0270] Fig. 9It is a structural diagram of a server provided in an embodiment of the present application. The server may have relatively large differences due to different configurations or performances, and may include one or more processors (Central Processing Units, CPU) 901 and one or more memories 902, wherein at least one computer program is stored in the one or more memories 902, and the at least one computer program is loaded and executed by the one or more processors 901, so that the server implements the image classification method or the training method of the image classification model provided in the above-mentioned various method embodiments. Of course, the server may also have components such as a wired or wireless network interface, a keyboard, and an input and output interface for input and output. The server may also include other components for implementing device functions, which will not be repeated here.

[0271] Fig.10 1 is a schematic diagram of a terminal structure provided by an embodiment of the present application. The terminal may be: a PC, a mobile phone, a smart phone, a PDA, a wearable device, a PPC, a tablet computer, a smart car machine, a smart TV, a smart speaker, a smart voice interaction device, a smart home appliance, or a vehicle-mounted terminal. The terminal may also be referred to as a user device, a portable terminal, a laptop terminal, a desktop terminal, or other names.

[0272] Typically, the terminal includes: a processor 1001 and a memory 1002 .

[0273] The processor 1001 may include one or more processing cores, such as a 4-core processor, an 8-core processor, etc. The processor 1001 may be implemented in at least one hardware form of DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), and PLA (Programmable Logic Array). The processor 1001 may also include a main processor and a coprocessor. The main processor is a processor for processing data in an awake state, also known as a CPU (Central Processing Unit); the coprocessor is a low-power processor for processing data in a standby state. In some embodiments, the processor 1001 may be integrated with a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the display screen. In some embodiments, the processor 1001 may also include an AI (Artificial Intelligence) processor, which is used to process computing operations related to machine learning.

[0274] The memory 1002 may include one or more computer-readable storage media, which may be non-transitory. The memory 1002 may also include a high-speed random access memory, and a non-volatile memory, such as one or more disk storage devices, flash memory storage devices. In some embodiments, the non-transitory computer-readable storage medium in the memory 1002 is used to store at least one instruction, which is used to be executed by the processor 1001 so that the terminal implements the image classification method or the image classification model training method provided in the method embodiment of the present application.

[0275] In some embodiments, the terminal may further optionally include: a peripheral device interface 1003 and at least one peripheral device. The processor 1001, the memory 1002 and the peripheral device interface 1003 may be connected via a bus or a signal line. Each peripheral device may be connected to the peripheral device interface 1003 via a bus, a signal line or a circuit board. Specifically, the peripheral device includes: at least one of a radio frequency circuit 1004, a display screen 1005, a camera assembly 1006, an audio circuit 1007 and a power supply 1008.

[0276] The peripheral device interface 1003 may be used to connect at least one peripheral device related to I / O (Input / Output) to the processor 1001 and the memory 1002. In some embodiments, the processor 1001, the memory 1002, and the peripheral device interface 1003 are integrated on the same chip or circuit board; in some other embodiments, any one or two of the processor 1001, the memory 1002, and the peripheral device interface 1003 may be implemented on a separate chip or circuit board, which is not limited in this embodiment.

[0277] The radio frequency circuit 1004 is used to receive and transmit RF (Radio Frequency) signals, also known as electromagnetic signals. The radio frequency circuit 1004 communicates with the communication network and other communication devices through electromagnetic signals. The radio frequency circuit 1004 converts the electrical signal into an electromagnetic signal for transmission, or converts the received electromagnetic signal into an electrical signal. Optionally, the radio frequency circuit 1004 includes: an antenna system, an RF transceiver, one or more amplifiers, a tuner, an oscillator, a digital signal processor, a codec chipset, a user identity module card, etc. The radio frequency circuit 1004 can communicate with other terminals through at least one wireless communication protocol. The wireless communication protocol includes but is not limited to: a metropolitan area network, various generations of mobile communication networks (2G, 3G, 4G and 5G), a wireless local area network and / or a WiFi (Wireless Fidelity) network. In some embodiments, the radio frequency circuit 1004 may also include circuits related to NFC (Near Field Communication), which is not limited in this application.

[0278] The display screen 1005 is used to display a UI (User Interface). The UI may include graphics, text, icons, videos, and any combination thereof. When the display screen 1005 is a touch display screen, the display screen 1005 also has the ability to collect touch signals on the surface or above the surface of the display screen 1005. The touch signal can be input to the processor 1001 as a control signal for processing. At this time, the display screen 1005 can also be used to provide virtual buttons and / or virtual keyboards, also known as soft buttons and / or soft keyboards. In some embodiments, the display screen 1005 can be one, which is set on the front panel of the terminal; in other embodiments, the display screen 1005 can be at least two, which are respectively set on different surfaces of the terminal or are folded; in other embodiments, the display screen 1005 can be a flexible display screen, which is set on the curved surface or folded surface of the terminal. Even, the display screen 1005 can also be set to a non-rectangular irregular shape, that is, a special-shaped screen. The display screen 1005 can be made of materials such as LCD (Liquid Crystal Display), OLED (Organic Light-Emitting Diode, organic light-emitting diode).

[0279] The camera assembly 1006 is used to capture images or videos. Optionally, the camera assembly 1006 includes a front camera and a rear camera. Typically, the front camera is arranged on the front panel of the terminal, and the rear camera is arranged on the back of the terminal. In some embodiments, there are at least two rear cameras, which are any one of a main camera, a depth of field camera, a wide-angle camera, and a telephoto camera, so as to realize the fusion of the main camera and the depth of field camera to realize the background blur function, the fusion of the main camera and the wide-angle camera to realize the panoramic shooting and VR (Virtual Reality) shooting function or other fusion shooting functions. In some embodiments, the camera assembly 1006 may also include a flash. The flash can be a monochrome temperature flash or a dual-color temperature flash. A dual-color temperature flash refers to a combination of a warm light flash and a cold light flash, which can be used for light compensation at different color temperatures.

[0280] The audio circuit 1007 may include a microphone and a speaker. The microphone is used to collect sound waves from the user and the environment, and convert the sound waves into electrical signals and input them into the processor 1001 for processing, or input them into the RF circuit 1004 to achieve voice communication. For the purpose of stereo acquisition or noise reduction, there may be multiple microphones, which are respectively arranged at different parts of the terminal. The microphone may also be an array microphone or an omnidirectional acquisition microphone. The speaker is used to convert the electrical signal from the processor 1001 or the RF circuit 1004 into sound waves. The speaker may be a traditional film speaker or a piezoelectric ceramic speaker. When the speaker is a piezoelectric ceramic speaker, it can not only convert the electrical signal into sound waves audible to humans, but also convert the electrical signal into sound waves inaudible to humans for purposes such as ranging. In some embodiments, the audio circuit 1007 may also include a headphone jack.

[0281] The power supply 1008 is used to power various components in the terminal. The power supply 1008 can be an alternating current, a direct current, a disposable battery, or a rechargeable battery. When the power supply 1008 includes a rechargeable battery, the rechargeable battery can support wired charging or wireless charging. The rechargeable battery can also be used to support fast charging technology.

[0282] In some embodiments, the terminal further includes one or more sensors 1009 , including but not limited to: an acceleration sensor 1010 , a gyroscope sensor 1011 , a pressure sensor 1012 , an optical sensor 1013 , and a proximity sensor 1014 .

[0283] The acceleration sensor 1010 can detect the magnitude of acceleration on the three coordinate axes of the coordinate system established by the terminal. For example, the acceleration sensor 1010 can be used to detect the components of gravity acceleration on the three coordinate axes. The processor 1001 can control the display screen 1005 to display the user interface in a horizontal view or a vertical view according to the gravity acceleration signal collected by the acceleration sensor 1010. The acceleration sensor 1010 can also be used for collecting game or user motion data.

[0284] The gyro sensor 1011 can detect the body direction and rotation angle of the terminal, and the gyro sensor 1011 can cooperate with the acceleration sensor 1010 to collect the user's 3D actions on the terminal. The processor 1001 can implement the following functions based on the data collected by the gyro sensor 1011: motion sensing (such as changing the UI according to the user's tilt operation), image stabilization during shooting, game control, and inertial navigation.

[0285] The pressure sensor 1012 can be set in the side frame of the terminal and / or the lower layer of the display screen 1005. When the pressure sensor 1012 is set in the side frame of the terminal, the user's holding signal of the terminal can be detected, and the processor 1001 performs left and right hand recognition or shortcut operation according to the holding signal collected by the pressure sensor 1012. When the pressure sensor 1012 is set in the lower layer of the display screen 1005, the processor 1001 controls the operability controls on the UI interface according to the user's pressure operation on the display screen 1005. The operability controls include at least one of a button control, a scroll bar control, an icon control, and a menu control.

[0286] The optical sensor 1013 is used to collect the ambient light intensity. In one embodiment, the processor 1001 can control the display brightness of the display screen 1005 according to the ambient light intensity collected by the optical sensor 1013. Specifically, when the ambient light intensity is high, the display brightness of the display screen 1005 is increased; when the ambient light intensity is low, the display brightness of the display screen 1005 is reduced. In another embodiment, the processor 1001 can also dynamically adjust the shooting parameters of the camera assembly 1006 according to the ambient light intensity collected by the optical sensor 1013.

[0287] The proximity sensor 1014, also known as a distance sensor, is usually arranged on the front panel of the terminal. The proximity sensor 1014 is used to collect the distance between the user and the front of the terminal. In one embodiment, when the proximity sensor 1014 detects that the distance between the user and the front of the terminal is gradually decreasing, the processor 1001 controls the display screen 1005 to switch from the screen-on state to the screen-off state; when the proximity sensor 1014 detects that the distance between the user and the front of the terminal is gradually increasing, the processor 1001 controls the display screen 1005 to switch from the screen-off state to the screen-on state.

[0288] Those skilled in the art will understand that Fig.10 The structure shown in the figure does not constitute a limitation on the terminal, and may include more or less components than those shown in the figure, or combine certain components, or adopt a different component arrangement.

[0289] In an exemplary embodiment, a computer-readable storage medium is also provided, in which at least one computer program is stored. The at least one computer program is loaded and executed by a processor of a computer device so that the computer implements any of the above-mentioned image classification methods or image classification model training methods.

[0290] In one possible implementation, the computer-readable storage medium may be a read-only memory (ROM), a random access memory (RAM), a compact disc (CD-ROM), a magnetic tape, a floppy disk, an optical data storage device, etc.

[0291] In an exemplary embodiment, a computer program product is also provided, which includes a computer program or computer instructions, which are loaded and executed by a processor to enable a computer to implement any of the above-mentioned image classification methods or image classification model training methods.

[0292] It should be noted that the terms "first", "second", etc. in this application are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. The embodiments described in the above exemplary embodiments do not represent all embodiments consistent with the present application. On the contrary, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims.

[0293] It should be understood that the "plurality" mentioned in this article refers to two or more. "And / or" describes the association relationship of the associated objects, indicating that there can be three relationships. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the associated objects are in an "or" relationship.

[0294] The above description is only an exemplary embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent substitutions, improvements, etc. made within the principles of the present application shall be included in the protection scope of the present application.

Claims

1. An image classification method, characterized in that: The method comprises: Get the target image; Performing feature extraction on the target image to obtain image features corresponding to the target image at a first classification level, wherein the image features corresponding to the target image at different classification levels are different; obtaining association information between the first classification level and the association level based on the image features corresponding to the target image at the first classification level, wherein the association information includes association sub-information between the first classification level and each candidate category corresponding to the association level, and the association sub-information between the first classification level and any candidate category corresponding to the association level includes, under the premise that the target image belongs to any candidate category, a matching probability or a normalized matching probability between the target image and each candidate category corresponding to the first classification level; Based on the classification result corresponding to the target image at the association level and the association information, obtaining the classification result corresponding to the target image at the first classification level; Among them, the first classification level is any classification level among at least two classification levels that meets the selection conditions, different classification levels have different granularities, and the associated level is the classification level among the at least two classification levels that is located in the order of granularity from fine to coarse and is located a reference number of places before the first classification level.

2. The method according to claim 1, characterized in that The acquiring, based on the image features corresponding to the target image at the first classification level, association information between the first classification level and the association level includes: Based on the image features corresponding to the target image at the first classification level, obtaining, under the premise that the target image belongs to any candidate category corresponding to the association level, a matching probability between the target image and each candidate category corresponding to the first classification level; Based on the matching probability between the target image and each candidate category corresponding to the first classification level, acquiring the associated sub-information between the first classification level and any candidate category; Based on the associated sub-information between the first classification level and each candidate category corresponding to the associated level, the associated information between the first classification level and the associated level is acquired.

3. The method according to claim 2, characterized in that The acquiring, based on the matching probability between the target image and each candidate category corresponding to the first classification level, associated sub-information between the first classification level and any candidate category includes: Normalizing the matching probabilities between the target image and each candidate category corresponding to the first classification level to obtain normalized matching probabilities between the target image and each candidate category corresponding to the first classification level; Based on the normalized matching probability between the target image and each candidate category corresponding to the first classification level, the associated sub-information between the first classification level and any candidate category is obtained.

4. The method according to claim 1, characterized in that: The reference number is one, the classification level that meets the selection condition is a classification level other than the classification level with the finest granularity among the at least two classification levels, and the method further includes: According to the order of granularity of the at least two classification levels from fine to coarse, classification results corresponding to the target image at the at least two classification levels are obtained in sequence.

5. The method according to claim 4, characterized in that The step of sequentially obtaining classification results of the target image at the at least two classification levels respectively in the order of granularity of the at least two classification levels from fine to coarse includes: The image features corresponding to the target image at the finest classification level are classified to obtain a classification result corresponding to the target image at the finest classification level.

6. The method according to any one of claims 1 to 5, characterized in that: The acquiring, based on the image features corresponding to the target image at the first classification level, association information between the first classification level and the association level includes: Calling the target image classification model to obtain association information between the first classification level and the association level based on the image features corresponding to the target image at the first classification level; The acquiring, based on the classification result corresponding to the target image at the association level and the association information, the classification result corresponding to the target image at the first classification level includes: The target image classification model is called to obtain the classification result corresponding to the target image at the first classification level based on the classification result corresponding to the target image at the association level and the association information.

7. The method according to any one of claims 1 to 5, characterized in that: The target image is an image for biological classification, and the at least two classification levels are at least two biological classification levels; or, the target image is an image for medical classification, and the at least two classification levels are at least two medical classification levels.

8. A method for training an image classification model, characterized in that: The method comprises: Obtaining a sample image and classification labels corresponding to the sample image at at least two classification levels; Calling the image classification model to obtain the classification results corresponding to the sample image at the at least two classification levels, respectively, wherein the classification result corresponding to the sample image at the first classification level is obtained based on the classification result corresponding to the sample image at the association level and the association information between the first classification level and the association level; Based on the classification results corresponding to the sample image at the at least two classification levels and the classification labels corresponding to the sample image at the at least two classification levels, a target loss is obtained; and the image classification model is trained using the target loss to obtain a target image classification model; The first classification level is any classification level that meets the selection condition among the at least two classification levels, different classification levels have different granularities, and the associated level is a classification level that is located before the first classification level by a reference number of digits in the order of granularity from fine to coarse among the at least two classification levels; The association information is obtained based on the image features corresponding to the sample image at the first classification level, and the image features corresponding to the sample image at different classification levels are different; the association information includes association sub-information between the first classification level and the various candidate categories corresponding to the association level, and the association sub-information between the first classification level and any candidate category corresponding to the association level includes, under the premise that the sample image belongs to any candidate category, the matching probability or normalized matching probability of the sample image and each candidate category corresponding to the first classification level.

9. The method according to claim 8, characterized in that The acquiring the target loss based on the classification results corresponding to the sample image at the at least two classification levels and the classification labels corresponding to the sample image at the at least two classification levels, comprises: Obtaining a first loss based on the classification results corresponding to the sample image at the at least two classification levels and the classification labels corresponding to the sample image at the at least two classification levels; Based on the association information between the first classification level and the association level, obtaining a second loss corresponding to the first classification level; The target loss is obtained based on the first loss and the second losses corresponding to each classification level that meets the selection condition.

10. The method according to claim 8 or 9, characterized in that: The classification labels corresponding to the sample images at at least two classification levels are extracted from the label tree; the method further includes: The label tree is constructed based on the arrangement order of the granularity of the at least two classification levels and the classification labels corresponding to each labeled image at the at least two classification levels, and each labeled image includes the sample image.

11. An image classification device, characterized in that: The device comprises: A first acquisition unit, used for acquiring a target image; A second acquisition unit is used to perform feature extraction on the target image to obtain image features corresponding to the target image at a first classification level, wherein the target image has different corresponding image features at different classification levels; based on the image features corresponding to the target image at the first classification level, obtain association information between the first classification level and the association level, wherein the association information includes association sub-information between the first classification level and each candidate category corresponding to the association level, and the association sub-information between the first classification level and any candidate category corresponding to the association level includes, under the premise that the target image belongs to any candidate category, the matching probability or normalized matching probability between the target image and each candidate category corresponding to the first classification level; based on the classification result corresponding to the target image at the association level and the association information, obtain the classification result corresponding to the target image at the first classification level; Among them, the first classification level is any classification level among at least two classification levels that meets the selection conditions, different classification levels have different granularities, and the associated level is the classification level among the at least two classification levels that is located in the order of granularity from fine to coarse and is located a reference number of places before the first classification level.

12. A training device for an image classification model, characterized in that: The device comprises: A first acquisition unit, used to acquire a sample image and classification labels corresponding to the sample image at at least two classification levels; A second acquisition unit is used to call the image classification model to obtain the classification results corresponding to the sample image at the at least two classification levels, wherein the classification result corresponding to the sample image at the first classification level is obtained based on the classification result corresponding to the sample image at the association level and the association information between the first classification level and the association level; A third acquisition unit, configured to acquire a target loss based on the classification results corresponding to the sample image at the at least two classification levels and the classification labels corresponding to the sample image at the at least two classification levels; A training unit, used to train the image classification model using the target loss to obtain a target image classification model; The first classification level is any classification level that meets the selection condition among the at least two classification levels, different classification levels have different granularities, and the associated level is a classification level that is located before the first classification level by a reference number of digits in the order of granularity from fine to coarse among the at least two classification levels; The association information is obtained based on the image features corresponding to the sample image at the first classification level, and the image features corresponding to the sample image at different classification levels are different; the association information includes association sub-information between the first classification level and the various candidate categories corresponding to the association level, and the association sub-information between the first classification level and any candidate category corresponding to the association level includes, under the premise that the sample image belongs to any candidate category, the matching probability or normalized matching probability of the sample image and each candidate category corresponding to the first classification level.

13. A computer device, characterized in that: The computer device includes a processor and a memory, wherein at least one computer program is stored in the memory, and the at least one computer program is loaded and executed by the processor so that the computer device implements the image classification method as described in any one of claims 1 to 7, or the training method of the image classification model as described in any one of claims 8 to 10.

14. A computer-readable storage medium, characterized in that: At least one computer program is stored in the computer-readable storage medium, and the at least one computer program is loaded and executed by the processor so that the computer implements the image classification method as described in any one of claims 1 to 7, or the training method of the image classification model as described in any one of claims 8 to 10.

15. A computer program product, characterized in that The computer program product includes a computer program or computer instructions, which are loaded and executed by a processor so that a computer implements the image classification method as described in any one of claims 1 to 7, or the training method of the image classification model as described in any one of claims 8 to 10.

Citation Information

Patent Citations

  • Image classification network training method, image classification method and device, and server

    CN109685110A

  • Object category recognition model construction method and application based on granularity and associated information

    CN111814554A

  • Image segmentation

    CN112088380A

  • Image processing model training method, image processing model processing method, image processing model training device, image processing model processing device, equipment and medium

    CN113569895A

  • Image recognition model training method and device, electronic equipment and storage medium

    CN113887581A