Multi-granularity image recognition method and device based on large model cooperation

Abstract

The disclosure provides a multi-granularity image recognition method and device based on large model cooperation, which can be applied to the technical field of deep learning image recognition. The method comprises: in response to an image recognition task for a target image, calling a thinking tree for image recognition; extracting a plurality of image features of the target image by using a feature extraction model, and performing feature fusion according to a granularity level to obtain multi-granularity fusion features; for each granularity level, predicting the classification confidence of the level by using the fusion features corresponding to the level; then determining the optimal classification path in the thinking tree to finally obtain the recognition result of the target image.

Description

Technical Field

[0001] This disclosure relates to the field of deep learning image recognition technology, and more specifically to a multi-granularity image recognition method and apparatus based on large model collaboration. Background Technology

[0002] With the rapid development of modern automation and intelligence, there is a need for accurate recognition and classification of complex visual images, enabling subsequent processing based on the classification results. However, traditional recognition and classification methods can only identify and classify simple images at a single granularity, making it difficult to correctly recognize multi-granularity images.

[0003] In the process of realizing the above-mentioned inventive concept, it was found through research that when faced with multiple features of multi-granularity images, existing recognition and classification methods are difficult to extract high-precision fine-grained features from complex images, resulting in difficulties in performing high-precision recognition and classification of complex images, as well as poor adaptability and flexibility in complex environments. Summary of the Invention

[0004] In view of the above problems, this disclosure provides a multi-granularity image recognition method and apparatus based on large model collaboration.

[0005] According to the first aspect of this disclosure, a multi-granularity image recognition method based on large-model collaboration is provided, comprising: responding to an image recognition task for a target image, invoking a mind tree for image recognition, wherein the mind tree is obtained by classifying a pre-defined image dataset using a large-scale language model, the mind tree includes an image classification root node and at least one image category node with hierarchical relationship, the mind tree includes M levels, M≥2; extracting image extraction features at M granularity levels for the target image using a feature extraction model, and fusing the M image extraction features according to the granularity level to obtain a multi-granularity image recognition method with M levels. Each level has its own corresponding fusion features; for the i-th level among the M levels, using the image category node corresponding to the i-th level and the fusion features corresponding to the i-th level, the classification confidence corresponding to each image category node contained in the i-th level is predicted, where the classification confidence represents the probability that the image category of the target image belongs to the image category node, 1≤i≤M. Based on the classification confidence corresponding to each image category node in the mind tree, the optimal classification path in the mind tree is determined, and the recognition result for the target image is generated based on the M image category nodes contained in the optimal classification path, where the recognition result includes M image categories.

[0006] The second aspect of this disclosure provides a multi-granularity image recognition device based on large-model collaboration, comprising: a mind tree module, used to invoke a mind tree for image recognition in response to an image recognition task for a target image, wherein the mind tree is obtained by classifying a preset image dataset using a large-scale language model, the mind tree includes an image classification root node and at least one image category node with hierarchical relationship, the mind tree includes M levels, M≥2; and a feature extraction module, used to extract image extraction features at M granularity levels for the target image using a feature extraction model, and to perform feature fusion on the M image extraction features according to the granularity level to obtain features corresponding to each of the M levels. The system comprises the following modules: a feature fusion module, which, for the i-th level among M levels, uses the image category node corresponding to the i-th level and the fusion feature corresponding to the i-th level to predict the classification confidence of each image category node contained in the i-th level, where the classification confidence represents the probability that the image category of the target image belongs to the image category node, 1≤i≤M; and a classification path determination module, which determines the optimal classification path in the mind tree based on the classification confidence of each image category node, and generates a recognition result for the target image based on the M image category nodes contained in the optimal classification path, where the recognition result includes M image categories.

[0007] A third aspect of this disclosure provides an electronic device comprising: one or more processors; and a memory for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors perform the methods described above.

[0008] A fourth aspect of this disclosure also provides a computer-readable storage medium having executable instructions stored thereon, which, when executed by a processor, cause the processor to perform the methods described above.

[0009] The fifth aspect of this disclosure also provides a computer program product, including a computer program that, when executed by a processor, implements the above-described method.

[0010] According to the multi-granularity image recognition method and apparatus based on large model collaboration disclosed herein, a pre-defined image dataset is first classified using a language large model to generate a mind tree containing M levels, multiple root nodes, and multiple image category nodes. This mind tree provides high-precision granularity information and multiple classification paths for subsequent image classification and recognition, and assists in the feature extraction and feature fusion processes of the supervised feature extraction model. Then, in response to an image recognition task for a target image, the mind tree for image recognition is invoked. Based on the M levels of the mind tree, the feature extraction model extracts image features at M granularity levels for the target image. Based on the granularity level, the M image features are fused to obtain fused features corresponding to each of the M levels. This achieves multi-level, multi-dimensional, multi-granularity image feature extraction from the target image, from coarse to fine granularity. This allows for detailed classification results based on the extracted multi-granularity image features, combined with the mind tree, improving the accuracy of image classification and reducing classification risk. Then, for the i-th level among the M levels, using the image category node corresponding to the i-th level and the fusion feature corresponding to the i-th level, the classification confidence corresponding to each image category node contained in the i-th level is predicted. Based on the classification confidence corresponding to each image category node in the mind tree, the optimal classification path in the mind tree is determined, and the recognition result for the target image is generated. Thus, based on the confidence calculated from the fusion feature, high-accuracy and fast feature recognition is achieved, especially in complex task environments, for simple images and high-fine-grained images, improving the calculation speed and application effectiveness, and providing classification information containing multiple coarse and fine granularities to assist in judgment.

[0011] According to embodiments of this disclosure, by further combining the mind tree with a feature extraction model incorporating deep learning technology, mutual supervision and learning can be achieved during the image classification and recognition process. This allows the feature extraction model in the image recognition process to both reference and constrain the mind tree, further improving the interpretability of the image recognition process, enhancing the practicality and applicability of recognition, and improving the robustness and stability of the model. Attached Figure Description

[0012] The foregoing contents, as well as other objects, features, and advantages of this disclosure, will become clearer from the following description of embodiments with reference to the accompanying drawings, in which:

[0013] Figure 1 An application scenario diagram of the multi-granularity image recognition method based on large model collaboration according to an embodiment of the present disclosure is shown;

[0014] Figure 2 A flowchart of a multi-granularity image recognition method based on large model collaboration according to an embodiment of the present disclosure is shown;

[0015] Figure 3 A schematic diagram of an initial thought tree generated using a large language model according to an embodiment of the present disclosure is shown;

[0016] Figure 4 A schematic diagram illustrating feature fusion of image features extracted using a feature extraction model according to an embodiment of the present disclosure is shown;

[0017] Figure 5 A schematic diagram of a multi-granularity image recognition process based on large model collaboration according to an embodiment of the present disclosure is shown;

[0018] Figure 6 A structural block diagram of a multi-granularity image recognition device based on large model collaboration according to an embodiment of the present disclosure is shown;

[0019] Figure 7 A block diagram of an electronic device using a multi-granularity image recognition method based on large model collaboration according to an embodiment of the present disclosure is shown. Detailed Implementation

[0020] The embodiments of the present disclosure will now be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the disclosure. In the following detailed description, numerous specific details are set forth to provide a thorough understanding of the embodiments of the present disclosure for ease of explanation. However, it will be apparent that one or more embodiments may be practiced without these specific details. Furthermore, descriptions of well-known structures and techniques are omitted in the following description to avoid unnecessarily obscuring the concepts of the present disclosure.

[0021] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit this disclosure. The terms “comprising,” “including,” etc., as used herein indicate the presence of the stated features, steps, operations, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, or components.

[0022] All terms used herein (including technical and scientific terms) have the meanings commonly understood by those skilled in the art, unless otherwise defined. It should be noted that the terms used herein are to be interpreted in a manner consistent with the context of this specification, and not in an idealized or overly rigid way.

[0023] When using expressions such as "at least one of A, B and C", they should generally be interpreted in accordance with the meaning that is commonly understood by those skilled in the art (e.g., "a system having at least one of A, B and C" should include, but is not limited to, a system having A alone, a system having B alone, a system having C alone, a system having A and B, a system having A and C, a system having B and C, and / or a system having A, B and C, etc.).

[0024] In the technical solution disclosed herein, the user information (including but not limited to user personal information, user image information, user device information, such as location information) and data (including but not limited to data used for analysis, stored data, and displayed data) involved are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, storage, use, processing, transmission, provision, disclosure, and application of related data all comply with relevant laws, regulations, and standards, necessary confidentiality measures have been taken, and they do not violate public order and good morals. Corresponding operation entry points are provided for users to choose to authorize or refuse.

[0025] With the rapid development of modern automation and intelligence, machines are playing an increasingly important role in complex vision tasks, especially in image recognition and classification. Machines need to process and analyze large amounts of image data quickly and accurately. In multi-granularity image classification, machines need to understand and classify images at different levels of abstraction to adapt to the needs of various application scenarios. Traditional machine vision systems typically only perform image classification at a single granularity, limiting their adaptability and flexibility in complex environments. Furthermore, with the development of deep learning technology, multi-granularity image classification has begun to be introduced into machine vision systems. Because multi-granularity classification can provide continuous visual recognition information from coarse to fine, it helps machines to understand images more comprehensively.

[0026] Based on this, the inventors of this application further discovered that mind trees can simulate the thought process. By constructing a multi-level decision tree, the logic and hierarchy of thought are simulated. Each thought node represents an intermediate level of thinking, allowing the decision-making process to occur at multiple levels, providing machines with a deeper level of understanding and decision support. Combining mind trees with multi-granularity image classification can greatly improve the classification performance and adaptability of images, especially in the face of complex and dynamic environments.

[0027] During the research and development process, it was discovered that when faced with multiple features of multi-granularity images, existing recognition and classification methods struggle to extract high-precision fine-grained features from complex images, resulting in difficulties in performing high-precision recognition and classification of complex images, as well as poor adaptability and flexibility in complex environments.

[0028] In view of this, embodiments of this disclosure provide a multi-granularity image recognition method based on large-model collaboration, comprising: responding to an image recognition task for a target image, invoking a mind tree for image recognition, wherein the mind tree is obtained by classifying a preset image dataset using a large-scale language model, the mind tree includes an image classification root node and at least one image category node with hierarchical relationship, the mind tree includes M levels, M≥2; extracting image extraction features at M granularity levels for the target image using a feature extraction model, and performing feature fusion on the M image extraction features according to the granularity level to obtain a multi-granularity image recognition method based on the M levels. Each level has its corresponding fusion features; for the i-th level among the M levels, using the image category node corresponding to the i-th level and the fusion features corresponding to the i-th level, the classification confidence corresponding to each image category node contained in the i-th level is predicted, where the classification confidence represents the probability that the image category of the target image belongs to the image category node, 1≤i≤M; based on the classification confidence corresponding to each image category node in the mind tree, the optimal classification path in the mind tree is determined, and the recognition result for the target image is generated based on the M image category nodes contained in the optimal classification path, where the recognition result includes M image categories.

[0029] Figure 1 An application scenario diagram of the multi-granularity image recognition method based on large model collaboration according to an embodiment of the present disclosure is shown.

[0030] like Figure 1 As shown, the application scenario according to this embodiment may include a vision sensor 101, an image 102, a processor 103, and a memory 104.

[0031] Users can use the visual sensor 101 to interact with the image 102 and the processor 103 to perform processing such as image classification and recognition.

[0032] The processor 103 can be a processor that receives various image parameter information, such as receiving and processing image granularity information obtained from image 102 using the vision sensor 101 (this is just an example). The processor 103 can analyze and process the received image granularity information and other data, and feed back the processing results (such as obtaining or generating relevant analysis data according to user requests) to the terminal device.

[0033] The memory 104 can be used to store image granularity information obtained from image 102 by the vision sensor 101, as well as the processing results after processing by the processor.

[0034] It should be noted that the multi-granularity image recognition method based on large model collaboration provided in this embodiment of the invention can generally be executed by processor 103. Correspondingly, the multi-granularity image recognition device based on large model collaboration provided in this embodiment of the invention can generally be housed in processor 103. The multi-granularity image recognition method based on large model collaboration provided in this embodiment of the invention can also be executed by a processor or processor cluster that is different from processor 103 and capable of communicating with visual sensor 101, image 102, and / or processor 103. Correspondingly, the multi-granularity image recognition device based on large model collaboration provided in this embodiment of the invention can also be housed in a processor or processor cluster that is different from processor 103 and capable of communicating with visual sensor 101, image 102, and / or processor 103.

[0035] It should be understood that Figure 1 The number of vision sensors, processors, and memory shown is merely illustrative. Depending on implementation requirements, any number of vision sensors, processors, and memory can be included.

[0036] The following will be based on Figure 1 The described scene, through Figures 2-5 The multi-granularity image recognition method based on large model collaboration, according to the disclosed embodiments, is described in detail.

[0037] Figure 2 A flowchart of a multi-granularity image recognition method based on large model collaboration according to an embodiment of the present disclosure is shown.

[0038] like Figure 2 As shown, the multi-granularity image recognition method based on large model collaboration in this embodiment includes operations S210~S240.

[0039] In operation S210, in response to an image recognition task for a target image, a mind tree for image recognition is invoked.

[0040] According to embodiments of this disclosure, the mind tree is obtained by classifying a preset image dataset using a large language model. The mind tree includes an image classification root node and at least one image category node with a hierarchical relationship. The mind tree includes M levels, where M≥2.

[0041] According to embodiments of this disclosure, each level of the mind tree can be represented as an image granularity level, with a larger number of levels representing finer image granularity.

[0042] In operation S220, the feature extraction model is used to extract image features at M granularity levels for the target image, and the M image features are fused according to the granularity level to obtain fused features corresponding to each of the M levels.

[0043] According to embodiments of this disclosure, the granularity level corresponds to the M levels of the mind tree. Using the feature extraction model, M image extraction features can be extracted according to the M granularity levels. Then, through feature fusion, M fused features containing all the details of the image can be obtained.

[0044] In operation S230, for the i-th level among the M levels, the classification confidence corresponding to each image category node in the i-th level is predicted by using the image category node corresponding to the i-th level and the fusion feature corresponding to the i-th level.

[0045] According to embodiments of this disclosure, the classification confidence level represents the probability that the image category of the target image belongs to the image category node, 1≤i≤M.

[0046] According to embodiments of this disclosure, by predicting the probability score corresponding to all category nodes in that level based on the fusion features of each level, i.e., the classification confidence, it is possible to obtain the classification confidence for each image category node in the mind tree under the current target image, so as to determine the recognition result.

[0047] In operation S240, the optimal classification path in the mind tree is determined based on the classification confidence corresponding to each image category node in the mind tree, and the recognition result for the target image is generated based on the M image category nodes contained in the optimal classification path.

[0048] According to embodiments of this disclosure, the identification results include M image categories.

[0049] According to embodiments of this disclosure, a greedy algorithm can be used to determine the optimal classification path based on the classification confidence level corresponding to each image category node in the mind tree, thereby generating the recognition result of the target image.

[0050] According to embodiments of this disclosure, the identification result of a target image may include multiple classification rules to which the target image belongs. For example, the current identification result of the target image is animal-mammal-terrestrial-omnivore-dog.

[0051] According to embodiments of this disclosure, a pre-defined image dataset is first classified using a large language model to generate a mind tree containing M levels, multiple root nodes, and multiple image category nodes. This mind tree provides highly granular information and multiple classification paths for subsequent image classification and recognition, and assists in the feature extraction and feature fusion processes of the supervised feature extraction model. Then, in response to an image recognition task for a target image, the mind tree for image recognition is invoked. Based on the M levels of the mind tree, the feature extraction model extracts image features at M granular levels for the target image. Based on the granularity level, the M image features are fused to obtain fused features corresponding to each of the M levels. This achieves the extraction of multi-level, multi-dimensional, multi-granular image features from coarse to fine granularity, enabling detailed classification results to be provided based on the extracted multi-granular image features combined with the mind tree, improving the accuracy of image classification and reducing classification risk. Then, for the i-th level among the M levels, using the image category node corresponding to the i-th level and the fusion feature corresponding to the i-th level, the classification confidence corresponding to each image category node contained in the i-th level is predicted. Based on the classification confidence corresponding to each image category node in the mind tree, the optimal classification path in the mind tree is determined, and the recognition result for the target image is generated. Thus, based on the confidence calculated from the fusion feature, high-accuracy and fast feature recognition is achieved, especially in complex task environments, for simple images and high-fine-grained images, improving the calculation speed and application effectiveness, and providing classification information containing multiple coarse and fine granularities to assist in judgment.

[0052] According to embodiments of this disclosure, by further combining the mind tree with a feature extraction model incorporating deep learning technology, mutual supervision and learning can be achieved during the image classification and recognition process. This allows the feature extraction model in the image recognition process to both reference and constrain the mind tree, further improving the interpretability of the image recognition process, enhancing the practicality and applicability of recognition, and improving the robustness and stability of the model.

[0053] According to embodiments of this disclosure, each image category node has a corresponding evaluation identifier, which characterizes the degree of classification correctness of the image category node; the language large model includes a thought generator and a state evaluator.

[0054] According to embodiments of this disclosure, a method for classifying a pre-defined image dataset using a large language model includes the following operations.

[0055] According to embodiments of this disclosure, an image dataset and an image classification root node are input into a thought generator, and new image classification nodes are recursively generated from the image classification root node downwards until a stopping condition is met, thus obtaining an initial thought tree.

[0056] According to embodiments of this disclosure, the image dataset may include images of various granularities. The stopping condition may include constraints on the generation of the initial thought tree, such as the hierarchy of the initial thought tree to be generated and the number of image category nodes. Based on the stopping condition, the image category root node and the image dataset are input into the thought generator to generate the initial thought tree. The thought generator can be based on formula (1).

[0057] (1);

[0058] Among them, z (1) This can be represented as the first newly generated thought, z (k) This can be represented as the newly generated k-th thought, and "propose" can be represented as the stopping condition of the language model, that is, the cue word of the language model. This can be characterized by using "propose" as the prompt word parameter. Large-scale language model It can be represented as an intermediate thought state, representing the (i+1)th thought to the kth thought, and s can be represented as the image dataset and image classification root node of the large language model.

[0059] According to embodiments of this disclosure, for each image classification node in the initial mind tree, the image classification node is input into a state evaluator for correctness evaluation, thereby obtaining an evaluation label for the image classification node.

[0060] According to embodiments of this disclosure, the evaluation identifier can be a symbol table identifier corresponding to the evaluation result. For example, if the evaluation result is correct, the evaluation identifier of the image classification node is a green checkmark; if the evaluation result is possible, the evaluation identifier of the image classification node is a yellow circle; and if the evaluation result is correct, the evaluation identifier of the image classification node is a red cross. Each image classification node is evaluated by the state evaluator, and the image classification nodes are labeled according to the evaluation results, so that each image classification node has an evaluation identifier, facilitating the organization and updating of the initial mind tree based on the evaluation identifier. The state evaluator can be based on formula (2).

[0061] (2);

[0062] in, It can be characterized as a state estimator. This can be characterized as parameters are The large language model, namely the one described above. The prompt word parameter is Large-scale language model This can be represented as the thought process in the i-th path being evaluated. The result of the thought evaluation is a scalar that can be used to measure whether the current thought is classified into the correct category (e.g., correct / possible / wrong).

[0063] According to embodiments of this disclosure, the initial mind tree is updated using the evaluation identifier of each image classification node to obtain the mind tree.

[0064] According to embodiments of this disclosure, each image classification node in the mind tree contains a corresponding evaluation identifier.

[0065] According to embodiments of this disclosure, an initial thought tree is obtained by inputting an image dataset and an image classification root node into a thought generator. New image classification nodes are recursively generated downwards from the root node until a stopping condition is met. Then, for each image classification node in the initial thought tree, its correctness is evaluated in a state evaluator, resulting in an evaluation label for each node. Finally, the initial thought tree is updated using the evaluation labels of each node, yielding the final thought tree. This achieves the generation of a thought tree with evaluated thought states using a large language model. This allows the thought tree to provide highly granular information and multiple classification paths for subsequent image classification and recognition, thereby assisting the supervised feature extraction model in feature extraction and feature fusion, leading to accurate classification results.

[0066] Figure 3 A schematic diagram of an initial thought tree generated using a large language model according to an embodiment of this disclosure is shown.

[0067] like Figure 3 As shown, Figure 3 The initial thought tree generated using the large language model is shown in the figure. It has three levels: the first level (301) has the highest granularity, with the image classification root node representing coarse-grained features; the second level (302) has a medium granularity; and the third level (303) has the lowest granularity, with the image category nodes representing fine-grained features. Based on the input image dataset and the image classification root node, the thought tree can automatically generate a thought tree containing multiple perspectives.

[0068] According to embodiments of this disclosure, the feature extraction model includes an image patch segmentation module and M feature extraction modules, each of which has a corresponding feature dimension.

[0069] According to embodiments of this disclosure, a method for extracting image features at M granularity levels for a target image using a feature extraction model includes the following operations.

[0070] According to embodiments of this disclosure, the target image is input into an image block segmentation module to perform image block segmentation on the target image, thereby obtaining image input features.

[0071] According to embodiments of this disclosure, an image block segmentation module is used to segment a target image into an image sequence, and image input features are obtained based on the segmented images in the image sequence.

[0072] According to an embodiment of this disclosure, for the m-th feature extraction module among M feature extraction modules, the image input features are upgraded and the number of channels is reduced based on the m-th feature dimension corresponding to the m-th feature extraction module, and then feature extraction is performed to output the m-th image extracted feature corresponding to the m-th granularity level, 1≤m≤M.

[0073] According to embodiments of this disclosure, the feature dimension corresponding to each feature extraction module can be predetermined by human intervention.

[0074] According to embodiments of this disclosure, based on the linear embedding module, and according to the feature dimension of each module, in the m-th feature extraction module, the image input features are first upscaled and the number of channels is reduced to progressively extract abstract image representation features. These features are then mapped to the self-attention network module to facilitate the extraction of image input features, resulting in extracted image features. For example, if the second layer contains 64 image category nodes, the feature dimension corresponding to the second feature extraction module is 64-dimensional. Upscaling the image input features and reducing the number of channels, followed by feature extraction processing, yields 64-dimensional extracted image features.

[0075] According to embodiments of this disclosure, a method for extracting features from M images and fusing them according to granularity levels to obtain fused features corresponding to each of the M levels includes the following operations.

[0076] According to embodiments of this disclosure, the extracted features from M images are sorted in descending order of granularity to obtain a feature sequence.

[0077] According to embodiments of this disclosure, the first image extraction feature in the feature sequence is the largest coarse-grained image extraction feature, and the last image extraction feature in the feature sequence is the smallest fine-grained image extraction feature. For example, the first image extraction feature in the feature sequence is the feature extracted by the Mth feature extraction module, and the last image extraction feature is the feature extracted by the 1st feature extraction module.

[0078] According to an embodiment of this disclosure, a feature is extracted for the kth image in the feature sequence, and the extracted feature of the kth image is fused with the (k-1)th fusion feature to obtain the kth fusion feature.

[0079] According to embodiments of this disclosure, the k-th fusion feature corresponds to the k-th layer in the mind tree, where 1≤k≤M.

[0080] According to the embodiments of this disclosure, the first image extraction feature in the feature sequence is defaulted to the first fusion feature, and the first fusion feature corresponds to the first level in the mind tree. The first fusion feature corresponding to the Mth granularity level can be fused with the kth image extraction feature and the (k-1)th fusion feature to obtain the kth fusion feature corresponding to the kth granularity level (the kth fusion feature corresponds to the i-th level in the mind tree). This can be done according to the formula (4) until the fusion feature corresponding to the first granularity level is obtained. That is, the last fine-grained feature that can represent the smallest fine-grained feature can be done according to the formula (5).

[0081] (3);

[0082] in, This can be characterized as the first image feature extracted in the feature sequence, that is, the shallowest feature extracted using the Mth feature extraction module in the feature extraction model. It can be characterized as a fusion feature (coarse-grained feature) corresponding to the image category node in the first level of the mind tree.

[0083] (4);

[0084] in, This can be represented as a fusion feature corresponding to the image category node in the i-th level of the mind tree. This can be characterized as extracting features from the k-th image in the feature sequence, that is, using the features extracted by the k-th feature extraction module in the feature extraction model. It can be represented as the fusion feature corresponding to the image category node in the (i-1)th level of the mind tree (the (k-1)th fusion feature).

[0085] (5);

[0086] in, Represents the smallest fine-grained feature. This represents the fusion feature (kth fusion feature) corresponding to the image category node in the Mth level of the mind tree.

[0087] According to embodiments of this disclosure, the target image is first input into an image block segmentation module to perform image block segmentation, obtaining image input features. Then, for the m-th feature extraction module among M feature extraction modules, the image input features are dimensionality reduced according to the m-th feature dimension corresponding to the m-th feature extraction module. Feature extraction is then performed on the dimensionality-reduced image input features, outputting the m-th image extraction feature corresponding to the m-th granularity level. The M image extraction features are then sorted in descending order of granularity to obtain a feature sequence. The k-th image extraction feature is fused with the (k-1)-th fusion feature to obtain the k-th fusion feature. Thus, by first obtaining the image extraction features of the target image corresponding to the granularity, and then fusing the image extraction features corresponding to the granularity in a coarse-to-fine order, the fused feature can cover all the details of the target image, completing the extraction of multi-level, multi-dimensional, multi-granularity image features of the target image from coarse to fine granularity. Meanwhile, during the feature fusion process, it is closely linked to each level of the mind tree and the image category node, so that the mind tree can assist in supervising the feature extraction and feature fusion process. Based on the extracted multi-granularity image features, combined with the mind tree, it provides detailed classification results for subsequent confidence and optimal path, thereby improving the accuracy of image classification and reducing classification risk.

[0088] Figure 4 A schematic diagram is shown illustrating feature fusion of an image using a feature extraction model according to an embodiment of the present disclosure.

[0089] like Figure 4 As shown, Figure 4To facilitate feature fusion in image feature extraction using a feature extraction model, the mind tree comprises four levels, resulting in four feature extraction modules. First, the target image is input to the image patch segmentation module 401, which segments the target image into image patches to obtain image input features. Then, these image input features are input to the first feature extraction module 402, where they undergo dimensionality reduction via a linear embedding module 4021 and are mapped to a self-attention network module 4022. The self-attention network module 4022 extracts features from the dimensionality-reduced image input features, outputting the first extracted image feature f1. Next, the first extracted image feature f1 undergoes dimensionality reduction via the linear embedding module 4031 in the second feature extraction module 403 and is mapped to the self-attention network module 4032. The self-attention network module 4032 then extracts features from the dimensionality-reduced first image feature f1. The input feature f1 is used for feature extraction, and the second image extracted feature f2 is output. The second image extracted feature f2 is then passed through the linear embedding module 4041 in the third feature extraction module 404 for dimensionality reduction and mapped to the self-attention network module 4042. The self-attention network module 4042 extracts features from the second image input feature f2 and outputs the third image extracted feature f3. The third image extracted feature f3 is then passed through the linear embedding module 4051 in the fourth feature extraction module 405 for dimensionality reduction and mapped to the self-attention network module 4052. The self-attention network module 4052 extracts features from the third image input feature f3 and outputs the fourth image extracted feature f4.

[0090] Then, the four image extraction features are sorted in descending order of granularity to obtain a feature sequence. The first image extraction feature f4 in the feature sequence is defaulted to the first fusion feature F1. The first image extraction feature f4 in the feature sequence is fused with the first fusion feature F1 to obtain the second fusion feature F2 corresponding to the second granularity level. The third image extraction feature f2 in the feature sequence is fused with the second fusion feature F2 to obtain the third fusion feature F3 corresponding to the third granularity level. The fourth image extraction feature f1 in the feature sequence is fused with the third fusion feature F3 to obtain the fourth fusion feature F4 corresponding to the fourth granularity level.

[0091] According to embodiments of this disclosure, a method for determining the optimal classification path in a mind tree based on the classification confidence corresponding to each image category node in the mind tree includes the following operations.

[0092] According to embodiments of this disclosure, starting from the root node of the image classification in the mind tree, the image category node with the highest classification confidence is searched layer by layer from top to bottom to obtain M candidate nodes.

[0093] According to embodiments of this disclosure, the thinking in the mind tree can be represented by formula (6).

[0094] (6);

[0095] in, This can be represented as the i-th thought in a mind tree. It can be represented as an image category node at a certain granularity level in the i-th thought (the granularity level of any level in the thought tree except for the M-th level). It can be represented as all image category nodes included at the current granularity level, and p can be represented as the sum of the classification confidence of each image category node in the i-th thought of the target image.

[0096] (7);

[0097] in, This can be represented as the j-th image category node included at the current granularity level. It can be represented as the m-th image category node included at the current granularity level.

[0098] According to embodiments of this disclosure, a further method for searching the image category node with the highest classification confidence in each level from top to bottom, starting from the root node of the image classification in the mind tree, to obtain M candidate nodes includes the following operations.

[0099] According to embodiments of this disclosure, for the first layer of the mind tree, the image classification root node is determined as a candidate node for the first layer.

[0100] According to embodiments of this disclosure, since the first layer contains only one image category node, i.e., the image classification root node, it is not necessary to use the highest classification confidence as the selection criterion; the node with the highest confidence is directly confirmed as a candidate node in the first layer. After determining the candidate nodes in the first layer, it is convenient to determine the optimal classification path from the image category nodes (child nodes) in each layer under the candidate nodes.

[0101] According to an embodiment of this disclosure, for the j-th level in the mind tree, the image category node with the highest classification confidence is determined from the child nodes corresponding to the candidate nodes of the (j-1)-th level as the candidate node of the j-th level, 2≤j≤M, and the (j-1)-th level is the level above the j-th level.

[0102] According to embodiments of this disclosure, a linear layer can be used to dimensionally transform image features, enabling the fused features to predict labels at different granularities. Then, cross-entropy is used layer by layer to calculate the classification confidence of the fused features at the current granularity level for different image category nodes based on the fused features processed by the linear layer. The classification confidence of each image category node at each level can be calculated first based on the fused features corresponding to each level in the mind tree. The calculation process of the classification confidence can be based on formula (8).

[0103] (8);

[0104] in, This represents the set of classification confidence scores for each image category node in the j-th level. Represents the thread layer. Represents the cross-entropy layer. It can be characterized as a fusion feature corresponding to the image category node in the j-th level of the mind tree.

[0105] According to embodiments of this disclosure, before determining candidate nodes from the j-th level of the mind tree, it is necessary to first determine multiple child nodes in the j-th level that have connected paths with candidate nodes in the (j-1)-th level. Then, the image category node with the highest classification confidence is determined from the multiple child nodes as the candidate node of the j-th level, and so on, the candidate nodes in each level are confirmed recursively. For example, before determining candidate nodes from the 4th level of the mind tree, it is necessary to first determine multiple child nodes in the 4th level that have connected paths with omnivores in the 3rd level, including: dog, cat, and mouse, where the classification confidence of dog is 80, the classification confidence of cat is 68, and the classification confidence of mouse is 20, thereby determining the image category node with the image category of dog as the candidate node of the 4th level.

[0106] According to embodiments of this disclosure, the optimal classification path is determined based on M candidate nodes.

[0107] According to embodiments of this disclosure, the path where the candidate nodes in each level determined above are located together is the optimal classification path.

[0108] According to embodiments of this disclosure, the starting point of the entire optimal path is determined by first identifying the root node of the image classification as a candidate node of the first layer. Then, for the j-th level in the mind tree, the image category node with the highest classification confidence is selected from the child nodes corresponding to the candidate nodes of the (j-1)-th level as the candidate node of the j-th level. This process is repeated layer by layer to confirm the candidate nodes in each level, ensuring that each candidate node is confirmed based on the candidate nodes in the previous level, thus guaranteeing the correlation between two candidate nodes in two adjacent levels, resulting in M ​​candidate nodes. The optimal classification path is then determined based on the path shared by the M candidate nodes. Therefore, by introducing the three dimensions of mind tree, deep learning, and confidence, even in complex task environments with high-fine-grained images, the optimal classification path in the mind tree can be determined quickly, efficiently, and with high quality. Based on the image category nodes on the optimal classification path, accurate and highly interpretable recognition results for the target image are generated.

[0109] According to embodiments of this disclosure, the method for determining the optimal classification path in a mind tree further includes the following operations.

[0110] According to embodiments of this disclosure, when the classification confidence of the target image classification node in the optimal classification path is less than the classification confidence threshold, the image category node in the mind tree whose evaluation label is the target evaluation label is determined, and the node to be pruned is obtained.

[0111] According to embodiments of this disclosure, the target image classification node is a leaf node in the optimal classification path.

[0112] According to embodiments of this disclosure, when the sum of all classification confidence scores corresponding to the target image classification nodes in the optimal classification path is less than the classification confidence threshold, it is assumed that there are image category nodes in the path whose evaluation results are possible or incorrect as determined by the state evaluator. Therefore, based on the target evaluation identifier corresponding to the possible or incorrect evaluation results, the image category nodes whose evaluation results are possible or incorrect are identified as nodes to be pruned.

[0113] According to embodiments of this disclosure, the nodes to be pruned are pruned to obtain a pruned mind tree.

[0114] According to embodiments of this disclosure, pruning is equivalent to cutting off the path, so that all paths in the thought process are updated to obtain a pruned thought tree.

[0115] According to embodiments of this disclosure, the optimal classification path in the pruned mind tree is determined based on the classification confidence level corresponding to each image category node in the pruned mind tree.

[0116] According to embodiments of this disclosure, based on the classification confidence of each image category node in the pruned mind tree after updating the thinking path, the optimal classification path is re-determined according to the method described above. The sum of all classification confidences corresponding to the target image category nodes in the re-determined optimal classification path is then compared with the classification confidence threshold. If the classification confidence is still less than the classification confidence threshold, pruning continues. If there are no branches to prune in the current mind tree, the current optimal classification path can be output and an alarm prompt can be displayed. If the classification confidence is greater than the classification confidence threshold, the current optimal classification path can be directly output.

[0117] According to embodiments of this disclosure, when the classification confidence of the target image classification node in the optimal classification path is less than a classification confidence threshold, the image category node in the mind tree whose evaluation label is the target evaluation label is determined, resulting in a node to be pruned. This node is then pruned to obtain a pruned mind tree. Based on the classification confidence of each image category node in the pruned mind tree, the optimal classification path in the pruned mind tree is determined. The determined optimal classification path is further constrained and controlled by the classification confidence threshold to ensure that the final optimal classification path is a high-accuracy recognition result that meets the target requirements. Furthermore, the mind tree can be optimized during the determination of the optimal classification path, allowing for multi-directional control over the correctness of the mind tree's path.

[0118] According to embodiments of this disclosure, while determining the optimal classification path, optimization algorithms such as greedy algorithms can be used to find other reference classification paths that are greater than the classification confidence threshold, in addition to the current optimal classification path. These reference classification results can then be used together as reference recognition results for the current target image, further improving the interpretability and diversity of the image recognition process, enhancing the practicality and applicability of the recognition. Even in complex environments and when faced with more complex instruction requirements, multiple recognition classification results with high accuracy and stability can be output for reference.

[0119] According to embodiments of this disclosure, when recognizing the next target image, an unpruned mind tree can be invoked. During the recognition process, personalized and targeted adjustments can be made based on the current target image, only corresponding to the current target image. Alternatively, a pruned mind tree from the previous target image recognition process can be invoked to improve recognition efficiency and speed.

[0120] Figure 5 A schematic diagram of a multi-granularity image recognition process based on large model collaboration according to an embodiment of the present disclosure is shown.

[0121] like Figure 5 As shown, Figure 5The process of multi-granularity image recognition based on large model collaboration is shown. The image dataset and the image classification root node 501 are input into the thought generator and the state evaluator 502 to obtain the thought tree 503. The target image 504 is input into the feature extraction model 505 for feature extraction to obtain multiple image extraction features 506. The multiple image extraction features 506 are input into the feature fusion module 507 for feature fusion to obtain multiple fused features 508. Then, the classification confidence of the multiple fused features 508 is calculated 509 to determine the optimal classification path in the thought tree and the recognition result of the target image 510.

[0122] According to embodiments of this disclosure, the feature extraction model is trained using the following operations.

[0123] According to embodiments of this disclosure, sample images are input into a feature extraction model to be trained, and M granularity-level sample image features are output.

[0124] According to embodiments of this disclosure, sample images can be obtained from a training set, which can be represented as... , where x n It can be characterized as y n This can be represented as the nth fine-grained label, and the extracted features from the sample image can be expressed as f. coarse .

[0125] According to embodiments of this disclosure, the network parameters of the feature extraction model to be trained can be initialized in advance.

[0126] According to embodiments of this disclosure, features are extracted from M sample images according to granularity level and then fused to obtain sample fusion features corresponding to each of the M levels.

[0127] According to embodiments of this disclosure, the sample fusion feature can be represented as F coarse .

[0128] According to embodiments of this disclosure, for the i-th level among M levels, the classification confidence of the sample corresponding to each image category node in the i-th level is predicted by using the image category node corresponding to the i-th level and the fusion feature corresponding to the i-th level.

[0129] According to embodiments of this disclosure, the sample classification confidence level characterizes the probability that the image category of a sample image belongs to an image category node.

[0130] According to embodiments of this disclosure, the sample classification confidence level can be expressed as P. coarse .

[0131] According to embodiments of this disclosure, the loss value of the i-th level is determined based on the sample classification confidence of each image category node in the i-th level.

[0132] According to embodiments of this disclosure, the feature extraction model to be trained is trained based on the loss value of each level in the mind tree to obtain the feature extraction model.

[0133] According to the embodiments of this disclosure, the loss values ​​of M levels are summed to obtain the total loss value, which can be expressed as shown in formula (9). Then, the backpropagation loss is calculated using the total loss value, which can be expressed as shown in formula (10). The network parameters of the feature extraction model are then trained and updated using the backpropagation loss and the pre-set learning rate to obtain the updated network parameters, which can be expressed as shown in formula (11).

[0134] (9);

[0135] Where L can be represented as the total loss value, This can be represented as the loss of the i-th level. Value

[0136] (10);

[0137] in, This can be characterized as backpropagation loss. This can be characterized as gradient calculation.

[0138] (11);

[0139] in, This can be represented as the updated network parameters in the (t+1)th update. This can be represented as the updated network parameters in the t-th update. This can be represented by the learning rate, which is typically set to 1×102. −3 .

[0140] According to the embodiments of this disclosure, during the training of the feature extraction model, an optimizer can also be used to assist in the training of the feature extraction model. In the process of using the optimizer to assist in training, the training period can be set to 100, the input size of the sample image is 224×224, the batch size is 64, and the weight decay coefficient can be set to 0.05. The updated network parameters obtained by training the feature extraction model with the optimizer can be obtained according to the formula (12).

[0141] (12);

[0142] in, This can be characterized as a parameter to prevent division by zero. This can be represented as a weight decay coefficient. It can be characterized as a first-order correction parameter. It can be characterized as a second-order correction parameter.

[0143] According to embodiments of this disclosure, the network parameters of the feature extraction model are updated and trained by calculating the loss value, thereby obtaining a feature extraction model that can extract accurate image features.

[0144] According to embodiments of this disclosure, after training the feature extraction model, the trained feature extraction model can be tested using sample images. Following the above method, based on the mind tree generated using a large language model, the sample images are input into the trained feature extraction model. The output sample classification confidence is compared with the original reference classification confidence of the sample image to confirm the stability, robustness, and accuracy of the trained feature extraction model.

[0145] According to embodiments of this disclosure, sample images are input into the feature extraction model to be trained, outputting M granularity-level sample image extracted features. Then, based on the granularity level, the M sample image extracted features are fused to obtain sample fusion features corresponding to each of the M levels. Using the image category nodes corresponding to the i-th level and the fusion features corresponding to the i-th level, the sample classification confidence corresponding to each image category node in the i-th level is predicted. Finally, the feature extraction model to be trained is trained based on the loss values ​​of each level in the mind tree, resulting in the feature extraction model. Thus, by combining the mind tree and calculating the loss values, the network parameters of the feature extraction model are continuously adjusted, resulting in a highly robust feature extraction model that can extract accurate image features.

[0146] According to embodiments of this disclosure, the parameters obtained after classifying general images using the multi-granularity image recognition method based on large model collaboration described in this application and using methods in the prior art can be shown in Table 1.

[0147] Table 1. Comparison of different methods for general datasets under accuracy and risk coefficient evaluation metrics.

[0148]

[0149] As can be seen from Table 1, in the process of recognizing and classifying general images, regardless of the magnitude of the parameters of the image to be recognized, the number of parameters used in this application is moderate, resulting in the highest classification accuracy and the lowest risk coefficient.

[0150] According to embodiments of this disclosure, the parameters obtained after recognizing and classifying fine-grained images using the multi-granularity image recognition method based on large model collaboration described in this application and using methods in the prior art can be shown in Table 2.

[0151] Table 2. Comparison of different methods for fine-grained datasets under accuracy and risk coefficient evaluation metrics.

[0152]

[0153] As can be seen from Table 2, this application has the highest classification accuracy and the lowest risk coefficient in the process of recognizing and classifying fine-grained images.

[0154] As can be seen from Tables 1 and 2, based on the embodiments of this disclosure, the identification method of this application improves the classification ability of general images by 0.5% and the classification ability of fine-grained images by 1.1% compared with the prior art.

[0155] Figure 6 A structural block diagram of a multi-granularity image recognition device based on large model collaboration according to an embodiment of the present disclosure is shown.

[0156] like Figure 6 As shown, the apparatus for determining the structural parameters of a nonlinear particle damper in this embodiment includes: a calling module 610, a feature extraction module 620, a feature fusion module 630, and a classification path determination module 640.

[0157] Module 610 is used to invoke a thought tree for image recognition in response to an image recognition task targeting a target image. The thought tree is obtained by classifying a pre-defined image dataset using a large language model. The thought tree includes a root node for image classification and at least one image category node with a hierarchical relationship. The thought tree has M levels, where M ≥ 2. Module 610 can be used to execute the operation S210 described above, which will not be repeated here.

[0158] The feature extraction module 620 is used to extract image features at M granularity levels for the target image using a feature extraction model, and to fuse the M image features according to the granularity level to obtain fused features corresponding to each of the M levels. The feature extraction module 620 can be used to perform the operation S220 described above, which will not be repeated here.

[0159] The feature fusion module 630 is used to predict the classification confidence of each image category node in the i-th level out of M levels, using the image category node corresponding to the i-th level and the fused features corresponding to the i-th level. The classification confidence represents the probability that the image category of the target image belongs to the image category node, 1 ≤ i ≤ M. The feature fusion module 630 can be used to perform the operation S230 described above, which will not be repeated here.

[0160] The classification path determination module 640 is used to determine the optimal classification path in the mind tree based on the classification confidence corresponding to each image category node, and to generate a recognition result for the target image based on the M image category nodes contained in the optimal classification path. The recognition result includes M image categories. The classification path determination module 640 can be used to perform the operation S240 described above, which will not be repeated here.

[0161] According to embodiments of this disclosure, any plurality of modules among the invocation module 610, feature extraction module 620, feature fusion module 630, and classification path determination module 640 can be combined into one module, or any one of these modules can be split into multiple modules. Alternatively, at least part of the functionality of one or more of these modules can be combined with at least part of the functionality of other modules and implemented in one module. According to embodiments of this disclosure, at least one of the invocation module 610, feature extraction module 620, feature fusion module 630, and classification path determination module 640 can be at least partially implemented as hardware circuitry, such as a field-programmable gate array (FPGA), a programmable logic array (PLA), a system-on-a-chip, a system-on-a-substrate, a system-on-package, an application-specific integrated circuit (ASIC), or implemented in hardware or firmware by any other reasonable means of integrating or packaging the circuitry, or implemented in any one of software, hardware, and firmware methods, or in a suitable combination of any of these methods. Alternatively, at least one of the calling module 610, feature extraction module 620, feature fusion module 630, and classification path determination module 640 can be at least partially implemented as a computer program module, which can perform corresponding functions when the computer program module is run.

[0162] Figure 7 A block diagram of an electronic device using a multi-granularity image recognition method based on large model collaboration according to an embodiment of the present disclosure is shown.

[0163] like Figure 7 As shown, an electronic device according to an embodiment of the present disclosure includes a processor 701, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 702 or a program loaded from a storage portion 708 into a random access memory (RAM) 703. The processor 701 may include, for example, a general-purpose microprocessor (e.g., a CPU), an instruction set processor and / or an associated chipset and / or a special-purpose microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 701 may also include onboard memory for caching purposes. The processor 701 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of the present disclosure.

[0164] RAM 703 stores various programs and data required for the operation of the electronic device. Processor 701, ROM 702, and RAM 703 are interconnected via bus 704. Processor 701 performs various operations of the method flow according to embodiments of the present disclosure by executing programs in ROM 702 and / or RAM 703. It should be noted that the programs may also be stored in one or more memories other than ROM 702 and RAM 703. Processor 701 may also perform various operations of the method flow according to embodiments of the present disclosure by executing programs stored in said one or more memories.

[0165] According to embodiments of this disclosure, the electronic device may further include an input / output (I / O) interface 705, which is also connected to a bus 704. The electronic device may also include one or more of the following components connected to the I / O interface 705: an input section 706 including a keyboard, mouse, etc.; an output section 707 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 708 including a hard disk, etc.; and a communication section 709 including a network interface card such as a LAN card, modem, etc. The communication section 709 performs communication processing via a network such as the Internet. A drive 710 is also connected to the I / O interface 705 as needed. A removable medium 711, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on the drive 710 as needed so that computer programs read from it can be installed into the storage section 708 as needed.

[0166] This disclosure also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments; or it may exist independently and not assembled into the device / apparatus / system. The computer-readable storage medium carries one or more programs that, when executed, implement the method according to the embodiments of this disclosure.

[0167] According to embodiments of this disclosure, the computer-readable storage medium can be a non-volatile computer-readable storage medium, such as including, but not limited to: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this disclosure, the computer-readable storage medium can be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. For example, according to embodiments of this disclosure, the computer-readable storage medium may include ROM 702 and / or RAM 703 and / or one or more memories other than ROM 702 and RAM 703 described above.

[0168] Embodiments of this disclosure also include a computer program product comprising a computer program containing program code for performing the methods shown in the flowchart. When the computer program product is run on a computer system, the program code enables the computer system to implement the multi-granularity image recognition method based on large model collaboration provided in embodiments of this disclosure.

[0169] When the computer program is executed by the processor 701, it performs the functions defined in the system / apparatus of this disclosure embodiments. According to embodiments of this disclosure, the systems, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0170] In one embodiment, the computer program may rely on a tangible storage medium such as an optical storage device or a magnetic storage device. In another embodiment, the computer program may also be transmitted and distributed in the form of signals over a network medium, and may be downloaded and installed via the communication section 709, and / or installed from a removable medium 711. The program code contained in the computer program can be transmitted using any suitable network medium, including but not limited to: wireless, wired, etc., or any suitable combination thereof.

[0171] In such an embodiment, the computer program can be downloaded and installed from a network via the communication section 709, and / or installed from the removable medium 711. When the computer program is executed by the processor 701, it performs the functions defined in the system of this disclosure embodiment. According to embodiments of this disclosure, the systems, devices, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0172] Those skilled in the art will understand that the features described in the various embodiments of this disclosure can be combined or combined in various ways, even if such combinations or combinations are not explicitly described in this disclosure. In particular, the features described in the various embodiments of this disclosure can be combined and / or combined in various ways without departing from the spirit and teachings of this disclosure. All such combinations and / or combinations fall within the scope of this disclosure.

Claims

1. A multi-granularity image recognition method based on large model collaboration, characterized in that, The method includes: In response to an image recognition task for a target image, a mind tree for image recognition is invoked. The mind tree is obtained by classifying a pre-defined image dataset using a large language model. The mind tree includes an image classification root node and at least one image category node with a hierarchical relationship. The mind tree includes M levels, where M≥2. The feature extraction model is used to extract image features at M granularity levels for the target image, and the M image features are fused according to the granularity level to obtain fused features corresponding to each of the M levels. For the i-th level among M levels, the classification confidence of each image category node in the i-th level is predicted by using the image category node corresponding to the i-th level and the fusion feature corresponding to the i-th level. The classification confidence represents the probability that the image category of the target image belongs to the image category node, 1≤i≤M. Based on the classification confidence level corresponding to each image category node in the mind tree, the optimal classification path in the mind tree is determined, and a recognition result for the target image is generated based on the M image category nodes contained in the optimal classification path, wherein the recognition result includes M image categories.

2. The method according to claim 1, characterized in that, The feature extraction model includes an image patch segmentation module and M feature extraction modules, each of which has a corresponding feature dimension. The step of extracting image features at M granularity levels for the target image using a feature extraction model includes: The target image is input into the image block segmentation module, and the target image is segmented into image blocks to obtain image input features; For the m-th feature extraction module among the M feature extraction modules, the image input features are dimensionality reduced according to the m-th feature dimension corresponding to the m-th feature extraction module, and feature extraction is performed on the dimensionality-reduced image input features to output the m-th image extracted feature corresponding to the m-th granularity level, 1≤m≤M.

3. The method according to claim 1, characterized in that, The step of fusing the extracted features from the M images according to the granularity level to obtain fused features corresponding to each of the M levels includes: The features extracted from the M images are sorted in descending order of granularity to obtain a feature sequence; For the k-th image in the feature sequence, extract the feature, and fuse the k-th image feature with the (k-1)-th fusion feature to obtain the k-th fusion feature, where the k-th fusion feature corresponds to the k-th layer in the mind tree, and 1≤k≤M.

4. The method according to claim 1, characterized in that, The step of determining the optimal classification path in the mind tree based on the classification confidence score corresponding to each image category node in the mind tree includes: Starting from the root node of the image classification in the mind tree, search the image category node with the highest classification confidence in each level from top to bottom to obtain M candidate nodes; The optimal classification path is determined based on the M candidate nodes.

5. The method according to claim 4, characterized in that, Starting from the root node of the image classification in the mind tree, the search proceeds from top to bottom, layer by layer, to find the image category node with the highest classification confidence at each level, resulting in M ​​candidate nodes, including: For the first layer of the mind tree, the root node of the image classification is determined as a candidate node of the first layer; For the j-th level in the mind tree, from the child nodes corresponding to the candidate nodes of the (j-1)-th level, the image category node with the highest classification confidence is determined as the candidate node of the j-th level, 2≤j≤M, and the (j-1)-th level is the level above the j-th level.

6. The method according to claim 1, characterized in that, Each image category node has a corresponding evaluation identifier, which characterizes the correctness of the classification of the image category node; The large-scale language model includes a thought generator and a state evaluator; The classification of the pre-defined image dataset using a large language model includes: The image dataset and the image classification root node are input into the mind generator. Starting from the image classification root node, new image classification nodes are recursively generated downwards until the stopping condition is met, thus obtaining the initial mind tree. For each image classification node in the initial mind tree, the image classification node is input into the state evaluator for correctness evaluation, and an evaluation label for the image classification node is obtained. The initial mind tree is updated using the evaluation identifier of each image classification node to obtain the mind tree.

7. The method according to claim 6, characterized in that, The method further includes: If the classification confidence of the target image classification node in the optimal classification path is less than the classification confidence threshold, the image category node in the mind tree whose evaluation label is the target evaluation label is determined, and the node to be pruned is obtained, wherein the target image classification node is the leaf node in the optimal classification path; The nodes to be pruned are pruned to obtain the pruned mind tree; The optimal classification path in the pruned mind tree is determined based on the classification confidence score corresponding to each image category node.

8. The method according to claim 1, wherein the feature extraction model is trained using the following operation: Input the sample images into the feature extraction model to be trained, and output M sample images with extracted features at different granularity levels; Based on the granularity level, features are extracted from the M sample images and fused to obtain sample fusion features corresponding to each of the M levels. For the i-th level out of M levels, using the image category nodes corresponding to the i-th level and the fusion features corresponding to the i-th level, the classification confidence of the samples corresponding to each image category node in the i-th level is predicted, where, The sample classification confidence score represents the probability that the image category of the sample image belongs to the image category node; The loss value of the i-th level is determined based on the classification confidence of the sample of each image category node in the i-th level; The feature extraction model is trained by using the loss value of each level in the mind tree.

9. A multi-granularity image recognition device based on large-model collaboration, characterized in that, The device includes: The Mind Tree module is used to invoke a mind tree for image recognition in response to an image recognition task targeting a target image. The mind tree is obtained by classifying a pre-defined image dataset using a large language model. The mind tree includes an image classification root node and at least one image category node with a hierarchical relationship. The mind tree includes M levels, where M≥2. The feature extraction module is used to extract image features at M granularity levels for the target image using a feature extraction model, and to fuse the M image features according to the granularity level to obtain fused features corresponding to each of the M levels. The feature fusion module is used to predict the classification confidence of each image category node in the i-th level among M levels by using the image category node corresponding to the i-th level and the fusion feature corresponding to the i-th level. The classification confidence represents the probability that the image category of the target image belongs to the image category node, 1≤i≤M. The classification path determination module is used to determine the optimal classification path in the mind tree based on the classification confidence level corresponding to each image category node in the mind tree, and to generate a recognition result for the target image based on the M image category nodes contained in the optimal classification path, wherein the recognition result includes M image categories.

10. An electronic device comprising a memory and a processor, the memory storing instructions executable by the processor, the instructions, when executed by the processor, causing the processor to perform the method as described in any one of claims 1 to 8.

Images