Image classification method, model training method, device, equipment and medium

Abstract

The present disclosure provides an image classification method, a training method for a deep learning model, an apparatus, an electronic device, a storage medium, and a program product, which relate to the field of artificial intelligence technology, especially to the technical fields of computer vision, deep learning, and large models, and can be applied to scenes such as image processing. The specific implementation scheme is: extracting features from the image to be processed to obtain image features; clustering the image features to obtain image clustering marking features; obtaining text prompt features based on the image clustering marking features and the task prompt marking features, and the task prompt marking features are used to characterize the knowledge related to the image classification task; and obtaining the image classification result of the image to be processed based on the text prompt features and the image to be processed.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence technology, in particular to computer vision, deep learning, large models and other technical fields, and can be applied to scenes such as image processing. Specifically, it relates to an image classification method, a training method for a deep learning model, a device, an electronic device, a storage medium and a program product. Background Art

[0002] With the explosive growth of image data, massive data sources and rich data layers, it is becoming increasingly difficult to rely on manual analysis and processing of these images. Computer vision technology offers great potential for liberating manpower. Computer vision is a science that studies how to use electronic devices to "see", that is, the science and technology of using cameras and computers instead of human eyes to classify and search for target objects in images. It can realize intelligent interaction between humans and machines, such as visual question and answer and visual dialogue. Summary of the invention

[0003] The present disclosure provides an image classification method, a deep learning model training method, a device, an electronic device, a storage medium, and a program product.

[0004] According to one aspect of the present disclosure, there is provided an image classification method, comprising: extracting features of an image to be processed to obtain image features; clustering the image features to obtain image clustering labeling features; obtaining text prompt features based on the image clustering labeling features and task prompt labeling features, wherein the task prompt labeling features are used to characterize knowledge related to an image classification task; and obtaining an image classification result of the image to be processed based on the text prompt features and the image to be processed.

[0005] According to another aspect of the present disclosure, a training method for a deep learning model is provided, wherein the deep learning model includes a task prompt module, the task prompt module includes adjustable parameters, and the method includes: extracting features from a sample image to obtain sample image features; clustering the sample image features to obtain sample image clustering labeling features; obtaining sample task prompt labeling features based on the task prompt module; obtaining sample text prompt features based on the sample image clustering labeling features and the sample task prompt labeling features; and adjusting the adjustable parameters in the task prompt module based on the sample text prompt features, the sample image features and labels to obtain a trained deep learning model, wherein the labels are used to characterize the real image classification results of the sample images.

[0006] According to another aspect of the present disclosure, there is provided an image classification device, comprising: a feature extraction module, used to extract features from an image to be processed to obtain image features; a clustering module, used to cluster the above image features to obtain image clustering marking features; a text prompt generation module, used to obtain text prompt features based on the above image clustering marking features and task prompt marking features, wherein the above task prompt marking features are used to characterize knowledge related to an image classification task; and a classification module, used to obtain an image classification result of the above image to be processed based on the above text prompt features and the above image to be processed.

[0007] According to another aspect of the present disclosure, a training device for a deep learning model is provided, wherein the deep learning model includes a task prompt module, the task prompt module includes adjustable parameters, and the device includes: a sample feature extraction module, used to extract features from sample images to obtain sample image features; a sample clustering module, used to cluster the sample image features to obtain sample image clustering labeling features; a sample task prompt generation module, used to obtain sample task prompt labeling features based on the task prompt module; a sample text prompt generation module, used to obtain sample text prompt features based on the sample image clustering labeling features and the sample task prompt labeling features; and a parameter adjustment module, used to adjust the adjustable parameters in the task prompt module based on the sample text prompt features, the sample image features and labels to obtain a trained deep learning model, wherein the labels are used to characterize the real image classification results of the sample images.

[0008] According to another aspect of the present disclosure, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute a method as disclosed in the present disclosure.

[0009] According to another aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable the computer to execute the method of the present disclosure.

[0010] According to another aspect of the present disclosure, a computer program product is provided, including a computer program, and when the computer program is executed by a processor, the method of the present disclosure is implemented.

[0011] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure.

[0013] Figure 1 An exemplary system architecture to which the image classification method and apparatus according to an embodiment of the present disclosure can be applied is schematically shown;

[0014] Figure 2 The flowchart of the image classification method according to the embodiment of the present disclosure is schematically shown;

[0015] Figure 3 A schematic diagram schematically shows a method for generating a text prompt feature according to an embodiment of the present disclosure;

[0016] Figure 4 A schematic diagram schematically shows a visual prompt feature for generating text reorganization according to an embodiment of the present disclosure;

[0017] Figure 5 A schematic diagram schematically shows a method of generating an image classification result according to an embodiment of the present disclosure;

[0018] Figure 6 A flowchart of a deep learning model training method according to an embodiment of the present disclosure is schematically shown;

[0019] Figure 7 A schematic diagram schematically shows a method of adjusting adjustable parameters according to an embodiment of the present disclosure;

[0020] Figure 8 A block diagram of an image classification device according to an embodiment of the present disclosure is schematically shown;

[0021] Fig. 9 A block diagram schematically shows a training device for a deep learning model according to an embodiment of the present disclosure; and

[0022] Fig.10 A block diagram of an electronic device suitable for implementing an image classification method according to an embodiment of the present disclosure is schematically shown. DETAILED DESCRIPTION

[0023] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0024] The present disclosure provides an image classification method, a deep learning model training method, a device, an electronic device, a storage medium, and a program product.

[0025] According to an embodiment of the present disclosure, an image classification method may include: extracting features from an image to be processed to obtain image features; clustering the image features to obtain image clustering marking features; obtaining text prompt features based on the image clustering marking features and the task prompt marking features, wherein the task prompt marking features are used to characterize knowledge related to the image classification task; and obtaining an image classification result of the image to be processed based on the text prompt features and the image to be processed.

[0026] In the technical solution of the present disclosure, the collection, storage, use, processing, transmission, provision, disclosure and application of user personal information involved comply with the provisions of relevant laws and regulations, take necessary confidentiality measures, and do not violate public order and good morals.

[0027] In the technical solution of the present disclosure, the user's authorization or consent is obtained before obtaining or collecting the user's personal information.

[0028] Figure 1 An exemplary system architecture to which the image classification method and apparatus according to an embodiment of the present disclosure can be applied is schematically shown.

[0029] It should be noted that Figure 1 The examples shown are only examples of system architectures to which the embodiments of the present disclosure can be applied, in order to help those skilled in the art understand the technical content of the present disclosure, but do not mean that the embodiments of the present disclosure cannot be used in other devices, systems, environments or scenarios. For example, in another embodiment, the exemplary system architecture to which the image classification method and apparatus can be applied may include a terminal device, but the terminal device may implement the image classification method and apparatus provided by the embodiments of the present disclosure without interacting with the server.

[0030] like Figure 1 As shown, the system architecture 100 according to this embodiment may include terminal devices 101, 102, 103, a network 104 and a server 105. The network 104 is used to provide a medium for communication links between the terminal devices 101, 102, 103 and the server 105. The network 104 may include various connection types, such as wired and / or wireless communication links, etc.

[0031] Users can use terminal devices 101, 102, 103 to interact with server 105 through network 104 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 101, 102, 103, such as knowledge reading applications, web browser applications, search applications, instant messaging tools, email clients and / or social platform software, etc. (only as examples).

[0032] The terminal devices 101 , 102 , and 103 may be various electronic devices having a display screen and supporting web browsing, including but not limited to smart phones, tablet computers, laptop computers, and desktop computers.

[0033] The server 105 may be a server that provides various services, such as a background management server (only an example) that provides support for the content browsed by the user using the terminal devices 101, 102, and 103. The background management server may analyze and process the received image data such as the user request, and feed back the processing result (such as the web page, information, or data obtained or generated according to the user request) to the terminal device.

[0034] It should be noted that the image classification method provided in the embodiment of the present disclosure can generally be executed by the terminal device 101, 102, or 103. Accordingly, the image classification apparatus provided in the embodiment of the present disclosure can also be set in the terminal device 101, 102, or 103.

[0035] Alternatively, the image classification method provided in the embodiment of the present disclosure may also be generally performed by the server 105. Accordingly, the image classification device provided in the embodiment of the present disclosure may generally be arranged in the server 105. The image classification method provided in the embodiment of the present disclosure may also be performed by a server or server cluster that is different from the server 105 and can communicate with the terminal devices 101, 102, 103 and / or the server 105. Accordingly, the image classification device provided in the embodiment of the present disclosure may also be arranged in a server or server cluster that is different from the server 105 and can communicate with the terminal devices 101, 102, 103 and / or the server 105.

[0036] For example, the terminal devices 101, 102, and 103 can obtain the image to be processed uploaded by the user, and then send the obtained image to be processed to the server 105, and the server 105 extracts features of the image to be processed to obtain image features; clusters the image features to obtain image clustering marking features; obtains text prompt features based on the image clustering marking features and the task prompt marking features, and the task prompt marking features are used to characterize the knowledge related to the image classification task; and obtains the image classification result of the image to be processed based on the text prompt features and the image to be processed. Alternatively, a server or server cluster that can communicate with the terminal devices 101, 102, 103 and / or the server 105 analyzes the image to be processed and finally obtains the image classification result of the image to be processed.

[0037] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to the implementation requirements.

[0038] It should be noted that the sequence numbers of the operations in the following method are only used as representations of the operations for the purpose of description, and should not be regarded as representing the execution order of the operations. Unless explicitly stated, the method does not need to be executed completely in the order shown.

[0039] Figure 2 The flowchart of the image classification method according to the embodiment of the present disclosure is schematically shown.

[0040] like Figure 2 As shown, the method includes operations S210 to S240.

[0041] In operation S210, feature extraction is performed on the image to be processed to obtain image features.

[0042] In operation S220, the image features are clustered to obtain image clustering label features.

[0043] In operation S230 , a text prompt feature is obtained based on the image clustering mark feature and the task prompt mark feature.

[0044] In operation S240, an image classification result of the image to be processed is obtained based on the text prompt feature and the image to be processed.

[0045] According to an embodiment of the present disclosure, task hint tag features are used to characterize knowledge related to an image classification task.

[0046] According to an embodiment of the present disclosure, an image encoder in a visual language model can be used to extract features from an image to be processed to obtain image features. The image encoder can be an image encoder based on a convolutional neural network (CNN). As long as it is a feature extraction module that can extract image features from an image to be processed, there is no limitation on this.

[0047] According to an embodiment of the present disclosure, clustering image features to obtain image clustering label features may refer to: semantically decoupling the overall semantic features of the image features to obtain multiple sub-image features. Clustering multiple sub-image features with the same semantics to obtain image clustering label features.

[0048] According to an embodiment of the present disclosure, the image clustering tag feature may refer to a fine-grained image tag feature. The fine-grainedness may refer to the same vector dimension of the image tag feature as the vector dimension of the task prompt tag feature. This allows the image tag feature and the task prompt tag feature to be semantically aligned, avoiding the semantic ambiguity of the text prompt feature caused by directly combining the image feature with the task prompt tag feature to obtain the text prompt feature.

[0049] According to the embodiments of the present disclosure, based on the text prompt features and the image to be processed being used as the input data of the pre-trained visual language model, an image classification result of the image to be processed with guaranteed accuracy can be obtained on the basis of fixing the model parameters of the pre-trained visual language model. The specific representation function of the pre-trained visual language model is reactivated by adding additional text prompt features to the image to be processed without modifying the model parameters of the pre-trained visual language model.

[0050] According to the embodiments of the present disclosure, the text prompt feature is obtained based on the image clustering mark feature and the task prompt mark feature. It combines the knowledge related to the image classification task with the image clustering mark feature that is aligned with the task prompt mark feature in the vector dimension, so that the prompt information is sufficient while avoiding semantic ambiguity, thereby improving the classification performance and recognition accuracy of the pre-trained visual language model.

[0051] According to the embodiments of the present disclosure, Figure 2 The operation S220 shown, clustering the image features to obtain image clustering label features, may include the following operations.

[0052] For example, semantic decoupling of image features is performed to obtain multiple sub-image features, and multiple sub-image features are clustered to obtain clustered label features.

[0053] According to an embodiment of the present disclosure, semantically decoupling an image feature to obtain a plurality of sub-image features may include: splitting the image feature according to a predetermined vector dimension to obtain a plurality of sub-image features. The vector dimension may include a horizontal vector dimension and a vertical vector dimension.

[0054] According to an embodiment of the present disclosure, clustering multiple sub-image features to obtain clustering label features may include: clustering multiple sub-image features to obtain multiple clustering clusters. Based on the multiple clustering clusters, obtaining clustering label features.

[0055] According to an embodiment of the present disclosure, obtaining clustering label features based on multiple clustering clusters may include: for each clustering cluster, performing weighted summation on multiple sub-image features in the clustering cluster to obtain an image clustering label vector. Obtaining image clustering label features based on multiple image clustering label vectors corresponding one-to-one to multiple clustering clusters.

[0056] According to an embodiment of the present disclosure, a weighted sum is performed on features of multiple sub-images in a cluster to obtain an image cluster label vector, which can be referred to in the following formula (1).

[0057]

[0058] Among them, t kThe image cluster label vector representing the Kth cluster, H k represents the Kth cluster, z j represents the jth sub-image feature in the Kth cluster, ω j represents the weight of the j-th sub-image feature in the K-th cluster.

[0059] According to an embodiment of the present disclosure, j The importance score, weight, used to characterize the j-th sub-image feature. The weight of each sub-image feature can be obtained in a predetermined manner, but is not limited to this. The weight can also be determined based on the classification confidence of the j-th sub-image feature in the K-th cluster, as long as it is an importance score that can characterize the sub-image feature.

[0060] According to other embodiments of the present disclosure, it can be considered that multiple sub-image features in the same cluster have the same semantic information, and it can be determined that the weights of the multiple sub-image features are the same. The multiple sub-image features in the cluster can be directly averaged to obtain the image clustering label vector. However, it is not limited to this. The sub-image feature as the cluster center in the cluster can also be used as the image clustering label vector.

[0061] According to the embodiments of the present disclosure, the importance of the semantic information of each sub-image feature in the cluster is represented by a weight, thereby enhancing the features of the sub-image features with high importance, thereby making the image clustering marking features closer to the category of the actual image to be processed, and improving the prompt effect of the image clustering marking features.

[0062] According to an embodiment of the present disclosure, clustering multiple sub-image features to obtain multiple cluster clusters includes: clustering multiple sub-image features using a density peak clustering algorithm to obtain multiple cluster clusters. However, it is not limited to this. Other clustering algorithms can also be used to cluster multiple sub-image features to obtain multiple cluster clusters. Other clustering algorithms can be, for example, a K-means clustering algorithm.

[0063] According to an embodiment of the present disclosure, a plurality of sub-image features are clustered using a density peak clustering algorithm, and it can be assumed that the cluster center of a cluster is surrounded by low-density neighbors, and the distance between the cluster center and any high-density sub-image feature is relatively large. The density of each sub-image feature and the minimum distance to a sub-image feature with a higher density can be determined. Density ρ i The determination method of can refer to formula (2), the minimum distance δ i The determination method of can refer to formula (3).

[0064]

[0065] Among them, Z represents the sub-image feature set, zi represents the i-th sub-image feature, z j represents the j-th sub-image feature.

[0066]

[0067] According to an embodiment of the present disclosure, the cluster center score of each sub-image feature can be determined based on density and minimum distance, and a higher cluster center score means a greater potential to become a cluster center. The sub-image feature with the highest cluster center score is used as the cluster center. The remaining sub-image features are assigned to the clusters that are closest to the cluster center and have a higher density, thereby completing the generation of cluster clusters.

[0068] According to the embodiments of the present disclosure, compared with other clustering algorithms, the density peak clustering algorithm is used to cluster multiple sub-image features, and the density connectivity of the clustering clusters can be used to quickly determine clustering clusters of arbitrary shapes, avoiding iteration, and generating a clustering cluster generation algorithm that has neither an iterative process nor a large number of parameters involved, thereby improving processing efficiency.

[0069] According to the embodiments of the present disclosure, Figure 2 Operation S220 shown, obtaining text prompt features based on image clustering mark features and task prompt mark features, may include the following operations.

[0070] For example, the image clustering mark feature and the task prompt mark feature are fused to obtain the text mark feature. Based on the text mark feature and the predetermined category mark vector, the text prompt mark feature is obtained. The text prompt mark feature is text-encoded to obtain the text prompt feature.

[0071] According to the embodiments of the present disclosure, the image features representing the overall image semantic information can be decoupled into more fine-grained semantic information, such as image clustering tag features. The image clustering tag features can be added to the task prompt tag features so that the image semantic information is added to the task prompt tag features, thereby allowing text and vision to prompt each other and fully unleash the potential representation capabilities of the two modalities.

[0072] According to an embodiment of the present disclosure, clustering multiple sub-image features to obtain multiple clusters may also include: determining the vector dimension of the task prompt mark feature. Determining the number of clusters of the clusters based on the vector dimension of the task prompt mark feature. Based on the number of clusters of the clusters, clustering the multiple sub-image features using a density peak clustering algorithm to obtain multiple clusters.

[0073] According to an embodiment of the present disclosure, setting the number of clusters of the clustering clusters and the vector dimension of the task prompt mark feature to be the same can achieve semantic alignment between the image clustering mark feature and the task prompt mark feature, thereby avoiding semantic ambiguity.

[0074] Figure 3 A schematic diagram of generating text prompt features according to an embodiment of the present disclosure is schematically shown.

[0075] like Figure 3 As shown, the image 310 to be processed can be input into the image encoder M310 to obtain the image feature 320.

[0076] like Figure 3 As shown, the image feature 320 can be semantically decoupled to obtain multiple sub-image features 330.

[0077] like Figure 3 As shown, multiple sub-image features 330 may be clustered to obtain multiple clusters 340, such as {C1, C2, C3, C4}.

[0078] like Figure 3 As shown, the task prompt mark feature 350 may include 4 task prompt mark vectors, such as {v1, v2, v3, v4}. Based on the vertical vector dimension 4, the number of clusters of the clustering clusters may be determined to be 4. Based on the clustering clusters, the image clustering mark feature 360 ​​may be obtained. The image clustering mark feature 360 ​​may include 4 image clustering mark vectors, such as {t1, t2, t3, t4}.

[0079] like Figure 3 As shown, the corresponding elements of {v1, v2, v3, v4} and {t1, t2, t3, t4} can be added to obtain the text mark feature. The text mark feature can include {v1+t1, v2+t2, v3+t3, v4+t4}.

[0080] like Figure 3 As shown, a text prompt mark feature can be obtained based on the text mark feature and the predetermined category mark vector 370. The text prompt mark feature is input into the text encoder M320 to obtain a text prompt feature 380.

[0081] According to an embodiment of the present disclosure, the text encoder may be a Transformer (codec) or a TextEncoder (text encoder). However, it is not limited thereto. Any text encoder that can perform feature processing on text mark features to obtain text prompt features will suffice.

[0082] According to an embodiment of the present disclosure, the predetermined class label vector [class] may refer to a class label vector used to indicate an image animal class result, such as “dog”, “cat” or “fish”.

[0083] According to an embodiment of the present disclosure, {v1+t1, v2+t2, v3+t3, v4+t4} is concatenated with a predetermined category tag vector [class] to obtain a text prompt tag feature, for example, {v1+t1, v2+t2, v3+t3, v4+t4, class}.

[0084] According to an embodiment of the present disclosure, a density peak clustering algorithm is used to decouple the overall image semantics, such as image features, into finer-grained image clustering label features. For example, a plurality of sub-image features in the same cluster can be visualized as a color by a density peak clustering algorithm. For example, all sub-image features constituting the sky are displayed in blue. The cluster can be converted into a new label, which is represented as an image clustering label vector of the [sky] semantics.

[0085] The text prompt feature provided by the embodiment of the present disclosure can be used as a marker-aware text prompt feature, which is obtained based on the image clustering marker feature that represents finer-grained image semantics, rather than being generated based on the complete image feature. This can improve the classification accuracy of unseen categories without introducing additional learning semantic information ambiguity.

[0086] According to the embodiments of the present disclosure, Figure 2 The operation S240 shown, obtaining an image classification result of the image to be processed based on the text prompt feature and the image to be processed, may include the following operations.

[0087] For example, based on the text prompt features and the image features, the visual prompt features of the text reorganization are obtained. Based on the visual prompt features of the text reorganization and the image to be processed, the image classification result is obtained.

[0088] According to an embodiment of the present disclosure, obtaining visual prompt features for text reorganization based on text prompt features and image features may include: fusing the text prompt features and image features to obtain visual prompt features for text reorganization. The fusion may refer to weighted summation of corresponding elements. However, it is not limited to this. It may also include: performing cross-attention processing on the text prompt features and image features to obtain visual prompt features for text reorganization. Any method that can reorganize the text prompt features and image features will suffice.

[0089] According to the embodiments of the present disclosure, the image features can be reorganized using the text prompt features to obtain visual prompts that align the text prompt features and the image features in the feature space, thereby improving the accuracy of the image classification results.

[0090] According to other embodiments of the present disclosure, trained image prompt features with fixed parameters may be used as visual prompt features.

[0091] Compared with the method of using image cue features as visual cue features, using visual cue features reorganized from text can avoid the problem of unstable recognition results caused by disturbances to the processed image due to fixed parameters in the visual cue features.

[0092] According to an optional embodiment of the present disclosure, obtaining visual prompt features for text reorganization based on text prompt features and image features may include: performing cross-attention processing on the text prompt features and image features to obtain an attention weight matrix. Based on the attention weight matrix, the text prompt features and the image features, the visual prompt features for text reorganization are obtained.

[0093] According to an embodiment of the present disclosure, cross-attention processing is performed on text prompt features and image features to obtain an attention weight matrix, which can be obtained by the following formula (4).

[0094] A x =f x (g t ) T ; Formula (4)

[0095] Among them, A x represents the attention weight matrix, f x The image patch vector representing the image patch corresponding to the i-th category label in the image feature, g t Represents the feature corresponding to the j-th category label in the text prompt feature.

[0096] According to an embodiment of the present disclosure, the attention weight matrix is ​​used to characterize the degree of correlation between different tag categories and image features. The image features and text prompt features are aligned in a common output space through cross-attention processing to obtain visual prompt features of text reorganization. The potential representation capacity of the visual prompt features of text reorganization can be fully released through cross-attention processing, thereby improving the generalization performance in image classification tasks.

[0097] According to an embodiment of the present disclosure, obtaining visual prompt features for text reorganization based on the attention weight matrix, text prompt features, and image features may include: obtaining weighted text prompt features based on the attention weight matrix and the text prompt features. Based on the weighted text prompt features, performing feature enhancement on image features to obtain enhanced image features. Based on the image features and the enhanced image features, obtaining visual prompt features for text reorganization.

[0098] According to an embodiment of the present disclosure, obtaining a weighted text prompt feature based on an attention weight matrix and a text prompt feature may include: normalizing the attention weight matrix to obtain a weight coefficient vector. Obtaining a weighted text prompt feature based on the weight coefficient vector and the text prompt feature.

[0099] For example, the attention weight matrix is ​​input into an activation function such as the Softmax function, and a weight coefficient vector is output. The weight coefficient vector is multiplied by the text prompt feature to obtain the weighted text prompt feature.

[0100] According to an embodiment of the present disclosure, based on the weighted text prompt feature, the image feature is enhanced to obtain the enhanced image feature, which may include: element-wise summing the weighted text prompt feature and the image feature to obtain the enhanced image feature.

[0101] According to an embodiment of the present disclosure, the enhanced image feature f a It can be obtained by the following formula (5).

[0102] f a =softmax(A x ) t +f x ; Formula (5)

[0103] According to an embodiment of the present disclosure, the obtained weight coefficient is used to weight the text prompt feature, so that the text prompt feature close to the image in the feature space is enhanced, thereby improving the prompt performance of the visual prompt feature of the text reorganization.

[0104] According to the embodiment of the present disclosure, the image feature and the enhanced image feature can be element-added to obtain the visual prompt feature of the text reorganization. However, it is not limited to this. The enhanced image feature f can also be directly added to obtain the visual prompt feature of the text reorganization. a Features as visual cues for text reorganization.

[0105] According to an embodiment of the present disclosure, the visual prompt feature f of text reorganization x a It can be obtained by the following formula (6).

[0106] f x a =f a +f x ; Formula (6)

[0107] According to the embodiments of the present disclosure, visual prompt features of text reorganization are obtained based on image features and enhanced image features, so as to avoid the visual prompt features of text reorganization destroying the original image information. As a result, the visual prompt features of text reorganization contain sufficient prompt information and prominent features, which is beneficial to improving the prompt performance of the prompt information as the image classification result.

[0108] According to the embodiments of the present disclosure, Figure 2The operation S210 shown, extracting features from the image to be processed to obtain image features, may include: segmenting the image to be processed to obtain multiple image blocks. Extracting features from the multiple image blocks respectively to obtain multiple image block vectors. Encoding the multiple image block vectors to obtain image features.

[0109] Figure 4 A schematic diagram schematically shows a visual prompt feature for generating text reorganization according to an embodiment of the present disclosure.

[0110] like Figure 4 As shown, the image to be processed 410 is segmented to obtain multiple image blocks 420. The multiple image blocks 420 are input to the input layer M410, such as the Embedding layer, to obtain multiple image block vectors 430. The multiple image block vectors 430 are input to the image encoder M420 to obtain image features 440.

[0111] like Figure 4 As shown, the image feature 440 and the text prompt feature 450 are input into the cross attention module M430 to obtain the visual prompt feature 460 of text reorganization.

[0112] According to an embodiment of the present disclosure, the image to be processed is segmented to obtain multiple image blocks, including: determining the vector dimension of the text prompt feature. According to the vector dimension of the text prompt feature, the image to be processed is segmented to obtain multiple image blocks.

[0113] For example, the text prompt feature may include {g1, g2, g3, g4}, and the vertical vector dimension is 4. Then the image to be processed is divided into 4 blocks to obtain 4 image blocks.

[0114] According to an embodiment of the present disclosure, setting the vertical vector dimension of the text prompt feature and the number of blocks of the image block to be the same can simplify the fusion process between the text prompt feature and the image feature, improve the fusion efficiency, and thus improve the classification speed.

[0115] According to an embodiment of the present disclosure, obtaining an image classification result based on the visual cue features of text reorganization and the image to be processed may include: segmenting the image to be processed to obtain multiple image blocks. Performing feature extraction on the multiple image blocks respectively to obtain multiple image block vectors. Fusing the visual cue features of text reorganization and the multiple image block vectors to obtain image fusion features. Performing image encoding on the image fusion features to obtain image encoding features. Based on the image encoding features, obtaining an image classification result.

[0116] Figure 5 A schematic diagram of generating image classification results according to an embodiment of the present disclosure is schematically shown.

[0117] like Figure 5As shown, the generation method of multiple image block vectors 510 is as follows Figure 4 The generation methods of the multiple image block vectors shown are the same and will not be repeated here.

[0118] like Figure 5 As shown, the visual prompt feature 520 for text reorganization is generated in the same manner as shown in Figure 4 The generation method of the visual prompt features of the text reorganization shown is the same and will not be repeated here.

[0119] like Figure 5 As shown, the visual cue feature 520 of the text reorganization and the multiple image block vectors 510 can be element-wise added to obtain an image fusion feature 530. The image fusion feature 530 is input into the image encoder M510 to obtain an image encoding feature 540. The image encoding feature 540 is input into the image classifier M520 to obtain an image classification result 550.

[0120] According to an embodiment of the present disclosure, the image encoder may be a Transformer (codec) or a Visual Encoder (visual encoder). However, it is not limited thereto. Any image encoder capable of performing feature processing on image fusion features may be used.

[0121] According to an embodiment of the present disclosure, the image classifier may include a fully connected layer and an activation function, but is not limited thereto, and may also be a Transformer (coder-decoder) or a Decoder (decoder), as long as it is a classifier that can obtain an image classification result based on image coding features.

[0122] According to the embodiments of the present disclosure, the image to be processed is classified in combination with the visual cue features of text reorganization to obtain an image classification result, which can fully utilize the prompt capabilities of both text prompts and visual prompts while fully releasing the potential representation capabilities of the two modalities, thereby improving the classification accuracy of unseen categories and expanding the scope of application.

[0123] According to an embodiment of the present disclosure, a visual language model (VLM) can be obtained by pre-training using visual language data. The visual language model is a powerful neural network that may include an image encoder and a text encoder to jointly encode image information and text information for various downstream tasks, performing downstream tasks such as image captioning, visual question answering, and visual dialogue. However, directly applying the pre-trained visual language model to downstream tasks may not achieve optimal performance. This is because there may be a gap between the pre-training goal and the task-specific goal.

[0124] According to other embodiments of the present disclosure, the pre-trained visual language model can be trained with downstream task training data, and the parameters in the pre-trained visual language model can be fine-tuned to bridge this gap, thereby adapting the visual language model to the results of the downstream task. However, fine-tuning the parameters in the pre-trained visual language model has the following disadvantages: for example, there are hundreds of millions or billions of parameters in visual language models such as the Contrastive Language-Image Pre-Training (CLIP) model or the A Large-scale Image and Noisy-text Embedding model, and the fine-tuning calculation cost is expensive and prone to overfitting; in addition, it may also cause the problem of forgetting learning features.

[0125] According to an embodiment of the present disclosure, for downstream tasks, prompt learning can be an effective method to improve the performance of a pre-trained visual language model. Visual prompts can be dynamically generated by processing the image to be processed in combination with a lightweight neural network. However, it is not limited to this. Text prompts can also be generated. Using the learnable prompt vectors in the visual prompts or text prompts, the knowledge of the image classification task in the downstream task is learned to improve the processing performance of the downstream task by using the learnable prompt vectors while ensuring that the model parameters in the pre-trained visual language model are fixed.

[0126] According to the embodiments of the present disclosure, the following can be generated based on the combination of the pre-trained visual language model and the task prompt module: Figure 6 The deep learning model shown. The task prompt module is used to generate sample task prompt marking features, and the sample task prompt marking features are used as text prompts. The task prompt module may include a learnable prompt vector, and the learnable prompt vector includes adjustable parameters. While ensuring that the model parameters in the pre-trained visual language model are fixed, the adjustable parameters in the task prompt module are adjusted through training so that the task prompt marking features obtained based on the task prompt module have knowledge of downstream tasks such as image classification tasks. Thereby improving the classification performance of the trained deep learning model for downstream tasks such as image classification tasks.

[0127] Figure 6 A flowchart of a deep learning model training method according to an embodiment of the present disclosure is schematically shown.

[0128] like Figure 6 As shown, the method 600 may include operations S610 to S650.

[0129] In operation S610, feature extraction is performed on the sample image to obtain sample image features.

[0130] In operation S620, the sample image features are clustered to obtain sample image clustering label features.

[0131] In operation S630 , based on the task prompt module, a sample task prompt mark feature is obtained.

[0132] In operation S640, sample text prompt features are obtained based on the sample image clustering mark features and the sample task prompt mark features.

[0133] In operation S650, based on the sample text prompt features, the sample image features and the labels, the adjustable parameters in the task prompt module are adjusted to obtain a trained deep learning model.

[0134] According to an embodiment of the present disclosure, the label is used to characterize the real image classification result of the sample image. The label and the image classification result output by the deep learning model can be used to determine the loss value, and the adjustable parameters in the task prompt module can be adjusted based on the loss value to obtain a trained deep learning model.

[0135] According to an embodiment of the present disclosure, sample text prompt features and sample image features may be input into a deep learning model to obtain image classification results.

[0136] According to other embodiments of the present disclosure, sample text prompt features may be obtained based on sample task prompt tag features.

[0137] Compared with the method of obtaining sample text prompt features based on sample task prompt marking features, the method of obtaining sample text prompt features based on sample image clustering marking features and sample task prompt marking features can make the generation of sample task prompt marking features dynamic, and use sample image clustering marking features to activate the task-related representation capabilities of sample task prompt marking features, thereby releasing the potential of visual language models.

[0138] According to other embodiments of the present disclosure, sample text prompt features may be obtained directly based on sample image features and sample task prompt mark features.

[0139] Compared with the method of obtaining sample text prompt features based on sample image features and sample task prompt marking features, the method of obtaining sample text prompt features based on sample image clustering marking features and sample task prompt marking features can cluster the sample image features so that the obtained sample image clustering marking features and the sample task prompt marking features are aligned in image semantics to avoid semantic ambiguity, thereby improving the prompt representation capability of the sample text prompt features.

[0140] According to the embodiments of the present disclosure, the above-mentioned training method can utilize the sample image clustering mark features that are aligned with the sample task prompt mark features in image semantics to improve the prompt representation capability of the sample text prompt features, thereby improving the stability of the deep learning model during the training process and improving the generalization capability of the deep learning model.

[0141] According to an optional embodiment of the present disclosure, based on the sample text prompt features and features, the sample image features and labels, adjusting the adjustable parameters in the task prompt module to obtain a trained deep learning model may include: obtaining a cross loss value based on the sample text prompt features and the sample image features. Based on the cross loss value and the label, adjusting the adjustable parameters in the task prompt module to obtain a trained deep learning model.

[0142] According to an embodiment of the present disclosure, a deep learning model may include a pre-trained language vision model. The pre-trained language vision model includes an image encoder. A sample image may be input into the image encoder to obtain sample image features. However, this is not limited to this. The sample image may also be segmented to obtain a plurality of sample image blocks. Feature extraction is performed on the plurality of sample image blocks respectively to obtain a plurality of sample image block vectors. The plurality of sample image block vectors are input into the image encoder to obtain sample image features.

[0143] According to an embodiment of the present disclosure, the training method of the deep learning model may further include: generating sample text prompt features.

[0144] According to an embodiment of the present disclosure, generating a sample text prompt feature may include: fusing a sample image clustering mark feature and a sample task prompt mark feature to obtain a sample text mark feature. Based on the sample text mark feature and a predetermined sample category mark vector, obtaining a sample text prompt mark feature. Text encoding the sample text prompt mark feature to obtain a sample text prompt feature.

[0145] According to an embodiment of the present disclosure, clustering sample image features to obtain sample image clustering label features may include: semantically decoupling the sample image features to obtain multiple sample sub-image features. Clustering the multiple sample sub-image features to obtain multiple sample clustering clusters. Based on the multiple sample clustering clusters, obtaining the sample image clustering label features.

[0146] According to an embodiment of the present disclosure, semantically decoupling a sample image feature to obtain a plurality of sample sub-image features may include: splitting the sample image feature according to a predetermined sample vector dimension to obtain a plurality of sample sub-image features. The sample vector dimension may include a sample horizontal vector dimension and a sample vertical vector dimension.

[0147] According to an embodiment of the present disclosure, clustering multiple sub-sample image features to obtain initial sample clustering label features may include: clustering multiple sample sub-image features to obtain multiple sample clustering clusters. Based on the multiple sample clustering clusters, obtaining sample clustering label features.

[0148] According to an embodiment of the present disclosure, obtaining a sample clustering label feature based on multiple sample clusters may include: for each sample cluster, performing weighted summation on multiple sample sub-image features in the sample cluster to obtain a sample image clustering label vector. Based on multiple sample image clustering label vectors corresponding one-to-one to multiple sample clusters, obtaining a sample image clustering label feature.

[0149] According to an embodiment of the present disclosure, clustering multiple sub-sample image features to obtain multiple sample clusters may include: clustering multiple sample sub-image features using a density peak clustering algorithm to obtain multiple sample clusters.

[0150] According to an embodiment of the present disclosure, a cross loss value is obtained based on sample text prompt features and sample image features, including: obtaining sample visual prompt features of text reorganization based on sample text prompt features and sample image features. A cross loss value is obtained based on sample text prompt features and sample visual prompt features of text reorganization.

[0151] According to the embodiments of the present disclosure, sample visual prompt features for text reorganization are constructed based on sample text prompt features and sample image features, thereby making the two different modal branches tightly coupled and mutually beneficial in the entire training process, thereby improving the adjustment efficiency of the adjustable parameters in the task prompt module and the training efficiency during the training process.

[0152] According to an embodiment of the present disclosure, obtaining sample visual cue features for text reorganization based on sample text cue features and sample image features may include: performing cross-attention processing on the sample text cue features and the sample image features to obtain a sample attention weight matrix. Based on the sample attention weight matrix, the sample text cue features and the sample image features, obtaining sample visual cue features for text reorganization.

[0153] According to an embodiment of the present disclosure, obtaining sample visual cue features for text reorganization based on a sample attention weight matrix, sample text cue features, and sample image features may include: obtaining weighted sample text cue features based on the sample attention weight matrix and the sample text cue features. Based on the weighted sample text cue features, performing feature enhancement on sample image features to obtain enhanced sample image features. Based on the sample image features and the enhanced sample image features, obtaining sample visual cue features for text reorganization.

[0154] According to an embodiment of the present disclosure, obtaining a weighted sample text prompt feature based on a sample attention weight matrix and a sample text prompt feature may include: normalizing the sample attention weight matrix to obtain a sample weight coefficient vector. Based on the sample weight coefficient vector and the sample text prompt feature, obtaining a weighted sample text prompt feature.

[0155] According to an embodiment of the present disclosure, based on the weighted sample text prompt feature, the sample image feature is enhanced to obtain the enhanced sample image feature, which may include: element-wise summing the weighted sample text prompt feature and the sample image feature to obtain the enhanced sample image feature.

[0156] Figure 7 A schematic diagram of adjusting adjustable parameters according to an embodiment of the present disclosure is schematically shown.

[0157] like Figure 7 As shown, the sample image 710 can be input into the image encoder M710 to obtain the sample image feature 720.

[0158] like Figure 7 As shown, the sample image features 720 may be clustered to obtain sample image clustering label features 730 .

[0159] like Figure 7 As shown, the sample task prompt labeling feature 740 may include 4 sample task prompt labeling vectors. Based on the sample vector dimension 4, the number of sample clusters of the sample clustering cluster may be determined to be 4. Based on the sample clustering clusters, the sample image clustering labeling feature 730 may be obtained. The sample image clustering labeling feature may include 4 sample image clustering labeling vectors.

[0160] like Figure 7 As shown, the sample task prompt mark feature 740 and the sample image cluster mark feature 730 may be added together by corresponding elements to obtain the sample text mark feature.

[0161] like Figure 7 As shown, the sample text prompt mark feature can be obtained based on the sample text mark feature and the predetermined sample category mark vector 750. The sample text prompt mark feature is input into the text encoder M720 to obtain the sample text prompt feature 760.

[0162] like Figure 7 As shown, the sample image feature 720 and the sample text prompt feature 760 are input into the cross attention module M730 to obtain the sample visual prompt feature 770 of text reorganization.

[0163] like Figure 7As shown, a cross loss value can be obtained based on the sample text prompt feature 760 and the sample visual prompt feature 770 of text reorganization. Based on the cross loss value and the label, the adjustable parameter is adjusted until the probability of the cross loss value corresponding to the label is the highest or converges.

[0164] According to an optional embodiment of the present disclosure, obtaining a cross loss value based on a sample text prompt feature and a sample image feature may include the following operations.

[0165] For example, based on the sample task prompt mark feature and the predetermined sample category mark vector, the misaligned sample text prompt mark feature is obtained. The misaligned sample text prompt mark feature is text-encoded to obtain the misaligned sample text prompt feature. Based on the sample image feature and the misaligned sample text prompt feature, a first cross loss value is obtained. Based on the sample text prompt feature and the sample image feature, a second cross loss value is obtained. Based on the sample text prompt feature and the sample visual prompt feature of text reorganization, a third cross loss value is obtained. Based on the first cross loss value, the second cross loss value and the third cross loss value, a cross loss value is obtained.

[0166] According to an embodiment of the present disclosure, obtaining a misaligned sample text prompt mark feature based on a sample task prompt mark feature and a predetermined sample category mark vector may include: concatenating the sample task prompt mark feature and the predetermined sample category mark vector to obtain a misaligned sample text prompt mark feature.

[0167] According to an embodiment of the present disclosure, text encoding is performed on the misaligned sample text prompt mark feature to obtain the misaligned sample text prompt feature, and the misaligned sample text prompt mark feature is input into a text encoder to obtain the misaligned sample text prompt feature.

[0168] According to an embodiment of the present disclosure, the misaligned sample text prompt feature may refer to a sample text prompt feature generated in a static manner, which is a feature without adding a sample image clustering mark feature.

[0169] According to an embodiment of the present disclosure, a cross loss value can be obtained based on one or more combinations of the first cross loss value, the second cross loss value, and the third cross loss value. The cross loss value is obtained based on the first cross loss value, the second cross loss value, and the third cross loss value, so that the reference factor of the loss value can be more sufficient, thereby making the adjustment of the adjustable parameters efficient, accurate and effective.

[0170] Figure 8 The block diagram of an image classification device according to an embodiment of the present disclosure is schematically shown.

[0171] like Figure 8As shown, the image classification device 800 includes: a feature extraction module 810 , a clustering module 820 , a text prompt generation module 830 and a classification module 840 .

[0172] The feature extraction module 810 is used to extract features from the image to be processed to obtain image features.

[0173] The clustering module 820 is used to cluster image features to obtain image clustering label features.

[0174] The text prompt generating module 830 is used to obtain text prompt features based on the image clustering mark features and the task prompt mark features. The task prompt mark features are used to represent the knowledge related to the image classification task.

[0175] The classification module 840 is used to obtain an image classification result of the image to be processed based on the text prompt feature and the image to be processed.

[0176] According to an embodiment of the present disclosure, the clustering module includes: a decoupling submodule, a clustering submodule and a clustering label generation submodule.

[0177] The decoupling submodule is used to semantically decouple image features to obtain multiple sub-image features.

[0178] The clustering submodule is used to cluster the features of multiple sub-images to obtain multiple cluster clusters.

[0179] The clustering mark generation submodule is used to obtain image clustering mark features based on multiple clustering clusters.

[0180] According to an embodiment of the present disclosure, the clustering submodule includes: a clustering unit.

[0181] The clustering unit is used to cluster the features of multiple sub-images using a density peak clustering algorithm to obtain multiple cluster clusters.

[0182] According to an embodiment of the present disclosure, the clustering label generation submodule includes: a clustering fusion unit and a clustering label generation unit.

[0183] The clustering fusion unit is used to perform weighted summation of multiple sub-image features in each cluster to obtain an image clustering label vector.

[0184] The clustering label generation unit is used to obtain image clustering label features based on multiple image clustering label vectors.

[0185] According to an embodiment of the present disclosure, the classification module includes: a recombination submodule and a classification submodule.

[0186] The reorganization submodule is used to obtain visual prompt features for text reorganization based on text prompt features and image features.

[0187] The classification submodule is used to obtain image classification results based on the visual cue features reorganized from the text and the image to be processed.

[0188] According to an embodiment of the present disclosure, the reorganization submodule includes: an attention processing unit and a reorganization unit.

[0189] The attention processing unit is used to perform cross-attention processing on text prompt features and image features to obtain an attention weight matrix.

[0190] The reorganization unit is used to obtain visual cue features for text reorganization based on the attention weight matrix, text cue features and image features.

[0191] According to an embodiment of the present disclosure, the recombination unit includes: a weighting subunit, an enhancement subunit and a recombination subunit.

[0192] The weighted subunit is used to obtain weighted text prompt features based on the attention weight matrix and the text prompt features.

[0193] The enhancement subunit is used to enhance the image features based on the weighted text prompt features to obtain enhanced image features.

[0194] The reorganization subunit is used to obtain visual prompt features for text reorganization based on image features and enhanced image features.

[0195] According to an embodiment of the present disclosure, the classification submodule includes: a segmentation unit, a feature extraction unit, an image feature fusion unit, an image encoding unit and a classification unit.

[0196] The segmentation unit is used to segment the image to be processed to obtain multiple image blocks.

[0197] The feature extraction unit is used to extract features from multiple image blocks respectively to obtain multiple image block vectors.

[0198] The image feature fusion unit is used to fuse the visual prompt features of text reorganization with multiple image block vectors to obtain image fusion features.

[0199] The image encoding unit is used to perform image encoding on the image fusion features to obtain image encoding features.

[0200] The classification unit is used to obtain image classification results based on image encoding features.

[0201] According to an embodiment of the present disclosure, the segmentation unit includes: a dimension determination subunit and a segmentation subunit.

[0202] The dimension determination subunit is used to determine the vector dimension of the text prompt feature.

[0203] The segmentation subunit is used to segment the image to be processed according to the vector dimension to obtain multiple image blocks.

[0204] According to an embodiment of the present disclosure, the text prompt generation module includes: a text feature fusion submodule, a splicing submodule and a text encoding submodule.

[0205] The text feature fusion submodule is used to fuse the image clustering mark features and the task prompt mark features to obtain the text mark features.

[0206] The splicing submodule is used to obtain text prompt tag features based on text tag features and predetermined category tag vectors.

[0207] The text encoding submodule is used to perform text encoding on the text prompt mark features to obtain text prompt features.

[0208] According to an embodiment of the present disclosure, the feature extraction module includes: an image segmentation submodule, a feature extraction submodule and an image encoding submodule.

[0209] The image segmentation submodule is used to segment the image to be processed to obtain multiple image blocks.

[0210] The feature extraction submodule is used to extract features from multiple image blocks respectively to obtain multiple image block vectors.

[0211] The image encoding submodule is used to encode multiple image block vectors to obtain image features.

[0212] Fig. 9 A block diagram of a deep learning model training device according to an embodiment of the present disclosure is schematically shown.

[0213] According to an embodiment of the present disclosure, the deep learning model includes a task prompt module, and the task prompt module includes adjustable parameters.

[0214] like Fig. 9 As shown, the training device 900 of the deep learning model includes: a sample feature extraction module 910, a sample clustering module 920, a sample task prompt generation module 930, a sample text prompt generation module 940 and a parameter adjustment module 950.

[0215] The sample feature extraction module 910 is used to extract features from the sample image to obtain sample image features.

[0216] The sample clustering module 920 is used to cluster the sample image features to obtain sample image clustering label features.

[0217] The sample task prompt generating module 930 is used to obtain the sample task prompt marking feature based on the task prompt module.

[0218] The sample text prompt generating module 940 is used to obtain the sample text prompt feature based on the sample image clustering mark feature and the sample task prompt mark feature.

[0219] The parameter adjustment module 950 is used to adjust the adjustable parameters in the task prompt module based on the sample text prompt features, sample image features and labels to obtain a trained deep learning model. The labels are used to characterize the real image classification results of the sample images.

[0220] According to an embodiment of the present disclosure, the parameter adjustment module includes: a loss value generation submodule and a parameter adjustment submodule.

[0221] The loss value generation submodule is used to obtain the cross loss value based on the sample text prompt features and the sample image features.

[0222] The parameter adjustment submodule is used to adjust the adjustable parameters in the task prompt module based on the cross loss value and the label to obtain a trained deep learning model.

[0223] According to an embodiment of the present disclosure, the training device of the deep learning model also includes: a sample text feature fusion module, a sample text feature splicing module and a sample text feature encoding module.

[0224] The sample text feature fusion module is used to fuse the sample image clustering mark features and the sample task prompt mark features to obtain the sample text mark features.

[0225] The sample text feature concatenation module is used to obtain the sample text prompt mark feature based on the sample text mark feature and the predetermined sample category mark vector.

[0226] The sample text feature encoding module is used to perform text encoding on the sample text prompt mark features to obtain the sample text prompt features.

[0227] According to an embodiment of the present disclosure, the loss value generation submodule includes: a sample visual cue generation unit and a loss value generation unit.

[0228] The sample visual cue generating unit is used to obtain sample visual cue features for text reorganization based on sample text cue features and sample image features.

[0229] The loss value generating unit is used to obtain a cross loss value based on the sample text prompt feature and the sample visual prompt feature of the text reorganization.

[0230] According to an embodiment of the present disclosure, the loss value generation submodule includes: an unaligned text mark generation unit, an unaligned text prompt generation unit, a first loss value generation unit, a second loss value generation unit, a third loss value generation unit, and a cross loss value generation unit.

[0231] The misaligned text tag generation unit is used to obtain misaligned sample text prompt tag features based on the sample task prompt tag features and the predetermined sample category tag vector.

[0232] The misaligned text prompt generating unit is used to perform text encoding on the misaligned sample text prompt mark features to obtain the misaligned sample text prompt features.

[0233] The first loss value generating unit is used to obtain a first cross loss value based on the sample image feature and the unaligned sample text prompt feature.

[0234] The second loss value generating unit is used to obtain a second cross loss value based on the sample text prompt feature and the sample image feature.

[0235] The third loss value generating unit is used to obtain a third cross loss value based on the sample text prompt feature and the sample visual prompt feature of the text reorganization.

[0236] The cross loss value generating unit is used to obtain a cross loss value based on the first cross loss value, the second cross loss value and the third cross loss value.

[0237] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium and a computer program product.

[0238] According to an embodiment of the present disclosure, an electronic device includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute a method as in the embodiment of the present disclosure.

[0239] According to an embodiment of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to cause a computer to execute a method as described in an embodiment of the present disclosure.

[0240] According to an embodiment of the present disclosure, a computer program product includes a computer program, and when the computer program is executed by a processor, the method according to the embodiment of the present disclosure is implemented.

[0241] Fig.10A schematic block diagram of an example electronic device 1000 that can be used to implement an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0242] like Fig.10 As shown, the device 1000 includes a computing unit 1001, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 1002 or a computer program loaded from a storage unit 1008 into a random access memory (RAM) 1003. In the RAM 1003, various programs and data required for the operation of the device 1000 can also be stored. The computing unit 1001, the ROM 1002, and the RAM 1003 are connected to each other via a bus 1004. An input / output (I / O) interface 1005 is also connected to the bus 1004.

[0243] A number of components in the device 1000 are connected to an input / output (I / O) interface 1005, including: an input unit 1006, such as a keyboard, a mouse, etc.; an output unit 1007, such as various types of displays, speakers, etc.; a storage unit 1008, such as a disk, an optical disk, etc.; and a communication unit 1009, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 1009 allows the device 1000 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0244] The computing unit 1001 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 1001 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 1001 performs the various methods and processes described above, such as image classification methods. For example, in some embodiments, the image classification method may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 1008. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 1000 via the ROM 1002 and / or the communication unit 1009. When the computer program is loaded into the RAM 1003 and executed by the computing unit 1001, one or more steps of the image classification method described above may be performed. Alternatively, in other embodiments, the computing unit 1001 may be configured to perform the image classification method in any other appropriate manner (e.g., by means of firmware).

[0245] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0246] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0247] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0248] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0249] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0250] A computer system may include a client and a server. The client and the server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises through computer programs running on respective computers and having a client-server relationship to each other. The server may be a cloud server, a server in a distributed system, or a server combined with a blockchain.

[0251] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0252] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. An image classification method, comprising: Perform feature extraction on the image to be processed to obtain image features; Semantically decoupling the image features to obtain multiple sub-image features; Clustering the multiple sub-image features to obtain multiple clusters; Based on the multiple clusters, an image clustering mark feature having the same vector dimension as the task prompt mark feature is obtained, wherein the task prompt mark feature is used to represent knowledge related to the image classification task; Based on the image clustering mark feature and the task prompt mark feature, a text prompt feature is obtained, wherein the text prompt feature activates the representation capability of the knowledge related to the image classification task through the image clustering mark feature; Based on the text prompt feature and the image feature, obtaining a visual prompt feature of text reorganization; as well as Based on the visual prompt features of the text reorganization and the image to be processed, an image classification result is obtained.

2. The method according to claim 1, wherein: The clustering of the plurality of sub-image features to obtain a plurality of cluster clusters includes: The plurality of sub-image features are clustered using a density peak clustering algorithm to obtain the plurality of cluster clusters.

3. The method according to claim 1, wherein: The obtaining of image clustering labeling features based on the multiple clusters includes: For each of the clusters, performing weighted summation on the features of multiple sub-images in the cluster to obtain an image cluster label vector; and Based on the multiple image clustering label vectors, the image clustering label feature is obtained.

4. The method according to claim 1, wherein: The step of obtaining visual prompt features for text reorganization based on the text prompt features and the image features includes: Performing cross attention processing on the text prompt feature and the image feature to obtain an attention weight matrix; and Based on the attention weight matrix, the text prompt feature and the image feature, a visual prompt feature of the text reorganization is obtained.

5. The method according to claim 4, wherein: The step of obtaining the visual prompt feature of the text reorganization based on the attention weight matrix, the text prompt feature and the image feature includes: Based on the attention weight matrix and the text prompt feature, obtaining a weighted text prompt feature; Based on the weighted text prompt feature, the image feature is enhanced to obtain an enhanced image feature; and Based on the image features and the enhanced image features, visual prompt features of the text reorganization are obtained.

6. The method according to claim 4 or 5, wherein: The image classification result is obtained based on the visual prompt features of the text reorganization and the image to be processed, including: Segmenting the image to be processed to obtain a plurality of image blocks; Performing feature extraction on the multiple image blocks respectively to obtain multiple image block vectors; Fusing the text reorganized visual prompt feature with the multiple image block vectors to obtain an image fusion feature; Performing image encoding on the image fusion feature to obtain an image encoding feature; and Based on the image coding features, the image classification result is obtained.

7. The method according to claim 6, wherein: The step of segmenting the image to be processed to obtain a plurality of image blocks includes: Determining the vector dimension of the text prompt feature; and The image to be processed is segmented according to the vector dimension to obtain the multiple image blocks.

8. The method according to any one of claims 1 to 5, wherein: The obtaining of text prompt features based on the image clustering mark features and the task prompt mark features includes: fusing the image clustering mark feature and the task prompt mark feature to obtain a text mark feature; Based on the text mark feature and the predetermined category mark vector, a text prompt mark feature is obtained; and The text prompt mark feature is text-encoded to obtain the text prompt feature.

9. The method according to any one of claims 1 to 5, wherein: The step of extracting features from the image to be processed to obtain image features includes: Segmenting the image to be processed to obtain a plurality of image blocks; Performing feature extraction on the multiple image blocks respectively to obtain multiple image block vectors; and The multiple image block vectors are encoded to obtain the image features.

10. A method for training a deep learning model, wherein: The deep learning model includes a task prompt module, the task prompt module includes adjustable parameters, and the method includes: Extract features from the sample image to obtain features of the sample image; Semantically decoupling the sample image features to obtain multiple sample sub-image features; Clustering the features of the multiple sample sub-images to obtain multiple sample clusters; Based on the multiple sample clusters, a sample image clustering mark feature having the same vector dimension as the sample task prompt mark feature is obtained; Based on the task prompt module, a sample task prompt mark feature is obtained, wherein the sample task prompt mark feature is used to represent knowledge related to the image classification task; Based on the sample image clustering mark feature and the sample task prompt mark feature, a sample text prompt feature is obtained, wherein the sample text prompt feature activates the representation capability of knowledge related to the image classification task through the sample image clustering mark feature; Obtaining a cross loss value based on the sample text prompt feature and the sample image feature; and Based on the cross loss value and the label, the adjustable parameters in the task prompt module are adjusted to obtain a trained deep learning model, wherein the label is used to characterize a real image classification result of a sample image.

11. The method according to claim 10, further comprising: The sample image clustering mark feature and the sample task prompt mark feature are merged to obtain a sample text mark feature; Based on the sample text mark feature and the predetermined sample category mark vector, obtaining the sample text prompt mark feature; and Text encoding is performed on the sample text prompt mark feature to obtain the sample text prompt feature.

12. The method according to claim 10, wherein: The obtaining the cross loss value based on the sample text prompt feature and the sample image feature includes: Based on the sample text prompt feature and the sample image feature, obtaining a sample visual prompt feature for text reorganization; and The cross loss value is obtained based on the sample text prompt feature and the sample visual prompt feature of the text reorganization.

13. The method according to claim 12, wherein: The obtaining the cross loss value based on the sample text prompt feature and the sample image feature includes: Based on the sample task prompt mark feature and the predetermined sample category mark vector, obtaining the misaligned sample text prompt mark feature; Performing text encoding on the misaligned sample text prompt mark feature to obtain the misaligned sample text prompt feature; Obtaining a first cross loss value based on the sample image feature and the unaligned sample text prompt feature; Obtaining a second cross loss value based on the sample text prompt feature and the sample image feature; Obtaining a third cross loss value based on the sample text prompt feature and the sample visual prompt feature of the text reorganization; and The cross loss value is obtained based on the first cross loss value, the second cross loss value and the third cross loss value.

14. An image classification device, comprising: A feature extraction module is used to extract features from the image to be processed to obtain image features; A clustering module, used for clustering the image features to obtain image clustering marking features; a text prompt generating module, configured to obtain a text prompt feature based on the image clustering mark feature and the task prompt mark feature, wherein the task prompt mark feature is used to represent knowledge related to the image classification task, and the text prompt feature activates the representation capability of the knowledge related to the image classification task through the image clustering mark feature; and A classification module, used for obtaining an image classification result of the image to be processed based on the text prompt feature and the image to be processed; Wherein, the clustering module includes: A decoupling submodule, used for semantically decoupling the image features to obtain a plurality of sub-image features; A clustering submodule, used for clustering the plurality of sub-image features to obtain a plurality of cluster clusters; and A clustering mark generation submodule, used for obtaining the image clustering mark feature having the same vector dimension as the task prompt mark feature based on the multiple clusters; Wherein, the classification module includes: a reorganization submodule, used to obtain visual prompt features for text reorganization based on the text prompt features and the image features; and The classification submodule is used to obtain the image classification result based on the visual prompt features of the text reorganization and the image to be processed.

15. The device according to claim 14, wherein: The clustering submodule includes: The clustering unit is used to cluster the multiple sub-image features using a density peak clustering algorithm to obtain the multiple cluster clusters.

16. The device according to claim 14, wherein: The cluster marker generation submodule includes: A clustering fusion unit, configured to perform weighted summation on the features of multiple sub-images in each cluster to obtain an image clustering label vector; and A clustering label generating unit is used to obtain the image clustering label feature based on a plurality of the image clustering label vectors.

17. The device according to claim 14, wherein: The recombinant module comprises: An attention processing unit is used to perform cross attention processing on the text prompt feature and the image feature to obtain an attention weight matrix; and A reorganization unit is used to obtain the visual prompt feature of the text reorganization based on the attention weight matrix, the text prompt feature and the image feature.

18. The device according to claim 17, wherein: The recombination unit comprises: A weighting subunit, used for obtaining a weighted text prompt feature based on the attention weight matrix and the text prompt feature; an enhancement subunit, configured to enhance the image feature based on the weighted text prompt feature to obtain an enhanced image feature; and The reorganization subunit is used to obtain the visual prompt feature of the text reorganization based on the image feature and the enhanced image feature.

19. The device according to claim 14 or 15, wherein: The classification submodule includes: A segmentation unit, used for segmenting the image to be processed to obtain a plurality of image blocks; A feature extraction unit, used to extract features from the multiple image blocks respectively to obtain multiple image block vectors; An image feature fusion unit, used for fusing the visual prompt feature of the text reorganization with the multiple image block vectors to obtain an image fusion feature; an image encoding unit, configured to perform image encoding on the image fusion feature to obtain an image encoding feature; and A classification unit is used to obtain the image classification result based on the image coding feature.

20. The device according to claim 19, wherein The segmentation unit comprises: a dimension determination subunit, used to determine the vector dimension of the text prompt feature; and The segmentation subunit is used to segment the image to be processed according to the vector dimension to obtain the multiple image blocks.

21. The device according to any one of claims 14 to 18, wherein: The text prompt generation module comprises: A text feature fusion submodule, used for fusing the image clustering mark feature and the task prompt mark feature to obtain a text mark feature; A splicing submodule, configured to obtain a text prompt mark feature based on the text mark feature and a predetermined category mark vector; and The text encoding submodule is used to perform text encoding on the text prompt mark feature to obtain the text prompt feature.

22. The device according to any one of claims 14 to 18, wherein: The feature extraction module comprises: An image segmentation submodule, used for segmenting the image to be processed to obtain a plurality of image blocks; a feature extraction submodule, configured to perform feature extraction on the plurality of image blocks respectively to obtain a plurality of image block vectors; and The image encoding submodule is used to encode the multiple image block vectors to obtain the image features.

23. A training device for a deep learning model, wherein: The deep learning model includes a task prompt module, the task prompt module includes adjustable parameters, and the device includes: A sample feature extraction module is used to extract features from sample images to obtain sample image features; A sample clustering module, used to cluster the sample image features to obtain sample image clustering marking features; A sample task prompt generating module, used for obtaining a sample task prompt marking feature based on the task prompt module; a sample text prompt generating module, configured to obtain a sample text prompt feature based on the sample image clustering mark feature and the sample task prompt mark feature, wherein the sample text prompt feature activates the representation capability of knowledge related to the image classification task through the sample image clustering mark feature; and A parameter adjustment module, used to adjust the adjustable parameters in the task prompt module based on the sample text prompt feature, the sample image feature and the label to obtain a trained deep learning model, wherein the label is used to characterize the real image classification result of the sample image; Wherein, the parameter adjustment module includes: A loss value generating submodule, used for obtaining a cross loss value based on the sample text prompt feature and the sample image feature; and A parameter adjustment submodule, used for adjusting the adjustable parameters in the task prompt module based on the cross loss value and the label to obtain the trained deep learning model; The step of clustering the sample image features to obtain sample image clustering label features includes: Semantically decoupling the sample image features to obtain multiple sample sub-image features; Clustering the features of the multiple sample sub-images to obtain multiple sample clusters; Based on the multiple sample clusters, a sample image clustering mark feature having the same vector dimension as the sample task prompt mark feature is obtained.

24. The apparatus according to claim 23, further comprising: A sample text feature fusion module, used to fuse the sample image clustering mark feature and the sample task prompt mark feature to obtain a sample text mark feature; A sample text feature concatenation module, used to obtain a sample text prompt mark feature based on the sample text mark feature and a predetermined sample category mark vector; and The sample text feature encoding module is used to perform text encoding on the sample text prompt mark feature to obtain the sample text prompt feature.

25. The device according to claim 23, wherein: The loss value generation submodule includes: A sample visual cue generating unit, configured to obtain sample visual cue features for text reorganization based on the sample text cue features and the sample image features; and The loss value generating unit is used to obtain the cross loss value based on the sample text prompt feature and the sample visual prompt feature of the text reorganization.

26. The device according to claim 25, wherein The loss value generation submodule includes: An unaligned text mark generating unit, used for obtaining unaligned sample text prompt mark features based on the sample task prompt mark features and the predetermined sample category mark vector; A misaligned text prompt generating unit, used for performing text encoding on the misaligned sample text prompt mark feature to obtain a misaligned sample text prompt feature; A first loss value generating unit, configured to obtain a first cross loss value based on the sample image feature and the misaligned sample text prompt feature; A second loss value generating unit, configured to obtain a second cross loss value based on the sample text prompt feature and the sample image feature; A third loss value generating unit, configured to obtain a third cross loss value based on the sample text prompt feature and the sample visual prompt feature of the text reorganization; and A cross loss value generating unit is used to obtain the cross loss value based on the first cross loss value, the second cross loss value and the third cross loss value.

27. An electronic device comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 13.

28. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1 to 13.

29. A computer program product comprising a computer program which, when executed by a processor, implements the method according to any one of claims 1 to 13.

Images