Image classification method and device, and computer readable storage medium
Patent Information
- Application Number
- PCT/CN2024/118204
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2024-03-08
- Filing Date
- 2024-09-11
- Publication Date
- 2025-10-02
AI Technical Summary
The existing image classification technology has low efficiency and high cost, is unable to automatically obtain and mark the point information of the surveillance camera, and cannot achieve high-accuracy camera location information positioning.
The Lance and Williams Distance uniform distribution probability prompt learning method based on a large model is adopted. The feature vector is obtained through the text encoder and image encoder. The uniform probability distribution and reparameterization module are used to improve the accuracy and efficiency of image classification, and the automatic acquisition and labeling of camera point information is realized.
It improves the accuracy and efficiency of image classification, reduces costs, and enables automatic acquisition and labeling of point information from millions of cameras in the visual network.
Smart Images
Figure CN2024118204_02102025_PF_FP_ABST
Abstract
Description
Image classification method, device and computer-readable storage medium
[0001] CROSS-REFERENCE TO RELATED APPLICATIONS
[0002] This application is based on the application with CN application number 202410268855.6 and application date March 8, 2024, and claims its priority. The disclosed content of the CN application is hereby introduced as a whole into this application. Technical Field
[0003] The present disclosure relates to the field of image processing technology, and in particular to an image classification method, an image classification device, a computer-readable storage medium, and a computer program product. Background Art
[0004] With the rapid development of 5G, big data, and artificial intelligence, video has become ubiquitous in our daily lives and work. However, in some video application scenarios, such as video surveillance, camera location information is often missing. For example, if a building has 50 cameras, it's impossible to determine their locations, including which cameras are located in stairways, hallways, meeting rooms, or kitchens.
[0005] In the related art, the images of the surveillance video are first observed manually, and then the images are manually classified to mark the point information for each camera one by one.
[0006] Summary of the Invention
[0007] According to some embodiments of the present disclosure, an image classification method is provided, comprising: inputting a plurality of prompt templates into a text encoder to obtain a first text feature vector corresponding to each of the plurality of prompt templates, each prompt template including text prompt information of an image category; inputting an image to be classified into an image encoder to obtain an image feature vector; and determining the image category to which the image to be classified belongs based on a degree of similarity between the image feature vector and the first text feature vector corresponding to each prompt template.
[0008] In some embodiments, the image to be classified is an image frame in a video; the image category includes a preset scene category, which is used to determine the position information of the image acquisition device for acquiring the video.
[0009] In some embodiments, inputting multiple prompt templates into a text encoder to obtain a first text feature vector corresponding to each of the multiple prompt templates includes: inputting multiple prompt templates into a text encoder to obtain a second text feature vector corresponding to each prompt template; sampling the first text feature vector from the probability distribution obeyed by multiple components in the second text feature vector.
[0010] In some embodiments, sampling the first text feature vector from the probability distribution obeyed by multiple components in the second text feature vector includes: determining a uniform probability distribution obeyed by the multiple components based on the maximum component and the minimum component in the first text feature vector, for sampling the first text feature vector.
[0011] In some embodiments, the uniform probability distribution is negatively correlated with the difference between the maximum component and the minimum component, and positively correlated with the difference between the component and the minimum component.
[0012] In some embodiments, sampling the first text feature vector from the probability distribution obeyed by multiple components in the second text feature vector includes: sampling the first text feature vector according to statistical characteristics of the probability distribution.
[0013] In some embodiments, sampling the first text feature vector according to the statistical characteristics of the probability distribution includes: determining the first text feature vector according to a weighted sum of a mean and a variance of the probability distribution.
[0014] In some embodiments, determining the image category to which the image to be classified belongs based on the degree of similarity between the image feature vector and the first text feature vector corresponding to each prompt template includes: calculating the distance between the image feature vector and each first text feature vector; determining the membership probability of the image to be classified belonging to each image category based on the distance; and determining the image category with the largest membership probability as the image category to which the image to be classified belongs.
[0015] In some embodiments, calculating the distance between the image feature vector and each first text feature vector includes: calculating the distance based on the sum of the absolute values of the components in the image feature vector and the absolute values of the components in each first text feature vector, and the absolute values of the components in the image feature vector and the sum of the components in each first text feature vector.
[0016] According to other embodiments of the present disclosure, an image classification device is provided, including: a text encoder for obtaining a first text feature vector corresponding to each of the multiple prompt templates based on multiple input prompt templates, each prompt template including text prompt information of an image category; an image encoder for obtaining an image feature vector based on an input image to be classified; and a classifier for determining the image category to which the image to be classified belongs based on the degree of similarity between the image feature vector and the first text feature vector corresponding to each prompt template.
[0017] In some embodiments, the image to be classified is an image frame in a video; the image category includes a preset scene category, which is used to determine the position information of the image acquisition device for acquiring the video.
[0018] In some embodiments, the text encoder obtains a second text feature vector corresponding to each prompt template based on multiple input prompt template texts, and samples a first text feature vector from the probability distribution obeyed by multiple components in the second text feature vector.
[0019] In some embodiments, the text encoder determines, based on the maximum component and the minimum component in the first text feature vector, a uniform probability distribution obeyed by multiple components for sampling the first text feature vector.
[0020] In some embodiments, the uniform probability distribution is negatively correlated with the difference between the maximum component and the minimum component, and positively correlated with the difference between the component and the minimum component.
[0021] In some embodiments, the text encoder samples the first text feature vector based on statistical characteristics of the probability distribution.
[0022] In some embodiments, the text encoder determines the first text feature vector based on a weighted sum of a mean and a variance of a probability distribution.
[0023] In some embodiments, the classifier calculates the distance between the image feature vector and each first text feature vector, determines the probability of the image to be classified belonging to each image category based on the distance, and determines the image category with the largest probability of belonging as the image category to which the image to be classified belongs.
[0024] In some embodiments, the classifier calculates the distance based on the absolute value of the component in the image feature vector and the absolute value of the component in each first text feature vector, and the absolute value of the sum of the component in the image feature vector and the component in each first text feature vector.
[0025] According to some further embodiments of the present disclosure, an image classification device is provided, comprising: a memory; and a processor coupled to the memory, wherein the processor is configured to execute the image classification method in any of the above embodiments based on instructions stored in the memory device.
[0026] According to some further embodiments of the present disclosure, a computer-readable storage medium is provided, on which a computer program is stored. When the program is executed by a processor, the image classification method in any of the above embodiments is implemented.
[0027] According to still further embodiments of the present disclosure, a computer program product is provided, comprising instructions, which, when executed by a processor, cause the processor to perform the image classification method in any one of the above embodiments. BRIEF DESCRIPTION OF THE DRAWINGS
[0028] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments of the present disclosure and, together with the description, serve to explain the principles of the present disclosure.
[0029] The present disclosure can be more clearly understood from the following detailed description with reference to the accompanying drawings:
[0030] FIG1 shows a flowchart of some embodiments of the image classification method of the present disclosure;
[0031] FIG2 is a schematic diagram showing some embodiments of the image classification method of the present disclosure;
[0032] FIG3 shows a flowchart of another embodiment of the image classification method of the present disclosure;
[0033] FIG4 shows a block diagram of some embodiments of the image classification apparatus of the present disclosure;
[0034] FIG5 shows a block diagram of some other embodiments of the image classification apparatus disclosed herein;
[0035] FIG6 shows a block diagram of some further embodiments of the image classification apparatus disclosed herein. DETAILED DESCRIPTION
[0036] Various exemplary embodiments of the present disclosure will now be described in detail with reference to the accompanying drawings. It should be noted that unless otherwise specifically stated, the relative arrangement of components and steps, numerical expressions and numerical values set forth in these embodiments do not limit the scope of the present disclosure.
[0037] At the same time, it should be understood that for the convenience of description, the sizes of the various parts shown in the drawings are not drawn according to the actual proportional relationship.
[0038] The following description of at least one exemplary embodiment is merely illustrative in nature and is in no way intended to limit the present disclosure, its application, or uses.
[0039] Technologies, methods, and equipment known to ordinary technicians in the relevant art may not be discussed in detail, but where appropriate, the technologies, methods, and equipment should be considered part of the specification.
[0040] In all examples shown and discussed herein, any specific values should be interpreted as merely exemplary and not limiting. Therefore, other examples of the exemplary embodiments may have different values.
[0041] It should be noted that like reference numerals and letters refer to like items in the following figures, and therefore, once an item is defined in one figure, it need not be further discussed in subsequent figures.
[0042] The inventors of the present disclosure have discovered that the above-mentioned related art has the following problems: the efficiency of image classification is low and the cost is high.
[0043] In view of this, the present disclosure proposes an image classification technology solution that can improve the efficiency of image classification and reduce costs.
[0044] As mentioned earlier, the operator's visual network management platform has approximately 30 million front-end cameras. Manual video monitoring is extremely time-consuming and labor-intensive. Furthermore, simply using AI and deep learning algorithms for classification cannot meet the accuracy, low latency, and abstract semantics requirements of intelligent video analysis tasks.
[0045] The 5G era has spawned a vast array of video-based applications, such as smart and safe cities, the internet, autonomous driving, video surveillance, and other image- and video-based content. Video applications are extremely broad, and image- and video-related applications will become a major source of incremental traffic in the 5G and post-5G eras. However, video systems face the following technical challenges: Video surveillance systems cannot automatically acquire location information for each camera; they cannot automatically label location information for each camera; and they cannot accurately locate camera locations based on keywords in location information.
[0046] To address the above technical issues, the disclosed technical solution employs a point determination method and system based on Lance and Williams Distance uniformly distributed probabilistic cue learning using a large model. This method utilizes context-aware probabilistic cue learning to fine-tune the parameters of the large model, mitigating the adverse effects of uncertainty. This probabilistic cue learning method enables the model to better understand and process the connection between vision and language, resulting in high performance and generalization capabilities.
[0047] To automatically acquire the location information of each camera in a visual network system, this paper designs a large-scale model-based uniformly distributed probability prompt learning method and an image classification method based on Lance and Williams Distance. This enables automatic acquisition and labeling of location information for each camera in the visual network. This system solves the problem of acquiring the location information of millions of cameras in the visual network.
[0048] For example, the technical solutions of the present disclosure can be implemented through the following embodiments.
[0049] FIG1 shows a flowchart of some embodiments of the image classification method disclosed herein.
[0050] As shown in FIG1 , in step 110 , a plurality of prompt templates are input into a text encoder to obtain a first text feature vector corresponding to each of the plurality of prompt templates, where each prompt template includes text prompt information of an image category.
[0051] In some embodiments, the image to be classified is an image frame in a video; the image category includes a preset scene category, which is used to determine the position information of the image acquisition device for acquiring the video.
[0052] For example, text prompt information can be obtained through model learning, and a prompt template T can be constructed using multiple text prompt information:
[0053] t 1 ,t 2 ,…t k ,…t K As shown below:
[0054] L is the set prompt word length. T in formulas (1) and (2) constitutes a prompt template including K scene categories to be judged.
[0055] For example: 1 ={This is the kitchen scene}; t 2 ={This is the scene of the community carport}; t 3 ={This is the scene of the entrance and exit of the venue}; t 4 ={This is a scene of underground space, parking lot, basement, etc.}
[0056] In some embodiments, multiple hint templates are input into a text encoder to obtain a second text feature vector corresponding to each hint template.
[0057] For example, a text encoder It can be a Transformer model, prompting the template T to pass through the text encoder After processing, the output second text feature vector is e:
[0058] K is the number of scene categories to be judged, e1,e2,e3,…e k ,…e K etc. are as follows:
[0059] e k is the prompt template for the kth scene category.
[0060] In some embodiments, the first text feature vector is sampled from a probability distribution obeyed by multiple components in the second text feature vector. For example, a uniform probability distribution obeyed by multiple components can be determined based on the maximum and minimum components in the first text feature vector to be used for sampling the first text feature vector.
[0061] For example, in the Probabilistic Embedding module, each e k The probability distribution function p(u|e k ) is determined as a uniform probability distribution with parameters α and β: p(u|e k )~U(α,β) (5)
[0062] α is the text encoder The second text feature e of the output k The components of The minimum value in , β is the text encoder The second text feature e of the output k The components of The maximum value in .
[0063] In the above embodiment, the uniform probability model can be understood as a probability distribution of possible representations to be classified, which can reflect the visual context knowledge from the input image. In this way, by considering the visual context knowledge, the accuracy of image classification can be improved.
[0064] In some embodiments, the uniform probability distribution is negatively correlated with the difference between the maximum component and the minimum component, and positively correlated with the difference between the component and the minimum component.
[0065] For example, p(u|e k ) obeys the uniform probability distribution function:
[0066] For example, take e1 as an example:
[0067] α is The minimum value in , β is The maximum value in .
[0068] In some embodiments, the first text feature vector is sampled from the probability distribution obeyed by multiple components in the second text feature vector. For example, from p(u|e k ) The first text feature vector u sampled from can be expressed as follows:
[0069] For example, u1,u2,u3,…u k ,…u K It can be expressed as follows:
[0070] u k Hint templates representing k scene categories.
[0071] In some embodiments, the first text feature vector is sampled based on statistical characteristics of the probability distribution, for example, the first text feature vector is determined based on a weighted sum of a mean and a variance of the probability distribution.
[0072] For example, based on the reparameterization module, e k Reparameterize to u k : u k =θ(e k )+ρ(e k ) (11)
[0073] θ(e k ) is p(u|e k ), ρ(e k ) is p(u|e k For example, visual and textual features can be input into a visual context probability encoder to define a probabilistic embedding space for image classification, thereby improving the accuracy of image classification.
[0074] In step 120, the image to be classified is input into an image encoder to obtain an image feature vector.
[0075] For example, the image to be classified can be input into the image encoder ε, and the image feature vector m of the image to be classified can be obtained through encoding calculation. b .
[0076] In step 130 , the image category to which the image to be classified belongs is determined based on the similarity between the image feature vector and the first text feature vector corresponding to each prompt template.
[0077] In the above embodiment, image classification is performed based on pre-set text prompt information for describing different image categories by comparing the similarity between the text prompt information and the features of the image, thereby improving the efficiency of image classification and reducing costs.
[0078] In some embodiments, the distance between the image feature vector and each first text feature vector is calculated; based on the distance, the probability of the image to be classified belonging to each image category is determined; and the image category with the largest membership probability is determined as the image category to which the image to be classified belongs.
[0079] For example, after processing by the reparameterization module, the first text feature vector can be obtained Calculate the output features m of the image encoder ε b with u kThe Lance and Williams Distance (Lance and Williams Distance) between them is used to calculate the similarity function based on the Lance and Williams distance to obtain the final result of the scene to be judged.
[0080] In some embodiments, the distance is calculated based on the sum of the absolute values of the components in the image feature vector and the absolute values of the components in each first text feature vector, and the absolute value of the sum of the components in the image feature vector and the components in each first text feature vector.
[0081] For example, we can first calculate the image feature vector m of the image to be classified b and the first text feature vector u k The distance between (b,k) :
[0082] is m b The vector representation of for u k The vector representation of m is calculated b and u k γ between (b,k) .
[0083] In some embodiments, the probability of the image to be classified belonging to each image category is determined based on the distance. For example, the probability of the image to be classified belonging to each image category can be determined based on the distance. (b,k) After the value, the value of Lance and Williams Distance can be calculated
[0084] For example, we can calculate the membership probability δ of image m belonging to category k in text u k |(m,u):
[0085] K is the total number of text categories to be understood, δ k |(m,u) is the probability that image m belongs to category k in document u. The category with the highest probability is the scene category in document u to which the image belongs.
[0086] In the above embodiments, in order to solve the technical problem that the visual network cannot automatically obtain and label the point information of each camera and can only judge the point scene information of the camera by watching the video with the human eye, a point determination method and system based on the Lance and Williams Distance uniform distribution probability prompt learning based on a large model is proposed.
[0087] Context-aware probabilistic learning is used to fine-tune the parameters of large models to mitigate the adverse effects of uncertainty. This probabilistic learning approach allows the model to better understand and process the connection between vision and language, with higher performance and generalization capabilities, thereby improving image classification accuracy.
[0088] Furthermore, through a large-scale model-based uniformly distributed probability hint learning method and a pixel-to-text matching loss calculation method based on the Lance and Williams Distance, the system automatically acquires and labels the location information of each camera in the visual network. This system solves the problem of determining the location of millions of cameras in the visual network.
[0089] FIG2 is a schematic diagram showing some embodiments of the image classification method disclosed herein.
[0090] As shown in Figure 2, the uniform distribution probability hint learning module structure based on the large model includes the image encoder ε and the text encoder For example, the input image to the image encoder could be a picture of a kitchen scene, and the input to the text encoder could be K types of prompt templates. After passing through the text encoder, the K types of prompt templates, along with the encoded features from the image encoder, are input into a uniformly distributed deterministic embedding module, a reparameterization module, and other modules.
[0091] Text Encoder It can be a Transformer model, prompting the template T to pass through the text encoder After processing, the output second text feature vector is e.
[0092] We can embed the module in a probabilistic way, k The probability distribution function p(u|e k ) is determined as a uniform probability distribution with parameters α and β. From p(u|e k ) to sample the first text feature vector u.
[0093] Based on the reparameterization module, e k Reparameterize to u k After the reparameterization module is processed, the first text feature vector can be obtained Calculate the output features m of the image encoder ε b with u k The Lance and Williams Distance between them is used to calculate the similarity function and obtain the final result of the scene to be judged.
[0094] For example, we can first calculate the image feature vector m of the image to be classified band the first text feature vector u k γ (b,k) ; Then calculate the value of Lance and Williams Distance Finally, calculate the membership probability δ of image m belonging to category k in text u k |(m,u), the category with the highest probability, is the scene category in the text u to which the image belongs.
[0095] FIG3 shows a flowchart of other embodiments of the image classification method disclosed herein.
[0096] As shown in FIG3 , in steps 310 to 330, the text encoder It can be a Transformer model, prompting the template T to pass through the text encoder After processing, the output second text feature vector is e.
[0097] In step 340, the image to be classified is input into the image encoder ε, and the image feature vector m of the image to be classified is obtained through encoding calculation. b .
[0098] In step 350, in the probabilistic embedding module, each e k The probability distribution function p(u|e k ) is determined as a uniform probability distribution with parameters α and β.
[0099] In step 360, based on the reparameterization module, e k Reparameterize to u k After the reparameterization module is processed, the first text feature vector can be obtained
[0100] In step 370, the image feature vector m of the image to be classified is first calculated. b and the first text feature vector u k γ (b,k) ; Then calculate the value of Lance and Williams Distance
[0101] In step 380, the membership probability δ of the image m belonging to the category k in the text u is calculated. k |(m,u), the category with the highest probability, is the scene category in the text u to which the image belongs.
[0102] In step 390 , the scene category in the text u to which the image belongs is output.
[0103] FIG4 shows a block diagram of some embodiments of the image classification apparatus of the present disclosure.
[0104] As shown in Figure 4, the image classification device 4 includes: a text encoder 41, which is used to obtain a first text feature vector corresponding to each of the multiple prompt templates based on the input of multiple prompt templates, and each prompt template includes text prompt information of an image category; an image encoder 42, which is used to obtain an image feature vector based on the input of the image to be classified; and a classifier 43, which is used to determine the image category to which the image to be classified belongs based on the degree of similarity between the image feature vector and the first text feature vector corresponding to each prompt template.
[0105] In some embodiments, the image to be classified is an image frame in a video; the image category includes a preset scene category, which is used to determine the position information of the image acquisition device for acquiring the video.
[0106] In some embodiments, the text encoder 41 obtains a second text feature vector corresponding to each prompt template based on multiple input prompt template texts, and samples a first text feature vector from the probability distribution obeyed by multiple components in the second text feature vector.
[0107] In some embodiments, the text encoder 41 determines, based on the maximum component and the minimum component in the first text feature vector, a uniform probability distribution obeyed by multiple components for sampling the first text feature vector.
[0108] In some embodiments, the uniform probability distribution is negatively correlated with the difference between the maximum component and the minimum component, and positively correlated with the difference between the component and the minimum component.
[0109] In some embodiments, the text encoder 41 samples the first text feature vector according to the statistical characteristics of the probability distribution.
[0110] In some embodiments, the text encoder 41 determines the first text feature vector according to a weighted sum of a mean and a variance of the probability distribution.
[0111] In some embodiments, the classifier 43 calculates the distance between the image feature vector and each first text feature vector, determines the probability of the image to be classified belonging to each image category based on the distance, and determines the image category with the largest probability of belonging as the image category to which the image to be classified belongs.
[0112] In some embodiments, the classifier 43 calculates the distance based on the absolute value of the component in the image feature vector and the absolute value of the component in each first text feature vector, and the absolute value of the sum of the component in the image feature vector and the component in each first text feature vector.
[0113] FIG5 shows a block diagram of some other embodiments of the image classification apparatus disclosed herein.
[0114] As shown in FIG5 , the image classification device 5 of this embodiment includes: a memory 51 and a processor 52 coupled to the memory 51 , and the processor 52 is configured to execute the image classification method in any embodiment of the present disclosure based on instructions stored in the memory 51 .
[0115] The memory 51 may include, for example, a system memory, a fixed non-volatile storage medium, etc. The system memory may store, for example, an operating system, application programs, a boot loader, a database, and other programs.
[0116] FIG6 shows a block diagram of some further embodiments of the image classification apparatus disclosed herein.
[0117] As shown in FIG6 , the image classification device 6 of this embodiment includes: a memory 610 and a processor 620 coupled to the memory 610 . The processor 620 is configured to execute the image classification method of any of the aforementioned embodiments based on instructions stored in the memory 610 .
[0118] The memory 610 may include, for example, a system memory, a fixed non-volatile storage medium, etc. The system memory may store, for example, an operating system, application programs, a boot loader, and other programs.
[0119] The image classification device 6 may further include an input / output interface 630, a network interface 640, a storage interface 650, and the like. These interfaces 630, 640, 650, as well as the memory 610 and the processor 620, may be connected, for example, via a bus 660. The input / output interface 630 provides a connection interface for input / output devices such as a display, mouse, keyboard, touch screen, microphone, and speakers. The network interface 640 provides a connection interface for various networked devices. The storage interface 650 provides a connection interface for external storage devices such as SD cards and USB flash drives.
[0120] Those skilled in the art will appreciate that embodiments of the present disclosure may be provided as methods, systems, or computer program products. Thus, the present disclosure may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware. Furthermore, the present disclosure may take the form of a computer program product implemented on one or more computer-usable non-transitory storage media, including but not limited to magnetic disk storage, CD-ROMs, optical storage, and the like, containing computer-usable program code.
[0121] The image classification method, image classification device, computer-readable storage medium, and computer program product according to the present disclosure have been described in detail. To avoid obscuring the concepts of the present disclosure, some details known in the art have been omitted. Based on the above description, those skilled in the art will fully understand how to implement the technical solutions disclosed herein.
[0122] The methods and systems of the present disclosure may be implemented in many ways. For example, the methods and systems of the present disclosure may be implemented by software, hardware, firmware, or any combination of software, hardware, and firmware. The above order of steps for the method is for illustration only, and the steps of the method of the present disclosure are not limited to the order specifically described above unless otherwise specified. In addition, in some embodiments, the present disclosure may also be implemented as programs recorded in a recording medium, which include machine-readable instructions for implementing the methods according to the present disclosure. Therefore, the present disclosure also covers recording media that store programs for executing the methods according to the present disclosure.
[0123] Although some specific embodiments of the present disclosure have been described in detail by way of examples, those skilled in the art will appreciate that the above examples are for illustrative purposes only and are not intended to limit the scope of the present disclosure. Those skilled in the art will appreciate that modifications may be made to the above embodiments without departing from the scope and spirit of the present disclosure. The scope of the present disclosure is defined by the appended claims.
Claims
1. An image classification method, comprising: Inputting a plurality of prompt templates into a text encoder to obtain a first text feature vector corresponding to each of the plurality of prompt templates, wherein each prompt template includes text prompt information of an image category; Input the image to be classified into the image encoder to obtain the image feature vector; The image category to which the image to be classified belongs is determined according to the similarity between the image feature vector and the first text feature vector corresponding to each prompt template.
2. The image classification method according to claim 1, wherein: The image to be classified is an image frame in a video; The image category includes a preset scene category, which is used to determine the position information of the image acquisition device that acquires the video.
3. The image classification method according to claim 1, wherein: Inputting the plurality of prompt templates into a text encoder to obtain a first text feature vector corresponding to each of the plurality of prompt templates includes: Inputting the plurality of prompt templates into the text encoder to obtain a second text feature vector corresponding to each prompt template; The first text feature vector is sampled from the probability distribution obeyed by multiple components in the second text feature vector.
4. The image classification method according to claim 3, wherein: The sampling of the first text feature vector from the probability distribution obeyed by the plurality of components in the second text feature vector comprises: According to the maximum component and the minimum component in the first text feature vector, a uniform probability distribution obeyed by the multiple components is determined for sampling the first text feature vector.
5. The image classification method according to claim 4, wherein: The uniform probability distribution is negatively correlated with the difference between the maximum component and the minimum component, and positively correlated with the difference between the component and the minimum component.
6. The image classification method according to any one of claims 3 to 5, wherein: The sampling of the first text feature vector from the probability distribution obeyed by the plurality of components in the second text feature vector comprises: The first text feature vector is sampled according to the statistical characteristics of the probability distribution.
7. The image classification method according to any one of claims 4 to 6, wherein: The step of sampling the first text feature vector according to the statistical characteristics of the probability distribution includes: The first text feature vector is determined according to a weighted sum of a mean and a variance of the probability distribution.
8. The image classification method according to any one of claims 1 to 7, wherein: Determining the image category to which the image to be classified belongs based on the similarity between the image feature vector and the first text feature vector corresponding to each prompt template includes: Calculating the distance between the image feature vector and each first text feature vector; Determining the membership probability of the image to be classified belonging to each image category according to the distance; The image category with the highest membership probability is determined as the image category to which the image to be classified belongs.
9. The image classification method according to claim 8, wherein: Calculating the distance between the image feature vector and each first text feature vector includes: The distance is calculated based on the sum of the absolute values of the components in the image feature vector and the absolute values of the components in each first text feature vector, and the absolute value of the sum of the components in the image feature vector and the components in each first text feature vector.
10. An image classification device, comprising: a text encoder configured to obtain, based on a plurality of prompt templates input, a first text feature vector corresponding to each of the plurality of prompt templates, wherein each prompt template includes text prompt information of an image category; An image encoder is used to obtain an image feature vector according to an input image to be classified; The classifier is used to determine the image category to which the image to be classified belongs based on the similarity between the image feature vector and the first text feature vector corresponding to each prompt template.
11. An image classification device, comprising: Memory; and A processor coupled to the memory, wherein the processor is configured to execute the image classification method according to any one of claims 1 to 9 based on instructions stored in the memory.
12. A computer-readable storage medium having a computer program stored thereon, wherein when the program is executed by a processor, the image classification method according to any one of claims 1 to 9 is implemented.
13. A computer program product comprising instructions, which, when executed by a processor, cause the processor to perform the image classification method according to any one of claims 1 to 9.