Image classification method based on prompt increment aggregation of energy field and distance space

Abstract

The invention provides an image classification method based on prompt incremental aggregation of an energy field and a distance space, and the method comprises the steps: obtaining a data set of a field reinforcement learning scene, and the data set comprises the data of at least one field; training is carried out through a data set, and during training, the data of the current field and the frozen CLIP model are utilized to optimize the personalized prompt model of the field through minimizing a loss function; during reasoning, a to-be-classified image is coded through the CLIP model to obtain a feature vector, the distance between the feature vector and each field is calculated, for each field, the personalized prompt model of the field is fused with the CLIP model to generate energy and prediction probability of the to-be-classified image, a distance factor and an energy factor are determined based on the distance and the energy, and the prediction probability of the to-be-classified image is calculated. And combining the distance factor and the energy factor to obtain a domain weight, calculating a weighted sum of prediction probabilities by using the domain weight, and generating a classification mixing probability. By adopting the scheme, the cross-domain increment adaptation capability of the model is improved.

Description

Technical Field

[0001] The present application relates to the field of incremental learning technology, and in particular to an image classification method and device based on prompt incremental aggregation of energy field and distance space. Background Art

[0002] In order for computer graphics classification models to be successful in real-world applications, as new data is acquired, the model needs to continuously learn incrementally to avoid outdated and declining performance. At the same time, data sampled in different regions and at different times have different characteristics, which is the phenomenon of domain data drift. Data is attributed to multiple domains. The data between domains are different, but have the same task objectives. Due to security and privacy restrictions, data between domains is not interoperable. The model performs incremental learning in units of domains. When learning a new domain, the old domain model can be used, but the old domain data cannot be used.

[0003] From the perspective of model learning mode, this is domain incremental learning without data playback, which presents two major challenges:

[0004] (1) Avoid catastrophic forgetting and ensure stability, that is, the knowledge acquired from the previous field cannot be forgotten after learning a new field;

[0005] (2) Allowing knowledge to be transferred forward and improving plasticity, that is, using knowledge from previous fields to effectively learn new fields.

[0006] For reasons of data security and privacy, it is usually not possible to store old data, so data replay methods cannot be used during incremental learning. Typical methods of incremental learning currently include: regularization-based methods, which regularize the model based on past knowledge while training a new domain, such as EWC, which introduces uncertainty in the Bayesian framework and uses the posterior distribution of previous data to limit parameter changes; architecture-based methods, which expand or modify the architecture of the model when encountering a new domain, such as DyTox, which shares the parameter architecture of early layers in all domains but dynamically expands task-specific tokens on the last layer. At the same time, CUE learns powerful representations based on multimodal pre-trained models (such as images, speech, and language) and performs personalized knowledge extraction for downstream tasks, demonstrating advantages such as pluggable, secure, and lightweight. In the domain incremental learning scenario without data replay, prompt-based methods have demonstrated superior performance: for example, L2P designs a shared prompt pool where each domain samples a set of prompts for training based on key-value matching; S-Prompts learns prompts independently for each domain and uses a distance query mechanism to select domain-related prompts during evaluation; ESN also adopts independent learning and introduces energy-based self-normalization and voting-based robust reasoning enhancement strategies to improve query efficiency.

[0007] However, existing cue-based methods still do not have a good stability-plasticity balance: sequential learning with shared cues may lead to cue weight drift, forgetting, and poor stability; learning cues from each domain independently ensures optimal performance, but selecting cues from only one domain at evaluation time prevents knowledge reuse and discards contributions from other domains, resulting in poor plasticity. Summary of the invention

[0008] The present application aims to solve one of the technical problems in the related art at least to some extent.

[0009] To this end, the first purpose of this application is to propose an image classification method based on prompt incremental aggregation of energy field and distance space, which solves the conflict problem of catastrophic forgetting and continuous learning faced by existing incremental learning technologies.

[0010] The second objective of the present application is to propose an image classification device based on prompt incremental aggregation of energy field and distance space.

[0011] To achieve the above-mentioned purpose, the first embodiment of the present application proposes an image classification method based on prompt incremental aggregation of energy field and distance space, including:

[0012] Acquire a data set of a domain-enhanced learning scenario, wherein the data set includes data of at least one domain;

[0013] Train with the dataset. During training, use the data in the current field and the frozen CLIP model to optimize the personalized prompt model in the field by minimizing the loss function.

[0014] During inference, the CLIP model is used to encode the image to be classified to obtain a feature vector, and the distance between the feature vector and each field is calculated. For each field, the personalized prompt model of the field is fused with the CLIP model to generate the energy and prediction probability of the image to be classified. The distance factor and energy factor are determined based on the distance and energy. The distance factor and energy factor are combined to obtain the field weight. The field weight is used to calculate the weighted sum of the prediction probability to generate the classification mixed probability.

[0015] The image classification method based on incremental aggregation of prompts in energy field and distance space in the embodiment of the present application adopts a hybrid energy-distance measurement strategy to fairly calculate weights across different fields, thereby integrating the classification capabilities of prompt models in various fields. However, the main difficulty of this paradigm is how to assign weights to each trained field in a fair manner when the distribution of field data is unbalanced. The present application converts it into a distribution inside and outside detection problem of each field, and obtains the probability of prediction falling within the field by detecting the similarity between the test data point and the field training data distribution. At the same time, it is necessary to ensure that the detection probability in each field is within the same output scale range in order to perform normalized conversion weights.

[0016] Specifically, the data distribution of each domain is mapped to a shared feature space using the same backbone model feature extractor (CLIP), and the distance factor is obtained based on the minimum distance from each sample to the centroid of each domain. Compared with the generalization of the backbone model, the personalization of the cue model is used to better represent the data of the current domain, and a regularization loss is added to limit the lower Helmholtz free energy generated in the cue model of similar domains, which is converted into an energy factor. Finally, the energy factor and the distance factor are combined in a reasonable proportion, and the obtained weights are normalized, so that the classification ability of each cue model is integrated in the evaluation stage, showing stable and efficient performance. Under this design: (1) It is resistant to forgetting: the parameters of the pre-trained model and the cue from the previous domain are frozen and maintain high performance; (2) It allows knowledge transfer: all participating domains participate in decision-making to varying degrees, allowing knowledge reuse.

[0017] Optionally, in one embodiment of the present application, the distance between the feature vector and each domain is the minimum distance from the feature vector to the centroid of each domain, and the distance factor is expressed as:

[0018] DF i (x) = D min -D i (x)

[0019] Among them, DF i (x) is the distance factor of the i-th field, DF i (x)∈(-∞,0],D min =min i {D i (x)}, D i (x) is the distance to the i-th area,

[0020]

[0021] Among them, f θ (x) is the feature vector of image x, the KNN center point of the i-th field is the jth KNN centroid of the i-th field, and k is the number of centroids.

[0022] Optionally, in one embodiment of the present application, the energy factor is Helmholtz free energy, and the energy factor is expressed as:

[0023]

[0024] Where H(x)[y] is the logit value of y∈{1,...,N}, y is the predicted label value number, N is the number of predicted labels, and T is the given temperature.

[0025] Optionally, in one embodiment of the present application, the weight is expressed as:

[0026]

[0027] Among them, EF(x) is the energy factor, DF(x) is the distance factor, α and β are coefficients, F(x) is the mixing factor, and W i is the mixing weight of the ith field, F i (x) is the mixing factor of the ith domain, and s means there are s domains.

[0028] Optionally, in one embodiment of the present application, the loss function is expressed as:

[0029] min L total =L ce +λL reg

[0030] Among them, the classification loss L ce is the cross entropy loss between the probability of the current field and the corresponding target label, λ is the coefficient, L reg is the regularization loss, L reg It is expressed as:

[0031] L reg =L border +L midline

[0032] Among them, L border is the boundary loss, L midline is the median loss,

[0033]

[0034] Among them, Θ is the set energy boundary threshold, Δ is the energy median value, is a data set, where (x, y) is the image feature vector and y is the label value corresponding to the image.

[0035] To achieve the above-mentioned purpose, the second aspect of the present invention proposes an image classification device based on prompt incremental aggregation of energy field and distance space, on which a trained image classification model is deployed, and the image classification model is used to implement image classification, wherein the training process of the image classification model includes:

[0036] Acquire a data set of a domain-enhanced learning scenario, wherein the data set includes data of at least one domain;

[0037] Train with the dataset. During training, use the data in the current field and the frozen CLIP model to optimize the personalized prompt model in the field by minimizing the loss function.

[0038] The process of using the image classification model includes:

[0039] During inference, the CLIP model is used to encode the image to be classified to obtain a feature vector, and the distance between the feature vector and each field is calculated. For each field, the personalized prompt model of the field is fused with the CLIP model to generate the energy and prediction probability of the image to be classified. The distance factor and energy factor are determined based on the distance and energy. The distance factor and energy factor are combined to obtain the field weight. The field weight is used to calculate the weighted sum of the prediction probability to generate the classification mixed probability.

[0040] Optionally, in one embodiment of the present application, the distance between the feature vector and each domain is the minimum distance from the feature vector to the centroid of each domain, and the distance factor is expressed as:

[0041] DF i (x) = D min -D i (x)

[0042] Among them, DF i (x) is the distance factor of the i-th field, DF i (x)∈(-∞,0],D min =min i {D i (x)}, D i (x) is the distance to the i-th area,

[0043]

[0044] Among them, f θ (x) is the feature vector of image x, the KNN center point of the i-th field is the jth KNN centroid of the i-th field, and k is the number of centroids.

[0045] Optionally, in one embodiment of the present application, the energy factor is Helmholtz free energy, and the energy factor is expressed as:

[0046]

[0047] Where H(x)[y] is the logit value of y∈{1,...,N}, y is the predicted label value number, N is the number of predicted labels, and T is the given temperature.

[0048] Optionally, in one embodiment of the present application, the weight is expressed as:

[0049]

[0050] Among them, EF(x) is the energy factor, DF(x) is the distance factor, α and β are coefficients, F(x) is the mixing factor, and W i is the mixing weight of the ith field, F i (x) is the mixing factor of the ith domain, and s means there are s domains.

[0051] Optionally, in one embodiment of the present application, the loss function is expressed as:

[0052] lm total =L ce +λL reg

[0053] Among them, the classification loss L ce is the cross entropy loss between the probability of the current field and the corresponding target label, λ is the coefficient, L reg is the regularization loss, L reg It is expressed as:

[0054] L reg =L border +L midline

[0055] Among them, L border is the boundary loss, L midline is the median loss,

[0056]

[0057] Among them, Θ is the set energy boundary threshold, Δ is the energy median value, is a data set, where (x, y) is the image feature vector and y is the label value corresponding to the image.

[0058] Additional aspects and advantages of the present application will be given in part in the description below, and in part will become apparent from the description below, or will be learned through the practice of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:

[0060] Figure 1 A flowchart of an image classification method based on prompt incremental aggregation of energy field and distance space provided in the first embodiment of the present application. ;

[0061] Figure 2 An overall architecture diagram of an image classification system based on an energy field and distance space prompt incremental aggregation technology provided in an embodiment of the present application;

[0062] Figure 3 This is a regularization loss effect diagram of this embodiment. DETAILED DESCRIPTION

[0063] Embodiments of the present application are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present application, and should not be construed as limiting the present application.

[0064] The following describes an image classification method and device based on prompt incremental aggregation of energy field and distance space according to an embodiment of the present application with reference to the accompanying drawings.

[0065] Figure 1 A flowchart of an image classification method based on prompt incremental aggregation of energy field and distance space provided in Example 1 of the present application.

[0066] like Figure 1 As shown, the image classification method based on prompt incremental aggregation of energy field and distance space includes the following steps:

[0067] Step 101, obtaining a data set of a domain enhanced learning scenario, wherein the data set includes data of at least one domain;

[0068] Step 102, training is performed using the data set. During training, the data of the current field and the frozen CLIP model are used to optimize the personalized prompt model of the field by minimizing the loss function;

[0069] Step 103, during inference, the image to be classified is encoded by the CLIP model to obtain a feature vector, and the distance between the feature vector and each field is calculated. For each field, the personalized prompt model of the field is fused with the CLIP model to generate the energy and prediction probability of the image to be classified. The distance factor and energy factor are determined based on the distance and energy. The distance factor and the energy factor are combined to obtain the field weight. The field weight is used to calculate the weighted sum of the prediction probability to generate the classification mixed probability.

[0070] The image classification method based on incremental aggregation of prompts in energy field and distance space in the embodiment of the present application adopts a hybrid energy-distance measurement strategy to fairly calculate weights across different fields, thereby integrating the classification capabilities of prompt models in various fields. However, the main difficulty of this paradigm is how to assign weights to each trained field in a fair manner when the distribution of field data is unbalanced. The present application converts it into a distribution inside and outside detection problem of each field, and obtains the probability of prediction falling within the field by detecting the similarity between the test data point and the field training data distribution. At the same time, it is necessary to ensure that the detection probability in each field is within the same output scale range in order to perform normalized conversion weights.

[0071] Specifically, the data distribution of each domain is mapped to a shared feature space using the same backbone model feature extractor (CLIP), and the distance factor is obtained based on the minimum distance from each sample to the centroid of each domain. Compared with the generalization of the backbone model, the personalization of the cue model is used to better represent the data of the current domain, and a regularization loss is added to limit the lower Helmholtz free energy generated in the cue model of similar domains, which is converted into an energy factor. Finally, the energy factor and the distance factor are combined in a reasonable proportion, and the obtained weights are normalized, so that the classification ability of each cue model is integrated in the evaluation stage, showing stable and efficient performance. Under this design: (1) It is resistant to forgetting: the parameters of the pre-trained model and the cue from the previous domain are frozen and maintain high performance; (2) It allows knowledge transfer: all participating domains participate in decision-making to varying degrees, allowing knowledge reuse.

[0072] See also Figure 2 , Figure 2 The overall architecture diagram of an image classification system based on an energy field and distance space prompt incremental aggregation technology provided by an embodiment of the present application is shown. The method is executed by an electronic device.

[0073] This embodiment is designed and adds prompts based on the pre-trained model to achieve the cross-domain incremental adaptation capability of the model. The key idea is independent learning and hybrid reasoning.

[0074] like Figure 2 As shown, first, the domain incremental learning scenario involves a series of datasets Where S is the current domain number. This embodiment can only access one domain at a time and cannot store samples from previous domains, but can retain the hint parameters of previous domains during reasoning to apply prior knowledge. For the i-th domain, the domain incremental learning method improves D i While improving model performance, it also retains i-1 ,D i-2 etc.) knowledge.

[0075] During training, each domain independently optimizes personalized cues and freezes parameters after training to resist forgetting. Specifically, domain-specific data and the frozen CLIP model are used to minimize the full loss function L total (classification loss and energy regularization loss) to optimize the hint parameters for this specific domain.

[0076] During inference, the predicted probabilities of the personalized hint model for each domain are weighted to complete the classification task, thereby achieving knowledge reuse and transfer. Specifically, CLIP encodes the data to obtain a feature vector to calculate the distance to each domain. For each domain, the hint of the domain is fused with CLIP to produce energy and predicted probabilities. The weights are derived from the energy factor and the distance factor, and the final classification mixed probability is obtained by the weighted sum of the probabilities.

[0077] (1) Hybrid Energy-Distance Weighting

[0078] Assigning a weight to each domain for an unknown sample is equivalent to finding one or more factors to determine the similarity between the sample and the data in each domain. The factors proposed in this embodiment need to have the following characteristics: (1) The factor generated by the sample in a certain domain is a scalar that can be used to represent the similarity match between the sample and the domain. (2) By comparing the factors generated by samples in different domains within the same scale range, the relative similarity between domains can be determined. There are two types of models in the domain incremental learning process, namely the generalized cross-domain CLIP model and the personalized cross-domain prompt model. This embodiment uses these two different domain data representations and distribution in- and out-of-domain detection methods to generate two factors, namely distance and energy.

[0079] 1) Distance factor

[0080] Non-parametric nearest neighbor distance methods have shown effectiveness in detecting out-of-domain samples. These methods utilize feature embeddings derived from the model and are based on the assumption that out-of-domain samples are significantly closer to in-domain samples in feature space. In this embodiment, the L1-norm is used to calculate the minimum distance D(x) between the test sample embedding and the kth nearest neighbor embedding of the training set, and compared in the same feature space to determine which domain the input is most likely to belong to. In simple terms, it is expressed as: D i (xi ) <D j (x i ), where x i ∈D i This embodiment uses only the CLIP image encoder without using prompts to extract a KNN center point of the i-th area Then calculate D i (x) is as follows:

[0081]

[0082] Calculate the distance factor DF of the i-th domain (i≤s) i (x), as the minimum value D min =min i {D i (x)}, as follows:

[0083] DF i (x) = D min -D i (x)

[0084] Obviously DF(x)∈(-∞,0], larger values ​​of DF correspond to closer distances and more similar neighborhoods.

[0085] 2) Energy Factor

[0086] The Helmholtz free energy value can be aligned with the input density and improve the in-distribution and out-of-distribution detection performance. The key idea is to use the regularization loss to establish an energy threshold that assigns low energy values ​​to in-distribution data and high energy values ​​to out-of-distribution data. As done in EBM, this example calculates the Helmholtz free energy E(x) at a given temperature T, where H(x)[y] is the logit value of y∈{1,...,N}:

[0087]

[0088] In this embodiment, the regularization loss L reg The refinement of the current domain-specific cue model is promoted to better align it with the in-domain data, while the data from other domains are mainly mapped to out-of-distribution examples in the current feature space. Assuming that the energy range of the cross-domain model is consistent for the in-domain data, the free energy of the data in the correct cue model should be the lowest among all domains. This can be expressed by the following equation: i (x i )=E j (x j ),E j (x j ) <E j (xi ) → E i (x i ) <E j (x i ), where x i The data corresponding to the i-th domain is in-domain data, and the data corresponding to the j-th domain is out-of-domain data.

[0089] Boundary loss L border Move the energy field of the data in the domain to one side of the fixed boundary θ, and the midline loss L midljne Move the average value of the energy field of the data within the domain closer to the same center line Δ. Figure 3 is the regular loss action diagram, such as Figure 3 As shown, the combination of borders and midlines ensures that each field has a similar energy distribution range within the field, not too concentrated in the midline or too far from the borders, allowing the similarity of fields to be compared based on the energy values ​​of samples in different fields.

[0090] L reg =L border +L midline

[0091]

[0092] This generates the energy factor EF of the i-th field i (x), where i≤s, by comparing it with the minimum free energy value E min =min i {E i (x)} to scale the confidence score of each domain, EF(x)∈(-∞,0], where larger EF values ​​correspond to lower free energy values ​​and more similar domains, calculated as follows:

[0093] EF i (x) = F mi9n -E i (x)

[0094] 3) Mixed weights and overall optimization goals

[0095] The energy factor EF(x) and the distance factor DF(x) are combined to obtain the mixing factor F(x), and α and β are used to balance the relative weights of the two terms. Then, Softmax normalization is performed to obtain the mixing weight W of the i-th community. i , as shown below:

[0096]

[0097] Our complete optimization objective is expressed as follows:

[0098] lm total =Lce +λL reg

[0099] Here, the classification loss L ce is the cross entropy loss between the probability of the current domain and the corresponding target label, while λ is used to adjust the regularization loss L reg proportion.

[0100] In order to implement the above embodiment, the present application also proposes an image classification device based on prompt incremental aggregation of energy field and distance space, on which a trained image classification model is deployed, and the image classification model is used to implement image classification, wherein the training process of the image classification model includes:

[0101] Acquire a data set of a domain-enhanced learning scenario, wherein the data set includes data of at least one domain;

[0102] Train with the dataset. During training, use the data in the current field and the frozen CLIP model to optimize the personalized prompt model in the field by minimizing the loss function.

[0103] The process of using the image classification model includes:

[0104] During inference, the CLIP model is used to encode the image to be classified to obtain a feature vector, and the distance between the feature vector and each field is calculated. For each field, the personalized prompt model of the field is fused with the CLIP model to generate the energy and prediction probability of the image to be classified. The distance factor and energy factor are determined based on the distance and energy. The distance factor and energy factor are combined to obtain the field weight. The field weight is used to calculate the weighted sum of the prediction probability to generate the classification mixed probability.

[0105] Optionally, in one embodiment of the present application, the distance between the feature vector and each domain is the minimum distance from the feature vector to the centroid of each domain, and the distance factor is expressed as:

[0106] DF i (x) = D min -D i (x)

[0107] Among them, DF i (x) is the distance factor of the i-th field, DF i (x)∈(-∞,0],D min =min i {D i (x)}, D i (x) is the distance to the i-th area,

[0108]

[0109] Among them, fθ (x) is the feature vector of image x, the KNN center point of the i-th field is the jth KNN centroid of the i-th field, and k is the number of centroids.

[0110] Optionally, in one embodiment of the present application, the energy factor is Helmholtz free energy, and the energy factor is expressed as:

[0111]

[0112] Where H(x)[y] is the logit value of y∈{1,...,N}, y is the predicted label value number, N is the number of predicted labels, and T is the given temperature.

[0113] Optionally, in one embodiment of the present application, the weight is expressed as:

[0114]

[0115] Among them, EF(x) is the energy factor, DF(x) is the distance factor, α and β are coefficients, F(x) is the mixing factor, and W i is the mixing weight of the ith field, F i (x) is the mixing factor of the ith domain, and s means there are s domains.

[0116] Optionally, in one embodiment of the present application, the loss function is expressed as:

[0117] min L total =L ce +λL reg

[0118] Among them, the classification loss L ce is the cross entropy loss between the probability of the current field and the corresponding target label, λ is the coefficient, L reg is the regularization loss, L reg It is expressed as:

[0119] L reg =L border +L midline

[0120] Among them, L border is the boundary loss, L midline is the median loss,

[0121]

[0122] Among them, Θ is the set energy boundary threshold, Δ is the energy median value, is a data set, where (x, y) is the image feature vector and y is the label value corresponding to the image.

[0123] It should be noted that the aforementioned explanation of the embodiment of the image classification method based on incremental aggregation of prompts in energy field and distance space is also applicable to the image classification device based on incremental aggregation of prompts in energy field and distance space of this embodiment, which will not be repeated here.

[0124] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example" or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, without contradiction.

[0125] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In the description of this application, the meaning of "plurality" is at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

[0126] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, fragment or portion of code comprising one or more executable instructions for implementing the steps of a custom logical function or process, and the scope of the preferred embodiments of the present application includes alternative implementations in which functions may not be performed in the order shown or discussed, including performing functions in a substantially simultaneous manner or in the reverse order depending on the functions involved, which should be understood by technicians in the technical field to which the embodiments of the present application belong.

[0127] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute the instructions), or in combination with these instruction execution systems, devices or apparatuses. For the purpose of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in combination with these instruction execution systems, devices or apparatuses. More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection with one or more wires (electronic device), a portable computer disk box (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), a fiber optic device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium and then editing, interpreting or processing in other suitable ways if necessary, and then stored in a computer memory.

[0128] It should be understood that the various parts of the present application can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, multiple steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0129] A person skilled in the art may understand that all or part of the steps in the method for implementing the above-mentioned embodiment may be completed by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, which, when executed, includes one or a combination of the steps of the method embodiment.

[0130] In addition, each functional unit in each embodiment of the present application may be integrated into a processing module, or each unit may exist physically separately, or two or more units may be integrated into one module. The above-mentioned integrated module may be implemented in the form of hardware or in the form of a software functional module. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it may also be stored in a computer-readable storage medium.

[0131] The storage medium mentioned above may be a read-only memory, a magnetic disk or an optical disk, etc. Although the embodiments of the present application have been shown and described above, it can be understood that the above embodiments are exemplary and cannot be understood as limiting the present application. A person of ordinary skill in the art may change, modify, replace and modify the above embodiments within the scope of the present application.

Claims

1. An image classification method based on prompt incremental aggregation of energy field and distance space, characterized in that: include: Acquire a data set of a domain enhanced learning scenario, wherein the data set includes data of at least one domain; Train with the dataset. During training, use the data in the current field and the frozen CLIP model to optimize the personalized prompt model in the field by minimizing the loss function. During inference, the CLIP model is used to encode the image to be classified to obtain a feature vector, and the distance between the feature vector and each field is calculated. For each field, the personalized prompt model of the field is fused with the CLIP model to generate the energy and prediction probability of the image to be classified. The distance factor and energy factor are determined based on the distance and energy. The distance factor and energy factor are combined to obtain the field weight. The field weight is used to calculate the weighted sum of the prediction probability to generate the classification mixed probability.

2. The method according to claim 1, characterized in that The distance between the feature vector and each field is the minimum distance from the feature vector to the centroid of each field. The distance factor is expressed as: DF i (x)=D min -D i (x) Among them, DF i (x) is the distance factor of the i-th field, DF i (x)∈(-∞,0],D min =min i {D i (x)}, D i (x) is the distance to the i-th area, Among them, f θ (x) is the feature vector of image x, the KNN center point of the i-th field is the jth KNN centroid of the i-th field, and k is the number of centroids.

3. The method according to claim 1, characterized in that The energy factor is the Helmholtz free energy, which is expressed as: Where H(x)[y] is the logit value of y∈{1,...,N}, y is the predicted label value number, N is the number of predicted labels, and T is the given temperature.

4. The method according to any one of claims 3 or 4, characterized in that: The weight is expressed as: Among them, EF(x) is the energy factor, DF(x) is the distance factor, α and β are coefficients, F(x) is the mixing factor, and W i is the mixing weight of the ith field, F i (x) is the mixing factor of the ith domain, and s means there are s domains.

5. The method according to claim 4, characterized in that The loss function is expressed as: from L total =L ce +λL reg Among them, the classification loss L ce is the cross entropy loss between the probability of the current field and the corresponding target label, λ is the coefficient, L reg is the regularization loss, L reg It is expressed as: L reg =L border +L midline Among them, L border is the boundary loss, L midline is the median loss, Among them, Φ is the set energy boundary threshold, Δ is the energy median value, is a data set, where (x, y) is the image feature vector and y is the label value corresponding to the image.

6. An image classification device based on prompt incremental aggregation of energy field and distance space, characterized in that: A trained image classification model is deployed on the device, and the image classification model is used to implement image classification, wherein the training process of the image classification model includes: Acquire a data set of a domain enhanced learning scenario, wherein the data set includes data of at least one domain; Train with the dataset. During training, use the data in the current field and the frozen CLIP model to optimize the personalized prompt model in the field by minimizing the loss function. The use process of the image classification model includes: During inference, the CLIP model is used to encode the image to be classified to obtain a feature vector, and the distance between the feature vector and each field is calculated. For each field, the personalized prompt model of the field is fused with the CLIP model to generate the energy and prediction probability of the image to be classified. The distance factor and energy factor are determined based on the distance and energy. The distance factor and energy factor are combined to obtain the field weight. The field weight is used to calculate the weighted sum of the prediction probability to generate the classification mixed probability.

7. The device according to claim 6, characterized in that The distance between the feature vector and each field is the minimum distance from the feature vector to the centroid of each field. The distance factor is expressed as: DF i (x)=D min -D i (x) Among them, DF i (x) is the distance factor of the i-th field, DF i (x)∈(-∞,0],D min =min i {D i (x)}, D i (x) is the distance to the i-th area, Among them, f θ (x) is the feature vector of image x, is the jth KNN centroid of the i-th field, and k is the number of centroids.

8. The device according to claim 6, characterized in that The energy factor is the Helmholtz free energy, which is expressed as: Where H(x)[y] is the logit value of y∈{1,...,N}, y is the predicted label value number, N is the number of predicted labels, and T is the given temperature.

9. The device according to any one of claims 7 or 8, characterized in that: The weight is expressed as: Among them, EF(x) is the energy factor, DF(x) is the distance factor, α and β are coefficients, F(x) is the mixing factor, and W i is the mixing weight of the ith field, F i (x) is the mixing factor of the ith domain, and s means there are s domains.

10. The device according to claim 6, characterized in that The loss function is expressed as: from L total =L ce +λL reg Among them, the classification loss L ce is the cross entropy loss between the probability of the current field and the corresponding target label, λ is the coefficient, L reg is the regularization loss, L reg It is expressed as: L reg =L border +L midline Among them, L border is the boundary loss, L midline is the median loss, Among them, Θ is the set energy boundary threshold, Δ is the energy median value, is a data set, where (x, y) is the image feature vector and y is the label value corresponding to the image.

Images