Image classification method based on energy field and distance space prompt incremental aggregation

Abstract

The application provides an image classification method based on energy field and distance space prompt incremental aggregation, including: obtaining a data set of a domain reinforcement learning scene, wherein the data set includes data of at least one domain; training through the data set, in the training, using the data of the current domain and the frozen CLIP model, optimizing the personalized prompt model of the domain by minimizing the loss function; in the inference, the CLIP model is used to encode the to-be-classified image to obtain a feature vector, and the distance between the feature vector and each domain is calculated, for each domain, the personalized prompt model of the domain is used to generate the energy and the prediction probability of the to-be-classified image in combination with the CLIP model, the distance factor and the energy factor are determined based on the distance and the energy, the domain weight is obtained by combining the distance factor and the energy factor, the weighted sum of the prediction probability is calculated by using the domain weight, and the classification mixed probability is generated. The application adopting the above scheme improves the cross-domain incremental adaptation capability of the model.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of incremental learning, in particular to an image classification method and device based on energy field and distance space prompt incremental aggregation. BACKGROUND

[0002] In order to make the computer graphics classification model successful in the application of the real world, the model needs to be continuously learned incrementally as new data is obtained, so as to avoid outdated and decaying performance. At the same time, the data characteristics of different regions and different time periods are different, that is, the domain data offset phenomenon, the data belongs to multiple domains, the data is different between domains, but has the same task target. And due to security and privacy restrictions, data between domains cannot be shared, and the model learns incrementally in units of domains, and when learning a new domain, the old domain model can be used, but the old domain data cannot be used.

[0003] From the perspective of model learning mode, this belongs to domain incremental learning without data replay, and two main challenges arise:

[0004] (1) Avoid catastrophic forgetting and ensure stability, that is, do not forget the knowledge obtained from previous domains after learning a new domain;

[0005] (2) Allow knowledge to be transferred forward and improve plasticity, that is, use the knowledge in the previous domain to effectively learn the new domain.

[0006] Due to data security and privacy reasons, it is usually not possible to store old data, so data replay methods cannot be used when incrementally learning. Typical methods of incremental learning currently include: regularization-based methods, which regularize the model according to past knowledge while training new domains, such as EWC, which introduces uncertainty in the Bayesian framework and uses the posterior distribution of previous data to limit the change of parameters; architecture-based methods, which extend or modify the architecture of the model when encountering a new domain, such as DyTox, which shares the parameter architecture of the early layers in all domains, but dynamically expands the task-specific tokens on the last layer. At the same time, prompt learning is based on the powerful representation of multi-modal pre-training models (such as images, speech and language), and performs personalized knowledge extraction for downstream tasks, showing advantages such as pluggable, secure and lightweight. In the field of domain incremental learning without data replay, prompt-based methods have shown superior performance: for example, L2P designs a shared prompt pool, and each domain samples a set of prompts for training according to key-value matching; S-Prompts learns prompts independently for each domain, and uses a distance query mechanism to select prompts related to the domain during evaluation; ESN also uses independent learning, and introduces energy-based self-normalization and voting-based robust inference enhancement strategies to improve query efficiency.

[0007] However, the existing prompt-based methods still do not balance stability-plasticity: sequential learning using shared prompts can lead to prompt weight drift, forgetting and poor stability; learning prompts independently from each domain can ensure optimal performance, but selecting only one domain's prompts at evaluation prevents knowledge reuse and discards the contributions of other domains, resulting in poor plasticity. SUMMARY

[0008] The present application aims to at least partially solve one of the technical problems in the related art.

[0009] To this end, the first object of the present application is to propose an image classification method based on energy field and distance space prompt incremental aggregation, which solves the conflict problem of catastrophic forgetting and continuous learning in existing incremental learning technology.

[0010] The second object of the present application is to propose an image classification device based on energy field and distance space prompt incremental aggregation.

[0011] To achieve the above object, the first aspect of the present application proposes an image classification method based on energy field and distance space prompt incremental aggregation, comprising:

[0012] Obtaining a dataset of a domain reinforcement learning scene, wherein the dataset includes data of at least one domain;

[0013] Training through the dataset, during training, using the data of the current domain and the frozen CLIP model, optimizing the personalized prompt model of the domain by minimizing the loss function;

[0014] During inference, the CLIP model is used to encode the image to be classified to obtain a feature vector, and the distance between the feature vector and each domain is calculated. For each domain, the personalized prompt model of the domain and the CLIP model are fused to generate the energy and prediction probability of the image to be classified, and the distance factor and energy factor are determined based on the distance and energy. The distance factor and energy factor are combined to obtain the domain weight, the weighted sum of the prediction probability is calculated using the domain weight, and the classification mixed probability is generated.

[0015] The image classification method based on energy field and distance space prompt incremental aggregation of the embodiments of the application adopts a hybrid energy-distance measurement strategy, performs fair calculation of weights across different fields, and thus integrates the classification capabilities of various field prompt models. However, the main difficulty of this paradigm is how to allocate weights to each trained field in a fair manner in the case of unbalanced field data distribution. The application converts this into an intra-distribution and extra-distribution detection problem for each field, obtains the probability of falling within the field by detecting the similarity between the test data points and the field training data distribution, and at the same time ensures that the detection probabilities in each field are within the same output scale range, so as to perform normalization to convert the weights.

[0016] Specifically, the data distribution of each field is mapped to a shared feature space using the same backbone model feature extractor (CLIP), and a distance factor is obtained based on the minimum distance of each sample to each field centroid. Compared with the generalization of the backbone model, the individualization of the prompt model is better to represent the data of the current field, and a regularization loss is added to limit the Helmholtz free energy generated in the similar field prompt model to be low, thereby converting to an energy factor. Finally, the energy factor and the distance factor are combined in a reasonable proportion, and the obtained weight is normalized, so as to integrate the classification capabilities of various prompt models in the evaluation stage and exhibit stable and efficient performance. Under this design: (1) resistant to forgetting: the pre-training model and the prompt parameters from the previous field are frozen and maintain high performance; (2) allow knowledge transfer: all participating fields participate in decision-making to varying degrees, allowing knowledge reuse.

[0017] Optionally, in an embodiment of the application, the distance of the feature vector to each field is the minimum distance of the feature vector to the centroid of each field, and the distance factor is represented as:

[0018] DF i (x)=D min -D i (x)

[0019] wherein DF i (x) is the distance factor of the i-th field, DF i (x)∈(-∞,0], D min =min i {D i (x)},D i (x) is the distance to the i-th field,

[0020]

[0021] wherein f θ (x) is the feature vector of the image x, and the KNN center point of the i-th field is the jth KNN centroid of the ith domain, k is the number of centroids.

[0022] Optionally, in an embodiment of the present application, the energy factor is Helmholtz free energy, and the energy factor is represented as:

[0023]

[0024] where H(x)[y] is the logit value of y, y is the predicted label value index, N is the number of predicted labels, and T is a given temperature.

[0025] Optionally, in an embodiment of the present application, the weight is represented as:

[0026]

[0027] where EF(x) is the energy factor, DF(x) is the distance factor, a and b are coefficients, F(x) is the mixing factor, W i is the mixing weight of the ith domain, F i (x) is the mixing factor of the ith domain, and s represents s domains.

[0028] Optionally, in an embodiment of the present application, the loss function is represented as:

[0029] min L total = L ce + l L reg

[0030] where L ce is the cross-entropy loss between the probability of the current domain and the corresponding target label, l is a coefficient, L reg is the regularization loss, and L reg is represented as:

[0031] L reg = L border + L midline

[0032] where L border is the boundary loss, L midline is the median loss,

[0033]

[0034] where Q is the set energy boundary threshold, and A is the energy median value, is a data set, and (x, y) is a picture feature vector, and y is the corresponding label value of the picture.

[0035] To achieve the above purpose, the second aspect embodiment of the present application proposes an image classification device based on energy field and distance space prompt incremental aggregation, which is deployed with a trained image classification model, and the image classification model is used to realize image classification, wherein the training process of the image classification model includes:

[0036] obtaining a data set of the field reinforcement learning scene, wherein the data set includes data of at least one field;

[0037] training through the data set, and in the training, the personalized prompt model of the current field is optimized by using the data of the current field and the frozen CLIP model and minimizing the loss function;

[0038] The use process of the image classification model includes:

[0039] In the inference, the feature vector of the image to be classified is obtained by encoding the image to be classified through the CLIP model, and the distance between the feature vector and each field is calculated, for each field, the personalized prompt model of the field is used to generate the energy and the prediction probability of the image to be classified by fusing the CLIP model, the distance factor and the energy factor are determined based on the distance and the energy, the field weight is obtained by combining the distance factor and the energy factor, the weighted sum of the prediction probability is calculated by using the field weight, and the classification mixed probability is generated.

[0040] Optionally, in an embodiment of the present application, the distance between the feature vector and each field is the minimum distance from the feature vector to the centroid of each field, and the distance factor is represented as:

[0041] DF i (x)=D min -D i (x)

[0042] Wherein, DF i (x) is the distance factor of the i-th field, DF i (x)∈(-∞,0], D min =min i {D i (x)},D i (x) is the distance to the i-th field,

[0043]

[0044] Wherein, f θ (x) is the feature vector of the image x, and the KNN center point of the i-th field is the j-th KNN centroid of the i-th field, and k is the centroid number.

[0045] Optionally, in an embodiment of the present application, the energy factor is Helmholtz free energy, and the energy factor is represented as:

[0046]

[0047] where H(x)[y] is the logit value of y e {1,...,N}, y is the predicted label value index, N is the number of predicted labels, and T is a given temperature.

[0048] Optionally, in an embodiment of the present application, the weight is represented as:

[0049]

[0050] where EF(x) is the energy factor, DF(x) is the distance factor, a and b are coefficients, F(x) is the mixing factor, W i is the mixing weight of the i-th domain, F i (x) is the mixing factor of the i-th domain, and s represents s domains.

[0051] Optionally, in an embodiment of the present application, the loss function is represented as:

[0052] min L total = L ce + l L reg

[0053] where L ce is the cross-entropy loss between the probability of the current domain and the corresponding target label, l is a coefficient, L reg is the regularization loss, and L reg is represented as:

[0054] L reg = L border + L midline

[0055] where L border is the boundary loss, and L midline is the median loss,

[0056]

[0057] where Q is the set energy boundary threshold, and A is the energy median value, is a data set, and x in (x, y) is a picture feature vector, and y is the label value corresponding to the picture.

[0058] Additional aspects and advantages of the present application will be in part apparent and in part pointed out hereinafter. BRIEF DESCRIPTION OF DRAWINGS

[0059] The above and / or additional aspects and advantages of the present application will become apparent and more readily appreciated from the following description, taken in conjunction with the following drawings of exemplary embodiments of the present application, wherein:

[0060] Figure 1 A flowchart of an image classification method based on energy field and distance space prompt incremental aggregation provided by the first embodiment of the present application.

[0061] Figure 2 A general architecture diagram of an image classification system based on energy field and distance space prompt incremental aggregation technology provided by the embodiment of the present application.

[0062] Figure 3 The regular loss function diagram for the embodiment. DETAILED DESCRIPTION

[0063] The embodiments of the present application are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals represent the same or similar elements or elements having the same or similar functions throughout. The embodiments described below by reference to the accompanying drawings are exemplary and are intended to explain the present application, and cannot be understood as limiting the present application.

[0064] The image classification method and device based on energy field and distance space prompt incremental aggregation of the embodiments of the present application are described below with reference to the accompanying drawings.

[0065] Figure 1 A flowchart of an image classification method based on energy field and distance space prompt incremental aggregation provided by the first embodiment of the present application.

[0066] As shown in the image classification method based on energy field and distance space prompt incremental aggregation includes the following steps: Figure 1

[0067] Step 101, obtaining a data set of a domain reinforcement learning scene, wherein the data set includes data of at least one domain;

[0068] Step 102, training through the data set, wherein during the training, the personalized prompt model of the domain is optimized by minimizing the loss function using the data of the current domain and the frozen CLIP model;

[0069] ​Step 103, in reasoning, the feature vector of the image to be classified is encoded by the CLIP model, and the distance between the feature vector and each field is calculated. For each field, the personalized prompt model of the field is used to generate the energy and prediction probability of the image to be classified by fusing with the CLIP model. The distance factor and the energy factor are determined based on the distance and the energy, and the field weight is obtained by combining the distance factor and the energy factor. The weighted sum of the prediction probability is calculated using the field weight to generate the classification mixed probability.

[0070] The image classification method based on energy field and distance space prompt incremental aggregation of the embodiment of the application adopts a hybrid energy-distance measurement strategy to calculate the weight fairly across different fields, thereby integrating the classification capabilities of the prompt models of each field. However, the main difficulty of this paradigm is how to allocate weights to each trained field in a fair manner in the case of unbalanced field data distribution. The application converts this into an in-distribution and out-of-distribution detection problem for each field, and obtains the probability of predicting that the data point falls within the field by detecting the similarity between the test data point and the field training data distribution. At the same time, the detection probability in each field must be within the same output scale range to be able to normalize the converted weight.

[0071] Specifically, the data distribution of each field is mapped to a shared feature space using the same backbone model feature extractor (CLIP), and a distance factor is obtained based on the minimum distance of each sample to the centroid of each field. Compared with the generalization of the backbone model, the individualization of the prompt model is better to represent the data of the current field, and a regularization loss is added to limit the Helmholtz free energy generated in the similar field prompt model to be low, thereby converting to an energy factor. Finally, the energy factor and the distance factor are combined in a reasonable proportion, and the obtained weight is normalized, thereby integrating the classification capabilities of each prompt model in the evaluation stage and exhibiting stable and efficient performance. Under this design: (1) resistant to forgetting: the pre-trained model and the prompt parameters from the previous field are frozen and maintain high performance; (2) allow knowledge transfer: all participating fields participate in decision-making to varying degrees, allowing knowledge reuse.

[0072] Please refer to Figure 2 , Figure 2 The overall architecture diagram of an image classification system based on the energy field and distance space prompt incremental aggregation technology provided by the embodiment of the application is shown. The method is executed by an electronic device.

[0073] The embodiment designs and adds prompts on the basis of the pre-trained model to realize the cross-domain incremental adaptation capability of the model. The key idea is independent learning and mixed reasoning.

[0074] As shown in Figure 2 , first, the field incremental learning scenario involves a series of data sets where S is the current domain number. This embodiment can only access one domain at a time and cannot store samples of previous domains, but can retain prompt parameters of previous domains during inference to apply prior knowledge. For the ith domain, the domain-incremental learning method improves the D i upper model performance while preserving knowledge from other domains (e.g., D i-1 , D i-2 , etc.).

[0075] During training, individualized prompts are optimized for each domain independently, and the parameters are frozen after training to resist forgetting, specifically, prompt parameters for a particular domain are optimized by minimizing the complete loss function L total (classification loss and energy regularization loss) using the data of the particular domain and the frozen CLIP model.

[0076] During inference, the predicted probabilities of the individualized prompt models for each domain are weighted to complete the classification task, thereby achieving knowledge reuse and transfer, specifically, CLIP encodes the data to obtain a feature vector, thereby calculating the distance to each domain. For each domain, the prompt of the domain is fused with CLIP to produce energy and predicted probability. The weight is derived from the energy factor and the distance factor, and the final classification mixed probability is obtained by the weighted sum of the probabilities.

[0077] (1) Mixed energy-distance weighting

[0078] Assigning each domain a weight for an unknown sample is equivalent to finding one or more factors to determine the similarity of the sample to the data within each domain. The factors proposed in this embodiment need to have the following characteristics: (1) the factor generated by a sample in a certain domain is a scalar that can be used to represent the similarity match of the sample to the domain. (2) By comparing the factors generated by samples in different domains, it can be determined that the relative similarity between the domains is within the same scale range. There are two types of models in the domain-incremental learning process, namely the generalized cross-domain CLIP model and the individualized cross-domain prompt model. This embodiment uses these two different domain data representations and distribution inside-out detection methods to generate two factors, namely distance and energy.

[0079] 1) Distance factor

[0080] Non-parametric nearest neighbor distance methods have shown effectiveness in detecting out-of-domain samples, which use feature embeddings derived from models and are based on the assumption that out-of-domain samples are significantly further away from in-domain samples in feature space. In this embodiment, the L1-norm is used to calculate the minimum distance D(x) between the test sample embedding and the kth nearest neighbor embedding of the training set, and the comparison is made within the same feature space to determine which domain the input is most likely to belong to. Simply put, it is expressed as: D i (xi )<D j (x i ), where x i ∈D i . This example extracts KNN center points of a i-th domain using only the image encoder of CLIP without prompts Then compute D i (x) as follows:

[0081]

[0082] Compute the distance factor DF i (x) of the i-th domain (i≤s) as a relative distance measure to the minimum D min = min i {D i (x)} as follows:

[0083] DF i (x) = D min - D i (x)

[0084] It is clear that DF(x) ∈ (-∞, 0], and larger values of DF correspond to closer distances and more similar domains.

[0085] 2) Energy factor

[0086] Helmholtz free energy values can be aligned with the input density and improve in-distribution and out-of-distribution detection performance. The key idea is to use a regularization loss to establish an energy threshold that assigns low energy values to in-distribution data and high energy values to out-of-distribution data. As done in EBM, this example computes the Helmholtz free energy E(x) at a given temperature T, where H(x)[y] is the logit value for y ∈ {1,...,N}:

[0087]

[0088] In this example, the regularization loss L reg promotes the fine-tuning of the current domain-specific prompt model to better align with in-distribution data, while data from other domains are mainly mapped to out-of-distribution examples in the current feature space. Assuming that the energy range of the cross-domain model is consistent for in-distribution data, the free energy of the data in the correct prompt model should be the lowest among all domains. This can be represented by the following equation: E i (x i ) = E j (x j ), E j (x j ) < E j (xi )→E i (x i )<E j (x i ), where x i corresponds to in-domain data for the ith domain and out-of-domain data for the jth domain.

[0089] Boundary loss L border Shifts the energy field of in-domain data towards one side of the fixed boundary Θ, midline loss L midljne Brings the average value of the energy field of in-domain data closer to the same midline Δ. Figure 3 For regular loss acting graph, as Figure 3 shown, the combination of boundary and midline ensures that each domain has a similar range of energy distribution within the domain, not too concentrated in the midline or too far away from the boundary, allowing the comparison of the similarity of domains based on the energy values of samples in different domains.

[0090] L reg = L border + L midline

[0091]

[0092] Here, the energy factor EF i (x) for the ith domain is generated, where i≤s, by scaling the confidence score of each domain according to its distance from the minimum free energy value E min = min i {E i (x)}, EF(x)∈(-∞,0], a larger EF value corresponds to a lower free energy value and a more similar domain, calculated as follows:

[0093] EF i (x) = F mi9n -E i (x)

[0094] 3) Mixing weights with optimized total objective

[0095] Combining the energy factor EF(x) and the distance factor DF(x) to obtain the mixing factor F(x), using α and β to balance the relative weights of the two terms. Then, Softmax normalization is performed to obtain the mixing weight W i for the ith domain, as follows:

[0096]

[0097] Our complete optimization objective is represented as follows:

[0098] min L total = Lce +λL reg

[0099] Here, the classification loss L ce is the cross-entropy loss between the probability of the current field and the corresponding target label, and λ is used to adjust the proportion of the regularization loss L reg .

[0100] To achieve the above-mentioned embodiments, the present application also proposes an image classification device based on prompt incremental aggregation of energy field and distance space, which is deployed with a trained image classification model, and the image classification model is used to implement image classification, wherein the training process of the image classification model comprises:

[0101] Obtaining a data set of a field reinforcement learning scene, wherein the data set comprises data of at least one field;

[0102] Training through the data set, wherein during the training, the data of the current field and the frozen CLIP model are used to optimize the personalized prompt model of the field by minimizing the loss function;

[0103] The use process of the image classification model comprises:

[0104] During inference, the CLIP model is used to encode the image to be classified to obtain a feature vector, and the distance between the feature vector and each field is calculated, for each field, the personalized prompt model of the field is used to fuse with the CLIP model to generate the energy and the prediction probability of the image to be classified, the distance factor and the energy factor are determined based on the distance and the energy, the field weight is obtained by combining the distance factor and the energy factor, the weighted sum of the prediction probability is calculated using the field weight, and the classification mixed probability is generated.

[0105] Optionally, in an embodiment of the present application, the distance between the feature vector and each field is the minimum distance from the feature vector to the centroid of each field, and the distance factor is expressed as:

[0106] DF i (x)=D min -D i (x)

[0107] Wherein, DF i (x) is the distance factor of the i-th field, DF i (x)∈(-∞,0], D min =min i {D i (x)},D i (x) is the distance to the i-th field,

[0108]

[0109] Wherein, fθ (x) is a feature vector of image x, KNN center point of i-th field is the j-th KNN centroid of i-th field, k is the number of centroids.

[0110] Optionally, in an embodiment of the present application, the energy factor is Helmholtz free energy, and the energy factor is expressed as:

[0111]

[0112] Wherein, H(x)[y] is the logit value of y∈{1,...,N}, y is the predicted label value sequence number, N is the number of predicted labels, and T is a given temperature.

[0113] Optionally, in an embodiment of the present application, the weight is expressed as:

[0114]

[0115] Wherein, EF(x) is the energy factor, DF(x) is the distance factor, α and β are coefficients, F(x) is the mixing factor, and W i is the mixing weight of i-th field, F i (x) is the mixing factor of i-th field, and s represents s fields.

[0116] Optionally, in an embodiment of the present application, the loss function is expressed as:

[0117] min L total =L ce +λL reg

[0118] Wherein, the classification loss L ce is the cross-entropy loss between the probability of the current field and the corresponding target label, λ is a coefficient, and L reg is the regularization loss, and L reg is expressed as:

[0119] L reg =L border +L midline

[0120] Wherein, L border is the boundary loss, L midline is the median loss,

[0121]

[0122] Wherein, Θ is the set energy boundary threshold, and Δ is the energy median value, For a dataset, x is a picture feature vector and y is a label value corresponding to the picture in (x, y).

[0123] It should be noted that the foregoing description of the embodiment of the image classification method based on the energy field and the distance space prompt incremental aggregation is also applicable to the embodiment of the image classification device based on the energy field and the distance space prompt incremental aggregation, and details are not repeated here.

[0124] In the description of the present specification, the description of the terms "one embodiment", "some embodiments", "an example", "a specific example" or "some examples" and the like means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In the present specification, the illustrative description of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any appropriate manner in any one or more embodiments or examples. In addition, the person skilled in the art can combine and combine the different embodiments or examples described in the present specification and the features of the different embodiments or examples without contradiction.

[0125] In addition, the terms "first", "second" are only for descriptive purposes and cannot be understood as indicating or implying relative importance or implicitly indicating the number of indicated technical features. Therefore, the features defined with "first", "second" can explicitly or implicitly include at least one of the features. In the description of the present application, the meaning of "multiple" is at least two, for example, two, three, etc., unless otherwise explicitly and specifically limited.

[0126] Any process or method descriptions in flow charts or otherwise described herein can be understood as representing code modules, segments, or portions of code that include one or more executable instructions for implementing the specified logic functions or processes, and the preferred embodiments of the present application also include additional implementation examples, in which the functions can be performed in different orders, in different ways, or in reverse, and the described embodiments should not be construed as limited to the described or discussed order or sequence of functions, and the skilled person in the art should understand that the described embodiments can be implemented in other ways, according to the functions involved.

[0127] The logic and / or steps represented in flow diagrams or otherwise described herein, for example, can be considered as a sequence of executable instructions, and can be embodied in any computer-readable medium for use by or in connection with an instruction execution system, apparatus, or device, such as a computer-based system, processor-containing system, or other system that can fetch the instructions from the instruction execution system, apparatus, or device and execute the instructions. For purposes of this specification, a "computer-readable medium" can be any apparatus that can contain, store, communicate, propagate, or transport the program for use by or in connection with the instruction execution system, apparatus, or device. The computer-readable medium can be a product of the manufacturing and / or processing. The computer-readable medium can include, but is not limited to, the following: an electronic connection (an electronic device having one or more wires), a portable computer diskette (a magnetic device), a RAM (random access memory), a ROM (read-only memory), an EPROM (erasable programmable ROM) or a Flash memory, an optical fiber, and a portable CD-ROM (compact disk ROM). In addition, the computer-readable medium can even be paper or another suitable medium upon which the program is printed, as the program can be electronically captured, for example, via the optical scanning of the paper or other medium, then compiled, interpreted, or otherwise processed in a suitable manner, if necessary, and then stored in the computer memory.

[0128] It should be understood that aspects of the application can be implemented in hardware, software, firmware or combinations thereof. In the above embodiments, various steps or methods can be implemented in software or firmware that is stored in memory and executed by a suitable instruction execution system. As such, in some embodiments, the steps or methods can be implemented in a combination of hardware and software. If implemented in hardware, as in another embodiment, any of the above techniques can be implemented with or without the use of a programmable digital signal processor (DSP) or other programmable device. In some embodiments, the steps or methods can be implemented using a combination of different hardware devices.

[0129] Those of skill in the art would understand that information and signals can be represented using any of a variety of technologies and techniques. For example, data, instructions, commands, information, signals, bits, symbols, and chips that can be referenced throughout the above description can be represented by voltages, currents, electromagnetic waves, magnetic fields or particles, optical fields or particles, or any combination thereof.

[0130] In addition, each of the functional units in the various embodiments of the present application can be integrated in one processing module, or each of the units can be physically present separately, or two or more units can be integrated in one module. The integrated module can be realized in the form of hardware or in the form of a software functional module. When the integrated module is realized in the form of a software functional module and sold or used as an independent product, it can also be stored in a computer readable storage medium.

[0131] The storage medium mentioned above can be a read-only memory, a magnetic disk or an optical disk, etc. Although the embodiments of the present application have been shown and described above, it should be understood that the above embodiments are exemplary and should not be construed as limiting the present application, and those skilled in the art can make changes, modifications, replacements and variations to the above embodiments within the scope of the present application.

Claims

1. An image classification method based on energy field and distance space prompt incremental aggregation, characterized in that, Comprise: obtain a dataset of a domain reinforcement learning scenario, wherein the dataset comprises data of at least one domain; training through the dataset, in the training, using the data of the current domain and the frozen CLIP model, optimizing the personalized prompt model of the domain by minimizing the loss function; in inference, the feature vector of the image to be classified is obtained by encoding the image to be classified through the CLIP model, the distance between the feature vector and each domain is calculated, for each domain, the energy and the prediction probability of the image to be classified are generated by fusing the personalized prompt model of the domain and the CLIP model, the distance factor and the energy factor are determined based on the distance and the energy, the domain weight is obtained by combining the distance factor and the energy factor, the weighted sum of the prediction probability is calculated using the domain weight, and the classification mixed probability is generated; wherein the distance between the feature vector and each domain is the minimum distance from the feature vector to the centroid of each domain, the distance factor is represented as: DF i (x) = D min - D i (x) where DF i (x) is the distance factor for the ith domain, DF i (x) e (-∞, 0], D min = min i {D i (x)}, D i (x) is the distance to the ith domain, where f θ (x) is the feature vector of image x, KNN center points of the i-th domain is the j-th KNN centroid of the i-th domain, and k is the number of centroids. the energy factor is Helmholtz free energy, and the energy factor is represented as: wherein H(x)[y] is the logit value of y∈{1,...,N}, y is the predicted label value sequence number, N is the number of predicted labels, and T is a given temperature; the weight is represented as: wherein EF(x) is an energy factor, DF(x) is a distance factor, a, b are coefficients, F(x) is a mixing factor, W i is a mixing weight for the i-th field, F i (x) is a mixing factor for the i-th field, and s indicates that there are s fields.

2. The method of claim 1, wherein, the loss function is represented as: min L total = L ce + λL reg wherein the classification loss L ce is the cross-entropy loss between the probability of the current field and the corresponding target label, and λ is a coefficient, L reg is the regularization loss, and L reg is represented as: L reg = L border + L midline where L border is the boundary loss, L midline is the midline loss, Wherein, Θ is the set energy boundary threshold, Δ is the energy median value, is the dataset, where x is the picture feature vector and y is the corresponding label value of the picture.

3. An image classification apparatus based on energy field and distance space hint incremental aggregation, characterized in that, The image classification model is deployed on the device, and the image classification model is used to realize image classification, wherein the training process of the image classification model comprises: obtain a dataset of a domain reinforcement learning scenario, wherein the dataset comprises data of at least one domain; training through the dataset, in the training, using the data of the current domain and the frozen CLIP model, optimizing the personalized prompt model of the domain by minimizing the loss function; the use process of the image classification model comprises: in inference, the feature vector of the image to be classified is obtained by encoding the image to be classified through the CLIP model, the distance between the feature vector and each domain is calculated, for each domain, the energy and the prediction probability of the image to be classified are generated by fusing the personalized prompt model of the domain and the CLIP model, the distance factor and the energy factor are determined based on the distance and the energy, the domain weight is obtained by combining the distance factor and the energy factor, the weighted sum of the prediction probability is calculated using the domain weight, and the classification mixed probability is generated; wherein the distance between the feature vector and each domain is the minimum distance from the feature vector to the centroid of each domain, the distance factor is represented as: DF i (x) = D min - D i (x) where DF i (x) is the distance factor of the ith domain, DF i (x) e (-∞, 0], D min = min i {D i (x)}, D i (x) is the distance to the ith domain, where f θ (x) is the feature vector of image x, KNN center points of the i-th domain is the j-th KNN centroid of the i-th domain, k is the number of centroids the energy factor is Helmholtz free energy, and the energy factor is represented as: wherein H(x)[y] is the logit value of y∈{1,...,N}, y is the predicted label value sequence number, N is the number of predicted labels, and T is a given temperature; the weight is represented as: wherein EF(x) is an energy factor, DF(x) is a distance factor, a, b are coefficients, F(x) is a mixing factor, W i is a mixing weight for the i-th field, F i (x) is a mixing factor for the i-th field, and s indicates that there are s fields.

4. The apparatus of claim 3, wherein, the loss function is represented as: min L total = L ce + λL reg wherein the classification loss L ce is the cross-entropy loss between the probability of the current field and the corresponding target label, and λ is a coefficient, L reg is the regularization loss, and L reg is represented as: L reg = L border + L midline where L borger is the boundary loss, L midline is the midline loss, Wherein, Θ is the set energy boundary threshold, Δ is the energy median value, is the dataset, x in (x, y) is the picture feature vector, and y is the label value corresponding to the picture.

Images