Image classification method and data processing method of classification model

Abstract

Embodiments of the present specification provide an image classification method and a data processing method of a classification model, wherein the image classification method comprises: obtaining an image to be classified; extracting image feature information of the image to be classified; inputting the image feature information into the classification model to obtain category information of the image to be classified, the classification model being obtained by training a plurality of sample data, the sample data comprising original feature information and enhanced feature information, and the enhanced feature information being obtained by performing cross-space feature enhancement on sample data belonging to different attribute type spaces. The original feature information of the sample data is subjected to cross-space feature enhancement by using sample data of different attribute type spaces to obtain enhanced feature information, and the model is trained based on the original feature information and the enhanced feature information of the plurality of sample data to obtain the classification model, so that the classification model is trained based on high-abundance sample features, and the accuracy of image classification is improved.

Description

Technical Field

[0001] The embodiments in this specification relate to the field of computer technology, and in particular to an image classification method. Background Technology

[0002] In recent years, deep learning has become a popular research direction in the field of machine learning and has been widely used in computer vision, natural language processing and other fields.

[0003] Traditional image classification methods require feature extraction from sample data, followed by training of a classification model based on these extracted features. The accuracy of the classification model is affected by the richness of the sample features. Therefore, a highly accurate image classification method is urgently needed. Summary of the Invention

[0004] In view of this, embodiments of this specification provide an image classification method. One or more embodiments of this specification also relate to a remote sensing image classification method, a data processing method for a classification model, an image classification device, a remote sensing image classification device, a data processing device for a classification model, a computing device, a computer-readable storage medium, and a computer program, to address the technical deficiencies existing in the prior art.

[0005] According to a first aspect of the embodiments of this specification, an image classification method is provided, applied to an edge device, comprising:

[0006] Obtain the image to be classified;

[0007] Extract image feature information from the image to be classified;

[0008] Image feature information is input into a classification model to obtain the category information of the image to be classified. The classification model is trained using multiple sample data, which includes original feature information and enhanced feature information. The enhanced feature information is obtained by cross-space feature enhancement of sample data belonging to different attribute type spaces.

[0009] According to a second aspect of the embodiments of this specification, a remote sensing image classification method is provided, applied to an edge device, comprising:

[0010] Acquire the remote sensing image to be classified selected by the user on the front end;

[0011] Extract image feature information from the remote sensing image to be classified;

[0012] Image feature information is input into a classification model to obtain category information of the remote sensing image to be classified. The classification model is trained using multiple sample data, which includes original feature information and enhanced feature information. Enhanced feature information is obtained by cross-spatial feature enhancement of sample data belonging to different attribute type spaces.

[0013] The category information is fed back to the front end for display.

[0014] According to a third aspect of the embodiments of this specification, a data processing method for a classification model is provided, applied to a cloud-side device, comprising:

[0015] Obtain a sample set, which includes multiple sample data, each carrying a category label;

[0016] The feature information of each sample data is obtained. The feature information includes original feature information and enhanced feature information. The enhanced feature information is obtained by cross-space feature enhancement of sample data belonging to different attribute type spaces.

[0017] Based on the feature information and category labels of each sample data, the preset classification network is trained to obtain the model parameters of the trained classification model.

[0018] The model parameters of the classification model are sent to the edge device.

[0019] According to a fourth aspect of the embodiments of this specification, an image classification apparatus is provided, applied to an end-side device, comprising:

[0020] The first acquisition module is configured to acquire the image to be classified.

[0021] The first extraction module is configured to extract image feature information from the image to be classified.

[0022] The first input module is configured to input image feature information into the classification model to obtain the category information of the image to be classified. The classification model is trained using multiple sample data, which includes original feature information and enhanced feature information. The enhanced feature information is obtained by performing cross-space feature enhancement on sample data belonging to different attribute type spaces.

[0023] According to a fifth aspect of the embodiments of this specification, a remote sensing image classification device is provided, applied to an end-side device, comprising:

[0024] The second acquisition module is configured to acquire the remote sensing image to be classified selected by the user on the front end;

[0025] The second extraction module is configured to extract image feature information from the remote sensing image to be classified.

[0026] The second input module is configured to input image feature information into the classification model to obtain the category information of the remote sensing image to be classified. The classification model is trained using multiple sample data. The sample data includes original feature information and enhanced feature information. The enhanced feature information is obtained by performing cross-spatial feature enhancement on sample data belonging to different attribute type spaces.

[0027] The feedback module is configured to send category information back to the front end for display.

[0028] According to a sixth aspect of the embodiments of this specification, a data processing apparatus for a classification model is provided, applied to a cloud-side device, comprising:

[0029] The third acquisition module is configured to acquire a sample set, which includes multiple sample data, each carrying a category label.

[0030] The acquisition module is configured to acquire feature information of each sample data. The feature information includes original feature information and enhanced feature information. The enhanced feature information is obtained by performing cross-space feature enhancement on sample data belonging to different attribute type spaces.

[0031] The training module is configured to train a preset classification network based on the feature information and category labels of each sample data, and obtain the model parameters of the trained classification model.

[0032] The sending module is configured to send the model parameters of the classification model to the edge device.

[0033] According to a seventh aspect of the embodiments of this specification, a computing device is provided, comprising:

[0034] Memory and processor;

[0035] The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions. When the computer-executable instructions are executed by the processor, they implement the steps of the above-described image classification method, the above-described remote sensing image classification method, or the above-described classification model data processing method.

[0036] According to an eighth aspect of the embodiments of this specification, a computer-readable storage medium is provided that stores computer-executable instructions, which, when executed by a processor, implement the steps of the above-described image classification method, the above-described remote sensing image classification method, or the above-described classification model data processing method.

[0037] According to a ninth aspect of the embodiments of this specification, a computer program is provided, wherein when the computer program is executed in a computer, it causes the computer to perform the steps of the above-described image classification method or the above-described remote sensing image classification method or the above-described classification model data processing method.

[0038] In this embodiment, an image to be classified is acquired; image feature information of the image to be classified is extracted; and the image feature information is input into a classification model to obtain the category information of the image to be classified. The classification model is trained using multiple sample data sets. These sample data sets include original feature information and enhanced feature information. The enhanced feature information is obtained by performing cross-spatial feature enhancement on sample data belonging to different attribute type spaces. By using sample data from different attribute type spaces, cross-spatial feature enhancement is performed on the original feature information of the sample data to obtain the enhanced feature information corresponding to that sample data. This process yields enhanced feature information for multiple sample data sets. The model is then trained based on the original feature information and enhanced feature information of these multiple sample data sets to obtain the classification model. This ensures that the classification model is trained based on highly rich sample features, thus improving the accuracy of image classification. Attached Figure Description

[0039] Figure 1 This is a schematic diagram of the interaction process under a classification model training system architecture provided in one embodiment of this specification;

[0040] Figure 2 This is a framework diagram of a classification model training system architecture provided in one embodiment of this specification;

[0041] Figure 3 This is a flowchart illustrating an image classification method provided in one embodiment of this specification;

[0042] Figure 4a This is a schematic diagram of a feature enhancement method for sample images during the training of a classification model in a class space, provided in one embodiment of this specification.

[0043] Figure 4b This is a schematic diagram of a feature enhancement method for sample images during the training of a classification model in a domain space, provided in one embodiment of this specification.

[0044] Figure 5a This is a class conditional covariance matrix data graph obtained during the training of a classification model provided in one embodiment of this specification without using the CrossSmooth statistic;

[0045] Figure 5b This is a graph of class-conditional covariance matrix data obtained during the training of a classification model provided in one embodiment of this specification, using the CrossSmooth statistic.

[0046] Figure 6 This is an overall flowchart of the training process of a classification model provided in one embodiment of this specification;

[0047] Figure 7 This is a histogram of A-distances for features learned on different major generalization tasks during the training of a classification model provided in one embodiment of this specification.

[0048] Figure 8 This is a flowchart illustrating a remote sensing image classification method provided in one embodiment of this specification;

[0049] Figure 9 This is a flowchart illustrating a data processing method for a classification model provided in one embodiment of this specification;

[0050] Figure 10 This is a schematic diagram of the interactive process of an image classification method provided in one embodiment of this specification;

[0051] Figure 11 This is a schematic diagram of the structure of an image classification device provided in one embodiment of this specification;

[0052] Figure 12 This is a schematic diagram of the structure of a remote sensing image classification device provided in one embodiment of this specification;

[0053] Figure 13 This is a schematic diagram of the structure of a data processing device for a classification model provided in one embodiment of this specification;

[0054] Figure 14 This is a structural block diagram of a computing device provided in one embodiment of this specification. Detailed Implementation

[0055] Many specific details are set forth in the following description to provide a full understanding of this specification. However, this specification can be implemented in many other ways than those described herein, and those skilled in the art can make similar extensions without departing from the spirit of this specification. Therefore, this specification is not limited to the specific implementations disclosed below.

[0056] The terminology used in one or more embodiments of this specification is for the purpose of describing particular embodiments only and is not intended to be limiting of the one or more embodiments of this specification. The singular forms “a,” “described,” and “the” as used in one or more embodiments of this specification and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used in one or more embodiments of this specification refers to and includes any or all possible combinations of one or more associated listed items.

[0057] It should be understood that although the terms first, second, etc., may be used to describe various information in one or more embodiments of this specification, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, first may also be referred to as second without departing from the scope of one or more embodiments of this specification, and similarly, second may also be referred to as first. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to a determination."

[0058] First, the terms and concepts used in one or more embodiments of this specification will be explained.

[0059] Domain Generalization (DG) and Domain Adaptation (DA): Due to the significant advancements in deep neural networks, they have become ubiquitous across various fields, especially in computer vision. However, the success of deep neural networks in various fields relies on a strong assumption: that training and testing data are independent and identically distributed. But in deep learning training scenarios, training and testing data often come from different distributions, and training data comes from multiple data sources with distribution shifts between them. Since deep learning models inherently have the tendency to overfit to the training set, fine-tuning with data from different distributions can lead to "catastrophic forgetting" of the deep learning model. Therefore, this domain shift severely affects the model's generalization ability. To address this issue, researchers have conducted extensive research on DA and DG. In DA, labeled images in the source domain and unlabeled images in the target domain are accessible during model training. However, in real-world scenarios, the target domain is often inaccessible. DG addresses a more challenging problem: learning domain robustness without access to data in the target domain. Recent research shows that data augmentation is highly beneficial for DG. These methods can be categorized into image-level augmentation, feature-level augmentation, and semantic-level augmentation. Deep features in a network are typically linear; specifically, there are many semantic directions in the deep feature space. Therefore, transforming data sample A along one of these directions will produce a feature representation corresponding to another data sample with the same class label but different semantics. Thus, by searching for many such semantic directions, the training set can be effectively augmented in a way that complements traditional data augmentation techniques (random flipping, rotation, cropping, deformation scaling, adding noise, color perturbation, etc.).

[0060] In DG, typically K fields are given, D = {D} 1 D 2 D K During training, K domains can be used for training. K Contains NK Each image and label pair is jointly distributed in Where x represents the image and y represents the label. This represents the i-th image in the k-th domain. Let represent the i-th label in the k-th domain. By training a model using multiple training domains and then using the trained model to obtain data in the target domain, the semantic-level augmentation of DG is called Semantic Augmentation for Distance Metric Learning (SADML).

[0061] SADML performs implicit semantic augmentation based on Fourier Augmented Co-Teacher (FACT), and the augmented features can be used for distance metric learning. FACT developed a framework consisting of implicit constraints derived from Fourier data augmentation and explicit constraints derived from co-teacher regularization. SADML inherits the loss from FACT, denoted as L... FACT However, it not only inherits the loss from FACT, but also adds a loss from Distance Metric Learning (DML). To achieve this, the total loss of SADML can be written as: Where β is a positive coefficient.

[0062] Following semantic enhancement, SADML hypothesizes enhanced features in the K-domain. It is a random directional translation sampled from a Gaussian distribution. get, This refers to the i-th feature in the k-th domain. When... When it is enhanced an infinite number of times, its enhanced features It can be expressed by the following formula (1):

[0063]

[0064] Where W = {w1, w2, ..., w C} T ∈R C×A It is the weight matrix corresponding to the final fully connected layer. Is with y i A weight matrix with classes having the same weight, where λ is a positive coefficient, I∈R C×C A diagonal matrix, J∈R C It is a matrix, where A is the dimension of the features before layering, C is the dimension, and T is the transpose. It means SA stands for Semantic Augmentation.

[0065] The class-conditional covariance matrix of the K-domain is By y i Feature estimation of all samples in the class, y i This refers to the label in the i-th class space. The covariance matrix is ​​calculated online by aggregating the statistical information of all mini-batches.

[0066] SADML uses the structure boosting loss as the loss for DML to improve performance. However, the structure boosting loss is insufficient in regulating the control margin. Recently, supervised contrastive learning (CL) has shown that simultaneous fusion of image augmentation operations improves the performance of contrastive learning models on this task. To further improve model performance, we utilize the loss L of contrastive learning... CL To replace the loss of the lifting structure DML in SADML, we construct the modified total loss of SADML. The overall formula can then be written as: β is a positive coefficient. This indicates that contrastive learning is used instead of DML loss, where the contrastive learning loss can be written as the following formula (2):

[0067]

[0068] Among them, a i a j a b Let i, j, and b be the augmented features. This indicates that the mini-batch belongs to y i Number of images in the class. B represents the number of images in the mini-batch; τ is the temperature coefficient.

[0069] SADML considers sampling within class and domain semantic directions, which can enrich the state of sample data. However, it does not consider inter-class and inter-domain sampling, which may result in limited diversity of sample data.

[0070] Implicit and explicit constraints: If the constraint function can reflect the obvious functional relationship between the design variables, it is called an explicit constraint; otherwise, it is called an implicit constraint.

[0071] Image-level augmentation: Image-level augmentation applies different inter-domain styles to synthetic data at the image level to improve generalization. For example, FACT uses the Fourier transform amplitude mixing method to perform linear interpolation between two different images from an image-level perspective. Since image-level augmentation is based on manual design, the augmented image may be limited and may not appear in real-world scenarios. Therefore, designing data augmentation at the feature level has a broader design prospect for Data Gaining (DG).

[0072] Feature-level augmentation: Feature-level augmentation uses random noise to perturb the feature embeddings of inter-class orientations. However, random noise lacks meaningful semantic information and loses its physical meaning.

[0073] Semantic-level augmentation: Semantic-level augmentation introduces semantic orientations from the Implicit Semantic Data Augmentation (ISDA) algorithm into Data Generation Groups (DGs) and theoretically proves that features obtained from ISDA can be used as novel features in Data Modeling (DML). Furthermore, it samples an infinite number of semantic orientations from intra-class and intra-domain distributions for augmentation. While semantic-level augmentation brings significant progress to DGs, we argue that it neglects inter-class and inter-domain cues, which are considered important for DGs. Simultaneously, due to the lack of real-world semantics, existing image-based inter-domain methods and feature-based inter-class methods cannot be directly applied to semantic-level augmentation.

[0074] Data augmentation is a technique that artificially expands a training dataset by generating more equivalent data from a limited dataset.

[0075] ISDA aims to perform more "advanced," "semantic" transformations on samples, such as changing the background, color, and viewpoint of objects, without altering the task label. From the perspective of semantic data augmentation, the resulting algorithm ultimately reduces to a completely new loss function. This algorithm can also be applied to any visual task using the Soft-Max cross-entropy loss function, such as image segmentation.

[0076] Traditional semantic-level augmentation introduces semantic orientations from ISDA into Data Generation Groups (DGs) and theoretically proves that features obtained from ISDA can be used as novel features in Data Modeling (DML). Furthermore, it samples an infinite number of semantic orientations from intra-class and intra-domain distributions for augmentation. While semantic-level augmentation has brought significant progress to DGs, we argue that it neglects inter-class and inter-domain cues, which are considered crucial for DGs. Simultaneously, due to the lack of real-world semantics, existing image-based inter-domain methods and feature-based inter-class methods cannot be directly applied to semantic-level augmentation; therefore, a method to improve the richness of sample features is urgently needed.

[0077] To address the aforementioned issues, this embodiment of the specification acquires an image to be classified; extracts image feature information from the image to be classified; and inputs the image feature information into a classification model to obtain the category information of the image to be classified. The classification model is trained using multiple sample data sets. These sample data sets include original feature information and enhanced feature information. The enhanced feature information is obtained by performing cross-spatial feature enhancement on sample data belonging to different attribute type spaces. By utilizing sample data from different attribute type spaces, cross-spatial feature enhancement is performed on the original feature information of the sample data to obtain the enhanced feature information corresponding to that sample data. This process yields enhanced feature information for multiple sample data sets. The model is then trained based on the original feature information and enhanced feature information of these multiple sample data sets to obtain the classification model. This ensures that the classification model is trained based on highly rich sample features, significantly improving the classification efficiency and accuracy of the model.

[0078] This specification provides an image classification method, and also relates to a remote sensing image classification method, a data processing method for a classification model, an image classification device, a remote sensing image classification device, a data processing device for a classification model, a computing device, and a computer-readable storage medium, which will be described in detail in the following embodiments.

[0079] See Figure 1 , Figure 1 This specification illustrates a schematic diagram of the interaction process under a classification model training system architecture provided in one embodiment, as shown below. Figure 1 As shown, the classification model training system includes cloud-side devices and edge-side devices;

[0080] The edge device is used to build a sample set and send the sample set to the cloud device. The sample set includes multiple sample data, and the sample data carries category labels.

[0081] The cloud-based device is used to obtain feature information of each sample data. The feature information includes original feature information and enhanced feature information. The enhanced feature information is obtained by cross-space feature enhancement of sample data belonging to different attribute type spaces. Based on the feature information and category labels of each sample data, the preset classification network is trained to obtain the model parameters of the trained classification model.

[0082] Furthermore, the cloud-side device is also used to send the model parameters of the classification model to the edge-side device.

[0083] According to the scheme of the embodiments in this specification, the edge device constructs a sample set and sends the sample set to the cloud device. The sample set includes multiple sample data, each carrying a category label. The cloud device obtains feature information for each sample data, including original feature information and enhanced feature information. The enhanced feature information is obtained by performing cross-space feature enhancement on sample data belonging to different attribute type spaces. Based on the feature information and category labels of each sample data, a preset classification network is trained to obtain the model parameters of the trained classification model. The cloud device sends the model parameters of the classification model to the edge device. By utilizing sample data from different attribute type spaces, cross-space feature enhancement is performed on the original feature information of the sample data to obtain the enhanced feature information corresponding to that sample data. This results in obtaining enhanced feature information for multiple sample data. The model is then trained based on the original and enhanced feature information of these multiple sample data to obtain a classification model. This improves the sample richness for training the classification model. The edge device can perform image classification based on the model parameters sent by the cloud device, thus improving the efficiency and accuracy of image classification.

[0084] See Figure 2 , Figure 2 This specification illustrates a framework diagram of a classification model training system architecture according to an embodiment of the present specification. The system may include a cloud-side device and multiple edge devices. The multiple edge devices can establish communication connections through the cloud-side device. In a classification model training scenario, the cloud-side device is used to provide classification model training services between the multiple edge devices. The multiple edge devices can act as either senders or receivers, achieving real-time communication through the cloud-side device.

[0085] Users can interact with cloud devices through their edge devices to receive data from other edge devices or send data to other edge devices. In a classification model training scenario, users can publish data streams to cloud devices through their edge devices. The cloud devices can then train classification models based on these data streams and push the trained classification models to other edge devices that have established communication with them.

[0086] In this process, the edge device and the cloud device establish a connection through a network. The network provides the medium for communication between the edge device and the cloud device. The network can include various connection types, such as wired, wireless communication links, or fiber optic cables. The data transmitted by the edge device may need to undergo encoding, transcoding, compression, and other processing before being published to the cloud device.

[0087] Edge devices can be browsers, apps (applications), web applications such as H5 (HyperText Markup Language 5) applications, lightweight applications (also known as mini-programs), or cloud applications. Edge devices can be developed using software development kits (SDKs) provided by cloud-based devices, such as real-time communication (RTC) SDKs. Edge devices can be deployed in electronic devices and depend on the device's operation or certain apps running on it. Electronic devices may have displays and support information browsing, such as personal mobile terminals like smartphones, tablets, and personal computers. Various other types of applications can also be configured in electronic devices, such as human-computer interaction applications, model training applications, text processing applications, web browser applications, shopping applications, search applications, instant messaging tools, email clients, and social media platforms.

[0088] Cloud-side devices can include servers providing various services, such as servers providing communication services to multiple clients, servers supporting backend training of models used on clients, and servers processing data sent by clients. It should be noted that cloud-side devices can be implemented as a distributed server cluster composed of multiple servers, or as a single server. Servers can also be servers in a distributed system, or servers incorporating blockchain technology. Servers can also be cloud servers providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms, or intelligent cloud computing servers or intelligent cloud hosts with artificial intelligence technology.

[0089] It is worth noting that the data processing method for the classification model provided in the embodiments of this specification can be executed by a cloud-side device. However, in other embodiments of this specification, the edge device may also have similar functions to the cloud-side device, thereby executing the data processing method for the classification model provided in the embodiments of this specification. In other embodiments, the data processing method for the classification model provided in the embodiments of this specification may also be executed jointly by the edge device and the cloud-side device.

[0090] See Figure 3 , Figure 3 This specification illustrates a flowchart of an image classification method provided in one embodiment, applied to an edge device, which specifically includes the following steps:

[0091] Step 302: Obtain the image to be classified.

[0092] When there is a need for image classification, the edge device will acquire the image to be classified. The image to be classified can be input by the user at the front end, or it can be obtained by the edge device from the image library to be classified.

[0093] In one possible implementation of this specification, the terminal device can open a channel for uploading images to be classified by the user's click (e.g., via Bluetooth, network, etc.), allowing the user to upload images and the terminal device to classify them.

[0094] In another possible implementation of this specification, there may be many images in the terminal device, and the user clicks on the specific image that needs to be classified, and then the terminal device classifies the clicked image.

[0095] Step 304: Extract the image feature information of the image to be classified.

[0096] Specifically, image feature information refers to information that characterizes the features of an image. Data can be obtained through the image feature information of an image. For example, the image feature information extracted from the image to be classified is: dog, yellow fur, standing, doorway, painting style, etc.

[0097] In one or more embodiments of this specification, after the end-side device acquires the image to be classified, it can further extract the image feature information of the image to be classified.

[0098] Feature extraction is necessary because the images to be classified may contain a lot of redundant information or the feature dimensions of these images may be very sparse. If the images to be classified are processed directly, the output results may be inefficient. Therefore, it is necessary to extract features from the images to be classified in order to obtain the image feature information of the images to be classified.

[0099] By applying the scheme of the embodiments of this specification, feature extraction is performed on multiple images to be classified to obtain image feature information of each image to be classified, so that subsequent classification can be performed based on the image feature information of the images to be classified.

[0100] Step 306: Input the image feature information into the classification model to obtain the category information of the image to be classified. The classification model is trained using multiple sample data. The sample data includes original feature information and enhanced feature information. The enhanced feature information is obtained by cross-space feature enhancement of sample data belonging to different attribute type spaces.

[0101] Specifically, category information refers to the classification result obtained when an image to be classified is input into a classification model. For example, inputting a photograph of a cat into a classification model will result in the classification result as "True Style, Cat," meaning the category information is "True Style, Cat." Cross-spatial feature enhancement refers to feature enhancement using different attribute type spaces. For example, feature enhancement can be performed using the original feature information of sample data from different class spaces. Enhanced feature information refers to the enhanced feature information obtained after cross-spatial feature enhancement. For example, performing feature enhancement on sample data in a first attribute type space across a second attribute type space yields enhanced feature information of the first sample data based on the second attribute type space. The first and second attribute type spaces belong to different attribute type spaces.

[0102] In one or more embodiments of this specification, after the end-side device acquires the image to be classified and extracts the image feature information of the image to be classified, it inputs the image feature information of the image to be classified into the classification model to obtain the category information of the image to be classified.

[0103] For example, the first image to be classified and the second image to be classified are any two images belonging to different attribute type spaces. The image feature information of the first image to be classified can be a dog, yellow fur, standing, and at the door of the house. The image feature information of the second image to be classified can be a cat, yellow and white striped fur, sitting, and on the stairs. Based on the image feature information of the first image to be classified, cross-space feature enhancement is performed on the second image to be classified, and the enhanced feature information of the second image to be classified can be a cat, yellow and white striped fur, standing, and on the stairs.

[0104] The scheme described in this specification involves acquiring an image to be classified; extracting image feature information from the image to be classified; and inputting the image feature information into a classification model to obtain the category information of the image to be classified. The classification model is trained using multiple sample data sets. These sample data sets include original feature information and enhanced feature information. The enhanced feature information is obtained by performing cross-spatial feature enhancement on sample data belonging to different attribute type spaces. By using sample data from different attribute type spaces, cross-spatial feature enhancement is performed on the original feature information of the sample data to obtain the enhanced feature information corresponding to that sample data. This process yields enhanced feature information for multiple sample data sets. The model is then trained based on the original feature information and enhanced feature information of these multiple sample data sets to obtain the classification model. This ensures that the classification model is trained based on highly rich sample features, improving the classification efficiency and accuracy of image classification.

[0105] In one optional embodiment of this specification, the image classification method described above further includes the following specific steps:

[0106] Obtain a sample set, wherein the sample set includes multiple sample data, and the sample data carries category labels;

[0107] Obtain feature information for each of the sample data, wherein the feature information includes original feature information and enhanced feature information, and the enhanced feature information is obtained by performing cross-space feature enhancement on sample data belonging to different attribute type spaces;

[0108] Based on the feature information and category labels of each sample data, a preset classification network is trained to obtain a trained classification model.

[0109] In one or more embodiments of this specification, to improve the classification efficiency and accuracy of the trained classification model, a sample set can be obtained when model training begins. Furthermore, to enhance the generalizability of the trained classification model, multiple sample data can be used to train the model.

[0110] Specifically, a sample set refers to the set of samples obtained when adjusting the parameters of the model; it is the collection of training samples, typically including the training set and the test set. A category label refers to the category to which the sample data belongs, carried in the sample data in the form of a label. For example, one portrait might be labeled as "painting style," while another might be labeled as "realistic style," meaning it was taken as a photograph, and so on.

[0111] In one or more embodiments of this specification, the sample set contains multiple sample data, each carrying a corresponding label. For example, the multiple sample data can be sample data in multiple different attribute type spaces. Specifically, the sample data can be sample images, where one sample data can be: a sitting cat; another sample data can be: a standing dog, etc.

[0112] Optionally, obtaining a sample set can be based on the need to train the model, or it can be based on the need to perform data augmentation or feature enhancement on the sample data in the sample set.

[0113] By acquiring a sample set containing multiple sample data points, and with each sample data point carrying a category label, a classification model can be trained based on the acquired sample set.

[0114] The original feature information and enhanced feature information of each sample data are used together as the feature information of the sample data when training the classification model.

[0115] By obtaining feature information composed of the original feature information and the enhanced feature information of each sample data, the classification model is trained. This ensures that the subsequent model training includes not only the original feature information of the sample data, but also the enhanced feature information, thereby improving the accuracy of the classification model.

[0116] Optionally, training the preset classification network based on the feature information and category labels of each sample data can be done by sequentially training the preset classification network based on the original feature information and category labels of multiple sample data; alternatively, each sample data can be divided into two parts, one part for training the preset classification network and the other part for testing the trained classification model.

[0117] Based on the feature information and category labels of each sample data, a pre-defined classification network is trained to obtain a trained classification model. The feature information includes both original feature information and enhanced feature information, making the trained classification model adaptable to unknown data. It can classify sample data in unknown class spaces or domain spaces and obtain accurate classification results.

[0118] In one optional embodiment of this specification, the process may involve extracting feature information and target category labels from target data samples, inputting the feature information of the target data samples into a preset classification network to obtain classification results for the target data samples, calculating a total loss value based on the feature information, classification results, and target category labels of the target data samples, adjusting the network parameters of the preset classification network based on the total loss value, and returning to the step of extracting feature information and target category labels from the target sample data. If the training stopping condition is met, the training of the preset classification network is considered complete, and a classification model is obtained. Specifically, the steps of training the preset classification network based on the feature information and category labels of each sample data to obtain a trained classification model include the following steps:

[0119] Extract feature information and target category label from target sample data, wherein the target sample data is any one of the plurality of sample data;

[0120] The feature information of the target sample data is input into a preset classification network to obtain the classification result of the target sample data;

[0121] The total loss value is calculated based on the feature information of the target sample data, the classification result, and the target category label;

[0122] The network parameters of the preset classification network are adjusted based on the total loss value, and the process of extracting feature information and target category labels from the target sample data is returned.

[0123] If the training stopping condition is met, the training of the preset classification network is completed, and a classification model is obtained.

[0124] Specifically, the training stopping condition refers to stopping the training of the model when a certain condition is met. For example, training stops when the total loss value reaches a preset loss threshold. The training stopping condition is whether the preset loss threshold is met.

[0125] Optionally, any one of the multiple sample data is extracted as the feature information and category label carried by the target sample data. The feature information of the target sample data is input into a preset classification network to obtain a classification result output by the preset classification network. The original feature information of the sample data, the classification result and the category label carried by the target sample data are calculated to obtain the total loss value.

[0126] Optionally, after calculating the total loss value, it is determined whether the total loss value meets the preset loss threshold. If it does, training is stopped; if not, the network parameters of the preset classification network are adjusted based on the total loss value. After adjustment, the process returns to extracting one sample from multiple sample data as the target sample data and continuing to train the preset classification network with adjusted network parameters until the total loss value meets the preset loss threshold.

[0127] By applying the scheme of the embodiments of this specification, the total loss value is obtained by inputting the feature information of the extracted target sample data into the classification result obtained by the preset classification network, the category label carried by the first sample data and the original feature information of the first sample data, and training the network parameters of the preset classification network based on the total loss value until the training stopping condition is reached. Then, a classification model is obtained, that is, the obtained classification model can classify data in unknown class space or domain space.

[0128] In one optional embodiment of this specification, the above steps, which calculate the total loss value based on the feature information of the target sample data, the classification result, and the target category label, include the following specific steps:

[0129] Based on the original feature information and enhanced feature information of the target sample data, calculate the contrastive learning loss;

[0130] Calculate the classification loss based on the classification result and the target category label;

[0131] The total loss value is calculated based on the contrastive learning loss and the classification loss.

[0132] Specifically, contrastive learning loss refers to the loss obtained by comparing two pieces of information of the same type and calculating the difference using a loss function. Classification loss refers to the loss calculated by comparing the classification result output by a pre-defined classification network with the class labels of the sample data itself.

[0133] After applying traditional data augmentation techniques to the original feature information of the sample data, the resulting contrastive learning loss is: The contrastive learning loss generated after cross-spatial feature enhancement It is the contrastive learning loss calculated based on the original feature information and enhanced feature information of the target sample data.

[0134] Calculating the classification loss based on the classification result and the target category label can be done by first determining whether the model's output classification result meets preset conditions. If it does, a classification probability is calculated based on the category to which the classification result belongs. This probability is then compared with the target category label to calculate the loss value. For example, the classification loss L can be calculated using a loss function based on the classification probability and the target category label. FACT The loss function can be the binary cross-entropy loss function.

[0135] Among them, the overall enhanced feature information is generated based on cross-space feature enhancement in class space and domain space.

[0136] The total loss value is calculated based on the contrastive learning loss and the classification loss, as shown in the following formula (3):

[0137]

[0138] The scheme implemented in this specification calculates the contrastive learning loss based on the original feature information and enhanced feature information of the target sample data; it calculates the classification loss based on the classification result and category label of the target sample data output by the preset classification network; it calculates the total loss value based on the contrastive learning loss and the classification loss; it calculates the contrastive learning loss using the original feature information and enhanced feature information, and the classification loss using the classification result and category label; and it obtains the total loss value based on the two losses. The classification model trained based on the total loss value will improve the generalization ability overall and further improve the accuracy of the classification model.

[0139] In one optional embodiment of this specification, the steps for obtaining feature information of each sample data include the following specific steps:

[0140] Extract the original feature information of each sample data.

[0141] Based on the original feature information of the first sample data, cross-spatial feature enhancement is performed on the second sample data to obtain the enhanced feature information of the second sample data, wherein the first sample data and the second sample data are any two sample data belonging to different attribute type spaces in the sample set.

[0142] Specifically, original feature information refers to information that characterizes the features of data. Data can be obtained through the original feature information of data. For example, if the sample data is a sample image, the original feature information extracted from the sample image is: dog, yellow fur, standing, doorway, painting style, etc.

[0143] Optionally, the extraction of the original feature information of each sample data is based on the fact that each sample data belongs to a different attribute type space, and feature enhancement is performed by using the original feature information of sample data from multiple different attribute type spaces.

[0144] Optionally, there are many ways to extract features from sample data to obtain the original feature information. One possible approach is to reduce the dimensionality of the sample data to obtain its original feature information. For example, if the sample data is a sentence, "A lazy, yellow-haired dog is sitting against the wall at the front door," by reducing the dimensionality, the key information extracted as the original feature information of the sample data could be "dog, yellow hair, sitting." Another possible approach is to use the information in the sample data as the original feature information obtained by describing it verbally. For example, if there is an image as a sample, and the image shows a sitting yellow-haired dog, then by describing the image verbally, the original feature information obtained could be "dog, yellow hair, sitting."

[0145] Feature extraction is necessary because sample data may contain a lot of redundant information or the feature dimensions of these sample data may be very sparse. If the sample data is processed directly, the output results may be inefficient. Therefore, it is necessary to extract features from the sample data first to obtain the original feature information of the sample data.

[0146] By extracting features from multiple sample data, the original feature information of each sample data is obtained, which enables subsequent feature enhancement based on the original feature information of each sample data.

[0147] Optionally, based on the original feature information of the first sample data, cross-spatial feature enhancement is performed on the second sample data to obtain the feature information of the attribute type space to which the first sample data belongs. The obtained feature information is combined with some of the original feature information to obtain the enhanced feature information of the sample data. The enhanced feature information obtained is different from the original feature information and also different from the feature information of the other attribute type space, but the label corresponding to the original feature information remains unchanged.

[0148] See Figure 4a and Figure 4b , Figure 4a This diagram illustrates a method for feature enhancement of sample images during the training of a classification model in a class space, according to an embodiment of this specification. Figure 4b This diagram illustrates a feature enhancement method for sample images during the training of a classification model in a domain space, as provided in one embodiment of this specification.

[0149] Figure 4a In the example, the original feature information of sample data 1 is: a sitting dog; the original feature information of sample data 2 is: a standing cat. Based on the original feature information of sample data 1, cross-class spatial feature enhancement is performed on sample data 2 to obtain the enhanced feature information: a sitting cat; based on the original feature information of sample data 2, cross-class spatial feature enhancement is performed on sample data 1 to obtain the enhanced feature information: a standing dog. The enhanced image is obtained based on the generated enhanced feature information.

[0150] Figure 4b In this context, the distribution directions within different domain spaces include: painting style, sketching style, and rapid drawing style. The original feature information of sample data 1 is: a real cat. Using a generator, the real cat (real domain) is used to enhance the painting style cat (painting domain), generating an enhanced painting domain cat.

[0151] In one optional embodiment of this specification, the above steps, based on the original feature information of the first sample data, perform cross-spatial feature enhancement on the second sample data to obtain the enhanced feature information of the second sample data, including the following specific steps:

[0152] Based on the original feature information of the first sample data, cross-class spatial feature enhancement is performed on the second sample data in the target semantic direction to obtain the enhanced feature information of the second sample data in the target semantic direction, wherein the target semantic direction is any specific semantic direction contained in the first class space.

[0153] Specifically, the target semantic direction refers to any specific semantic direction among multiple semantic directions in a class space. For example, the target semantic direction of a sample image in the class space could be pose, position, color, background, etc. "Specific" means specifically designated; for example, it could be the pose specifically designated within the target semantic direction of the sample image in the class space.

[0154] For example, the original feature information of the first sample data is dog, yellow fur, standing, and at the door of the house, and the original feature information of the second sample data is cat, yellow and white striped fur, sitting, and on the stairs. The target semantic direction is posture. Then, based on the original feature information of the first sample data, cross-class spatial feature enhancement is performed on the second sample data in the posture direction to obtain the enhanced feature information of the second sample data in the posture direction as cat, yellow and white striped fur, standing, and on the stairs.

[0155] Optionally, based on the original feature information of the first sample data, cross-spatial feature enhancement is performed on the second sample data in the target semantic direction to obtain the number of enhanced feature information of the second sample data in the target semantic direction, which varies with the change of the target semantic direction. For example, if the first target semantic direction is pose, then the feature information of pose in the original feature information of the first sample data can be obtained; if the second target semantic direction is position, then the feature information of position in the original feature information of the first sample data can be obtained, and so on.

[0156] By applying the scheme of the embodiments of this specification, based on the original feature information of the first sample data, cross-class spatial feature enhancement is performed on the second sample data in the target semantic direction to obtain the enhanced feature information of the second sample data in the target semantic direction. This results in the enhanced feature information of the second sample data containing both some of the original feature information of the second sample data and the feature information in the specific target semantic direction of the first sample data, thereby improving the richness of the feature information contained in the sample data and further enhancing the richness of the sample data.

[0157] In one optional embodiment of this specification, the above steps, based on the original feature information of the first sample data, perform cross-class spatial feature enhancement on the second sample data in the target semantic direction to obtain the enhanced feature information of the second sample data in the target semantic direction, including the following specific steps:

[0158] Based on the original feature information of the first sample data, determine the sampling probability of the first sample data in the target semantic direction;

[0159] Based on the original feature information of the second sample data and the sampling probability, calculate the inter-class enhancement index;

[0160] Based on the original feature information of the second sample data and the inter-class enhancement index, the enhanced feature information of the second sample data in the target semantic direction is obtained.

[0161] Specifically, the sampling probability refers to the probability of sampling a certain semantic direction from a certain class space of sample data, usually denoted by λ0. The inter-class enhancement index is an index calculated based on the original feature information of the first sample data and the original feature information of the second sample data, through the weight matrix of the fully connected layer corresponding to their respective class spaces.

[0162] The inter-class enhancement index is calculated using the original feature information of the second sample data and the sampling probability. This is achieved through a CrossSmooth sampling method, using a diagonal matrix I to sample the target semantic direction between classes. Specifically, the sampling probability λ0 of the first sample data in the target semantic direction is first determined using the original feature information of the first sample data in the first class space. Then, the inter-class enhancement index is calculated based on the original feature information of the second sample data and the sampling probability λ0.

[0163] Based on the original feature information of the second sample data Using the inter-class enhancement metrics, we obtain enhanced feature information of the second sample data in the target semantic direction.

[0164] For example, CrossSmooth can be used to increase the sampling probability between classes. That is, the first sample data is a picture of a cat and the second sample data is a picture of a dog. The state of the dog is enriched by the sampling probability of the cat's direction. Based on the sitting cat, cross-class spatial feature enhancement is performed on the standing dog to obtain the sitting dog.

[0165] By applying the scheme of the embodiments of this specification, the sampling probability of the first sample data in the target semantic direction is determined based on the original feature information of the first sample data. The inter-class enhancement index is calculated based on the sampling probability and the original feature information of the second sample data. Then, the enhanced feature information of the second sample data in the target semantic direction is obtained based on the original feature information of the second sample data and the inter-class enhancement index. This results in the enhanced feature information of the generated second sample data containing the feature information of the original feature information of the first sample data in the first class space in the target semantic direction, which enriches the feature information of the sample data and further improves the richness of the sample data.

[0166] In one optional embodiment of this specification, the above steps for calculating the inter-class enhancement index based on the original feature information of the second sample data and the sampling probability include the following specific steps:

[0167] The original feature information of the second sample data is statistically analyzed to obtain the class conditional covariance matrix of the second class space;

[0168] Based on the sampling probability and the preset diagonal matrix, construct the inter-class sampling matrix;

[0169] Based on the class-conditional covariance matrix and the inter-class sampling matrix, determine the correlation matrix between the first class space and the second class space;

[0170] Based on the preset feature enhancement intensity and the correlation matrix, the inter-class enhancement index is calculated.

[0171] Specifically, the class-conditional covariance matrix refers to the statistical analysis of sample data within a class space. The pre-set diagonal matrix is ​​a pre-defined diagonal matrix, where the values ​​on opposite sides of the matrix are the same; it is usually denoted by I and is a diagonal matrix set according to the CrossSmooth sampling method. Covariance refers to the overall error between the original feature information. The inter-class sampling matrix is ​​an inter-class sampling matrix constructed from information in the class space belonging to the target semantic direction of cross-space feature enhancement. The correlation matrix is ​​a matrix between two class spaces (where there is a correlation for feature enhancement along the target semantic direction). The pre-set feature enhancement intensity is a pre-defined feature enhancement intensity, usually denoted by λ.

[0172] Statistical analysis of the original features of the second sample data yields the class-conditional covariance of the second class space as follows: Based on the sampling probability and the preset diagonal matrix, multiply them to construct the inter-class sampling matrix λ0I; based on the class conditional covariance matrix and the inter-class sampling matrix, the weight matrix of the fully connected layer in the first classification network corresponding to the first sample data, and the weight matrix of the fully connected layer in the second classification network corresponding to the second class space, determine the correlation matrix between the first class space and the second class space, where λ0 represents the probability intensity of sampling from the inter-class space, and I is the diagonal matrix, as shown in the following equation (4):

[0173]

[0174] Based on the preset feature enhancement intensity and correlation matrix, the inter-class enhancement index is calculated, as shown in the following formula (5):

[0175]

[0176] After obtaining the inter-class enhancement index, based on the original feature information of the sample data... Using inter-class enhancement metrics, we can determine the enhanced feature information of the second sample data in the target semantic direction. For: the original feature information of the second sample data Adding it to the inter-class enhancement index yields enhanced feature information. CS stands for CrossSmooth, as shown in equation (6) below:

[0177]

[0178] Optionally, the original feature information of the second sample data is statistically analyzed to obtain the class conditional covariance matrix of the second class space. Specifically, based on the second class space to which the second sample data belongs, the original feature information of multiple sample data in the second class space is statistically analyzed to obtain the class conditional covariance matrix of the second class space.

[0179] See Figure 5a and Figure 5b , Figure 5a This diagram illustrates a class-conditional covariance matrix obtained during training of a classification model provided in one embodiment of this specification without using the CrossSmooth statistic. Figure 5b The diagram illustrates a class-conditional covariance matrix obtained during the training of a classification model provided in one embodiment of this specification, using the CrossSmooth statistic.

[0180] Figure 5a Without using the CrossSmooth statistic, the class conditional covariance matrix data obtained using dogs and cats is 0.23, the class conditional covariance matrix data obtained using dogs and giraffes is 0.34, ..., and the class conditional covariance matrix data obtained using domestic animals and humans is 0.28.

[0181] Figure 5b When using the CrossSmooth statistic, the class conditional covariance matrix data obtained using dogs and cats is 0.64, the class conditional covariance matrix data obtained using dogs and giraffes is 0.73, ..., and the class conditional covariance matrix data obtained using domestic animals and humans is 0.09.

[0182] Optionally, the CrossSmooth statistic was used to improve the inter-class enhancement index.

[0183] By applying the scheme of the embodiments of this specification, the class conditional covariance matrix of the second class space is obtained by statistically analyzing the original feature information of the second sample data; an inter-class sampling matrix is ​​constructed based on the sampling probability and a preset diagonal matrix; the correlation matrix between the first and second class spaces is determined based on the class conditional covariance matrix and the inter-class sampling matrix; and an inter-class enhancement index is calculated based on the preset enhancement intensity and the correlation matrix. Subsequently, the enhanced feature information of the second sample data can be determined based on the inter-class enhancement index and the original feature information of the second sample data. By calculating the original information of the second sample data with the sampling probability obtained from the original feature information of the first sample data, enhanced feature information of the sample data is generated. Based on the enhanced feature information of the sample data, sample data corresponding to the enhanced feature information is formed, further improving the richness of the sample data.

[0184] In one optional embodiment of this specification, the above steps, based on the original feature information of the first sample data, perform cross-spatial feature enhancement on the second sample data to obtain the enhanced feature information of the second sample data, including the following specific steps:

[0185] Based on the original feature information of the first sample data, cross-domain spatial feature enhancement is performed on the second sample data in the target distribution direction to obtain the enhanced feature information of the second sample data in the target distribution direction, wherein the target distribution direction is the distribution direction of the first domain space.

[0186] Specifically, the target distribution direction refers to the distribution direction in a certain domain space. For example, the target distribution direction of a sample image in the domain space can be a style: painting, photography, compositing, etc. The domain space is a space of a certain domain. For example, the domain space includes the distribution direction, usually only one, such as style or pattern: painting, photography, compositing, etc.

[0187] For example, the original feature information of the first sample data in the first domain space is painting style and cat; the original feature information of the second sample data in the second domain space is real style and dog, and the target distribution direction is the distribution direction of painting style in the first domain space. Then, based on the original feature information of the first sample data, cross-domain space feature enhancement is performed on the painting style of the second sample data to obtain the enhanced feature information of the second sample data in the painting style direction as painting style and dog.

[0188] By using CrossVariance to sample the distribution direction from different domains, and adding a painting style to cats in the real style domain based on the painting style domain.

[0189] By applying the scheme of the embodiments of this specification, based on the original feature information of the first sample data, cross-domain spatial feature enhancement is performed on the second sample data in the target distribution direction to obtain the enhanced feature information of the second sample data in the target distribution direction. This makes the enhanced feature information of the generated second sample data the original feature information along the distribution direction of the domain space to which the first sample data belongs, thereby increasing the richness of the original feature information of the sample data and thus greatly improving the richness of the sample data.

[0190] In one optional embodiment of this specification, the above steps, based on the original feature information of the first sample data, perform cross-domain spatial feature enhancement on the second sample data in the target distribution direction to obtain the enhanced feature information of the second sample data in the target distribution direction, including the following specific steps:

[0191] The original feature information of the first sample data is statistically analyzed to obtain the covariance matrix of the first domain space;

[0192] Based on the covariance matrix, determine the intra-domain correlation matrix of the first domain space;

[0193] Based on the preset feature enhancement strength and the intra-domain correlation matrix, the inter-domain enhancement index is calculated;

[0194] Based on the original feature information of the second sample data and the inter-domain enhancement index, the enhanced feature information of the second sample data in the target distribution direction is obtained.

[0195] Specifically, the covariance matrix of the first domain space is obtained by statistically analyzing the sample data within the first domain space. The intra-domain correlation matrix is ​​determined by the intra-domain covariance and the original feature information of multiple sample data within the domain. The inter-domain enhancement index is an index calculated based on the original feature information of the first sample data and the original feature information of the second sample data, using the weight matrices of the fully connected layers corresponding to their respective domain spaces.

[0196] Optionally, the original feature information of the first sample data is statistically analyzed to determine the covariance matrix of the first domain space. Specifically, based on the first domain space to which the first sample data belongs, the original feature information of multiple sample data within the first domain space is statistically analyzed to obtain the covariance matrix of the first domain space.

[0197] The intra-domain correlation matrix of the first domain space is derived from the covariance matrix of the first domain space and the weight matrix W of the fully connected layer in the first classification network corresponding to the first sample data, and the weight matrix of the fully connected layer in the second classification network corresponding to the second domain space. The result is confirmed, as shown in the following formula (7):

[0198]

[0199] Based on the preset feature enhancement intensity λ and the intra-domain correlation matrix of the first domain space, the inter-domain enhancement index is calculated, as shown in the following equation (8):

[0200]

[0201] Based on the original feature information and inter-domain enhancement index of the second sample data, the enhanced feature information of the second sample data in the target distribution direction is obtained. Specifically, the original feature information of the second sample data is... By adding the inter-domain enhancement index, the enhanced feature information of the second sample data in the target distribution direction can be obtained. CV refers to CrossVariance, as shown in equation (9) below:

[0202]

[0203] Optionally, based on the original feature information of the second sample data and the inter-domain enhancement index, the enhanced feature information of the second sample data can be obtained by adding the original feature information of the second sample data and the enhanced feature information generated along the distribution direction of the first domain space to which the first sample data belongs.

[0204] By applying the scheme of the embodiments of this specification, the covariance matrix of the first domain space is obtained by statistically analyzing the original feature information of the first sample data; the intra-domain correlation matrix of the first domain space is determined based on the covariance matrix; the inter-domain enhancement index is calculated based on the preset feature enhancement intensity and the intra-domain correlation matrix; and the enhanced feature information of the second sample data in the target distribution direction is obtained based on the original feature information of the second sample data and the inter-domain enhancement index. This ensures that the enhanced feature information of the second sample data in the target distribution direction is obtained through rigorous calculation using the original feature information of the first and second sample data, thus integrating the distribution direction of the domain space to which the first sample data belongs, and greatly improving the richness of sample features.

[0205] Optionally, the attribute type space includes a class space and a domain space; then, based on the original feature information of the first sample data, cross-space feature enhancement is performed on the second sample data, generating feature enhancement information. To obtain the total feature enhancement information, we can summarize the enhanced feature information from the attribute type space (including the class space) and the attribute type space (including the domain space). CSCV refers to CrossSmooth and CrossVariance, as shown in the following formula (10):

[0206]

[0207] When the attribute type space includes class space and domain space, the enhanced feature information of the generated second sample data is richer, which in turn increases the richness of the sample data. Furthermore, when the model is trained using the highly rich sample data, the accuracy of the model can be improved.

[0208] The scheme implemented in this specification involves acquiring a sample set comprising multiple sample data, extracting the original feature information of each sample data, and performing cross-spatial feature enhancement on the second sample data based on the original feature information of the first sample data to obtain enhanced feature information of the second sample data. The first and second sample data are any two sample data belonging to different attribute type spaces within the sample set. By extracting the original feature information of multiple sample data and performing cross-spatial feature enhancement on the second sample data based on the original feature information of the first sample data in different attribute type spaces, the enhanced feature information of the second sample data is generated based on the original feature information of the first sample data in different attribute type spaces. In other words, the enhanced feature information of the second sample data contains feature information from the first sample data in different attribute type spaces. This allows the original feature information of the second sample data to obtain enhanced feature information through cross-spatial feature enhancement, achieving cross-spatial feature enhancement of the sample data and improving the richness of the sample features.

[0209] See Figure 6 , Figure 6 This specification illustrates an overall flowchart of the training process for a classification model provided in one embodiment, as shown below. Figure 6 As shown.

[0210] The process involves acquiring the original image (second sample data) as the learner and the enhanced image (first sample data) as the teacher. A framework is constructed using implicit constraints arising from Fourier-based data augmentation and explicit constraints arising from co-teacher regularization. Based on the original feature information of the original and enhanced images, random direction sampling is performed on different classes within the same domain and on different domains within the same class, generating covariance matrices in the first domain, ..., and the Kth domain. The enhanced feature information of the sample data is obtained based on these multiple domain covariance matrices, where the sample data is the second sample data.

[0211] An experimental data recording table of a feature enhancement method for sample data provided according to an embodiment of this specification.

[0212] The enhanced feature information obtained from the feature enhancement method based on the above sample data is evaluated experimentally on four benchmark datasets: Digits-DG dataset, PACS dataset, Office-Home dataset, and DomainNet dataset.

[0213] Among them, the Digits-DG dataset is a digital dataset used for domain generalization;

[0214] Photo Art Cartoon Sketch (PACS) dataset: This is an image dataset for domain generalization, containing 4 domains: photographs (1670 images), artwork (2048 images), animation (2344 images), and sketches (3929 images), with 7 categories in each domain;

[0215] The Office-Home dataset is a benchmark dataset for domain adaptation. It contains 4 domains, each consisting of 65 categories, with an average of 70 images per category, for a total of 15,500 images.

[0216] The DomainNet dataset is a dataset consisting of common objects from six different domains, containing 345 categories and 600,000 images.

[0217] Table 1: Data statistics of experiments based on a single domain on the Digits-DG dataset

[0218] method MNIST MNIST-M SVHN SYN average value DeepAll 95.80％ 58.80％ 61.70％ 78.60％ 73.70％ JiGen 96.50％ 61.40％ 63.70％ 74.00％ 73.90％ CCSA 95.20％ 58.20％ 65.50％ 79.10％ 74.50％ MMD-AAE 96.50％ 58.40％ 65.00％ 78.40％ 74.60％ DDAIG 96.60％ 64.10％ 68.60％ 81.00％ 77.60％ L2A-OT 96.70％ 63.90％ 68.60％ 83.20％ 78.10％ FACT 97.90％ 65.60％ 72.40％ 90.30％ 81.50％ COMEN 97.10％ 67.60％ 75.10％ 91.30％ 82.30％ SADML 98.90％ 68.20％ 74.30％ 92.50％ 83.50％ Ours 99.10％ 70.70％ 76.40％ 94.10％ 85.10％

[0219] The methods listed in Table 1 include: DeepAll (a method that trains all source domain samples together), JiGen (a self-supervised training method based on image patch order prediction), CCSA (Classification and Contrastive Semantic Alignment), MMD-AAE (MMD-based Adversarial Autoencoder), DDAIG (Deep Domain-Adversarial Image Generation), L2A-OT (a deep domain adversarial image generation method for domain generalization), FACT, COMEN (Compound Domain Generalization via Meta-Knowledge Encoding), and SADML and Ours (feature enhancement schemes for sample data in the embodiments of this specification). The datasets used include: MNIST (Mixed National Institute of Standards and Technology database, a large-scale handwritten digit database), MNIST-M (composed of MNIST digits and BSDS500 (Berkeley Segmentation Data)). The datasets used for image segmentation and object edge detection (such as Set, which consists of random color patches), SVHN (Street-View House Number, a real-world image dataset), and SYN (Synthese) were tested on multiple datasets. The results and averages are shown in the table. Specifically, Table 1 shows the results of feature augmentation on the Digits-DG dataset, which improved the accuracy of the sample data by 85.1% - 83.5% = 1.6% compared to the SADML method. This clearly demonstrates that the generated augmented feature information can be effectively used for generalizable and transferable representations in domain generalization. In particular, when the source domain is significantly different from the unseen target domain, the feature augmentation method on the sample data improves the classification accuracy of the domain generalization task, for example, the results based on the SVHN dataset (from 74.3% to 76.4%).

[0220] Table 2: Statistical Table of Experiments Based on a Single Domain on the PACS Dataset

[0221]

[0222]

[0223] The methods listed in Table 2 include: DeepAll, MetaReg (Meta-Regularization), CrossGrad (Cross-Gradient), JiGen, DDAIG (Deep Domain-Adversarial Image Generation), L2A-OT, MixStyle (Mixed Style Domain Generalization), EISNet (Extrinsic and Intrinsic Supervisions Net), FACT, COMEN, and ATSRL (Adversarial Teacher-Student Representation). Learning, adversarial teacher-student representation learning methods, SADML, and Ours were used. The datasets used included Art (painting image dataset), Cartoon (cartoon image dataset), Photo (photo image dataset), and Sketch (sketch image dataset). Experiments were conducted on ResNet18 (accepting 224×224 image data as input) and ResNet50 (accepting 224×224 3-channel image data as input). Results and averages from experiments with multiple methods and datasets are shown in the table. Specifically, Table 2 shows the results of feature augmentation on sample data in the PACS dataset. Due to its similarity to the pre-trained ImageNet dataset, the FACT baseline achieves significant accuracy in the photo domain. However, there are significant differences between the photo domain and the painting, cartoon, and sketch domains. Even so, CDSA (inter-Class and inter-Domain Semantic Augmentation) still outperforms FACT in drawing, cartooning, and sketching by 2.86% (88.23%-85.37%), 4.99% (83.37%-78.38%), and 7.59% (86.74%-79.15%), respectively.

[0224] Table 3: Data Statistics of Experiments Based on a Domain on the Office-Home Dataset

[0225]

[0226]

[0227] The methods listed in Table 3 include: DeepAll, CCSA (Classification and Contrastive Semantic Alignment), MMD-AAE, CrossGrad, MixStyle, L2A-OT, FACT, SWAD (Stochastic Weight Averaging Densely), SADML, and Ours. The datasets used include: Art, Clipar (art pattern image dataset), Product (product image dataset), and Real (real image dataset). This demonstrates the results of feature augmentation on the Office-Home dataset, which can be used to add samples to more diverse datasets. Furthermore, the Office-Home dataset shows greater improvement than other datasets.

[0228] Table 4: Data Statistics of Experiments Based on a Domain on the DomainNet Dataset

[0229] method Clp Info Pat Qud Real Skt average value C-DANN 54.60％ 17.30％ 43.70％ 12.10％ 56.20％ 45.90％ 38.30％ RSC 55.00％ 18.30％ 44.40％ 12.20％ 55.70％ 47.80％ 38.90％ Mixup 55.70％ 18.50％ 44.30％ 12.50％ 55.80％ 48.20％ 39.20％ SagNet 57.70％ 19.00％ 45.30％ 12.70％ 58.10％ 48.80％ 40.30％ MLDG 59.10％ 19.10％ 45.80％ 13.40％ 59.60％ 50.20％ 41.20％ ERM 58.10％ 18.80％ 46.70％ 12.20％ 59.60％ 49.80％ 40.90％ MetaReg 59.77％ 25.58％ 50.19％ 11.52％ 64.56％ 50.09％ 43.62％ DMG 65.24％ 22.15％ 50.03％ 15.68％ 59.63％ 49.02％ 43.63％ SelfReg 62.40％ 22.60％ 51.80％ 14.30％ 62.50％ 53.80％ 44.60％ SADML 63.72％ 23.36％ 51.92％ 15.93％ 63.01％ 54.34％ 45.38％ Ours 64.28％ 25.51％ 53.04％ 16.66％ 64.28％ 55.87％ 46.61％

[0230] The methods listed in Table 4 include: C-DANN (Deep domain generalization via conditional invariant adversarial networks), RSC (Rep-resentation Self-Challenging), Mixup (Beyond empirical risk minimization), SagNet (Style-Agnostic Network), MLDG (Meta-Learning for Domain Generalization), ERM (Empirical Risk Minimization), MetaReg, DMG (Domain-specific Masks for Generalization), and SelfReg (Self-supervised Contrastive... Regularization (a self-supervised contrastive regularization method), SADML, and Ours were used. The datasets used included Clp (Clipart), Info (Inforgraph), Pat (Painting), Qud (Quickdraw), Real (Real), and Skt (Sketch). The results of feature enhancement on the DomainNet dataset are shown, improving SADML's accuracy by 64.28% - 63.72% = 1.23%, achieving an average maximum accuracy of 46.61%. This allows for stable enhancement of the classifier's generalization ability across different spatial datasets in unknown spaces.

[0231] Table 5: Statistical data of ablation studies on different components of the feature enhancement schemes in the sample data of the embodiments in this specification on the PACS dataset (RESNET-50).

[0232] method Cartoon Sketch Photo Art average value CDSA 89.12％ 92.06％ 99.20％ 93.38％ 93.44％ CDSA (with or without CL) 86.38％ 87.64％ 99.17％ 91.84％ 91.26％ CDSA (without CV) 85.29％ 87.37％ 98.90％ 92.24％ 90.95％ CDSA (with / o CS) 87.38％ 88.21％ 99.23％ 92.03％ 91.71％ CDSA (w / o CS+CL) 84.06％ 85.21％ 97.11％ 90.79％ 89.30％ CDSA (without CV+CL) 83.31％ 84.67％ 97.92％ 91.25％ 89.29％ CDSA (without CS+CV+CL) 81.77％ 84.46％ 96.75％ 89.63％ 88.15％

[0233] The methods listed in Table 5 are: CDSA, CDSA(w / o CL), CDSA(w / o CV), CDSA(w / o CS), CDSA(w / o CS+CL), CDSA(w / o CV+CL), and CDSA(w / o CS+CV+CL), where w / o indicates without. The datasets used are: Cartoon, Sketch, Photo, and Art, where CS, CV, and CL represent cross-smoothing, cross-variance, and contrastive learning losses, respectively. To investigate the relative contribution of each component in the proposed methods, extensive ablation analysis was performed on the task PACS. The results are reported in Table 5. Specifically, all possible combinations of the three proposed components were evaluated: CrossSmooth (CS), CrossVariance (CV), and Contrastive Learning (CL). It was determined that all possible combinations are reasonably designed, and the performance degrades when any one of these components is removed, provided that any one of them is reasonably designed. Removing the CS and CV modules resulted in a significant performance drop, validating the importance of improving domain generalization diversity. In this case, the results for the CS and CV modules were observed to be robust.

[0234] See Figure 7 , Figure 7 This document shows a histogram of A-distances for features learned on different major generalization tasks during the training process of a classification model provided in one embodiment of this specification.

[0235] Figure 7 In the examples of From PAS to A and From PAC to S, the A-distances of IRM (Invariant Risk Minimization, a new method for cross-domain image classification), FACT, SADML, and CDSA are presented. Specific numerical data for the A-distances are shown in the figures. "From PAS to A" refers to the A-distance from PAS (Photo Art painting Sketch) to A (Art painting), representing the A-distance from PAS to A under the IRM, FACT, SADML, and CDSA methods, respectively. "From PAC to S" refers to the A-distance from PAC (Photo Art painting Cartoon) to S (Sketch), representing the A-distance from PAC to S under the IRM, FACT, SADML, and CDSA methods, respectively.

[0236] Alternatively, the A-distance can be used to measure domain divergence. The A-distance is a widely used measure of distribution divergence. The A-distance is defined as A- = 2(1-∈), where ∈ represents the test error source domain and target domain of the classifier trained to distinguish between them. The smaller the A-distance, the better the distribution alignment. Figure 7 This demonstrates that our method can learn more invariant features to minimize the difference between the source and target domains.

[0237] See Figure 8 , Figure 8 This specification illustrates a flowchart of a remote sensing image classification method according to an embodiment, applied to an edge device, specifically including the following steps:

[0238] Step 802: Obtain the remote sensing image to be classified selected by the user on the front end.

[0239] In one or more embodiments of this specification, the end-side device can acquire the remote sensing image to be classified selected by the user at the front end.

[0240] Specifically, remote sensing images, also known as remote sensing photographs, refer to images that record the magnitude of electromagnetic waves emitted by various ground features. Remote sensing images include, but are not limited to, optical, radar satellite images, and historical remote sensing images, etc., selected according to the actual situation. This specification does not impose any limitations on these selections in the embodiments. Remote sensing images awaiting classification refer to remote sensing images that are waiting to be classified, such as radar satellite images awaiting classification, etc.

[0241] It should be noted that there are many ways for the edge device to acquire the remote sensing image to be classified selected by the user at the front end. The specific method should be selected according to the actual situation. This specification does not limit the specific method used in this embodiment.

[0242] In one possible implementation of this specification, the user may have pre-uploaded or stored multiple remote sensing images on the terminal device, and then the user may directly select these remote sensing images on the front end for classification.

[0243] In another possible implementation of this specification, the user may randomly select a remote sensing image while browsing the front-end webpage, and the end-side device may classify it.

[0244] Step 804: Extract the image feature information of the remote sensing image to be classified.

[0245] In one or more embodiments of this specification, after the end-side device acquires the remote sensing image to be classified selected by the user at the front end, it further extracts the image feature information of the remote sensing image to be classified.

[0246] Step 806: Input the image feature information into the classification model to obtain the category information of the remote sensing image to be classified.

[0247] The classification model is trained using multiple sample data, which includes original feature information and enhanced feature information. The enhanced feature information is obtained by cross-space feature enhancement of sample data belonging to different attribute type spaces.

[0248] In one or more embodiments of this specification, after the end-side device acquires the remote sensing image to be classified selected by the user at the front end, it extracts the image feature information of the remote sensing image to be classified, and then inputs the image feature information into the classification model to obtain the category information of the remote sensing image to be classified.

[0249] The specific implementation methods of steps 802, 804, and 806 are the same as those described above. Figure 3 The image classification methods shown are implemented in a similar manner, and will not be described in detail here.

[0250] Step 808: Feed back the category information to the front end display.

[0251] In one or more embodiments of this specification, after the end-side device acquires the remote sensing image to be classified selected by the user at the front end, it extracts the image feature information of the remote sensing image to be classified, inputs the image feature information into the classification model, obtains the category information of the remote sensing image to be classified, and then feeds the category information back to the front end for display to the user.

[0252] It should be noted that the method of feeding back the category information to the front end for display can be to display only the category information or to display the remote sensing image to be classified at the same time. The specific choice depends on the actual situation, and this specification does not limit this in any way.

[0253] In one optional embodiment of this specification, after step 808 above, the following specific steps are further included:

[0254] Receive the classification and evaluation information provided by the user;

[0255] Based on the classification evaluation information, the model parameters of the classification model are adjusted.

[0256] In one or more embodiments of this specification, after the end-side device feeds back the category information to the front end for display to the user, it can also receive classification evaluation information from the user based on the displayed category information, and adjust the model parameters of the classification model according to the classification evaluation information.

[0257] It should be noted that there are many types of user feedback ratings. One could be a "Rate" button on the device's category information screen. Clicking this button would display a 10-digit number from 0 to 9, with 0 being the lowest rating and 9 the highest. Users would then click the button corresponding to that 10 to rate the category. Another option is an input box below the category information displayed on the front end, showing "Please rate." Users could enter their rating in the input box as their category feedback.

[0258] See Figure 9 , Figure 9 This specification illustrates a flowchart of a data processing method for a classification model provided in one embodiment, applied to a cloud-side device, and specifically includes the following steps:

[0259] Step 902: Obtain the sample set.

[0260] The sample set includes multiple sample data, and the sample data carries category labels.

[0261] Step 904: Obtain the feature information of each sample data.

[0262] The feature information includes original feature information and enhanced feature information. The enhanced feature information is obtained by cross-space feature enhancement of sample data belonging to different attribute type spaces.

[0263] Step 906: Based on the feature information and category labels of each sample data, train the preset classification network to obtain the model parameters of the trained classification model.

[0264] Step 908: Send the model parameters of the classification model to the end device.

[0265] In practical applications, there are various ways for cloud-side devices to acquire sample sets, and the specific method should be selected according to the actual situation. This specification does not limit the specific methods used in this embodiment.

[0266] In one possible implementation of this specification, the cloud-side device can obtain a sample set of Chinese image-text pairs including multiple different scenarios from a cloud database.

[0267] In another possible implementation of this specification, the cloud testing device can receive a sample set constructed and uploaded by the end-side device.

[0268] It should be noted that the specific implementation methods of steps 902, 904, and 906 are the same as those described above. Figure 3 The training methods for the image and text recognition models in the illustrated image and text recognition methods are the same, so this manual will not repeat them further.

[0269] Furthermore, after the cloud-side device trains and obtains the model parameters of the classification model, it can send the model parameters of the classification model to multiple end-side devices that have established connections with the cloud-side device, so that the end-side devices can use the classification model to perform image classification.

[0270] According to the scheme of the embodiments in this specification, the cloud-side device acquires a sample set, wherein the sample set includes multiple sample data, and the sample data carries category labels; it obtains feature information of each sample data, wherein the feature information includes original feature information and enhanced feature information, and the enhanced feature information is obtained by cross-space feature enhancement of sample data belonging to different attribute type spaces; based on the feature information and category labels of each sample data, a preset classification network is trained to obtain the model parameters of the trained classification model; the model parameters of the classification model are sent to the edge device. By using sample data from different attribute type spaces to perform cross-space feature enhancement on the original feature information of the sample data to obtain the enhanced feature information corresponding to the sample data, the enhanced feature information of multiple sample data is obtained accordingly, and the model is trained based on the original feature information and enhanced feature information of multiple sample data to obtain a classification model. This makes the classification model trained based on highly rich sample features, improving the classification efficiency and accuracy of the classification model.

[0271] The following is in conjunction with the appendix Figure 10 Taking the image classification method provided in this specification as an example in the application of remote sensing image classification, the image classification method will be further explained. Among other things, Figure 10 The diagram illustrates the interactive flow of an image classification method according to an embodiment of this specification, specifically including the following steps:

[0272] The edge device acquires multiple sample data belonging to different domains, constructs a sample set based on these data, and sends the sample set to the cloud device. The sample data carries category labels, and the multiple sample data belonging to different domains include sample data from the painting domain, the realistic painting domain, and the sketching domain.

[0273] After acquiring the sample set, the cloud-side device obtains the feature information of each sample data. Based on the feature information and the category label carried by each sample data, it trains a preset classification network to obtain the model parameters of the trained classification model, and sends the model parameters of the classification model to the edge device. The feature information includes original feature information and enhanced feature information.

[0274] The edge device acquires the image to be classified selected by the user at the front end, extracts the image feature information of the image to be classified, inputs the image feature information into the classification model, obtains the category information of the image to be classified, and displays the category information to the user through the front end.

[0275] By applying the scheme of the embodiments of this specification, sample data from different attribute type spaces are used to perform cross-space feature enhancement on the original feature information of the sample data to obtain the enhanced feature information corresponding to the sample data. Thus, enhanced feature information of multiple sample data is obtained, and the model is trained based on the original feature information and enhanced feature information of multiple sample data to obtain a classification model. This makes the classification model trained based on highly rich sample features, which greatly improves the classification efficiency and accuracy of the classification model.

[0276] Corresponding to the above method embodiments, this specification also provides embodiments of image classification devices. Figure 11 A schematic diagram of an image classification device according to one embodiment of this specification is shown. Figure 11 As shown, the device includes:

[0277] The first acquisition module 1102 is configured to acquire the image to be classified.

[0278] The first extraction module 1104 is configured to extract image feature information of the image to be classified.

[0279] The first input module 1106 is configured to input image feature information into a classification model to obtain the category information of the image to be classified. The classification model is trained using multiple sample data, which includes original feature information and enhanced feature information. The enhanced feature information is obtained by performing cross-space feature enhancement on sample data belonging to different attribute type spaces.

[0280] Optionally, the image classification device further includes a classification model training module, configured to acquire a sample set, wherein the sample set includes multiple sample data, and the sample data carries category labels; acquire feature information of each sample data, wherein the feature information includes original feature information and enhanced feature information, the enhanced feature information being obtained by cross-space feature enhancement of sample data belonging to different attribute type spaces; and train a preset classification network based on the feature information and category labels of each sample data to obtain a trained classification model.

[0281] Optionally, the classification model training module is further configured to extract feature information and target category labels from the target sample data, wherein the target sample data is any one of multiple sample data; input the feature information of the target sample data into a preset classification network to obtain the classification result of the target sample data; calculate the total loss value based on the feature information, classification result, and target category label of the target sample data; adjust the network parameters of the preset classification network based on the total loss value, and return to the steps of extracting feature information and target category labels from the target sample data; and determine that the training of the preset classification network is complete and a classification model is obtained when the training stopping condition is met.

[0282] Optionally, the classification model training module is further configured to extract the original feature information of each sample data; and to perform cross-space feature enhancement on the second sample data based on the original feature information of the first sample data to obtain the enhanced feature information of the second sample data, wherein the first sample data and the second sample data are any two sample data belonging to different attribute type spaces in the sample set.

[0283] Optionally, the attribute type space includes a class space, the first sample data belongs to the first class space, and the second sample data belongs to the second class space; the classification model training module is further configured to perform cross-class space feature enhancement on the second sample data in the target semantic direction based on the original feature information of the first sample data, so as to obtain the enhanced feature information of the second sample data in the target semantic direction, wherein the target semantic direction is any specific semantic direction contained in the first class space.

[0284] Optionally, the classification model training module is further configured to determine the sampling probability of the first sample data in the target semantic direction based on the original feature information of the first sample data; calculate the inter-class enhancement index based on the original feature information and sampling probability of the second sample data; and obtain the enhanced feature information of the second sample data in the target semantic direction based on the original feature information and inter-class enhancement index of the second sample data.

[0285] Optionally, the classification model training module is further configured to perform statistical analysis on the original feature information of the second sample data to obtain the class conditional covariance matrix of the second class space; construct the inter-class sampling matrix based on the sampling probability and the preset diagonal matrix; determine the correlation matrix between the first class space and the second class space based on the class conditional covariance matrix and the inter-class sampling matrix; and calculate the inter-class enhancement index based on the preset feature enhancement intensity and the correlation matrix.

[0286] Optionally, the attribute type space includes a domain space, the first sample data belongs to the first domain space, and the second sample data belongs to the second domain space; the classification model training module is further configured to perform cross-domain space feature enhancement on the second sample data in the target distribution direction based on the original feature information of the first sample data, and obtain the enhanced feature information of the second sample data in the target distribution direction, wherein the target distribution direction is the distribution direction of the first domain space.

[0287] Optionally, the classification model training module is further configured to perform statistical analysis on the original feature information of the first sample data to obtain the covariance matrix of the first domain space; determine the intra-domain correlation matrix of the first domain space based on the covariance matrix; calculate the inter-domain enhancement index based on the preset feature enhancement intensity and the intra-domain correlation matrix; and obtain the enhanced feature information of the second sample data in the target distribution direction based on the original feature information of the second sample data and the inter-domain enhancement index.

[0288] The above is a schematic scheme of an image classification device according to this embodiment. It should be noted that the technical solution of this image classification device and the technical solution of the image classification method described above belong to the same concept. For details not described in detail in the technical solution of the image classification device, please refer to the description of the technical solution of the image classification method described above.

[0289] Corresponding to the above method embodiments, this specification also provides embodiments of remote sensing image classification devices. Figure 12 A schematic diagram of the structure of a remote sensing image classification device according to one embodiment of this specification is shown. Figure 12 As shown, the device includes:

[0290] The second acquisition module 1202 is configured to acquire the remote sensing image to be classified selected by the user at the front end;

[0291] The second extraction module 1204 is configured to extract image feature information from the remote sensing image to be classified.

[0292] The second input module 1206 is configured to input image feature information into a classification model to obtain category information of the remote sensing image to be classified. The classification model is trained using multiple sample data, which includes original feature information and enhanced feature information. The enhanced feature information is obtained by performing cross-spatial feature enhancement on sample data belonging to different attribute type spaces.

[0293] Feedback module 1208 is configured to feed back category information to the front end for display.

[0294] Optionally, an adjustment module is included after the feedback module 1208, configured to receive classification evaluation information from user feedback; and adjust the model parameters of the classification model based on the classification evaluation information.

[0295] The above is a schematic scheme of a remote sensing image classification device according to this embodiment. It should be noted that the technical solution of this remote sensing image classification device and the technical solution of the remote sensing image classification method described above belong to the same concept. For details not described in detail in the technical solution of the remote sensing image classification device, please refer to the description of the technical solution of the remote sensing image classification method described above.

[0296] Corresponding to the above method embodiments, this specification also provides embodiments of a data processing apparatus for a classification model. Figure 13 A schematic diagram of the structure of a data processing apparatus for a classification model provided in one embodiment of this specification is shown. Figure 13 As shown, the device includes:

[0297] The third acquisition module 1302 is configured to acquire a sample set, wherein the sample set includes multiple sample data, and the sample data carries category labels;

[0298] The module 1304 is configured to obtain feature information of each sample data, wherein the feature information includes original feature information and enhanced feature information. The enhanced feature information is obtained by performing cross-space feature enhancement on sample data belonging to different attribute type spaces.

[0299] Training module 1306 is configured to train a preset classification network based on the feature information and category labels of each sample data to obtain the model parameters of the trained classification model.

[0300] The sending module 1308 is configured to send the model parameters of the classification model to the end device.

[0301] The above is a schematic scheme of a data processing device for a classification model according to this embodiment. It should be noted that the technical solution of the data processing device for the classification model and the technical solution of the data processing method for the classification model described above belong to the same concept. For details not described in detail in the technical solution of the data processing device for the classification model, please refer to the description of the technical solution of the data processing method for the classification model described above.

[0302] Figure 14 A structural block diagram of a computing device according to one embodiment of this specification is shown. The components of the computing device 1400 include, but are not limited to, a memory 1410 and a processor 1420. The processor 1420 is connected to the memory 1410 via a bus 1430, and a database 1450 is used to store data.

[0303] The computing device 1400 also includes an access device 1440, which enables the computing device 1400 to communicate via one or more networks 1460. Examples of these networks include Public Switched Telephone Network (PSTN), Local Area Network (LAN), Wide Area Network (WAN), Personal Area Network (PAN), or combinations of communication networks such as the Internet. The access device 1440 may include one or more of any type of wired or wireless network interface (e.g., Network Interface Card (NIC)), such as an IEEE 802.11 Wireless Local Area Networks (WLAN) interface, a Wi-MAX (World Interoperability for Microwave Access) interface, an Ethernet interface, a Universal Serial Bus (USB) interface, a cellular network interface, a Bluetooth interface, a Near Field Communication (NFC) interface, and so on.

[0304] In one embodiment of this specification, the above-described components of the computing device 1400 and Figure 14 Other components, not shown, can also be connected to each other, for example, via a bus. It should be understood that... Figure 14 The block diagram of the computing device shown is for illustrative purposes only and is not intended to limit the scope of this specification. Those skilled in the art can add or replace other components as needed.

[0305] The computing device 1400 can be any type of stationary or mobile computing device, including mobile computers or mobile computing devices (e.g., tablet computers, personal digital assistants, laptop computers, notebook computers, netbooks, etc.), mobile phones (e.g., smartphones), wearable computing devices (e.g., smartwatches, smart glasses, etc.) or other types of mobile devices, or stationary computing devices such as desktop computers or personal computers (PCs). The computing device 1400 can also be a mobile or stationary server.

[0306] The processor 1420 is configured to execute the following computer-executable instructions, which, when executed by the processor, implement the steps of the above-described image classification method, the above-described remote sensing image classification method, or the above-described classification model data processing method.

[0307] The above is a schematic representation of a computing device according to this embodiment. It should be noted that the technical solution of this computing device belongs to the same concept as the technical solution of the image classification method, the remote sensing image classification method, or the data processing method of the classification model described above. Details not described in detail in the technical solution of the computing device can be found in the description of the technical solution of the image classification method, the remote sensing image classification method, or the data processing method of the classification model described above.

[0308] An embodiment of this specification also provides a computer-readable storage medium storing computer-executable instructions that, when executed by a processor, implement the steps of the above-described image classification method, the above-described remote sensing image classification method, or the above-described classification model data processing method.

[0309] The above is an illustrative scheme of a computer-readable storage medium according to this embodiment. It should be noted that the technical solution of this storage medium belongs to the same concept as the technical solution of the image classification method, the remote sensing image classification method, or the data processing method of the classification model described above. Details not described in detail in the technical solution of the storage medium can be found in the descriptions of the technical solutions of the image classification method, the remote sensing image classification method, or the data processing method of the classification model described above.

[0310] An embodiment of this specification also provides a computer program, wherein when the computer program is executed in a computer, it causes the computer to perform the steps of the above-described image classification method, the above-described remote sensing image classification method, or the above-described classification model data processing method.

[0311] The above is an illustrative scheme of a computer program according to this embodiment. It should be noted that the technical solution of this computer program belongs to the same concept as the technical solution of the image classification method, the remote sensing image classification method, or the data processing method of the classification model described above. For details not described in detail in the technical solution of the computer program, please refer to the description of the technical solution of the image classification method, the remote sensing image classification method, or the data processing method of the classification model described above.

[0312] The foregoing has described specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired result. In some embodiments, multitasking and parallel processing are possible or may be advantageous.

[0313] The computer instructions include computer program code, which may be in the form of source code, object code, executable file, or some intermediate form. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording media, USB flash drive, portable hard drive, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc.

[0314] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that the embodiments in this specification are not limited to the described order of actions, because according to the embodiments in this specification, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in this specification are all preferred embodiments, and the actions and modules involved are not necessarily essential to the embodiments in this specification.

[0315] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0316] The preferred embodiments disclosed above are merely illustrative of this specification. The optional embodiments do not exhaustively describe all details, nor do they limit the invention to the specific implementations described. Clearly, many modifications and variations can be made based on the embodiments described herein. These embodiments are selected and specifically described in this specification to better explain the principles and practical applications of the embodiments, thereby enabling those skilled in the art to better understand and utilize this specification. This specification is limited only by the claims and their full scope and equivalents.

Claims

1. An image classification method, applied to an edge device, comprising: Obtain the image to be classified; Extract the image feature information of the image to be classified; The image feature information is input into a classification model to obtain the category information of the image to be classified. The classification model is trained using multiple sample data. The sample data includes original feature information and enhanced feature information. The enhanced feature information is obtained by cross-space feature enhancement of sample data belonging to different attribute type spaces. The attribute type spaces include class space and domain space.

2. The method according to claim 1, further comprising: Obtain a sample set, wherein the sample set includes multiple sample data, and the sample data carries category labels; Obtain feature information for each of the sample data, wherein the feature information includes original feature information and enhanced feature information, and the enhanced feature information is obtained by performing cross-space feature enhancement on sample data belonging to different attribute type spaces; Based on the feature information and category labels of each sample data, a preset classification network is trained to obtain a trained classification model.

3. The method according to claim 2, wherein training a preset classification network based on the feature information and category labels of each of the sample data to obtain a trained classification model includes: Extract feature information and target category label from target sample data, wherein the target sample data is any one of the plurality of sample data; The feature information of the target sample data is input into a preset classification network to obtain the classification result of the target sample data; The total loss value is calculated based on the feature information of the target sample data, the classification result, and the target category label; The network parameters of the preset classification network are adjusted based on the total loss value, and the process of extracting feature information and target category labels from the target sample data is returned. If the training stopping condition is met, the training of the preset classification network is completed, and a classification model is obtained.

4. The method according to claim 2, wherein obtaining the feature information of each of the sample data includes: Extract the original feature information of each sample data; Based on the original feature information of the first sample data, cross-spatial feature enhancement is performed on the second sample data to obtain the enhanced feature information of the second sample data, wherein the first sample data and the second sample data are any two sample data belonging to different attribute type spaces in the sample set.

5. The method according to claim 4, wherein the first sample data belongs to a first type of space, and the second sample data belongs to a second type of space; The step of performing cross-spatial feature enhancement on the second sample data based on the original feature information of the first sample data to obtain the enhanced feature information of the second sample data includes: Based on the original feature information of the first sample data, cross-class spatial feature enhancement is performed on the second sample data in the target semantic direction to obtain the enhanced feature information of the second sample data in the target semantic direction, wherein the target semantic direction is any specific semantic direction contained in the first class space.

6. The method according to claim 5, wherein the step of performing cross-class spatial feature enhancement on the second sample data in the target semantic direction based on the original feature information of the first sample data to obtain the enhanced feature information of the second sample data in the target semantic direction includes: Based on the original feature information of the first sample data, determine the sampling probability of the first sample data in the target semantic direction; Based on the original feature information of the second sample data and the sampling probability, calculate the inter-class enhancement index; Based on the original feature information of the second sample data and the inter-class enhancement index, the enhanced feature information of the second sample data in the target semantic direction is obtained.

7. The method according to claim 6, wherein calculating the inter-class enhancement index based on the original feature information of the second sample data and the sampling probability comprises: The original feature information of the second sample data is statistically analyzed to obtain the class conditional covariance matrix of the second class space; Based on the sampling probability and the preset diagonal matrix, construct the inter-class sampling matrix; Based on the class-conditional covariance matrix and the inter-class sampling matrix, determine the correlation matrix between the first class space and the second class space; Based on the preset feature enhancement intensity and the correlation matrix, the inter-class enhancement index is calculated.

8. The method according to claim 4, wherein the first sample data belongs to a first domain space, and the second sample data belongs to a second domain space; The step of performing cross-spatial feature enhancement on the second sample data based on the original feature information of the first sample data to obtain the enhanced feature information of the second sample data includes: Based on the original feature information of the first sample data, cross-domain spatial feature enhancement is performed on the second sample data in the target distribution direction to obtain the enhanced feature information of the second sample data in the target distribution direction, wherein the target distribution direction is the distribution direction of the first domain space.

9. The method according to claim 8, wherein the step of performing cross-domain spatial feature enhancement on the second sample data in the target distribution direction based on the original feature information of the first sample data to obtain enhanced feature information of the second sample data in the target distribution direction comprises: The original feature information of the first sample data is statistically analyzed to obtain the covariance matrix of the first domain space; Based on the covariance matrix, determine the intra-domain correlation matrix of the first domain space; Based on the preset feature enhancement strength and the intra-domain correlation matrix, the inter-domain enhancement index is calculated; Based on the original feature information of the second sample data and the inter-domain enhancement index, the enhanced feature information of the second sample data in the target distribution direction is obtained.

10. A remote sensing image classification method, applied to an edge device, comprising: Acquire the remote sensing image to be classified selected by the user on the front end; Extract image feature information from the remote sensing image to be classified; The image feature information is input into a classification model to obtain the category information of the remote sensing image to be classified. The classification model is trained using multiple sample data. The sample data includes original feature information and enhanced feature information. The enhanced feature information is obtained by cross-space feature enhancement of sample data belonging to different attribute type spaces. The attribute type spaces include class space and domain space. The category information is fed back to the front end for display.

11. The method according to claim 10, further comprising, after feeding back the category information to the front-end display: Receive the classification and evaluation information provided by the user; Based on the classification evaluation information, the model parameters of the classification model are adjusted.

12. A data processing method for a classification model, applied to cloud-side devices, comprising: Obtain a sample set, wherein the sample set includes multiple sample data, and the sample data carries category labels; Obtain feature information for each of the sample data, wherein the feature information includes original feature information and enhanced feature information, and the enhanced feature information is obtained by cross-space feature enhancement of sample data belonging to different attribute type spaces, wherein the attribute type spaces include class space and domain space; Based on the feature information and category labels of each sample data, a preset classification network is trained to obtain the model parameters of the trained classification model. The model parameters of the classification model are sent to the edge device.

13. A computing device, comprising: Memory and processor; The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions. When the computer-executable instructions are executed by the processor, they implement the steps of the image classification method according to any one of claims 1 to 9, the remote sensing image classification method according to claims 10 to 11, or the data processing method of the classification model according to claim 12.

14. A computer-readable storage medium storing computer-executable instructions that, when executed by a processor, implement the steps of the image classification method of any one of claims 1 to 9, the remote sensing image classification method of claims 10 to 11, or the data processing method of the classification model of claim 12.

15. A computer program product comprising computer instructions that, when executed by a processor, implement the steps of the image classification method of any one of claims 1 to 9, the remote sensing image classification method of claims 10 to 11, or the data processing method of the classification model of claim 12.

Images