Data processing method, device and non-volatile storage medium

Abstract

The application discloses a data processing method and device and a nonvolatile storage medium. The method comprises the following steps: acquiring at least a training sample set and a test sample set; extracting target features from the training sample set and the test sample set respectively, wherein the target features are used to determine the features of training samples in the training sample set and the features of test samples in the test sample set; performing fuzzy-mean clustering processing on the training sample set and the test sample set based on the extracted target features, to obtain a membership matrix; and determining weight data based on the membership matrix, wherein the sample weight data is used to adjust the weight of a loss function of an initial model, to obtain a target model, and the target model is used for data calibration. The application solves the technical problem of unbalanced training samples in a classifier model.

Description

Technical Field

[0001] This invention relates to the field of computer technology, and more specifically, to data processing methods, apparatus, and non-volatile storage media. Background Technology

[0002] In the era of intelligent big data, the increasing volume and types of data have led to a growing demand for rapidly building machine learning models with strong generalization capabilities. Theoretically and practically, supervised learning models trained on a small number of labeled samples cannot achieve a good balance between empirical risk and expected risk, especially when training and test data do not adhere to the independent and identically distributed assumption. Obtaining a large number of labeled samples often requires enormous human, material, and financial resources. These high costs in time, money, and effort force researchers to explore other forms of machine learning modeling when labeled data is insufficient. Reusing existing labeled data and models has become possible; therefore, how to quickly build models based on existing data with different distributions has become a crucial issue.

[0003] Transfer learning networks consist of a feature extractor, a class predictor, and a domain classifier. Feature extraction extracts the features needed for subsequent network tasks and maps and mixes source and target domain samples. The class predictor classifies the extracted information, and the domain classifier determines whether the extracted information comes from the source or target domain. The inability to correctly identify the domain creates an adversarial relationship. Typically, the source domain data has a relatively large amount of data and labels, while the target domain data is less abundant. When the data is severely imbalanced, the machine learning model's predictions often favor the side with more data. Therefore, classifier models suffer from the technical problem of imbalanced training samples.

[0004] There is currently no effective solution to the above problems. Summary of the Invention

[0005] The present invention provides a data processing method, apparatus, and non-volatile storage medium to at least solve the technical problem of imbalanced training samples in classifier models.

[0006] According to one aspect of the present invention, a data processing method is provided. The method may include: acquiring at least a training sample set and a test sample set; extracting target features from the training sample set and the test sample set respectively, wherein the target features are used to determine the features of training samples in the training sample set and the features of test samples in the test sample set; performing fuzzy-means clustering on the training sample set and the test sample set based on the extracted target features to obtain a membership matrix; and determining weight data based on the membership matrix, wherein the sample weight data is used to adjust the weights of the loss function of the initial model to obtain a target model, and the target model is used for data calibration.

[0007] Optionally, extracting target features from the training sample set and the test sample set respectively may include: performing median padding on the training sample set and the test sample set respectively to obtain the padded dataset; replacing the data in the padded dataset that are not within the threshold range, and normalizing the dataset obtained after replacement to obtain the target features.

[0008] Optionally, determining weight data based on the membership matrix may include: determining, based on the membership matrix, the first average membership degree and the first class center of training samples whose membership degree is less than a first membership degree threshold, and determining the second average membership degree and the second class center of training samples whose membership degree is greater than a second membership degree threshold; determining the first distance between the test sample and the training sample based on the first average membership degree, the first class center, the second average membership degree, and the second class center; and determining the weight data of the test sample based on the first distance.

[0009] Optionally, determining the weight data of the test sample based on distance may include: determining a second distance between the test sample and the center point of the training sample; and determining the ratio between the second distance and the first distance as the weight data.

[0010] Optionally, the above method may also include: modifying the parameters of the loss function based on the weight data to obtain the modified loss function.

[0011] Optionally, modifying the parameters of the loss function based on the weight data to obtain the modified loss function may include: modifying the weight coefficients of the domain classifier loss function in the loss function to the weight data; determining the product of the domain classifier loss function and the weight data, and determining the difference between the category predictor loss function and the product in the loss function as the modified loss function.

[0012] According to another aspect of the present invention, a data processing apparatus is also provided. The apparatus may include: an acquisition unit, configured to acquire at least a training sample set and a test sample set; an extraction unit, configured to extract target features from the training sample set and the test sample set respectively, wherein the target features are used to determine the features of training samples in the training sample set and the features of test samples in the test sample set; a processing unit, configured to perform fuzzy-means clustering processing on the training sample set and the test sample set based on the extracted target features to obtain a membership matrix; and a determination unit, configured to determine weight data based on the membership matrix, wherein the sample weight data is used to adjust the weights of the loss function of the initial model to obtain a target model, and the target model is used for data calibration.

[0013] According to another aspect of the present invention, a non-volatile storage medium is also provided, wherein the non-volatile storage medium stores a plurality of instructions, the instructions being adapted to be loaded by a processor and executed any one of the above-described data processing methods.

[0014] According to another aspect of the present invention, an electronic device is also provided, including one or more processors and a memory, wherein the memory is used to store one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors cause the one or more processors to implement any of the above-described data processing methods.

[0015] According to another aspect of the present invention, a computer program product is also provided, including a computer program that, when executed by a processor, implements any of the above-described data processing methods.

[0016] In this embodiment of the invention, a training sample set and a test sample set are first obtained. Target features are extracted from both sets, whereby the target features are used to determine the features of the training samples in the training sample set and the features of the test samples in the test sample set. Based on the extracted target features, fuzzy-means clustering is performed on both the training and test sample sets to obtain a membership matrix. Weight data is determined based on the membership matrix, whereby the sample weight data is used to adjust the weights of the loss function of the initial model to obtain the target model, which is then used for data labeling. This method increases the amount of sample data with different labels in the training sample set, effectively solving the technical problem of imbalanced training samples in classifier models and achieving a balanced distribution of training samples in classification models. Attached Figure Description

[0017] The accompanying drawings, which are included to provide a further understanding of the invention and form part of this application, illustrate exemplary embodiments of the invention and, together with their description, serve to explain the invention and do not constitute an undue limitation thereof. In the drawings:

[0018] Figure 1 This is a flowchart of a data processing method according to an embodiment of the present invention;

[0019] Figure 2 This is a flowchart of an adaptive transfer learning method according to an embodiment of the present invention;

[0020] Figure 3 This is a schematic diagram of the network structure of a transfer adaptive learning model according to an embodiment of the present invention;

[0021] Figure 4 This is a schematic diagram of a data processing apparatus according to an embodiment of the present invention. Detailed Implementation

[0022] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0023] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0024] Example 1

[0025] According to an embodiment of the present invention, a data processing method is provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0026] Figure 1 This is a flowchart of a data processing method according to an embodiment of the present invention, such as... Figure 1 As shown, the method includes the following steps:

[0027] Step S101: Obtain at least the training sample set and the test sample set.

[0028] In the technical solution provided by step S101 of the present invention, a training sample set and a test sample set are obtained. The training sample set and the test sample set can be image samples, text samples or traffic samples, and no specific limitation is made here.

[0029] Step S102: Extract target features from the training sample set and the test sample set respectively. The target features are used to determine the features of the training samples in the training sample set and the features of the test samples in the test sample set.

[0030] In the technical solution provided by step S102 of the present invention, the target features of the training sample set and the test sample set are used to obtain the target features, wherein the target features can reflect the specific attributes of the samples, and the target features can be continuous or discrete.

[0031] Step S103: Based on the extracted target features, perform fuzzy-means clustering on the training sample set and the test sample set respectively to obtain the membership matrix.

[0032] In the technical solution provided by step S103 of the present invention, the target features obtained in the training sample set and the test sample set are subjected to fuzzy-mean clustering to obtain a membership matrix, wherein the membership matrix is ​​used to calculate the similarity between samples.

[0033] Step S104: Determine the weight data based on the membership matrix. The sample weight data is used to adjust the weights of the loss function of the initial model to obtain the target model. The target model is used for data calibration.

[0034] In the technical solution provided by step S104 of the present invention, after the membership matrix determines the similarity between samples, the model parameters can be adjusted to obtain the target model and used for data calibration.

[0035] In steps S101 to S104 of this application, a training sample set and a test sample set are obtained; target features are extracted from the training sample set and the test sample set respectively, wherein the target features are used to determine the features of the training samples in the training sample set and the features of the test samples in the test sample set; based on the extracted target features, fuzzy-means clustering is performed on the training sample set and the test sample set respectively to obtain a membership matrix; weight data is determined based on the membership matrix, wherein the sample weight data is used to adjust the weights of the loss function of the initial model to obtain the target model, and the target model is used for data labeling. Through the data processing method of the above steps, labels can be assigned to samples that are unlabeled in the training samples, so as to increase the number of labeled samples in the training sample set, thereby solving the technical problem of imbalanced training samples in the classifier model and effectively achieving a balanced training sample effect.

[0036] The method described in this embodiment will be further described below.

[0037] As an optional embodiment, step S102, extracting target features from the training sample set and the test sample set respectively, may include: performing median padding on the training sample set and the test sample set respectively to obtain a padded dataset; replacing the data in the padded dataset that are not within the threshold range, and normalizing the dataset obtained after replacement to obtain the target features.

[0038] In this embodiment, target features are extracted from the training and test sample sets. The median imputation method can be used on the training and test sample sets to replace data in the dataset that are not within the threshold range. The target features can be continuous or discrete. The median imputation process can replace data with missing values ​​in the dataset with the median. Replacing data in the dataset that are not within the threshold range means replacing data that are outside the threshold range with values ​​that are within the threshold range.

[0039] For example, let sample D be... S The median m of continuous feature j j The missing data is represented by m. j Fill in the blanks. The value of percentile n1, P1. j And the value of percentile n2 P2 j If 1 ≥ n2 >>> n1 ≥ 0, then it is less than P1. i The number is P1 i Replacement, greater than P2 i The number is P2 i replace.

[0040] As an optional embodiment, step S104, determining weight data based on the membership matrix, may include: determining the first average membership degree and first class center of training samples whose membership degree is less than a first membership degree threshold in the membership matrix, and determining the second average membership degree and second class center of training samples whose membership degree is greater than a second membership degree threshold in the membership matrix; determining the first distance between the test sample and the training sample based on the first average membership degree, the first class center, the second average membership degree, and the second class center; and determining the weight data of the test sample based on the first distance.

[0041] In this embodiment, based on whether the membership degree in the membership degree matrix is ​​less than the membership degree threshold, the first average membership degree and the first class center, the second average membership degree and the second class center can be determined, and based on the above four types of data, the distance between the test sample and the training sample can be determined to determine the sample weight.

[0042] For example, to calculate the average membership degree u(0,k) and class center c(0,k) of source domain samples in class k with membership degree less than δ, and the average membership degree u(1,k) and class center c(1,k) of all source domain samples in class k with membership degree greater than 1-δ, and the cluster label l of sample i, we can calculate the average membership degree u(1,k) and class center c(1,k) of all source domain samples in class k with membership degree greater than 1-δ. i The data label with the highest membership degree can be used to determine the weight data of the test sample based on these four types of data.

[0043] As an optional embodiment, step S104, determining the weight data of the test sample based on distance, may include: determining a second distance between the test sample and the center point of the training sample; and determining the ratio between the second distance and the first distance as the weight data.

[0044] In this embodiment, the distance between the center points of the test sample and the training sample is determined as the second distance, and the ratio of the second distance to the first distance is determined as the sample weight. The second distance and the first distance are obtained through a fixed calculation method.

[0045] For example, to obtain the data label with the highest membership degree, where... Calculate the distance dist(i) from sample i to the sample in the source domain. The specific formula for calculating this distance is dist(i) = u(0, l). i )*d(i,c(0,l i ))+u(1,l i )*d(i,c(1,l i The specific formula for calculating the other distance is... The formula for calculating the sample weights of the domain classifier model is w. i =d i / dist(i).

[0046] As an optional embodiment, the above method may include: modifying the parameters of the loss function based on the weight data to obtain the modified loss function.

[0047] In this embodiment, the weight data obtained by the above method is used to guide the determination of the loss function parameters. The initial values ​​of the loss function parameters are defined manually. As the model is trained and the loss function is minimized, the parameters of the loss function can eventually be determined.

[0048] For example, if a neural network structure is constructed at this point, the weights calculated using the above method are assigned to each sample in the classifier. These weights can be used to guide the model's loss function. The specific calculation method for the loss function can be as follows: in This is expressed as the loss function of the classifier. This can be represented as the loss function of the domain classifier, where λ is the weight of the domain classifier's loss function. Based on the calculated weight data, as the model converges, the loss function can be guided to obtain parameters that are adapted to the model.

[0049] As an optional embodiment, step S104, modifying the parameters of the loss function based on the weight data to obtain the modified loss function, may include: modifying the weight coefficients of the domain classifier loss function in the loss function to the weight data; determining the product of the domain classifier loss function and the weight data, and determining the difference between the category predictor loss function and the product in the loss function as the modified loss function.

[0050] In this embodiment, based on the obtained weights of the loss function, the weight coefficients of the domain classifier loss function in the loss function are modified, and the product of the domain classifier loss function and the weight data is determined. The difference obtained by subtracting the category classifier loss function in the loss function from the obtained product can be used as the final modified loss function of the model.

[0051] This embodiment performs median imputation on both the training and test sample sets to obtain an imputed dataset. Data outside the threshold range in the imputed dataset are then replaced, and the resulting dataset is normalized to obtain the target features. A second distance is determined between the center points of the test and training samples, and the ratio of this second distance to the first distance is used as the weight data. The parameters of the loss function are modified based on this weight data, thereby solving the technical problem of imbalanced training samples in the classifier model and improving its performance.

[0052] Example 2

[0053] The technical solutions of the embodiments of the present invention will be illustrated below with reference to preferred embodiments.

[0054] With societal progress and technological advancements, the era of big data has arrived. The volume and types of data are constantly increasing, leading to a greater demand for machine learning models with strong generalization capabilities. Theoretically and practically, training a model with a small number of samples will not result in high classification accuracy, and the model will lack strong learning ability. This problem is particularly pronounced when the training and testing data are unevenly distributed. Acquiring a large amount of labeled sample data is extremely costly in terms of human and financial resources. Therefore, how to obtain a large amount of labeled data while conserving resources, or how to improve the model's generalization ability, has become a key issue to be addressed.

[0055] Transfer learning is a crucial technique in machine learning for addressing the fundamental problem of obtaining sufficient labeled data. It reduces the distributional disparity between training samples (source domain) and test samples (target domain), transferring knowledge learned from a large number of labeled samples in the source domain to the target domain, thus solving similar problems and achieving data labeling. Transfer learning involves a domain and a task. The domain, composed of data features and feature distribution, is the core of the learning process and includes the source domain (the domain with existing knowledge, typically containing a large amount of labeled data) and the target domain (the domain to be learned, typically with sparse labeled samples). The task consists of an objective function and the learning result. In practical applications, system features often change over time, leading to variations in the feature space and distribution patterns of the data. Domain-adaptive transfer learning attempts to address the problem of inconsistent data distribution. Commonly used domain adaptation methods, such as sample adaptation, feature adaptation, and model adaptation, can extract similarity relationships between the source and target domains at different stages of the task, improving generalization in the target domain. Generally, the source domain has abundant data with many labeled samples, while the target domain has less data. Therefore, addressing the technical problem of imbalanced training samples in classifier models is currently a key focus.

[0056] Therefore, in order to overcome the above problems, this invention discloses a method based on FCM clustering and transfer adaptive learning. The method calculates the similarity between each sample and the source domain sample according to the membership matrix, assigns different weights to each sample, reduces the weight of samples with similar distribution to the source domain sample, and increases the weight of samples with different distribution to the source domain sample. This effectively solves the problem of unbalanced sample distribution in the domain classifier model in transfer adaptive learning, thereby improving the accuracy of category prediction.

[0057] In this related technology, a method based on FCM clustering and transfer adaptive learning may include:

[0058] Obtain source domain samples and target domain samples, preprocess the data, use FCM clustering to obtain membership degrees, assign weights to the domain classifier model samples based on the membership degrees, and construct a transfer adaptive learning model.

[0059] First, prepare the source domain sample D. S and target domain sample D T The source domain sample D S :(X S ,Y S The dataset is abundant and contains sufficient labeled data; the target domain sample D T :(X T The data is sparse and unlabeled. Calculate the source domain sample D. S Center point c: Calculate the source domain samples D respectively S Mean of continuous feature j The mode of discrete feature v The center point is Where p represents the number of samples with continuous features, and m represents the total number of features. The center distance from each sample to the source domain samples is... x ij This represents the value of the j-th feature of the i-th sample.

[0060] Then, merge the source domain samples D S and target domain sample D T For sample D, perform data preprocessing on sample D to obtain the target features. Statistical source domain sample D S The median m of continuous feature j j Missing values ​​are represented by m j Filler, percentile n1 value and the value of percentile n2 Where 1≥n2>>>n1≥0, the discrete features in the source domain samples are encoded using LabelEncoder. Median imputation, outlier handling, and data normalization are performed on both the source and target domain samples, respectively. That is, missing data is replaced with the median, and values ​​less than 0 are normalized. The number used Replace, greater than The number used Instead, linear function normalization is performed; discrete features are replaced with LabelEncoder values, and the target features are finally obtained.

[0061] Furthermore, FCM clustering is performed on the samples based on the target features. Random numbers within the interval [0,1] are selected to initialize the membership matrix U0. The number of clusters C, the fuzzy weighting exponent, and the threshold ε are set. Iteration stops when the absolute deviation between the previous and current distances is less than ε. The updated membership matrix U = (u i,k ), where i represents the sample and k represents the category.

[0062] At this point, the sample weights of the domain classifier model can be obtained. First, the average membership degree u(0,k) and class center c(0,k) of all source domain samples with membership degree less than δ in class k are calculated, and the average membership degree u(1,k) and class center c(1,k) of all source domain samples with membership degree greater than 1-δ are calculated, along with the cluster label l of sample i. i That is, the data label with the highest membership degree, among which Next, calculate the distance dist(i) from sample i to the sample in the source domain:

[0063] dist(i) = u(0,l) i )*d(i,c(0,l i ))+u(1,l i )*d(i,c(1,l i ))

[0064]

[0065] Calculate the sample weight coefficient w in the domain classifier model during transfer adaptive learning. i =d i / dist(i).

[0066] Finally, a transfer adaptive learning model is constructed, and the loss function is modified to assign sample weights to the domain classifier model samples based on their similarity to the source domain samples. The expression for the domain adaptive model loss function is then:

[0067]

[0068] in, This represents the loss function of the category predictor. Let λ represent the domain classifier loss function, and let λ represent the weight coefficients of the domain classifier loss function.

[0069] In this embodiment, the sample data is first preprocessed, and then the similarity between each sample domain and the source domain samples is calculated based on the membership matrix. Sample weights based on the similarity between the sample data and the classifier samples are assigned, thereby guiding the parameter values ​​of the loss function. This effectively solves the problem of unbalanced sample distribution in the domain classifier model in transfer adaptive learning, thereby improving the accuracy of category prediction.

[0070] According to an embodiment of the present invention, Figure 2 This is a flowchart of an adaptive transfer learning method according to an embodiment of the present invention. As shown in the figure, the adaptive transfer learning method may include:

[0071] Step S201: Obtain source domain data and target domain data.

[0072] In the technical solution provided by step S201 of the present invention, source domain sample data and target domain sample data are first obtained.

[0073] For example, prepare source domain samples D S and target domain sample D T Select the MQTT-IOT-IDS 2020 dataset as the source domain sample D. S The task labels include normal traffic labels and four types of attack labels. Traffic data is captured using tcpdump, which collects Ethernet traffic packets by recording them, and then exported to a pcap file. t The system analyzes 30 features, including source IP and port, destination IP and port, protocol number, and MQTT tag. In the experimental environment, pyshark is used to capture real-time MQTT traffic data from port 224 of the virtual machine ens as the target domain sample D. T Its feature logic is consistent with that of the source domain samples. In the actual modeling process, for the source domain samples D... S Sampling is used, and the source domain samples D are obtained after sampling. S The dataset contains 164,554 samples, including 80,872 normal samples and 83,682 attack samples, with 1,392 samples from the target domain. Twenty-four features were selected: 3 continuous features and 21 discrete features. The source domain sample D was calculated. S Center point c: Calculate the source domain samples D respectively S Mean of continuous feature j and the mode of discrete feature ν The coordinates of the center point are The center distance from each sample to the source region samples is: x ij Let μ represent the value of the j-th feature of the i-th sample, where μ = 1.

[0074] Step S202, data preprocessing.

[0075] In the technical solution provided by step S202 of the present invention, data preprocessing can be used to effectively process data when there are missing values ​​or abnormal data formats, so that the data can be used for model training.

[0076] For example, consider the source domain sample D. S and target domain sample D T Merged into sample D, statistical source domain sample D S The median m of continuous feature j j Missing values ​​are represented by m j Filler, percentile n1 value P1 j and the value of percentile n2 Where 1≥n2>>>n1≥0, the discrete features in the source domain samples are encoded using LabelEncoder. Median imputation, outlier handling, and data normalization are performed on both the source and target domain samples, including replacing missing data with the median and handling values ​​less than 0. The number used Replace, greater than The number used The discrete features are replaced by LabelEncoder encoded values ​​to obtain the target features.

[0077] Step S203, FCM clustering.

[0078] In the technical solution provided by step S203 of the present invention, FCM clustering is fuzzy-means clustering.

[0079] For example, when performing FCM clustering on the target features in sample D, the membership matrix U0 is initialized with random numbers selected from the interval [0,1]. The number of clusters is set to 2, and the fuzzy weighting exponent is 1.7. The iteration stops when the absolute difference between the previous distance and the current distance is less than 0.001. The updated membership matrix is: U = (u i,k ), where i represents the sample and k represents the category.

[0080] Step S204: Calculate the sample weights of the domain classifier model.

[0081] In the technical solution provided by step S204 of the present invention, the sample weights of the domain classifier model are calculated. The sample weights are the weight data of the samples in the model obtained through a specific calculation formula.

[0082] For example, first, calculate the average membership degree u(0,k) and class center c(0,k) of all source domain samples with membership degree less than δ in class k, and the average membership degree u(1,k) and class center c(1,k) of all source domain samples with membership degree greater than 1-δ, as well as the cluster label l of sample i. i This refers to the data label with the highest membership degree. In this paper, δ = 0.5. The distance from sample i to the source domain sample is then calculated using the following formula:

[0083] dist(i) = u(0,l) i )*d(i,c(0,l i ))+u(1,l i )*d(i,c(1,l i ))

[0084]

[0085] The above formula can be used to calculate the sample weight coefficient w in the domain classifier model during transfer adaptive learning.i =d i / dist(i).

[0086] Step S205: Construct a transfer adaptive learning model.

[0087] In the technical solution provided in step S205 of the present invention, a transfer adaptive learning model is constructed, wherein... Figure 3 This is a schematic diagram of the network structure of a transfer adaptive learning model according to an embodiment of the present invention, as shown below. Figure 3 As shown in the diagram, the network structure of a transfer adaptive learning model can include: a feature extractor, a domain classifier, and a category predictor.

[0088] For example, consider an adaptive learning model where the feature extractor has three hidden layers with 32, 40, and 48 neurons respectively. The domain classifier has 48 input neurons, 2 output neurons, and two hidden layers with 60 and 100 neurons respectively. The class predictor has 48 input neurons, 2 output neurons, and one hidden layer with 60 neurons. The loss function of this model can be modified as follows:

[0089]

[0090]

[0091]

[0092] in This represents the loss function of the category predictor. Let λ represent the loss function of the domain classifier, λ represent the weight coefficients of the domain classifier loss function, and δ represent the activation function. λ = 0.5 is used. The source domain samples and target domain samples are processed by a feature extractor to obtain mapped features. These mapped features are then fed into the class discriminator. when This indicates that the flow is normal; when This indicates abnormal traffic and the presence of an attack; when When it indicates that the sample belongs to the source domain; when This indicates that the sample belongs to the target domain; after 300 epochs, the model parameters are solved using the Adam method. Compared to directly using transfer learning, the accuracy is improved by 7.5%.

[0093] In this embodiment, samples from the source domain and the target domain are first obtained and preprocessed. Then, FCM clustering is used to obtain the membership matrix. Based on the membership matrix, the sample weights of the domain classifier model are obtained, thereby constructing an adaptive learning model. This model can calibrate the data, thus solving the technical problem of imbalanced training samples in the classifier model and improving the technical effect of imbalanced training samples in the classifier model.

[0094] Example 3

[0095] According to an embodiment of the present invention, a data processing apparatus is also provided. It should be noted that this data processing apparatus can be used to execute a data processing method as described in Embodiment 1.

[0096] Figure 4 This is a schematic diagram of a data processing apparatus according to an embodiment of the present invention. Figure 4 As shown, the data processing device 400 may include: an acquisition unit 401, an extraction unit 402, a processing unit 403, and a determination unit 404.

[0097] Acquisition unit 401 is used to acquire at least the training sample set and the test sample set.

[0098] Extraction unit 402 is used to extract target features from the training sample set and the test sample set respectively, wherein the target features are used to determine the features of the training samples in the training sample set and the features of the test samples in the test sample set.

[0099] The processing unit 403 is used to perform fuzzy-means clustering on the training sample set and the test sample set based on the extracted target features to obtain the membership matrix.

[0100] The determination unit 404 is used to determine the weight data based on the membership matrix. The sample weight data is used to adjust the weights of the loss function of the initial model to obtain the target model, and the target model is used for data calibration.

[0101] In this embodiment, an acquisition unit acquires a training sample set and a test sample set; an extraction unit extracts target features from the training sample set and the test sample set respectively, wherein the target features are used to determine the features of the training samples in the training sample set and the features of the test samples in the test sample set; a processing unit performs fuzzy-means clustering on the training sample set and the test sample set based on the extracted target features to obtain a membership matrix; and a determination unit determines weight data based on the membership matrix, wherein the sample weight data is used to adjust the weights of the loss function of the initial model to obtain a target model, which is used for data calibration. This solves the technical problem of imbalanced training samples in the classifier model and achieves the technical effect of improving the imbalanced training samples in the classifier model.

[0102] Example 4

[0103] According to an embodiment of the present invention, a computer-readable storage medium is also provided, the storage medium including a stored program, wherein the program executes the data processing method in embodiment 1.

[0104] Example 5

[0105] According to an embodiment of the present invention, an electronic device is also provided for running a program, wherein the electronic device includes one or more processors and a memory for storing one or more programs, wherein when one or more programs are executed by one or more processors, the programs execute the data processing method in embodiment 1.

[0106] Example 6

[0107] According to another aspect of the present invention, a computer program product is also provided, including a computer program that, when executed by a processor, performs the data processing method in embodiment 1.

[0108] The sequence numbers of the above embodiments of the present invention are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0109] In the above embodiments of the present invention, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0110] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. The device embodiments described above are merely illustrative; for example, the division of units can be a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual couplings, direct couplings, or communication connections may be through some interfaces; indirect couplings or communication connections between units or modules may be electrical or other forms.

[0111] The units described as separate components may or may not be physically separate. Similarly, the components shown as units may or may not be physical units; they may be located in one place or distributed across multiple units. Some or all of the units can be selected to achieve the purpose of this embodiment, depending on actual needs.

[0112] Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0113] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, read-only memory (ROM), random access memory (RAM), portable hard drives, magnetic disks, or optical disks.

[0114] The above are merely preferred embodiments of the present invention. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principle of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A data processing method, characterized in that, include: At least a training sample set and a test sample set are obtained, wherein the training sample set and the test sample set are image samples; Target features are extracted from the training sample set and the test sample set, respectively, wherein the target features are used to determine the features of the training samples in the training sample set and the features of the test samples in the test sample set; Based on the extracted target features, fuzzy-means clustering is performed on the training sample set and the test sample set respectively to obtain a membership matrix, which is used to calculate the similarity between samples; The weight data is determined based on the membership matrix, wherein the sample weight data is used to adjust the weights of the loss function of the initial model to obtain the target model, and the target model is used for data calibration. The determination of the weight data based on the membership matrix includes: determining, based on the membership matrix, the first average membership degree and the first class center of the training samples whose membership degree is less than a first membership degree threshold, and determining the second average membership degree and the second class center of the training samples whose membership degree is greater than a second membership degree threshold; determining the first distance between the test sample and the training sample based on the first average membership degree, the first class center, the second average membership degree, and the second class center; and determining the weight data of the test sample based on the first distance.

2. The method according to claim 1, characterized in that, Extracting target features from the training sample set and the test sample set respectively, including: The training sample set and the test sample set are respectively subjected to median imputation to obtain the imputed dataset; Replace the data in the filled dataset that are not within the threshold range, and normalize the dataset after replacement to obtain the target feature.

3. The method according to claim 1, characterized in that, The weight data of the test sample is determined based on the first distance, including: Determine a second distance between the center point of the test sample and the center point of the training sample; The ratio between the second distance and the first distance is determined as the weight data.

4. The method according to claim 1, characterized in that, The method further includes: The parameters of the loss function are modified based on the weight data to obtain the modified loss function.

5. The method according to claim 1, characterized in that, Based on the weight data, the parameters of the loss function are modified to obtain the modified loss function, including: Modify the weight coefficients of the domain classifier loss function in the loss function to the weight data; The product of the domain classifier loss function and the weight data is determined, and the difference between the category predictor loss function and the product is determined as the modified loss function.

6. A data processing apparatus, characterized in that, include: An acquisition unit is configured to acquire at least a training sample set and a test sample set, wherein the training sample set and the test sample set are image samples; An extraction unit is configured to extract target features from the training sample set and the test sample set, respectively, wherein the target features are used to determine the features of the training samples in the training sample set and the features of the test samples in the test sample set; The processing unit is used to perform fuzzy-means clustering on the training sample set and the test sample set based on the extracted target features to obtain a membership matrix, which is used to calculate the similarity between samples. A determining unit is used to determine weight data based on the membership matrix, wherein the sample weight data is used to adjust the weights of the loss function of the initial model to obtain the target model, and the target model is used for data calibration; The determining unit is further configured to, based on the membership matrix, determine the first average membership degree and the first class center of the training samples whose membership degree is less than a first membership degree threshold in the membership matrix, and determine the second average membership degree and the second class center of the training samples whose membership degree is greater than a second membership degree threshold in the membership matrix; determine the first distance between the test sample and the training sample based on the first average membership degree, the first class center, the second average membership degree, and the second class center; and determine the weight data of the test sample based on the first distance.

7. A non-volatile storage medium, characterized in that, The non-volatile storage medium stores a plurality of instructions adapted for loading by a processor and executing the method of any one of claims 1 to 5.

8. An electronic device, characterized in that, The method includes one or more processors and a memory for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors cause the one or more processors to implement the method of any one of claims 1 to 5.

9. A computer program product comprising a computer program that, when executed by a processor, implements the method according to any one of claims 1-5.

Images