Unbalanced big data classification method, system and device and storage medium

Abstract

The embodiment of the invention provides an unbalanced big data classification method, system and device and a storage medium, and is applied to the field of big data classification. The method comprises the following steps: acquiring unbalanced high-dimensional dynamic data in real time, and classifying the unbalanced high-dimensional dynamic data based on an unbalanced big data classification model to obtain a real-time classification result; wherein the unbalanced big data classification model is obtained by training through the following steps: pre-processing pre-obtained unbalanced high-dimensional dynamic data, generating minority class synthesis samples through an online clustering algorithm and a conditional generative adversarial network, and mixing the minority class synthesis samples with original data to obtain a balanced data set; training the deep variational belief network to obtain an unbalanced big data classification model; the deep variational belief network comprises N hidden layers, and N is greater than or equal to 10. In this way, the capturing capacity of high-dimensional nonlinear features and minority class classification performance can be enhanced, noise and distribution deviation are reduced, and the retraining cost and calculation complexity are reduced.

Description

Technical Field

[0001] This disclosure relates to the field of big data classification, and in particular to an imbalanced big data classification method, system, device, and storage medium. Background Technology

[0002] With the widespread adoption of the Internet of Things (IoT), real-time monitoring, and big data analytics, highly imbalanced, high-dimensional data is increasingly prevalent in applications across finance, industry, and healthcare, posing a significant challenge to data classification tasks. In real-time financial transactions, fraudulent samples account for less than 0.01%, and transaction data is dynamically generated in streaming form, requiring real-time fraud detection. In industrial scenarios, abnormal samples involved in equipment fault monitoring are extremely rare, and the data features are highly dimensional and complexly distributed, making it difficult to accurately capture fault characteristics using conventional methods. Similar problems exist in scenarios such as disease anomaly diagnosis in the medical field, where the scarcity of minority class samples and the high-dimensionality of the data coexist, placing stringent demands on the performance of classification models.

[0003] Traditional classification methods, such as random forests and support vector machines, struggle to effectively capture key features of minority class samples when dealing with highly imbalanced high-dimensional data. This is due to the skewed distribution of samples and the sparsity of high-dimensional data, leading to classification results biased towards the majority class and extremely low minority class identification accuracy, which fails to meet practical application needs. Deep belief networks (DBNs) have shown certain advantages in complex data modeling due to their hierarchical feature extraction capabilities. However, traditional DBNs have relatively shallow layers, making it difficult to fully capture complex nonlinear patterns in high-dimensional data. Furthermore, they lack adaptability to dynamic data flows and cannot cope with scenarios where data changes dynamically over time.

[0004] Among existing sampling methods, traditional algorithms such as SMOTE and ADASYN have certain effects in static data scenarios. However, the generated synthetic samples are prone to introducing noise or distribution bias in dynamic data streams, leading to a decrease in classification performance. Although Generative Adversarial Networks (GANs) and Variational Autoencoders (VAEs) can generate synthetic samples to balance data distribution, they suffer from poor training stability and high computational overhead, making them difficult to adapt to application scenarios with high real-time requirements.

[0005] Furthermore, existing methods lack comprehensive optimization of deeper network structures and online learning strategies, and cannot simultaneously solve the problems of sample balancing for extremely imbalanced data, feature extraction for high-dimensional data, and real-time adaptation to dynamic data streams. As a result, they are unable to meet the requirements of high performance, high robustness, and real-time performance for classification tasks of extremely imbalanced high-dimensional dynamic data. Summary of the Invention

[0006] This disclosure provides a method, system, device, and storage medium for classifying imbalanced big data, which solves the technical problems of low minority class recognition accuracy, poor model adaptability, and insufficient real-time performance in the classification of extremely imbalanced high-dimensional dynamic data.

[0007] According to a first aspect of this disclosure, a method for classifying imbalanced big data is provided. The method includes: Real-time acquisition of imbalanced high-dimensional dynamic data; The imbalanced high-dimensional dynamic data is classified based on an imbalanced big data classification model to obtain real-time classification results; wherein, the imbalanced big data classification model is trained through the following steps: Preprocessing is performed on the pre-acquired imbalanced high-dimensional dynamic data to obtain multi-scale feature representation data; Based on the multi-scale feature representation data, minority class synthetic samples are generated through online clustering algorithms and conditional generative adversarial networks. The minority class synthetic samples are then mixed with the pre-acquired imbalanced high-dimensional dynamic data to obtain a balanced dataset. Based on the balanced dataset, a deep variational belief network is trained to obtain an imbalanced big data classification model; the deep variational belief network includes N hidden layers, with the number of neurons in each hidden layer decreasing layer by layer; wherein N is greater than or equal to 10.

[0008] In addition to the aspects and any possible implementations described above, a further implementation is provided, wherein the preprocessing of the pre-acquired imbalanced high-dimensional dynamic data to obtain multi-scale feature representation data includes: The imbalanced high-dimensional dynamic data includes: numerical data, text data, and time-series data; Perform discrete wavelet transform on the numerical data to extract frequency features; The semantic vectors of the text data are extracted using the DistilBERT model; The temporal features of the time-series data are extracted using a Transformer encoder. The frequency features, semantic vectors, and temporal features are normalized to obtain multi-scale feature representation data.

[0009] As described above and in any possible implementation, a further implementation is provided, wherein generating synthetic minority class samples based on the multi-scale feature representation data using an online clustering algorithm and a conditional generative adversarial network includes: Minority class samples are selected from the multi-scale feature representation data to obtain a minority class feature set; The minority class feature set is clustered using an online clustering algorithm, and the cluster centers are dynamically adjusted and updated. Based on the feature representation of the cluster centers, a conditional generative adversarial network is used to generate synthetic minority class samples.

[0010] In addition to the aspects and any possible implementations described above, a further implementation is provided, wherein the method further includes: Calculate the Euclidean distance between the minority class composite sample and the corresponding cluster center, remove minority class composite samples whose Euclidean distance exceeds a preset threshold, and adjust the sampling ratio of the minority class composite samples according to the data flow imbalance rate, thereby increasing the number of minority class samples to meet the preset ratio threshold with the number of majority class samples.

[0011] In addition to the aspects and any possible implementations described above, a further implementation is provided in which the training of a deep variational belief network based on the balanced dataset to obtain an imbalanced big data classification model includes: A variational restricted Boltzmann machine is used to perform hierarchical variational pre-training on the deep variational belief network, and the number of adaptive sampling steps is adjusted according to the data sparsity. Based on the loss changes during the training process of deep variational belief networks, dynamic topology adjustment is performed on deep variational belief networks; Based on a balanced dataset, a deep variational belief network is fine-tuned to obtain an imbalanced big data classification model.

[0012] In addition to the aspects and any possible implementations described above, a further implementation is provided in which the deep variational belief network is fine-tuned based on a balanced dataset to obtain an imbalanced big data classification model, including: Based on a balanced dataset, the weights of the deep variational belief network are trained online using variational methods. During the online variational training process, an attention mechanism is introduced to allocate feature weights according to feature information gain and update the attention weights according to a preset iteration cycle. Training is terminated when the verification performance meets the preset conditions, and the trained imbalanced big data classification model is obtained.

[0013] According to a second aspect of this disclosure, an imbalanced big data classification system is provided. The system includes: an acquisition module for acquiring imbalanced high-dimensional dynamic data in real time; The classification module is used to classify the imbalanced high-dimensional dynamic data based on an imbalanced big data classification model to obtain real-time classification results; wherein, the imbalanced big data classification model is trained through the following steps: Preprocessing is performed on the pre-acquired imbalanced high-dimensional dynamic data to obtain multi-scale feature representation data; Based on the multi-scale feature representation data, minority class synthetic samples are generated through online clustering algorithms and conditional generative adversarial networks. The minority class synthetic samples are then mixed with the pre-acquired imbalanced high-dimensional dynamic data to obtain a balanced dataset. Based on the balanced dataset, a deep variational belief network is trained to obtain an imbalanced big data classification model; the deep variational belief network includes N hidden layers, with the number of neurons in each hidden layer decreasing layer by layer; wherein N is greater than or equal to 10.

[0014] According to a third aspect of this disclosure, an electronic device is provided. The electronic device includes a memory and a processor, wherein the memory stores a computer program, and the processor executes the program to implement the methods according to the first and / or second aspects of this disclosure.

[0015] According to a fourth aspect of this disclosure, a computer-readable storage medium is provided having a computer program stored thereon that, when executed by a processor, implements the methods according to the first and / or second aspects of this disclosure.

[0016] In this application, imbalanced high-dimensional dynamic data is acquired in real time, and the imbalanced big data classification model is used to classify the imbalanced high-dimensional dynamic data to obtain real-time classification results. The imbalanced big data classification model is trained through the following steps: preprocessing the pre-acquired imbalanced high-dimensional dynamic data to obtain multi-scale feature representation data; generating minority class synthetic samples using an online clustering algorithm and a conditional generative adversarial network, and mixing these samples with the pre-acquired imbalanced high-dimensional dynamic data to obtain a balanced dataset; and training a deep variational belief network based on the balanced dataset to obtain the imbalanced big data classification model. The deep variational belief network includes N hidden layers, with the number of neurons in each hidden layer decreasing progressively; where N is greater than or equal to 10. This approach enhances the ability to capture high-dimensional nonlinear features, reduces noise and distribution bias, lowers retraining costs, improves minority class classification performance, and reduces computational complexity.

[0017] It should be understood that the description in the Summary of the Invention is not intended to limit the key or essential features of the embodiments of this disclosure, nor is it intended to restrict the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0018] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. The drawings are provided for a better understanding of the invention and are not intended to limit the scope of this disclosure. In the drawings, the same or similar reference numerals denote the same or similar elements, wherein: Figure 1A flowchart of an imbalanced big data classification method according to an embodiment of the present disclosure is shown; Figure 2 A flowchart illustrating the training of an imbalanced big data classification model according to an embodiment of the present disclosure is shown; Figure 3 A block diagram of an imbalanced big data classification system according to an embodiment of the present disclosure is shown; Figure 4 A block diagram of an exemplary electronic device capable of implementing embodiments of the present disclosure is shown. Detailed Implementation

[0019] To make the objectives, technical solutions, and advantages of the embodiments of this disclosure clearer, the technical solutions of the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this disclosure, and not all embodiments. Based on the embodiments of this disclosure, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this disclosure.

[0020] Furthermore, the term "and / or" in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. Additionally, the character " / " in this article generally indicates that the preceding and following related objects have an "or" relationship.

[0021] This disclosure acquires imbalanced high-dimensional dynamic data in real time and classifies it based on an imbalanced big data classification model to obtain real-time classification results. The imbalanced big data classification model is trained through the following steps: preprocessing the pre-acquired imbalanced high-dimensional dynamic data to obtain multi-scale feature representation data; generating minority class synthetic samples using an online clustering algorithm and a conditional generative adversarial network, and mixing these samples with the pre-acquired imbalanced high-dimensional dynamic data to obtain a balanced dataset; and training a deep variational belief network based on the balanced dataset to obtain the imbalanced big data classification model. The deep variational belief network includes N hidden layers, with the number of neurons in each hidden layer decreasing progressively; where N is greater than or equal to 10. This approach enhances the ability to capture high-dimensional nonlinear features, reduces noise and distribution bias, lowers retraining costs, improves minority class classification performance, and reduces computational complexity.

[0022] Figure 1 A flowchart of an imbalanced big data classification method 100 according to an embodiment of the present disclosure is shown. Figure 1 As shown, method 100 includes: S101, real-time acquisition of imbalanced high-dimensional dynamic data; In one possible implementation, imbalanced high-dimensional dynamic data includes numerical data, text data, and time-series data.

[0023] For example, the imbalanced high-dimensional dynamic data originates from scenarios such as financial transactions, industrial equipment monitoring, and medical diagnosis. Data types include numerical data (such as transaction amounts, equipment operating parameters, and physiological indicator values), text data (such as transaction notes, equipment maintenance records, and medical records), and time-series data (such as continuous transaction time series, equipment operating status time-series data, and patient vital sign monitoring time-series data). The data features have more than 1000 dimensions, with minority class samples accounting for less than 1%, and the data stream is dynamically generated at a rate of 1000 data points per second.

[0024] For example, data streams are received in real time through the data input interface, supporting multiple data input formats such as Kafka streams, CSV files, and JSON. The data is parsed using the Apache Spark framework, with a preprocessing time of approximately 20 seconds (100,000 data entries), ensuring the real-time nature and compatibility of data reception.

[0025] S102, classify imbalanced high-dimensional dynamic data based on an imbalanced big data classification model to obtain real-time classification results; like Figure 2 As shown, the imbalanced big data classification model is trained through the following steps: S201 involves preprocessing the pre-acquired imbalanced high-dimensional dynamic data to obtain multi-scale feature representation data.

[0026] In one possible implementation, for the input imbalanced high-dimensional dynamic data, missing values ​​are detected by sliding window, missing values ​​are filled by k-nearest neighbor interpolation, missing text is completed based on a pre-trained language model, and outlier samples are removed by kernel density estimation.

[0027] For example, a sliding window (size 10, step size 5) is used to detect missing values ​​in dynamic streaming data. For numerical missing values, k-nearest neighbor interpolation (k=5) is used to fill in the missing values; for textual missing values, the missing values ​​are filled in based on a pre-trained language model.

[0028] For example, the kernel density estimation (KDE) method is applied to detect outliers. A threshold of 0.01 is set to remove about 0.3% of noisy data (e.g., about 9,000 outliers out of 3 million data points) to ensure data quality.

[0029] In one possible implementation, discrete wavelet transform is performed on numerical data to extract frequency features; the DistilBERT model is used to extract semantic vectors from text data; a Transformer encoder is used to extract temporal features from time-series data; and the frequency features, semantic vectors, and temporal features are normalized to obtain multi-scale feature representation data.

[0030] For example, in numerical data processing, discrete wavelet transform (DWT, DaubechiesD4, three-level decomposition) is used to extract low-frequency and high-frequency features, reducing the initial 1000-dimensional numerical data to 200-dimensional, while retaining key frequency features.

[0031] For example, in text data processing, the DistilBERT model (512-dimensional embedding) is used to extract text semantic vectors, and the dimensionality is reduced to 50 dimensions through t-SNE. The processing time for 1 million text data is about 1 hour, which efficiently extracts text semantic information.

[0032] For example, in the processing of time series data, the Transformer encoder (hidden unit 128) extracts time series features and outputs a 64-dimensional vector to capture the dynamic change patterns of the time series data.

[0033] For example, for numerical data: the Z-score normalization method is used to transform the data to the range [-3,3] to eliminate the influence of differences in numerical magnitude.

[0034] For example, for text data and time series data: the Min-Max normalization method is used to normalize the data to the [0,1] interval to ensure the fusion of different types of features.

[0035] For example, imbalanced high-dimensional dynamic data of multiple modalities can be transformed into multi-scale feature representation data of 264 dimensions (200-dimensional numerical features + 50-dimensional text features + 64-dimensional temporal features) to provide high-quality input for subsequent sampling and model training.

[0036] S202, based on multi-scale feature representation data, generates minority class synthetic samples through online clustering algorithms and conditional generative adversarial networks. The minority class synthetic samples are then mixed with pre-acquired imbalanced high-dimensional dynamic data to obtain a balanced dataset.

[0037] In one possible implementation, minority class samples are selected from multi-scale feature representation data to obtain a minority class feature set; the minority class feature set is clustered using an online clustering algorithm, and the cluster centers are dynamically adjusted and updated; based on the feature representation of the cluster centers, a conditional generative adversarial network is used to generate synthetic minority class samples.

[0038] In one possible implementation, the Euclidean distance between the minority class composite sample and the corresponding cluster center is calculated, minority class composite samples whose Euclidean distance exceeds a preset threshold are removed, and the sampling ratio of minority class composite samples is adjusted according to the data flow imbalance rate, thereby increasing the number of minority class samples to meet the preset ratio threshold with the number of majority class samples.

[0039] For example, minority class samples (accounting for <1%) are selected from multi-scale feature representation data to form a minority class feature set.

[0040] For example, an online k-means clustering algorithm is applied to a minority class feature set. The initial k value is set to 5. The cluster quality is evaluated based on the silhouette coefficient, and the k value is dynamically adjusted to the range of 3-10 (e.g., k=4 in 1 million data points). The cluster centers are updated once every 1,000 new data points are received to ensure that the clustering results adapt to changes in the data flow.

[0041] For example, based on the feature representation of cluster centers, a conditional generative adversarial network (cGAN) is used to generate synthetic minority class samples. The cGAN generator outputs 32-dimensional latent features, and the discriminator uses a 3-layer fully connected structure to distinguish between real and synthetic samples. The cGAN is trained using an Adam optimizer (learning rate 0.0002) for 50 epochs with a batch size of 64, aiming to increase the number of minority class samples to 10% of the majority class samples (e.g., from 5000 minority class samples to 50,000).

[0042] For example, the Euclidean distance between the synthetic sample and the corresponding cluster center is calculated, and low-quality synthetic samples with a distance greater than 1.5 times the mean are removed (about 3%). The sampling ratio is dynamically adjusted according to the data flow imbalance rate (IR). When IR>1000, the synthetic sample is increased by 20% to ensure that the number of minority class samples and the number of majority class samples meet the preset ratio threshold (e.g., 1:10).

[0043] For example, validated minority class synthetic samples are mixed with the original samples of pre-acquired imbalanced high-dimensional dynamic data to obtain a balanced dataset, providing a balanced sample distribution for model training.

[0044] S203, based on a balanced dataset, trains a deep variational belief network to obtain an imbalanced big data classification model.

[0045] In one possible implementation, the deep variational belief network comprises N hidden layers, with the number of neurons in each hidden layer decreasing progressively; where N is greater than or equal to 10.

[0046] In one possible implementation, a variational restricted Boltzmann machine is used to perform hierarchical variational pre-training on the deep variational belief network, and the number of adaptive sampling steps is adjusted according to the data sparsity; the deep variational belief network is dynamically topologically adjusted based on the loss changes during the training process; and the deep variational belief network is fine-tuned based on a balanced dataset to obtain an imbalanced big data classification model.

[0047] In one possible implementation, the weights of a deep variational belief network are trained online based on a balanced dataset. In one possible implementation, an attention mechanism is introduced during online variational training. Feature weights are assigned based on feature information gain, and the attention weights are updated according to a preset iteration cycle. Training is terminated when the verification performance meets the preset conditions, resulting in a trained imbalanced big data classification model.

[0048] For example, a deep variational belief network is initialized using a DVBN network comprising an input layer, N hidden layers (N≥10), and an output layer. The input layer is 264-dimensional (receiving multi-scale feature representation data); the number of neurons in the 10 hidden layers decreases progressively: 2048, 1024, 512, 256, 128, 64, 32, 16, 8, 4; the output layer is binary (minority / major). The network uses the LeakyReLU activation function (slope 0.2) and a Dropout rate of 0.4 to prevent overfitting.

[0049] For example, a variational restricted Boltzmann machine (VRBM) is used to pre-train DVBN layer by layer, with each layer trained for 30 epochs and a batch size of 256. The number of adaptive sampling steps is dynamically adjusted according to the data sparsity. The initial number of sampling steps is 100, and when the data sparsity is <0.1, it is dynamically reduced to 10 to optimize the feature extraction effect.

[0050] For example, the model loss is evaluated every 20 epochs of training. If the loss decreases by less than 0.005, the network topology is adjusted by adding one hidden layer (up to a maximum of 12 layers) or by increasing the number of neurons in the existing hidden layer by 10%. At the same time, a dynamic pruning strategy is applied to remove network connections with weights whose absolute values ​​are less than 0.1, reducing the model parameters by about 15% and lowering the computational complexity.

[0051] For example, DVBN is trained online using a balanced dataset. Every 1000 new data points, the network weights are incrementally updated using a mini-batch Adam optimizer (learning rate 0.0005). The optimization objective is weighted cross-entropy loss, and the weights of minority class samples are dynamically adjusted (weight range 5-20) according to the data flow imbalance rate (IR). A self-attention mechanism is introduced, which assigns feature weights based on feature information gain (threshold > 0.2), focusing on key features of the minority class, and updates the attention weights every 10 epochs. After training for 20 epochs, the learning rate is halved to 0.00025 to continuously optimize model performance.

[0052] For example, the validation performance metrics of the real-time computing model (including precision, recall, F1 score, and AUPRC) are calculated. When the recall is ≥0.95 and the F1 score is ≥0.94, the training is terminated, and the trained imbalanced big data classification model is obtained.

[0053] For example, the real-time acquired imbalanced high-dimensional dynamic data is preprocessed and converted into multi-scale feature representation data, which is then input into the trained imbalanced big data classification model. The model outputs real-time classification results (minority class / majority class), and the output format supports JSON. The output time for processing 1 million data points is about 5 minutes, which meets the real-time requirements.

[0054] In one possible implementation, in addition to outputting real-time classification results, the system also generates performance reports (PDF format) and visualizations (based on Plotly), including a real-time confusion matrix heatmap (showing minority class accuracy >92%), a 2D scatter plot after UMAP dimensionality reduction (verifying class separation effect), and performance metrics such as precision, recall, F1 score, G-mean, and AUPRC, making it convenient for users to monitor the model's running status.

[0055] In this application, imbalanced high-dimensional dynamic data is acquired in real time, and the imbalanced big data classification model is used to classify the imbalanced high-dimensional dynamic data to obtain real-time classification results. The imbalanced big data classification model is trained through the following steps: preprocessing the pre-acquired imbalanced high-dimensional dynamic data to obtain multi-scale feature representation data; generating minority class synthetic samples using an online clustering algorithm and a conditional generative adversarial network, and mixing these samples with the pre-acquired imbalanced high-dimensional dynamic data to obtain a balanced dataset; and training a deep variational belief network based on the balanced dataset to obtain the imbalanced big data classification model. The deep variational belief network includes N hidden layers, with the number of neurons in each hidden layer decreasing progressively; where N is greater than or equal to 10. This approach enhances the ability to capture high-dimensional nonlinear features, reduces noise and distribution bias, lowers retraining costs, improves minority class classification performance, and reduces computational complexity.

[0056] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that this disclosure is not limited to the described order of actions, because according to this disclosure, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are all optional embodiments, and the actions and modules involved are not necessarily essential to this disclosure.

[0057] The above is an introduction to the method embodiments. The following describes the solution described in this disclosure further through device embodiments.

[0058] Figure 3 A block diagram of an imbalanced big data classification system 300 according to an embodiment of the present disclosure is shown. Figure 3 As shown, the device 300 includes: The acquisition module 301 is used to acquire imbalanced high-dimensional dynamic data in real time. Classification module 302 is used to classify imbalanced high-dimensional dynamic data based on an imbalanced big data classification model to obtain real-time classification results; wherein, the imbalanced big data classification model is trained through the following steps: Preprocessing is performed on the pre-acquired imbalanced high-dimensional dynamic data to obtain multi-scale feature representation data; Based on multi-scale feature representation data, a minority class synthetic sample is generated through online clustering algorithm and conditional generative adversarial network. The minority class synthetic sample is then mixed with pre-acquired imbalanced high-dimensional dynamic data to obtain a balanced dataset. Based on a balanced dataset, a deep variational belief network is trained to obtain an imbalanced big data classification model. The deep variational belief network consists of N hidden layers, with the number of neurons in each hidden layer decreasing progressively. N is greater than or equal to 10.

[0059] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the described module can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0060] The acquisition, storage, and application of user personal information involved in the technical solution disclosed herein comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0061] According to embodiments of this disclosure, this disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0062] Figure 4A schematic block diagram of an electronic device 400 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0063] Electronic device 400 includes a computing unit 401, which can perform various appropriate actions and processes according to a computer program stored in ROM 402 or a computer program loaded into RAM 403 from storage unit 408. RAM 403 can also store various programs and data required for the operation of electronic device 400. The computing unit 401, ROM 402, and RAM 403 are interconnected via bus 404. I / O interface 405 is also connected to bus 404.

[0064] Multiple components in electronic device 400 are connected to I / O interface 405, including: input unit 406, such as keyboard, mouse, etc.; output unit 407, such as various types of displays, speakers, etc.; storage unit 408, such as disk, optical disk, etc.; and communication unit 409, such as network card, modem, wireless transceiver, etc. Communication unit 409 allows electronic device 400 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0065] The computing unit 401 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 401 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 401 performs the various methods and processes described above, such as method 100. For example, in some embodiments, method 100 may be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 408. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 400 via ROM 402 and / or communication unit 409. When the computer program is loaded into RAM 403 and executed by the computing unit 401, one or more steps of method 100 described above may be performed. Alternatively, in other embodiments, the computing unit 401 may be configured to perform method 100 by any other suitable means (e.g., by means of firmware).

[0066] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0067] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0068] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0069] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including voice input, speech input, or tactile input).

[0070] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0071] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact via communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other. Servers can be cloud servers, servers in distributed systems, or servers incorporating blockchain technology.

[0072] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0073] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. An unbalanced big data classification method, characterized in that, The method comprises: real-time acquisition of unbalanced high-dimensional dynamic data; classification of the unbalanced high-dimensional dynamic data based on an unbalanced big data classification model to obtain a real-time classification result; wherein the unbalanced big data classification model is obtained by the following steps: preprocessing of pre-acquired unbalanced high-dimensional dynamic data to obtain multi-scale feature representation data; based on the multi-scale feature representation data, generating minority class synthetic samples by an online clustering algorithm and a conditional generative adversarial network, mixing the minority class synthetic samples with the pre-acquired unbalanced high-dimensional dynamic data to obtain a balanced data set; training a deep variational belief network based on the balanced data set to obtain an unbalanced big data classification model; the deep variational belief network comprises N layers of hidden layers, and the number of hidden layer neurons decreases layer by layer; wherein N is greater than or equal to 10.

2. The method of claim 1, wherein, The preprocessing of the pre-acquired unbalanced high-dimensional dynamic data to obtain multi-scale feature representation data comprises: the unbalanced high-dimensional dynamic data comprises numerical data, text data, and time series data; discrete wavelet transform is performed on the numerical data to extract frequency features; a DistilBERT model is used to extract semantic vectors of the text data; a Transformer encoder is used to extract time series features of the time series data; the frequency features, the semantic vectors, and the time series features are normalized to obtain multi-scale feature representation data.

3. The method of claim 1, wherein, The generation of minority class synthetic samples based on the multi-scale feature representation data by an online clustering algorithm and a conditional generative adversarial network comprises: minority class samples are selected from the multi-scale feature representation data to obtain a minority class feature set; the minority class feature set is clustered by an online clustering algorithm, and the cluster centers are dynamically adjusted and updated; minority class synthetic samples are generated based on the feature representation of the cluster centers by a conditional generative adversarial network.

4. The method of claim 3, wherein, The method further comprises: calculating the Euclidean distance between the minority class synthetic samples and the corresponding cluster centers, removing the minority class synthetic samples whose Euclidean distance exceeds a preset threshold, and adjusting the sampling ratio of the minority class synthetic samples according to the data stream imbalance rate, so as to increase the number of minority class samples to satisfy a preset proportion threshold with the number of majority class samples.

5. The method of claim 4, wherein, The training of a deep variational belief network based on the balanced data set to obtain an unbalanced big data classification model comprises: a variational restricted Boltzmann machine is used to perform hierarchical variational pre-training of the deep variational belief network, and the adaptive sampling step number is adjusted according to data sparsity; based on the loss change in the deep variational belief network training process, the deep variational belief network is dynamically topologically adjusted; based on the balanced data set, the deep variational belief network is fine-tuned to obtain an unbalanced big data classification model.

6. The method of claim 5, wherein, The fine-tuning of the deep variational belief network based on the balanced data set to obtain an unbalanced big data classification model comprises: The deep variational confidence network weights are trained online based on the balanced data set; wherein, in the online variational training process, an attention mechanism is introduced, feature weights are allocated according to feature information gain, and attention weights are updated according to a preset iteration period; When the verification performance meets the preset condition, the training is terminated, and the unbalanced big data classification model trained is obtained.

7. An imbalanced big data classification system, characterized by, Comprise: An acquisition module is configured to acquire unbalanced high-dimensional dynamic data in real time; A classification module is configured to classify the unbalanced high-dimensional dynamic data based on an unbalanced big data classification model to obtain a real-time classification result; wherein, the unbalanced big data classification model is trained by the following steps: The pre-acquired unbalanced high-dimensional dynamic data is preprocessed to obtain multi-scale feature representation data; Based on the multi-scale feature representation data, a few-class synthetic sample is generated by an online clustering algorithm and a conditional generative adversarial network, the few-class synthetic sample is mixed with the pre-acquired unbalanced high-dimensional dynamic data to obtain a balanced data set; Based on the balanced data set, a deep variational confidence network is trained to obtain an unbalanced big data classification model; the deep variational confidence network comprises N layers of hidden layers, and the number of hidden layer neurons decreases layer by layer; wherein, N is greater than or equal to 10.

8. An electronic device comprising: At least one processor; And A memory connected in communication with the at least one processor; wherein, The memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-6.

9. A non-transitory computer readable storage medium having stored thereon computer instructions, wherein, The computer instructions are used to enable the computer to perform the method according to any one of claims 1 to 6.

Images