Digital transmission method for deep learning of documents
Through the deep learning model, the maximum available broadband of the transmission path is judged and document compression and transmission is carried out, which solves the problems of poor robustness and low transmission rate in the prior art under variable channel conditions, and achieves efficient and low-cost document transmission.
Patent Information
- Application Number
- CN202411737368.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-11-29
- Publication Date
- 2025-03-07
AI Technical Summary
The existing deep learning-based literature transmission methods lack robustness under variable channel conditions and cannot effectively adapt to dynamic bit rate constraints in time and space, resulting in low transmission rates and high broadband costs.
By collecting network performance data, performing data cleaning and feature extraction, using convolutional neural networks to train deep learning models, judging the maximum available broadband of the transmission path, and performing literature compression and transmission, and dynamically adjusting the transmission strategy to adapt to channel changes.
Efficient literature transmission under variable channel conditions is realized, broadband costs are reduced, transmission rates are improved, and adaptability to dynamic bit rate constraints is improved.
Smart Images
Figure CN119583479A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the field of digital transmission technology, and in particular relates to a digital transmission method for deep learning of documents. Background Art
[0002] According to a survey of the Internet industry report, the literature data on the Internet has increased exponentially since 2020. Although the computing power of mobile terminals has been greatly improved and 4G networks have become increasingly perfect, the explosive growth of literature data has brought new challenges to the storage and transmission of literature on the mobile Internet.
[0003] At present, in mainstream document transmission methods, document compression and bit transmission are considered to be two indispensable parts for achieving the transmission task. If it is difficult to adapt to the changes in the channel in a timely and effective manner during the data transmission process, the quality of the document signal may be degraded, or even the document transmission task may fail. At the same time, in many applications, the communication network or storage device may impose constraints on the maximum bit rate, which requires the document encoder to adapt to the given bit rate budget. In some applications, this constraint may even change dynamically over time. The current document compression transmission method based on deep learning can adapt to the transmission conditions in specific situations through neural networks, but it lacks certain robustness for channel conditions in different situations. At the same time, it cannot be deployed on a large scale in real life. The reason is that the high complexity in time and space limits the practicality of this method, such as unsustainable computational burden and memory consumption. In addition, in order to obtain the best rate-distortion performance, most deep learning-based methods need to retrain separate models for a single bit rate target, which is inefficient in both time and space. Therefore, it is of profound practical significance to study how documents are transmitted under variable channel conditions. At the same time, the multi-rate transmission problem is also an urgent problem to be solved in document transmission based on deep learning. Summary of the invention
[0004] The purpose of the present invention is to provide a digital transmission method for deep learning of documents, which trains a model through a deep learning algorithm, uses the model to determine the maximum available bandwidth of the transmission path, compresses the document, and uses the maximum available bandwidth to transmit the document, thereby solving the problems of high cost and low transmission rate of existing document transmission bandwidth.
[0005] In order to solve the above technical problems, the present invention is achieved through the following technical solutions:
[0006] The present invention is a digital transmission method for deep learning of documents, comprising the following steps:
[0007] Step S1, collecting network performance data: collecting a large amount of data about network performance, including: bandwidth usage, delay data and packet loss rate data;
[0008] Step S2, data cleaning: clean the collected data to remove outliers and noise;
[0009] Step S3, feature extraction: extracting features that can reflect the maximum available bandwidth of the transmission path from the cleaned network performance data, specifically including: minimum bandwidth, average bandwidth and delay jitter on the link;
[0010] Step S4, deep learning model training: select a convolutional neural network to train the extracted feature data set;
[0011] Step S5, model verification: Use the test set to verify the trained deep learning model;
[0012] Step S6, document transmission: using encoding technology to compress the document to be transmitted;
[0013] Step S7, transmission identification: input the trained deep learning model to determine the maximum available bandwidth of the transmission path;
[0014] Step S8: Output display: The deep learning model outputs and displays the decompressed document.
[0015] As a preferred technical solution, in step S1, network performance data is collected in real time using a network monitoring tool or by deploying hardware or software probes at key nodes in the network to capture and analyze the network performance data of the nodes; network monitoring tools such as Wireshark, SolarWinds, Nagios, Zabbix, etc., and many network devices such as routers and switches support SNMP (Simple Network Management Protocol), and the operating status, traffic statistics and other information of the device can be obtained through the SNMP protocol; the probe can provide more accurate network performance data, but requires certain hardware investment and maintenance costs;
[0016] When collecting network performance data, you need to pay attention to the following points: 1. Ensure the accuracy and completeness of data collection to avoid data loss or errors; 2. Select appropriate data collection methods and tools based on actual needs; 3. Regularly analyze and evaluate the collected data to promptly identify and resolve network performance issues; 4. Protect the security and privacy of the collected data to avoid data leakage or abuse.
[0017] As a preferred technical solution, in step S2, the specific process of data cleaning of network performance data is as follows:
[0018] Step S21, deleting data beyond the time range: determining the time range of the data, and deleting the data beyond the time range;
[0019] Step S22, processing duplicate data: identifying and deleting duplicate data records;
[0020] Step S23, processing missing data: for missing data, fill, interpolate or delete according to actual conditions;
[0021] Step S24, filtering outliers: using statistical methods to identify and filter out outliers;
[0022] After the cleaning is completed, it is necessary to apply data transformation: transform the data as needed, such as logarithmic transformation, square root transformation, etc., to make the data more consistent with the assumptions of the analysis model; verify data quality: after the cleaning is completed, perform a quality check on the data to ensure the accuracy and completeness of the data; back up the original data: before performing any data cleaning operation, be sure to back up the original data to prevent accidental loss or error; record the cleaning process: record the data cleaning process and the methods used in detail for subsequent review and verification.
[0023] As a preferred technical solution, in step S3, the principal component analysis method is used to extract features that can reflect the maximum available bandwidth of the transmission path from the cleaned network performance data. The specific method is as follows:
[0024] Step S31: centralize the data and convert it into a data matrix;
[0025] Step S32: Calculate the covariance matrix of the centralized data:
[0026]
[0027] In the formula, X is the centralized data matrix, and n is the number of samples;
[0028] Step S33: performing eigenvalue decomposition on the covariance matrix ∑ to obtain eigenvalues and eigenvectors;
[0029] Step S34: Select the eigenvectors corresponding to the first k largest eigenvalues as principal components;
[0030] Step S35: Project the original data onto the selected principal component to obtain the data after dimensionality reduction.
[0031] As a preferred technical solution, in step S42, for each weight and bias in the convolutional neural network, a small random number is used for initialization, and for each sample in the input data set, the following operations are performed: starting from the input layer, the output of the neuron is calculated layer by layer. The output of each layer of neurons is the product of its input and the weight plus the bias, and then a nonlinear transformation is performed through an activation function; the output of the current layer is used as the input of the next layer, and continues to propagate forward until it reaches the output layer; the result of the output layer is the network's predicted value for the current input sample.
[0032] As a preferred technical solution, in step S6, the document uses encoding technology to compress the document and construct a binary tree to encode the characters. Characters with high frequency use short codes, and characters with low frequency use long codes. Specifically, the average code length of the file encoding is calculated first: Calculate the source entropy for a given character set: In the formula, p i is the probability of the i-th character appearing, l i is the length of the character, and n represents the total number of characters in the character set;
[0033] The efficiency of document encoding is:
[0034] As a preferred technical solution, in step S7, the trained deep learning model outputs the maximum available bandwidth of the path and calculates whether the bandwidth is congested; when the congestion level exceeds a certain threshold, the receiver returns ACK, and the sender adopts a low-priority data retransmission scheme. When the congestion level is less than the threshold, there is no need to retransmit data, and the receiver does not return ACK; for data determined to be lost, the sender adopts a high-priority data transmission scheme.
[0035] The present invention has the following beneficial effects:
[0036] The present invention trains a model through a deep learning algorithm, uses the model to determine the maximum available bandwidth of a transmission path, compresses the document, uses the maximum available bandwidth to transmit the document and calculates the congestion level of the maximum bandwidth, thereby reducing the broadband cost of document transmission and improving the transmission rate.
[0037] Of course, any product implementing the present invention does not necessarily need to achieve all of the advantages described above at the same time. BRIEF DESCRIPTION OF THE DRAWINGS
[0038] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for describing the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.
[0039] Figure 1 This is a flow chart of a digital transmission method for deep learning of documents according to the present invention. DETAILED DESCRIPTION
[0040] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0041] In addition, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.
[0042] In order to make the purpose, technical solution and advantages of this application clearer, the following Figure 1 The implementation methods of the present application are described in further detail.
[0043] Before introducing the embodiments of the present application, the deep learning technology is first described.
[0044] Deep learning is an artificial intelligence technology that allows computers to learn complex patterns and regularities from large amounts of data, thereby achieving various intelligent tasks such as document recognition, speech recognition, natural language processing, machine translation, etc.
[0045] The core of deep learning is the neural network, which is a mathematical model that imitates the structure and function of neurons in the human brain. The neural network consists of multiple layers of neurons, each of which can receive input signals from the previous layer, and after certain calculations and activation functions, it generates output signals and passes them to the next layer.
[0046] Deep learning refers to learning using a neural network with multiple hidden layers, which increases the network's expressiveness and generalization capabilities, allowing the network to learn deeper and more complex features. Deep learning requires a large amount of data and computing resources to train the parameters in the network, which determine how the network establishes a mapping relationship between input and output. Deep learning continuously adjusts parameters through optimization algorithms and back-propagation mechanisms, allowing the network to achieve optimal performance on a given task.
[0047] The working process of deep learning can be divided into two stages: training and inference.
[0048] Training refers to using a large amount of labeled data to train the parameters of a deep neural network so that the network can achieve the best performance on a given task. Labeled data refers to data that has given the correct answer or expected output, such as documents and labels in document recognition tasks, and speech and text in speech recognition tasks.
[0049] The goal of training is to make the output of the network as close as possible to the answer or expected output of the labeled data. This can be achieved by defining a loss function to measure the gap between the network output and the labeled data, and using an optimization algorithm to minimize the loss function. Optimization algorithms usually use gradient descent or its variants, which gradually reduce the value of the loss function by calculating the gradient of the loss function to the parameters and updating the parameters in the opposite direction of the gradient. The calculation of the gradient requires the use of a backpropagation mechanism, which is a method of transferring error signals from the output layer to the input layer layer by layer and calculating the gradient.
[0050] Training deep neural networks requires a lot of data and computing resources, because the network usually contains millions or even billions of parameters, and the data usually contains thousands or even millions of samples. Therefore, training deep neural networks usually requires the use of high-performance computers or cloud servers, and the use of parallel computing and distributed computing to accelerate the training process.
[0051] In addition, training deep neural networks also requires attention to some issues, such as overfitting, underfitting, gradient vanishing, gradient exploding, etc. These problems will affect the learning effect and generalization ability of the network. In order to solve these problems, some technologies and methods have been proposed in the field of deep learning, such as regularization, batch normalization, residual connection, attention mechanism, etc.
[0052] Inference refers to using a trained deep neural network to predict new input data or generate output. Inference is relatively simple compared to training. It only requires inputting the input data into the network and forward propagating according to the network structure and parameters to obtain the output result.
[0053] Inference does not require the use of labeled data or back-propagation mechanisms, so there is no need to update parameters or calculate gradients. Inference also does not require the use of large amounts of data or computing resources, because only a single or small amount of input data needs to be processed, and only one forward propagation is required. Therefore, inference can be performed on lower-performance devices, such as mobile phones, tablets, and embedded devices.
[0054] The goal of reasoning is to make the network output meet the needs of users or application scenarios as much as possible, such as accurately identifying objects in documents, fluently translating languages, and naturally generating text. In order to achieve this goal, some issues need to be paid attention to during reasoning, such as the interpretability, credibility, and diversity of the output results, which will affect the user or application scenario's acceptance and satisfaction with the network output results. In order to solve these problems, some technologies and methods have been proposed in the field of deep learning, such as visualization, uncertainty estimation, and generative models.
[0055] In order to make the purpose, technical solutions and advantages of this application more clear, the following Figure 1 It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.
[0056] See also Figure 1 As shown, the present invention is a digital transmission method for deep learning of documents, comprising the following steps:
[0057] Step S1, collecting network performance data: collecting a large amount of data about network performance, including: bandwidth usage, delay data and packet loss rate data, etc. These data can be obtained through network monitoring tools or special measurement tools (such as iperf3);
[0058] Step S2, data cleaning: clean the collected data, remove outliers and noise, and perform standardization to facilitate subsequent model training;
[0059] Step S3, feature extraction: extracting features that can reflect the maximum available bandwidth of the transmission path from the cleaned network performance data, specifically including: minimum bandwidth, average bandwidth and delay jitter on the link; the most influential features for predicting the maximum available bandwidth of the transmission path can be screened out through a feature selection algorithm to improve the training efficiency and prediction accuracy of the model;
[0060] Step S4, deep learning model training: select a convolutional neural network to train the extracted feature data set; select a suitable deep learning model, such as a convolutional neural network (CNN), a recurrent neural network (RNN) or a long short-term memory network (LSTM), etc., according to the characteristics of the problem and the characteristics of the data. At the same time, during the model training process, the prediction accuracy of the model is improved by adjusting the model parameters and structure. During the training process, methods such as cross-validation can be used to evaluate the performance of the model;
[0061] Step S5, model verification: Use the test set to verify the trained deep learning model and evaluate the prediction accuracy and generalization ability of the model; if the model performs poorly, you can try to adjust the model parameters, increase the training data, or replace it with a more suitable model; for the poorly performing model, according to the verification results, you can further optimize the model, such as adjusting the learning rate, adding regularization terms, etc., to improve the performance and stability of the model;
[0062] Deploy the trained deep learning model to the actual network environment to monitor and predict the maximum available bandwidth of the transmission path in real time. Based on the prediction results, dynamically adjust the allocation and routing strategies of network resources to improve the overall performance of the network and user experience;
[0063] Step S6, document transmission: using encoding technology to compress the document to be transmitted;
[0064] Step S7, transmission identification: input the trained deep learning model to determine the maximum available bandwidth of the transmission path;
[0065] Step S8: Output display: The deep learning model outputs and displays the decompressed document.
[0066] In step S2, the specific process of data cleaning of network performance data is as follows:
[0067] Step S21, deleting data beyond the time range: determining the time range of the data, and deleting the data beyond the time range;
[0068] Step S22, processing duplicate data: identifying and deleting duplicate data records;
[0069] Step S23, processing missing data: for missing data, fill, interpolate or delete according to actual conditions;
[0070] Step S24, filtering outliers: using statistical methods to identify and filter out outliers;
[0071] After the cleaning is completed, it is necessary to apply data transformation: transform the data as needed, such as logarithmic transformation, square root transformation, etc., to make the data more consistent with the assumptions of the analysis model; verify data quality: after the cleaning is completed, perform a quality check on the data to ensure the accuracy and completeness of the data; back up the original data: before performing any data cleaning operation, be sure to back up the original data to prevent accidental loss or error; record the cleaning process: record the data cleaning process and the methods used in detail for subsequent review and verification.
[0072] In step S3, the principal component analysis method is used to extract features that can reflect the maximum available bandwidth of the transmission path from the cleaned network performance data. The specific method is as follows:
[0073] Step S31: centralize the data and convert it into a data matrix;
[0074] Step S32: Calculate the covariance matrix of the centralized data:
[0075]
[0076] In the formula, X is the centralized data matrix, and n is the number of samples;
[0077] Step S33: performing eigenvalue decomposition on the covariance matrix ∑ to obtain eigenvalues and eigenvectors;
[0078] Step S34: Select the eigenvectors corresponding to the first k largest eigenvalues as principal components;
[0079] Step S35: Project the original data onto the selected principal component to obtain the data after dimensionality reduction.
[0080] In step S32, when calculating the covariance matrix, for a data set, it is first necessary to calculate its covariance matrix. The covariance matrix is a symmetric matrix, in which each element represents the covariance between two variables.
[0081] If an m×n data matrix X, where each row represents a sample and each column represents a feature. The covariance matrix C can be obtained by X T X / (n-1) is calculated.
[0082] Therefore, when performing eigenvalue decomposition, eigenvalue decomposition is performed on the calculated covariance matrix C. Eigenvalue decomposition is the process of decomposing a square matrix into the product of an eigenvector and an eigenvalue.
[0083] Specifically, if there exists a non-zero vector v and a real number λ such that Cv=λv, then v is the eigenvector of C and λ is the eigenvalue of C.
[0084] In step S42, for each weight and bias in the convolutional neural network, a small random number is used for initialization, and for each sample in the input data set, the following operations are performed: starting from the input layer, the output of the neuron is calculated layer by layer. The output of each layer of neurons is the product of its input and the weight plus the bias, and then a nonlinear transformation is performed through the activation function; the output of the current layer is used as the input of the next layer and continues to propagate forward until the output layer is reached; the result of the output layer is the network's predicted value for the current input sample. Back propagation starts from the output layer and calculates the gradient of the loss function with respect to each weight and bias layer by layer. This process involves the application of the chain rule, that is, calculating the gradient by the derivative of the composite function. Specifically, for each layer, the gradient of the loss function with respect to the output of the layer (also called the error term) is first calculated, and then this error term is used to calculate the gradient of the loss function with respect to the weights and biases of the layer. These gradients are accumulated so that the weights and biases can be updated in subsequent steps.
[0085] In step S6, the document uses encoding technology to compress the document and construct a binary tree to encode the characters. Characters with high frequency use short codes, and characters with low frequency use long codes. Specifically, the average code length of the file encoding is calculated first: Calculate the source entropy for a given character set: In the formula, p i is the probability of the i-th character appearing, l i is the length of the character, and n represents the total number of characters in the character set;
[0086] The efficiency of document encoding is: When the coding efficiency is close to 1, it means that Huffman coding works very well because it is close to the theoretically optimal compression rate.
[0087] It is worth noting that in the above system embodiment, the various units included are only divided according to functional logic, but are not limited to the above division, as long as the corresponding functions can be achieved; in addition, the specific names of the functional units are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of the present invention.
[0088] In addition, those skilled in the art can understand that all or part of the steps in the above-mentioned embodiments can be completed by instructing related hardware through a program, and the corresponding program can be stored in a computer-readable storage medium.
[0089] The preferred embodiments of the present invention disclosed above are only used to help illustrate the present invention. The preferred embodiments do not describe all the details in detail, nor do they limit the invention to the specific implementation methods described. Obviously, many modifications and changes can be made according to the content of this specification. This specification selects and specifically describes these embodiments in order to better explain the principles and practical applications of the present invention, so that those skilled in the art can understand and use the present invention well. The present invention is limited only by the claims and their full scope and equivalents.
Claims
1. A digital transmission method for deep learning of documents, characterized in that: The steps include: Step S1, collecting network performance data: collecting a large amount of data about network performance, including: bandwidth usage, delay data and packet loss rate data; Step S2, data cleaning: clean the collected data to remove outliers and noise; Step S3, feature extraction: extracting features that can reflect the maximum available bandwidth of the transmission path from the cleaned network performance data, specifically including: minimum bandwidth, average bandwidth and delay jitter on the link; Step S4, deep learning model training: select a convolutional neural network to train the extracted feature data set; Step S5, model verification: Use the test set to verify the trained deep learning model; Step S6, document transmission: using encoding technology to compress the document to be transmitted; Step S7, transmission identification: input the trained deep learning model to determine the maximum available bandwidth of the transmission path; Step S8: Output display: The deep learning model outputs and displays the decompressed document.
2. A digital transmission method for deep learning of documents according to claim 1, characterized in that: In step S1, network performance data is collected by using a network monitoring tool to obtain in real time or by deploying hardware or software probes at key nodes in the network to capture and analyze the network performance data of the nodes.
3. A digital transmission method for deep learning of documents according to claim 1, characterized in that: In step S2, the specific process of data cleaning of network performance data is as follows: Step S21, deleting data beyond the time range: determining the time range of the data, and deleting the data beyond the time range; Step S22, processing duplicate data: identifying and deleting duplicate data records; Step S23, processing missing data: for missing data, fill, interpolate or delete according to actual conditions; Step S24, filtering outliers: using statistical methods to identify and filter out outliers.
4. A digital transmission method for deep learning of documents according to claim 1, characterized in that: In step S3, the principal component analysis method is used to extract features that can reflect the maximum available bandwidth of the transmission path from the cleaned network performance data. The specific method is as follows: Step S31: centralize the data and convert it into a data matrix; Step S32: Calculate the covariance matrix of the centralized data: In the formula, X is the centralized data matrix, and n is the number of samples; Step S33: performing eigenvalue decomposition on the covariance matrix ∑ to obtain eigenvalues and eigenvectors; Step S34: Select the eigenvectors corresponding to the first k largest eigenvalues as principal components; Step S35: Project the original data onto the selected principal component to obtain the data after dimensionality reduction.
5. A digital transmission method for deep learning of documents according to claim 1, characterized in that: In step S4, the specific process of deep learning model training is as follows: Step S41: setting the deep learning model to a convolutional neural network model, the convolutional layer and activation function of the convolutional neural network model are both 20 layers, and a regularized linear unit function is used as the activation function; Step S42: Select the data pair (x, r) as the training set, input the data pair (x, r) into the convolutional neural network model, and obtain the objective function: In the formula, f(w,b,x) is the prediction result of the neural network model, and w and b are both network model parameters; Step S43: using the stochastic gradient descent method to solve the parameters w and b of the neural network model, in each iteration process, calculating the prediction result error and back-propagating it to the convolutional neural network model, calculating the gradient and updating the parameters of the convolutional neural network model; Step S44: Generate a mapping relationship from a low-resolution image to a high-resolution image based on the model parameters of each convolutional layer in the convolutional neural network model and complete the training of the neural network model.
6. A digital transmission method for deep learning of documents according to claim 5, characterized in that: In step S42, each weight and bias in the convolutional neural network is initialized with a small random number, and for each sample in the input data set, the following operations are performed: starting from the input layer, the output of the neuron is calculated layer by layer; the output of each layer of neurons is the product of its input and the weight plus the bias, and then a nonlinear transformation is performed through an activation function; The output of the current layer is used as the input of the next layer and continues to propagate forward until it reaches the output layer; the result of the output layer is the network's predicted value for the current input sample.
7. A digital transmission method for deep learning of documents according to claim 1, characterized in that: In step S6, the document uses encoding technology to compress the document and construct a binary tree to encode the characters. Characters with high frequency use short codes, and characters with low frequency use long codes. Specifically, the average code length of the file encoding is calculated first: Calculate the source entropy for a given character set: In the formula, p i is the probability of the i-th character appearing, l i is the length of the character, and n represents the total number of characters in the character set; The efficiency of document encoding is:
8. A digital transmission method for deep learning of documents according to claim 1, characterized in that: In step S7, the trained deep learning model outputs the maximum available bandwidth of the path and calculates whether the bandwidth is congested; when the congestion level exceeds a certain threshold, the receiver returns ACK, and the sender adopts a low-priority data retransmission scheme. When the congestion level is less than the threshold, there is no need to retransmit data, and the receiver does not return ACK; for data determined to be lost, the sender adopts a high-priority data transmission scheme.