Data transmission method and device, electronic equipment and storage medium

Abstract

The invention relates to the technical field of computers, and provides a data transmission method and device, electronic equipment and a storage medium. The method comprises the steps of obtaining to-be-transmitted data, and performing importance grade division on the to-be-transmitted data through a target classification model to obtain importance grades of the to-be-transmitted data; selecting a corresponding encryption algorithm based on the importance level of the to-be-transmitted data, and encrypting the to-be-transmitted data based on the encryption algorithm corresponding to the to-be-transmitted data to obtain encrypted data corresponding to the to-be-transmitted data; and determining a to-be-transmitted channel according to the importance level of the to-be-transmitted data, and transmitting the encrypted data to a receiving end through the to-be-transmitted channel. The problem of low data transmission efficiency in the prior art is solved.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a data transmission method, device, electronic device and storage medium. Background Art

[0002] A network gatekeeper is a network security device that is mainly used to establish a secure isolation between two networks to prevent unauthorized access and data leakage. It can be used in various industries and institutions to keep data confidential. For example, a network gatekeeper can prevent external networks from directly accessing internal networks; filter data passing through to prevent the transmission of sensitive information and malicious code; control data selectively passing through; record data and access behavior passing through the network gatekeeper; encrypt transmitted data, etc.

[0003] However, in the prior art, when implementing data transmission through a network gate, a transparent transmission method is usually adopted, which means that all data are transmitted in the original form, which results in low efficiency in data transmission. Summary of the invention

[0004] In view of this, the present application provides a data transmission method, device, electronic device and storage medium to solve the problem of low data transmission efficiency in the prior art.

[0005] In a first aspect of the present application, a data transmission method is provided, which is applied to a sending end, including: obtaining data to be transmitted, dividing the data to be transmitted into importance levels through a target classification model, and obtaining the importance level of the data to be transmitted; selecting a corresponding encryption algorithm based on the importance level of the data to be transmitted, and encrypting the data to be transmitted based on the encryption algorithm corresponding to the data to be transmitted, and obtaining encrypted data corresponding to the data to be transmitted; determining a transmission channel according to the importance level of the data to be transmitted, and transmitting the encrypted data to a receiving end through the transmission channel.

[0006] The second aspect of the present application provides a data transmission method, which is applied to a receiving end, including: receiving encrypted data, decrypting the encrypted data, and obtaining decrypted data; responding to a consumption request from a client, transmitting the decrypted data corresponding to the consumption request to the client for consumption, deleting the decrypted data corresponding to the consumption request, and storing the unconsumed decrypted data in a target database.

[0007] According to a third aspect of the present application, a data transmission device is provided, which is applied to a sending end and includes: an acquisition module, configured to acquire data to be transmitted, and classify the data to be transmitted into importance levels through a target classification model to obtain the importance level of the data to be transmitted; an encryption module, configured to select a corresponding encryption algorithm based on the importance level of the data to be transmitted, and encrypt the data to be transmitted based on the encryption algorithm corresponding to the data to be transmitted to obtain encrypted data corresponding to the data to be transmitted; and a transmission module, configured to determine a channel to be transmitted according to the importance level of the data to be transmitted, and transmit the encrypted data to a receiving end through the channel to be transmitted.

[0008] The fourth aspect of the present application provides a data transmission device, which is applied to a receiving end, including: a decryption module, which receives encrypted data, decrypts the encrypted data, and obtains decrypted data; a forwarding module, which responds to a consumption request from a client, transmits the decrypted data corresponding to the consumption request to the client for consumption, deletes the decrypted data corresponding to the consumption request, and stores the unconsumed decrypted data in a target database.

[0009] In a fifth aspect of the present application, an electronic device is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above method when executing the computer program.

[0010] In a sixth aspect of the present application, a storage medium is provided, which stores a computer program, and when the computer program is executed by a processor, the steps of the above method are implemented.

[0011] At least one of the above technical solutions adopted in this application can achieve the following beneficial effects:

[0012] The data to be transmitted is divided into importance levels through the target classification model of the sending end to obtain the importance level of the data to be transmitted, and the corresponding encryption algorithm is further selected based on the importance equivalence of the data to be transmitted, and the corresponding data to be transmitted is encrypted to obtain encrypted data, and the data transmission channel to be transmitted is determined according to the importance level of the data to be transmitted, and the encrypted data of the corresponding importance level is transmitted to the receiving end through the transmission channel to be transmitted, so that the data to be transmitted can be graded according to importance, and different transmission priorities and encryption security measures can be set, so that the data to be transmitted with high importance level can be transmitted through high priority and encryption channels, and different transmission channels are set for data to be transmitted with different importance levels to achieve logical distinction of the data to be transmitted, and the data to be transmitted with different importance levels can be transmitted through different transmission channels to improve the transmission efficiency and security of the data to be transmitted. And after the receiving end receives the encrypted data sent by the sending end, the encrypted data can be decrypted and stored, and deleted and stored according to the consumption of the restored data to be transmitted by the client, so as to further achieve the security of the application of the data to be transmitted after data transmission. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0014] Figure 1 It is a flowchart of a data transmission method provided by an embodiment of the present application;

[0015] Figure 2 It is a flowchart of another data transmission method provided in an embodiment of the present application;

[0016] Figure 3 is a flowchart of another data transmission method provided in an embodiment of the present application;

[0017] Figure 4 It is a structural schematic diagram of a data transmission device provided in an embodiment of the present application;

[0018] Figure 5 is a structural schematic diagram of another data transmission device provided in an embodiment of the present application;

[0019] Figure 6 It is a structural schematic diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0020] In the following description, specific details such as specific system structures, technologies, etc. are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present application. However, it should be clear to those skilled in the art that the present application may also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to prevent unnecessary details from obstructing the description of the present application.

[0021] The terms "first", "second", etc. in the specification and claims of the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable under appropriate circumstances, so that the embodiments of the present application can be implemented in an order other than those illustrated or described here, and the objects distinguished by "first", "second", etc. are generally of one type, and the number of objects is not limited. For example, the first object can be one or more. In addition, "and / or" in the specification and claims represents at least one of the connected objects, and the character " / " generally indicates that the objects associated with each other are in an "or" relationship.

[0022] In addition, it should be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "includes..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0023] In the embodiments of the present application, a network gate represents a security device for isolating and protecting a network, which can be deployed between different networks (such as an intranet and an extranet) to control and filter data transmission, ensure network security and data integrity, and a network gate can usually be set with two network interfaces, one connected to the intranet and the other connected to the extranet, to physically isolate the internal and external networks through the network gate, and to check the data entering and leaving the intranet. A network gate can be applied to various scenarios that require secure processing of data transmission, such as applying a network gate between an enterprise's internal network and the Internet to protect the enterprise's internal network from external threats and ensure secure data transmission; applying a network gate in the financial industry to ensure the security of customer information and transaction data, etc.

[0024] Although the network firewall can isolate the internal and external networks, it can also be used to transmit data, such as transmitting data between the internal network and the Internet, or transmitting data between people of different levels within the enterprise. In order to ensure the security of the data transmission process and prevent the data from being eavesdropped or tampered with, the existing technology has set up certain strategies to protect the security of data transmission. However, the existing technology transparently transmits the real-time data of the network firewall without making any logical distinction based on the importance of the data. Therefore, the present application proposes a data transmission method that distinguishes the data by importance level and transmits it logically in isolation.

[0025] It should also be understood that the specific data transmitted through the network gate may include various types of data, such as documents, images, videos, software installation packages, etc., and may also be system logs, security audit records and other data. The embodiments of the present application do not limit the data types.

[0026] A data transmission method and device according to an embodiment of the present application will be described in detail below with reference to the accompanying drawings.

[0027] Figure 1 It is a flow chart of a data transmission method provided in an embodiment of the present application. Figure 1 The data transmission method is executed by the sender. Figure 1 As shown, the data transmission method includes:

[0028] S101, obtaining data to be transmitted, and classifying the data to be transmitted into importance levels using a target classification model to obtain importance levels of the data to be transmitted;

[0029] S102, selecting a corresponding encryption algorithm based on the importance level of the data to be transmitted, and encrypting the data to be transmitted based on the encryption algorithm corresponding to the data to be transmitted, to obtain encrypted data corresponding to the data to be transmitted;

[0030] S103, determining a transmission channel according to the importance level of the data to be transmitted, and transmitting the encrypted data to the receiving end through the transmission channel.

[0031] Specifically, first, the data to be transmitted can be obtained. These data can be any information that needs to be transmitted in the network, which will not be expanded here. Then, the target classification model is used to classify the importance level of the data to be transmitted to obtain the importance level of the data to be transmitted, where the importance level can be: ordinary, secret, confidential, and top secret, which represent more important data in turn. For example, top secret can represent the most important data in the pre-set importance level, and the ordinary importance level can be used to represent general information, which can be made public or only used internally. In this way, the data can be classified into importance levels to intelligently process the priority of data transmission, and more important data can be transmitted first to improve the efficiency of data transmission.

[0032] Furthermore, a corresponding encryption algorithm can be selected based on the importance level of the data to be transmitted, where the encryption algorithm can be various encryption algorithms pre-set for the data to be transmitted, and a stronger encryption algorithm is selected to encrypt the data to be transmitted with a higher importance level. In this way, the encryption algorithms of the data to be transmitted are matched one by one, and the data to be transmitted with the corresponding importance level is encrypted to obtain encrypted data.

[0033] Furthermore, the transmission channel can be determined based on the importance level of the data to be transmitted, and the encrypted data of a specific importance level can be transmitted through the transmission channel of a specific level. It should also be understood that the encrypted data is obtained by encrypting the data to be transmitted, so the data to be transmitted is consistent with the corresponding encrypted data, and the transmission channel determined based on the importance level of the data to be transmitted is the transmission channel corresponding to the importance level of the encrypted data, so that the encrypted data can be logically isolated and transmitted through the transmission channel, wherein the transmission channel can be a channel of a specific security level, and the corresponding transmission channel of a high importance level has high security, such as a dedicated line, a virtual private network (VPN), etc. Encrypted data of the same importance level can be transmitted through the same transmission channel.

[0034] According to the method of the above embodiment, by dividing the data into importance levels, selecting the encryption algorithm, and determining the channel to be transmitted, the data to be transmitted can be intelligently classified to achieve targeted security processing, thereby ensuring the security of the encrypted data transmission process. By transmitting encrypted data of different importance levels through the corresponding channel to be transmitted, priority transmission of data can be achieved, thereby improving the efficiency of data transmission.

[0035] In some embodiments, before the importance level of the data to be transmitted is classified by the target classification model to obtain the importance level of the data to be transmitted, it also includes: collecting training data, the training data includes training features and the importance levels corresponding to the training features, and the importance levels corresponding to the training data are determined in response to manual annotations; cleaning the training data to obtain cleaned training data, and dividing the cleaned training data into a training set, a validation set and a test set; using the training set to train a preset classification model to obtain an initial classification model; using the validation set and the test set to evaluate the initial classification model to obtain a target classification model.

[0036] Specifically, a target classification model can be trained to determine the importance of the data to be transmitted. A classification model architecture can be pre-selected, such as selecting a random forest as a preset classification model, and training it, that is, by building multiple decision trees and averaging or voting on them to improve the accuracy and generalization ability of the random forest. Before training it, the training data can be obtained and cleaned to be divided into a training set, a validation set, and a test set. The cleaning process may include processing missing values, outliers, and erroneous data, which will not be expanded here.

[0037] In order to train the target classification model, the training data may include characteristic information of the data (for example, data type, content, source, etc.) and the corresponding importance level. The importance level can be determined by manual annotation to ensure the accuracy of the training data label. The importance levels may include ordinary, secret, confidential, and top secret. The importance levels of the corresponding training data may be annotated by professionals based on their professional knowledge and experience. For example, the data may be first classified according to factors such as its nature, purpose, and sensitivity to obtain initial classifications of ordinary data, secret data, confidential data, and top secret data. Risk assessments may then be conducted on each of them to assess the potential risks and impacts of data leakage or tampering, and then the importance level of the data may be comprehensively evaluated based on user needs and data protection strategies.

[0038] The preset classification model is trained using the training set. The training process may include adjusting the model parameters so that the classification model can accurately predict the importance level of the data based on the input features. After the preset classification model is trained using the training set, an initial classification model is obtained. The initial classification model can be further evaluated using the validation set to observe the performance of the initial classification model. The parameters and structure of the initial classification model are adjusted based on the validation results to further optimize the model. Finally, the optimized model can be finally evaluated using the test set to ensure that the model can perform well on unseen data, increase the robustness and generalization ability of the classification model, and obtain the target classification model.

[0039] According to the method of the above embodiment, by acquiring training data, cleaning and dividing the training data to obtain a training set, a validation set, and a test set, the preset classification model is trained, and finally a target classification model is obtained. In this way, the target classification model can be used to accurately divide the importance level of the transmitted data, providing a solid foundation for the subsequent encryption and transmission process.

[0040] In some embodiments, the data to be transmitted includes at least two sub-transmission data, and each sub-transmission data corresponds to a different importance level. The data to be transmitted is classified into importance levels through a target classification model to obtain the importance level of the data to be transmitted, including: classifying each sub-transmission data in the data to be transmitted into importance levels to obtain the importance level of each sub-transmission data; then, compressing the sub-transmission data of the same importance level to obtain compressed data corresponding to each importance level, and encrypting the compressed data corresponding to each importance level based on the encryption algorithm corresponding to each importance level to obtain encrypted data corresponding to each importance level.

[0041] Specifically, the data to be transmitted may include multiple sub-transmission data, each of which may represent data of different importance levels. The sub-transmission data may judge the importance level of each data to be transmitted based on the target classification model, and obtain the importance level of each data to be transmitted, and then classify the data to be transmitted of the same importance level into one sub-transmission data. After obtaining the importance level of the sub-transmission data, the sub-transmission data of different importance levels may be compressed based on the importance level of the sub-transmission data to be transmitted, and multiple compressed data of the same importance level, i.e., compressed data of each importance level, may be obtained. The compression process may use a lossless compression algorithm, so that the integrity of the data to be transmitted is maintained while reducing the data volume. In this way, after the sub-transmission data of the same importance level are compressed separately, different compressed data corresponding to different importance levels may be obtained, and further, the data to be transmitted may be logically isolated based on the importance level, and multi-channel transmission may be realized, thereby improving data security and transmission efficiency.

[0042] Before compressing the sub-transmission data, each sub-transmission data can also be serialized using a preset method, where the preset method can be Google's Protocol Buffers (Protobuf) method, which converts the data to be transmitted into a binary format (i.e., the target format). The binary format of Protobuf can make the data to be transmitted more compact, thereby improving the transmission efficiency.

[0043] Of course, before the data to be transmitted is transmitted, it can also be encrypted, that is, different encryption algorithms are selected to encrypt the sub-transmission data of different importance levels respectively to obtain encrypted data. The encryption algorithm can be a national secret encryption algorithm. A high-intensity encryption algorithm is used for the data to be transmitted with a high importance level, and a weaker encryption algorithm is used for the data to be transmitted with a low importance level. Of course, in order to reduce the system load, for data that can be made public, you can choose not to use an encryption algorithm for encryption. Of course, in order to avoid the disclosure of important information due to incorrect classification of importance, all data can also be encrypted. No specific restrictions are made here.

[0044] According to the method of the above embodiment, by serializing, compressing and encrypting the sub-transmission data, each sub-transmission data can be made more compact, the data volume can be reduced to improve the transmission speed, and the data confidentiality can be increased to prevent unauthorized access, thereby further achieving the security and efficiency of data transmission. In this way, the data transmission system can improve the transmission efficiency while ensuring the confidentiality and integrity of the data, and meet the security requirements of different importance levels.

[0045] In some embodiments, the compressed data corresponding to each importance level is encrypted based on the encryption algorithm corresponding to each importance level to obtain encrypted data corresponding to each importance level, including: selecting a target encryption algorithm from a preset encryption algorithm set according to the importance level of the sub-transmission data; based on the target encryption algorithm, encrypting the compressed data corresponding to the sub-transmission data to obtain encrypted data, and the security level of the target encryption algorithm corresponds to the importance level of the sub-transmission data.

[0046] Specifically, the corresponding encryption algorithm can be selected based on the importance level of the sub-transmission data. A high-level encryption algorithm can be applied to encrypt sub-transmission data of a high importance level to ensure the security of the data. The corresponding encryption algorithm (i.e., the target encryption algorithm) can be selected from a preset encryption algorithm set, which can be a national encryption algorithm library or other encryption algorithm libraries based on personalized settings.

[0047] As an example, sub-transmission data is divided into four levels (i.e., normal, secret, confidential, and top secret). If the importance level of the sub-transmission data is normal, the target encryption algorithm with the lowest security level can be selected for encryption. If the importance level of the sub-transmission data is top secret, the target encryption algorithm with the highest security level can be selected for encryption.

[0048] Furthermore, the encryption algorithms in the preset encryption algorithm set may include: SM1 (symmetric encryption algorithm), SM2 (asymmetric encryption algorithm), SM3 (message digest algorithm), SM4 (symmetric block cipher algorithm), SM9 (identity-based encryption), etc. The SM1 algorithm can be used for undisclosed specific details, the SM2 algorithm can represent the public key cryptography of the elliptic curve, which is used for digital signatures, key exchange and data encryption mechanisms, SM3 can represent the encryption mechanism for generating a fixed-length hash value (256 bits) for data integrity verification and digital signatures, SM4 can represent a group algorithm for wireless LAN products, an encryption mechanism for data encryption transmission and file encryption, SM9 can represent an encryption mechanism that combines identity information to generate a company key pair to simplify the management of the public key infrastructure, which is used for identity authentication, key exchange and data encryption.

[0049] After determining the preset encryption algorithm set, the target encryption algorithm can be selected based on the importance level of the sub-transmission data, such as: the data to be transmitted at the ordinary level can use the basic encryption algorithm, such as SM4 to encrypt and decrypt the data to be transmitted at the ordinary level to provide basic confidentiality protection; the data to be transmitted at the secret level can choose the medium-strength encryption algorithm, such as SM4 and SM3, encrypt and decrypt with SM4, and verify the data integrity through SM3 to ensure that the data has not been tampered with during the transmission process; the data to be transmitted at the confidential level can use a stronger encryption algorithm, such as SM2 combined with SM4 and SM3, encrypt through SM4, exchange keys through SM2 to ensure the security of the encryption key, and verify the integrity of the data through SM3; the data to be transmitted at the top secret level can also be encrypted using SM2 combined with SM4 and SM3, encrypted through SM4, exchange keys and digitally sign through SM2 to ensure the source and non-repudiation of the data, and ensure the data integrity and generate digital signatures through SM3 to ensure that the data has not been tampered with. Of course, other more suitable encryption methods can be selected based on the type of data to be transmitted and the encryption requirements, and this embodiment does not limit it.

[0050] According to the method of the above embodiment, by applying different encryption algorithms to sub-transmission data of different importance levels, it can be ensured that the data can be effectively protected under different security requirements, so that the data can be safely transmitted through the transmission channel.

[0051] In some embodiments, determining the channel to be transmitted according to the importance level of the data to be transmitted includes: selecting the channel to be transmitted corresponding to the importance level of the data to be transmitted from a pre-set mapping table according to the importance level of the data to be transmitted, and the mapping table is used to associate the mapping relationship between the importance level of the data to be transmitted and the corresponding channel to be transmitted.

[0052] Specifically, after the data is encrypted, a corresponding transmission channel can be selected to transmit the corresponding encrypted data. The determination of the transmission channel can be mapped based on a pre-set mapping table. That is, a mapping table can be pre-set to associate the importance level of the data to be transmitted with a specific transmission channel level (such as a network channel with different security levels), and the mapping table can be cached in the memory, thereby facilitating the mapping relationship between the data to be transmitted of each importance level and the corresponding transmission channel through the mapping table.

[0053] As an example, the contents of the mapping table may include: ordinary level - public network; secret level - encrypted VPN channel; confidential level - high security VPN channel; top secret level - dedicated encrypted channel. In this way, the corresponding transmission channel can be selected according to the importance level of the data to be transmitted.

[0054] According to the method of the above embodiment, by selecting the corresponding channel to be transmitted according to the different importance levels of the data to be transmitted, the security and confidentiality of the data during the transmission process can be ensured. The pre-set mapping table can also be used to realize the automatic selection of the channel to be transmitted, reduce the errors in human judgment, and improve the data transmission efficiency. In addition, the preset mapping table can be dynamically adjusted according to the corresponding demand changes, which is convenient for management.

[0055] Figure 2 It is a flowchart of another data transmission method provided in an embodiment of the present application. Figure 2 The data transmission method is executed by the receiving end. Figure 2 As shown, the data transmission method includes:

[0056] S201, receiving encrypted data, decrypting the encrypted data to obtain decrypted data;

[0057] S202, in response to a consumption request from the client, transmitting the decrypted data corresponding to the consumption request to the client for consumption, deleting the decrypted data corresponding to the consumption request, and storing the unconsumed decrypted data in the target database.

[0058] Specifically, after the receiving end receives the encrypted data, the encrypted data can be decrypted first. The decryption process is determined based on the corresponding encryption algorithm. For example, if SM4 is used to encrypt the data at the sending end, the same secret key is used to decrypt the encrypted data to obtain the decrypted data. This process can be executed by the server at the receiving end.

[0059] It should be understood that when the data to be transmitted is transmitted, it is compressed and serialized in order to reduce the amount of data. Therefore, after decryption at the receiving end, the decrypted data can also be decompressed and deserialized to restore the data to its original structure for further processing.

[0060] Furthermore, it can be determined whether the client has sent a consumption request. If the client sends a consumption request, the decrypted data corresponding to the consumption request will be sent to the client for processing, and the consumed decrypted data will be deleted based on the processing result, and the unconsumed decrypted data will be stored in the target database. The target database can be various types of databases, depending on the application requirements, data characteristics, performance requirements, etc., such as relational databases, NoSQL databases, etc., which are not limited here.

[0061] Of course, the decrypted data can also be directly sent to the target database for storage. When the database receives a consumption request from the client, the decrypted data requested by the client is sent to the client for consumption. If no consumption request is received from the client, the target database is kept for storage. Similarly, the consumed data in the database can be deleted after consumption is completed to reduce the accumulation of redundant data.

[0062] According to the method of the above embodiment, by encrypting and forwarding the encrypted data at the receiving end and processing the consumption needs of the client, the security and integrity of the data during the data processing and the efficient operation of the system can be ensured.

[0063] In some embodiments, deleting the decrypted data corresponding to the consumption request includes: deleting the decrypted data corresponding to the consumption request in response to confirmation information from the client, wherein the confirmation information indicates that the client has successfully consumed the decrypted data corresponding to the consumption request; if no confirmation information from the client is received, storing the decrypted data corresponding to the consumption request in the target database.

[0064] Specifically, when judging whether the server is to delete the corresponding data, it mainly judges whether the data has been consumed, that is, if the client has successfully consumed it, the corresponding data can be deleted. However, in order to avoid data loss, the corresponding data can be deleted only after the client returns confirmation information, that is, the decrypted data corresponding to the consumption request.

[0065] Among them, the confirmation information can be in the form of a message queue. When the client consumes the message, it can send a confirmation message to notify the server that the message has been successfully processed. It can also be a database update request, that is, the client sends a database update request to the server, marking the corresponding data as "consumed status". The server regularly cleans up the data marked as "consumed". If the client fails in the process of processing the data and does not send a confirmation message, the corresponding information will not be deleted. We will not expand on this here, as long as it is reasonable.

[0066] According to the method of the above embodiment, the client returns confirmation information to confirm that the data has been consumed, and then deletes the corresponding consumed data, which can avoid data loss. Timely deletion of consumed data can also reduce system storage pressure.

[0067] Figure 3 is a flowchart of another data transmission method provided in an embodiment of the present application, combined with Figure 3 The embodiments of the present application are further described.

[0068] During the data transmission process of the network gate, the sender can first encrypt and serialize the data to be transmitted. The encryption algorithm can be SM2, SM4, etc., and then the encrypted data is encrypted and transmitted. The data transmission can be dynamically synchronized from one Kafka cluster to another Kafka cluster through the mapping tool MirrorMaker, so that smooth data migration can be achieved, and the security of the encrypted data transmission process can be further ensured. After the receiving end receives the encrypted data, it filters and forwards it. The filtering can include decryption, decompression, and deserialization of the encrypted data. Forwarding can send the corresponding decrypted data to the client for consumption based on the client's consumption request, or store the unconsumed decrypted data in the target database, such as a relational database (i.e. MySQL), a distributed column storage database (i.e. Star Rocks, Apache Doris), etc.

[0069] It should also be understood that during the data transmission process, the Pod in the container can be processed through the Kubernetes (K8s) container management tool. The Pod is a basic scheduling unit and the smallest deployment unit that can be created and managed in K8s. It can contain one or more closely related containers, and each container shares network and storage resources. The K8s container management tool can be used to dynamically adjust the data transmission rate by expanding and contracting nodes, adjusting network conditions and data priorities, thereby improving data processing capabilities and avoiding resource waste, achieving data load balancing, and improving resource utilization.

[0070] To be more specific, k8s can be used to deploy and manage containerized applications in the intranet and extranet, encapsulate applications in containers, ensure a consistent operating environment, and automatically expand or shrink the number of pods according to the central processing unit (CPU) utilization, memory utilization, or custom indicators (such as data transmission rate), so as to dynamically adjust resource allocation according to actual load. You can also adjust network conditions and data priorities, that is, k8s allows network policies and service quality rules, through which you can adjust network conditions and data priorities, for example, you can allocate more bandwidth or lower latency to critical applications to ensure that other performance is not affected by non-critical tasks, ensure that data transmission tasks are processed first, and improve data transmission efficiency. You can also dynamically adjust the data transmission rate, that is, perform bandwidth management and current limiting, adjust the data transmission rate according to the network conditions, prevent network congestion, and improve the stability and efficiency of data transmission.

[0071] According to the method of the above embodiment, the internal and external networks are isolated by a network firewall. Through the automatic expansion, network optimization, priority management and bandwidth control functions of k8s, the rate of data transmission between the internal and external networks can be dynamically adjusted to improve data processing capabilities while avoiding waste of resources.

[0072] All the above optional technical solutions can be arbitrarily combined to form optional embodiments of the present application, which will not be described one by one here.

[0073] The following is an embodiment of the device of the present application, which can be used to execute the embodiment of the method of the present application. For details not disclosed in the embodiment of the device of the present application, please refer to the embodiment of the method of the present application.

[0074] Figure 4 Schematic diagram of a data transmission device provided in an embodiment of the present application. Figure 4 As shown, the data transmission device includes:

[0075] The acquisition module 401 is configured to acquire the data to be transmitted, classify the data to be transmitted into importance levels by using the target classification model, and obtain the importance level of the data to be transmitted;

[0076] The encryption module 402 is configured to select a corresponding encryption algorithm based on the importance level of the data to be transmitted, and encrypt the data to be transmitted based on the encryption algorithm corresponding to the data to be transmitted to obtain encrypted data corresponding to the data to be transmitted;

[0077] The transmission module 403 is configured to determine a transmission channel according to the importance level of the data to be transmitted, and transmit the encrypted data to the receiving end through the transmission channel.

[0078] In some embodiments, the acquisition module 401 is specifically used to collect training data, the training data includes training features and importance levels corresponding to the training features, and the importance levels corresponding to the training data are determined in response to manual annotations; the training data is cleaned to obtain cleaned training data, and the cleaned training data is divided into a training set, a validation set, and a test set; the training set is used to train a preset classification model to obtain an initial classification model; the validation set and the test set are used to evaluate the initial classification model to obtain a target classification model.

[0079] In some embodiments, the acquisition module 401 is specifically used to classify the importance level of each sub-transmission data in the transmission data to obtain the importance level of each sub-transmission data; then, compress the sub-transmission data of the same importance level to obtain compressed data corresponding to each importance level, and encrypt the compressed data corresponding to each importance level based on the encryption algorithm corresponding to each importance level to obtain encrypted data corresponding to each importance level.

[0080] In some embodiments, the encryption module 402 is specifically used to select a target encryption algorithm from a preset encryption algorithm set according to the importance level of the sub-transmission data; based on the target encryption algorithm, the compressed data corresponding to the sub-transmission data is encrypting to obtain encrypted data, and the security level of the target encryption algorithm corresponds to the importance level of the sub-transmission data.

[0081] In some embodiments, the transmission module 403 is specifically used to select a channel to be transmitted corresponding to the importance level of the data to be transmitted from a pre-set mapping table according to the importance level of the data to be transmitted, and the mapping table is used to associate the mapping relationship between the importance level of the data to be transmitted and the corresponding channel to be transmitted.

[0082] Figure 5 Schematic diagram of another data transmission device provided in an embodiment of the present application. Figure 5 As shown, the data transmission device includes:

[0083] The decryption module 501 receives the encrypted data and decrypts the encrypted data to obtain the decrypted data;

[0084] The forwarding module 502, in response to the consumption request of the client, transmits the decrypted data corresponding to the consumption request to the client for consumption, deletes the decrypted data corresponding to the consumption request, and stores the unconsumed decrypted data to the target database.

[0085] In some embodiments, the forwarding module 502 is specifically used to delete the decrypted data corresponding to the consumption request in response to the client's confirmation information, and the confirmation information indicates that the client has successfully consumed the decrypted data corresponding to the consumption request; if the client's confirmation information is not received, the decrypted data corresponding to the consumption request is stored in the target database.

[0086] It should be understood that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0087] Figure 6 Schematic diagram of an electronic device 6 provided in an embodiment of the present application. Figure 6 As shown, the electronic device 6 of this embodiment includes: a processor 601, a memory 602, and a computer program 603 stored in the memory 602 and executable on the processor 601. When the processor 601 executes the computer program 603, the steps in the above-mentioned method embodiments are implemented. Alternatively, when the processor 601 executes the computer program 603, the functions of the modules / units in the above-mentioned device embodiments are implemented.

[0088] The electronic device 6 may be a desktop computer, a notebook, a PDA, a cloud server, or other electronic device. The electronic device 6 may include, but is not limited to, a processor 601 and a memory 602. Those skilled in the art will appreciate that Figure 6 The electronic device 6 is merely an example and does not limit the electronic device 6 . The electronic device 6 may include more or less components than those shown in the figure, or different components.

[0089] The processor 601 can be a CPU, or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc.

[0090] The memory 602 may be an internal storage unit of the electronic device 6, for example, a hard disk or memory of the electronic device 6. The memory 602 may also be an external storage device of the electronic device 6, for example, a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the electronic device 6. The memory 602 may also include both an internal storage unit of the electronic device 6 and an external storage device. The memory 602 is used to store computer programs and other programs and data required by the electronic device.

[0091] Those skilled in the art can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example. In actual applications, the above-mentioned functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units.

[0092] If the integrated module / unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a storage medium. Based on this understanding, the present application implements all or part of the process in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. The computer program may include computer program code, which may be in source code form, object code form, executable file or some intermediate form. Computer-readable media may include: any entity or device capable of carrying computer program code, recording medium, U disk, mobile hard disk, disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium, etc.

[0093] The above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application, and should all be included in the protection scope of the present application.

Claims

1. A data transmission method, applied to a sending end, characterized in that: include: Acquire data to be transmitted, and classify the data to be transmitted into importance levels using a target classification model to obtain the importance levels of the data to be transmitted; Selecting a corresponding encryption algorithm based on the importance level of the data to be transmitted, and encrypting the data to be transmitted based on the encryption algorithm corresponding to the data to be transmitted to obtain encrypted data corresponding to the data to be transmitted; A transmission channel is determined according to the importance level of the data to be transmitted, and the encrypted data is transmitted to a receiving end through the transmission channel.

2. The method according to claim 1, characterized in that Before classifying the importance level of the data to be transmitted by using the target classification model to obtain the importance level of the data to be transmitted, the method further includes: Collecting training data, the training data including training features and importance levels corresponding to the training features, the importance levels corresponding to the training features being determined in response to manual annotation; Cleaning the training data to obtain cleaned training data, and dividing the cleaned training data into a training set, a validation set, and a test set; Using the training set to train a preset classification model to obtain an initial classification model; The initial classification model is evaluated using the validation set and the test set to obtain the target classification model.

3. The method according to claim 1, characterized in that The data to be transmitted includes at least two sub-transmission data, and each of the sub-transmission data corresponds to a different importance level. The importance level of the data to be transmitted is classified by the target classification model to obtain the importance level of the data to be transmitted, including: Classifying the importance level of each of the sub-transmission data in the data to be transmitted to obtain the importance level of each of the sub-transmission data; after, The sub-transmission data of the same importance level are compressed to obtain compressed data corresponding to each importance level, and the compressed data corresponding to each importance level are encrypted based on the encryption algorithm corresponding to each importance level to obtain encrypted data corresponding to each importance level.

4. The method according to claim 3, characterized in that The encryption algorithm corresponding to each importance level is used to encrypt the compressed data corresponding to each importance level to obtain the encrypted data corresponding to each importance level, including: Selecting a target encryption algorithm from a set of preset encryption algorithms according to the importance level of the sub-transmission data; Based on the target encryption algorithm, the compressed data corresponding to the sub-transmission data is encrypted to obtain the encrypted data, and the security level of the target encryption algorithm corresponds to the importance level of the sub-transmission data.

5. The method according to claim 1, characterized in that The determining of the channel to be transmitted according to the importance level of the data to be transmitted includes: According to the importance level of the data to be transmitted, a channel to be transmitted corresponding to the importance level of the data to be transmitted is selected from a preset mapping table, wherein the mapping table is used to associate a mapping relationship between the importance level of the data to be transmitted and the corresponding channel to be transmitted.

6. A data transmission method, applied to a receiving end, characterized in that: include: Receiving encrypted data, and decrypting the encrypted data to obtain decrypted data; In response to a consumption request from a client, the decrypted data corresponding to the consumption request is transmitted to the client for consumption, the decrypted data corresponding to the consumption request is deleted, and the unconsumed decrypted data is stored in a target database.

7. The method according to claim 6, characterized in that The deleting the decrypted data corresponding to the consumption request includes: In response to confirmation information from the client, deleting the decrypted data corresponding to the consumption request, wherein the confirmation information indicates that the client has successfully consumed the decrypted data corresponding to the consumption request; If the confirmation information from the client is not received, the decrypted data corresponding to the consumption request is stored in the target database.

8. A data transmission device, applied to a sending end, characterized in that: include: An acquisition module is configured to acquire data to be transmitted, and classify the data to be transmitted into importance levels by using a target classification model to obtain the importance levels of the data to be transmitted; an encryption module, configured to select a corresponding encryption algorithm based on the importance level of the data to be transmitted, and encrypt the data to be transmitted based on the encryption algorithm corresponding to the data to be transmitted to obtain encrypted data corresponding to the data to be transmitted; The transmission module is configured to determine a transmission channel according to the importance level of the data to be transmitted, and transmit the encrypted data to a receiving end through the transmission channel.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the processor implements the steps of the method according to any one of claims 1 to 5, or implements the steps of the method according to claim 6 or 7.

10. A storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the computer program implements the steps of the method according to any one of claims 1 to 5, or implements the steps of the method according to claim 6 or 7.

Images