Data transmission method and device, computer equipment, readable storage medium and program product
By filtering and encoding processing based on the frequency of data usage, the efficiency and accuracy of data transmission are achieved, the problem of inefficiency in traditional storage solutions is solved, and it is suitable for large-scale data storage needs and reduces storage costs.
Patent Information
- Application Number
- CN202411130450.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-16
- Publication Date
- 2024-11-22
AI Technical Summary
Traditional heterogeneous storage solutions are inefficient in data storage and management, difficult to cope with the needs of large-scale data, and the migration is complex, so they cannot effectively reduce storage costs.
By determining preset transmission conditions based on the frequency of data usage, filtering the data to be transmitted, and encoding it, data is transmitted from the first data storage environment to the second data storage environment, reducing data redundancy, reducing storage space requirements and Transmission time.
It improves the efficiency and accuracy of data transmission and migration, reduces the cost of the overall storage system, has better scalability, and is suitable for large-scale data storage needs.
Smart Images

Figure CN119011604A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of data processing technology, and in particular to a data transmission method, apparatus, computer equipment, computer-readable storage medium, and computer program product. Background Art
[0002] With the advancement of science and technology and the development of informatization, in order to ensure the security and reliability of data and reduce the cost of use, the requirements for data storage are becoming increasingly higher.
[0003] In traditional technology, different types of storage media and technologies can be combined to perform heterogeneous data storage. However, heterogeneous storage solutions usually require storing multiple copies to ensure data fault tolerance, resulting in low data storage efficiency and high migration difficulty, making it difficult to cope with large-scale data storage and management needs. Summary of the invention
[0004] Based on this, it is necessary to provide a data transmission method, apparatus, computer equipment, computer-readable storage medium and computer program product that can simplify the process and improve efficiency in response to the above technical problems.
[0005] In a first aspect, the present application provides a data transmission method, comprising:
[0006] In response to receiving the data transmission instruction, determining data to be transmitted that meets a preset transmission condition from the first data storage environment, wherein the preset transmission condition is determined based on a usage frequency of the data;
[0007] Encoding the data to be transmitted to obtain synchronous transmission data;
[0008] The to-be-transmitted data and the synchronous transmission data are transmitted from the first data storage environment to the second data storage environment.
[0009] In one embodiment, the method for determining the preset transmission condition includes:
[0010] Acquire a data usage frequency corresponding to the data in the first data storage environment;
[0011] Determining a frequency threshold according to the frequency of use of the data;
[0012] The preset transmission condition is determined as the data usage frequency being less than or equal to the frequency threshold.
[0013] In one embodiment, the storage method of data in the first data storage environment includes:
[0014] Obtaining data usage frequency corresponding to the data to be stored;
[0015] Determine a data storage level corresponding to the data usage frequency;
[0016] The data to be stored is stored in the data storage area corresponding to the data storage level.
[0017] In one embodiment, the data storage area corresponds to a data storage tag, and storing the data to be stored in the data storage area corresponding to the data storage level includes:
[0018] Determine a data storage area corresponding to the data storage tag to which the data storage level belongs;
[0019] The data to be stored is stored in the data storage area.
[0020] In one embodiment, encoding the data to be transmitted to obtain synchronous transmission data includes:
[0021] Dividing the data to be transmitted to obtain multiple data groups to be transmitted;
[0022] The multiple data groups to be transmitted are encoded to obtain synchronous transmission data, wherein the synchronous transmission data includes multiple synchronous transmission data groups.
[0023] In one of the embodiments, the first data storage environment and the second data storage environment correspond to the same data access node.
[0024] In a second aspect, the present application also provides a data transmission device, including:
[0025] A determination module, configured to determine, in response to receiving a data transmission instruction, data to be transmitted that meets a preset transmission condition from the first data storage environment, wherein the preset transmission condition is determined based on a usage frequency of the data;
[0026] The encoding module is used to encode the data to be transmitted to obtain synchronous transmission data;
[0027] The transmission module is used to transmit the data to be transmitted and the synchronous transmission data from the first data storage environment to the second data storage environment.
[0028] In one embodiment, the device further includes a module for determining the preset transmission condition, and the module for determining the preset transmission condition includes:
[0029] A first acquisition submodule, used to acquire a data usage frequency corresponding to the data in the first data storage environment;
[0030] A first determination submodule, configured to determine a frequency threshold according to the data usage frequency;
[0031] The second determining submodule is used to determine that the preset transmission condition is that the data usage frequency is less than or equal to the frequency threshold.
[0032] In one embodiment, the device further comprises a storage module for data in the first data storage environment, the storage module comprising:
[0033] The second acquisition submodule is used to acquire the data usage frequency corresponding to the data to be stored;
[0034] A third determination submodule is used to determine the data storage level corresponding to the data usage frequency;
[0035] The storage submodule is used to store the data to be stored in the data storage area corresponding to the data storage level.
[0036] In one embodiment, the data storage area corresponds to a data storage tag, and the storage submodule includes:
[0037] A determining unit, configured to determine a data storage area corresponding to a data storage tag to which the data storage level belongs;
[0038] The storage unit is used to store the data to be stored in the data storage area corresponding to the data storage level.
[0039] In one embodiment, the encoding module is further used for:
[0040] Dividing the data to be transmitted to obtain multiple data groups to be transmitted;
[0041] The multiple data groups to be transmitted are encoded to obtain synchronous transmission data, wherein the synchronous transmission data includes multiple synchronous transmission data groups.
[0042] In one of the embodiments, the first data storage environment and the second data storage environment correspond to the same data access node.
[0043] In a third aspect, an embodiment of the present disclosure further provides a computer device, wherein the computer device comprises a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the steps of any one of the methods in the embodiments of the present disclosure are implemented.
[0044] In a fourth aspect, the embodiments of the present disclosure further provide a computer-readable storage medium, wherein a computer program is stored thereon, and when the computer program is executed by a processor, the steps of any one of the methods in the embodiments of the present disclosure are implemented.
[0045] In a fifth aspect, the embodiments of the present disclosure further provide a computer program product, wherein the computer program product includes a computer program, and when the computer program is executed by a processor, the steps of any one of the methods in the embodiments of the present disclosure are implemented.
[0046] The above-mentioned data transmission method, apparatus, computer equipment, computer-readable storage medium and computer program product, when performing data transmission between a first data storage environment and a second data storage environment, first determine the data to be transmitted according to preset transmission conditions, and can screen the transmission data to ensure the efficiency and accuracy of data transmission and migration; when performing transmission, encode the data to be transmitted to obtain synchronous transmission data, and transmit the synchronous transmission data and the data to be transmitted to the second data storage environment. Compared with the copy replication method, data redundancy is reduced, storage space requirements are reduced, and the cost of the overall storage system is reduced. At the same time, since there is no need to transmit multiple complete copies, transmission time is saved and transmission efficiency is improved. By encoding data for data transmission, compared with the copy replication method, it has better scalability, adapts to the needs of large-scale data storage, and is suitable for more application scenarios. BRIEF DESCRIPTION OF THE DRAWINGS
[0047] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the drawings required for use in the embodiments of the present application or related technical descriptions will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.
[0048] Figure 1 A schematic diagram of a flow chart of a data transmission method in an embodiment;
[0049] Figure 2 A schematic diagram of a flow chart of a method for determining a preset transmission condition in an embodiment;
[0050] Figure 3 It is a flowchart of a method for storing data in a first data storage environment in an embodiment;
[0051] Figure 4 A schematic diagram of a flow chart of a data transmission method in another embodiment;
[0052] Figure 5 is a structural block diagram of a data transmission device in one embodiment;
[0053] Figure 6 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION
[0054] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.
[0055] In one embodiment, Figure 1 As shown, a data transmission method is provided. This embodiment takes the method applied to a terminal as an example. It can be understood that the method can also be applied to a server, and can also be applied to a system including a terminal and a server, and is implemented through the interaction between the terminal and the server. In this embodiment, the method includes the following steps:
[0056] Step S110, in response to receiving the data transmission instruction, determining data to be transmitted that meets a preset transmission condition from the first data storage environment, wherein the preset transmission condition is determined based on the usage frequency of the data;
[0057] Exemplarily, the data transmission instruction may be determined according to an actual application scenario. In some possible implementations, the data transmission instruction may be actively sent by a user or automatically triggered by a data storage environment. When a data transmission instruction is received, it may be considered that data transmission is required.
[0058] Optionally, when a data transmission instruction is received, data needs to be transmitted from the first data storage environment to the second data storage environment.
[0059] The preset transmission condition can be determined based on the frequency of data use, which may include but is not limited to data access frequency, data call frequency, data modification frequency, etc., which can be determined based on actual application scenarios. The preset transmission condition is determined based on the frequency of data use, and data to be transmitted that meets the preset transmission condition is determined from the first data storage environment.
[0060] Step S120, encoding the data to be transmitted to obtain synchronous transmission data;
[0061] Exemplarily, after determining that the data to be transmitted is obtained, the data to be transmitted is coded to obtain synchronous transmission data. In one example, the coding algorithm used in the coding process can be determined according to the actual application scenario, and the coding algorithm can include but is not limited to the Reed-Solomon algorithm, the LDPC algorithm, the Luby Transform algorithm, etc. In one example, the data to be transmitted can be directly coded, or the data to be transmitted can be divided to obtain multiple data groups, and the coding process is performed on the groups.
[0062] Step S130: transmitting the data to be transmitted and the synchronous transmission data from the first data storage environment to the second data storage environment.
[0063] Exemplarily, after the synchronous transmission data is obtained, the data to be transmitted and the synchronous transmission data are transmitted from the first data storage environment to the second data storage environment.
[0064] The first data storage environment and the second data storage environment may be the same data storage architecture or different data storage architectures. The data storage architectures may include but are not limited to Hadoop 2 clusters, Hadoop 3 clusters, etc. In one example, the first data storage environment may be a Hadoop 2 cluster, and the second data storage environment may be a Hadoop 3 cluster.
[0065] In the disclosed embodiments, when data is transmitted between a first data storage environment and a second data storage environment, the data to be transmitted is first determined according to preset transmission conditions, and the transmission data can be screened to ensure the efficiency and accuracy of data transmission and migration; when transmitting, the data to be transmitted is encoded to obtain synchronous transmission data, and the synchronous transmission data and the data to be transmitted are transmitted to the second data storage environment. Compared with the copy replication method, data redundancy is reduced, storage space requirements are reduced, and the cost of the overall storage system is reduced. At the same time, since there is no need to transmit multiple complete copies, transmission time is saved and transmission efficiency is improved. Data transmission through data encoding has better scalability than the copy replication method, adapts to the needs of large-scale data storage, and is suitable for more application scenarios.
[0066] In one embodiment, Figure 2 As shown, the method for determining the preset transmission condition includes:
[0067] Step S210, obtaining the data usage frequency corresponding to the data in the first data storage environment;
[0068] Step S220, determining a frequency threshold according to the data usage frequency;
[0069] Step S230: determining that the preset transmission condition is that the data usage frequency is less than or equal to the frequency threshold.
[0070] Exemplarily, the preset transmission condition is determined according to the data usage frequency of the data in the first data storage environment. The data usage frequency corresponding to the data in the first data storage environment is obtained. In one example, the data usage frequency may include but is not limited to data access frequency, data call frequency, data modification frequency, etc. The higher the data usage frequency, the more times the corresponding data is used, and the more important the data is.
[0071] In one example, data usage in the first data storage environment can be collected and recorded in real time to determine the data usage frequency corresponding to each data. A frequency threshold is determined based on the data usage frequency, and the frequency threshold can be a smaller usage frequency. When the data usage frequency is less than the frequency threshold, it can be considered that the data usage frequency is low and the data activity is low. Data migration can be performed to improve the utilization rate of the storage space in the first data storage environment.
[0072] The preset transmission condition is determined to be that the data usage frequency is less than or equal to a frequency threshold, and based on the preset transmission condition, data to be transmitted with a lower data usage frequency can be screened out.
[0073] In the embodiments of the present disclosure, when setting preset transmission conditions, a frequency threshold is determined based on the data usage frequency of the data in the first data storage environment, and data with a data usage frequency less than or equal to the frequency threshold is screened through the preset transmission conditions as data to be transmitted, thereby being able to obtain data with a lower usage frequency in the first data storage environment. When performing data transmission, data with a lower usage frequency is transmitted, thereby ensuring the reliability of data transmission and, at the same time, reducing the impact on other data in the first data storage environment, further improving the storage efficiency of the first data storage environment, and effectively improving the utilization rate of the storage space in the first data storage environment, which is suitable for more application scenarios.
[0074] In one embodiment, Figure 3 As shown, the storage method of data in the first data storage environment includes:
[0075] Step S310, obtaining the data usage frequency corresponding to the data to be stored;
[0076] Step S320, determining the data storage level corresponding to the data usage frequency;
[0077] Step S330: storing the data to be stored in the data storage area corresponding to the data storage level.
[0078] Exemplarily, data is stored hierarchically in the first data storage environment. When data is stored in the first data storage environment, the data usage frequency corresponding to the data to be stored is obtained, and the data storage level corresponding to the data usage frequency is determined. In one example, multiple data storage levels can be set according to actual application scenarios, and different data storage levels correspond to different data usage frequencies.
[0079] In an example, the first data storage environment may include data storage areas with different storage performances, such as solid-state hard disks, ordinary disks, tape libraries, cloud storage, etc., which can be determined based on actual application scenarios.
[0080] Different data storage areas may correspond to different data storage levels based on their different storage performances, and the data to be stored may be stored in the data storage area corresponding to the data storage level.
[0081] In one possible implementation, data with high usage frequency can be stored on a solid-state drive, data with relatively high usage frequency can be stored on a common disk, and data with relatively low usage frequency can be stored in a tape library or cloud storage. In some examples, data storage levels can be divided according to the importance of data, data access patterns, etc., and data can be stored in a hierarchical manner.
[0082] Optionally, the storage method of data in the second data storage environment may be the same as the storage method of data in the first data storage environment.
[0083] In the embodiments of the present disclosure, in the first data storage environment, data is stored in a hierarchical manner according to its usage frequency. Different data usage frequencies correspond to different data storage levels, and the corresponding data are stored in data storage areas corresponding to the data storage levels. This enables hierarchical storage of data in the first data storage environment, ensures the orderliness of data storage, and can further improve the utilization rate of data storage space by dividing different data storage areas.
[0084] In one embodiment, the data storage area corresponds to a data storage tag, and storing the data to be stored in the data storage area corresponding to the data storage level includes:
[0085] Determine a data storage area corresponding to the data storage tag to which the data storage level belongs;
[0086] The data to be stored is stored in the data storage area.
[0087] Exemplarily, different data storage levels correspond to different data storage tags. In one example, data storage areas of different data storage levels can be distinguished by data storage tags to facilitate subsequent data migration and scheduling.
[0088] In an example, the data storage tag may include but is not limited to a digital tag, a string tag, an identification tag, etc., which may be determined based on the actual application scenario.
[0089] In the embodiments of the present disclosure, when performing data storage, the corresponding data storage area can be determined according to the data storage label belonging to the data storage level, and the data to be stored is stored in the data storage area. The data storage area is divided according to the data storage label, which is beneficial to the subsequent adjustment of data storage. In the subsequent data processing and access process, data migration and scheduling can be performed quickly and effectively, further improving the efficiency of data transmission.
[0090] In one embodiment, Figure 4 As shown, the encoding process of the data to be transmitted to obtain synchronous transmission data includes:
[0091] Step S121, dividing the data to be transmitted to obtain a plurality of data groups to be transmitted;
[0092] Step S122: encoding the plurality of data groups to be transmitted to obtain synchronous transmission data, wherein the synchronous transmission data includes a plurality of synchronous transmission data groups.
[0093] Exemplarily, when performing encoding processing, the data blocks are encoded in groups. The data to be transmitted is divided to obtain multiple data groups to be transmitted, wherein the data division method can be determined according to the actual application scenario. In one example, the data to be transmitted can be divided into equal parts to obtain multiple data groups to be transmitted.
[0094] Multiple data groups to be transmitted are encoded to obtain synchronous transmission data, wherein the synchronous transmission data includes multiple synchronous transmission data groups, wherein the number of synchronous transmission data groups and the number of data groups to be transmitted may be the same or different. In one example, the synchronous transmission data can be used to verify data recovery. During data transmission, if part of the data to be transmitted is lost or erroneous, the data can be recovered and corrected by decoding the synchronous transmission data, thereby effectively ensuring the reliability and security of data transmission.
[0095] In the disclosed embodiment, when performing data encoding, the data to be transmitted is divided into multiple data groups to be transmitted, and the groups are encoded to obtain synchronous transmission data including multiple synchronous transmission data groups, thereby realizing the division and encoding of data blocks, ensuring the reliability of data transmission, reducing the data processing amount of data encoding, improving data transmission efficiency, and being suitable for more application scenarios.
[0096] In one embodiment, the first data storage environment and the second data storage environment correspond to the same data access node.
[0097] Exemplarily, the first data storage environment and the second data storage environment may correspond to the same data access node. In one example, data intercommunication may be achieved by configuring the data access nodes of the first data storage environment and the second data storage environment. In a possible implementation, the Nameservice function may be configured and used to make the first data storage environment and the second data storage environment correspond to the same data access node.
[0098] In one example, since the first data storage environment and the second data storage environment correspond to the same data access node, when data is accessed, data access and search can be performed through the same data access node to achieve data sharing.
[0099] In some possible implementations, the configuration of the data access node may include but is not limited to DistCp, NFS Gateway, Hadoop Federation, data synchronization tools, etc.
[0100] In the disclosed embodiment, the first data storage environment and the second data storage environment correspond to the same data access node, thereby realizing data sharing between the first data storage environment and the second data storage environment, enabling mutual access and sharing of data, improving the flexibility of data management and resource calculation, and enhancing data utilization and data storage efficiency.
[0101] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.
[0102] Based on the same inventive concept, the embodiment of the present application also provides a data transmission device for implementing the data transmission method involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in one or more data transmission device embodiments provided below can refer to the limitations on the data transmission method above, and will not be repeated here.
[0103] In an exemplary embodiment, Figure 5 As shown, a data transmission device 500 is provided, comprising:
[0104] A determination module 510 is configured to determine, in response to receiving a data transmission instruction, data to be transmitted that meets a preset transmission condition from the first data storage environment, wherein the preset transmission condition is determined based on a usage frequency of the data;
[0105] The encoding module 520 is used to encode the data to be transmitted to obtain synchronous transmission data;
[0106] The transmission module 530 is used to transmit the data to be transmitted and the synchronous transmission data from the first data storage environment to the second data storage environment.
[0107] In one embodiment, the device further includes a module for determining the preset transmission condition, and the module for determining the preset transmission condition includes:
[0108] A first acquisition submodule, used to acquire a data usage frequency corresponding to the data in the first data storage environment;
[0109] A first determination submodule, configured to determine a frequency threshold according to the data usage frequency;
[0110] The second determining submodule is used to determine that the preset transmission condition is that the data usage frequency is less than or equal to the frequency threshold.
[0111] In one embodiment, the device further comprises a storage module for data in the first data storage environment, the storage module comprising:
[0112] The second acquisition submodule is used to acquire the data usage frequency corresponding to the data to be stored;
[0113] A third determination submodule is used to determine the data storage level corresponding to the data usage frequency;
[0114] The storage submodule is used to store the data to be stored in the data storage area corresponding to the data storage level.
[0115] In one embodiment, the data storage area corresponds to a data storage tag, and the storage submodule includes:
[0116] A determining unit, configured to determine a data storage area corresponding to a data storage tag to which the data storage level belongs;
[0117] The storage unit is used to store the data to be stored in the data storage area corresponding to the data storage level.
[0118] In one embodiment, the encoding module is further used for:
[0119] Dividing the data to be transmitted to obtain multiple data groups to be transmitted;
[0120] The multiple data groups to be transmitted are encoded to obtain synchronous transmission data, wherein the synchronous transmission data includes multiple synchronous transmission data groups.
[0121] In one of the embodiments, the first data storage environment and the second data storage environment correspond to the same data access node.
[0122] Each module in the above data transmission device can be implemented in whole or in part by software, hardware, or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute operations corresponding to each module.
[0123] In an exemplary embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as shown in FIG. Figure 6 As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. Among them, the processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store data to be transmitted and other data involved in the method described in this embodiment. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a data transmission method is implemented.
[0124] Those skilled in the art will understand that Figure 6 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.
[0125] In one embodiment, a computer device is further provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above method embodiments when executing the computer program.
[0126] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.
[0127] In one embodiment, a computer program product is provided, including a computer program, which implements the steps in the above method embodiments when executed by a processor.
[0128] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.
[0129] A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, an artificial intelligence (AI) processor, etc., but are not limited to this.
[0130] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.
[0131] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.
Claims
1. A data transmission method, It is characterized in that The method comprises: In response to receiving the data transmission instruction, determining data to be transmitted that meets a preset transmission condition from the first data storage environment, wherein the preset transmission condition is determined based on a usage frequency of the data; Encoding the data to be transmitted to obtain synchronous transmission data; The to-be-transmitted data and the synchronous transmission data are transmitted from the first data storage environment to the second data storage environment.
2. The method according to claim 1, It is characterized in that The method for determining the preset transmission condition includes: Acquire a data usage frequency corresponding to the data in the first data storage environment; Determining a frequency threshold according to the frequency of use of the data; The preset transmission condition is determined as the data usage frequency being less than or equal to the frequency threshold.
3. The method according to claim 1, It is characterized in that The storage method of data in the first data storage environment includes: Obtaining data usage frequency corresponding to the data to be stored; Determine a data storage level corresponding to the data usage frequency; The data to be stored is stored in the data storage area corresponding to the data storage level.
4. The method according to claim 3, It is characterized in that The data storage area corresponds to a data storage tag, and storing the data to be stored in the data storage area corresponding to the data storage level includes: Determine a data storage area corresponding to the data storage tag to which the data storage level belongs; The data to be stored is stored in the data storage area.
5. The method according to claim 1, It is characterized in that The encoding process of the data to be transmitted to obtain synchronous transmission data includes: Dividing the data to be transmitted to obtain multiple data groups to be transmitted; The multiple data groups to be transmitted are encoded to obtain synchronous transmission data, wherein the synchronous transmission data includes multiple synchronous transmission data groups.
6. The method according to claim 1, It is characterized in that The first data storage environment and the second data storage environment correspond to the same data access node.
7. A data transmission device, It is characterized in that The device comprises: A determination module, configured to determine, in response to receiving a data transmission instruction, data to be transmitted that meets a preset transmission condition from the first data storage environment, wherein the preset transmission condition is determined based on a usage frequency of the data; The encoding module is used to encode the data to be transmitted to obtain synchronous transmission data; The transmission module is used to transmit the data to be transmitted and the synchronous transmission data from the first data storage environment to the second data storage environment.
8. A computer device comprising a memory and a processor, wherein the memory stores a computer program. It is characterized in that When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.
9. A computer-readable storage medium having a computer program stored thereon, It is characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.
10. A computer program product comprising a computer program, It is characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.
