Data transmission method, device, electronic device and computer readable medium

Abstract

The present application discloses a data transmission method, device, electronic device and computer-readable medium, and relates to the field of computer technology. The method includes: receiving a data transmission request, obtaining corresponding scene data, and then determining the data source identifier and the corresponding data magnitude corresponding to the scene data; determining the data transmission channel corresponding to the scene data according to the data source identifier and the data magnitude; obtaining the channel weight corresponding to each data transmission channel, and then determining the priority of each data transmission channel according to each channel weight; based on the priority, executing the transmission of the corresponding scene data through the corresponding data transmission channel. By pushing data of different scenes to different channels and assigning different weights to each channel, it can be ensured that data with high priority is transmitted to the library first, and other data are queued and transmitted to the library in sequence, thereby improving the overall throughput of data transmission.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a data transmission method, device, electronic device and computer-readable medium. Background Art

[0002] During data transmission, all messages that need to be transmitted are sent directly to a channel for consumption by consumers. The data transmission channel is single, the overall throughput is low, and the priority between data cannot be controlled. The data in each scene affects each other. Because there is only one channel, other data will be blocked when the message consumption is full or consumption errors are reported.

[0003] In the process of implementing this application, the inventors found that there are at least the following problems in the prior art:

[0004] When data transmission is performed, the data transmission channel is single and the overall data transmission throughput is low. Summary of the invention

[0005] In view of this, the embodiments of the present application provide a data transmission method, device, electronic device and computer-readable medium, which can solve the existing problem of a single data transmission channel and low overall data transmission throughput during data transmission.

[0006] To achieve the above object, according to one aspect of an embodiment of the present application, a data transmission method is provided, including:

[0007] Receive a data transmission request, obtain corresponding scene data, and then determine a data source identifier and a corresponding data magnitude corresponding to the scene data;

[0008] Determine the data transmission channel corresponding to the scene data according to the data source identifier and data magnitude;

[0009] Obtaining the channel weight corresponding to each data transmission channel, and then determining the priority of each data transmission channel according to each channel weight;

[0010] Based on the priority, the transmission of the corresponding scene data is performed through the corresponding data transmission channel.

[0011] Optionally, determining a data source identifier corresponding to the scene data includes:

[0012] Get the field value corresponding to the source field in the scene data;

[0013] Determine the data source identifier corresponding to the scene data based on the field value.

[0014] Optionally, determining a data transmission channel corresponding to the scene data includes:

[0015] In response to the data source identifier corresponding to the first data source, it is determined that the data transmission channel corresponding to the scene data is the first data transmission channel.

[0016] Optionally, determining a data transmission channel corresponding to the scene data includes:

[0017] In response to the data source identifier corresponding to the second data source, obtaining a task type identifier corresponding to the data source identifier;

[0018] In response to the task type identifier corresponding to the user real-time behavior, the universal channel corresponding to the user real-time behavior is determined as the data transmission channel corresponding to the scene data.

[0019] Optionally, determining a data transmission channel corresponding to the scene data includes:

[0020] In response to the task type identification corresponding to the timed directional task, the scene data is segmented according to the data magnitude to obtain data segments;

[0021] According to each data slice, the corresponding universal channel is determined.

[0022] Optionally, based on the priority, performing transmission of corresponding scene data through a corresponding data transmission channel includes:

[0023] Determine the data transmission channel corresponding to the highest priority as the target data transmission channel;

[0024] Calculate the time period corresponding to when the corresponding scene data is pulled through the target transmission channel, and determine the data transmission speed corresponding to the time period;

[0025] Get the data transmission speed threshold corresponding to the time period;

[0026] In response to the data transmission speed being greater than the data transmission speed threshold, performing the scene data pulling operation again after a preset time interval;

[0027] In response to the data transmission speed being less than or equal to the data transmission speed threshold, the scene data pulling operation continues to be performed until the scene data corresponding to the target transmission channel is completely pulled.

[0028] Optionally, before determining the data transmission channel corresponding to the scene data, the method further includes:

[0029] In response to the data level exceeding a preset data level threshold, an alarm process is executed.

[0030] Optionally, before determining the data transmission channel corresponding to the scene data, the method further includes:

[0031] In response to the data source identifier including a preset routing identifier, the scene data corresponding to the preset routing identifier is filtered.

[0032] In addition, the present application also provides a data transmission device, including:

[0033] A receiving unit is configured to receive a data transmission request, obtain corresponding scene data, and then determine a data source identifier and a corresponding data magnitude corresponding to the scene data;

[0034] A data transmission channel determination unit is configured to determine a data transmission channel corresponding to the scene data according to the data source identifier and the data magnitude;

[0035] an acquisition unit, configured to acquire a channel weight corresponding to each data transmission channel, and then determine a priority of each data transmission channel according to each channel weight;

[0036] The data transmission unit is configured to perform transmission of corresponding scene data through a corresponding data transmission channel based on the priority.

[0037] Optionally, the receiving unit is further configured to:

[0038] Get the field value corresponding to the source field in the scene data;

[0039] Determine the data source identifier corresponding to the scene data based on the field value.

[0040] Optionally, the data transmission channel determination unit is further configured to:

[0041] In response to the data source identifier corresponding to the first data source, it is determined that the data transmission channel corresponding to the scene data is the first data transmission channel.

[0042] Optionally, the data transmission channel determination unit is further configured to:

[0043] In response to the data source identifier corresponding to the second data source, obtaining a task type identifier corresponding to the data source identifier;

[0044] In response to the task type identifier corresponding to the user real-time behavior, the universal channel corresponding to the user real-time behavior is determined as the data transmission channel corresponding to the scene data.

[0045] Optionally, the data transmission channel determination unit is further configured to:

[0046] In response to the task type identification corresponding to the timed directional task, the scene data is segmented according to the data magnitude to obtain data segments;

[0047] According to each data slice, the corresponding universal channel is determined.

[0048] Optionally, the data transmission unit is further configured to:

[0049] Determine the data transmission channel corresponding to the highest priority as the target data transmission channel;

[0050] Calculate the time period corresponding to when the corresponding scene data is pulled through the target transmission channel, and determine the data transmission speed corresponding to the time period;

[0051] Get the data transmission speed threshold corresponding to the time period;

[0052] In response to the data transmission speed being greater than the data transmission speed threshold, performing the scene data pulling operation again after a preset time interval;

[0053] In response to the data transmission speed being less than or equal to the data transmission speed threshold, the scene data pulling operation continues to be performed until the scene data corresponding to the target transmission channel is completely pulled.

[0054] Optionally, the data transmission device further includes an alarm unit configured to:

[0055] In response to the data level exceeding a preset data level threshold, an alarm process is executed.

[0056] Optionally, the data transmission device further includes a filtering unit configured to:

[0057] In response to the data source identifier including a preset routing identifier, the scene data corresponding to the preset routing identifier is filtered.

[0058] In addition, the present application also provides a data transmission electronic device, including: one or more processors; a storage device for storing one or more programs, when the one or more programs are executed by one or more processors, the one or more processors implement the data transmission method as described above.

[0059] In addition, the present application also provides a computer-readable medium on which a computer program is stored, and when the program is executed by a processor, the data transmission method as described above is implemented.

[0060] One embodiment of the above invention has the following advantages or beneficial effects: the present application obtains the corresponding scene data by receiving a data transmission request, and then determines the data source identifier and the corresponding data magnitude corresponding to the scene data; determines the data transmission channel corresponding to the scene data according to the data source identifier and the data magnitude; obtains the channel weight corresponding to each data transmission channel, and then determines the priority of each data transmission channel according to each channel weight; based on the priority, the corresponding scene data is transmitted through the corresponding data transmission channel. By pushing data of different scenes to different channels and assigning different weights to each channel, it can be ensured that data with high priority is transmitted to the database first, and other data are queued and transmitted to the database in turn, thereby improving the overall throughput of data transmission.

[0061] The further effects of the above-mentioned non-conventional optional manner will be described below in conjunction with specific implementation examples. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] The accompanying drawings are used to better understand the present application and do not constitute an improper limitation on the present application.

[0063] Figure 1 is a schematic diagram of the main process of the data transmission method according to the first embodiment of the present application;

[0064] Figure 2 is a schematic diagram of the main process of the data transmission method according to the second embodiment of the present application;

[0065] Figure 3 is a schematic diagram of an application scenario of a data transmission method according to a third embodiment of the present application;

[0066] Figure 4 is a global architecture diagram of a data transmission method according to an embodiment of the present application;

[0067] Figure 5 is a schematic diagram of main units of a data transmission device according to an embodiment of the present application;

[0068] Figure 6 is an exemplary system architecture diagram to which the embodiments of the present application can be applied;

[0069] Figure 7 It is a structural diagram of a computer system of a terminal device or server suitable for implementing an embodiment of the present application. DETAILED DESCRIPTION

[0070] The following is a description of exemplary embodiments of the present application in conjunction with the accompanying drawings, including various details of the embodiments of the present application to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present application. Similarly, for the sake of clarity and conciseness, the description of well-known functions and structures is omitted in the following description. The acquisition, storage, use, and processing of data in the technical solution of the present application are in compliance with the relevant provisions of national laws and regulations.

[0071] Figure 1 is a schematic diagram of the main process of the data transmission method according to the first embodiment of the present application, such as Figure 1 As shown, the data transmission method includes:

[0072] Step S101, receiving a data transmission request, acquiring corresponding scene data, and then determining a data source identifier and a corresponding data level corresponding to the scene data.

[0073] In this embodiment, the execution subject of the data transmission method (for example, it can be a server) can receive the data transmission request through a wired connection or a wireless connection to obtain the corresponding scene data. The scene data corresponding to the data transmission request can be, for example, user policy information or user tag information. Among them, the user policy information can be the policy information of the user's real-time behavior generated by the user browsing an article or adding a product. The user tag information can be user tag information processed based on a certain algorithm and data, such as the region is Shanghai, etc. The data transmission request can also carry an identifier representing the source of the scene data, that is, a data source identifier, such as api, which represents the api interface scenario, that is, it represents the api source; the data source identifier, for example, excel, represents the manual import scenario, that is, it represents the manual import source. After determining the data source identifier corresponding to the scene data, the execution subject can determine the data level of the scene data corresponding to each data source identifier. For example, the data level of the scene data with the data source identifier of api is 1 million, and the data level of the scene data with the data source identifier of excel is 2 million.

[0074] Step S102: determining a data transmission channel corresponding to the scene data according to the data source identifier and the data magnitude.

[0075] The execution subject may pre-set multiple data channels, for example, the specific channels may be:

[0076] a) Interface data channel: receives data from API call scenarios;

[0077] b) Manual derivative channel: receives data manually imported into the Excel scene;

[0078] c) Appointment channel: Receives data on appointments made by customers through apps, phone calls and other channels.

[0079] The above a), b), and c) can determine the corresponding data transmission channel (such as interface data channel, manual derivative channel, and reservation data channel) through the data source identifier (such as API, Excel, App).

[0080] d) Jmq Progress 1 data channel: used to receive policy data generated by real-time user behavior;

[0081] Specifically, when the data source identifier source = molo and the task_type of the molo scene = 1, it means that the scene data obtained by the execution subject is the data generated by the user's real-time behavior. At this time, the data transmission channel corresponding to the scene data is determined to be the Jmq input number 1 data channel, that is, Figure 4 The global architecture diagram shows the generic channel 1.

[0082] e) Jmq Progress 2 data channel: used to receive user policy data generated by scheduled directional tasks (batch tasks that are executed regularly at a specified time). Here, the data will be sharded according to the amount of task data, and only the first 1 million (configurable) data pushed by the task will be received;

[0083] f) Jmq Progress 3 data channel: used to receive user policy data generated by scheduled directional tasks (batch tasks that are executed regularly at a specified time). Here, the data will be sharded according to the amount of task data, and only the first 1 million to 2 million (configurable) data pushed by the task will be received;

[0084] g) Jmq feed 4 data channel: used to receive user policy data generated by scheduled directional tasks (batch tasks that are executed regularly at a specified time). Here, the data will be fragmented according to the amount of task data, and only the first 2 million to 3 million (configurable) data pushed by the task will be received;

[0085] h) Jmq Progress 5 data channel: used to receive user policy data generated by scheduled directional tasks (batch tasks that are executed regularly at a specified time). The data will be sharded according to the amount of task data, and the data pushed by the task at that time that is greater than the amount of data configured above will be received.

[0086] Specifically, when the data source identifier source=molo and the task_type of the molo scenario is 2, it means that the scene data corresponding to the data transmission request is derived from the timing directional information. In this scenario, the execution subject can determine that the data transmission channel corresponding to the scene data can be one or more of the Jmq input number 2 data channel, Jmq input number 3 data channel, Jmq input number 4 data channel, and Jmq input number 5 data channel shown in the above e), f), g), and h). Which channel is specific needs to be determined based on the data level corresponding to the molo scene data. For example, when the data source identifier source=molo and the task_type of the molo scenario is 2, it means that the scene data is derived from the timing directional information, and then it is necessary to further determine the corresponding channel based on the data level, for example: data.priority.max=1000000, 2000000, 3000000, which means that data less than 1000000 is pushed to the Jmq input number 2 data channel (that is, Figure 4 If the value is greater than 1,000,000 and less than 2,000,000, it is pushed to the Jmq data channel 3 (that is, Figure 4 If the value is greater than 2000000 and less than 3000000, it is pushed to the Jmq forwarding channel 4 (that is, Figure 4 Otherwise, it is pushed to the Jmq data channel 5 (that is, Figure 4 Universal channel 5).

[0087] Specifically, before determining the data transmission channel corresponding to the scene data, the method further includes: in response to the data level exceeding a preset data level threshold, executing an alarm process.

[0088] The execution subject can determine the magnitude of the received scene data (which may include data of one or more scenes), for example, scene data of 2 scenes is received, and the total magnitude is 5 million. If 5 million exceeds the preset data magnitude threshold (for example, 1 million), the execution subject can execute the alarm process interception.

[0089] As another implementation method, the execution subject can also execute the alarm process when the amount of scene data of a single scene exceeds a preset data level threshold. For example, the alarm threshold configuration is: routeMaxDayCount=7000000, which means that an alarm will be issued when the amount of data pushed by a scene in one day exceeds this value.

[0090] Specifically, before determining the data transmission channel corresponding to the scene data, the method further includes: in response to the data source identifier including the preset routing identifier, filtering the scene data corresponding to the preset routing identifier.

[0091] For example, the data filtering configuration is: filteredRouteId = aaa, ccc, which means that the data with routeId = aaa and ccc is filtered out and not consumed.

[0092] Specifically, determining the data transmission channel corresponding to the scene data includes:

[0093] In response to the data source identifier corresponding to the second data source, obtain the task type identifier corresponding to the data source identifier; by way of example, the second data source may specifically include a user behavior data source and a timing directional data source. For example, the source field carried in the data transmission request received by the execution subject, source=molo, represents that it comes from the molo scenario. The molo scenario includes the user real-time behavior scenario and the timing directional scenario. The corresponding data sources are the user behavior data source and the timing directional data source, respectively. The user real-time behavior data source and the timing directional data source both belong to the second data source. The data source identifier, such as molo, represents the molo scenario; the task type identifier corresponding to the data source identifier, such as task_type=1, task_type=2.

[0094] In response to the task type identifier corresponding to the user real-time behavior, the universal channel corresponding to the user real-time behavior is determined as the data transmission channel corresponding to the scene data.

[0095] When the task type is 1, that is, task_type = 1, it means that the acquired scene data is the user's real-time behavior data. The universal channel corresponding to the user's real-time behavior is as follows: Figure 4 The universal channel 1 shown is the data transmission channel corresponding to the scene data generated by the user's actual behavior.

[0096] Specifically, determining the data transmission channel corresponding to the scene data includes:

[0097] In response to the task type identifying the corresponding timing and directional task, the scene data is sliced ​​according to the data magnitude to obtain various data slices; and the corresponding universal channel is determined according to each data slice.

[0098] For example, when the executing entity determines that the task type identifier corresponds to a timed and directional task, the scene data is segmented according to the data level of the scene data obtained by the timed and directional task. For example, if the data level of the scene data is 4 million, the segmentation results are: <1 million, 1 million to 2 million, 2 million to 3 million, and 3 million to 4 million.

[0099] According to each data slice, the corresponding universal channel is determined, which can be specifically: the data <1 million in the obtained data slice is allocated to Figure 4 The data channel 2 shown in FIG. 1 distributes 1 million to 2 million data in the obtained data slices to the data channel 2 shown in FIG. Figure 4 The universal channel 3 shown in FIG. 1 distributes 2 million to 3 million data in the obtained data slices to the following: Figure 4 The universal channel 4 shown in FIG. 1 distributes 3 million to 4 million data in the obtained data slices to the following: Figure 4 The general channel 5 is shown. The above is only an example and is not intended to limit the embodiments of the present application.

[0100] Step S103: Acquire the channel weight corresponding to each data transmission channel, and then determine the priority of each data transmission channel according to each channel weight.

[0101] For example, the execution subject can set different weights for each message channel in the concrete memory management system. The weights of channels with high consumption speed and high data real-time requirements are set to be larger. The larger the weight, the higher the priority of the data transmission channel corresponding to the weight.

[0102] For example, the weight of the artificial derivative channel is set to 50, the weight of the reserved input channel is set to 50, the weight of the interface input channel is set to 40, the weight of the universal channel 1 is set to 50, the weight of the universal channel 2 is set to 40, the weight of the universal channel 3 is set to 20, the weight of the universal channel 4 is set to 20, and the weight of the universal channel 5 is set to 10. Then the priority of the data transmission channel can be artificial derivative channel = reserved input channel = universal channel 1 > interface input channel > universal channel 2 > universal channel 3 = universal channel 4 > universal channel 5.

[0103] Step S104: Based on the priority, the corresponding scene data is transmitted through the corresponding data transmission channel.

[0104] Specifically, based on the priority, the corresponding scene data is transmitted through the corresponding data transmission channel, including:

[0105] The data transmission channel corresponding to the highest priority is determined as the target data transmission channel; the time period corresponding to pulling the corresponding scene data through the target transmission channel is calculated, and the data transmission speed corresponding to the time period is determined; the data transmission speed threshold corresponding to the time period is obtained; in response to the data transmission speed being greater than the data transmission speed threshold, the scene data pulling operation is performed again after a preset time interval; in response to the data transmission speed being less than or equal to the data transmission speed threshold, the scene data pulling operation is continued until the scene data corresponding to the target transmission channel is pulled.

[0106] Based on the priorities of each data transmission channel, the execution entity first transmits the corresponding scene data to the data transmission channel with the highest priority. After the transmission is completed, the execution entity then transmits the corresponding scene data to the data transmission channel with the second priority. This continues until the scene data corresponding to the last priority data transmission channel is transmitted, at which point the data transmission ends.

[0107] Furthermore, in the process of transmitting the scene data corresponding to each data transmission channel, the execution subject can verify in real time the current data transmission speed of each data transmission channel and the data transmission speed threshold corresponding to each data transmission channel, and adjust the transmission of the scene data of each data transmission channel according to the verification result. The specific examples are as follows:

[0108] The threshold verification process of the embodiment of the present application is as follows: for each data transmission channel, the execution subject changes the consumption program of the scene data from the listening mode to the active pulling mode: data pulling is performed once every 1 second, and the total speed of the current time period is obtained when the data pulling is performed, and then it is determined whether the total speed obtained exceeds the threshold configured in the time-sharing speed control function. If the threshold is reached, the data pulling work will be continued at an interval of 100ms (the embodiment of the present application does not specifically limit the interval time).

[0109] This embodiment receives a data transmission request to obtain the corresponding scene data, and then determines the data source identifier and the corresponding data level corresponding to the scene data; determines the data transmission channel corresponding to the scene data according to the data source identifier and the data level; obtains the channel weight corresponding to each data transmission channel, and then determines the priority of each data transmission channel according to each channel weight; based on the priority, the corresponding scene data is transmitted through the corresponding data transmission channel. By pushing data of different scenes to different channels and assigning different weights to each channel, it can be ensured that data with high priority is transmitted to the library first, and other data are queued and transmitted to the library in turn, thereby improving the overall throughput of data transmission.

[0110] Figure 2 is a schematic diagram of the main flow of the data transmission method according to the second embodiment of the present application, such as Figure 2 As shown, the data transmission method includes:

[0111] Step S201, receiving a data transmission request and acquiring corresponding scene data.

[0112] Step S202, obtaining a field value corresponding to a source field in the scene data. The source field, for example, the source field. The field value corresponding to the source field, for example, source=excel, where excel is the field value corresponding to the source field.

[0113] Step S203: Determine the data source identifier corresponding to the scene data according to the field value. For example, the data source identifier may be api, excel, etc.

[0114] Step S204, determining the data level corresponding to the scene data.

[0115] Step S205 , in response to the data source identifier corresponding to the first data source, determining that the data transmission channel corresponding to the scene data is the first data transmission channel.

[0116] The first data source of the embodiment of the present application includes data input by the API interface (data in the scenario of API interface call), manually imported data (data in the scenario of manual import of Excel), and data entered by appointment (data entered by customers through channels such as apps and telephones). The first data transmission channel includes: Figure 4 The artificial derivative channel, reservation input channel and interface input channel shown. For example, when the data source identifier corresponds to API, it is determined that the data source identifier corresponds to the first data source, and the data transmission channel corresponding to the scene data corresponding to the data source identifier (API) is determined to be the interface input channel in the first data transmission channel. When the data source identifier corresponds to Excel, it is determined that the data source identifier corresponds to the first data source, and the data transmission channel corresponding to the scene data corresponding to the data source identifier (Excel) is determined to be the artificial derivative channel in the first data transmission channel.

[0117] Step S206: Acquire the channel weight corresponding to each data transmission channel, and then determine the priority of each data transmission channel according to each channel weight.

[0118] The channel weight represents the consumption speed, the scene's requirements for data real-time, and other information. The higher the consumption speed and the higher the scene's requirements for data real-time, the higher the corresponding channel weight of the data transmission channel. The channel weight can be pre-set according to actual conditions.

[0119] Step S207: Based on the priority, the corresponding scene data is transmitted through the corresponding data transmission channel.

[0120] The embodiment of the present application pushes data from different scenarios to different channels, which can reduce the problem of mutual influence between data. Different weights are assigned to each channel to ensure that high-priority data is stored first, and other data are queued and stored in sequence, thereby improving the overall throughput of data transmission.

[0121] Figure 3 Schematic diagram of the application scenario of the data transmission method according to the third embodiment of the present application. The data transmission method of the embodiment of the present application is applied to a multi-channel, multi-priority mass data transmission scenario with controllable total speed and time-sharing speed increase. Figure 3 As shown, the overall data consumption process is as follows:

[0122] a) Pushing scene data to the data channel: After receiving the scene data, the parameters, business, token and other data are verified, and the verified data is pushed to the corresponding data channel for consumption; specifically, the scene data needs to be Figure 4The scene entry shown is transmitted to the execution body. In the embodiment of the present application, "input" means "importing data". The scene entry includes the jmq input entry (input through the molo-c end), the interface input entry (input through the molo-d end, gold bars, etc.), the import input entry (input through manual excel), and the appointment input entry (input through appointment). After receiving the data, the execution body executes data distribution, specifically judging the source of the data according to the source field in the data, source = molo represents the source from the molo scene, task_type = 1 of the molo scene represents user behavior data, task_type = 2 represents timed directional data, source = api represents the api interface scene, source = excel represents the manual import of excel, source = appointment represents the appointment input scene; verify the business scenario and the data volume threshold configured by concrete to distribute the data to each data channel.

[0123] b) Check whether the data is pushed by mistake, if so, intercept it; after receiving the scene data through the scene entrance, the execution subject can call Figure 4 The front module shown in the figure intercepts the mis-push. The front module can also perform parameter verification, message duplication prevention, extended warranty service, token verification, binding relationship verification, scene / level identification, blacklist filtering, service duplication prevention and excessive alarm and other services.

[0124] c) Check whether the data volume has reached the alarm threshold. If so, Figure 4 The front module in the process executes the overload alarm process to send an alarm;

[0125] d) Counting: Push data counts to redis (a high-performance key-value database) according to the scenario;

[0126] e) Threshold check: Change the consumer program from monitoring mode to active pull mode: perform data pull once every 1 second. When executing data pull, check whether the total speed of the current time period exceeds the threshold configured in the time-sharing speed control function. If the threshold is reached, rest for 100ms before continuing to execute data pull;

[0127] f) Pull action: After passing the overall threshold check, the configured channels and channel weights are obtained from concrete (concrete runs ciphertext quickly by extending TfhE. It is a fully homomorphic encryption (FHE) library that implements Zama's TFHE variant. The Hyperledgerfabric-config library is used to update the channel configuration) and the weight coefficient data is pulled from the channel;

[0128] g) Core consumption: After pulling the data, perform core business verification and storage operations. Among them, core consumption includes: mispush interception, unified log, relationship valve verification and data entry receipt, as well as transaction maintenance and execution. Among them, transaction maintenance and execution include: customer basic information, agent task information, customer relationship information, customer strategy information, customer extended attributes and batch detail information maintenance and execution.

[0129] The embodiments of the present application can realize multi-priority channel design, time-sharing consumption, and speed safety control. It can realize time-sharing speed adjustment: reduce the speed during business peaks, increase the consumption speed during business off-peaks, and improve the overall throughput; total speed control: the total speed of storage is controlled within a safe range; alarm function: alarm when it exceeds expectations, and find problems in time; abnormal data filtering: discard useless data to avoid wasting system resources; weighted consumption: specify different weights for each channel, which can ensure that high-priority data is stored first, and other data is queued and stored in turn; data decoupling: push data from different scenarios to different channels, which can reduce the mutual influence between data.

[0130] Figure 5 Schematic diagram of the main units of the data transmission device according to an embodiment of the present application. Figure 5 As shown, the data transmission device 500 includes a receiving unit 501 , a data transmission channel determining unit 502 , an acquiring unit 503 and a data transmission unit 504 .

[0131] The receiving unit 501 is configured to receive a data transmission request, obtain corresponding scene data, and further determine a data source identifier and a corresponding data level corresponding to the scene data.

[0132] The data transmission channel determination unit 502 is configured to determine the data transmission channel corresponding to the scene data according to the data source identifier and the data magnitude.

[0133] The acquisition unit 503 is configured to acquire the channel weight corresponding to each data transmission channel, and then determine the priority of each data transmission channel according to each channel weight.

[0134] The data transmission unit 504 is configured to perform transmission of corresponding scene data through a corresponding data transmission channel based on the priority.

[0135] In some embodiments, the receiving unit 501 is further configured to: obtain a field value corresponding to a source field in the scene data; and determine a data source identifier corresponding to the scene data according to the field value.

[0136] In some embodiments, the data transmission channel determination unit 502 is further configured to: in response to the data source identifier corresponding to the first data source, determine that the data transmission channel corresponding to the scene data is the first data transmission channel.

[0137] In some embodiments, the data transmission channel determination unit 502 is further configured to: in response to the data source identifier corresponding to the second data source, obtain the task type identifier corresponding to the data source identifier; in response to the task type identifier corresponding to the user's real-time behavior, determine the universal channel corresponding to the user's real-time behavior as the data transmission channel corresponding to the scene data.

[0138] In some embodiments, the data transmission channel determination unit 502 is further configured to: in response to the task type identifying the corresponding timing and directional task, slice the scene data according to the data magnitude to obtain each data slice; and determine the corresponding universal channel according to each data slice.

[0139] In some embodiments, the data transmission unit 504 is further configured to: determine the data transmission channel corresponding to the highest priority as the target data transmission channel; calculate the time period corresponding to pulling the corresponding scene data through the target transmission channel, and determine the data transmission speed corresponding to the time period; obtain the data transmission speed threshold corresponding to the time period; in response to the data transmission speed being greater than the data transmission speed threshold, perform the scene data pulling operation again after a preset time interval; in response to the data transmission speed being less than or equal to the data transmission speed threshold, continue to perform the scene data pulling operation until the scene data corresponding to the target transmission channel is pulled.

[0140] In some embodiments, the data transmission device further includes Figure 5 The alarm unit not shown in the figure is configured to: execute an alarm process in response to the data level exceeding a preset data level threshold.

[0141] In some embodiments, the data transmission device further includes Figure 5 The filtering unit not shown in the figure is configured to: in response to the data source identifier containing a preset routing identifier, filter the scene data corresponding to the preset routing identifier.

[0142] It should be noted that the data transmission method and the data transmission device of the present application have a corresponding relationship in terms of specific implementation contents, so the repeated contents will not be described again.

[0143] Figure 6 An exemplary system architecture 600 to which the data transmission method or data transmission device according to the embodiment of the present application can be applied is shown.

[0144] like Figure 6As shown, system architecture 600 may include terminal devices 601, 602, 603, network 604 and server 605. Network 604 is used to provide a medium for communication links between terminal devices 601, 602, 603 and server 605. Network 604 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc.

[0145] Users can use terminal devices 601, 602, and 603 to interact with server 605 through network 604 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 601, 602, and 603, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social platform software, etc. (only examples).

[0146] The terminal devices 601, 602, 603 may be various electronic devices having a data transmission processing screen and supporting web browsing, including but not limited to smart phones, tablet computers, laptop computers, desktop computers, and the like.

[0147] Server 605 can be a server that provides various services, such as a background management server that provides support for data transmission requests submitted by users using terminal devices 601, 602, and 603 (only as an example). The background management server can receive data transmission requests, obtain corresponding scene data, and then determine the data source identifier and the corresponding data magnitude corresponding to the scene data; determine the data transmission channel corresponding to the scene data according to the data source identifier and the data magnitude; obtain the channel weight corresponding to each data transmission channel, and then determine the priority of each data transmission channel according to the weight of each channel; based on the priority, perform the transmission of the corresponding scene data through the corresponding data transmission channel. By pushing data of different scenes to different channels and assigning different weights to each channel, it can be ensured that data with high priority is transmitted to the database first, and other data are queued and transmitted to the database in turn, thereby improving the overall throughput of data transmission.

[0148] It should be noted that the data transmission method provided in the embodiment of the present application is generally executed by the server 605 , and accordingly, the data transmission device is generally arranged in the server 605 .

[0149] It should be understood that Figure 6 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to implementation requirements.

[0150] Reference below Figure 7 , which shows a schematic diagram of the structure of a computer system 700 of a terminal device suitable for implementing an embodiment of the present application. Figure 7The terminal device shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0151] like Figure 7 As shown, the computer system 700 includes a central processing unit (CPU) 701, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 702 or a program loaded from a storage part 708 into a random access memory (RAM) 703. In the RAM 703, various programs and data required for the operation of the computer system 700 are also stored. The CPU 701, the ROM 702, and the RAM 703 are connected to each other via a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704.

[0152] The following components are connected to the I / O interface 705: an input section 706 including a keyboard, a mouse, etc.; an output section 707 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 708 including a hard disk, etc.; and a communication section 709 including a network interface card such as a LAN card, a modem, etc. The communication section 709 performs communication processing via a network such as the Internet. A drive 710 is also connected to the I / O interface 705 as needed. A removable medium 711, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 710 as needed, so that a computer program read therefrom is installed into the storage section 708 as needed.

[0153] In particular, according to the embodiments disclosed in the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present application include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 709, and / or installed from the removable medium 711. When the computer program is executed by the central processing unit (CPU) 701, the above-mentioned functions defined in the system of the present application are executed.

[0154] It should be noted that the computer-readable medium shown in the present application may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. Computer-readable storage media may include, for example, but are not limited to electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or devices, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present application, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device. In the present application, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, in which a computer-readable program code is carried. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than computer-readable storage media, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer-readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, optical cable, RF, etc., or any suitable combination of the above.

[0155] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present application. In this regard, each box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the above-mentioned module, program segment or a part of a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flow chart, and the combination of the boxes in the block diagram or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0156] The units involved in the embodiments described in the present application may be implemented by software or hardware. The units described may also be set in a processor, for example, it may be described as: a processor includes a receiving unit, a data transmission channel determination unit, an acquisition unit, and a data transmission unit. The names of these units do not, in some cases, constitute limitations on the units themselves.

[0157] As another aspect, the present application also provides a computer-readable medium, which may be included in the device described in the above embodiment; or it may exist independently without being assembled into the device. The above computer-readable medium carries one or more programs. When the above one or more programs are executed by a device, the device includes receiving a data transmission request, obtaining corresponding scene data, and then determining the data source identifier and the corresponding data level corresponding to the scene data; determining the data transmission channel corresponding to the scene data according to the data source identifier and the data level; obtaining the channel weight corresponding to each data transmission channel, and then determining the priority of each data transmission channel according to each channel weight; based on the priority, performing the transmission of the corresponding scene data through the corresponding data transmission channel.

[0158] According to the technical solution of the embodiment of the present application, by pushing data of different scenarios to different channels and assigning different weights to each channel, it can be ensured that high-priority data is transmitted to the database first, and other data are queued and transmitted to the database in turn, thereby improving the overall throughput of data transmission.

[0159] The above specific implementations do not constitute a limitation on the protection scope of this application. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions may occur depending on design requirements and other factors. Any modifications, equivalent substitutions and improvements made within the spirit and principles of this application should be included in the protection scope of this application.

Claims

1. A data transmission method, characterized in that: include: Receiving a data transmission request, acquiring corresponding scene data, and then determining a data source identifier and a corresponding data magnitude corresponding to the scene data; Determining a data transmission channel corresponding to the scene data according to the data source identifier and the data magnitude; The determining of the data transmission channel corresponding to the scene data includes: in response to the data source identifier corresponding to the second data source, obtaining the task type identifier corresponding to the data source identifier; in response to the task type identifier corresponding to the timing and directional task, slicing the scene data according to the data magnitude to obtain each data slice; and determining the corresponding universal channel according to each data slice; Obtaining a channel weight corresponding to each of the data transmission channels, and then determining a priority of each of the data transmission channels according to each of the channel weights, wherein the greater the channel weight, the higher the priority of the data transmission channel corresponding to the channel weight; Based on the priority, transmission of the corresponding fragments of the scene data is performed through the corresponding data transmission channel.

2. The method according to claim 1, characterized in that The determining of the data source identifier corresponding to the scene data includes: Obtaining a field value corresponding to a source field in the scene data; A data source identifier corresponding to the scene data is determined according to the field value.

3. The method according to claim 1, characterized in that: The determining the data transmission channel corresponding to the scene data includes: In response to the data source identifier corresponding to the first data source, it is determined that the data transmission channel corresponding to the scene data is the first data transmission channel.

4. The method according to claim 1, characterized in that The determining the data transmission channel corresponding to the scene data includes: In response to the task type identifier corresponding to the user real-time behavior, the universal channel corresponding to the user real-time behavior is determined as the data transmission channel corresponding to the scene data.

5. The method according to claim 1, characterized in that The performing transmission of corresponding scene data through a corresponding data transmission channel based on the priority includes: Determine the data transmission channel corresponding to the highest priority as the target data transmission channel; Calculating a time period corresponding to when the corresponding scene data is pulled through the target data transmission channel, and determining a data transmission speed corresponding to the time period; Obtaining a data transmission speed threshold corresponding to the time period; In response to the data transmission speed being greater than the data transmission speed threshold, performing the scene data pulling operation again after a preset time interval; In response to the data transmission speed being less than or equal to the data transmission speed threshold, the scene data pulling operation continues to be performed until the scene data corresponding to the target data transmission channel is completely pulled.

6. The method according to claim 1, characterized in that Before determining the data transmission channel corresponding to the scene data, the method further includes: In response to the data level exceeding a preset data level threshold, an alarm process is executed.

7. The method according to claim 1, characterized in that Before determining the data transmission channel corresponding to the scene data, the method further includes: In response to the data source identifier including a preset routing identifier, the scene data corresponding to the preset routing identifier is filtered.

8. A data transmission device, characterized in that: include: A receiving unit is configured to receive a data transmission request, obtain corresponding scene data, and then determine a data source identifier and a corresponding data magnitude corresponding to the scene data; a data transmission channel determining unit, configured to determine a data transmission channel corresponding to the scene data according to the data source identifier and the data magnitude; The determining of the data transmission channel corresponding to the scene data includes: in response to the data source identifier corresponding to the second data source, obtaining the task type identifier corresponding to the data source identifier; in response to the task type identifier corresponding to the timing and directional task, slicing the scene data according to the data magnitude to obtain each data slice; and determining the corresponding universal channel according to each data slice; an acquisition unit, configured to acquire a channel weight corresponding to each of the data transmission channels, and then determine a priority of each of the data transmission channels according to each of the channel weights, wherein the greater the channel weight, the higher the priority of the data transmission channel corresponding to the channel weight; The data transmission unit is configured to perform transmission of corresponding scene data segments through corresponding data transmission channels based on the priority.

9. The device according to claim 8, characterized in that The receiving unit is further configured to: Obtaining a field value corresponding to a source field in the scene data; A data source identifier corresponding to the scene data is determined according to the field value.

10. An electronic device for data transmission, characterized in that: include: one or more processors; a storage device for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 7.

11. A computer readable medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images