Data transmission methods, apparatus, electronic devices, media and software products

By obtaining the feature values ​​of the data from the data buffer container, identifying idle processing nodes and performing modulo operations, the problem of low processing efficiency caused by large data volume is solved, and more efficient data processing is achieved.

CN116600026BActive Publication Date: 2025-12-02INDUSTRIAL AND COMMERCIAL BANK OF CHINA
View PDF 8 Cites 0 Cited by

Patent Information

Application Number
CN202310463322.9
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2023-04-26
Publication Date
2025-12-02
Estimated Expiration
2043-04-26

AI Technical Summary

Technical Problem

The large volume of data and its continuous transmission result in low efficiency for subsequent nodes in processing the data.

Method used

Data is retrieved from the data buffer container, its feature values ​​are obtained, the number of idle processing nodes is determined, and the modulo value is obtained by taking the modulo value. The data is then sent to the processing node associated with the modulo value for processing.

Benefits of technology

This reduces the amount of data processed by each processing node, improves data processing efficiency, and distributes the data processing workload among multiple processing nodes, thus reducing the processing load on a single node.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN116600026B_ABST
    Figure CN116600026B_ABST
Patent Text Reader

Abstract

This application provides a data transmission method, apparatus, electronic device, medium, and program product, belonging to the field of big data technology. The method includes: obtaining first data to be processed from a data buffer container and obtaining the feature value corresponding to the first data; determining the number of idle first processing nodes and taking the modulo of the feature value with the number to obtain a modulo value; and sending the first data to a second processing node for data processing. In this application, the data is obtained from the data buffer container, avoiding continuous data transmission to processing nodes, reducing the amount of data processed by each processing node, allowing processing nodes to have more computing resources to process data, and improving data processing efficiency. In addition, determining the processing node for processing the data based on the feature value of the data among multiple processing nodes allows multiple processing nodes to share the data processing work, reducing the amount of data processed by a single processing node, and further improving data processing efficiency.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This application relates to the field of big data technology, and in particular to a data transmission method, apparatus, electronic device, medium, and program product. Background Technology

[0002] With the development of network technology, many large-scale network systems have implemented modular architecture. Modular architecture consists of multiple units, which can be deployed in different locations. Multiple units can be combined to complete various sub-tasks within a given task.

[0003] Each unit generates a large amount of data every moment. This data is filtered, analyzed, and then stored in the database for easy querying.

[0004] In the exemplary technology, data is continuously and indiscriminately sent to subsequent nodes for processing. Due to the large amount of data and the continuous transmission of data, the amount of data processed by subsequent nodes is extremely large, which reduces the efficiency of node data processing, that is, the data processing efficiency is low. Summary of the Invention

[0005] This application provides a data transmission method, apparatus, electronic device, medium, and program product to solve the problem of low data processing efficiency.

[0006] On one hand, this application provides a data transmission method applied to a data transmission device, the data transmission method comprising:

[0007] Obtain the first data to be processed from the data buffer container, and obtain the feature value corresponding to the first data;

[0008] Determine the number of first processing nodes in the idle state, and take the modulo of the number of feature values ​​to obtain the modulo value;

[0009] The first data is sent to a second processing node for data processing. The second processing node is the first processing node associated with the modulo value.

[0010] On the other hand, this application also provides a data transmission apparatus, comprising:

[0011] The acquisition module is used to acquire the first data to be processed from the data buffer container and acquire the feature value corresponding to the first data;

[0012] A determination module is used to determine the number of first processing nodes in the idle state, and to take the modulo of the feature value by the number of nodes to obtain a modulo value.

[0013] The processing module is used to send the first data to a second processing node for data processing, wherein the second processing node is the first processing node associated with the modulo value.

[0014] On the other hand, this application also provides an electronic device, including: a processor, and a memory communicatively connected to the processor;

[0015] The memory stores computer-executed instructions;

[0016] The processor executes computer execution instructions stored in the memory to implement the method described above.

[0017] On the other hand, this application also provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement the method described above.

[0018] On the other hand, this application also provides a computer program product, including a computer program that, when executed by a processor, implements the method described above.

[0019] The data transmission method, apparatus, electronic device, medium, and program product provided in this application obtain data from a data buffer container, acquire the corresponding feature values ​​of the data, determine the number of idle processing nodes, obtain a modulo value by taking the modulo of the feature values, and then send the data to the processing node associated with the modulo value for data processing. In this application, data is obtained from a data buffer container, avoiding continuous data transmission to processing nodes, reducing the amount of data processed by each processing node, allowing processing nodes to have more computing resources to process data, and improving data processing efficiency. In addition, determining the processing node for processing data based on the feature values ​​of the data among multiple processing nodes allows multiple processing nodes to share the data processing workload, reducing the amount of data processed by a single processing node, and further improving data processing efficiency. Attached Figure Description

[0020] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0021] Figure 1 This is a schematic diagram illustrating a scenario involving the data transmission method described in this application.

[0022] Figure 2 A flowchart illustrating the first embodiment of the data transmission method provided in this application;

[0023] Figure 3 A flowchart illustrating a second embodiment of the data transmission method provided in this application;

[0024] Figure 4 A flowchart illustrating a third embodiment of the data transmission method provided in this application;

[0025] Figure 5 A flowchart illustrating the fourth embodiment of the data transmission method provided in this application;

[0026] Figure 6 This is a schematic diagram of the data transmission device of this application;

[0027] Figure 7 This is a schematic diagram of the data transmission device of this application.

[0028] The accompanying drawings illustrate specific embodiments of this application, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concept of this application to those skilled in the art through reference to particular embodiments. Detailed Implementation

[0029] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0030] With the development of network technology, many large-scale network systems have implemented modular architecture. Modular architecture consists of multiple units, which can be deployed in different locations. Multiple units can be combined to complete various sub-tasks within a given task.

[0031] Each unit generates a large amount of data every moment. This data is filtered, analyzed, and then stored in the database for easy querying.

[0032] The inventors of this application discovered that data is continuously and indiscriminately sent to subsequent nodes for processing. Due to the large amount of data and the continuous transmission of data, the amount of data processed by subsequent nodes is extremely large, resulting in low data processing efficiency.

[0033] Therefore, the inventors of this application conceived of obtaining data from a data buffer container to avoid continuously sending data to processing nodes, thereby reducing the amount of data processed by processing nodes and allowing processing nodes to have more computing resources to process data, thus improving data processing efficiency. In addition, determining the processing node for processing data among multiple processing nodes based on the feature values ​​of the data allows multiple processing nodes to share the data processing work, reducing the amount of data processed by a single processing node and further improving data processing efficiency.

[0034] Reference Figure 1 , Figure 1 This is a schematic diagram illustrating the scenario involved in the data transmission method of this application. The data transmission device 100 includes a data buffer container 110, from which the data transmission device 100 obtains data. The data transmission device 100 is communicatively connected to multiple processing nodes 200, and each processing node 200 sends its own operating information to the data transmission device 100 in real time. The data transmission device 100 can determine the idle processing nodes 200 based on the operating information. The data carries a feature value. The data transmission device 100 modulo the feature value with the number of idle processing nodes to obtain a modulo value, and then sends the data to the idle processing node 200 associated with the modulo value for data processing.

[0035] The technical solution of this application and how the technical solution of this application solves the above-mentioned technical problems are described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will now be described with reference to the accompanying drawings.

[0036] In addition, the data involved in this application can be data authorized by the user or fully authorized by all parties. The collection, dissemination and use of the data comply with the requirements of relevant national laws and regulations. The implementation methods / executives disclosed herein can be combined with each other.

[0037] It should be noted that the data transmission method, apparatus, electronic device, medium, and program product of this application can be used in the field of big data technology, as well as in any field other than big data. The application fields of the data transmission method, apparatus, electronic device, medium, and program product of this application are not limited.

[0038] Reference Figure 2 , Figure 2 This is a flowchart illustrating the first embodiment of the data transmission method of this application. The data transmission method includes the following steps:

[0039] Step S201: Obtain the first data to be processed from the data buffer container, and obtain the feature value corresponding to the first data.

[0040] In this embodiment, the execution entity is a data transmission device. For ease of description, the term "device" will be used to refer to the data transmission device below. The device can be any terminal device with data processing capabilities.

[0041] During operation, the device allocates a small memory area to create a data buffer container. This buffer container stores the received data, and the device extracts data from it for allocation, determining the processing node to which the data will be sent. This data buffer container significantly reduces the data processing load on subsequent nodes. Furthermore, the size of the data buffer container is variable, meaning its capacity can be modified based on actual needs to ensure sufficient storage of the received data.

[0042] The device retrieves data to be processed from a data buffer container; this data is defined as the first data. The device then acquires the characteristic values ​​of the first data.

[0043] In one example, when acquiring the first data, the device assigns a feature value to the first data. The feature value is a numerical value, and the feature value can be determined by the device based on the previous feature value. For example, if the previous feature value was 10, then the feature value of the first data could be 11.

[0044] In another example, the first data may consist of multiple sub-data with the same data identifier. The device obtains the data identifier of the first data and performs a preset calculation on the data identifier to obtain a feature value. For example, a hash value can be obtained by performing a hash operation on the data identifier, and the hash value can be used as the feature value.

[0045] Step S202: Determine the number of first processing nodes in the idle state, and take the modulo of the number of feature values ​​to obtain the modulo value.

[0046] The device has multiple processing nodes in its communication connection, which are defined as the third processing node. The device will determine the first processing node that is idle among the third processing nodes.

[0047] In one example, the third processing node sends its resource information to the device. The device obtains the resource information of the third processing node, including information such as the currently remaining computing resources of the third processing node. Based on the resource information, the device can determine the remaining resources of the third processing node. The device then determines whether the remaining resources of the third processing node are greater than a preset threshold. If the remaining resources are greater than the preset threshold, the third processing node is determined to be the first processing node in an idle state. That is, the third processing node with remaining resources greater than the preset threshold is determined to be the first processing node in an idle state.

[0048] In another example, the device records data sent to each third processing node. Based on the records, the device determines the first time point in time when each third processing node last performed data processing, calculates the interval between each first time point and the current time point, and identifies the third processing node corresponding to the first time point whose interval is longer than a preset time point as an idle first processing node. The preset time point is the maximum duration for a processing node to perform data processing.

[0049] After determining the first idle processing node, the device determines the number of first idle processing nodes and takes the modulo of that number to obtain a modulo value. For example, if the feature value is 10 and the number of first processing nodes is 5, then taking the modulo of 10 with 5 yields a modulo value of 0.

[0050] Step S203: The first data is sent to the second processing node for data processing. The second processing node is the first processing node associated with the modulo value.

[0051] The device configures a corresponding modulo value for each first processing node based on the number of first processing nodes. The device determines the first processing node associated with the currently determined modulo value as the second processing node; that is, the second processing node is the first processing node associated with the currently determined modulo value. For example, there are five first processing nodes in an idle state: node a, node b, node c, node d, and node e. Node a is associated with a modulo value of 0, node b with a modulo value of 1, node c with a modulo value of 2, node d with a modulo value of 3, and node e with a modulo value of 0. If the modulo value obtained by taking the modulo of the feature value is 0, then the first data is sent to the first processing node associated with the modulo value of 0.

[0052] The second processing node processes the first data, performing operations such as aggregation and feature value modification. After processing the first data, the second processing node obtains the fourth data, which is then sent to the device; that is, the device receives the fourth data sent by the second processing node. This fourth data needs to be sent to the target terminal. To reduce network bandwidth pressure, the device packages the fourth data into fifth data, which is then sent to the target terminal. Alternatively, the device can package multiple data sets to reduce network latency. For example, the device packages the previously processed data sent by the processing node along with the fourth data to obtain the fifth data.

[0053] In this embodiment, data is retrieved from a data buffer container, and the corresponding feature values ​​are obtained. The number of idle processing nodes is then determined. A modulo value is obtained by taking the modulo of the feature values, and the data is then sent to the processing node associated with the modulo value for processing. In this embodiment, data is retrieved from the data buffer container, avoiding continuous data transmission to processing nodes, reducing the amount of data processed by each node, allowing them to allocate more computing resources, and improving data processing efficiency. Furthermore, determining the processing node based on the data's feature values ​​among multiple processing nodes allows for a distribution of data processing workload across multiple nodes, reducing the amount of data processed by a single node and further improving data processing efficiency.

[0054] Reference Figure 3 , Figure 3 This is a second embodiment of the data transmission method of this application. Based on the first embodiment, before step S201, it further includes:

[0055] Step S301: Obtain multiple second data, and obtain third data with the same data identifier from each of the second data.

[0056] Step S302: Merge the third data with the same data identifier to obtain the first data, and store the first data in the data buffer container.

[0057] In this embodiment, the device acquires multiple second data sets, each of which is data generated by a unit device. The device sets routing rules to extract data with the same data identifier. Based on the routing rules, the device acquires third data sets with the same data identifier from among the second data sets.

[0058] Routing rules are configured through the configuration center. When a device detects a rule change, it will actively obtain the changed data. Alternatively, when a rule in the configuration center changes, the configuration center will actively send the changed content to the device.

[0059] Third data with the same data identifier refers to data generated by the same transaction on different unit devices. For example, a transaction includes multiple processes, each distributed across different unit devices. Each unit device generates data when executing its corresponding process, and this generated data carries a transaction identifier, which serves as the data identifier for the generated data. The device merges the various third data to obtain first data, and then stores the first data in a data buffer container so that subsequent processing nodes can process the data belonging to the same transaction.

[0060] For example, the device obtains the unique identifier field in the message, and obtains the data identifier of the message through the identifier field. The message can be a second data.

[0061] For example, the message format is {key1:value1,key2:value2,traceId}, which has a field used to distinguish unique identifiers, such as traceId. This field is used to represent a transaction, that is, traceId is a data identifier.

[0062] Furthermore, after acquiring multiple sets of second data, the device determines the total amount of data for each set of second data, and then adjusts the capacity of the data buffer container based on the total amount of data. For example, if the sum of the total amount of data for each set of second data and the remaining capacity of the data buffer container is less than the maximum capacity of the data buffer, then the data buffer container needs to be expanded to prevent the second data from being unable to be stored in the data buffer container.

[0063] In this embodiment, the device acquires multiple second data, and then acquires each third data with the same data identifier from the multiple second data. The third data are then merged to obtain the first data, which is stored in a data buffer container. This allows subsequent processing nodes to directly process the first data, avoiding the need for processing nodes to sort out and process data with the same data identifier. This reduces the processing time of the processing nodes and improves the data processing efficiency.

[0064] Reference Figure 4 , Figure 4 This is a third embodiment of the data transmission method of this application. Based on the second embodiment, step S301 includes:

[0065] Step S401: Obtain the filtering rules and filter each second data according to the filtering rules to obtain multiple third data.

[0066] Step S401: Determine the third data with the same data identifier.

[0067] In this embodiment, the device is equipped with filtering rules. The device filters each second data according to the rules to obtain multiple third data, thereby identifying each third data with the same data identifier.

[0068] The device supports setting various filtering rules, and these rules can be adjusted in real time. For example, the device has a built-in filtering rule container to store various filtering rules. Each rule is separated by a special character (such as "," or "|"). When a filtering rule is executed, it is parsed according to this special character, resulting in individual filtering rules. For instance, if the filtering rule is set to {type:1,length:200,errorTag:true}, it will be parsed into three different rules: filtering by type 1, filtering by length less than 200, and filtering by whether the error tag `errorTag` is true. The relationship between these three filtering rules can be any combination of OR and AND, and the combination of filtering rules can be configured through the device's display interface.

[0069] In this embodiment, the device reduces the pressure on the configuration center and minimizes operational risks by merging multiple rules into a single configuration. If only one rule is configured, and all rules fall under the same theme, omissions and misconfigurations are avoided. Flexible and diverse filtering rules allow users to filter data according to their desired settings. The second data filtering process filters out data that does not meet the conditions, reducing interference from illegal data with subsequent processing logic and improving the efficiency of subsequent data processing.

[0070] Reference Figure 5 , Figure 5 This is a fourth embodiment of the data transmission method of this application. Based on the second or third embodiment, step S301 includes:

[0071] Step S501: Upon receiving data subscription information, determine the node device to be monitored based on the data subscription information.

[0072] In this embodiment, the device is equipped with a data consumption function. The data consumption function includes subscription, monitoring, and consumption. Subscription refers to the device subscribing to monitor data from node devices. Monitoring refers to the device configuring message middleware to listen for data changes on node devices; these data changes can be data generated by the node devices. Consumption refers to the device transmitting the monitored data to subsequent nodes, that is, storing the monitored data in a data buffer container.

[0073] Node devices can subscribe to the device, meaning they send data subscription information to the device. This data subscription information includes the node device's identifier. When the device receives the data subscription information, it can retrieve the data identifier of the node device to be monitored from it. A node device can be a physical device corresponding to a unit or a virtual device.

[0074] Step S502: Configure the corresponding message middleware for the node device. The message middleware is used to monitor whether the data in the node device has changed.

[0075] After identifying the node device to be monitored, the device configures a corresponding message middleware for the node device. The message middleware is used to monitor whether the data in the node device has changed. For example, when the data in the node device changes, the changed data is sent to the message middleware.

[0076] Step S503: When the data of the node device is changed based on the message middleware, all the changed data of the node device are used as the second data.

[0077] When the message middleware detects a change in the data of a node device, the changed data is used as the second data. In other words, the device obtains the changed data of the node device from the message middleware as the second data.

[0078] In this embodiment, the node device subscribes to the device, enabling the device to obtain data on changes made by the node device and then process the changed data in a timely manner.

[0079] This application also provides a data transmission apparatus, referring to... Figure 6 The data transmission device 600 includes:

[0080] The acquisition module 610 is used to acquire the first data to be processed from the data buffer container and acquire the feature value corresponding to the first data;

[0081] The determination module 620 is used to determine the number of first processing nodes in the idle state and to obtain the modulo value by taking the number of feature values.

[0082] The processing module 630 is also used to send the first data to the second processing node for data processing. The second processing node is the first processing node associated with the modulo value.

[0083] In one embodiment, the acquisition module 610 includes:

[0084] The first acquisition unit is used to acquire the data identifier corresponding to the first data;

[0085] The calculation unit is used to perform preset calculations on the data identifier to obtain feature values.

[0086] In one embodiment, the data transmission device 600 includes:

[0087] The acquisition module 610 is also used to acquire multiple second data and acquire third data with the same data identifier in each of the second data;

[0088] The merging module is used to merge third data with the same data identifier to obtain first data, and store the first data in a data buffer container.

[0089] In one embodiment, the acquisition module 610 includes:

[0090] The second acquisition unit is used to acquire filtering rules and perform data filtering on each second data according to the filtering rules to obtain multiple third data.

[0091] The first determining unit is used to determine the third data that has the same data identifier.

[0092] In one embodiment, the acquisition module 610 includes:

[0093] The second determining unit is used to determine the node device to be monitored based on the data subscription information when the data subscription information is received.

[0094] The configuration unit is used to configure the corresponding message middleware for the node device. The message middleware is used to monitor whether the data in the node device has changed.

[0095] The third determining unit is used to take multiple changed data from the node device as the second data when the data of the node device is changed based on the message middleware.

[0096] In one embodiment, the acquisition module 610 includes:

[0097] The fourth determining unit is used to determine the total amount of data for each of the second data items;

[0098] The adjustment unit is used to adjust the capacity of the data buffer container according to the total amount of data.

[0099] In one embodiment, the determining module 620 includes:

[0100] The third acquisition unit is used to acquire resource information of the third processing node;

[0101] The fifth determining unit is used to determine the remaining resources of the third processing node based on the resource information;

[0102] The sixth determining unit is used to determine the third processing node as the first processing node in an idle state when the remaining resources are greater than a preset threshold.

[0103] In one embodiment, the data transmission device 600 further includes:

[0104] The receiving module is used to receive the fourth data sent by the second processing node and package the fourth data to obtain the fifth data. The fourth data is the data processed by the first processing node after processing the first data.

[0105] The sending module is used to send the fifth data to the target terminal.

[0106] Figure 7This is a hardware structure diagram of a data transmission device according to an exemplary embodiment.

[0107] The data transmission device 700 may include: a processor 71, such as a CPU, a memory 72, and a transceiver 73. Those skilled in the art will understand that... Figure 7 The structure shown does not constitute a limitation on the data transmission device and may include more or fewer components than illustrated, or combine certain components, or have different component arrangements. Memory 72 may be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0108] The processor 71 can call the computer program stored in the memory 72 or execute the computer instructions to complete all or part of the steps of the above-described data transmission method.

[0109] Transceiver 73 is used to receive information sent by external devices and to send information to external devices.

[0110] An electronic device includes: a processor, and a memory communicatively connected to the processor;

[0111] The memory stores the instructions that the computer executes;

[0112] The processor executes computer execution instructions stored in memory to implement the data transfer method as described in any of the above embodiments.

[0113] A non-transitory computer-readable storage medium, wherein when instructions (computer-executable instructions) in the storage medium are executed by a processor of a data transmission device, the data transmission device is able to perform the aforementioned data transmission method.

[0114] A computer program product includes a computer program that, when executed by a processor of a data transmission device, enables the data transmission device to perform the aforementioned data transmission method.

[0115] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the following claims.

[0116] It should be understood that this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is limited only by the appended claims.

Claims

1. A data transmission method, characterized in that, include: Upon receiving data subscription information, the node device to be monitored is determined based on the data subscription information; Configure a corresponding message middleware for the node device, and the message middleware is used to monitor whether the data in the node device has changed. When the message middleware detects a change in the data of the node device, all the changed data of the node device are used as the second data. Obtain third data with the same data identifier from each of the second data sets; the third data with the same data identifier refers to data generated by the same transaction at different nodes. Third data with the same data identifier are merged to obtain first data, and the first data is stored in a data buffer container; the capacity of the data buffer container is determined according to the total amount of each second data. Obtain the first data to be processed from the data buffer container, and obtain the feature value corresponding to the first data; Determine the number of first processing nodes in the idle state, and take the modulo of the number of feature values ​​to obtain the modulo value; The first data is sent to a second processing node for data processing. The second processing node is the first processing node associated with the modulo value.

2. The data transmission method according to claim 1, characterized in that, The step of obtaining the feature value corresponding to the first data includes: Obtain the data identifier corresponding to the first data; The feature value is obtained by performing a preset operation on the data identifier.

3. The data transmission method according to claim 2, characterized in that, The step of obtaining the third data with the same data identifier from each of the second data includes: Obtain the filtering rules, and filter each of the second data according to the filtering rules to obtain multiple third data; The third data is identified as having the same data identifier.

4. The data transmission method according to claim 2, characterized in that, After the step of acquiring multiple second data, the method further includes: Determine the total data volume of each of the second data items; Adjust the capacity of the data buffer container according to the total amount of data.

5. The data transmission method according to claim 1, characterized in that, Before the step of determining the number of first processing nodes in the idle state, the method further includes: Obtain resource information from the third processing node; The remaining resources of the third processing node are determined based on the resource information; When the remaining resources are greater than a preset threshold, the third processing node is determined to be the first processing node in an idle state.

6. The data transmission method according to any one of claims 1-5, characterized in that, After the step of sending the first data to the second processing node for data processing, the method further includes: The system receives the fourth data sent by the second processing node and packages the fourth data to obtain the fifth data, wherein the fourth data is the data processed by the second processing node after processing the first data. The fifth data is sent to the target terminal.

7. A data transmission device, characterized in that, include: The second determining unit is used to determine the node device to be monitored based on the data subscription information when receiving the data subscription information; A configuration unit is used to configure a corresponding message middleware for the node device, and the message middleware is used to monitor whether the data in the node device has changed. The third determining unit is used to, when the data of the node device is detected to have changed based on the message middleware, take multiple changed data of the node device as the second data. The acquisition module is used to acquire third data with the same data identifier from each of the second data sets; the third data with the same data identifier refers to data generated by the same transaction at different nodes; The merging module is used to merge third data with the same data identifier to obtain first data, and store the first data in a data buffer container; the capacity of the data buffer container is determined according to the total amount of data of each second data. The acquisition module is further configured to acquire the first data to be processed from the data buffer container, and acquire the feature value corresponding to the first data; A determination module is used to determine the number of first processing nodes in the idle state, and to take the modulo of the feature value by the number of nodes to obtain a modulo value. The processing module is used to send the first data to a second processing node for data processing, wherein the second processing node is the first processing node associated with the modulo value.

8. An electronic device, characterized in that, include: A processor, and a memory communicatively connected to the processor; The memory stores computer-executed instructions; The processor executes computer execution instructions stored in the memory to implement the method as described in any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the method as described in any one of claims 1-6.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the method of any one of claims 1-6.

Citation Information

Patent Citations

  • Load balancing method and system

    CN103067439A

  • Distributed task scheduling method and system and storage medium

    CN109669775A

  • Centralized task scheduling method and system and storage medium

    CN109684080A

  • Service data processing method and device, computer equipment and storage medium

    CN109815013A

  • Account checking method, account checking application cluster and related client

    CN113421163A