Multi-channel separation and distributed architecture data transmission method, device and equipment

Abstract

This application relates to a data transmission method, apparatus, and device for a multi-channel separation and distributed architecture. The method includes: a central master station retrieving raw data corresponding to a target data source from a database; the central master station performing multi-channel separation on the raw data to obtain data to be transmitted for each channel; wherein the multiple channels include an active upload channel, a business instruction channel, and an active supplementary acquisition channel; the central master station transmitting the data to be transmitted for each channel to an edge computing node through the multiple channels; the edge computing node processing the data to be transmitted to obtain processed data; the edge computing node returning the processed data to the central master station; and the central master station obtaining the data processing result based on the processed data. This method can improve the data transmission load capacity of a distributed architecture.

Description

Technical Field

[0001] This application relates to the field of data transmission technology, and in particular to a data transmission method, apparatus and device with a multi-channel separation and distributed architecture. Background Technology

[0002] With the development of computer technology, distributed technology has emerged. Distributed technology is a technology that distributes computing tasks to multiple computers or nodes for processing. Its core lies in integrating dispersed computing resources into a collaborative system through a network.

[0003] In this process, frequent data transmission is required between the central master station and the edge computing nodes, but the existing data transmission suffers from load imbalance. Summary of the Invention

[0004] Therefore, it is necessary to provide a data transmission method, apparatus, and device with a multi-channel separation and distributed architecture that can improve data transmission load capacity in response to the above-mentioned technical problems.

[0005] Firstly, this application provides a data transmission method with a multi-channel separation and distributed architecture, including:

[0006] The central main station retrieves the original data corresponding to the source of the target data from the database;

[0007] The central master station performs multi-channel separation on the raw data to obtain the data to be transmitted for each channel; among them, the multi-channel includes the active upload channel, the business instruction channel and the active supplementary acquisition channel;

[0008] The central master station transmits the data to be transmitted from each channel to the edge computing nodes through multiple channels.

[0009] The edge computing nodes process the data to be transmitted to obtain the processed data.

[0010] The processed data is returned to the central master station by the edge computing nodes;

[0011] The central main station obtains the data processing results based on the processed data.

[0012] In one embodiment, the central master station performs multi-channel separation on the raw data to obtain the data to be transmitted for each channel, including:

[0013] The central master station obtains business instructions for data processing tasks from the business instruction center.

[0014] The business instructions are parsed to obtain key information about the instructions; the key information about the instructions includes at least one of the following: instruction type, target object, and execution parameters.

[0015] Based on the key information of the instruction, the raw data is filtered, and the raw data containing the key information of the instruction is determined as the data to be transmitted corresponding to the business instruction channel.

[0016] In one embodiment, the central master station performs multi-channel separation on the raw data to obtain the data to be transmitted for each channel, including:

[0017] The central main station retrieves the actual supplementary data from the database;

[0018] The actual supplementary data is compared with the corresponding data in the original data to determine the gap data;

[0019] The gap data is identified as the data to be transmitted in the active data replenishment channel.

[0020] In one embodiment, before the central master station transmits the data to be transmitted for each channel to the edge computing node through multiple channels, the method includes:

[0021] Based on a preset instruction priority algorithm, the business instructions are analyzed for priority to obtain the first priority analysis result;

[0022] Based on the first priority analysis results, the transmission order of the data to be transmitted corresponding to the business instruction channel is determined.

[0023] Based on the collection objectives, collection conditions and collection requirements corresponding to the gap data, a priority analysis of the gap data is performed to obtain the second priority analysis results;

[0024] Based on the results of the second priority analysis, the transmission order of the gap data is determined.

[0025] In one embodiment, the central master station performs multi-channel separation on the raw data to obtain the data to be transmitted for each channel, including:

[0026] For each piece of raw data, if the data status of the raw data meets the preset data status standard, the raw data is determined as the data to be transmitted in the active upload channel.

[0027] In one embodiment, the processed data is returned to the central master station by the edge computing node, including:

[0028] The edge computing nodes determine the target compression algorithm based on the data characteristics of the processed data; these data characteristics include data type, data size, and real-time requirements.

[0029] The processed data is compressed using a target compression algorithm to obtain compressed data.

[0030] The compressed data is returned to the central main station via encrypted transmission.

[0031] In one embodiment, before the central master station retrieves the original data corresponding to the target data source from the database, the method further includes:

[0032] Acquire data processing tasks;

[0033] The central main station determines the target data source corresponding to the data processing task based on the data collection objectives and data scenarios included in the data processing task.

[0034] Secondly, this application also provides a data transmission device with a multi-channel separation and distributed architecture, comprising:

[0035] The acquisition module is used by the central main station to retrieve the original data corresponding to the target data source from the database;

[0036] The separation module is used by the central master station to separate the raw data into multiple channels to obtain the data to be transmitted for each channel; among them, the multiple channels include the active upload channel, the business instruction channel and the active supplementary acquisition channel;

[0037] The first transmission module is used to transmit the data to be transmitted from the central master station to the edge computing nodes through multiple channels.

[0038] The processing module is used to process the data to be transmitted by the edge computing nodes and obtain the processed data.

[0039] The second transmission module is used to return the processed data from the edge computing nodes to the central master station;

[0040] The output module is used by the central master station to obtain the data processing results based on the processed data.

[0041] Thirdly, this application also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of the above-described method.

[0042] Fourthly, this application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the above-described method.

[0043] Fifthly, this application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the above-described method.

[0044] The aforementioned multi-channel separation and distributed architecture data transmission method, apparatus, and equipment address the issue that data acquired through different data acquisition modes typically has different data transmission requirements. Therefore, after acquiring the raw data to be processed, the central master station performs channel separation based on the data characteristics of the active upload channel, business instruction channel, and active supplementary acquisition channel. This allows multiple channels to be used to transmit the data corresponding to each channel, enabling different types of data to be transmitted in channels that fully meet their transmission requirements. This avoids mutual interference between different data types and improves the parallelism and efficiency of data processing. Furthermore, by sending data to edge computing nodes for processing, parallel processing by multiple edge computing nodes can be achieved, further improving data processing efficiency. Therefore, the method provided in this application effectively avoids the load imbalance problem caused by excessive concentration of data streams in a single or suboptimal channel due to mixed data transmission, thereby improving data processing efficiency. Attached Figure Description

[0045] To more clearly illustrate the technical solutions in the embodiments of this application or related technologies, the drawings used in the description of the embodiments of this application or related technologies will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0046] Figure 1 This is an application environment diagram of a data transmission method with multi-channel separation and distributed architecture in one embodiment;

[0047] Figure 2 This is a flowchart illustrating a data transmission method using a multi-channel separation and distributed architecture in one embodiment.

[0048] Figure 3 This is a flowchart illustrating step 204 in one embodiment;

[0049] Figure 4 This is a schematic diagram illustrating the steps of a data transmission method using a multi-channel separation and distributed architecture in one embodiment;

[0050] Figure 5 This is an application flowchart of a data transmission method with multi-channel separation and distributed architecture in one embodiment;

[0051] Figure 6 This is a structural block diagram of a data transmission device with a multi-channel separation and distributed architecture in one embodiment;

[0052] Figure 7 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0053] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0054] It should be noted that the terms "first," "second," etc., used in this application can be used to describe various elements, but these elements are not limited by these terms. These terms are only used to distinguish the first element from the second element. The terms "comprising" and "having," and any variations thereof, used in this application, are intended to cover non-exclusive inclusion. The term "multiple" used in this application refers to two or more. The term "and / or" used in this application refers to one of the embodiments, or any combination of multiple embodiments.

[0055] The data transmission method for multi-channel separation and distributed architecture provided in this application embodiment can be applied to, for example... Figure 1 In the application environment shown, terminal 102 communicates with server 104 via a network. A data storage system can store the data that server 104 needs to process. The data storage system can be integrated onto server 104, or it can be located in the cloud or on another network server.

[0056] Data processing tasks can be initiated through terminal 102. The physical server in server 104 that receives the data processing tasks serves as the central master station. Other physical servers and terminals can serve as edge computing nodes. After receiving the data processing task, the central master station performs data processing by executing the method provided in this application and returns the data processing results to terminal 102.

[0057] Terminal 102 can be, but is not limited to, various personal computers, laptops, smartphones, tablets, drones, low-altitude aircraft, IoT devices, and portable wearable devices. IoT devices can include smart speakers, smart TVs, smart air conditioners, smart in-vehicle devices, and projection equipment. Portable wearable devices can include smartwatches, smart bracelets, and head-mounted displays. Head-mounted displays can be virtual reality (VR) devices, augmented reality (AR) devices, and smart glasses. Server 104 can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing cloud computing services.

[0058] In one exemplary embodiment, such as Figure 2 As shown, a data transmission method with multi-channel separation and distributed architecture is provided, which can be applied to... Figure 1Taking the server in the example, the explanation includes the following steps 202 to 212. Wherein:

[0059] Step 202: The central main station retrieves the original data corresponding to the target data source from the database.

[0060] The central master station and edge computing nodes together constitute the distributed computing network; furthermore, the central master station is the node in the distributed computing network that undertakes core coordination, management and control functions.

[0061] The database stores various types of raw data; furthermore, the database also stores data element information of the raw data; the data element information may include the collection target of the raw data, the application scenario of the raw data, etc.; among them, the data collection target indicates the data collection requirements, and the application scenario indicates the application direction of the data.

[0062] The target data source refers to the range of data determined for the current data processing task.

[0063] In some embodiments, the source of the target data can be determined based on the data acquisition target and data application scenario of the current data processing task; for example, if the current data processing task is a UAV flight data processing task, the acquisition target is the coordinates of the UAV, and the application scenario is flight trajectory planning, then the original data with the acquisition target being the coordinates of the UAV and the application scenario being flight trajectory planning can be obtained from the database. Obviously, the original data obtained is also the original data required for the UAV flight data processing task.

[0064] Raw data refers to the initial data that is directly acquired during the data acquisition process and is not filtered, cleaned, or aggregated. In this embodiment of the application, all types of raw data are stored in the database. Obviously, for the current data processing task, it is not necessary to use all the raw data in the database. Therefore, the target data source can be determined through the data processing task, and then the required raw data can be obtained from the database through the target data source.

[0065] Step 204: The central master station performs multi-channel separation on the raw data to obtain the data to be transmitted for each channel; among which, the multi-channel includes the active upload channel, the business instruction channel and the active supplementary acquisition channel.

[0066] The active upload channel is a data channel established for transmitting actively uploaded data. Actively uploaded data refers to data that is automatically transmitted to the target end (such as a server, database, or application) spontaneously, periodically, or based on specific conditions from the data source end (such as a terminal device) without external request. Examples include speed reported in real time by a speed sensor and temperature reported in real time by a temperature sensor.

[0067] A business instruction channel is a data transmission channel established for data obtained through business instructions. A business instruction is an instruction issued based on the needs of the current data processing task to obtain specific data; for example, historical transaction records obtained in response to a query instruction.

[0068] The active data acquisition channel is a data transmission channel established for data acquired through active data acquisition. Active data acquisition refers to a data reacquisition mechanism initiated by the data acquisition management terminal (e.g., the central master station in this embodiment) when the actively uploaded data is incomplete or missing due to certain reasons (such as network interruption, equipment failure, data loss, or failure to report on time).

[0069] Understandably, proactively uploaded data is characterized by high frequency, low value density, and continuity, and its transmission requirements focus on low-latency, high-throughput channels to avoid data backlog. Data acquired through business instructions is characterized by directional, high-value density, and on-demand triggering, and its transmission requirements emphasize reliable channels with precise response and bidirectional interaction to ensure data integrity and consistency. Data acquired through proactive supplementary collection is characterized by asynchronous and small-batch characteristics, and its transmission requirements focus on flexible scheduling and fault-tolerant recovery elastic channels to adapt to network fluctuations or data source anomalies.

[0070] Clearly, due to significant differences in real-time requirements, data volume, and transmission priority among various types of data, mixed transmission could lead to high-priority data being blocked by low-value data, or large volumes of data consuming bandwidth. Therefore, by separating the original data into dedicated channels and constructing dedicated transmission channels for different types of data, data contention can be avoided, thus achieving efficient data transmission.

[0071] In some embodiments, the specific transmission channel to which the original data is applicable can be distinguished based on the data characteristics of the original data, thereby achieving channel separation of the original data. For example, data obtained according to business instructions may contain key information related to the business instructions, such as the identifier of the business instructions, specific response fields, etc. Therefore, it can be determined whether the original data needs to be transmitted through the business instruction channel based on the presence or absence of key information related to the business instructions. As another example, actively uploaded data usually has a fixed data format, so actively uploaded data can be identified by regular expression matching, etc.

[0072] Step 206: The central master station transmits the data to be transmitted to the edge computing nodes through multiple channels.

[0073] There can be multiple edge computing nodes; edge computing nodes are used to perform predefined tasks such as data parsing, decoding, cleaning and feature extraction on the data to be processed.

[0074] In some embodiments, an edge computing node may have only one transmission channel or multiple transmission channels between it and the central master station. When there is only one transmission channel between the edge computing node and the central master station, it means that the edge computing node can only undertake specific data processing tasks.

[0075] In some embodiments, the central master station can first mark the data to be transmitted on each channel. Different transmission channels correspond to different data marks. When transmitting data, the corresponding transmission channel can be determined according to the data mark of the data to be transmitted.

[0076] In other embodiments, considering that the same type of data can be processed by different edge computing nodes, the data to be transmitted corresponding to the same channel can also be fragmented, so as to facilitate the transmission of different fragments to different edge computing nodes for processing.

[0077] Step 208: The edge computing node processes the data to be transmitted to obtain the processed data.

[0078] The data processing performed by edge computing nodes is predefined; for example, it may include data parsing, decoding, cleaning, and feature extraction.

[0079] Step 210: The edge computing nodes return the processed data to the central master station.

[0080] It is understandable that during the process of transmitting processed data from edge computing nodes to the central master station, the data is still transmitted through multiple channels.

[0081] Step 212: The central main station obtains the data processing result based on the processed data.

[0082] For example, the central main station can use big data analytics algorithms to perform trend prediction and behavior analysis based on the processed data, thereby obtaining the data processing results.

[0083] In the aforementioned multi-channel separation and distributed architecture data transmission method, since data obtained through different data acquisition modes typically have different data transmission requirements, after the central master station acquires the raw data to be processed, it performs channel separation on the raw data according to the data characteristics corresponding to the active upload channel, business instruction channel, and active supplementary acquisition channel. This allows multiple channels to be used to transmit the data corresponding to each channel separately, enabling different types of data to be transmitted in channels that fully meet their transmission requirements. This avoids mutual interference between different data and improves the parallelism and efficiency of data processing. At the same time, by sending the data to edge computing nodes for processing, parallel processing by multiple edge computing nodes can be achieved, improving the efficiency of data processing. Therefore, the method provided in this application can effectively avoid the load imbalance problem caused by excessive concentration of data streams in a single or suboptimal channel due to the mixed transmission of different data, thereby improving the efficiency of data processing.

[0084] In one exemplary embodiment, such as Figure 3 As shown, step 204 includes steps 302 to 306, wherein:

[0085] Step 302: The central master station obtains the business instructions corresponding to the data processing task from the business instruction center.

[0086] The business instruction center stores historical business instructions. Furthermore, the business instructions corresponding to data processing tasks can be determined by timestamp matching. For example, if the initiation time of a data processing task is the first time, and the distance between the initiation time of a business instruction and the first time is less than a preset time length, then the business instruction is considered to be the business instruction corresponding to the data processing task.

[0087] Step 304: Parse the business instruction to obtain key instruction information; the key instruction information includes at least one of the following: instruction type, target object, and execution parameters.

[0088] The instruction type refers to the way a business instruction works. Specifically, instruction types can include query, control, configuration, and interaction types.

[0089] The target object refers to the target of the business instruction, such as a business instruction issued to device a, a business instruction issued to device b, etc.

[0090] Execution parameters refer to the specific parameter conditions included in a business instruction.

[0091] In some embodiments, instruction parsing can be achieved through methods such as field extraction, keyword extraction, and regular expression extraction.

[0092] Step 306: Based on the key information of the instruction, the raw data is filtered, and the raw data containing the key information of the instruction is determined as the data to be transmitted corresponding to the business instruction channel.

[0093] It is understandable that the data obtained through business instructions is essentially the response data of the business instructions, and therefore it must contain content related to the business instructions. Obviously, if the original data contains key information of the instructions, it means that the original data is likely the response data of the business instructions, and the original data can be identified as the data to be transmitted corresponding to the business instruction channel.

[0094] In an exemplary embodiment, step 204 includes steps 402 to 406, wherein:

[0095] Step 402: The central main station retrieves the actual supplementary data from the database.

[0096] It should be noted that in the database, the supplementary data and the original data are stored separately; after the central master station obtains the original data corresponding to the source of the target data from the database, it can obtain the corresponding actual supplementary data based on the supplementary data record corresponding to the obtained original data.

[0097] It is worth mentioning that the actual supplementary data collection is initiated by the central station when it detects anomalies in the original data. When initiating supplementary data collection, the collection targets, conditions, and requirements of the supplementary data can be prioritized, which helps to rationally arrange collection tasks, improve collection efficiency, and ensure that key data can be collected first.

[0098] Step 404: Compare the actual supplementary data with the corresponding data in the original data to determine the gap data.

[0099] Specifically, after retrieving the actual supplementary data from the database, the actual supplementary data is compared with the original data to identify data in the actual supplementary data that is inconsistent with the original data, and the inconsistent data is marked as gap data.

[0100] Step 406: The gap data is identified as the data to be transmitted in the active data acquisition channel.

[0101] Clearly, gap data refers to data in the original data that contains anomalies.

[0102] In the above embodiments, by comparing the original data with the actual supplementary data, inconsistent screening data, i.e. gap data, can be identified and marked in a timely manner, which helps to ensure the accuracy and integrity of the data and reduce the impact of data errors and omissions on subsequent analysis and decision-making. At the same time, transmitting only the gap data makes the transmitted data more targeted, enabling the accurate acquisition of the required data and avoiding the waste of resources caused by blindly transmitting the actual supplementary data.

[0103] In an exemplary embodiment, steps 502 to 508 are included before step 206, wherein:

[0104] Step 502: Perform priority analysis on the business instructions based on the preset instruction priority algorithm to obtain the first priority analysis result.

[0105] The first priority analysis results include the priorities corresponding to various business instructions.

[0106] The preset instruction priority algorithm is pre-set; for example, different instruction types and different target objects can be preset. Then, the weighted score is obtained by weighting the instruction type and target object of the business instruction. The higher the weighted score, the more important the business instruction is, and the higher the transmission priority of the corresponding business instruction data.

[0107] Step 504: Based on the first priority analysis results, determine the transmission order of the data to be transmitted corresponding to the service instruction channel.

[0108] Specifically, the higher the priority of a business instruction, the higher the priority of its corresponding data transmission.

[0109] Step 506: Based on the collection target, collection conditions and collection requirements corresponding to the gap data, perform priority analysis on the gap data to obtain the second priority analysis result.

[0110] The second priority analysis results include the priority corresponding to each gap data.

[0111] The collection objective of gap data refers to the core business objective that needs to be achieved by supplementing the gap data; the collection conditions are the environmental or resource constraints that must be met when performing supplementary data collection, such as equipment online status and network bandwidth; the collection requirements refer to the specific standards for supplementary data collection, such as collection accuracy, time granularity, and completeness.

[0112] In some embodiments, different scores corresponding to different collection targets, collection conditions and collection requirements can be preset. Then, the corresponding scores are weighted according to the collection targets, collection conditions and collection requirements corresponding to the gap data to obtain a weighted score, which is used as the priority score of the gap data.

[0113] Step 508: Based on the second priority analysis results, determine the transmission order of the gap data.

[0114] In the above embodiments, a preset instruction priority algorithm is applied to prioritize the data to be transmitted corresponding to the business instruction channel, enabling the system to flexibly schedule according to the importance and urgency of the task; high-priority data can be processed first, which helps to ensure the timely execution of critical business and improve the overall performance and response speed of the system; at the same time, priority analysis of the gap data based on the collection target, collection conditions and collection requirements corresponding to the gap data helps to ensure that critical data can be processed first.

[0115] In one exemplary embodiment, step 206 may further include the step of:

[0116] For each piece of raw data, if the data status of the raw data meets the preset data status standard, the raw data is determined as the data to be transmitted in the active upload channel.

[0117] The preset data status standard is defined based on the data characteristics of the actively uploaded data. For example, if the actively uploaded data has a fixed data format, the preset data status standard can be defined as the fixed data format; if the actively uploaded data has a specific data status, the preset data status standard can be defined as conforming to the specific status, etc.

[0118] In an exemplary embodiment, step 210 may further include steps 602 to 606:

[0119] Step 602: The edge computing node determines the target compression algorithm based on the data characteristics of the processed data; whereby the data characteristics include data type, data size, and data real-time requirements.

[0120] In some embodiments, multiple compression algorithms and the correspondence between multiple compression algorithms and data features are pre-defined. Edge computing nodes can perform data parsing on the processed data to determine the data features. Then, by querying the correspondence between multiple compression algorithms and data features, the target compression algorithm is determined, and the target compression algorithm is called to compress the data.

[0121] Step 604: Compress the processed data using the target compression algorithm to obtain compressed data.

[0122] In some embodiments, the compression ratio and compression speed can be monitored in real time during the compression process. If the compression ratio does not meet the requirements, the algorithm parameters can be adjusted or the algorithm can be replaced, thereby ensuring that a good compression effect is always obtained and improving the flexibility and adaptability of the solution.

[0123] Step 606: The compressed data is returned to the central master station via encrypted transmission.

[0124] In some embodiments, asymmetric encryption technology can be used to encrypt the compressed data; wherein, asymmetric encryption refers to an encryption method that converts the data into ciphertext form, which can only be decrypted by the recipient who has the corresponding private key.

[0125] In the above embodiments, the compression algorithm is selected and its parameters are adjusted and optimized according to the data type, size and real-time requirements. This can ensure the compression ratio while taking into account the characteristics of different data, achieving efficient data compression and reducing data storage space and transmission bandwidth. At the same time, encryption technology is used to encrypt the compressed data, which effectively protects the security and confidentiality of the data and prevents the data from being stolen or tampered with during transmission.

[0126] In an exemplary embodiment, prior to step 202, the data transmission method for a multi-channel separation and distributed architecture further includes steps 702 to 704:

[0127] Step 702: Obtain the data processing task.

[0128] The data processing tasks can be user-inputted or initiated proactively by the central main station based on specific conditions.

[0129] Step 704: The central main station determines the target data source corresponding to the data processing task based on the data collection targets and data scenarios included in the data processing task.

[0130] In some embodiments, the data acquisition target and the data scenario can be combined to obtain the target data source.

[0131] In the above embodiments, the source of target data is determined by the data collection objectives and data scenarios included in the data processing task, ensuring that the collected data is closely integrated with actual needs, avoiding the collection of irrelevant or redundant data, and improving the targeting and effectiveness of data collection.

[0132] In some embodiments, please refer to Figure 4 , Figure 4 The diagram illustrates the steps of a data transmission method with a multi-channel separation and distributed architecture provided in this application embodiment, including:

[0133] Step 1: Collect raw data and then filter the data.

[0134] Step 2: Perform multi-channel separation and scheduling on the raw data.

[0135] Step 3: Perform distributed node collaborative processing on the scheduled data.

[0136] Step four involves compressing and encrypting the collaboratively processed data, and then transmitting the encrypted data.

[0137] Step 5: Secure the transmitted data.

[0138] Further, please refer to Figure 5 , Figure 5 This paper illustrates an application flowchart of a data transmission method with a multi-channel separation and distributed architecture provided in an embodiment of this application, including:

[0139] Data Acquisition: Determine the target data source based on the data acquisition objectives and application scenarios included in the data processing task. Then, collect raw data from the database according to the target data source. Of course, data preprocessing can also be performed on the raw data, such as data cleaning and transformation, noise removal, and duplicate value removal, to improve the cleanliness and usability of the data.

[0140] Identification of Normal and Abnormal Data: The collected raw data is matched using predefined normal data rules. If the matching result meets the predefined rules, the preprocessed data is considered normal data; otherwise, it is considered abnormal data. Finally, the normal and abnormal data are labeled, resulting in labeled raw data. Labeled raw data accurately identifies normal and abnormal data, promptly detecting anomalies and improving data accuracy and reliability, thus facilitating subsequent data analysis and processing.

[0141] Multi-channel separation scheduling: Specifically, raw data is separated and scheduled through acquisition channels, command channels, and supplementary channels, enabling different types of data to be transmitted and processed in their respective suitable channels. This avoids data mixing and interference, and improves the parallelism and efficiency of data processing. The acquisition channel is the task-initiated upload channel, the command channel is the business instruction channel, and the supplementary channel is the master station's proactive supplementary acquisition channel.

[0142] For example, preset anomaly judgment rules and key state identification standards are read from the database, and the read preset anomaly judgment rules and key state identification standards are parsed. Execution conditions are transformed based on the parsing results. The filtered raw data is then filtered according to the transformed execution conditions. Data that does not conform to the anomaly judgment rules and key state identification standards is obtained after data filtering and removed, resulting in first configuration data. This first configuration data is used as the transmission data for the task actively uploading channel. The business instruction channel first receives control instructions from the instruction control center and parses the received control instructions, obtaining key information in the control instructions, including instruction type, target object, and execution parameters. Based on the key information in the control instructions, data extraction is performed on the filtered raw data, where the extracted data is consistent with the key information in the control instructions. A preset priority scheduling algorithm is applied within the business instruction channel to prioritize the extracted data. The process involves several steps: First, the data is hierarchically divided. Based on the division results, the extracted data is sorted to form a priority-based instruction queue. This queue is designated as the second configuration data, which serves as the transmission data for the business instruction channel. Second, actual supplementary data is retrieved from the database and compared with the filtered data. Data discrepancies are identified and marked as gap data. The scope of supplementary data is determined based on the gap data, including its time interval, data source, and data type. This scope is then broken down to obtain the collection target, conditions, and requirements for each data point. Priorities are assigned to each data point based on these criteria. After prioritization, the third configuration data is obtained and serves as the transmission data for the main station's active supplementary data channel. Finally, the first, second, and third configuration data constitute the scheduled data.

[0143] Distributed node collaborative processing: Specifically, the first configuration data, the second configuration data, and the third configuration data are transmitted to the edge computing nodes through different channels for preliminary processing; on each edge computing node, predefined data processing tasks are performed, including data parsing, data decoding, data cleaning, and feature extraction.

[0144] Distributed node collaborative processing consists of two parts. The first part is as follows:

[0145] The compressed data is encrypted and fragmented: the compression ratio can be set to greater than 5:1; data encryption is achieved by generating a pair of asymmetric encryption keys, including a public key and a private key, and using the public key to encrypt the compressed data, converting the data into ciphertext; the encrypted data is fragmented according to fragmentation rules, including data size, data type, or time order; and each data fragment is given a unique identifier.

[0146] Data transmission is performed; the data processed by the edge computing nodes is transmitted to the central master station.

[0147] The transmitted data is decrypted as follows: The central master station receives data fragments transmitted from different paths and verifies each data fragment according to its unique identifier, including integrity and accuracy verification; according to the identifier and fragmentation rules of the data fragments, the received data fragments are reassembled and encrypted and compressed to restore the data; the reassembled data is then decrypted by using the generated private key to convert the ciphertext into plaintext.

[0148] After decryption, a dynamic distribution strategy is formulated: Specifically, the sensitivity and usage frequency of the data after decryption are analyzed, and a dynamic distribution strategy is formulated in combination with the security level, storage capacity and performance of the storage nodes. The formulated dynamic distribution strategy is then optimized based on the real-time status of the storage nodes and the data usage.

[0149] The data is stored across multiple different storage nodes, and index information is added: Specifically, based on a dynamic distribution strategy, the decrypted data is distributed and stored across multiple different storage nodes, and corresponding index information is added to the data portion of each storage node.

[0150] Detecting data loss and requesting retransmission: When receiving or retrieving data, the central master station detects whether the data is lost by comparing the data's index information and storage status. If data loss is detected, the scope and identification information of the lost data are confirmed, and the lost data fragments are located by querying the data's index information and transmission logs. The lost data is then marked as data to be retransmitted.

[0151] The second part is as follows:

[0152] Data processing is performed on the data on the edge computing nodes: Specifically, on each edge computing node, predefined data processing tasks are performed, including data parsing, data decoding, data cleaning, and feature extraction.

[0153] Storing in fast memory: Specifically, the processed data is temporarily stored in the fast memory of the edge computing node.

[0154] The central master station receives data from the memory: Specifically, the central master station receives preliminary processed data and status information from each edge computing node through the communication link.

[0155] Data aggregation: Specifically, data transmitted from different edge computing nodes is aggregated, including data integration and phased data aggregation.

[0156] Configure accordingly and set the time synchronization server and synchronization period: Specifically, use the time synchronization protocol of the multi-node environment to configure the edge computing nodes and the central master station accordingly, and set the time synchronization server and synchronization period; among them, each node actively requests time synchronization from the time synchronization server to obtain an accurate time reference and align its own clock with the server clock.

[0157] Time calibration: Specifically, during data processing, nodes periodically calibrate their time with the time synchronization server according to a set synchronization cycle. When a time deviation exceeds a threshold, the node clock is automatically adjusted.

[0158] Trend prediction and behavior analysis: Specifically, big data analytics algorithms are used to predict trends and analyze behavior in the data after it has been processed collaboratively by distributed nodes, and the analysis results are generated in real time to obtain the data processing results.

[0159] It should be understood that although the steps in the flowcharts of the above embodiments are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the above embodiments may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages in other steps. It is understood that the steps in different embodiments can be freely combined as needed, and all non-contradictory solutions formed by such combinations are within the scope of protection of this application.

[0160] Based on the same inventive concept, this application also provides a data transmission apparatus for implementing the data transmission method of the multi-channel separation and distributed architecture described above. The solution provided by this apparatus is similar to the implementation scheme described in the above method. Therefore, the specific limitations of one or more embodiments of the data transmission apparatus for the multi-channel separation and distributed architecture provided below can be found in the limitations of the data transmission method for the multi-channel separation and distributed architecture described above, and will not be repeated here.

[0161] In one exemplary embodiment, such as Figure 6 As shown, a data transmission device with a multi-channel separation and distributed architecture is provided, comprising:

[0162] The acquisition module is used by the central main station to retrieve the original data corresponding to the target data source from the database.

[0163] The separation module is used by the central master station to separate the raw data into multiple channels to obtain the data to be transmitted for each channel; among them, the multiple channels include the active upload channel, the business instruction channel and the active supplementary acquisition channel.

[0164] The first transmission module is used by the central master station to transmit the data to be transmitted to the edge computing nodes through multiple channels.

[0165] The processing module is used to process the data to be transmitted by the edge computing nodes and obtain the processed data.

[0166] The second transmission module is used by the edge computing nodes to return the processed data to the central master station.

[0167] The output module is used by the central master station to obtain the data processing results based on the processed data.

[0168] In one embodiment, the separation module is specifically used to: obtain business instructions for data processing tasks from the business instruction center by the central master station; parse the business instructions to obtain key instruction information; the key instruction information includes at least one of instruction type, target object, and execution parameters; and filter the raw data based on the key instruction information to determine the raw data containing the key instruction information as the data to be transmitted corresponding to the business instruction channel.

[0169] In one embodiment, the separation module is specifically used to: obtain actual supplementary data from the database by the central master station; compare the actual supplementary data with the corresponding data in the original data to determine the gap data; and determine the gap data as the data to be transmitted in the active supplementary data channel.

[0170] In one embodiment, the data transmission device of the multi-channel separation and distributed architecture further includes a sorting module, which is used to perform priority analysis on the business instructions based on a preset instruction priority algorithm to obtain a first priority analysis result; determine the transmission order of the data to be transmitted corresponding to the business instruction channel based on the first priority analysis result; perform priority analysis on the gap data based on the acquisition target, acquisition conditions and acquisition requirements corresponding to the gap data to obtain a second priority analysis result; and determine the transmission order of the gap data based on the second priority analysis result.

[0171] In one embodiment, the separation module is specifically used to determine the original data as data to be transmitted in the active uploading channel if the data state of the original data meets the preset data state standard.

[0172] In one embodiment, the second transmission module is specifically used to: have the edge computing node determine the target compression algorithm based on the data characteristics of the processed data; wherein the data characteristics include data type, data size, and data real-time requirements; use the target compression algorithm to compress the processed data to obtain compressed data; and return the compressed data to the central main station through encrypted transmission.

[0173] In one embodiment, the data transmission device with multi-channel separation and distributed architecture further includes a sorting module and a source confirmation module, used to obtain data processing tasks; the central master station determines the target data source corresponding to the data processing task based on the data acquisition target and data scenario included in the data processing task.

[0174] Each module in the aforementioned multi-channel separation and distributed data transmission device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the operations corresponding to each module.

[0175] In one exemplary embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 7As shown, this computer device includes a processor, memory, input / output interfaces (I / O), and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is also connected to the system bus via the I / O interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides the environment for the operating system and computer programs stored in the non-volatile storage media. The database stores various types of data involved in data processing tasks. The I / O interfaces are used for exchanging information between the processor and external devices. The communication interface is used for communicating with external terminals via a network connection. When the computer program is executed by the processor, it implements a data transmission method with a multi-channel, discrete, and distributed architecture.

[0176] Those skilled in the art will understand that Figure 7 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0177] In one exemplary embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of the method described above.

[0178] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the steps of the above-described method.

[0179] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps of the method described above.

[0180] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data must comply with relevant regulations.

[0181] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, artificial intelligence (AI) processors, etc., and are not limited to these.

[0182] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0183] The above embodiments are merely illustrative of several implementation methods of this application, and their descriptions are relatively specific and detailed. However, they should not be construed as limiting the scope of this application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A data transmission method with multi-channel separation and distributed architecture, characterized in that, The method includes: The central main station retrieves the original data corresponding to the source of the target data from the database; The central master station performs multi-channel separation on the raw data to obtain the data to be transmitted for each channel; wherein, the multi-channel includes an active upload channel, a service instruction channel, and an active supplementary acquisition channel; The central master station transmits the data to be transmitted corresponding to each channel to the edge computing node through the multiple channels; The edge computing node processes the data to be transmitted to obtain processed data. The edge computing node returns the processed data to the central master station; The central master station obtains the data processing result based on the processed data.

2. The method according to claim 1, characterized in that, The process of separating the raw data into multiple channels by the central master station to obtain the data to be transmitted for each channel includes: The central master station obtains the business instructions corresponding to the data processing task from the business instruction center; The business instruction is parsed to obtain key instruction information; the key instruction information includes at least one of the following: instruction type, target object, and execution parameters. Based on the key information of the instruction, the raw data is filtered, and the raw data containing the key information of the instruction is determined as the data to be transmitted corresponding to the business instruction channel.

3. The method according to claim 2, characterized in that, The process of separating the raw data into multiple channels by the central master station to obtain the data to be transmitted for each channel includes: The central master station retrieves the actual supplementary data from the database; The actual supplementary data is compared with the corresponding data in the original data to determine the gap data; The gap data is determined as the data to be transmitted in the active data acquisition channel.

4. The method according to claim 3, characterized in that, Before the central master station transmits the data to be transmitted corresponding to each channel to the edge computing node through the multiple channels, the method includes: The service instructions are analyzed for priority based on a preset instruction priority algorithm to obtain the first priority analysis result. Based on the first priority analysis result, the transmission order of the data to be transmitted corresponding to the service instruction channel is determined; Based on the collection targets, collection conditions and collection requirements corresponding to the gap data, a priority analysis of the gap data is performed to obtain the second priority analysis result; Based on the results of the second priority analysis, the transmission order of the gap data is determined.

5. The method according to any one of claims 1-4, characterized in that, The process of separating the raw data into multiple channels by the central master station to obtain the data to be transmitted for each channel includes: For each piece of raw data, if the data state of the raw data meets the preset data state standard, the raw data is determined as the data to be transmitted by the active upload channel.

6. The method according to claim 1, characterized in that, The step of the edge computing node returning the processed data to the central master station includes: The edge computing node determines the target compression algorithm based on the data characteristics of the processed data; wherein, the data characteristics include data type, data size, and data real-time requirements; The processed data is compressed using the target compression algorithm to obtain compressed data; The compressed data is returned to the central master station via encrypted transmission.

7. The method according to claim 1, characterized in that, Before the central master station retrieves the original data corresponding to the target data source from the database, the method further includes: Acquire data processing tasks; The central master station determines the target data source corresponding to the data processing task based on the data collection targets and data scenarios included in the data processing task.

8. A data transmission device with a multi-channel separation and distributed architecture, characterized in that, The device includes: The acquisition module is used by the central main station to retrieve the original data corresponding to the target data source from the database; The separation module is used by the central master station to perform multi-channel separation on the original data to obtain the data to be transmitted corresponding to each channel; wherein, the multi-channel includes an active upload channel, a service instruction channel, and an active supplementary acquisition channel; The first transmission module is used to transmit the data to be transmitted corresponding to each channel to the edge computing node through the multi-channel system by the central master station. The processing module is used to process the data to be transmitted by the edge computing node to obtain the processed data; The second transmission module is used to return the processed data from the edge computing node to the central master station; The output module is used by the central master station to obtain the data processing result based on the processed data.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 7.

Images