Distributed informatization data transmission method for digital visual platform

Abstract

The invention relates to the technical field of distributed informatization data transmission, in particular to a distributed informatization data transmission method for a digital visual platform, which comprises the following steps: firstly, acquiring historical transmission environment data, establishing a data transmission congestion degree evaluation index, judging whether a data transmission environment is a crowded data transmission environment according to the data transmission congestion degree evaluation index, and if yes, transmitting the data transmission environment; then data transmission is carried out; if the transmission fails, the data is retransmitted through a self-healing mechanism; after the data transmission is completed, the edge node preprocesses the received data according to the data requirement of the digital visual platform to obtain the preprocessed data, and stores the preprocessed data to the edge storage node; and the client node receives the data transmitted by the edge storage node according to the edge data flow control mechanism, and visually monitors the transmission state. According to the method, efficient and stable data transmission of the digital visual platform is realized by introducing a data transmission congestion degree evaluation index, a data priority and an edge data flow control mechanism.

Description

Technical Field

[0001] The present invention relates to the technical field of distributed information data transmission, and in particular to a distributed information data transmission method for a digital visual platform. Background Art

[0002] Digital visual platform refers to the use of data analysis, data visualization technology, artificial intelligence and deep learning and other digital technologies, as well as human-computer interaction interface, to achieve intelligent data processing and visual display. Its main function is to enable users to understand data more intuitively through data visualization, intelligent processing and analysis, and provide auxiliary information for decision-making and management. It is widely used in smart manufacturing, smart cities, medical health and financial services.

[0003] Digital visualization platforms usually involve massive amounts of data from multiple sources and require real-time, stable and efficient transmission. Existing data transmission methods prolong the transmission time when the data volume is large or the network is complex, affecting the real-time monitoring and response capabilities of the visualization platform. On the other hand, in a cross-regional multi-data source environment, the traditional centralized architecture transmission method is difficult to support large-scale distributed data synchronization.

[0004] These defects limit the real-time and stability of the digital visual platform. In order to solve these problems and improve the working efficiency of the digital visual platform, the modern digital visual platform needs to use distributed information data transmission methods, combined with edge computing, big data, dynamic load balancing and encrypted transmission technologies, to establish a system that can transmit stably, in real time and efficiently, thereby improving the data transmission capabilities of the digital visual platform.

[0005] Therefore, a distributed information data transmission method for digital visual platform is proposed. Summary of the invention

[0006] The purpose of the present invention is to provide a distributed information data transmission method for a digital visual platform. First, a distributed database node is set up to store the data to be transmitted, and then the data transmission environment is evaluated by establishing a data transmission congestion evaluation index, and different modes of data transmission are performed according to different transmission environments; then the edge node pre-processes the received data and stores it in the edge storage node, and the client node obtains data through the edge data flow control mechanism according to its own needs.

[0007] To achieve the above object, the present invention provides the following technical solutions:

[0008] A distributed information data transmission method for a digital visual platform, comprising:

[0009] The data receiving node acquires and processes the data collected by the sensor to obtain the collected data, and stores the collected data in the distributed database node;

[0010] Furthermore, the distributed database includes:

[0011] Acquire the collected data and clean it to obtain data to be transmitted, wherein the cleaning includes outlier filtering and missing value filling;

[0012] The data information of the data to be transmitted is obtained and processed to obtain the data priority; the data information includes real-time requirements and data importance; the calculation formula of the data priority is:

[0013]

[0014] Among them, P represents the data priority of the data to be transmitted, α1 represents the weight coefficient of the real-time requirement score T, α2 represents the weight coefficient of the data importance score I, α3 represents the weight coefficient of the data freshness score R, α4 represents the weight coefficient of the data type response frequency F, α5 represents the weight coefficient of the data volume S, and α6 represents the weight coefficient of the expected bandwidth consumption B.

[0015] Acquire and process historical transmission environment data to obtain a data transmission congestion evaluation index, and determine whether it is a congested data transmission environment according to the data transmission congestion evaluation index;

[0016] Furthermore, the data transmission congestion evaluation index includes:

[0017] The historical transmission environment data includes the transmission delay, data packet loss rate, bandwidth utilization, retransmission rate, number of connections, data volume and request frequency within t0 time before the current data transmission;

[0018] If the data transmission congestion evaluation index is less than or equal to the preset data transmission congestion evaluation index threshold, it is a non-congested data transmission environment; if the data transmission congestion evaluation index is greater than the preset data transmission congestion evaluation index threshold, it is a congested data transmission environment.

[0019] If it is a non-congested data transmission environment, the edge node receives the data transmitted by the distributed database node in combination with a dynamic optimal transmission mechanism;

[0020] Furthermore, the dynamic optimal transmission mechanism includes:

[0021] The initial queue is queued according to the data priority of the data to be transmitted;

[0022] The data priority of the data to be transmitted and the waiting time of the data to be transmitted are obtained and processed to obtain the queuing priority of the data to be transmitted, and the real-time transmission queue is updated according to the queuing priority.

[0023] If it is a crowded data transmission environment, the data to be transmitted is obtained and the data is segmented and integrated to obtain secondary data to be transmitted, and the data is transmitted;

[0024] Furthermore, the data segmentation and integration includes:

[0025] Get the data packets with data volume greater than the threshold and divide them into small data packets for parallel transmission. Each small data packet is sent within a certain time interval.

[0026] Merge multiple data requests and transfer data.

[0027] If the transmission fails, the data is retransmitted through a self-healing mechanism;

[0028] Furthermore, the self-healing mechanism includes:

[0029] Data is stored and redundantly backed up on the distributed database nodes, and if a network failure or node abnormality is detected, the backup path is switched to perform data transmission;

[0030] Acquire and process data transmission monitoring data to obtain the data packet comprehensive status indicator; if the data packet comprehensive status indicator is greater than a preset data packet comprehensive status indicator threshold, the distributed database node terminates the transmission of the current data to be transmitted and retransmits the current data to be transmitted.

[0031] After the data transmission is completed, the edge node performs abnormal data identification and data timeliness verification on the received data;

[0032] Furthermore, the abnormal data identification includes:

[0033] The edge node obtains the data to be transmitted and parses it. If the data parsing fails, a data request is initiated to re-acquire the data to be transmitted.

[0034] Furthermore, the data timeliness verification includes:

[0035] The data transmission time is obtained, where the data transmission time represents the total time duration for data to be transmitted from the data receiving node to the edge node; if the data transmission time is greater than a preset data transmission time threshold, it indicates that the data is invalid data.

[0036] The edge node acquires the received data and preprocesses it according to the requirements of the digital visual platform to obtain preprocessed data; the edge node stores the preprocessed data in an edge storage node;

[0037] An edge data flow control mechanism is established according to the load of the edge storage node and the priority of the pre-processed data; the client node receives the data transmitted by the edge storage node according to the edge data flow control mechanism, and visually monitors the data transmission status.

[0038] Furthermore, the edge data flow control mechanism includes:

[0039] Acquire and process the preprocessed data stored in the edge storage node to obtain a data transmission priority after preprocessing, and sort the data according to the data transmission priority after preprocessing;

[0040] An adaptive receiving rate limit and an adaptive concurrency limit are obtained according to the load; and the client node receives data according to the adaptive receiving rate limit and the adaptive concurrency limit.

[0041] Furthermore, the visual monitoring includes:

[0042] The digital visual platform builds a real-time monitoring panel to display information of all nodes; the logging function records errors and delays during the transmission process, and reminds through the alarm system when a failure occurs.

[0043] Compared with the prior art, the present invention has the following beneficial effects:

[0044] 1. First, by calculating the data priority, ensure that important data is transmitted first to avoid delays in key data due to transmission congestion; then, by using the data transmission congestion evaluation index, it is possible to determine in real time whether the transmission is in a congested state, and adjust the transmission strategy in time to improve transmission efficiency; the self-healing mechanism provides backup and automatic retransmission functions. In the event of network failure or node abnormality, data transmission is transmitted through redundant backup and backup paths, which maximizes the integrity and stability of data transmission.

[0045] 2. If data parsing fails, the edge node can automatically initiate a request to re-acquire the data to avoid data distortion or errors caused by errors or incompleteness during the transmission process, thereby ensuring that the data finally transmitted to the digital visual platform is more accurate and reliable, and improving data quality and accuracy; data timeliness verification can promptly identify expired or invalid data by checking the time of data transmission, ensuring that the received data meets the requirements of the digital visual platform for data timeliness.

[0046] 3. Sort the data according to the data transmission priority after preprocessing, and adjust the receiving rate limit and concurrency limit. The edge data flow control mechanism ensures that important data is transmitted first, while controlling the rate of data inflow to avoid network congestion caused by data influx under high load conditions, optimizing the system's resource allocation, and making transmission more orderly and efficient; visual monitoring provides real-time display of information such as the status of transmission nodes, data packet traffic, and transmission paths, and timely reminders through the alarm system, which helps to diagnose and solve problems faster and improve data transmission efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1 A flowchart of a distributed information data transmission method for a digital visual platform provided in Example 1 of the present invention;

[0048] Figure 2 A schematic diagram of a node structure provided in Example 1 of the present invention;

[0049] Figure 3 A schematic diagram of the functional structure of an edge node provided in Example 1 of the present invention. DETAILED DESCRIPTION

[0050] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0051] Example 1

[0052] In order to effectively improve production efficiency and reduce maintenance costs, real-time monitoring of production line status and output data is crucial during the production and operation of smart factories. However, with the increasing amount of production data, especially the data integration between multiple factories and production bases, cross-regional and cross-network data transmission has become a core challenge. To this end, a certain automobile manufacturer introduced a distributed information data transmission method for a digital visual platform provided by the present invention to ensure efficient and stable transmission of large-scale data between multiple locations, ensure seamless data connection between different equipment and processes, and ultimately acquire and process various data streams in real time on the digital visual platform. The method flow is as follows Figure 1 As shown, the specific implementation is as follows:

[0053] First, the nodes are distributed as follows Figure 2 As shown, the data receiving node obtains and processes the data collected by the sensor, and stores the data in the distributed database node;

[0054] Furthermore, the automobile manufacturer has a total of N smart production factories across the country, and each smart production factory has a data receiving node for obtaining production-related data collected by sensors in the smart production factory and storing the production data in the distributed database node corresponding to the smart production factory.

[0055] Furthermore, the distributed database includes:

[0056] The distributed database first cleans the production-related data of the intelligent production factory acquired by the data receiving node to obtain the data to be transmitted, wherein the cleaning includes outlier filtering and missing value filling;

[0057] Furthermore, data information of the data to be transmitted is obtained and processed to obtain data priority, and the data priority is used when the data to be transmitted is transmitted to the edge node; the data information includes real-time requirements, data importance, data freshness, data volume, estimated bandwidth consumption and transmission frequency.

[0058] Furthermore, the calculation formula of the data priority is:

[0059]

[0060] Among them, P represents the priority of the data to be transmitted, α1 represents the weight coefficient of the real-time requirement score T, α2 represents the weight coefficient of the data importance score I, α3 represents the weight coefficient of the data freshness score R, α4 represents the weight coefficient of the response frequency F of the data type, α5 represents the weight coefficient of the data volume S, and α6 represents the weight coefficient of the expected bandwidth consumption B.

[0061] Furthermore, the real-time demand score indicates the real-time demand for the data. For example, the real-time demand score for real-time data can be 1, quasi-real-time data can be 0.7, and historical data can be 0.3; the data importance score indicates whether the data is important. For example, the data importance score of key equipment data or warning data can be 1, and the data importance score of secondary equipment data can be 0.5; the data freshness score indicates how long the data has existed. The latest data has a high score, and then gradually decreases; the data type response frequency indicates the frequency of data of the same type as the data appearing in historical transmissions. The higher the frequency of occurrence, the higher the priority; the larger the data, the more bandwidth it occupies, and the more bandwidth it is expected to consume, which will cause congestion and occupy transmission resources, and the lower the priority.

[0062] Furthermore, some production data types and their real-time requirements, data importance, and data freshness values ​​in smart production factories are shown in Table 1.

[0063] Table 1. Production data related scoring values

[0064] Production data types Real-time demand scoring Data Importance Rating Equipment warning data 1 1 Production line operation status data 0.8 0.9 Operator behavior monitoring data 0.7 0.8 Quality test result data 0.5 0.7 Historical production data 0.3 0.5

[0065] A distributed database is established at the location of the smart production factory to perform preliminary processing and storage of production data, which reduces the burden on the central database, improves data processing efficiency, and enhances data security. On the other hand, based on the weight allocation of indicators such as real-time requirements, data importance, and response frequency, the system can give priority to transmitting key data. By calculating the data priority of the data to be transmitted, data support is provided for the subsequent improvement of data transmission efficiency.

[0066] Acquire historical transmission environment data and establish a data transmission congestion evaluation index, and determine whether it is a congested data transmission environment according to the data transmission congestion evaluation index;

[0067] Furthermore, the data transmission congestion evaluation index includes:

[0068] Acquire and process the historical transmission environment data to obtain the data transmission congestion evaluation index; the historical transmission environment data includes the transmission delay, data packet loss rate, bandwidth utilization, retransmission rate, number of connections, data volume and request frequency within t0 time before the current data transmission;

[0069] If the data transmission congestion evaluation index is less than or equal to the preset data transmission congestion evaluation index threshold, it is a non-congested data transmission environment; if the data transmission congestion evaluation index is greater than the preset data transmission congestion evaluation index threshold, it is a congested data transmission environment.

[0070] Furthermore, before data transmission is performed, historical transmission environment data in the past period of time is obtained, and a data transmission congestion evaluation index is calculated to measure the state of the current data transmission environment. The calculation formula of the data transmission congestion evaluation index is:

[0071]

[0072] Wherein, E represents the data transmission congestion evaluation index, β1 represents the weight coefficient of transmission delay T0, β2 represents the weight coefficient of data packet loss rate L0, and β3 represents the used bandwidth D used The weight coefficient, D total represents the total bandwidth, β4 represents the weight coefficient of the retransmission rate R0, β5 represents the weight coefficient of the number of connections C0, β6 represents the weight coefficient of the amount of transmitted data S0, β7 represents the weight coefficient of the request frequency F0, and k represents a preset parameter.

[0073] Furthermore, transmission delay indicates the time from data sending to receiving, and the lower the delay, the higher the data transmission congestion evaluation index; retransmission rate indicates the ratio of the number of retransmissions to the total number of transmissions, and the lower the retransmission rate, the higher the data transmission congestion evaluation index; more connections may mean higher concurrency, thus affecting transmission performance; the larger the data, the longer the transmission time may be, the higher the request frequency, and the heavier the transmission load.

[0074] Before data transmission, by obtaining comprehensive data to calculate the data transmission congestion evaluation index, the current data transmission environment status can be further accurately evaluated, which can provide data support for the subsequent system to effectively allocate resources, avoid transmission congestion, improve the transmission success rate and the overall response capability of the system.

[0075] The specific functions of edge nodes are as follows: Figure 3 As shown, if it is a non-congested data transmission environment, the edge node receives the data pushed in real time by the distributed database node in combination with the dynamic optimal transmission mechanism;

[0076] Furthermore, the dynamic optimal transmission mechanism includes:

[0077] First, the initial queue is queued according to the data priority of the data to be transmitted;

[0078] Then, the data priority of the data to be transmitted and the waiting time of the data to be transmitted are obtained and processed to obtain the queuing priority of the data to be transmitted, and the real-time transmission queue is updated according to the queuing priority. The calculation formula of the queuing priority is:

[0079] P queue =γ1·P0+γ2·T wait ;

[0080] Among them, P queue represents the queuing priority of the data, γ1 represents the weight coefficient of the initial data priority P0 of the data, and γ2 represents the waiting time T of the data in the queue wait The weight coefficient of

[0081] Furthermore, the untransmitted data to be transmitted can be reordered at fixed intervals to ensure that data with higher priority and data with longer waiting time can be transmitted in time, thereby ensuring efficient and real-time data transmission.

[0082] The initial queue is sorted according to data priority to ensure that important data can enter the transmission process first, meeting the digital visual platform's needs for efficient transmission of key data; on the other hand, through regular reordering, the queue priority of data with a longer waiting time is increased to avoid data being ignored due to long waiting times, thereby achieving balanced data transmission.

[0083] If it is a crowded data transmission environment, the data to be transmitted is obtained and the data is segmented and integrated to obtain secondary data to be transmitted, and the data is transmitted;

[0084] Furthermore, the data segmentation and integration includes:

[0085] Divide a large data packet into multiple small data packets and transmit them in parallel. Each small data packet is sent within a certain time interval.

[0086] Then, multiple data requests are merged to obtain secondary data to be transmitted, thereby reducing frequent data transmission.

[0087] Furthermore, if there is a congested data transmission environment, it is processed from two aspects. First, the data to be transmitted is obtained. If the size of the data to be transmitted exceeds the preset threshold, it is divided into multiple small data packets, which reduces the bandwidth occupied by a single transmission, so that each small packet occupies smaller resources in the transmission link. At the same time, the parallel transmission method can utilize multiple paths of the network to improve the overall efficiency of data transmission.

[0088] Furthermore, in a high concurrency state, frequent independent data requests may cause excessive network load, so multiple data requests in a short period of time are merged to reduce the bandwidth occupied by frequent requests through a merged single transmission.

[0089] Through the data segmentation and integration methods mentioned above, better utilization of network resources can be achieved, reducing the bandwidth load during a single transmission process, accelerating the overall data transmission speed in a high-concurrency transmission environment, and improving network stability.

[0090] If the transmission fails, the data is retransmitted through a self-healing mechanism;

[0091] Furthermore, the self-healing mechanism includes:

[0092] First, data is stored and redundantly backed up on distributed database nodes. If a network failure or node abnormality is detected, the backup path is switched to carry out data transmission.

[0093] Then, the data transmission monitoring data is obtained and processed to obtain a comprehensive data packet status indicator; if the comprehensive data packet status indicator is greater than a preset comprehensive data packet status indicator threshold, the uploading of the current data to be transmitted is terminated, and the current data to be transmitted is uploaded again.

[0094] Furthermore, a comprehensive data packet status indicator is established by comparing the ACK confirmation response, packet sequence number continuity and check code; first, when the distributed database sends a data packet, the edge node needs to return a confirmation response within a specified time. If the confirmation response time T ACKIf the time is greater than the preset confirmation response time, it is determined that the data packet is lost, and the data packet loss indicator F ACK =1, otherwise F ACK is 0.

[0095] Furthermore, the data packets sent by the distributed database usually have sequence numbers. The edge nodes will receive and confirm the data packets one by one. If the sequence number of the data packet is missing or discontinuous, it means that there may be packet loss or disorder. The disorder index F SEQ =1, otherwise F SEQ is 0.

[0096] Furthermore, each data packet sent by the distributed database is accompanied by a checksum to ensure the integrity of the data in the packet. The edge node recalculates the checksum after receiving the data. If the checksum does not match, it means that an error occurred during the data transmission process. The data check index F CRC =1, otherwise F CRC is 0.

[0097] Furthermore, the results of the above three determination conditions are obtained and processed to obtain the comprehensive status index of the data packet, and the calculation formula is:

[0098] F fail =θ1·F ACK +θ2·F SEQ +θ3·F CRC ;

[0099] Among them, F fail represents the comprehensive status index of the data packet, θ1, θ2 and θ3 represent weight coefficients; if the comprehensive status index of the data packet is greater than the preset comprehensive status index threshold of the data packet, it means that the data packet transmission fails, and the data packet is transmitted again.

[0100] By setting up a backup transmission path, it can quickly recover when the main path fails, ensuring that data transmission will not be interrupted or lost due to network problems, thereby improving the stability of data storage and transmission. On the other hand, by suspending and re-uploading data whose data packet comprehensive status index is greater than the preset data packet comprehensive status index threshold, network congestion and resource waste are reduced, thereby improving overall transmission efficiency.

[0101] After the data transmission is completed, the edge node first performs abnormal data identification and data timeliness verification, and performs preprocessing according to the data requirements of the digital visual platform to obtain preprocessed data, and then stores the preprocessed data to the edge storage node;

[0102] Furthermore, the abnormal data identification includes:

[0103] The edge node obtains the data to be transmitted and parses it. If the data parsing fails, it initiates a data request to re-acquire the current data.

[0104] Furthermore, after the edge node obtains data from the distributed database, it parses the obtained data, first ensuring the correctness of the data format and content. If the parsing is successful, the data timeliness verification is performed; if data anomalies are detected during the parsing process, such as data format mismatch and field missing errors, resulting in the data not meeting the data requirements of the digital visual platform, it means that the parsing operation has failed. At this time, the edge node initiates an automatic data request to the corresponding distributed database to re-acquire the current data.

[0105] By detecting the received data through the edge node, abnormal data that does not meet the requirements of the digital visual platform can be identified, and the data can be corrected in real time to prevent abnormal data from entering subsequent processing steps, improve the data processing capabilities of the edge node, make data transmission more self-correcting, and reduce the negative impact of abnormal data.

[0106] Furthermore, the data timeliness verification includes:

[0107] The data transmission time is obtained, where the data transmission time represents the total time it takes for data to be transmitted from the data receiving node to the edge node; if the data transmission time is greater than the preset data transmission time, it indicates that the data is expired or invalid.

[0108] Furthermore, the key to data timeliness verification is to ensure that the data does not lose timeliness due to delays during transmission. By calculating the time interval for data to arrive at the edge node from the receiving node to measure whether the data meets the timeliness requirements of the digital visual platform, it is possible to prevent data with large delays from entering the system and ensure that all processed data is up-to-date and valid, thereby ensuring the real-time nature of the decision-making, data analysis and display of the digital visual platform.

[0109] Furthermore, after the data is received, the data is first preprocessed at the edge node according to the specific data requirements of the digital visual platform to obtain preprocessed data that meets the requirements of the digital visual platform, thereby reducing the computing consumption of the client node. The preprocessed data will be stored in the edge storage node so that valid data can be provided quickly and reliably when data is requested.

[0110] Furthermore, the transmission rate, concurrency limit and data priority of the edge node are obtained and an edge data flow control mechanism is established; the client node receives the pre-processed data transmitted by the edge node according to the edge data flow control mechanism, and visually monitors and tracks the transmission status.

[0111] Furthermore, the edge data flow control mechanism includes:

[0112] Acquire and process the preprocessed data stored in the edge storage node to obtain a data transmission priority after preprocessing, and sort the data according to the data transmission priority after preprocessing;

[0113] Furthermore, the data transmission priority after preprocessing takes into account the real-time requirements of the digital visual platform for data and the transmission waiting time of the preprocessed data. The calculation formula of the data transmission priority after preprocessing is:

[0114] P re =η1·R re +η2·T re +η3·W re ;

[0115] Among them, P re represents the data transmission priority after preprocessing, and η1 represents the real-time requirement score R of the data. re The weight coefficient, η2 represents the transmission waiting time T of the data re The weight coefficient, η3 represents the importance score of the data W re The weight coefficient of .

[0116] Acquire and process transmission load data to obtain an adaptive receiving rate limit and an adaptive concurrency limit; the client node receives data according to the adaptive receiving rate limit and the adaptive concurrency limit;

[0117] Furthermore, the calculation formula of the adaptive receiving rate limit is:

[0118]

[0119] Among them, R new Represents the adaptive receiving rate limit adjusted in real time, R base represents the basic rate limit, ω1 represents the rate adjustment coefficient, L current Indicates the current system load, L base Preset baseline load.

[0120] Furthermore, the calculation formula for the adaptive concurrency limit is:

[0121]

[0122] Among them, C new represents the adaptive concurrency limit adjusted in real time, C base represents the basic concurrency limit, and ω2 represents the concurrency adjustment factor.

[0123] Table 2 shows the adaptive adjustment results of the transmission rate and concurrency limit under different load conditions. By limiting the transmission rate and concurrency, the system overload is avoided and the stable transmission of data is guaranteed.

[0124] Table 2. Limit data under different loads

[0125]

[0126] If the receiving rate reaches the limit, new data will be temporarily blocked from entering and the data will be kept in the message queue, waiting for the next round of processing; when the existing connections are processed and resources are released, new connections will be allowed to enter.

[0127] By calculating the transmission priority of pre-processed data, the pre-processed data with high real-time requirements, long waiting time and high importance is transmitted first, ensuring that critical data can reach the digital visualization platform faster, improving the platform's response speed to important data, and optimizing the timeliness of information acquisition; combined with adaptive receiving rate limitation and concurrency limitation, the node can always maintain efficient and stable data transmission in an environment with changing loads.

[0128] Furthermore, the visual monitoring includes:

[0129] The digital visual platform builds a real-time monitoring panel to display information such as the status of each transmission node, data packet flow, and transmission path; the log recording function records errors and delays in the transmission process, and reminds through the alarm system when a fault occurs.

[0130] By building a real-time monitoring panel and log recording function, the digital visual platform provides highly transparent and traceable transmission process management, improving the efficiency, accuracy and reliability of data transmission.

[0131] The method provided by the present invention effectively solves the requirements of the digital visual platform for data real-time and stable transmission through two steps. First, a distributed database node is set to pre-process and store the collected data, and then the data transmission environment between the edge node and the distributed database node is evaluated by calculating the data transmission congestion evaluation index, and is divided into high concurrency and non-high concurrency for processing, thereby improving the data transmission efficiency; after data transmission, the edge node also performs abnormal data identification and data timeliness verification to ensure the accuracy and real-time of the data; at the edge node, data pre-processing is performed according to the digital visual platform before data transmission, thereby reducing the computing power and storage consumption of the client node; data transmission is performed according to the edge data flow control mechanism, thereby ensuring the stability of data transmission of the client node.

[0132] Example 2

[0133] A large shopping mall has built a digital visual monitoring platform for processing and presenting a large amount of monitoring data. In order to achieve efficient transmission in the scenario where a large amount of monitoring data surges, a distributed information data transmission method for a digital visual platform provided by the present invention is introduced. The specific implementation method is as follows:

[0134] First, the data receiving node obtains and processes the monitoring data collected by the sensor, and stores the data in the distributed database node;

[0135] Furthermore, the distributed database acquires the collected monitoring data and cleans it to obtain the monitoring data to be transmitted, and then acquires the data information of the monitoring data to be transmitted and processes it to obtain the data priority; the data information includes real-time requirements, data importance, data freshness, data volume, expected bandwidth consumption and transmission frequency.

[0136] Acquire historical transmission environment data and establish a data transmission congestion evaluation index, and determine whether it is congested data transmission according to the data transmission congestion evaluation index;

[0137] Furthermore, before data transmission is performed, historical transmission environment data in the past period of time is obtained, and a data transmission congestion evaluation index is calculated to measure the state of the current data transmission environment. The calculation formula of the data transmission congestion evaluation index is:

[0138]

[0139] Wherein, E represents the data transmission congestion evaluation index, β1 represents the weight coefficient of transmission delay T0, β2 represents the weight coefficient of data packet loss rate L0, and β3 represents the used bandwidth D used The weight coefficient, D total represents the total bandwidth, β4 represents the weight coefficient of the retransmission rate R0, β5 represents the weight coefficient of the number of connections C0, β6 represents the weight coefficient of the amount of transmitted data S0, β7 represents the weight coefficient of the request frequency F0, and k represents a preset parameter.

[0140] By calculating the data transmission congestion evaluation index, we can first accurately evaluate the status of the current data transmission environment, and then perform data transmission according to different data transmission environment states, thereby improving the efficiency and stability of data transmission.

[0141] If it is a non-congested data transmission environment, the edge node receives the data pushed in real time by the distributed database node in combination with the dynamic optimal transmission mechanism;

[0142] Furthermore, when designing the transmission priority for video stream data, factors such as the timeliness, frame importance, data packet size, and bandwidth occupancy of the video data usually need to be considered. The transmission priority calculation formula for video stream data is:

[0143]

[0144] Among them, P sp represents the transmission priority, ρ1 represents the real-time score R of the video stream data sp The weight coefficient, ρ2 represents the frame importance of the video stream data F sp The weight coefficient, ρ3 represents the packet size S of the video stream data sp The weight coefficient of .

[0145] Furthermore, the real-time score of the video stream data is assigned according to the delay requirement. In the assignment of frame importance of the video stream data, key frames take precedence over difference frames. The assignment can be performed through a machine learning algorithm to first perform scene and character recognition and then assign frame importance. The priority of larger data packets is appropriately reduced.

[0146] If it is the congested data transmission, the data to be transmitted is obtained and the data is segmented and integrated before the data is transmitted;

[0147] If the transmission fails, the data is retransmitted through a self-healing mechanism;

[0148] Furthermore, the self-healing mechanism includes:

[0149] Data is stored and backed up redundantly on distributed database nodes. If a network failure or node abnormality is detected, the backup path is switched to carry out data transmission.

[0150] Then, the data transmission monitoring data is obtained and processed to obtain a comprehensive data packet status indicator; if the comprehensive data packet status indicator is greater than a preset comprehensive data packet status indicator threshold, the uploading of the current data to be transmitted is terminated, and the current data to be transmitted is uploaded again.

[0151] After the data transmission is completed, the edge node identifies abnormal data and verifies the timeliness of the received data; obtains the received data and preprocesses it according to the requirements of the digital visual platform to obtain preprocessed data; and stores the preprocessed data in the edge storage node;

[0152] Furthermore, after the data is received, the video stream data is preprocessed at the edge node according to the specific data requirements of the digital visual monitoring platform to obtain preprocessed data that meets the requirements of the digital visual platform, thereby reducing the computing consumption of the client node. The preprocessed data will be stored in the edge storage node so that valid data can be provided quickly and reliably when data is requested.

[0153] Further, the preprocessed data stored in the edge storage node is obtained and processed to obtain a preprocessed data transmission priority, and the data is sorted according to the preprocessed data transmission priority;

[0154] Table 3. Limit adjustment results under different loads

[0155]

[0156] Further, an adaptive receiving rate limit and an adaptive concurrency limit are obtained according to the load; the client node receives data according to the adaptive receiving rate limit and the adaptive concurrency limit;

[0157] Table 3 shows the adjustment results of adaptive receiving rate limit and concurrency limit under different loads. The above method improves the efficiency and stability of data transmission.

[0158] By applying the distributed information data transmission method for a digital visual platform provided by the present invention, the requirements of the digital visual monitoring platform for high efficiency and stability of data transmission are achieved, and the working efficiency of the digital visual monitoring platform is improved.

[0159] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A distributed information data transmission method for a digital visual platform, characterized in that: include: The data receiving node acquires and processes the data collected by the sensor to obtain the collected data, and stores the collected data in the distributed database node; Acquire and process historical transmission environment data to obtain a data transmission congestion evaluation index, and determine whether it is a congested data transmission environment according to the data transmission congestion evaluation index; If it is a non-congested data transmission environment, the edge node receives the data transmitted by the distributed database node in combination with a dynamic optimal transmission mechanism; If it is a crowded data transmission environment, the data to be transmitted is obtained and the data is segmented and integrated to obtain secondary data to be transmitted, and the data is transmitted; If data transmission fails, the data is retransmitted through a self-healing mechanism; the self-healing mechanism also includes a comprehensive status indicator of the data packet; After the data transmission is completed, the edge node performs abnormal data identification and data timeliness verification on the received data; The edge node acquires the received data and pre-processes it according to the requirements of the digital visual platform to obtain pre-processed data; The edge node stores the pre-processed data in an edge storage node; An edge data flow control mechanism is established according to the load of the edge storage node and the priority of the pre-processed data; the client node receives the data transmitted by the edge storage node according to the edge data flow control mechanism, and visually monitors the data transmission status.

2. A distributed information data transmission method for a digital visual platform according to claim 1, characterized in that: The distributed database includes: Acquire the collected data and clean it to obtain data to be transmitted, wherein the cleaning includes outlier filtering and missing value filling; The data information of the data to be transmitted is obtained and processed to obtain the data priority; the data information includes real-time requirements and data importance; the calculation formula of the data priority is: Among them, P represents the data priority of the data to be transmitted, α1 represents the weight coefficient of the real-time requirement score T, α2 represents the weight coefficient of the data importance score I, α3 represents the weight coefficient of the data freshness score R, α4 represents the weight coefficient of the data type response frequency F, α5 represents the weight coefficient of the data volume S, and α6 represents the weight coefficient of the expected bandwidth consumption B.

3. A distributed information data transmission method for a digital visual platform according to claim 1, characterized in that: The data transmission congestion evaluation index includes: The historical transmission environment data includes the transmission delay, data packet loss rate, bandwidth utilization, retransmission rate, number of connections, data volume and request frequency within t0 time before the current data transmission; The historical transmission environment data is acquired and processed to obtain the data transmission congestion evaluation index. If the data transmission congestion evaluation index is less than or equal to a preset data transmission congestion evaluation index threshold, it is a non-congested data transmission environment. If the data transmission congestion evaluation index is greater than the preset data transmission congestion evaluation index threshold, it is a congested data transmission environment.

4. A distributed information data transmission method for a digital visual platform according to claim 1, characterized in that: The dynamic optimal transmission mechanism includes: The initial queue is queued according to the data priority of the data to be transmitted; The data priority of the data to be transmitted and the waiting time of the data to be transmitted are obtained and processed to obtain the queuing priority of the data to be transmitted, and the real-time transmission queue is updated according to the queuing priority.

5. A distributed information data transmission method for a digital visual platform according to claim 1, characterized in that: The data segmentation and integration include: Get the data packets with data volume greater than the threshold and divide them into small data packets for parallel transmission. Each small data packet is sent at an interval of time0. Merge multiple data requests and transfer data.

6. A distributed information data transmission method for a digital visual platform according to claim 1, characterized in that: The self-healing mechanism includes: Performing data backup on the distributed database nodes, and switching to the backup path for data transmission if a network failure or node abnormality is detected; Acquire and process data transmission monitoring data to obtain the data packet comprehensive status indicator; if the data packet comprehensive status indicator is greater than a preset data packet comprehensive status indicator threshold, the distributed database node terminates the transmission of the current data to be transmitted and retransmits the current data to be transmitted.

7. A distributed information data transmission method for a digital visual platform according to claim 1, characterized in that: The abnormal data identification includes: The edge node obtains the data to be transmitted and parses it. If the data parsing fails, a data request is initiated to re-acquire the data to be transmitted.

8. A distributed information data transmission method for a digital visual platform according to claim 1, characterized in that: The data timeliness verification includes: The data transmission time is obtained, where the data transmission time represents the total time duration for data to be transmitted from the data receiving node to the edge node; if the data transmission time is greater than a preset data transmission time threshold, it indicates that the data is invalid data.

9. A distributed information data transmission method for a digital visual platform according to claim 1, characterized in that: The edge data flow control mechanism includes: Acquire and process the preprocessed data stored in the edge storage node to obtain a data transmission priority after preprocessing, and sort the data according to the data transmission priority after preprocessing; An adaptive receiving rate limit and an adaptive concurrency limit are obtained according to the load; and the client node receives data according to the adaptive receiving rate limit and the adaptive concurrency limit.

10. A distributed information data transmission method for a digital visual platform according to claim 1, characterized in that: The visual monitoring includes: The digital visual platform builds a real-time monitoring panel to display information of all nodes; the logging function records errors and delays during the transmission process, and reminds through the alarm system when a failure occurs.

Images