Configurable cloud edge real-time database transmission method and system

Abstract

The invention discloses a configurable cloud edge real-time database transmission method and system, and relates to the field of data transmission security, and the main scheme is as follows: collecting multiple items of data of a database; respectively calculating a comprehensive priority evaluation value, a network state evaluation index and a data updating frequency, and further calculating a network connection establishment tendency value; judging whether connection establishment is successful or not according to a preset threshold value; if the connection is established successfully, calculating a received data integrity evaluation value, and further obtaining a unique identifier of the received information; calculating a data storage time index, and further calculating a deletion decision index; calculating a transmission reliability index, calculating an anomaly judgment comprehensive index according to the received data integrity evaluation value and the transmission reliability index, and judging whether the data is normal or not according to a preset threshold value; the method can scientifically and reasonably dispatch data transmission, deal with abnormities in time, improve data transmission efficiency and safety, and guarantee stable operation of the system.

Description

Technical Field

[0001] The present invention relates to the field of data transmission security technology, and specifically to a configurable cloud-edge real-time database transmission method and system. Background Art

[0002] With the rapid development of information technology, the amount of data has exploded, and the security and efficient transmission of data face many challenges. For example, different types of data have different importance and transmission requirements. How to reasonably determine the priority of data to achieve efficient transmission has become a key issue. In addition, during the data storage process, how to scientifically manage the storage time and clean up abnormal data in a timely manner is also a problem that needs to be solved urgently.

[0003] The existing technology mainly adopts some relatively conventional methods to deal with these problems. For the determination of data priority, it is usually based on the source, type or simple business rules of the data for preliminary classification, and then the data is transmitted in the established priority order. In terms of data storage management, a fixed storage period is generally set, and when the data storage time reaches the period, the data is cleared.

[0004] Although existing technologies can play a role in data transmission and management to a certain extent, there are still many shortcomings. First, the data priority determined based on simple rules cannot fully consider the actual needs of data in different business scenarios, which can easily lead to delays in important data transmission or unnecessary data occupying transmission resources. In addition, the fixed storage period setting lacks flexibility, which will delete normal data while retaining a large amount of abnormal data, wasting storage resources. Summary of the invention

[0005] 1. Technical issues to be resolved In view of the shortcomings of the prior art, the present invention provides a configurable cloud-edge real-time database transmission method and system, which collects multiple data from the database; calculates the comprehensive priority evaluation value, network status evaluation index and data update frequency respectively, and then calculates the network connection establishment tendency value; presets a threshold to determine whether the connection establishment is successful; if the connection is successfully established, calculates the received data integrity evaluation value, and further obtains the received information unique identifier; calculates the data storage time index, and further calculates the deletion decision index; calculates the transmission reliability index, and calculates the abnormality judgment comprehensive index according to the received data integrity evaluation value and the transmission reliability index, and presets a threshold to determine whether the data is normal, thereby solving the problems of low data transmission efficiency and unreliable connection.

[0006] (II) Technical solution To achieve the above objectives, the present invention is implemented through the following technical solutions: a configurable cloud-edge real-time database transmission method, comprising: Step 1: Collect the encrypted data of the database and the computer system information data; Step 2: Calculate the data priority transmission index PDF of different types of data based on the encrypted data; calculate the comprehensive priority evaluation value PDG based on the data priority transmission index PDF and the computer system information data; calculate the network status evaluation index NSV based on the computer system information data; calculate the data update frequency UF based on the computer system information data; Step 3: Calculate the network connection establishment tendency value TDC based on the comprehensive priority evaluation value PDG, the network status evaluation index NSV and the data update frequency UF, preset the network connection establishment tendency value threshold, compare the network connection establishment tendency value TDC with the network connection establishment tendency value threshold, and judge whether the connection establishment is successful based on the comparison result; if the connection establishment fails, skip this data item and continue to check the next data item; Step 4: If the connection is established successfully, the received data integrity assessment value VBI is calculated based on the network connection establishment tendency value TDC, and the received information unique identifier UID is calculated based on the received data integrity assessment value VBI; the data storage time indicator WT is calculated based on the received information unique identifier UID and the encrypted data; the deletion decision indicator DQ is calculated based on the data storage time indicator WT, the received information unique identifier UID, the data priority transmission indicator PDF and the encrypted data; Step 5: Calculate the transmission reliability index TRI based on the deletion decision index DQ, encrypted data and computer system information data; calculate the abnormality judgment comprehensive index A based on the received data integrity assessment value VBI and the transmission reliability index TRI; preset the abnormality judgment comprehensive index threshold, compare the abnormality judgment comprehensive index A with the abnormality judgment comprehensive index threshold, and judge whether the data is normal based on the comparison result; if it is normal data, continue to transmit the data; if it is abnormal data, store it in a separate abnormal data storage area, and suspend the continued transmission of the data.

[0007] In the preferred embodiment of the above-mentioned configurable cloud-edge real-time database transmission method: the method for calculating the comprehensive priority evaluation value PDG is: The encrypted data includes the data size SD, data generation time TD and data access frequency AD of different types of data; Computer system information data includes system load index LO and system resource available evaluation value RP; According to the data size SD, data generation time TD and data access frequency AD, the data priority transmission index PDF of different types of data is calculated based on the formula: ; Among them, Smax is the maximum data size allowed by the system; Tmin is the minimum timestamp processed by the system; Tmax is the maximum timestamp processed by the system; e is a natural constant; According to the data priority transmission index PDF, the system load index LO and the system resource available evaluation value RP, the comprehensive priority evaluation value PDG is calculated based on the formula: ; Among them, MA is the system maintenance status coefficient; NA is the network status coefficient.

[0008] In the preferred embodiment of the above configurable cloud-edge real-time database transmission method: the method for calculating the network status evaluation index NSV is: The computer system information data also includes network bandwidth BW, network delay DL, packet loss rate PLR ​​and time interval RTR from the last successful transmission to the present; According to the network bandwidth BW, network delay DL, packet loss rate PLR ​​and the time interval RTR from the last successful transmission to the current time, the network status evaluation index NSV is calculated. The calculation formula is: ; Among them, BWmax is the maximum network bandwidth; DLmax is the maximum network delay; RTRmax is the maximum time interval from the last successful transmission to the current one; ω1 is The weight coefficient is 0.2~0.4; ω2 is The weight coefficient of ω3 is 0.1~0.3; ω4 is the weight coefficient of the packet loss rate PLR, which is 0.3~0.5; The weight coefficient is between 0.2 and 0.4; and ω1+ω2+ω3+ω4=1.

[0009] In the preferred embodiment of the above configurable cloud-edge real-time database transmission method: the method for calculating the data update frequency UF is: The computer system information data also includes the total number of data updates NG and the average data volume DA of the data updates within the time TE; According to the total number of data updates NG and the average data volume DA in the time TE, the data update frequency UF is calculated. The calculation formula is: ; Among them, CA is the average data processing capacity of the system.

[0010] In the preferred embodiment of the above configurable cloud-edge real-time database transmission method: the method for determining whether the connection establishment is successful is: According to the comprehensive priority evaluation value PDG, the network status evaluation index NSV and the data update frequency UF, the network connection establishment tendency value TDC is calculated. The calculation formula is: ; Among them, φ1 is the weight coefficient of the comprehensive priority evaluation value PDG, which is 0.2~0.4; φ2 is the weight coefficient of the network status evaluation index NSV, which is 0.3~0.5; φ3 is the weight coefficient of the data update frequency UF, which is 0.1~0.5; and φ1+φ2+φ3=1; A network connection establishment tendency value threshold Tth is preset; when TDC≤Tth, the network connection establishment is judged to have failed, the data is skipped, and the next data is checked; when TDC>Tth, the network connection is judged to have been successfully established, and the TCP protocol is used to establish a connection based on the target cloud database address and transmit the data.

[0011] In the preferred embodiment of the above-mentioned configurable cloud-edge real-time database transmission method: the method for calculating the unique identifier UID of the received information is: The trend value TDC is established based on the network connection, and the received data integrity assessment value VBI is calculated based on the formula: ; Among them, HR is the hash value of the received data; HS is the original hash value of the data when it was sent; DR is the amount of data received; DS is the amount of data sent; EF is the error count during data transmission; Based on the received data integrity assessment value VBI, the received information unique identifier UID is calculated according to the formula: ; Among them, TS is the timestamp of data reception.

[0012] In the preferred embodiment of the above configurable cloud-edge real-time database transmission method: the method for calculating the deletion decision index DQ is: The encrypted data also includes the storage time TT of the data in the local cache and the average size Savg of the cache data; According to the unique identifier UID of the received information and the storage time TT of the data in the local buffer area, the data storage time indicator WT is calculated. The calculation formula is: ; Among them, TCU is the current time; TImax is the maximum storage time allowed; According to the data storage time indicator WT, the average size of the cache data Savg, the data size SD, the data priority transmission indicator PDF and the unique identifier UID of the received information, the deletion decision indicator DQ is calculated. The calculation formula is: ; Where C is a constant.

[0013] In the preferred embodiment of the above-mentioned configurable cloud-edge real-time database transmission method: the method for calculating the transmission reliability index TRI is: According to the deletion decision index DQ, data size SD and network bandwidth BW, the transmission reliability index TRI is calculated. The calculation formula is: ; Wherein, t is the time to wait for receiving confirmation information; TL is the predetermined timeout period; and PN is the initial priority of the data.

[0014] In the preferred embodiment of the above-mentioned configurable cloud-edge real-time database transmission method: the method for determining whether the data is abnormal is: According to the received data integrity assessment value VBI and the transmission reliability index TRI, the abnormality judgment comprehensive index A is calculated based on the formula: ; Among them, CL is the data transmission delay time; CI is the network congestion index; TR is the data transmission rate; A threshold value Ath of the comprehensive index for abnormal determination is preset; when A≤Ath, the data is judged to be normal data and the data is continuously transmitted; when A>Ath, the data is judged to be abnormal data and stored in a separate abnormal data storage area, and the further transmission of the data is suspended.

[0015] The present invention also discloses a configurable cloud-edge real-time database transmission system, which is used to implement the above-mentioned configurable cloud-edge real-time database transmission method, including: Data collection module, used to collect encrypted data of database and computer system information data; A data processing module is used to calculate the data priority transmission index PDF of different types of data based on the encrypted data; calculate the comprehensive priority evaluation value PDG according to the data priority transmission index PDF and the computer system information data; calculate the network status evaluation index NSV based on the computer system information data; calculate the data update frequency UF based on the computer system information data; A connection module is used to calculate a network connection establishment tendency value TDC according to a comprehensive priority evaluation value PDG, a network status evaluation index NSV and a data update frequency UF, preset a network connection establishment tendency value threshold, compare the network connection establishment tendency value TDC with the network connection establishment tendency value threshold, and judge whether the connection establishment is successful according to the comparison result; if the connection establishment fails, skip this data item and continue to check the next data item; The scheduling module, if the connection is successfully established, calculates the received data integrity assessment value VBI based on the network connection establishment tendency value TDC, and calculates the received information unique identifier UID based on the received data integrity assessment value VBI; calculates the data storage time indicator WT according to the received information unique identifier UID and the encrypted data; calculates the deletion decision indicator DQ according to the data storage time indicator WT, the received information unique identifier UID, the data priority transmission indicator PDF and the encrypted data; The judgment module is used to calculate the transmission reliability index TRI according to the deletion decision index DQ, the encrypted data and the computer system information data; calculate the abnormality judgment comprehensive index A according to the received data integrity assessment value VBI and the transmission reliability index TRI; preset the abnormality judgment comprehensive index threshold, compare the abnormality judgment comprehensive index A with the abnormality judgment comprehensive index threshold, and judge whether the data is normal according to the comparison result.

[0016] (III) Beneficial effects The present invention provides a configurable cloud-edge real-time database transmission method and system, which has the following beneficial effects: (1) Collecting encrypted data from the database and computer system information data can provide comprehensive and accurate basic data for subsequent calculations and decision-making; this can provide real-time information on key information such as system operation status and network status, and provide a basis for reasonably evaluating the priority of data transmission and the feasibility of network connections.

[0017] (2) Calculating the data priority transmission index can clarify the importance of transmitting different types of data and facilitate the reasonable arrangement of the transmission order; calculating the comprehensive priority evaluation value can more comprehensively reflect the priority level of data in the entire transmission system; the calculation of the network status evaluation index and the data update frequency considers the network status and data change frequency respectively, providing a multi-dimensional reference for subsequent network connection and data transmission decisions, so that the system can flexibly adjust the transmission strategy according to different situations.

[0018] (3) Calculate the network connection establishment tendency value and compare it with the threshold to determine whether the connection establishment is successful. This method can scientifically decide whether to perform data transmission. If the connection establishment fails, skip the data item, which can avoid invalid connection attempts, save system resources, improve the overall efficiency of data transmission, and ensure that the system gives priority to those data that are more likely to be successfully transmitted.

[0019] (4) Calculate the received data integrity assessment value, the received information unique identifier, and further calculate the deletion decision index, which can manage and evaluate the data that has been successfully connected and transmitted in a more detailed manner. Through these calculations, the integrity of the data can be judged, the storage time of the data can be determined, and whether to delete the data can be decided, which helps to optimize data storage and management, improve the resource utilization efficiency of the system, and ensure the reliability and consistency of the data.

[0020] (5) Calculate the transmission reliability index, and calculate the anomaly judgment comprehensive index in combination with the received data integrity assessment value, and then compare it with the threshold to determine whether the data is normal. This method can effectively identify abnormal data and ensure the quality and stability of data transmission. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 A schematic diagram of the steps of the configurable cloud-edge real-time database transmission method of the present invention; Figure 2 It is a schematic diagram of the working steps of step 5 in the configurable cloud-edge real-time database transmission method of the present invention; Figure 3 This is a schematic diagram of the composition structure of the configurable cloud-edge real-time database transmission system of the present invention. DETAILED DESCRIPTION

[0022] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0023] See also Figure 1-Figure 2 The present invention provides a configurable cloud-edge real-time database transmission method, including: Step 1: Collect the encrypted data of the database and the computer system information data.

[0024] Comprehensive step one: By collecting encrypted data from the database and computer system information data, comprehensive and basic data support is provided for subsequent calculations and decisions. This can make subsequent analysis and judgment based on these data more accurate and reliable, and help the system fully understand the data status and the operating status of the computer system, so as to make more reasonable decisions.

[0025] Step 2: Calculate the data priority transmission index PDF of different types of data based on the encrypted data; calculate the comprehensive priority evaluation value PDG based on the data priority transmission index PDF and the computer system information data; calculate the network status evaluation index NSV based on the computer system information data; calculate the data update frequency UF based on the computer system information data.

[0026] Step 201: Calculate the comprehensive priority evaluation value PDG, the specific method is: The encrypted data includes data size SD, data generation time TD and data access frequency AD of different types of data.

[0027] It should be noted that the data size SD is obtained by using SQL statements to query relevant data records and using the functions provided by the database system. For example, in MySQL, for data stored as text, the LENGTH function can be used to obtain the byte length; for binary data, the CHAR-LENGTH function can be used to obtain the byte length; this method is used to obtain the data size SD.

[0028] When designing a database table, there is usually a field that records the creation time of the data. When inserting data, the current time is written into this field. When querying data later, the value of this time field can be directly obtained to obtain the data generation time TD.

[0029] Set up a logging function in the application program to record an access log each time data is accessed, including access time, accessed data ID, etc. By regularly analyzing the log file, the number of accesses to each data within a specific time period is counted, thus obtaining the data access frequency AD.

[0030] The computer system information data includes a system load index LO and a system resource available evaluation value RP.

[0031] It should be noted that in a computer system, the system CPU usage CPUusage is obtained by using the psutil.cpu_percent function in Python's psutil library; the number of CPU cores NCO is obtained by using the psutil.cpu_count function; and the system load index LO is calculated, LO=(CPUusage×NCO) / 100.

[0032] Use the psutil.virtual_memory.total function to obtain the system's total memory MEM and used memory MED; use the psutil.disk_usage(' / ').total function to obtain the system's total disk space MIS and used disk space ISK. Calculate the system resource available evaluation value RP using the following formula: .

[0033] According to the data size SD, data generation time TD and data access frequency AD, the data priority transmission index PDF of different types of data is calculated based on the formula: ; Among them, Smax is the maximum data size allowed by the system; Tmin is the minimum timestamp processed by the system; Tmax is the maximum timestamp processed by the system; e is a natural constant, and its value is 2.71828.

[0034] It should be noted that the maximum data size Smax allowed by the system is obtained by reading the system configuration file because the parameter value of the maximum data size is clearly specified in the system configuration file.

[0035] Get the minimum timestamp Tmin processed by the system. Since the system records and manages the timestamps of data, you can directly query the timestamps recorded in the system. In the database, use the SQL statement "SELECT MIN FROM table name" to get the minimum timestamp Tmin.

[0036] Get the maximum timestamp Tmax processed by the system. You can get the maximum timestamp Tmax by using the SQL statement "SELECT MAX FROM table name".

[0037] It should be noted that This part considers the effect of data size SD. As SD increases, The value of will increase, making the value of this part increase, indicating that the larger the data is, the higher its weight in the transmission priority index is, that is, it tends to give priority to the transmission of large data. This part reflects the role of data generation time TD. It reflects the relative position of the data generation time in the entire time range. It plays a normalization role and normalizes the sum of the two parts in the numerator so that the calculated PDF value is within a suitable range, which is convenient for comparing and sorting the transmission priorities of different data.

[0038] According to the data priority transmission index PDF, the system load index LO and the system resource available evaluation value RP, the comprehensive priority evaluation value PDG is calculated based on the formula: ; Among them, MA is the system maintenance status coefficient; NA is the network status coefficient.

[0039] It should be noted that the maintenance state coefficient MA of the system is obtained. According to the total number of maintenance tasks planned by the system in a time period TW is Npl, and the number of maintenance tasks actually completed is Nco, the maintenance state coefficient MA of the system is calculated by the following formula: ; Among them, k is the influence coefficient.

[0040] Obtain the network status coefficient NA, count the number of network connection interruptions IT and the total connection interruption duration TIN within the time period TY; calculate the network status coefficient NA, the calculation formula is: .

[0041] It should be noted that This reflects the adjustment of data priority by system resource conditions. When RP is higher and LO is lower, the value of this part will be larger, which means that when system resources are sufficient and the load is light, it will be more inclined to transmit according to the priority of the data itself. The influence of system maintenance status and network conditions on the comprehensive priority evaluation value is considered. This formula takes into account multiple factors, such as the priority of the data itself, system resource status, system maintenance status, and network status, to calculate a comprehensive priority evaluation value, so as to more reasonably determine the data transmission priority.

[0042] Step 202: Calculate the network status evaluation index NSV, the specific method is: The computer system information data also includes network bandwidth BW, network delay DL, packet loss rate PLR ​​and time interval RTR from the last successful transmission to the present.

[0043] It should be noted that the network bandwidth BW is obtained by using the iperf network test software and running the software at both ends of the network for testing.

[0044] Enter ping <target IP address> in the command line interface, the system will send ICMP data packets to the target address and return the response time. This response time is the network delay DL.

[0045] By using the "ping <target IP address>" command, while testing the network connectivity, the number of packets sent and received will be counted. By calculating the ratio of the number of lost packets to the total number of sent packets, the packet loss rate PLR ​​can be obtained.

[0046] In the data transmission system, the current timestamp is recorded each time data is successfully transmitted; the current time is obtained, and the timestamp of the last successful transmission is subtracted from the current time to obtain the time interval between the two, thereby obtaining the time interval RTR from the last successful transmission to the current time.

[0047] According to the network bandwidth BW, network delay DL, packet loss rate PLR ​​and the time interval RTR from the last successful transmission to the current time, the network status evaluation index NSV is calculated. The calculation formula is: ; Among them, BWmax is the maximum network bandwidth; DLmax is the maximum network delay; RTRmax is the maximum time interval from the last successful transmission to the current one; ω1 is The weight coefficient is 0.2~0.4; ω2 is The weight coefficient of ω3 is 0.1~0.3; ω4 is the weight coefficient of the packet loss rate PLR, which is 0.3~0.5; The weight coefficient is between 0.2 and 0.4; and ω1+ω2+ω3+ω4=1.

[0048] It should be noted that the maximum network bandwidth value BWmax is obtained by viewing the network specification provided by the network service provider to obtain the maximum bandwidth value BWmax.

[0049] To obtain the maximum network delay DLmax, enter ping <target IP address> in the command line interface, test the network delay multiple times in different time periods, record the delay value of each test, and select the maximum value as DLmax.

[0050] Get the maximum time interval RTRmax from the last successful transmission to the current time. Record the timestamp each time data is successfully transmitted. In the subsequent data transmission process, continue to calculate the interval between the current time and the last successful transmission time, and record the interval value calculated each time. After a period of monitoring, find the maximum value from these recorded values ​​as the maximum time interval RTRmax from the last successful transmission to the current time.

[0051] It should be noted that in the formula , and The network bandwidth, network delay, and the time interval from the last successful transmission to the current state are normalized respectively. By dividing by their respective maximum values, these parameters are compared and calculated at the same order of magnitude, which is convenient for subsequent comprehensive evaluation. It shows that each parameter has a corresponding weight coefficient, which indicates the importance of different parameters when calculating the network status evaluation index. The formula normalizes and weights the network bandwidth, network delay, packet loss rate and the time interval of the last successful transmission, and then uses the characteristics of the exponential function to combine these parameters to calculate the network status evaluation index NSV.

[0052] Step 203: Calculate the data update frequency UF, the specific method is: The computer system information data also includes the total number of data updates NG and the average data amount DA of the data updates within the time TE.

[0053] It should be noted that by adding a counter variable at the key code of data update, each time the data update operation is performed, the counter variable is increased by 1. At the end of the time, the value of the counter is read, which is the total number of data updates NG.

[0054] During each data update operation, the amount of data updated is recorded. The amount of data recorded each time is accumulated to obtain the total amount of data updated within the time TE. Then this total amount is divided by the total number of data updates NG to obtain the average amount of data updated DA.

[0055] According to the total number of data updates NG and the average data volume DA in the time TE, the data update frequency UF is calculated. The calculation formula is: ; Among them, CA is the average data processing capacity of the system.

[0056] It should be noted that to obtain the average data processing capacity CA of the system, a series of data processing requests, such as data insertion, query, update, and deletion, are sent to the system using the BenchmarkSQL performance testing tool for a certain period of time. The total amount of data successfully processed by the system during this period is recorded, and then the total amount of processed data is divided by the test time to obtain the average data processing capacity CA of the system during the test period.

[0057] It should be noted that It reflects the frequency of data updates. The larger the value is, the more times the data is updated in the same period of time, and the greater the contribution to the data update frequency. This means that when the amount of data updated each time is larger, it will have a greater impact on the frequency of data updates. Adding 1 and squaring it further amplifies this effect, emphasizing the role of data size on update frequency.

[0058] Comprehensive steps 201 to 203: Calculating the comprehensive priority evaluation value realizes the dual consideration of data priority and system environment, making the evaluation more comprehensive and in line with actual needs. Calculating network status evaluation indicators and data update frequency can provide real-time insight into the operation status of the network and the dynamic change characteristics of data. These indicators are interrelated and have their own focus, and together provide a rich and accurate basis for decision-making in data transmission, resource allocation, performance optimization, etc., effectively ensuring the efficient and stable operation of the system.

[0059] Step 3: Based on the comprehensive priority evaluation value PDG, the network status evaluation index NSV and the data update frequency UF, calculate the network connection establishment tendency value TDC, preset the network connection establishment tendency value threshold, compare the network connection establishment tendency value TDC with the network connection establishment tendency value threshold, and determine whether the connection establishment is successful based on the comparison result; if the connection establishment fails, skip this data item and continue to check the next data item.

[0060] Step 301: Determine whether the connection is established successfully. The specific method is: According to the comprehensive priority evaluation value PDG, the network status evaluation index NSV and the data update frequency UF, the network connection establishment tendency value TDC is calculated. The calculation formula is: ; Among them, φ1 is the weight coefficient of the comprehensive priority evaluation value PDG, which is 0.2~0.4; φ2 is the weight coefficient of the network status evaluation index NSV, which is 0.3~0.5; φ3 is the weight coefficient of the data update frequency UF, which is 0.1~0.5; and φ1+φ2+φ3=1.

[0061] It should be noted that this formula comprehensively considers three important parameters: the comprehensive priority evaluation value PDG, the network status evaluation index NSV, and the data update frequency UF. Each parameter has a corresponding weight coefficient, which indicates the importance of different parameters when calculating the network connection establishment tendency value TDC. The network connection establishment tendency value TDC is obtained by multiplying each parameter with its corresponding weight coefficient and then summing them.

[0062] A network connection establishment tendency value threshold Tth is preset; by collecting relevant data of all network connections of the system in the historical time, including the comprehensive priority evaluation value, the network status evaluation index, the data update frequency and the network status at that time, the network connection establishment tendency value TDC of each network connection is calculated, and the network connection establishment tendency value average value TDCavg is further calculated, and the network connection establishment tendency value average value TDCavg is used as the network connection establishment tendency value threshold Tth.

[0063] When TDC≤Tth, it is determined that the network connection fails to be established, the data is skipped, and the next data is checked; when TDC>Tth, it is determined that the network connection is successfully established, and the TCP protocol is used to establish a connection based on the target cloud database address and transmit the data.

[0064] Comprehensive step three: This method can effectively avoid wasting resources on bad connections. If the connection fails, skip the data item and check the next one, which can stop the loss in time, improve data processing efficiency, ensure that the system focuses on data transmission that can be successfully connected, and ensure the stability and efficiency of data transmission.

[0065] Step 4: If the connection is established successfully, calculate the received data integrity assessment value VBI based on the network connection establishment tendency value TDC, and calculate the received information unique identifier UID based on the received data integrity assessment value VBI; calculate the data storage time indicator WT based on the received information unique identifier UID and the encrypted data; calculate the deletion decision indicator DQ based on the data storage time indicator WT, the received information unique identifier UID, the data priority transmission indicator PDF and the encrypted data.

[0066] Step 401: Calculate the unique identifier UID of the received information. The specific method is: The trend value TDC is established based on the network connection, and the received data integrity assessment value VBI is calculated based on the formula: ; Among them, HR is the hash value of the received data; HS is the original hash value of the data when it was sent; DR is the amount of data received; DS is the amount of data sent; EF is the error count during data transmission.

[0067] It should be noted that to obtain the hash value HR of the received data, a specific hash algorithm, such as MD5, SHA-256, etc., is used at the data receiving end to calculate the received data, and the resulting hash value is HR. For example, when a piece of text data "Hello, World!" is received, the SHA-256 algorithm is used to calculate and obtain a hash value consisting of 64 hexadecimal digits, which is the hash value HR of the received data.

[0068] Get the original hash value HS of the data when it is sent. Before the data is sent, the data sender uses the same hash algorithm as the receiver to calculate the data to be sent, and the hash value obtained is HS. For example, before sending the text data "Hello, World!", use the SHA-256 algorithm to calculate a hash value and record it. This value is the original hash value HS of the data when it is sent.

[0069] Get the amount of data received DR. At the data receiving end, DR is determined by counting the number of bytes of the received data, the number of rows of data records, or other appropriate units of measurement. For example, if a file is received and the size of the file is 1024 bytes, then DR = 1024 bytes. Or if a data set containing 50 records is received, then DR = 50 records.

[0070] Get the amount of data to be sent DS. At the data sending end, record the number of bytes of the data to be sent, the number of rows of data records, and other information when sending data to determine DS. For example, to send a file of 2048 bytes, then DS = 2048 bytes.

[0071] Get the error count EF during data transmission. During data transmission, errors can be detected and counted through network protocols or program codes. For example, when using the TCP protocol to transmit data, the protocol itself will detect data transmission errors, such as checksums, packet loss, etc. The program can record the number of times these errors occur, which is the error count EF during data transmission.

[0072] It should be noted that This item compares the difference between the hash value HR of the received data and the original hash value HS of the data when it was sent, and performs a square operation. The hash value is a characteristic identifier of the data. The greater the difference between HR and HS, the greater the value of this item, indicating that the data has changed during the transmission process, which has a negative impact on the data integrity. The impact of this difference is further amplified by the square operation. This item means that if the difference between the two is large, it means that there is data loss or redundancy during the transmission process, which will also affect the integrity of the data. The larger the value of this item, the greater the negative impact on data integrity. It reflects the errors that occurred during the transmission process. The formula takes into account multiple factors such as the change in hash value during data transmission, the difference in data volume, error counts, and the tendency of network connection establishment, so as to quantitatively evaluate the integrity of the received data.

[0073] Based on the received data integrity assessment value VBI, the received information unique identifier UID is calculated according to the formula: ; Among them, TS is the timestamp of data reception.

[0074] It should be noted that the timestamp TS of data reception can be obtained because routers, switches and other devices in the network will record relevant logs of data transmission, which contain the timestamp of data reception. By checking the log files of these devices and finding the records of corresponding data reception events, the timestamp TS of data reception can be obtained.

[0075] It should be noted that this formula comprehensively considers the two values ​​of the received data integrity assessment value VBI and the data reception timestamp TS, and generates a unique identifier for each received information by dividing the numerator by the denominator to facilitate identification and distinction in data processing and management.

[0076] Step 402: Calculate the deletion decision index DQ, the specific method is: The encrypted data also includes the storage time TT of the data in the local cache area and the average size Savg of the cache area data.

[0077] It should be noted that by recording the current timestamp T1 when the data is stored in the local cache, when the storage time needs to be obtained, the current timestamp T2 is recorded again, and the storage time TT of the data in the local cache can be obtained by T2-T1.

[0078] Count the sizes of all data items in the cache, add them up to get the total size STO, then count the number of data items n in the cache, and calculate the average size Savg of the cache data, Savg=STO / n.

[0079] According to the unique identifier UID of the received information and the storage time TT of the data in the local buffer area, the data storage time indicator WT is calculated. The calculation formula is: ; Among them, TCU is the current time; TImax is the maximum allowed storage time.

[0080] It should be noted that the current time TCU is obtained by checking the current time from the clock setting of the computer system.

[0081] Get the maximum allowed storage time TImax by checking the system configuration file, which clearly stipulates the maximum allowed storage time of data in the local cache area, to obtain the maximum allowed storage time TImax.

[0082] It should be noted that Indicates the time difference from when the data is stored in the local cache to the current moment. The closer the data storage time is to the maximum allowed storage time, the smaller the WT value is, reflecting the impact of data storage time on the indicator. The purpose is to make a more comprehensive evaluation of the data storage time indicator based on the storage time and the characteristics of the received information. The formula is calculated by multiplying the exponential function result of the storage time difference by the UID information. Multiplying them together comprehensively considers the storage time of the data and the uniqueness of the received information, thereby obtaining an indicator WT that can measure the data storage time status.

[0083] According to the data storage time indicator WT, the average size of the cache data Savg, the data size SD, the data priority transmission indicator PDF and the unique identifier UID of the received information, the deletion decision indicator DQ is calculated. The calculation formula is: ; Where C is a constant.

[0084] It should be noted that to obtain the constant C, a simulation system environment is constructed to simulate the processes of data storage, transmission, and deletion. During the simulation process, different C values ​​are set to observe the operation of the system under different C values, such as the hit rate of the data cache, the data loss rate, and other indicators. Through multiple simulations and comparisons of the changes in these indicators under different C values, a C value that can achieve the optimal system performance is found. For example, after simulating 1,000 data operations, when C=0.2, the data cache hit rate is the highest and the data loss rate is the lowest, then C can be set to 0.2.

[0085] It should be noted that It means that when the data size SD is larger than the average size of the cache area Savg, the value of this item will increase, thereby increasing DQ. This shows that larger data has a higher weight in the decision and is more likely to be retained. Conversely, smaller data has a lower weight in the calculation and has less impact on DQ. It indicates that the larger the PDF, the smaller the DQ, which means that high-priority data is more likely to be retained and less affected in the deletion decision; Used to distinguish and process information of different sources or characteristics. The formula calculates DQ by combining factors such as data storage time indicator, data size and average buffer size, data priority transmission indicator, and unique identifier of received information.

[0086] Comprehensive step 401~step 402: Calculating the integrity assessment value of received data can evaluate data integrity based on the network connection status, providing a basis for subsequent operations. Calculating the unique identifier of received information and the data storage time index takes into account the relationship between data characteristics and storage time. Calculating the deletion decision index comprehensively considers various factors of the data, and can more reasonably determine whether to delete the decision data, improve the efficiency and accuracy of data storage and management, and optimize system resource utilization.

[0087] Step 5: Calculate the transmission reliability index TRI based on the deletion decision index DQ, encrypted data and computer system information data; calculate the abnormality judgment comprehensive index A based on the received data integrity assessment value VBI and the transmission reliability index TRI; preset the abnormality judgment comprehensive index threshold, compare the abnormality judgment comprehensive index A with the abnormality judgment comprehensive index threshold, and judge whether the data is normal based on the comparison result; if it is normal data, continue to transmit the data; if it is abnormal data, store it in a separate abnormal data storage area, and suspend the continued transmission of the data.

[0088] Step 501: Calculate the transmission reliability index TRI, the specific method is: According to the deletion decision index DQ, data size SD and network bandwidth BW, the transmission reliability index TRI is calculated. The calculation formula is: ; Wherein, t is the time to wait for receiving confirmation information; TL is the predetermined timeout period; and PN is the initial priority of the data.

[0089] It should be noted that the time t for waiting to receive confirmation information is obtained. During the data transmission process, when the sender sends data, a timer is started from the sending moment. When the confirmation information returned by the receiver is received, the timer is stopped. The time recorded by the timer is the time t for waiting to receive confirmation information.

[0090] Get the scheduled timeout TL. By checking the network protocol, it will clearly stipulate the maximum time to wait for confirmation after data is sent. This time is the scheduled timeout TL.

[0091] The initial priority PN of the data is obtained by checking the metadata information of the data.

[0092] It should be noted that It means that the error count and waiting time are combined and compared with the timeout time. If the error count is high or the waiting time is long and close to or exceeds the timeout time, the value of this part will increase, which means that the transmission reliability is reduced; otherwise, the transmission reliability is high. Represents the impact of deleting decision indicators on transmission reliability. Indicates the impact of data priority on transmission reliability. Taking into account the impact of error counts on reliability, the larger the denominator, the smaller the fraction value, indicating lower transmission reliability. The formula divides the numerator by the denominator, taking into account various factors in the data transmission process, including error conditions, waiting time, data deletion decisions, data priority, data size, and network bandwidth, and finally obtains an indicator TRI that can measure transmission reliability.

[0093] Step 502: Determine whether the data is abnormal. The specific method is: According to the received data integrity assessment value VBI and the transmission reliability index TRI, the abnormality judgment comprehensive index A is calculated based on the formula: ; Among them, CL is the data transmission delay time; CI is the network congestion index; TR is the data transmission rate.

[0094] It should be noted that to obtain the data transmission delay time CL, the data transmission time Ta and the receiving time Tb are recorded at the data transmitting end and the receiving end respectively, and the data transmission delay time CL is obtained by calculating Tb-Ta.

[0095] Obtain the network congestion index CI. In the network device, obtain the current length GC of the data queue and the maximum length GG of the queue through monitoring data, and calculate the network congestion index CI, CI=GC / GG×100%.

[0096] To obtain the data transmission rate TR, during the data transmission process, record the amount of data sent or received and the time it takes to transmit the data, then divide the amount of data by the time to get the data transmission rate TR. For example, if 10240 bytes of data are transmitted in 10 seconds, then the data transmission rate TR = 10240 / 10 = 1024 bytes / second.

[0097] It should be noted that This part substitutes the transmission reliability index TRI into the sine function calculation, reflecting the influence of the transmission reliability index TRI on the comprehensive index of abnormality judgment. This part comprehensively considers the data transmission delay time CL and the received data integrity assessment value VBI, reflecting the role of data integrity and transmission delay in abnormality judgment. It reflects the impact of network congestion and other conditions on abnormality determination. When network congestion is serious or other related factors are not conducive to data transmission, the value of this item will increase, thereby increasing A, indicating that the network condition has an important impact on abnormality determination. The formula adds the above three parts and comprehensively considers multiple factors such as transmission reliability, data integrity, network transmission delay, congestion, etc., to obtain a comprehensive indicator A that can comprehensively measure whether abnormalities occur during data transmission.

[0098] A threshold value Ath of the comprehensive indicator for abnormality determination is preset; in the network data transmission scenario, relevant data in different network environments such as different time periods, high bandwidth, low bandwidth environments, and different data types such as text data and image data are collected, including calculating the comprehensive indicator A for abnormality determination corresponding to each piece of data based on the received data integrity assessment value VBI, the transmission reliability index TRI, the data transmission delay time CL, the network congestion index CI, and the data transmission rate TR; the average value Aavg of the comprehensive indicator for abnormality determination is calculated, and the average value Aavg of the comprehensive indicator for abnormality determination is used as the threshold value Ath of the comprehensive indicator for abnormality determination.

[0099] When A≤Ath, the data is judged to be normal data and the data is continuously transmitted; when A>Ath, the data is judged to be abnormal data and stored in a separate abnormal data storage area, and the further transmission of the data is suspended.

[0100] Comprehensive step 501~step 502: By calculating the transmission reliability index, we can comprehensively measure the reliability of data transmission and timely discover transmission risks. Calculating the comprehensive index of abnormality judgment can comprehensively evaluate the data status. Preset thresholds and compare to determine whether the data is normal, accurately identify abnormal data, enable the system to handle abnormalities in a timely manner, ensure the accuracy and stability of data transmission and storage, improve the overall performance and reliability of the system, and reduce potential risks.

[0101] See also Figure 3 On the other hand, the present invention also discloses a configurable cloud-edge real-time database transmission system, which is used to implement the above-mentioned configurable cloud-edge real-time database transmission method, including: Data collection module, used to collect encrypted data of database and computer system information data; A data processing module is used to calculate the data priority transmission index PDF of different types of data based on the encrypted data; calculate the comprehensive priority evaluation value PDG according to the data priority transmission index PDF and the computer system information data; calculate the network status evaluation index NSV based on the computer system information data; calculate the data update frequency UF based on the computer system information data; A connection module is used to calculate a network connection establishment tendency value TDC according to a comprehensive priority evaluation value PDG, a network status evaluation index NSV and a data update frequency UF, preset a network connection establishment tendency value threshold, compare the network connection establishment tendency value TDC with the network connection establishment tendency value threshold, and judge whether the connection establishment is successful according to the comparison result; if the connection establishment fails, skip this data item and continue to check the next data item; The scheduling module, if the connection is successfully established, calculates the received data integrity assessment value VBI based on the network connection establishment tendency value TDC, and calculates the received information unique identifier UID based on the received data integrity assessment value VBI; calculates the data storage time indicator WT according to the received information unique identifier UID and the encrypted data; calculates the deletion decision indicator DQ according to the data storage time indicator WT, the received information unique identifier UID, the data priority transmission indicator PDF and the encrypted data; The judgment module is used to calculate the transmission reliability index TRI according to the deletion decision index DQ, the encrypted data and the computer system information data; calculate the abnormality judgment comprehensive index A according to the received data integrity assessment value VBI and the transmission reliability index TRI; preset the abnormality judgment comprehensive index threshold, compare the abnormality judgment comprehensive index A with the abnormality judgment comprehensive index threshold, and judge whether the data is normal according to the comparison result.

[0102] The above embodiments may be implemented in whole or in part by software, hardware, firmware or any other combination thereof. When implemented using software, the above embodiments may be implemented in whole or in part in the form of a computer program product. A person of ordinary skill in the art may appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein may be implemented in electronic hardware or in combination with computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution.

[0103] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0104] The above description is only a specific implementation manner of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application.

Claims

1. A configurable cloud-edge real-time database transmission method, characterized in that: include: Step 1: Collect the encrypted data of the database and the computer system information data; Step 2: Calculate the data priority transmission index PDF of different types of data based on the encrypted data; calculate the comprehensive priority evaluation value PDG according to the data priority transmission index PDF and the computer system information data; Calculate the network status evaluation index NSV based on the computer system information data; calculate the data update frequency UF based on the computer system information data; Step 3: Calculate the network connection establishment tendency value TDC based on the comprehensive priority evaluation value PDG, the network status evaluation index NSV and the data update frequency UF, preset the network connection establishment tendency value threshold, compare the network connection establishment tendency value TDC with the network connection establishment tendency value threshold, and judge whether the connection establishment is successful based on the comparison result; if the connection establishment fails, skip this data item and continue to check the next data item; Step 4: If the connection is established successfully, the received data integrity assessment value VBI is calculated based on the network connection establishment tendency value TDC, and the received information unique identifier UID is calculated based on the received data integrity assessment value VBI; Calculate the data storage time indicator WT according to the unique identifier UID of the received information and the encrypted data; Calculate the deletion decision indicator DQ according to the data storage time indicator WT, the unique identifier UID of the received information, the data priority transmission indicator PDF and the encrypted data; Step 5: Calculate the transmission reliability index TRI according to the deletion decision index DQ, the encrypted data and the computer system information data; Calculate the abnormality determination comprehensive index A according to the received data integrity assessment value VBI and the transmission reliability index TRI; Preset the abnormality determination comprehensive index threshold, compare the abnormality determination comprehensive index A with the abnormality determination comprehensive index threshold, and judge whether the data is normal according to the comparison result; If it is normal data, continue to transmit the data; If it is abnormal data, it is stored in a separate abnormal data storage area, and the further transmission of the data is suspended.

2. The configurable cloud-edge real-time database transmission method according to claim 1 is characterized in that: The method for calculating the comprehensive priority assessment value PDG is: The encrypted data includes the data size SD, data generation time TD and data access frequency AD of different types of data; Computer system information data includes system load index LO and system resource available evaluation value RP; According to the data size SD, data generation time TD and data access frequency AD, the data priority transmission index PDF of different types of data is calculated based on the formula: ； Among them, Smax is the maximum data size allowed by the system; Tmin is the minimum timestamp processed by the system; Tmax is the maximum timestamp processed by the system; e is a natural constant; According to the data priority transmission index PDF, the system load index LO and the system resource available evaluation value RP, the comprehensive priority evaluation value PDG is calculated based on the formula: ； Among them, MA is the system maintenance status coefficient; NA is the network status coefficient.

3. The configurable cloud-edge real-time database transmission method according to claim 2 is characterized in that: The method for calculating the network status evaluation index NSV is: The computer system information data also includes network bandwidth BW, network delay DL, packet loss rate PLR ​​and time interval RTR from the last successful transmission to the present; According to the network bandwidth BW, network delay DL, packet loss rate PLR ​​and the time interval RTR from the last successful transmission to the current time, the network status evaluation index NSV is calculated. The calculation formula is: ； Among them, BWmax is the maximum network bandwidth; DLmax is the maximum network delay; RTRmax is the maximum time interval from the last successful transmission to the current one; ω1 is The weight coefficient is 0.2~0.4; ω2 is The weight coefficient of ω3 is 0.1~0.3; ω4 is the weight coefficient of the packet loss rate PLR, which is 0.3~0.5; The weight coefficient is between 0.2 and 0.4; and ω1+ω2+ω3+ω4=1.

4. The configurable cloud-edge real-time database transmission method according to claim 3 is characterized in that: The method for calculating the data update frequency UF is: The computer system information data also includes the total number of data updates NG and the average data volume DA of the data updates within the time TE; According to the total number of data updates NG and the average data volume DA in the time TE, the data update frequency UF is calculated. The calculation formula is: ； Among them, CA is the average data processing capacity of the system.

5. The configurable cloud-edge real-time database transmission method according to claim 4 is characterized in that: The method to determine whether the connection is established successfully is: According to the comprehensive priority evaluation value PDG, the network status evaluation index NSV and the data update frequency UF, the network connection establishment tendency value TDC is calculated. The calculation formula is: ； Among them, φ1 is the weight coefficient of the comprehensive priority evaluation value PDG, which is 0.2~0.4; φ2 is the weight coefficient of the network status evaluation index NSV, which is 0.3~0.5; φ3 is the weight coefficient of the data update frequency UF, which is 0.1~0.5; and φ1+φ2+φ3=1; A network connection establishment tendency value threshold Tth is preset; when TDC≤Tth, the network connection establishment is judged to have failed, the data is skipped, and the next data is checked; when TDC>Tth, the network connection is judged to have been successfully established, and the TCP protocol is used to establish a connection based on the target cloud database address and transmit the data.

6. The configurable cloud-edge real-time database transmission method according to claim 5 is characterized in that: The method for calculating the unique identifier UID of the received information is: The trend value TDC is established based on the network connection, and the received data integrity assessment value VBI is calculated based on the formula: ； Among them, HR is the hash value of the received data; HS is the original hash value of the data when it was sent; DR is the amount of data received; DS is the amount of data sent; EF is the error count during data transmission; Based on the received data integrity assessment value VBI, the received information unique identifier UID is calculated according to the formula: ； Among them, TS is the timestamp of data reception.

7. The configurable cloud-edge real-time database transmission method according to claim 6 is characterized in that: The method for calculating the deletion decision index DQ is: The encrypted data also includes the storage time TT of the data in the local cache and the average size Savg of the cache data; According to the unique identifier UID of the received information and the storage time TT of the data in the local buffer area, the data storage time indicator WT is calculated. The calculation formula is: ； Among them, TCU is the current time; TImax is the maximum storage time allowed; According to the data storage time indicator WT, the average size of the cache data Savg, the data size SD, the data priority transmission indicator PDF and the unique identifier UID of the received information, the deletion decision indicator DQ is calculated. The calculation formula is: ； Where C is a constant.

8. The configurable cloud-edge real-time database transmission method according to claim 7 is characterized in that: The method for calculating the transmission reliability index TRI is: According to the deletion decision index DQ, data size SD and network bandwidth BW, the transmission reliability index TRI is calculated. The calculation formula is: ； Wherein, t is the time to wait for receiving confirmation information; TL is the predetermined timeout period; and PN is the initial priority of the data.

9. The configurable cloud-edge real-time database transmission method according to claim 8, characterized in that: The method to determine whether the data is abnormal is: According to the received data integrity assessment value VBI and the transmission reliability index TRI, the abnormality judgment comprehensive index A is calculated based on the formula: ； Among them, CL is the data transmission delay time; CI is the network congestion index; TR is the data transmission rate; A threshold value Ath of the comprehensive index for abnormal determination is preset; when A≤Ath, the data is judged to be normal data and the data is continuously transmitted; when A>Ath, the data is judged to be abnormal data and stored in a separate abnormal data storage area, and the further transmission of the data is suspended.

10. Configurable cloud-edge real-time database transmission system, characterized by: Data collection module, used to collect encrypted data of database and computer system information data; A data processing module is used to calculate data priority transmission indicators PDF of different types of data based on encrypted data; and to calculate a comprehensive priority evaluation value PDG according to the data priority transmission indicator PDF and computer system information data; Calculate the network status evaluation index NSV based on the computer system information data; calculate the data update frequency UF based on the computer system information data; A connection module is used to calculate a network connection establishment tendency value TDC according to a comprehensive priority evaluation value PDG, a network status evaluation index NSV and a data update frequency UF, preset a network connection establishment tendency value threshold, compare the network connection establishment tendency value TDC with the network connection establishment tendency value threshold, and judge whether the connection establishment is successful according to the comparison result; if the connection establishment fails, skip this data item and continue to check the next data item; The scheduling module calculates the received data integrity assessment value VBI based on the network connection establishment tendency value TDC if the connection is successfully established, and calculates the received information unique identifier UID based on the received data integrity assessment value VBI; Calculate the data storage time indicator WT according to the unique identifier UID of the received information and the encrypted data; Calculate the deletion decision indicator DQ according to the data storage time indicator WT, the unique identifier UID of the received information, the data priority transmission indicator PDF and the encrypted data; A judgment module, used for calculating a transmission reliability index TRI according to a deletion decision index DQ, encrypted data and computer system information data; According to the received data integrity assessment value VBI and the transmission reliability index TRI, the abnormality judgment comprehensive index A is calculated; the abnormality judgment comprehensive index threshold is preset, the abnormality judgment comprehensive index A is compared with the abnormality judgment comprehensive index threshold, and whether the data is normal is judged according to the comparison result.

Images