Scientific and technological intelligence resource integration method based on real-time data flow

Abstract

The invention provides a scientific and technological intelligence resource integration method based on a real-time data stream, which comprises the following steps: monitoring the task queue depth of each node through a load migration trigger condition, and if the queue depth of a certain node exceeds a preset upper limit value, starting a task migration process, selecting a target node with the shortest response time as a migration object in combination with the node response time statistical data, and obtaining a load redistribution instruction; and according to the load redistribution instruction, a processing result synchronization mechanism is adopted to coordinate the data processing progress among the nodes, if a source node completes task migration, a global task state table is updated, if a target node successfully receives a task, a confirmation signal is sent, and a closed-loop control mechanism for cooperative processing among the nodes is formed.

Description

Technical Field

[0001] The present invention relates to the technical field of data transmission, and in particular to a method for integrating scientific and technological intelligence resources based on real-time data streams. Background Art

[0002] As a core technology supporting modern information systems, the collection of scientific and technological intelligence plays a vital role in promoting technological innovation and supporting decision-making. With the accelerating pace of technological development, massive amounts of scientific and technological intelligence data need to be efficiently integrated, transmitted, and accurately processed in complex and ever-changing network environments, posing unprecedented challenges to data stream processing technologies.

[0003] Similar prior arts include a Chinese patent with publication number CN116567034B, which proposes a data transmission method for the Internet of Things, including: step S1, after the terminal acquisition device is matched with the terminal node, data is transmitted with the terminal node through a first short-range transmission method; step S2, the terminal node obtains the data transmission type of each terminal acquisition device and transmits the inspection data to the relay processing station through a second short-range transmission method; step S3, the relay processing station sets the number of terminal nodes, and converts the collected data into feature data for simplified representation; step S4, the relay processing station transmits each feature data to the cloud platform for storage through the first long-range transmission method; step S5, the control center periodically retrieves the feature data stored in the cloud platform for data stability analysis; by compressing the data volume of each collected data, the accuracy and efficiency of data transmission are improved.

[0004] Another similar prior art is a Chinese patent with publication number CN118590455A, which proposes a distributed organization interconnection method based on SDWAN, including the following steps: S1: determining a transmission classification and obtaining data packets to be transmitted; S2: determining a candidate path; and determining a transmission path based on selection parameters; S3: obtaining the transmission path of the highest priority data packet to be transmitted, calculating an evaluation score, and sorting the transmission paths according to the evaluation score; determining a reference path, fragmenting the data packet to be transmitted, and transmitting the first data fragment via the reference path; generating a new sorting order, and repeating the above steps to fragment and transmit data; S4: selecting an unused transmission path as a candidate path, and repeating the above steps to fragment and transmit all data packets to be transmitted. Multi-link load balancing can be achieved through routing selection and traffic optimization, ensuring the stability of data transmission between branches.

[0005] Although both of the above patent documents solve the data transmission problem, they lack a flexible modulation mechanism. The entire data stream transmission may encounter performance bottlenecks and even cause delays or failures in the processing of key scientific and technological intelligence. Summary of the Invention

[0006] The present invention provides a method for integrating scientific and technological intelligence resources based on real-time data streams, which mainly includes: Obtain the network transmission rate and round-trip delay parameters of the source data stream, determine the bandwidth status through the network monitoring module, and generate a bandwidth status identifier; According to the bandwidth status identifier, the data compression ratio and fragmentation strategy are adjusted, dynamic compression and preprocessing are performed on the data stream, an optimized data transmission unit is generated, and the optimized data transmission unit is sent to the forwarding unit; monitoring network stability of the data transmission unit during transmission to the forwarding unit, generating an adaptive flow control parameter configuration, and adjusting a data transmission rate; The forwarding unit receives and verifies the integrity of the data transmission unit and constructs a hierarchical priority queue; During the forwarding process, the resource usage status of each node in the forwarding unit is also evaluated, a node capability weight distribution table is generated, tasks are allocated based on the hierarchical priority queues, load balancing between nodes is coordinated, and the verified data stream is forwarded to the cloud server.

[0007] As a preferred technical solution of the present invention, the generation of the data transmission unit includes: According to the bandwidth status identifier, if the bandwidth is limited, the compression ratio is increased and the fine-grained fragmentation mode is selected; if the bandwidth is sufficient, the compression ratio is reduced and the standard fragmentation mode is maintained; a data packet is split according to the fragmentation mode to generate multiple data fragments; and differential compression processing is performed on the data fragments according to the degree of redundancy to generate compressed data transmission units.

[0008] As a preferred technical solution of the present invention, the adjustment of the data transmission rate includes: The packet loss rate and retransmission times of data transmission are obtained through the network congestion detection window; the comprehensive stability evaluation value is calculated in combination with the occupancy rate of the bandwidth reserved buffer; if the comprehensive stability evaluation value is lower than the preset threshold, it is marked as a network unstable state and the flow control time interval is increased; if the comprehensive stability evaluation value is higher than the preset threshold, it is marked as a network stable state and the flow control time interval is reduced.

[0009] As a preferred technical solution of the present invention, a hierarchical priority queue is constructed, including: The integrity of the data packet is detected through a checksum comparison mechanism. If the detection result is that the data is abnormal, the data compression ratio is reduced and the retransmission mechanism is triggered. If the detection result is that the data is complete, the current compression configuration is maintained; priority tags are assigned to the data packets based on the data subject and urgency, and inserted into the corresponding priority queue; the processing weight of the priority queue is dynamically adjusted according to the system load to construct a hierarchical priority queue.

[0010] As a preferred technical solution of the present invention, the generation of the node capability weight distribution table includes: Obtain resource utilization data of each forwarding node; compare the resource utilization data with a preset threshold value, and if it is higher than a high load threshold, reduce the node weight coefficient; if it is lower than a low load threshold, increase the node weight coefficient; generate the node capacity weight distribution table according to the adjusted weight coefficient.

[0011] As a preferred technical solution of the present invention, allocating tasks based on the hierarchical priority queue includes: According to the node capability weight distribution table, the task segmentation granularity control mechanism is used to subdivide the verified hierarchical priority queue according to data urgency and processing complexity. If the node weight is greater than the set weight, a large-grained task is assigned; if the node weight is less than or equal to the set weight, a small-grained task is assigned to obtain a task allocation plan.

[0012] As an optimal technical solution of the present invention, the weight acquisition time of each node in the node capability weight distribution table is obtained. When the difference between the weight acquisition times corresponding to any two nodes is less than or equal to the set time, task allocation is performed based on the priority queue corresponding to each node. Otherwise, the node capability weight distribution table is updated.

[0013] As a preferred technical solution of the present invention, coordinating load balancing between nodes also includes: The task queue depth of each node is monitored through the load migration trigger condition. If the queue depth of any node exceeds the preset upper limit, the task migration process is started. The node is used as the source node, and the target node with the shortest response time is selected as the migration object in combination with the node response time statistics to obtain the load redistribution instruction; according to the load redistribution instruction, the processing result synchronization mechanism is used to coordinate the data processing progress between the nodes. If the source node completes the task migration, the global task status table is updated. If the target node successfully receives the task, a confirmation signal is sent, forming a closed-loop control mechanism for collaborative processing between nodes.

[0014] The present invention also provides a scientific and technological information resource integration system based on real-time data stream, which is used to implement the above method. The system includes: A monitoring unit is used to obtain the network transmission rate and round-trip delay parameters of the source data stream, determine the bandwidth status through the network monitoring module, and generate a bandwidth status identifier; a processing unit, configured to adjust the data compression ratio and fragmentation strategy according to the bandwidth status identifier, perform dynamic compression and preprocessing on the data stream, generate an optimized data transmission unit, and send the optimized data transmission unit to the forwarding unit; The monitoring unit is further configured to monitor the network stability of the data transmission unit during the transmission process to the forwarding unit, generate an adaptive flow control parameter configuration, and adjust the data transmission rate; A forwarding unit is configured to receive and verify the integrity of the data transmission unit and construct a hierarchical priority queue; during the forwarding process, it also evaluates the resource usage status of each node in the forwarding unit, generates a node capacity weight distribution table, allocates tasks based on the hierarchical priority queue, coordinates load balancing between nodes, and forwards the verified scientific and technological intelligence data stream to the cloud server; The cloud server is used to receive the scientific and technological intelligence data stream.

[0015] The present invention also provides a computer-readable storage medium having instructions stored thereon, and the above-mentioned method is implemented when the instructions are executed by a processor.

[0016] Beneficial effects of the present invention: The present invention discloses a method or system for integrating scientific and technological intelligence resources. By monitoring the network transmission rate and the round-trip delay of data packets in real time, the system dynamically adjusts data compression parameters and fragmentation specifications, and combines network congestion detection and flow control strategies to ensure the reliability of data transmission. At the same time, the present invention evaluates the real-time processing capabilities of each node based on the CPU utilization and memory occupancy of the processing node, and performs task allocation and load balancing accordingly. Through task segmentation granularity control and load migration mechanisms, collaborative processing and optimal resource utilization between nodes are achieved. The present invention can effectively cope with network bandwidth fluctuations and changes in node processing capabilities, and improve the transmission efficiency and processing quality of scientific and technological intelligence data. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 Flowchart of a method for integrating scientific and technological intelligence resources based on real-time data streams according to an embodiment of the present invention; Figure 2 Flowchart of a method for adjusting data transmission rate according to an embodiment of the present invention; Figure 3 A flow chart of a method for allocating tasks based on hierarchical priority queues in an embodiment of the present invention; Figure 4 This is a structural diagram of the scientific and technological information resource integration system based on real-time data stream of the present invention. DETAILED DESCRIPTION

[0018] To help those skilled in the art better understand the technical solutions in this specification, the following will provide a clear and complete description of the technical solutions in the embodiments of this specification, in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of this specification, not all of them. All other embodiments derived by those skilled in the art based on the embodiments in this specification without creative effort shall fall within the scope of protection of this specification.

[0019] This embodiment is based on the technology information resource integration method of real-time data stream, such as Figure 1 As shown, including: Step S1: Obtain the network transmission rate and round-trip delay parameters of the source data stream, determine the bandwidth status through the network monitoring module, and generate a bandwidth status identifier; Specifically, when collecting the above-mentioned source data stream, i.e., the scientific and technological intelligence data stream, from the data source node, the above-mentioned data source node can be an academic website, a patent database, or a technical forum and academic conference, and sending it from the above-mentioned data collection source node to the forwarding unit, and then forwarding it to the cloud server by the forwarding unit, in the process of sending it from the data source node to the above-mentioned forwarding unit, the network transmission rate and the round-trip delay of the data packet are monitored and obtained in real time by the monitoring unit to evaluate the network bandwidth and delay conditions; and the above-mentioned network transmission rate and the round-trip delay of the data packet are used as performance indicator data of the transmission link. These parameters can reflect the bandwidth utilization of the network and the delay characteristics during the data transmission process. If the network bandwidth is lower than the preset threshold, that is, the network transmission rate is less than the set rate or the round-trip delay parameter is greater than the set delay, the system will determine that the bandwidth is limited; otherwise, it is considered that the bandwidth is sufficient. With this information, the system can determine the network load status and provide input data for subsequent data compression and fragmentation strategies. The above technical solution, by monitoring the network transmission rate and latency, lays the foundation for dynamically adjusting data transmission strategies, such as compression ratio and fragmentation strategy, thereby improving the efficiency and reliability of data transmission. It can also help the system respond quickly when the network status changes, avoiding data loss or transmission delays caused by network congestion or insufficient bandwidth.

[0020] Step S2: According to the bandwidth status identifier, adjust the data compression ratio and fragmentation strategy, perform dynamic compression and preprocessing on the data stream, generate an optimized data transmission unit, and send it to the forwarding unit; Specifically, when bandwidth is limited, the compression ratio of the source data node when sending the scientific and technological intelligence data is increased, and a fine-grained fragmentation mode is adopted to divide the data packets into smaller fragments to reduce the total amount of data transmission and alleviate bandwidth pressure; when bandwidth is sufficient, the system will reduce the compression ratio and maintain the standard fragmentation mode, thereby maintaining a high data transmission speed and low processing overhead. In this way, the system can automatically optimize the processing strategy of the data flow according to the real-time network conditions and obtain the above-mentioned data transmission unit, wherein the above-mentioned data transmission unit includes a plurality of the above-mentioned scientific and technological intelligence data. The above-mentioned technical solution, by increasing the data compression ratio and adopting fine-grained fragmentation, can ensure that data is transmitted as efficiently as possible in a lower bandwidth environment to avoid data loss; and when bandwidth is sufficient, appropriately reducing the compression ratio can increase data processing speed and reduce unnecessary computing resource consumption, thereby optimizing the utilization of network resources and significantly improving data transmission efficiency.

[0021] Step S3: monitoring the network stability of the data transmission unit during the transmission process to the forwarding unit, generating an adaptive flow control parameter configuration, and adjusting the data transmission rate; Specifically, the above-mentioned monitoring unit also continuously monitors the network stability of the above-mentioned data transmission unit during the transmission process through the network congestion detection window, focusing on the packet loss rate, the number of retransmissions, and the occupancy rate of the bandwidth reserved buffer. These parameters can reflect the network load and transmission stability. If the system detects that the packet loss rate exceeds the preset threshold, or the network is unstable, the system will increase the flow control time interval to slow down the data transmission speed; if the network is stable, the flow control time interval will be shortened to increase the data transmission speed. By dynamically optimizing the data transmission rate according to the network stability, the stable transmission of the data stream is ensured and network congestion is avoided. The above-mentioned technical solution, by monitoring the network stability in real time, can automatically adjust the transmission rate when the network becomes unstable, avoiding network congestion or packet loss caused by excessively high data transmission rates, thereby ensuring the accuracy and timeliness of the data. This adaptive flow control strategy can significantly improve the network's resource utilization and data transmission quality.

[0022] Step S4: The forwarding unit receives and verifies the integrity of the data transmission unit and constructs a hierarchical priority queue; Specifically, the integrity of the data packet is ensured through a check code comparison mechanism. If the check result shows that the data is abnormal, the system will trigger the retransmission mechanism and reduce the compression ratio. If the data verification passes, the system will assign different priority labels to the data packet according to the subject and urgency of the scientific and technological intelligence data. After that, the system will store the data in different queues according to the priority and process it according to the priority order of the queues. The above technical solution can not only detect and correct errors in data transmission in a timely manner, reduce the possibility of data loss or errors, but also ensure that critical tasks are processed in the shortest time, while reducing the processing delay of priority data.

[0023] Step S5: During the forwarding process, the resource usage status of each node in the forwarding unit is evaluated, a node capability weight distribution table is generated, tasks are allocated and load balancing between nodes is coordinated, and the verified data stream is forwarded.

[0024] Specifically, by monitoring the resource usage of each node in the forwarding unit, i.e. the forwarder, such as CPU usage, memory occupancy, etc., and comparing the resource usage data with the preset threshold, if the load of a node is high, the system will reduce the weight coefficient of the node and reduce its forwarding task allocation; if the node's resource usage is low, the weight coefficient of the node will be increased and more forwarding tasks will be allocated. Based on the weight distribution of node capabilities, the system adopts a task segmentation granularity control mechanism to adjust the granularity and complexity of the task, thereby achieving load balancing. The above technical solution, by dynamically adjusting node weights and task allocation, can prevent the overload of a single node, improve overall processing efficiency and response speed, and reduce performance bottlenecks caused by uneven node resources.

[0025] The generation of the data transmission unit includes: according to the bandwidth status identifier, if the bandwidth is limited, increasing the compression ratio and selecting the fine-grained fragmentation mode; if the bandwidth is sufficient, reducing the compression ratio and maintaining the standard fragmentation mode; performing a segmentation operation on the data packet according to the fragmentation mode to generate multiple data fragments; performing differential compression processing on the data fragments according to the degree of redundancy to generate a compressed data transmission unit.

[0026] Specifically, the data packet fragmentation strategy selector is started according to the network status identifier. If the bandwidth limited identifier is activated, the fine-grained fragmentation mode is selected and the small fragment specification parameters are set. If the bandwidth sufficient identifier is activated, the standard fragmentation mode is selected and the conventional fragment specification parameters are maintained to obtain the fragmentation mode configuration and fragment specification definition. The fragmentation mode configuration is used to perform a fragmentation operation on the scientific and technological intelligence data packet. The original data packet structure information is read through the data packet parser. The fragment boundary position and the number of fragments are calculated according to the fragment specification definition. The data packet content is fragmented according to the fragment boundary to generate multiple data fragments and a fragment index table. The size attributes and content characteristics of each data fragment are obtained through the fragment index table. The Bayesian classification algorithm is used to analyze the data type distribution and redundancy of the fragment content. If the fragment redundancy is higher than the preset threshold, it is marked as a high-redundancy fragment. If the redundancy is lower than the threshold, it is marked as a low-redundancy fragment to determine the fragment redundancy level classification. The compression ratio adjustment parameters are set according to the bandwidth status classification results and the fragment redundancy level classification. If the bandwidth is limited and the fragment is of high redundancy type, a high compression ratio range is set. If the bandwidth is sufficient and the fragment is of low redundancy type, a high compression ratio range is set. The adaptive compression ratio configuration scheme sets a low compression ratio range for each data segment. The adaptive compression ratio configuration scheme performs differential compression on each data segment, selecting corresponding compression strength parameters based on segment characteristics. A strong compression mode is applied to high-redundancy segments, while a light compression mode is applied to low-redundancy segments. This generates a compressed data segment set. The data segment reassembly module reads the compressed data segment set and the segment index table. Based on the segment sequence information and boundary markers in the index table, the compressed segments are reassembled according to the original data packet structure, restoring the complete compressed data packet format and obtaining an optimized scientific and technological intelligence data transmission unit. For example, the system monitors the current bandwidth status in real time and uses a sliding window algorithm to calculate the average bandwidth utilization over the last 10 seconds. When the utilization exceeds 75%, it is determined to be bandwidth-limited. In this bandwidth-limited situation, the packet fragmentation mechanism segments the original 1500-byte Ethernet frame into smaller 512-byte segments. The LZMA compression algorithm is then used to adjust the compression ratio to 87%. The system uses entropy coding to identify repetitive patterns in scientific and technological intelligence data and Huffman coding to map high-frequency words into shorter binary sequences, maximizing compression while ensuring data integrity. When network monitoring indicates sufficient bandwidth and utilization below 40%, the system maintains the original 1500-byte fragment size and uses the LZ77 fast compression algorithm at a 28% compression ratio, prioritizing processing speed over efficiency. The system then processes multiple data fragments simultaneously through a parallel processing mechanism, leveraging multi-threading to keep CPU utilization below 60% and ensure real-time transmission performance. Throughout the preprocessing process, the system continuously monitors network latency and packet loss rates, automatically triggering parameter reconfiguration when changes in network conditions are detected, dynamically matching bandwidth status with data processing strategies.

[0027] Furthermore, the data transmission rate is adjusted as follows: Figure 2 As shown, including: The packet loss rate and retransmission times of data transmission are obtained through the network congestion detection window; the comprehensive stability evaluation value is calculated in combination with the occupancy rate of the bandwidth reserved buffer; if the comprehensive stability evaluation value is lower than the preset threshold, it is marked as a network unstable state and the flow control time interval is increased; if the comprehensive stability evaluation value is higher than the preset threshold, it is marked as a network stable state and the flow control time interval is reduced.

[0028] Specifically, real-time monitoring data of the network congestion detection window is obtained, the packet loss rate statistics and retransmission count records of the current transmission link are read, and the occupancy rate and queue length status of the bandwidth reserved buffer are collected at the same time to obtain a set of network transmission performance indicators. A comprehensive stability evaluation value is calculated based on the network transmission performance indicator set. A weighted average algorithm is used to comprehensively score the packet loss rate, retransmission count, and buffer occupancy rate. If the evaluation value is lower than the preset stability threshold, the network is marked as unstable. If the evaluation value is higher than the threshold, the network is marked as stable, and the current network stability level is determined. The flow control strategy is triggered by the network stability level. If the network unstable state is activated, the conservative transmission mode is activated and a longer flow control time interval is set. If the network stable state is activated, the aggressive transmission mode is activated and a shorter time interval is set. An adaptive flow control parameter configuration is generated and the data transmission rate is dynamically adjusted using the adaptive flow control parameter configuration. The flow regulator controls the packet sending frequency according to the time interval parameter. In the conservative mode, the sending rate is reduced to reduce the network load. In the aggressive mode, the sending rate is increased to fully utilize bandwidth resources, and an optimized transmission rate control scheme is obtained. A packet sending queue management mechanism is established based on the transmission rate control scheme. The queue scheduler arranges the sending timing of the data packets according to the control parameters, assigns shorter waiting times to high-priority data packets, and longer waiting times to low-priority data packets, forming a hierarchical transmission queue structure. The hierarchical transmission queue structure is used to perform orderly data packet sending operations, and a time slice round-robin scheduling algorithm is used to control the data packet sending ratio of each priority queue. The time slice allocation weight of each queue is dynamically adjusted according to the network status to generate a real-time transmission scheduling instruction set. The real-time transmission scheduling instruction set is used to drive the network interface to execute the data packet sending task. The transmission status and confirmation reply of each data packet are monitored through the transmission controller. If the sending fails, the retransmission mechanism is triggered and the network stability evaluation parameters are updated. If the sending is successful, the transmission performance statistics are recorded to obtain a complete network flow control execution result. For example, the network congestion detection window uses a sliding window algorithm to continuously monitor data transmission status, with a window size of 1000 packets. The packet loss rate and retransmission count are calculated every second. The system assesses network quality in real time by calculating the packet loss rate (the number of lost packets divided by the total number of sent packets multiplied by 100%). When 25 out of 1000 packets are lost, the packet loss rate reaches 2.5%, exceeding the threshold. The system also records 38 retransmissions, indicating significant network congestion. The bandwidth reserved buffer usage analysis module assesses network stability by monitoring buffer occupancy. When buffer occupancy exceeds 80% and persists for more than three seconds, the network is deemed unstable. Given that the packet loss rate of 2.5% exceeds the preset threshold of 2%, the flow control policy automatically triggers the congestion avoidance mechanism, adjusting the original 100-millisecond flow control interval to 500 milliseconds, alleviating network pressure by reducing the frequency of data transmission. The system uses an exponentially weighted moving average algorithm to calculate the network stability index. The formula is: the current stability index equals 0.8 times the stability index at the previous moment plus 0.2 times the current measurement value. When the packet loss rate is less than 1.5% and the buffer utilization rate is less than 60% for five consecutive monitoring cycles, the system determines that the network has returned to a stable state and automatically adjusts the flow control interval back to 100 milliseconds to improve transmission efficiency.

[0029] Furthermore, a hierarchical priority queue is constructed, including: The integrity of the data packet is detected through a check code comparison mechanism. If the detection result is that the data is abnormal, the data compression ratio is reduced and a retransmission mechanism is triggered. If the detection result is that the data is complete, the current compression configuration is maintained; priority tags are assigned to the data packets according to the data subject and urgency, and inserted into the corresponding priority queue; the processing weight of the priority queue is dynamically adjusted according to the system load to obtain the hierarchical priority queue.

[0030] Specifically, each node in the forwarding unit receives the data packet corresponding to the data transmission unit, i.e., the scientific and technological intelligence data packet, extracts the data header information and payload content from the data packet, and uses a hash function to calculate the data packet's checksum value. If the calculated result matches the received checksum, the data is marked as complete; if not, the data is marked as abnormal, thereby obtaining a data integrity check result. Branch processing is performed based on the data integrity check result. If the data is abnormal, the compression parameter adjuster reduces the compression ratio and repackages the data packet. If the data is intact, the original compression setting is maintained. Through data packet reconstruction, a data packet corresponding to the scientific and technological intelligence data in a standard format is generated, and a data processing mode is determined. The data processing mode triggers a transmission control mechanism. If the abnormal data mode is activated, a retransmission request is initiated to the sender and the number of retransmissions is recorded. If the normal data mode is activated, the retransmission step is skipped and subsequent processing is directly performed to obtain transmission status confirmation information. The transmission status confirmation information is used for priority classification. The subject category and urgency of the scientific and technological intelligence data are analyzed using a Bayesian classification algorithm. Based on the classification results, the scientific and technological intelligence data is assigned three priority labels: high, medium, and low, generating data records with priority attributes. A multi-level queue storage structure is established based on scientific and technological intelligence data records with priority attributes. Through the queue manager, high-priority scientific and technological intelligence data is inserted into the fast processing queue, medium-priority scientific and technological intelligence data into the standard processing queue, and low-priority scientific and technological intelligence data into the delayed processing queue, forming a hierarchical data caching system. Data scheduling operations are performed through this hierarchical data caching system, using a timestamp sorting mechanism to arrange scientific and technological intelligence data within queues of the same priority level by reception time. The processing weight allocation of each queue in each forwarding unit is dynamically adjusted based on the load of each node in the corresponding node, resulting in an optimized scientific and technological intelligence data processing sequence. The optimized scientific and technological intelligence data processing sequence is used to start the data output controller, and the scientific and technological intelligence data in the queue is converted into the data format required by the cloud server through the interface adapter module. If the format conversion is successful, the scientific and technological intelligence data is pushed to the target processing module. If the conversion fails, the data is re-queued and waits for reprocessing to obtain a verified scientific and technological intelligence data stream; exemplarily, the scientific and technological intelligence data integrity verification system uses the CRC-32 cyclic redundancy check algorithm to verify the received data blocks, and the size of each data block is set to 4096 bytes. The system generates a 32-bit check code for each data block and compares it with the check code of the sending end. When the system receives a batch of scientific and technological intelligence data, it detects that the check codes of 12 data blocks do not match, and the check failure rate reaches 3.2%, triggering the data reconstruction mechanism.The system immediately invokes an adaptive compression algorithm to reduce the original GZIP compression ratio from 75% to 65%, reducing the compression depth to improve data transmission reliability. At the same time, it initiates a selective retransmission request to the sender, requesting the retransmission of data blocks that failed verification. The retransmission mechanism uses the ARQ automatic retransmission protocol, setting the maximum number of retransmissions to 3 times and the timeout to 200 milliseconds. After the retransmitted data passes verification, the system calculates the transmission priority weight value based on the data type and timeliness. For example: the weight of patent data is set to 0.8, the weight of academic papers is set to 0.6, and the weight of technical reports is set to 0.4. Combined with the data reception timestamp and urgency indicator, the system sorts the verified data from high to low priority scores and inserts them into the transmission queue, forming a verified data queue containing multiple scientific and technological intelligence records awaiting subsequent analysis and processing. The above technical solution effectively reduces the possibility of data loss or errors, ensures priority processing of critical data, and thus improves the stability of data transmission and system processing efficiency.

[0031] Furthermore, the generation of the node capability weight distribution table includes: Obtain resource utilization data of each forwarding node; compare the resource utilization data with a preset threshold value, and if it is higher than a high load threshold, reduce the node weight coefficient; if it is lower than a low load threshold, increase the node weight coefficient; generate the node capacity weight distribution table according to the adjusted weight coefficient.

[0032] Specifically, each forwarding device in the distributed forwarding unit is used as the above-mentioned forwarding node, and the resource usage status of the processing node is evaluated. The node's capacity weight is adjusted according to the resource utilization rate. That is, by monitoring the node's CPU utilization rate and memory usage, the resource utilization data of each node is obtained. If the resource utilization rate of a node exceeds the preset high load threshold, its weight coefficient is reduced; if the resource utilization rate is lower than the low load threshold, its weight coefficient is increased. The adjustment of the weight coefficient reflects the processing capacity of the node. Nodes with lower weights may face insufficient processing capacity, while nodes with higher weights indicate that they have sufficient resources and can undertake more or more complex tasks. Based on the node's weight coefficient, the system can dynamically adjust the distribution of tasks and load balancing. For highly loaded nodes, the system will reduce their task allocation to prevent overload; for low-load nodes, the system will increase task allocation to fully utilize their resources. The establishment of the node capacity weight distribution table provides a basis for task allocation, making task allocation more reasonable and avoiding resource waste or node overload. The above technical solution can dynamically adjust the node processing capacity weight according to real-time resource usage, ensuring that the load of each node is within a reasonable range, improving the system's resource utilization, and at the same time, by adjusting the distribution of tasks, ensuring the system's load balance, thereby improving the overall processing efficiency and response speed, and avoiding system performance bottlenecks caused by uneven resources of individual nodes.

[0033] Furthermore, tasks are assigned and load balancing between nodes is coordinated, e.g. Figure 3 As shown, including: According to the node capability weight distribution table, the task segmentation granularity control mechanism is used to subdivide the verified hierarchical priority queue according to data urgency and processing complexity. If the node weight is greater than the set weight, a large-grained task is assigned; if the node weight is less than or equal to the set weight, a small-grained task is assigned to obtain a task allocation plan.

[0034] Specifically, the above-mentioned scientific and technological intelligence data is forwarded to the cloud server through each node in the forwarding unit, wherein the forwarding unit is a distributed forwarding device, and each of the above-mentioned forwarding devices is regarded as a processing node. The resource usage of each node is evaluated through the node capacity weight distribution table. The above-mentioned resource usage can be CPU usage and memory occupancy. If a node has strong processing power, that is, low CPU usage or low memory occupancy, its assigned tasks are more complex and include more data transmission units in the above-mentioned hierarchical priority queue, so that its high-performance resources can be better utilized; if a node has poor processing power, the system will assign simpler and fewer data transmission units to avoid overload; in addition, the complexity and urgency of the data will be considered in the task allocation process, and critical tasks or urgent data will be given priority to ensure that the corresponding sending tasks of the high-priority queue are processed in a timely manner. During the actual execution process, the system will monitor the task queue depth of each node, that is, the depth of the above-mentioned hierarchical priority queue. If the queue depth of a node exceeds the preset upper limit, the task migration process will be triggered, and the target node with the shortest response time will be selected as the migration object, thereby maintaining the load balance of the forwarding unit. During the migration process, the task status of each node is synchronously updated to the global task status table, and the cloud server sends a confirmation signal to ensure the successful reception and processing of the task. This technical solution can dynamically adjust task allocation and load balancing, improve the collaborative processing capabilities between nodes, and thus achieve efficient and stable transmission and processing of scientific and technological intelligence data streams.

[0035] Furthermore, coordinating load balancing between nodes also includes: The task queue depth of each node is monitored through the load migration trigger condition. If the queue depth of any node exceeds the preset upper limit, the task migration process is started. Combined with the node response time statistics, the target node with the shortest response time is selected as the migration object to obtain the load redistribution instruction; according to the load redistribution instruction, the processing result synchronization mechanism is used to coordinate the data processing progress between the nodes. If the source node completes the task migration, the global task status table is updated. If the target node successfully receives the task, a confirmation signal is sent, forming a closed-loop control mechanism for collaborative processing between nodes.

[0036] Specifically, the task queue depth of each node is monitored using the aforementioned load migration conditions. The task migration executor is activated using a load redistribution strategy. A task priority sorting algorithm is used to prioritize the tasks corresponding to the migrating nodes based on urgency and resource requirements. High-priority tasks are preferentially migrated to high-performance nodes, while low-priority tasks are migrated to standard nodes. This determines the task migration sequence. Based on the task migration sequence, a data transfer controller is driven. A network bandwidth detection module is used to obtain the real-time bandwidth status between nodes. If sufficient bandwidth is available, data transfer is initiated directly. If insufficient bandwidth is detected, data transfer is delayed until network conditions improve. This results in a data transfer plan. The data transfer plan is used to activate the task state synchronizer. A state code mapping table is used to convert the source node's task execution status information into a standard format. If the task status is executing, state continuity is maintained. If the task status is paused, execution is resumed at the target node, generating a state synchronization record. This state synchronization record triggers a resource reservation mechanism. A resource occupancy prediction module is used to calculate the resource requirements of the migrating task at the target node. If the target node has sufficient available resources, resources are allocated directly. If insufficient resources are available, resources occupied by lower-priority tasks are released, thus determining a resource allocation plan. The task restart controller is started according to the resource allocation plan. The task execution environment is rebuilt on the target node through the environment configuration module. If the environment configuration is successful, the task execution process is started. If the configuration fails, the task is rolled back to the source node to continue execution and obtain the task execution status. The task execution status is used to drive the load balancing validator. The load distribution calculation module re-evaluates the load balancing degree of each node. If the load distribution reaches a balanced state, the migration process is completed. If there is still load imbalance, the task distribution is further adjusted to generate a load balancing report. The performance monitoring module is activated through the load balancing report. The K-means clustering algorithm is used to classify and analyze the performance indicators of each node. If the node performance indicators are concentrated in the high-performance range, they are marked as high-quality nodes. If they are concentrated in the low-performance range, they are marked as nodes to be optimized, and the node performance evaluation results are generated.

[0037] More specifically, the state monitoring module is activated through the data synchronization controller. A polling mechanism is used to periodically scan task execution status changes at each node. If a source node's task status changes from executing to migration completed, the state transition timestamp is recorded. If a target node's status changes from idle to task receiving, the node status change is marked, resulting in a node state change sequence. This node state change sequence is then driven by a confirmation response processor, which collects state confirmation signals sent by each node through a message queue mechanism. Upon receiving a migration completion confirmation from the source node, the task migration record table is updated. Upon receiving a task acceptance confirmation from the target node, the node load distribution table is modified, forming a set of bidirectional confirmation records. This set of bidirectional confirmation records is then used to initiate a global state updater, which synchronously updates the distributed state storage system through a database transaction control mechanism. If all nodes confirm the completeness of the information, the state update transaction is committed. If any confirmation information is missing, the transaction is rolled back and confirmation information is collected again to determine the global state consistency. Based on this global state consistency result, the collaborative processing verification module is triggered. A hash algorithm is used to calculate a checksum for each node's state data. If the checksum matches the expected value, inter-node data synchronization is verified. If the checksum does not match, a data repair process is initiated to resynchronize the node states and generate a collaborative processing verification report. The exception handling controller is activated based on the collaborative processing verification report. A state machine model is used to manage the inter-node exception recovery process. If node communication is interrupted, data transmission is resumed via a backup communication link. If a data transmission timeout is detected, the transmission request is re-initiated to obtain the exception handling execution result. Based on the exception handling execution result, a performance evaluation analyzer is launched. Statistical analysis is used to calculate the response time and success rate of inter-node collaborative processing. If the response time is below a preset threshold and the success rate is above a standard value, the collaborative processing performance is marked as good. If the indicators do not meet the standards, inter-node communication parameters are adjusted to generate collaborative processing performance evaluation data. The collaborative processing performance evaluation data drives the optimization strategy generator. A rule engine analyzes performance bottleneck patterns in historical collaborative processing data. If a specific node combination is identified as having low collaborative efficiency, the task allocation strategy is adjusted to avoid this combination. If the collaborative processing load is too high during certain periods, a staggered processing plan is developed to determine the collaborative processing optimization solution. This technical solution dynamically adjusts load distribution by monitoring the node task queue depth and response time in real time, ensuring optimal utilization of each node's processing capacity. At the same time, the system maintains the high priority and urgency of tasks during the load redistribution process, ensuring the timely processing and transmission of scientific and technological intelligence data. It can effectively cope with uneven network load and fluctuations in resource usage, improve the stability and efficiency of the entire data stream processing system, and reduce system bottlenecks caused by node overload or network congestion.

[0038] Furthermore, the resource utilization acquisition time corresponding to each node in the node capability weight distribution table is obtained. When the difference between the acquisition time corresponding to any two nodes is less than or equal to the set time, task allocation is performed based on the priority queue corresponding to each node. Otherwise, the node capability weight distribution table is updated again.

[0039] Specifically, since in the process of forwarding scientific and technological intelligence data, the load and processing capacity of each node in the forwarding unit are different, there may be a delay when obtaining the resource utilization data of each forwarding node for nodes with heavier loads. Therefore, the acquisition time of the above-mentioned resource utilization corresponding to each node in the above-mentioned node capability weight distribution table is obtained, and the difference in acquisition time between any two of the above-mentioned nodes is calculated. When the above-mentioned difference is less than or equal to the above-mentioned set time, it is considered that the above-mentioned resource utilization data between any two of the above-mentioned nodes are within the same time period, that is, synchronized. The node capability weight distribution table calculated based on the above-mentioned resource utilization data is accurate, so the next task allocation can be carried out. Otherwise, the above-mentioned node capability weight distribution table is considered inaccurate. Therefore, the above-mentioned node capability weight distribution table can be updated again. The above-mentioned technical solution can improve the accuracy of the above-mentioned node capability weight distribution table, thereby improving the accuracy of the above-mentioned task allocation, and further improving the transmission efficiency of scientific and technological intelligence data.

[0040] The present invention also provides a scientific and technological information resource integration system for implementing the above method, such as Figure 4 As shown, the system includes: A monitoring unit is used to obtain the network transmission rate and round-trip delay parameters of the source data stream, determine the bandwidth status through the network monitoring module, and generate a bandwidth status identifier; a processing unit, configured to adjust the data compression ratio and fragmentation strategy according to the bandwidth status identifier, perform dynamic compression and preprocessing on the data stream, generate an optimized data transmission unit, and send the optimized data transmission unit to the forwarding unit; The monitoring unit is further configured to monitor the network stability of the data transmission unit during the transmission process to the forwarding unit, generate an adaptive flow control parameter configuration, and adjust the data transmission rate; A forwarding unit is configured to receive and verify the integrity of the data transmission unit and construct a hierarchical priority queue; during the forwarding process, it also evaluates the resource usage status of each node in the forwarding unit, generates a node capacity weight distribution table, allocates tasks based on the hierarchical priority queue, coordinates load balancing between nodes, and forwards the verified scientific and technological intelligence data stream to the cloud server; The cloud server is used to receive the scientific and technological intelligence data stream.

[0041] The present invention also provides a computer-readable storage medium having instructions stored thereon, and the above-mentioned method is implemented when the instructions are executed by a processor.

[0042] In summary, the present invention monitors network transmission rate and round-trip delay in real time to assess bandwidth status, providing fundamental information for subsequent data processing and compression strategies. This provides a basis for the system to adjust data compression ratios and fragmentation strategies when facing bandwidth limitations. Next, based on the bandwidth status indicator, data compression and fragmentation strategies are dynamically adjusted to improve data transmission efficiency. When bandwidth is limited, data transmission volume is optimized by increasing the compression ratio and using a fine-grained fragmentation mode. When bandwidth is sufficient, the compression ratio is reduced and a standard fragmentation mode is maintained, ensuring efficient resource utilization. During data transmission, adaptive flow control parameters are generated by monitoring network stability, and the data transmission rate is adjusted based on network conditions to ensure data flow stability and efficiency. When data arrives at the forwarding unit, data integrity is verified and priority is assigned based on data importance and urgency, forming a priority queue to ensure timely processing of critical data. Finally, by evaluating the resource usage status of each node, a node capacity weight distribution table is generated. Load balancing is coordinated based on resource usage and task allocation requirements. Through mechanisms such as task migration, task splitting, and load redistribution between nodes, dynamic resource adjustment and optimal utilization are effectively achieved. The coordinated cooperation of this series of steps not only ensures the efficient transmission and processing of scientific and technological intelligence data, but also improves the adaptability and stability of the system in complex environments, thereby improving the performance and response speed of the overall system.

[0043] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the above embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A resource integration method based on real-time data stream, characterized in that: include: Obtain the network transmission rate and round-trip delay parameters of the source data stream, determine the bandwidth status through the network monitoring module, and generate a bandwidth status identifier; According to the bandwidth status identifier, the data compression ratio and fragmentation strategy are adjusted, dynamic compression and preprocessing are performed on the data stream, an optimized data transmission unit is generated, and the optimized data transmission unit is sent to the forwarding unit; monitoring network stability of the data transmission unit during transmission to the forwarding unit, generating an adaptive flow control parameter configuration, and adjusting a data transmission rate; The forwarding unit receives and verifies the integrity of the data transmission unit and constructs a hierarchical priority queue; During the forwarding process, the resource usage status of each node in the forwarding unit is also evaluated, a node capability weight distribution table is generated, tasks are allocated based on the hierarchical priority queues, load balancing between nodes is coordinated, and the verified data stream is forwarded to the cloud server.

2. The method according to claim 1, characterized in that The generation of the data transmission unit includes: According to the bandwidth status identifier, if the bandwidth is limited, the compression ratio is increased and the fine-grained fragmentation mode is selected; if the bandwidth is sufficient, the compression ratio is reduced and the standard fragmentation mode is maintained; a data packet is split according to the fragmentation mode to generate multiple data fragments; and differential compression processing is performed on the data fragments according to the degree of redundancy to generate compressed data transmission units.

3. The method according to claim 1, characterized in that The adjustment of the data transmission rate includes: The packet loss rate and retransmission times of data transmission are obtained through the network congestion detection window; the comprehensive stability evaluation value is calculated in combination with the occupancy rate of the bandwidth reserved buffer; if the comprehensive stability evaluation value is lower than the preset threshold, it is marked as a network unstable state and the flow control time interval is increased; if the comprehensive stability evaluation value is higher than the preset threshold, it is marked as a network stable state and the flow control time interval is reduced.

4. The method according to claim 1, wherein Build a hierarchical priority queue, including: The integrity of the data packet is detected through a checksum comparison mechanism. If the detection result is that the data is abnormal, the data compression ratio is reduced and the retransmission mechanism is triggered. If the detection result is that the data is complete, the current compression configuration is maintained; priority tags are assigned to the data packets based on the data subject and urgency, and inserted into the corresponding priority queue; the processing weight of the priority queue is dynamically adjusted according to the system load to construct a hierarchical priority queue.

5. The method according to claim 1, wherein Generation of node capability weight distribution table, including: Obtain resource utilization data of each forwarding node; compare the resource utilization data with a preset threshold value, and if it is higher than a high load threshold, reduce the node weight coefficient; if it is lower than a low load threshold, increase the node weight coefficient; generate the node capacity weight distribution table according to the adjusted weight coefficient.

6. The method according to claim 1, characterized in that Allocating tasks based on the hierarchical priority queues includes: According to the node capability weight distribution table, the verified hierarchical priority queues are subdivided according to data urgency and processing complexity using a task segmentation granularity control mechanism. If the node weight is higher, large-grained tasks are assigned; if the weight is lower, small-grained tasks are assigned, thereby obtaining a task allocation plan.

7. The method according to claim 6, characterized in that Obtain the weight acquisition time of each node in the node capability weight distribution table. Only when the difference between the weight acquisition times corresponding to any two nodes is less than or equal to the set time, perform task allocation based on the priority queue corresponding to each node. Otherwise, update the node capability weight distribution table.

8. The method according to claim 6, characterized in that Coordinate load balancing between nodes, including: The task queue depth of each node is monitored through the load migration trigger condition. If the queue depth of any node exceeds the preset upper limit, the task migration process is started. The node is used as the source node, and the target node with the shortest response time is selected as the migration object in combination with the node response time statistics to obtain the load redistribution instruction; according to the load redistribution instruction, the processing result synchronization mechanism is used to coordinate the data processing progress between the nodes. If the source node completes the task migration, the global task status table is updated. If the target node successfully receives the task, a confirmation signal is sent, forming a closed-loop control mechanism for collaborative processing between nodes.

9. A scientific and technological information resource integration system for implementing the method according to any one of claims 1 to 8, characterized in that: The system comprises: The monitoring unit is used to obtain the network transmission rate and round-trip delay parameters of the source data stream, determine the bandwidth status through the network monitoring module, and generate a bandwidth status identifier It is also used to monitor the network stability of the data transmission unit during the transmission process of the data transmission unit to the forwarding unit, generate an adaptive flow control parameter configuration, and adjust the data transmission rate; a processing unit, configured to adjust the data compression ratio and fragmentation strategy according to the bandwidth status identifier, perform dynamic compression and preprocessing on the data stream, generate an optimized data transmission unit, and send the optimized data transmission unit to the forwarding unit; A forwarding unit is configured to receive and verify the integrity of the data transmission unit and construct a hierarchical priority queue; during the forwarding process, it also evaluates the resource usage status of each node in the forwarding unit, generates a node capacity weight distribution table, allocates tasks based on the hierarchical priority queue, coordinates load balancing between nodes, and forwards the verified scientific and technological intelligence data stream to the cloud server; The cloud server is used to receive the scientific and technological intelligence data stream.

10. A computer-readable storage medium having instructions stored thereon, characterized in that: When the instructions are executed by a processor, the method according to any one of claims 1 to 8 is implemented.

Images