Heterogeneous data processing optimization system based on intelligent edge computing

Abstract

The invention relates to the technical field of intelligent edge computing, and discloses a heterogeneous data processing optimization system based on intelligent edge computing. Comprising a heterogeneous data acquisition module, an edge preprocessing module, an intelligent optimization scheduling module, a heterogeneous fusion analysis module and a self-adaptive feedback control module. According to the system, data streams of different formats and protocols can be collected from a plurality of heterogeneous data sources in real time, and multi-modal correlation analysis is carried out after preprocessing and intelligent scheduling. Meanwhile, the system performance is optimized through self-adaptive feedback control, and a resource efficiency evaluation module and a safety compliance verification module are further arranged. According to the method, the problems of delay, accuracy, resource allocation, safety compliance and the like in heterogeneous data processing are effectively solved, the data processing efficiency and the system stability are improved, and the method is suitable for heterogeneous data processing scenes in the fields of industry, traffic, medical treatment and the like.

Description

Technical Field

[0001] The present invention relates to the technical field of intelligent edge computing, and particularly to a heterogeneous data processing optimization system based on intelligent edge computing. Background Art

[0002] With the rapid development of information technology, data has grown explosively, with a wide range of sources and diverse formats, covering structured, semi-structured, and unstructured data. These heterogeneous data widely exist in various fields such as industrial Internet of Things, intelligent transportation, and medical health, posing extremely high challenges to data processing capabilities.

[0003] In the field of industrial Internet of Things, a large number of different types of sensors, devices, and control systems are distributed in factories, and the data they generate has different data formats and communication protocols. Traditional data processing methods often transmit all data to a central server for processing, which leads to a huge pressure on network bandwidth and data processing delays. A large amount of data with high real-time requirements will experience delay phenomena during transmission and cannot be timely fed back to the production system, thus affecting production efficiency and product quality. For example, in an automobile manufacturing production line, sensors monitor the operating status of equipment and product production parameters in real time. If the data is not processed in a timely manner, equipment failures may not be detected and processed in a timely manner, resulting in production line downtime and huge economic losses.

[0004] In the field of intelligent transportation, traffic data is also heterogeneous. Cameras, geomagnetic sensors, in-vehicle devices, etc. on the road continuously generate various formats of data such as videos, images, and vehicle driving trajectories. These data need to be analyzed in real time to achieve traffic flow monitoring, congestion prediction, and intelligent traffic signal control. However, existing processing systems are difficult to efficiently fuse and analyze this heterogeneous data, resulting in a lack of accurate basis for traffic management decisions and making it difficult to effectively alleviate traffic congestion problems. For example, during the morning and evening rush hours in big cities, due to the inability to accurately predict traffic flow, the signal timing of traffic lights is unreasonable, further exacerbating road congestion.

[0005] In the field of medical health, there are various heterogeneous data in hospitals, such as electronic medical records of patients, imaging examination data, and physiological monitoring data. Electronic medical records record patient diagnosis and treatment information in structured and semi-structured forms; imaging examination data such as X-ray and CT images are unstructured data; physiological monitoring data such as heart rate and blood pressure are structured data generated in real time. Currently, these data are often stored and managed separately, making it difficult to conduct comprehensive analysis and utilization. Doctors cannot quickly obtain comprehensive and accurate patient information during the diagnosis and treatment process, affecting the accuracy of diagnosis and treatment effects. For example, in the scenario of telemedicine, due to data processing and transmission problems, experts cannot timely obtain the complete imaging data and physiological monitoring data of patients, resulting in the inability to give accurate diagnosis suggestions in a timely manner.

[0006] In addition, when facing large-scale heterogeneous data, traditional data processing systems have unreasonable resource allocation, and it is easy to have a situation where some nodes are overloaded while some nodes are idle, resulting in low overall system performance. Moreover, the security and privacy issues in the data processing process are becoming increasingly prominent. During the data acquisition, transmission, and fusion processes, sensitive information may be leaked, bringing potential risks to users. To sum up, the existing data processing technologies can no longer meet the needs of heterogeneous data processing, and there is an urgent need for an efficient and intelligent heterogeneous data processing optimization system. Summary of the Invention

[0007] The purpose of the present invention is to provide a heterogeneous data processing optimization system based on intelligent edge computing to solve the problems proposed in the above background technology.

[0008] To achieve the above object, the present invention provides the following technical solution: A heterogeneous data processing optimization system based on intelligent edge computing, the system includes a heterogeneous data acquisition module, an edge preprocessing module, an intelligent optimization scheduling module, a heterogeneous fusion analysis module, and an adaptive feedback control module;

[0009] The heterogeneous data acquisition module is used to collect data streams of different formats and protocols from multiple heterogeneous data sources in real time and transmit the data streams to the edge preprocessing module;

[0010] The edge preprocessing module performs format standardization, noise filtering, and preliminary feature extraction on the received data stream to generate preprocessing data packets;

[0011] The intelligent optimization scheduling module dynamically allocates computing resources according to the real-time load status and data processing priorities of edge nodes to schedule the preprocessing data packets to the target processing unit;

[0012] The heterogeneous fusion analysis module performs multi-modal correlation analysis on the scheduled data packets and generates a fusion analysis result through spatio-temporal feature matching and semantic parsing;

[0013] The adaptive feedback control module monitors the operating status of each module in real time. If a data acquisition delay, preprocessing abnormality, scheduling conflict, or fusion error signal is detected, a dynamic parameter adjustment instruction is triggered to optimize the system performance.

[0014] Preferably, the specific operation process of the heterogeneous data acquisition module is as follows:

[0015] In the data acquisition stage, identify the transmission protocols and data structures of different data sources, label unstructured data as the first type of data stream, semi-structured data as the second type of data stream, and structured data as the third type of data stream; allocate acquisition weights to each type of data stream according to the preset data priority rules. If the real-time transmission rate of a certain type of data stream is detected to be lower than the preset threshold, generate a data acquisition delay signal; after data classification is completed, encapsulate the data stream into an original data packet with a timestamp and a source identifier, and transmit it to the edge preprocessing module.

[0016] Preferably, the specific processing logic of the edge preprocessing module includes:

[0017] Perform protocol parsing on the received original data packet, extract the payload data and perform format unification processing; use the sliding window algorithm to segment and intercept the continuous data stream, calculate the signal-to-noise ratio and feature density of each data segment. If the signal-to-noise ratio is lower than the preset threshold or the feature density exceeds the allowable range, mark it as noise data and trigger a filtering operation; vectorize the features of the filtered data segments to generate a preprocessing data packet containing key feature indexes.

[0018] Preferably, the scheduling strategy of the intelligent optimization scheduling module is as follows:

[0019] Obtain the available computing resources, network bandwidth and task queue length of the edge node in real time, and allocate an initial scheduling level to each preprocessing data packet based on the dynamic priority algorithm; if it is detected that the task queue length of the target processing unit exceeds the capacity limit, perform secondary scheduling and allocation according to the timeliness weight and resource requirements of the data packet, and preferentially route high-timeliness data packets to low-load nodes; if there are still resource conflicts after secondary scheduling, generate a scheduling conflict signal and trigger the adaptive feedback control module.

[0020] Preferably, the multi-modal correlation analysis process of the heterogeneous fusion analysis module includes:

[0021] Perform spatio-temporal alignment on the preprocessing data packets from different data sources, and extract the periodic features in the time series and the topological relationships in the spatial distribution; use the semantic graph construction method to associate the context information of structured data and unstructured data. If semantic conflicts or logical breaks are detected, generate a fusion error signal; input the associated data into a multi-modal neural network for feature-level fusion, and output a fusion analysis result containing cross-domain correlation indicators.

[0022] Preferably, the specific steps of the semantic graph construction method are:

[0023] Extract entity attributes and relationship triples from structured data, and extract event subjects and action descriptions from unstructured data through natural language processing techniques; perform semantic similarity matching between the triples and the event descriptions. If the matching degree is lower than the preset threshold, mark it as a potential logical fault; construct a hierarchical semantic network based on the matching results, and optimize the graph structure by removing redundant nodes through a knowledge graph pruning algorithm.

[0024] Preferably, the dynamic parameter adjustment logic of the adaptive feedback control module includes:

[0025] When receiving a data acquisition delay signal, adjust the polling frequency of the data source and increase the bandwidth quota of high-priority data streams; when detecting a preprocessing exception signal, reset the noise filtering threshold and expand the size of the sliding window; if there is a scheduling conflict signal, reduce the resource allocation weight of low-timeliness tasks; if receiving a fusion error signal, enable a backup semantic parsing model and update the training parameters of the multi-modal neural network.

[0026] Preferably, the system further includes a resource efficiency evaluation module. The resource efficiency evaluation module collects the resource utilization rate, task processing success rate, and energy consumption data of each edge node, compares the data with a preset efficiency baseline. If the resource utilization rate continuously exceeds the baseline value or the task processing success rate is lower than the baseline value, generate a resource overload signal; send the resource overload signal to the intelligent optimization scheduling module to trigger a global resource reallocation strategy.

[0027] Preferably, the specific implementation method of the global resource reallocation strategy is:

[0028] Divide resource cooperation clusters according to the geographical location and network topology of edge nodes, and dynamically migrate some computing tasks within the clusters using a load balancing algorithm; if the overall load of the cluster still exceeds the capacity limit, request elastic resource expansion from the cloud and temporarily store non-real-time tasks in a distributed cache queue; after the task migration is completed, update the resource efficiency evaluation data of each node and recalculate the efficiency baseline.

[0029] Preferably, the system further includes a security compliance verification module. The security compliance verification module performs desensitization processing on sensitive information during the data acquisition stage, and detects the privacy compliance of cross-domain association results during the data fusion stage; if unauthorized data associations or potential privacy leakage risks are identified, generate a security violation signal and interrupt the relevant data processing flow, and at the same time trigger a data traceability mechanism to locate the violating operation node.

[0030] Compared with the prior art, the beneficial effects of the present invention are:

[0031] In the data collection phase, the heterogeneous data collection module can collect data streams in different formats and protocols from multiple heterogeneous data sources in real time. By identifying the transmission protocols and data structures of the data sources, marking different types of data streams, and allocating collection weights according to preset rules, it can ensure that high-priority data is collected first, effectively improving the collection efficiency. When detecting data collection delays, it can generate signals in a timely manner, and the system adjusts the data source polling frequency and bandwidth quota accordingly to reduce delay phenomena and ensure the real-time and integrity of data. This is particularly crucial in industrial control scenarios. For example, in the chemical production process, sensor data reflects the production status in real time, and timely and accurate collection can avoid production accidents caused by data delays.

[0032] The edge preprocessing module standardizes the format, filters noise, and extracts preliminary features from the collected data. The format unification process enables subsequent modules to process data more efficiently and avoid compatibility issues caused by format differences. The sliding window algorithm combines signal-to-noise ratio and feature density analysis to accurately filter out noise data, improve data quality, and provide a reliable data basis for subsequent analysis. For example, in image data processing, after effectively removing noise, image feature extraction is more accurate, which is beneficial to subsequent image recognition and analysis.

[0033] The intelligent optimization scheduling module dynamically allocates computing resources according to the real-time load status of edge nodes and data processing priorities. By obtaining the available computing resources, network bandwidth, and task queue length of edge nodes in real time, it uses a dynamic priority algorithm to assign an initial scheduling level to data packets, and then performs secondary scheduling and allocation according to the actual situation to give priority to ensuring the processing of high-timeliness data packets. This intelligent scheduling strategy effectively avoids resource conflicts, improves task processing efficiency, and reduces processing delays. In cloud computing and edge computing scenarios, a large number of tasks with different priorities can be reasonably allocated to improve the response speed of the entire system.

[0034] The heterogeneous fusion analysis module performs multi-modal correlation analysis on the scheduled data packets through spatio-temporal feature matching and semantic parsing. Spatio-temporal alignment extracts periodic features and topological relationships, providing information in the time and space dimensions for data analysis; the semantic graph construction method associates the context information of structured and unstructured data, effectively avoiding semantic conflicts and logical breaks, and improving the accuracy of fusion analysis results. For example, in an intelligent transportation system, by combining vehicle driving trajectory data and road monitoring video data, it can more accurately predict traffic congestion and provide a scientific basis for traffic management.

[0035] The adaptive feedback control module monitors the operating status of each module in real time. Once abnormal signals are detected, such as data acquisition delay, preprocessing abnormality, scheduling conflict, or fusion error signal, it immediately triggers a dynamic parameter adjustment instruction. This self-regulating mechanism enables the system to optimize its performance in real time according to the actual operating conditions, improving the stability and reliability of the system. For example, when there is a data acquisition delay, it adjusts the data source polling frequency and bandwidth quota; when there is a preprocessing abnormality, it resets the noise filtering threshold and expands the sliding window size to ensure that the system is always in the best operating state.

[0036] The combination of the resource efficiency evaluation module and the global resource reallocation strategy further optimizes the system resource utilization. By collecting the resource utilization rate, task processing success rate, and energy consumption data of edge nodes, comparing them with the preset efficiency baseline, it can timely detect resource overloading situations and trigger global resource reallocation. This not only improves the resource utilization rate but also ensures the task processing success rate and reduces the energy consumption. For example, in large-scale data centers, through reasonable resource allocation, it reduces energy waste and improves the overall operation efficiency.

[0037] The security and compliance verification module performs desensitization processing on sensitive information during the data acquisition stage and detects privacy compliance during the data fusion stage. Once unauthorized data association or potential privacy leakage risks are found, it immediately interrupts the processing flow and triggers a data traceability mechanism, effectively ensuring the security and compliance of the data. In fields with extremely high requirements for data security and privacy, such as financial data processing and medical data management, this function can protect users' sensitive information from being leaked and safeguard users' rights and interests. Description of the Drawings

[0038] Figure 1 It is the working principle diagram of the heterogeneous data processing optimization system based on intelligent edge computing described in the present invention;

[0039] Figure 2 It is the working principle diagram of data preprocessing of the edge preprocessing module;

[0040] Figure 3 It is the schematic diagram of the task scheduling strategy of the intelligent optimization scheduling module;

[0041] Figure 4 It is the working principle diagram of multi-modal association analysis of the heterogeneous fusion analysis module. Detailed Embodiments

[0042] Next, the technical solutions in the embodiments of the present invention will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

[0043] Please refer to Figures 1-4 , the present invention provides a heterogeneous data processing optimization system based on intelligent edge computing, and its overall implementation solution is as follows:

[0044] The heterogeneous data acquisition module is responsible for real-time acquisition of data streams in different formats and protocols from multiple heterogeneous data sources. During acquisition, it identifies the transmission protocol and data structure of the data source, marks unstructured data as the first type of data stream, semi-structured data as the second type of data stream, and structured data as the third type of data stream. Then, according to the preset data priority rules, acquisition weights are assigned to each type of data stream. If the real-time transmission rate of a certain type of data stream is lower than the preset threshold, a data acquisition delay signal is generated. After data classification, the data stream is encapsulated into a raw data packet with a timestamp and a source identifier and transmitted to the edge preprocessing module.

[0045] The edge preprocessing module processes the received raw data packet. First, it performs protocol parsing, extracts the payload data and unifies the format. Then, the sliding window algorithm is used to segment the continuous data stream, and the signal-to-noise ratio and feature density of each segment are calculated. If the signal-to-noise ratio is lower than the preset threshold or the feature density exceeds the allowable range, it is marked as noise data and filtered. Finally, the filtered data segments are feature vectorized to generate a preprocessing data packet containing key feature indexes.

[0046] The intelligent optimization scheduling module schedules the preprocessing data packet according to the real-time load status and data processing priority of the edge node. It real-time obtains the available computing resources, network bandwidth and task queue length of the edge node, and uses the dynamic priority algorithm to assign an initial scheduling level to each preprocessing data packet. If the task queue length of the target processing unit exceeds the capacity limit, secondary scheduling allocation is performed according to the timeliness weight and resource requirements of the data packet, and high-timeliness data packets are preferentially routed to low-load nodes. If there are still resource conflicts after secondary scheduling, a scheduling conflict signal is generated and the adaptive feedback control module is triggered.

[0047] The heterogeneous fusion analysis module performs multi-modal correlation analysis on the scheduled data packet. First, the preprocessing data packets from different data sources are aligned in time and space, and the periodic features in the time series and the topological relationship in the spatial distribution are extracted. Then, the semantic graph construction method is used to associate the context information of structured data and unstructured data. If semantic conflicts or logical breaks are detected, a fusion error signal is generated. Finally, the associated data is input into a multi-modal neural network for feature-level fusion, and the fusion analysis result containing cross-domain correlation indexes is output.

[0048] The adaptive feedback control module monitors the operating status of each module in real time. Once a data acquisition delay, preprocessing anomaly, scheduling conflict, or fusion error signal is detected, a dynamic parameter adjustment instruction is triggered to optimize the system performance. For example, when a data acquisition delay signal is received, the polling frequency of the data source is adjusted and the bandwidth quota of high-priority data streams is increased; when a preprocessing anomaly signal is detected, the noise filtering threshold is reset and the sliding window size is expanded; when there is a scheduling conflict signal, the resource allocation weight of low-timeliness tasks is reduced; when a fusion error signal is received, a backup semantic parsing model is enabled and the training parameters of the multi-modal neural network are updated.

[0049] The following further describes the implementation of the present invention in conjunction with Embodiments 1 to 6.

[0050] Embodiment 1:

[0051] During the actual operation of the heterogeneous data acquisition module, in the face of numerous different types of data sources, such as sensor networks, databases, file systems, etc., it is necessary to accurately identify their transmission protocols and data structures. For sensor networks, the MQTT protocol may be used to transmit data, and the data structure is relatively simple but may contain a large number of real-time changing values; while the data in the database follows the specifications of structured query languages such as SQL, and the structure is rigorous. During the identification process, specialized protocol parsing tools and data structure identification algorithms are used to quickly determine the category of the data.

[0052] After marking the category of the data stream, the acquisition weight is allocated according to the preset data priority rules. For example, for sensor data with high real-time requirements, a higher acquisition weight is set to ensure priority acquisition. If during the acquisition process, the real-time transmission rate of a certain type of data stream is lower than the preset threshold, for example, when the transmission rate of a certain temperature sensor data stream is lower than the threshold of 10 data points per second, a data acquisition delay signal is immediately generated. At this time, the system will record the delay time and relevant data stream information for subsequent analysis and processing.

[0053] After completing data classification, the data stream is encapsulated into raw data packets with timestamps and source identifiers. The timestamp is accurate to the millisecond level to ensure the accurate time sequence of the data; the source identifier includes the device number, location information, etc. of the data source, which is convenient for tracing the data source. These raw data packets are transmitted to the edge preprocessing module in an orderly manner.

[0054] After receiving the raw data packet, the edge preprocessing module first performs protocol parsing. Taking the data packet transmitted by the HTTP protocol as an example, by parsing the HTTP header information, the effective payload part of the data is accurately extracted, and format normalization processing is performed according to the data type to convert it into a format that the system can uniformly process.

[0055] Next, the sliding window algorithm is used to segment and intercept the continuous data stream. The size of the sliding window is set according to the data characteristics and processing requirements. Assume the window size is 100 data points, and it slides 10 data points each time. In each data segment, the signal-to-noise ratio and feature density are calculated. The formula for calculating the signal-to-noise ratio is: where s i represents the i-th true data point, represents the estimated value of the i-th data point, and n is the number of data points in the data segment. The feature density is calculated based on the number of data features and the number of data points. For example, if there are 5 features in a data segment with a total of 100 data points, the feature density is 5÷100 = 0.05. If the signal-to-noise ratio is lower than the preset threshold (such as 10 dB) or the feature density exceeds the allowable range (assuming the allowable range is 0.01 - 0.1), it is marked as noise data and the filtering operation is triggered.

[0056] Finally, the filtered data segments are subjected to feature vectorization processing. For example, for data such as temperature and humidity, it is converted into vector form, such as [temperature value, humidity value], and a preprocessing data packet containing the key feature index is generated to facilitate the subsequent module to quickly locate and process the key data.

[0057] Example 2:

[0058] The intelligent optimization scheduling module plays a key role in resource allocation and task scheduling in the system. In the actual operating environment, the status of the edge nodes is constantly changing, and their available computing resources, network bandwidth, and task queue length may change at any time.

[0059] This module obtains these key information by interacting with the monitoring system of the edge nodes in real time. For example, it obtains the current CPU usage rate and memory occupancy of the edge node every 10 seconds as the available computing resource metrics, and at the same time obtains the real-time bandwidth occupancy and remaining bandwidth of the network interface as the network bandwidth information, as well as the number of pending tasks in the task queue as the task queue length.

[0060] Based on the dynamic priority algorithm, an initial scheduling level is assigned to each preprocessing data packet. The dynamic priority algorithm comprehensively considers factors such as the data type, data volume, and estimated processing time of the data packet. Assume that for data packets of the emergency alarm data type, the priority weight is set to 0.8; for data packets of the ordinary monitoring data type, the priority weight is set to 0.3. The larger the data volume and the longer the estimated processing time, the lower the priority. The calculation formula is: where the data type weight is preset according to the importance of the data type, and the maximum data volume and the maximum estimated processing time are obtained from the statistical analysis of the system historical data.

[0061] When it is detected that the task queue length of the target processing unit exceeds the capacity limit, secondary scheduling and allocation are performed. Adjustments are made according to the timeliness weight and resource requirements of the data packets. For example, for video surveillance data with high real-time requirements, the timeliness weight is set to 0.9 and it is preferentially routed to low-load nodes. When selecting low-load nodes, the remaining computing resources and network bandwidth of the nodes are comprehensively considered. Suppose the remaining CPU usage rate of node A is 60%, the remaining memory is 512MB, and the remaining network bandwidth is 10Mbps; the remaining CPU usage rate of node B is 40%, the remaining memory is 256MB, and the remaining network bandwidth is 20Mbps. For video processing tasks that require a large amount of computing resources, node A is preferentially selected.

[0062] If resource conflicts still exist after secondary scheduling, for example, multiple high-timeliness data packets compete for the resources of the same low-load node at the same time, a scheduling conflict signal is generated at this time and the adaptive feedback control module is triggered so that the system can optimize and adjust the overall performance.

[0063] Example 3:

[0064] The multi-modal association analysis process of the heterogeneous fusion analysis module involves multiple complex steps, and its specific implementation is introduced in detail in this example.

[0065] When performing spatio-temporal alignment on the preprocessed data packets from different data sources, taking traffic data as an example, which includes vehicle flow data collected by road sensors and video data captured by cameras. For vehicle flow data, the timestamp is accurate to the second, recording the time when the vehicle passes; for video data, the time corresponding to each frame is calculated through the video frame rate and the start time. Spatially, they are associated according to the geographical location information of the sensors and cameras. For example, it is determined which sections covered by a certain camera coincide with the sections monitored by which sensors. In this way, periodic features in the time series are extracted, such as the regular changes in vehicle flow during peak morning and evening hours; topological relationships in the spatial distribution are extracted, such as the connection relationships between roads and the driving directions of vehicles.

[0066] The semantic graph construction method is used to associate the context information of structured data and unstructured data. Entity attribute and relationship triples are extracted from the structured traffic database, such as (vehicle, driving on, Road A). Event subjects and action descriptions are extracted from the unstructured video description text through natural language processing techniques, such as "the vehicle turns at the intersection". The triples are matched with the event descriptions, and the semantic similarity calculation formula is: where, w i and w j are the weights of the words in the structured data and unstructured data respectively, sim(word i ,word j) is the similarity between two words, where m and n are the numbers of words in structured data and unstructured data respectively. If the matching degree is lower than a preset threshold (such as 0.6), it is marked as a potential logical fault.

[0067] Construct a hierarchical semantic network based on the matching results, with nodes such as vehicles, roads, and traffic events, and their relationships as edges. Remove redundant nodes through a knowledge graph pruning algorithm, such as some vehicle information nodes with repeated descriptions, to optimize the graph structure.

[0068] Input the associated data into a multi-modal neural network for feature-level fusion. The multi-modal neural network contains multiple input layers, which respectively receive data of different modalities such as structured data and unstructured data. Through complex calculations in the hidden layer, the data features of different modalities are fused, and a fusion analysis result containing cross-domain association indicators is output, such as vehicle driving path prediction, traffic congestion probability, etc.

[0069] Example 4:

[0070] The adaptive feedback control module is an important part to ensure the stable operation and performance optimization of the system. The specific implementation of its dynamic parameter adjustment logic is introduced in detail below.

[0071] When receiving a data acquisition delay signal, the system will respond quickly. For example, in an industrial production environment, if the sensor data acquisition of a certain production line is delayed, first adjust the polling frequency of the data source. Suppose the original polling frequency is once per second, and now it is increased to twice per second to increase the access frequency to the data source and ensure timely data acquisition. At the same time, increase the bandwidth quota of high-priority data streams. For the sensor data of key equipment on the production line, increase its bandwidth quota from the original 1Mbps to 2Mbps to ensure the timeliness of data transmission.

[0072] When detecting a preprocessing exception signal, make corresponding adjustments. Taking image data preprocessing as an example, if there is too much noise resulting in preprocessing exceptions, reset the noise filtering threshold. Suppose the original noise filtering threshold is a signal-to-noise ratio of 15dB, and now it is reduced to 10dB to filter more noise. At the same time, expand the size of the sliding window from the original 50 pixel points to 80 pixel points to make data processing more comprehensive and avoid missing key information.

[0073] If there is a scheduling conflict signal, for example, in a cloud computing edge node environment, multiple tasks compete for resources resulting in a scheduling conflict. At this time, reduce the resource allocation weight of low-timeliness tasks. Suppose the original resource allocation weight of low-timeliness tasks is 0.4, and it is reduced to 0.2 to give priority to ensuring the resource requirements of high-timeliness tasks and improve the overall performance of the system.

[0074] If a fused error signal is received, in an intelligent security system, when an error occurs in the fusion of video and image data, a backup semantic parsing model is enabled. Assume that the system originally uses a rule-based semantic parsing model and switches to a backup deep learning-based semantic parsing model at this time. At the same time, the training parameters of the multi-modal neural network are updated, and the weights and biases of the neural network are adjusted according to the latest fused error data to improve the accuracy of fusion analysis.

[0075] Example 5:

[0076] The resource efficiency evaluation module continuously collects relevant data of each edge node. Taking a system composed of multiple industrial site edge nodes as an example, these nodes are responsible for collecting and preliminarily processing various types of data on the production line. At fixed time intervals, such as every 15 minutes, the resource efficiency evaluation module collects the resource utilization rate, task processing success rate, and energy consumption data of each edge node. For the resource utilization rate, the CPU utilization rate and memory utilization rate are mainly monitored. The proportion of the current CPU busy level in the total processing capacity is obtained through specific system monitoring tools, as well as the proportion of the used memory space in the total memory space. The task processing success rate is determined by calculating the ratio of the number of successfully completed tasks to the total number of tasks within a certain period of time. The energy consumption data is collected by means of specialized energy consumption monitoring equipment to accurately measure the power consumption of each edge node per unit time.

[0077] The collected data is compared and analyzed with the preset efficiency baseline. The preset efficiency baseline is set according to the historical data of the long-term operation of the system and the expected performance goals. For example, through a large amount of historical data statistics and system performance analysis, the baseline value of the CPU utilization rate is determined to be 75%, the baseline value of the memory utilization rate is 80%, and the baseline value of the task processing success rate is 90%. When the resource efficiency evaluation module finds that the resource utilization rate of a certain edge node continuously exceeds the baseline value, such as the CPU utilization rate exceeds 75% in three consecutive collections, or the task processing success rate is lower than the baseline value, such as the task processing success rate of a certain node is continuously lower than 90%, a resource overload signal will be generated.

[0078] Once the resource overload signal is sent to the intelligent optimization scheduling module, the global resource reallocation strategy will be launched. First, resource collaboration clusters are divided according to the geographical location and network topology of the edge nodes. In an industrial network environment covering multiple factory areas, based on the geographical locations of different factory areas and the network connection relationships between edge nodes, edge nodes with close geographical locations, stable network connections, and complementary performance are divided into a cluster. For example, edge nodes located within the same factory area with low network latency are grouped into a cluster to facilitate efficient data interaction and task collaboration between them.

[0079] Inside the cluster, a load balancing algorithm is adopted to dynamically migrate some computing tasks. Suppose a dynamic load balancing algorithm based on task priority and node load is adopted. For high-priority tasks, they are preferentially assigned to nodes with relatively low load for processing. When calculating the node load, the CPU utilization rate, memory utilization rate, and network bandwidth occupancy are comprehensively considered. For example, if the CPU utilization rate of node A is 60%, the memory utilization rate is 70%, and the network bandwidth occupancy is 30%, and the CPU utilization rate of node B is 80%, the memory utilization rate is 85%, and the network bandwidth occupancy is 50%, then when allocating tasks, tasks will be preferentially assigned to node A.

[0080] If the overall load of the cluster still exceeds the capacity limit, the system will request elastic resource expansion from the cloud. For example, when the overall load of a certain cluster continuously exceeds the set capacity threshold, such as when the average CPU utilization rate of all nodes in the cluster is detected to exceed 90% for 5 consecutive times, the system will automatically send a resource expansion request to the cloud service provider to apply for additional computing resources and storage resources. At the same time, in order to relieve the pressure on the current cluster, non-real-time tasks will be temporarily stored in the distributed cache queue. For example, for some regularly generated production report tasks, since they have low requirements for real-time performance, they can be temporarily stored in the distributed cache queue and processed after the resource pressure on the cluster is relieved.

[0081] After the task migration is completed, it is necessary to update the resource performance evaluation data of each node in a timely manner. Re-collect the resource utilization rate, task processing success rate, and energy consumption data of each node, and recalculate the performance baseline based on the new data. This can enable the system to more accurately reflect the current operating state, provide a more reasonable basis for subsequent resource allocation and task scheduling, and ensure that the system always maintains a stable and efficient operating state.

[0082] Example 6:

[0083] In the data collection stage, the security and compliance verification module will perform strict desensitization processing on possible sensitive information. Taking the intelligent transportation system collecting vehicle and driver-related data as an example, it involves sensitive information such as the driver's ID number and mobile phone number. For the ID number, an encrypted hash function is used for processing, converting the original ID number into a hash value of a fixed length, which cannot restore the original ID number, thus protecting the driver's identity information. For the mobile phone number, a partial hiding method is adopted, only showing the first three digits and the last four digits of the mobile phone number, and replacing the middle part with asterisks, which not only does not affect the partial use of the data but also effectively protects privacy. In the financial data collection scenario, the same partial hiding desensitization strategy is adopted for bank card numbers, only showing the first 6 digits and the last 4 digits of the card number, effectively preventing the leakage of bank card number information.

[0084] In the data fusion stage, the security compliance verification module will conduct a comprehensive privacy compliance detection on the cross-domain association results. For example, in a system that combines the user purchase data of an e-commerce platform and the delivery data of a logistics platform, when conducting an association analysis of user purchase behavior and commodity delivery information, the security compliance verification module will carefully check the legality of data association according to relevant privacy regulations and user authorization agreements. If unauthorized data association is found, such as the e-commerce platform sharing users' purchase records with the logistics platform for precise marketing purposes without obtaining explicit user authorization, the security compliance verification module will immediately generate a security violation signal and quickly interrupt the relevant data processing process to prevent further expansion of privacy leakage.

[0085] Meanwhile, the security compliance verification module will trigger a data traceability mechanism to accurately locate the nodes of illegal operations. Using the data flow and operation logs recorded in the system, combined with the immutable and traceable characteristics of blockchain technology, it tracks each link from data collection, transmission, storage to fusion processing. Assuming a violation is found during data fusion, through information such as timestamps and operation subjects recorded on the blockchain, it is possible to quickly locate which edge node performed the illegal operation during data fusion, and obtain detailed information such as the device number, operation time, and data content involved in that node. This not only enables timely discovery and prevention of illegal acts, but also provides strong evidence for subsequent accountability and system improvement, ensuring that the data processing process of the entire system always complies with security compliance requirements and protecting users' privacy and data security.

[0086] It should be noted that in this article, relational terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variant thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements not only includes those elements, but also includes other elements not explicitly listed, or further includes elements inherent to such process, method, article or device.

[0087] Although the embodiments of the present invention have been shown and described, for those of ordinary skill in the art, it can be understood that various changes, modifications, substitutions and variations can be made to these embodiments without departing from the principles and spirit of the present invention. The scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. An optimization system for heterogeneous data processing based on intelligent edge computing, characterized in that It includes a heterogeneous data acquisition module, an edge preprocessing module, an intelligent optimization scheduling module, a heterogeneous fusion analysis module, and an adaptive feedback control module; The heterogeneous data acquisition module is used to collect data streams in different formats and protocols from multiple heterogeneous data sources in real time, and transmit the data streams to the edge preprocessing module; The edge preprocessing module performs format standardization, noise filtering, and preliminary feature extraction on the received data stream to generate a preprocessed data packet; The intelligent optimization scheduling module dynamically allocates computing resources according to the real-time load status and data processing priority of the edge node to schedule the preprocessed data packet to the target processing unit; The heterogeneous fusion analysis module performs multi-modal correlation analysis on the scheduled data packet, and generates a fusion analysis result through spatio-temporal feature matching and semantic parsing; The adaptive feedback control module monitors the operating status of each module in real time. If a data acquisition delay, preprocessing abnormality, scheduling conflict, or fusion error signal is detected, it triggers a dynamic parameter adjustment instruction to optimize the system performance.

2. The heterogeneous data processing optimization system based on intelligent edge computing according to claim 1, characterized in that, The specific operation process of the heterogeneous data acquisition module is as follows: In the data acquisition stage, identify the transmission protocols and data structures of different data sources, mark unstructured data as the first type of data stream, mark semi-structured data as the second type of data stream, and mark structured data as the third type of data stream; Allocate acquisition weights to each type of data stream according to the preset data priority rules. If the real-time transmission rate of a certain type of data stream is detected to be lower than the preset threshold, a data acquisition delay signal is generated; after data classification is completed, encapsulate the data stream into an original data packet with a timestamp and a source identifier, and transmit it to the edge preprocessing module.

3. An optimized heterogeneous data processing system based on intelligent edge computing according to claim 1, characterized in that, The specific processing logic of the edge preprocessing module includes: Perform protocol parsing on the received original data packet, extract the payload data and perform format unification processing; use a sliding window algorithm to segment and intercept the continuous data stream, calculate the signal-to-noise ratio and feature density of each data segment. If the signal-to-noise ratio is lower than the preset threshold or the feature density exceeds the allowable range, mark it as noise data and trigger a filtering operation; vectorize the filtered data segments to generate a preprocessed data packet containing key feature indexes.

4. An optimized heterogeneous data processing system based on intelligent edge computing according to claim 1, characterized in that, The scheduling strategy of the intelligent optimization scheduling module is as follows: Obtain the available computing resources, network bandwidth, and task queue length of the edge node in real time, and allocate an initial scheduling level to each preprocessed data packet based on the dynamic priority algorithm; if it is detected that the task queue length of the target processing unit exceeds the capacity limit, perform secondary scheduling allocation according to the timeliness weight and resource requirements of the data packet, and preferentially route high-timeliness data packets to low-load nodes; If there is still a resource conflict after secondary scheduling, generate a scheduling conflict signal and trigger the adaptive feedback control module.

5. An optimized heterogeneous data processing system based on intelligent edge computing according to claim 1, characterized in that, The multi-modal correlation analysis process of the heterogeneous fusion analysis module includes: Perform spatio-temporal alignment on preprocessed data packets from different data sources, and extract periodic features in the time series and topological relationships in the spatial distribution; use the semantic graph construction method to associate the context information of structured data and unstructured data. If semantic conflicts or logical breaks are detected, generate a fusion error signal; input the associated data into a multi-modal neural network for feature-level fusion, and output a fusion analysis result containing cross-domain association metrics.

6. The heterogeneous data processing optimization system based on intelligent edge computing according to claim 5, wherein, The specific steps of the semantic graph construction method are as follows: Extract entity attribute and relationship triples from structured data, and extract event subjects and action descriptions from unstructured data through natural language processing techniques; perform semantic similarity matching between the triples and the event descriptions. If the matching degree is lower than the preset threshold, mark it as a potential logical break. Construct a hierarchical semantic network based on the matching results, and optimize the graph structure by removing redundant nodes through a knowledge graph pruning algorithm.

7. An optimization system for heterogeneous data processing based on intelligent edge computing according to claim 1, characterized in that, The dynamic parameter adjustment logic of the adaptive feedback control module includes: When receiving a data acquisition delay signal, adjust the polling frequency of the data source and increase the bandwidth quota of high-priority data streams; when detecting a preprocessing anomaly signal, reset the noise filtering threshold and expand the size of the sliding window; if there is a scheduling conflict signal, reduce the resource allocation weight of low-timeliness tasks; if receiving a fusion error signal, enable a backup semantic parsing model and update the training parameters of the multi-modal neural network.

8. A heterogeneous data processing optimization system based on intelligent edge computing according to claim 1, characterized in that, It also includes a resource efficiency evaluation module. The resource efficiency evaluation module collects the resource utilization rate, task processing success rate, and energy consumption data of each edge node, compares the data with a preset efficiency baseline. If the resource utilization rate continuously exceeds the baseline value or the task processing success rate is lower than the baseline value, generate a resource overload signal; send the resource overload signal to the intelligent optimization scheduling module to trigger a global resource reallocation strategy.

9. The heterogeneous data processing optimization system based on intelligent edge computing according to claim 8, wherein The specific implementation method of the global resource reallocation strategy is: Divide resource cooperation clusters according to the geographical location and network topology of edge nodes, and dynamically migrate some computing tasks within the clusters using a load balancing algorithm; if the overall load of the cluster still exceeds the capacity limit, request elastic resource expansion from the cloud, and temporarily store non-real-time tasks in a distributed cache queue; after the task migration is completed, update the resource efficiency evaluation data of each node and recalculate the efficiency baseline.

10. An optimized heterogeneous data processing system based on intelligent edge computing according to claim 1, characterized in that It also includes a security and compliance verification module. The security and compliance verification module performs desensitization processing on sensitive information during the data acquisition stage, and detects the privacy compliance of cross-domain association results during the data fusion stage; if unauthorized data associations or potential privacy leakage risks are identified, generate a security violation signal and interrupt the relevant data processing process, and at the same time trigger a data tracing mechanism to locate the nodes of the violation operation.

Images