Multi-source heterogeneous data edge processing method and device for field monitoring and medium

Abstract

The application relates to the technical field of data processing, and discloses a multi-source heterogeneous data edge processing method and device for field monitoring and a medium, which comprises the following steps: based on a data access link, parallelly receiving original data generated by different monitoring devices in a field monitoring site; extracting multi-dimensional features to determine the data source types corresponding to the original data; calling corresponding analysis engines according to the data source types to generate analysis results through differential analysis; determining the time field priority according to the data source types, selecting target time fields from the analysis results, performing unified event time axis mapping, and generating standardized data; writing the standardized data into a classified cache queue, determining the time effectiveness level according to the time difference between the standard event time and the current time, and performing hierarchical sending scheduling based on the time effectiveness level and the network state. The edge access capability of multi-source heterogeneous data in the field ecological monitoring scene, the cross-source time correlation capability and the key data sending capability under the weak network condition are improved.

Description

Technical Field

[0001] This invention relates to the fields of ecological environment monitoring, embedded edge computing, and heterogeneous data acquisition and processing, and in particular to a method, device, and medium for edge processing of multi-source heterogeneous data for field monitoring. Background Technology

[0002] Field monitoring stations typically require long-term, continuous, and unattended collection of various ecological and environmental data, including meteorological data, flux data, hydrological data, soil data, and camera image / video monitoring data. Because different devices come from different manufacturers, their communication interfaces, protocol formats, field organization methods, and time stamping methods vary significantly, resulting in a highly heterogeneous and multi-source nature at the station front end. In actual deployments, common data sources include: devices outputting time-series monitoring data via proprietary data communication protocols; devices outputting manufacturer-defined serial port message data via serial interfaces; devices generating or transmitting image files, log files, or text files via file transfer; and devices providing video monitoring data via video signaling or media streaming.

[0003] In existing technologies, the processing of the aforementioned multi-source heterogeneous data typically employs a discrete approach, where dedicated acquisition hosts, protocol converters, or independent software are configured for different devices, and then the data is aggregated by a backend system. This approach suffers from the following problems: First, the large number of front-end devices, system fragmentation, and high average power consumption make it unsuitable for long-term operation under solar or battery power conditions. Second, the lack of a unified access and processing mechanism between different protocol links leads to complex fault location and high on-site maintenance costs. Third, most existing industrial gateways are oriented towards standard industrial protocols, typically relying on manual pre-specification of protocol types and parsing rules, making them ill-suited for adapting to the diverse heterogeneous data present in field monitoring scenarios. Fourth, for serial port data, most existing solutions require manual definition of fixed templates or field rules, resulting in poor adaptability to application scenarios with inconsistent message formats and frequent device model changes. Fifth, existing data aggregation is usually performed in the cloud or backend, lacking a unified event timeline mapping capability at the edge, making it difficult to correlate data from different sources under the same time reference. Sixth, most existing caching and forwarding methods only perform queue caching or breakpoint resumption based on arrival order, making it difficult to prioritize the transmission of critical monitoring data based on data timeliness and network status.

[0004] Therefore, there is an urgent need for a method, device, and medium for edge processing of multi-source heterogeneous data in field monitoring. Summary of the Invention

[0005] This invention provides a method, apparatus, and medium for edge processing of multi-source heterogeneous data for field monitoring, in order to solve the aforementioned problems existing in the prior art.

[0006] To achieve the above objectives, the present invention provides the following technical solution: A method for edge processing of multi-source heterogeneous data for field monitoring includes: S1: Establish multiple independent data access links and receive raw data generated by different monitoring devices at the field monitoring site in parallel based on the data access links; S2: Extract multi-dimensional features from the original data and determine the data source type corresponding to the original data based on the multi-dimensional features; S3: Call the corresponding parsing engine according to the data source type, and use the parsing engine to perform differential parsing on the original data to generate parsing results; S4: Determine the priority of the time field based on the data source type, select the target time field from the parsing results according to the time field priority, perform unified event timeline mapping based on the target time field, and generate standardized data containing data source type identifier and standard event time; S5: Write standardized data into a categorized cache queue, determine the timeliness level based on the time difference between the standard event time and the current time, and perform hierarchical sending scheduling based on the timeliness level and network status.

[0007] Furthermore, step S1 includes: S11: Establish data communication protocol data access links, serial port data access links, file transfer data access links, and video data access links, wherein the video data access links include signaling access links and media stream access links; S12: Allocate independent buffers and task queues for the data access links of the data communication protocol, serial port, file transfer, and video data access links respectively, and receive raw data generated by different monitoring devices at the field monitoring site in parallel through the independent buffers and task queues.

[0008] Furthermore, step S2 includes: S21: Extract link port features, message frame structure features, session interaction features, and data arrival time sequence features from the raw data. The link port features and message frame structure features constitute the link features, while the session interaction features and data arrival time sequence features constitute the data features. S22: Based on link port characteristics, message frame structure characteristics, session interaction characteristics, and data arrival time sequence characteristics, calculate the identification confidence of multiple candidate data source types respectively, and determine the data source type corresponding to the original data according to the identification confidence. S23: Use the data source type as the basis for controlling the selection of parsing path and time conversion strategy.

[0009] Furthermore, step S22, which involves determining the data source type based on the identification confidence level, includes: When the maximum value in the confidence score is greater than the preset confirmation threshold, the candidate data source type corresponding to the maximum value is determined as the data source type; If the maximum value does not reach the preset confirmation threshold, continue to collect subsequent features and return to step S22 to recalculate; When the confidence difference between two or more candidate data source types is less than the preset ambiguity threshold, the corresponding data access link is marked as pending judgment. After collecting subsequent message features, session information or arrival period features, the process returns to step S22 for re-judgment.

[0010] Furthermore, the differential parsing of serial port data in step S3 includes: S31: Establish a serial port message template fingerprint database, and match the received serial port messages with multiple templates in the template fingerprint database to form a candidate template set; S32: Score candidate templates in the candidate template set based on field position stability, numerical type consistency, time field resolvability, and verification pass rate. Select the candidate template with the highest score that is higher than the preset threshold as the target template. Complete field extraction based on the target template and generate serial port data parsing results.

[0011] Furthermore, the method for establishing a serial port message template fingerprint database in S31 includes: Obtain a historical serial port message dataset, and preprocess the historical serial port message dataset to obtain several initial message clusters; Select any initial message cluster as the target initial message cluster, and obtain all individual messages within the target initial message cluster; construct a single message byte position axis for each individual message, map each individual message to a position node sequence according to the byte transmission order, and determine the single message structured fingerprint element of each individual message based on the position node sequence; for each individual message within the target initial message cluster, calculate the multi-dimensional stability corresponding to each individual message; determine the single message confidence of each individual message based on the multi-dimensional stability corresponding to each individual message; obtain a preset confidence threshold and a multi-dimensional stability threshold, and filter messages in the target initial message cluster whose built-in confidence is higher than the preset confidence threshold and whose multi-dimensional stability is not lower than the corresponding multi-dimensional threshold, thus agglomerating to obtain a candidate template cluster; Obtain the start and end positions of semantically identical fields in all messages within the candidate template cluster, construct a unified field position distribution axis, map the start and end positions of all semantically identical fields onto the unified field position distribution axis, and generate a field position distribution sequence; use the field boundary position samples in the field position distribution sequence as statistical objects, classify them based on preset conditional classification rules, and obtain the first stable sample and the second stable sample; Based on the first and second stable samples, the standard field structure definition of the serial port message template corresponding to the candidate template cluster is determined. Simultaneously, the fast matching fingerprint element, field attribute label, four-dimensional scoring benchmark parameter and confidence label are encapsulated to obtain the template core fingerprint of the serial port message template corresponding to the candidate template cluster. The candidate template cluster aggregation and template core fingerprint generation steps are repeated for all initial message clusters to obtain the template core fingerprints corresponding to all serial port message templates. The template core fingerprint library is then constructed by summarizing the results. Obtain all template core fingerprints, combine them in pairs to generate template pairs, calculate the fast matching fingerprint meta-similarity of each template pair, filter template pairs with similarity higher than a preset conflict threshold to obtain potential conflict template pairs, build a template conflict resolution rule base based on the potential conflict template pairs, and call it in conjunction with the template core fingerprint base. By integrating the core template fingerprint library and the conflict resolution rule library, a serial port message template fingerprint library is obtained.

[0012] Furthermore, step S32 is followed by: S33: When a preset number of consecutive serial port messages fail to match the target template that meets the threshold condition, a new template candidate is generated based on the position of the repeating delimiter, fixed prefixes and suffixes, distribution of numerical segments, position of the suspected time field, and position of the suspected check digit. S34: Perform continuous verification on the new template candidate. When the continuous verification result meets the preset verification pass rate, write the new template candidate into the template fingerprint database. S35: After successfully matching the target template or verifying the new template candidate and writing it into the template fingerprint database, compare the corresponding template fingerprint matching result with the determined data source type. If the two are inconsistent, re-mark the corresponding data access link as pending judgment and re-determine the data source type.

[0013] Furthermore, step S4 includes: S41: Determine the priority of time fields based on the data source type, and select target time fields sequentially from the parsing results according to the priority of time fields; among them, data communication protocol data prioritizes the sampling time field within the message, serial port data prioritizes the parsed time field, file transfer data prioritizes the file content time field or filename time field, and video data prioritizes the signaling time field, stream establishment time field, or pull-stream time field; when no available time field is found in the parsing results, estimate the standard event time based on the data arrival time and device reporting cycle; S42: Generate standard event times based on the target time field, and map the parsing results from different sources to the same time base to generate standardized data containing data source type identifiers, standard event times, and standard fields.

[0014] Furthermore, step S5 includes: S51: Write the standardized data into the corresponding category cache queue according to the data type of the standardized data. The category cache queue includes the time series monitoring data cache queue, the file index cache queue, and the video index cache queue. S52: Determine the timeliness level based on the time difference between the standard event time of the standardized data and the current time. Determine the transmission priority based on at least two of the timeliness level, network status, cache occupancy rate, and remaining power. Retrieve standardized data from the classified cache queue according to the transmission priority and perform hierarchical transmission scheduling. When the network is restricted, prioritize the transmission of time-series monitoring data with a timeliness level higher than the preset threshold. After the network is restored, prioritize the transmission of key observation data within the preset event time window.

[0015] Furthermore, a multi-source heterogeneous data edge processing device for field monitoring includes: The multi-link access module is used to establish multiple independent data access links and receive raw data generated by different monitoring devices at the field monitoring site in parallel based on the data access links; The joint identification module is used to extract multi-dimensional features from the raw data and determine the data source type corresponding to the raw data based on the multi-dimensional features. The differential parsing module is used to call the corresponding parsing engine according to the data source type, and generate parsing results by performing differential parsing on the original data through the parsing engine; The standardization processing module is used to determine the priority of time fields based on the data source type, select the target time field from the parsing results according to the time field priority, perform unified event timeline mapping based on the target time field, and generate standardized data containing data source type identifier and standard event time. The cache scheduling module is used to write standardized data into a categorized cache queue, determine the timeliness level based on the time difference between the standard event time and the current time, and perform hierarchical sending scheduling based on the timeliness level and network status.

[0016] Furthermore, a computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement a multi-source heterogeneous data edge processing method for field monitoring.

[0017] Compared with the prior art, the present invention has the following advantages: This invention reduces the dispersion of front-end devices and system power consumption by establishing multiple independent data access links within a single embedded edge terminal and receiving raw data from different monitoring devices in the field in parallel. It improves the accuracy of identifying heterogeneous data sources by jointly extracting link port features, message frame structure features, session interaction features, and data arrival time sequence features from the raw data. Furthermore, it introduces a mechanism for calculating identification confidence, determining confirmation thresholds, and re-judging ambiguous thresholds to perform secondary verification of data sources with insufficient features or ambiguous identification, avoiding incorrect identification that could lead to incorrect subsequent path selection. Finally, it achieves template matching through a serial port message template fingerprint database. Multi-dimensional scoring and dynamic expansion reduce manual configuration costs and improve the adaptive parsing capability of non-standard serial port data; unified event timeline mapping enables the correlation of time-series monitoring data, file data, and video index data under the same time base; by combining timeliness level, network status, cache utilization, and remaining power to determine transmission priority and execute hierarchical transmission scheduling, edge terminals can prioritize the transmission of key monitoring data under network-limited conditions, improving data availability in weak network environments; joint identification, differentiated parsing, unified event timeline mapping, and hierarchical transmission scheduling form a collaborative processing chain, thereby improving the integrity and stability of multi-source heterogeneous data processing at the edge. Attached Figure Description

[0018] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings: Figure 1 This is a flowchart of a multi-source heterogeneous data edge processing method for field monitoring, as described in an embodiment of the present invention. Figure 2 This is a schematic diagram of the overall process of the multi-source heterogeneous data edge processing method for field monitoring in an embodiment of the present invention; Figure 3 This is a schematic diagram of the embedded edge terminal hardware structure in an embodiment of the present invention; Figure 4 This is a schematic diagram of the multi-link access and edge processing software module in an embodiment of the present invention; Figure 5 This is a flowchart of the joint identification and re-judgment of data source types in an embodiment of the present invention; Figure 6 This is a flowchart of serial port message template fingerprint matching and new template generation in an embodiment of the present invention; Figure 7 This is a schematic diagram of unified event timeline mapping and categorized cache scheduling in an embodiment of the present invention; Figure 8 This is a schematic diagram of a unified and standardized data recording structure in an embodiment of the present invention; Figure 9 This is a schematic diagram of the edge deployment of field ecological monitoring stations in an embodiment of the present invention. Detailed Implementation

[0019] The preferred embodiments of the present invention will be described below with reference to the accompanying drawings. It should be understood that the preferred embodiments described herein are for illustration and explanation only and are not intended to limit the present invention.

[0020] The embodiments of the present invention provide, as follows Figure 1 The method shown is a multi-source heterogeneous data edge processing method for field monitoring, including: S1: Establish multiple independent data access links and receive raw data generated by different monitoring devices at the field monitoring site in parallel based on the data access links; S2: Extract multi-dimensional features from the original data and determine the data source type corresponding to the original data based on the multi-dimensional features; S3: Call the corresponding parsing engine according to the data source type, and use the parsing engine to perform differential parsing on the original data to generate parsing results; S4: Determine the priority of the time field based on the data source type, select the target time field from the parsing results according to the time field priority, perform unified event timeline mapping based on the target time field, and generate standardized data containing data source type identifier and standard event time; S5: Write standardized data into a categorized cache queue, determine the timeliness level based on the time difference between the standard event time and the current time, and perform hierarchical sending scheduling based on the timeliness level and network status.

[0021] The working principle and beneficial effects of the above technical solution are as follows: In S1, multiple independent data access links are established, and raw data is received in parallel based on these links. Each access link establishes an independent channel for different types of monitoring equipment (PakBus devices, RS232 / RS485 serial port devices, FTP file transfer devices, and video devices) in the field monitoring scenario, avoiding processing blockages between data from different sources. This raw data serves as input data for subsequent joint identification. In the actual deployment of field ecological monitoring stations, different devices come from different manufacturers, and their communication interfaces, protocol formats, field organization methods, and time stamping methods differ significantly, resulting in significant multi-source heterogeneity at the station front end. The design of the aforementioned independent links enables multiple heterogeneous data sources to access in parallel within the same embedded edge terminal.

[0022] In S2, multi-dimensional features are extracted from the above-mentioned raw data. These multi-dimensional features cover four dimensions: link port features, message frame structure features, session interaction features, and data arrival time sequence features. Based on these multi-dimensional features, the data source type corresponding to the above-mentioned raw data is determined. The data source type serves as the trigger condition for differentiated parsing, which solves the problem of relying on manual pre-specification of protocol types in the prior art.

[0023] S3 calls the corresponding parsing engine according to the above data source type, and performs differential parsing on the above raw data to generate parsing results. The above parsing engines include PakBus parsing engine, RS232 / RS485 serial port template fingerprint parsing engine, FTP file parsing engine and video parsing engine. The above parsing results contain structured information such as device identifier, sampling time, monitoring parameter name, monitoring value and status identifier extracted from the raw data.

[0024] like Figure 7 As shown, in S4, the priority of the time field is determined according to the above data source type. The target time field is selected from the parsing results according to the priority of the time field. A unified event time axis mapping is performed based on the target time field to generate standardized data containing data source type identifier and standard event time. The above standardized data enables monitoring data from different sources to establish a correlation under the same time benchmark, which solves the problem of insufficient cross-source time normalization capability on the edge side.

[0025] In S5, the timeliness level is determined based on the time difference between the standard event time of the aforementioned standardized data and the current time. Based on this timeliness level and network status, a tiered transmission scheduling process is executed, enabling edge terminals to prioritize the transmission of critical time-series monitoring data with high timeliness levels even under network-constrained conditions. For example... Figure 2 As shown, the above steps constitute a complete technical link from data access, joint identification, differential parsing, time axis mapping to hierarchical transmission. The joint identification result determines the differential parsing path and further affects the determination of standard event time. Standard event time serves as the basis for classification caching and transmission priority determination, thus forming a collaborative processing closed loop from data source identification, field extraction, time normalization to weak network transmission, reducing the dispersion of front-end devices and realizing unified processing of multi-source heterogeneous data at the edge.

[0026] In another embodiment, step S1 includes: S11: Establish data communication protocol data access links, serial port data access links, file transfer data access links, and video data access links, wherein the video data access links include signaling access links and media stream access links; S12: Allocate independent buffers and task queues for the data access links of the data communication protocol, serial port, file transfer, and video data access links respectively, and receive raw data generated by different monitoring devices at the field monitoring site in parallel through the independent buffers and task queues.

[0027] Specifically, the aforementioned data communication protocol data access link is a PakBus data access link, used to receive timing monitoring data output through proprietary protocols such as PakBus; the aforementioned serial port data access link is an RS232 / RS485 serial port data access link, used to receive manufacturer-defined serial port message data output through RS232 or RS485 interfaces; the aforementioned file transfer data access link is an FTP file access link, used to receive image files, log files, or text files generated or transmitted via FTP; the aforementioned video data access link is used to receive video surveillance data provided through video signaling or media streaming, wherein the video data access link includes at least one of the GB / T28181 signaling access link and the RTSP media stream access link. The aforementioned GB / T28181 signaling access link is used to extract device registration information, keep-alive information, channel information, and media resource index; the aforementioned RTSP media stream access link is used to extract stream address, stream identifier, pull time, and video resource index.

[0028] The four types of data access links mentioned above cover the data communication needs of monitoring equipment in field monitoring scenarios, including meteorological, flux, evapotranspiration, sediment, soil observation, camera image, and video surveillance. Example equipment access methods are shown in the table below: Table 1. Sample Table of Device Connections ; The working principle and beneficial effects of the above technical solution are as follows: S11 establishes four independent data access links: PakBus data access link, RS232 / RS485 serial port data access link, FTP file access link, and video data access link. The video data access link is further subdivided into a GB / T28181 signaling access link and an RTSP media stream access link. The GB / T28181 signaling access link is used to extract device registration information, keep-alive information, channel information, and media resource index. The RTSP media stream access link is used to extract stream address, stream identifier, streaming time, and video resource index. These four links cover the data communication needs of monitoring equipment such as meteorological, flux, evapotranspiration, sediment, and soil observation, camera images, and video surveillance in field monitoring scenarios.

[0029] S12 allocates independent buffers and task queues for the four data access links mentioned above. The raw data generated by different monitoring devices at the field monitoring site are received in parallel through the independent buffers and task queues. The independent buffers ensure that data reception and processing of each link do not block each other, and the task queues ensure that message parsing and subsequent processing of each link are executed in sequence.

[0030] like Figure 2 As shown, by establishing independent access links and allocating independent buffers and task queues for PakBus data, RS232 / RS485 serial port data, FTP file data, and video data respectively, processing blockage between data from different sources is avoided. This enables parallel access of the above four types of data within the same embedded edge terminal, reduces the processing latency of a single terminal for multiple types of device data, reduces the number of front-end devices and system power consumption, thereby improving the edge access capability for multi-source heterogeneous data in field monitoring scenarios. The parsing of PakBus data packets, RS232 / RS485 serial port data, serial port template scoring, FTP file data, and video data are shown in the table below: Table 2 Sample PakBus Data Message Parsing Table ; Table 3. Sample Fingerprint Table for RS232 / RS485 Serial Port Message Templates ; Table 4 Sample Scoring Table for Serial Port Templates ; Table 5 Sample FTP File Parsing Table ; Table 6 Sample Table for Video Data Analysis ; In another embodiment, step S2 includes: S21: Extract link port features, message frame structure features, session interaction features, and data arrival time sequence features from the raw data. The link port features and message frame structure features constitute the link features, while the session interaction features and data arrival time sequence features constitute the data features. S22: Based on link port characteristics, message frame structure characteristics, session interaction characteristics, and data arrival time sequence characteristics, calculate the identification confidence of multiple candidate data source types respectively, and determine the data source type corresponding to the original data according to the identification confidence. S23: Use the data source type as the basis for controlling the selection of parsing path and time conversion strategy.

[0031] Specifically, the aforementioned link port characteristics reflect the physical channel attributes of the data source, including at least one of the following: source port, destination port, serial port number, baud rate, parity bit, stop bit, and data bits; the aforementioned message frame structure characteristics reflect the data encapsulation format, including at least one of the following: frame header identifier, frame trailer identifier, field separator, length field, and check field; the aforementioned session interaction characteristics reflect the interaction behavior patterns during data communication, including at least one of the following: registration message, handshake message, acknowledgment message, keep-alive message, and directory message; the aforementioned data arrival timing characteristics reflect the periodicity of data generation, including at least one of the following: message arrival period, file generation period, device reporting period, and stream keep-alive period. These four-dimensional characteristics constitute a multi-dimensional feature vector.

[0032] In practical applications, the identification focus differs for different data source types. PakBus data identification relies more on message frame structure characteristics and device reporting cycle; RS232 / RS485 serial port data identification relies more on serial port parameters, field separators, and numerical segment distribution; FTP file data identification relies more on file generation cycle and session interaction characteristics; GB / T28181 signaling data identification relies more on registration messages, keep-alive messages, and directory messages; and RTSP media stream data identification relies more on stream address establishment, pull session characteristics, and stream keep-alive cycle.

[0033] The working principle and beneficial effects of the above technical solution are as follows: S21 extracts features from the above raw data from four dimensions. The above four-dimensional features constitute a multi-dimensional feature vector, providing comprehensive input information for subsequent data source type determination.

[0034] S22 determines the data source type based on the above multi-dimensional feature vectors. In practical applications, the identification of PakBus data depends more on the message frame structure characteristics and device reporting cycle; the identification of RS232 / RS485 serial port data depends more on serial port parameters, field separators and numerical segment distribution; the identification of FTP file data depends more on the file generation cycle and session interaction characteristics; the identification of GB / T28181 signaling data depends more on registration messages, keep-alive messages and directory messages; and the identification of RTSP media stream data depends more on stream address establishment, pull session characteristics and stream keep-alive cycle.

[0035] S23 uses the aforementioned data source types as the control basis for parsing path selection and time conversion strategy selection, ensuring that subsequent parsing engine calls accurately match the data source type. By jointly extracting features from four dimensions—link port, message frame structure, session interaction, and data arrival time sequence—the data source type is determined. Compared with single-dimensional identification, this significantly improves the accuracy and robustness of heterogeneous data source identification, solving the problem of relying on manual pre-specification of protocol types in existing technologies. This enhances the adaptive identification capability of edge processing for multi-source heterogeneous data in field monitoring scenarios.

[0036] In another embodiment, step S22, which involves determining the data source type based on the identification confidence level, includes: When the maximum value in the confidence score is greater than the preset confirmation threshold, the candidate data source type corresponding to the maximum value is determined as the data source type; If the maximum value does not reach the preset confirmation threshold, continue to collect subsequent features and return to step S22 to recalculate; When the confidence difference between two or more candidate data source types is less than the preset ambiguity threshold, the corresponding data access link is marked as pending judgment. After collecting subsequent message features, session information or arrival period features, the process returns to step S22 for re-judgment.

[0037] The working principle and beneficial effects of the above technical solution are as follows: Based on link port features, message frame structure features, session interaction features and data arrival time sequence features, the identification confidence of multiple candidate data source types are calculated respectively. The identification confidence reflects the degree of matching between each candidate type and the current observed features. The confidence score of each candidate type is obtained by weighted fusion of the four-dimensional features. The confidence scores constitute the confidence ranking of candidate data source types.

[0038] The above determination process includes three branches: The first branch is that when the maximum value in the above identification confidence score is greater than the preset confirmation threshold, the candidate data source type corresponding to the maximum value is determined as the above data source type; the second branch is that when the maximum value does not reach the preset confirmation threshold, subsequent features are collected and the identification confidence score calculation and determination steps are re-executed; the third branch is that when the identification confidence score difference between two or more candidate data source types is less than the preset ambiguity threshold, the corresponding data access link is marked as pending determination, and subsequent message features are collected for re-determination.

[0039] like Figure 5 As shown, by introducing a mechanism for calculating identification confidence, determining confirmation thresholds, and re-judging ambiguous thresholds, secondary verification is performed on data sources with insufficient features or ambiguous identification. This avoids incorrect identification due to insufficient features, thereby preventing incorrect identification from leading to incorrect selection of subsequent parsing paths and improving the robustness of data source type determination in complex field monitoring scenarios.

[0040] In another embodiment, the differential parsing of serial port data in step S3 includes: S31: Establish a serial port message template fingerprint database, and match the received serial port messages with multiple templates in the template fingerprint database to form a candidate template set; S32: Score candidate templates in the candidate template set based on field position stability, numerical type consistency, time field resolvability, and verification pass rate. Select the candidate template with the highest score that is higher than the preset threshold as the target template. Complete field extraction based on the target template and generate serial port data parsing results.

[0041] Specifically, when generating the candidate template set in S31, if there are weakly conflicting template pairs in the set, they are first initially sorted according to the comprehensive confidence level in the weak conflict resolution rules, templates with low confidence are removed, and the remaining templates are used as the final candidate template set in S32. In S32, based on the field position stability, numerical type consistency, time field resolvability, and verification pass rate, the four-dimensional matching score is calculated in real time according to the actual matching situation between the current received message and each candidate template, and weighted normalization is performed in combination with the four-dimensional scoring benchmark parameters in the template to obtain the final score. The candidate template with the highest score and higher than the preset threshold is selected as the target template, and field extraction is completed according to the target template to generate serial port data parsing results. The real-time calculation of the four-dimensional matching score is specifically as follows: the absolute value of the deviation between the field position of the current received message and the standard field position of the template is calculated, and the position score is calculated in combination with the expected value of the field position stability. When the absolute value of the deviation is less than or equal to the reciprocal of the expected value, the full score is obtained; otherwise, it is reduced proportionally. The system calculates the matching status of the numerical type of the currently received message field with the template standard numerical type. If there is a complete match, full marks are awarded; if there is a match of a subtype, partial marks are awarded; otherwise, 0 marks are awarded. The type score is calculated based on the numerical type consistency weight. The system also calculates the parsing time or deviation of the currently received message time field. If the deviation is less than or equal to the time field's resolvability tolerance range, full marks are awarded; otherwise, the score is reduced proportionally. The time score is calculated based on the tolerance range. The system further calculates whether the verification calculation result of the currently received message is consistent with the template verification field. If they are consistent, full marks are awarded; otherwise, 0 marks are awarded. The verification score is calculated based on the verification pass rate benchmark value. The weighted normalization specifically involves dividing the location score, type score, time score, and verification score by the full marks of their respective dimensions to normalize them, resulting in four dimension scores between 0 and 1. The four dimension scores are then multiplied by preset dimension weights and summed to obtain the final score, with the location score having the highest weight, followed by the verification score, then the type score, and finally the time score.

[0042] The working principle and beneficial effects of the above technical solution are as follows: by establishing a serial port message template fingerprint library and combining it with a four-dimensional scoring mechanism to select the optimal target template from the candidate template set, the parsing error rate of non-standard RS232 / RS485 serial port data is reduced, the field extraction failure caused by manual template configuration errors or omissions is avoided, and the adaptive parsing capability of edge terminals for various vendor-customized serial port message formats is improved.

[0043] In another embodiment, the method for establishing a serial port message template fingerprint database in step S31 includes: Obtain a historical serial port message dataset, and preprocess the historical serial port message dataset to obtain several initial message clusters; Select any initial message cluster as the target initial message cluster, and obtain all individual messages within the target initial message cluster; construct a single message byte position axis for each individual message, map each individual message to a position node sequence according to the byte transmission order, and determine the single message structured fingerprint element of each individual message based on the position node sequence; for each individual message within the target initial message cluster, calculate the multi-dimensional stability corresponding to each individual message; determine the single message confidence of each individual message based on the multi-dimensional stability corresponding to each individual message; obtain a preset confidence threshold and a multi-dimensional stability threshold, and filter messages in the target initial message cluster whose built-in confidence is higher than the preset confidence threshold and whose multi-dimensional stability is not lower than the corresponding multi-dimensional threshold, thus agglomerating to obtain a candidate template cluster; Obtain the start and end positions of semantically identical fields in all messages within the candidate template cluster, construct a unified field position distribution axis, map the start and end positions of all semantically identical fields onto the unified field position distribution axis, and generate a field position distribution sequence; use the field boundary position samples in the field position distribution sequence as statistical objects, classify them based on preset conditional classification rules, and obtain the first stable sample and the second stable sample; Based on the first and second stable samples, the standard field structure definition of the serial port message template corresponding to the candidate template cluster is determined. Simultaneously, the fast matching fingerprint element, field attribute label, four-dimensional scoring benchmark parameter and confidence label are encapsulated to obtain the template core fingerprint of the serial port message template corresponding to the candidate template cluster. The candidate template cluster aggregation and template core fingerprint generation steps are repeated for all initial message clusters to obtain the template core fingerprints corresponding to all serial port message templates. The template core fingerprint library is then constructed by summarizing the results. Obtain all template core fingerprints, combine them in pairs to generate template pairs, calculate the fast matching fingerprint meta-similarity of each template pair, filter template pairs with similarity higher than a preset conflict threshold to obtain potential conflict template pairs, build a template conflict resolution rule base based on the potential conflict template pairs, and call it in conjunction with the template core fingerprint base. By integrating the core template fingerprint library and the conflict resolution rule library, a serial port message template fingerprint library is obtained.

[0044] Specifically, a historical serial port message dataset is acquired, and the dataset is preprocessed to obtain several initial message clusters. Abnormal frame removal and cleaning are performed on the historical serial port message dataset to obtain a cleaned message set. From the cleaned message set, a set of frame header identifiers, a message length statistical range, and a function code value range are statistically extracted. The cleaned messages are then clustered using the frame header identifiers, message length range, and function code value range as clustering dimensions to obtain several initial message clusters.

[0045] Select any initial message cluster as the target initial message cluster, and obtain all individual messages within the target initial message cluster; construct a single message byte position axis for each individual message, map each individual message to a position node sequence according to the byte transmission order, and determine the single message structured fingerprint element of each individual message based on the position node sequence; wherein, determining the single message structured fingerprint element of each individual message based on the position node sequence includes: each position node carrying byte value, semantic type, and bit width attribute; extract anchor point features from the byte position axis of each individual message, the anchor point features including frame header, frame tail, and check field; locate the corresponding anchor point coordinates on the byte position axis, based on the preset anchor point-capture The direction mapping rule extends bidirectionally along the byte axis within the frame to divide the data field range, obtaining the start and end boundaries of the data field for each individual message. Specifically, the anchor-capture direction mapping rule is as follows: if the anchor point is the frame header, the capture direction is forward, the data field start boundary is the end position of the frame header, and the end boundary is the start position of the frame tail; if the anchor point is the frame tail, the capture direction is reverse, the data field end boundary is the start position of the frame tail, and the start boundary is the end position of the frame header; if the anchor point is a checksum field, the start boundary is determined by extending towards the frame header and the end boundary is determined by extending towards the frame tail, based on the relative position of the checksum field in the message. Frame-level first-order features and fields are extracted within the data field range of each individual message. The system encapsulates the third-level features of a single message according to a hierarchical relationship, using second-level features at the level of byte and third-level features at the level of byte. This yields a structured fingerprint element for each message, carrying a unique identifier and its cluster label. Specifically, establishing a single message byte position axis involves mapping each message to a one-dimensional linear position axis according to its byte transmission order. Each byte corresponds to a position node on the axis, with node attributes including byte value, semantic type (fixed / variable / checksum), and bit width. This process transforms the message from a binary stream to a structured axis sequence. The frame-level first-level features include four global features: total message length, function code value, frame header sequence, and checksum algorithm type. These features generate a frame-level feature vector, which serves as the fingerprint for subsequent rapid template matching. The foundation consists of three levels: First, the field-level second-order features are initially defined within the data domain based on byte value fluctuation patterns. The starting position, byte length, numeric type (ASCII / hexadecimal enumeration / binary bit field / timestamp), and upper and lower value limits of each candidate field are extracted to generate a field-level feature set. Second, the byte-level third-order features are parsed within each candidate field to extract fixed byte bit distribution, variable byte fluctuation range, the associated calculation domain of the check byte, and the bit offset of the bit field, generating a byte-level feature set. Third, the three-level features are encapsulated at the frame-field-byte level to generate a single-message structured fingerprint element, with each fingerprint element carrying a unique identifier and its associated cluster label.

[0046] For each individual message within the target initial message cluster, the multi-dimensional stability of each individual message is statistically calculated. Based on the multi-dimensional stability of each individual message, the confidence level of that message is determined. Preset confidence thresholds and multi-dimensional stability thresholds are obtained, and messages with built-in confidence levels higher than the preset confidence thresholds and multi-dimensional stability levels not lower than their corresponding multi-dimensional thresholds are selected from the target initial message cluster to obtain a candidate template cluster. Specifically, for each individual message within the target initial message cluster, the multi-dimensional stability of each message is statistically calculated. Calculate the multi-dimensional stability corresponding to each individual message; the multi-dimensional stability includes frame-level stability, field-level stability, and byte-level stability; using the mode of the first-order frame-level features of all messages in the cluster as a benchmark, calculate the matching degree between the frame-level features of each individual message and the mode, to obtain the frame-level stability S1 of the individual message. The matching degree is specifically as follows: compare the four global features of the first-order frame-level features with the mode in the cluster respectively. If they are completely consistent, record it as 1; otherwise, record it as 0. Add the scores of the four features and divide by 4 to obtain the frame-level stability. Assign the first weight Using the mean of field-level features at the same position within the cluster as a benchmark, calculate the reciprocal of the deviation of each individual message's field-level features from this mean to obtain the field-level stability of that individual message. The reciprocal of the deviation is specifically calculated as follows: the Euclidean distance D between the field-level feature vector of each individual message and the cluster mean is calculated. If D=0, then the following is set. =1; if D>0, then calculate To obtain field-level stability Assign a second weight Based on the byte distribution within a cluster's fields, the degree of fit between the byte-level characteristics of each individual message and this distribution is calculated to obtain the byte-level stability of that individual message. The specific degree of agreement is as follows: The KL divergence algorithm (Kullback-Leibler Divergence) is used to calculate the difference value KL between the byte-level feature probability distribution of each individual message and the average probability distribution of byte-level features within the cluster. Byte-level stability is obtained Assigning a third weight ;in, And it satisfies the normalization constraint. Formula for calculating the confidence level of a single message: ; Filter the messages whose built-in confidence is higher than the preset confidence threshold and whose multi-dimensional stability is not lower than the corresponding sub-dimensional threshold, and agglomerate them to obtain candidate template clusters.

[0047] Obtain the start and end positions of semantically identical fields in all messages within the candidate template cluster, construct a unified field position distribution axis, and map all semantically identical field start and end positions onto the unified field position distribution axis to generate a field position distribution sequence. Use the field boundary position samples in the field position distribution sequence as statistical objects, and classify them based on a preset conditional classification rule to obtain a first stable sample and a second stable sample. Specifically, map the start and end positions of semantically identical fields in all messages within the candidate template cluster onto a unified byte position axis to generate a field position distribution sequence. Each position point corresponds to a field boundary of a message, and the concentration of the position distribution directly reflects the stability of the field. The preset conditional classification rule is as follows: words whose start positions are continuous without offset and whose corresponding position points are within a preset byte range are classified as stable. Segment boundary position samples are classified as first stable samples; field boundary position samples with a slight offset at the start position of the field and a semantic difference before and after the offset less than or equal to a preset difference threshold are classified as second stable samples. The semantic difference is specifically calculated as follows: extract the byte sequence within each preset length byte window before and after the field boundary position sample, calculate the information entropy of the byte values ​​in the two windows respectively, and calculate the absolute value of the difference between the two information entropies as the first difference value; at the same time, determine whether the parsed data types of the byte sequences in the two windows are consistent. If they are inconsistent, the second difference value is 1, and if they are consistent, it is 0; the first difference value and the second difference value are weighted and summed to obtain the semantic difference; field boundary position samples with a field start position offset greater than a preset slight offset threshold or a semantic difference before and after the offset greater than a preset difference threshold are removed.

[0048] Based on the first and second stable samples, the standard field structure definition of the serial port message template corresponding to the candidate template cluster is determined. Simultaneously, fast matching fingerprint elements, field attribute labels, four-dimensional scoring benchmark parameters, and confidence labels are encapsulated to obtain the core fingerprint of the serial port message template corresponding to the candidate template cluster. The candidate template cluster aggregation and core fingerprint generation steps are repeated for all initial message clusters to obtain the core fingerprints corresponding to all serial port message templates. These are then aggregated to construct a core fingerprint library. Specifically, the standard field structure definition of the serial port message template corresponding to the candidate template cluster is determined based on the first and second stable samples, including: extracting the field benchmark position, standard numerical type, standard value range, and time field label from the first stable sample. The quasi-parsing format and standard calculation rules for the check domain are used to generate a first field normal form and assign it a first fusion weight. From the second stable sample, the allowed field offset, type compatibility rules, value extension range, time field compatible format, and check domain fault tolerance rules are extracted to generate a second field normal form and assign it a second fusion weight. The first fusion weight is greater than the second fusion weight. The two types of field normal forms are fused to obtain the standard field structure definition of the serial port message template corresponding to the candidate template cluster. Specifically, for field positions, the reference position of the field in the first field normal form is taken as the standard position, and the allowed field offset of the field in the second field normal form is taken as the tolerance range of the standard position, and these are combined to generate a standard field position definition. For numeric types, the first field normal form... The standard numeric type is used as the primary type, and the type compatibility rules of the second field normal form are used as the secondary type to generate the standard numeric type definition. For the value range, the union of the standard value range of the first field normal form and the value extension range of the second field normal form is taken to generate the standard value range definition. For the time field and the check domain, the standard format or rule of the first field normal form is used as the default rule, and the compatibility format or fault tolerance rule of the second field normal form is used as the degradation rule, and the standard parsing definition is generated by combining them. Simultaneously, the fast matching fingerprint element, field attribute label, four-dimensional scoring benchmark parameters and confidence label are encapsulated to obtain the template core fingerprint of the serial port message template corresponding to the candidate template cluster. Among them, the fast matching fingerprint element includes the frame header sequence hash, length range, and function code. The three matching features of the value range are used for rapid initial screening of subsequent messages to generate a candidate template set; the field attribute labels include the fixed position, type, value range, and parsing rules of each field to support subsequent field extraction; the four-dimensional scoring benchmark parameters are specifically the built-in expected value of field position stability, weight of numerical type consistency, tolerance range of time field resolvability, and benchmark value of verification pass rate. The specific statistical steps of the four-dimensional scoring benchmark parameters are as follows: calculate the standard deviation of the field positions of all messages in the candidate template cluster, and use its reciprocal as the expected value of field position stability; calculate the proportion of consistent numerical types of each field in the cluster as the weight of numerical type consistency; calculate the maximum timestamp deviation when the time field is successfully parsed in the cluster as the tolerance range of time field resolvability.The percentage of messages that pass the verification calculation within a cluster is used as the benchmark value for the verification pass rate; during matching, it serves as the reference weight and normalization benchmark for calculating the actual matching score between the currently received message and the template; confidence label: stores the overall confidence of the template, serving as the priority basis for conflict resolution; traversing all initial message clusters, the steps of candidate template cluster aggregation and template core fingerprint generation are repeated to obtain the template core fingerprints corresponding to all serial port message templates, and the template core fingerprint library is constructed by summarizing them.

[0049] Obtain all core fingerprints of the templates, combine them in pairs to generate template pairs, calculate the fast matching fingerprint meta-similarity of each template pair, and filter template pairs with similarity higher than a preset conflict threshold to obtain potential conflict template pairs. Based on these potential conflict template pairs, construct a template conflict resolution rule base. Specifically: for each potential conflict template pair, define the template with higher overall confidence as the master template and the other template as the control template; construct a unified byte position axis, align the common anchor point positions of the two potential conflict templates to determine the baseline alignment coordinates, and traverse the position and type differences of non-anchor fields within a preset byte range before and after the baseline coordinates, uniformly statistically analyzing the master and control templates. The total difference count is calculated by counting the number of times the semantic difference of the template is greater than or equal to a preset threshold. A preset conflict level comparison table is obtained, and template pairs are divided into three levels—no conflict, weak conflict, and strong conflict—based on the total difference count. Specifically, if the total difference count is 0, the pair is classified as no conflict; if the total difference count is greater than 0 and less than or equal to a preset weak conflict threshold, the pair is classified as weak conflict; if the total difference count is greater than the preset weak conflict threshold, the pair is classified as strong conflict. For weak conflict template pairs, a matching priority is set according to the overall confidence level. For strong conflict template pairs, differential field features are extracted and added to the corresponding template. After quickly matching fingerprint elements, the similarity of the updated template pairs is recalculated. Specifically, this recalculation involves converting the supplemented differential field features into binary feature vectors, calculating the cosine similarity between the updated fingerprint element feature vectors of the two templates, and using this cosine similarity as the updated template pair similarity. If there are still potentially conflicting template pairs exceeding a preset conflict threshold, the conflict resolution rules are iteratively executed until there are no potential conflicts or the preset maximum number of iterations is reached. If strong conflicting template pairs still exist after reaching the preset maximum number of iterations, the template with the higher overall confidence is retained as the valid template, and those with higher overall confidence are discarded. Templates with low confidence are used. A template conflict resolution rule library is constructed by summarizing the conflict levels and resolution rules of all template pairs. Specifically, within a preset byte range before and after the baseline coordinates, the position and type differences of non-anchor fields are traversed. The number of times the semantic difference of the main template is greater than or equal to a preset threshold is uniformly counted to obtain the total difference count. This includes: regardless of whether the field boundary of the reference template appears before or after the baseline coordinates, the main template is used as the baseline, and the semantic difference of the main template's fields within a preset range after the baseline coordinates is counted. If the difference is ≥ the preset threshold, the count is incremented by 1. The count results of all fields are summarized to obtain the total difference count. Based on a preset conflict level comparison table, template pairs are divided into three levels: no conflict, weak conflict, and strong conflict, according to the total difference count. The weak conflict resolution rule includes comparing the overall confidence of the two templates, setting priority ranking, and prioritizing the template with higher confidence during matching. The strong conflict resolution rule includes extracting the differentiated field features of the two templates and supplementing them into their respective fast matching fingerprint elements to increase feature discrimination and eliminate matching conflicts at the root.By integrating the core template fingerprint library and the conflict resolution rule library, a serial port message template fingerprint library is obtained.

[0050] The beneficial effects of the above technical solution are as follows: Structured representation of messages is achieved by constructing a single message byte position axis to extract three-level features, and the purity of template clusters is ensured from the source by combining a screening mechanism based on individual multi-dimensional stability. Simultaneously, by using conditional classification rules to analyze field position distribution, the dual paradigms of absolute stability and elastic offset are organically integrated, significantly enhancing the template's fault tolerance to field transmission jitter and format variations. Furthermore, the data-driven built-in four-dimensional scoring benchmark parameters effectively reduce the real-time computational overhead at the edge, and the introduction of a deterministic master-slave conflict resolution and iterative update mechanism eliminates matching ambiguities between similar templates. Ultimately, a serial port message template fingerprint library with high accuracy, strong robustness, and low computational overhead is constructed, providing a solid foundation for real-time and reliable analysis of field edge monitoring.

[0051] In another embodiment, step S32 is followed by: S33: When a preset number of consecutive serial port messages fail to match the target template that meets the threshold condition, a new template candidate is generated based on the position of the repeating delimiter, fixed prefixes and suffixes, distribution of numerical segments, position of the suspected time field, and position of the suspected check digit. S34: Perform continuous verification on the new template candidate. When the continuous verification result meets the preset verification pass rate, write the new template candidate into the template fingerprint database. S35: After successfully matching the target template or verifying the new template candidate and writing it into the template fingerprint database, compare the corresponding template fingerprint matching result with the determined data source type. If the two are inconsistent, re-mark the corresponding data access link as pending judgment and re-determine the data source type.

[0052] When the data source type is determined to be FTP file data, file content parsing or file index extraction is performed. For text log files, time, monitoring, status, and alarm fields are extracted and structured records are formed. For image files, large files, or files that do not need to be uploaded in full, index information such as file path, file name, file size, file creation time, local cache location, and hash value is extracted. FTP file data can be parsed for file content or only the file index can be extracted to adapt to edge processing needs under different bandwidth conditions.

[0053] When the data source type is determined to be video data, the differential parsing module performs video signaling parsing or media stream resource index extraction. For GB / T28181 signaling access links, it extracts device registration information, keep-alive information, channel information, and media resource index; for RTSP media stream access links, it extracts stream address, stream identifier, streaming time, and video resource index. In some implementations, the terminal does not perform full decoding of the video content, but instead prioritizes extracting the media stream resource index to reduce computing power consumption and network usage.

[0054] The working principle and beneficial effects of the above technical solution are as follows: In S33, when a preset number of consecutive RS232 / RS485 serial port messages fail to match the target template that meets the threshold condition, it indicates that there is no template matching the current message format in the template fingerprint database. At this time, new template candidates are generated based on the feature analysis of five dimensions: the position of the repeating delimiter, fixed prefixes and suffixes, the distribution of numerical segments, the position of the suspected time field, and the position of the suspected check bit. The specific analysis methods for each dimension include: For each character in N consecutively collected serial port messages, the frequency of occurrence of each character at a fixed byte offset position is counted byte by byte. Characters whose frequency exceeds a preset frequency threshold and whose positions are fixed across multiple messages are identified as field separators. The position sequence recorded in the message is used as the field separator point definition for the new template. Extract the first M bytes and the last M bytes from the above N messages, calculate the consistency rate of the byte values ​​at each byte position, determine the prefix byte sequence with a consistency rate exceeding a preset frequency threshold as the message start identifier, determine the corresponding suffix byte sequence as the message end identifier, and record the corresponding byte values ​​and lengths as the frame boundary definition of the new template. The message segments between the field separators are parsed using integer, floating-point, and BCD encoding formats. The success rate of parsing the values ​​at each field position is counted. Field positions with a success rate exceeding a preset threshold are marked as numeric fields. The corresponding position, length, and numeric type are recorded as the numeric field definition for the new template. For each field fragment in the message, try regular expression matching or numerical range judgment for common time formats in turn. The field positions with a matching success rate exceeding the preset frequency threshold are identified as suspected time fields. Record the position and format type as the time field definition of the new template. For 1 to 4 byte segments at the end of the message or at a fixed offset position, try CRC16 (16-bit Cyclic Redundancy Check 16), CRC8 (8-bit Cyclic Redundancy Check 8), XOR (Exclusive OR), and summation check algorithms respectively. Byte positions with a check pass rate exceeding a preset frequency threshold are identified as suspected check bits, and their positions, lengths, and check algorithm types are recorded as the check rule definition for the new template.

[0055] Based on the above analysis results, new template candidates are generated, including start identifier, end identifier, field separator pattern, field length pattern, numeric type pattern, time field pattern, and verification rules, as input for subsequent continuous verification. Specifically, the recurring separator position analysis is used to identify repeatedly occurring separator characters and their positional patterns in the message; the fixed prefix and suffix analysis is used to identify the start and end markers of the message; the numeric segment distribution analysis is used to identify the position and length distribution of numeric fields in the message; the suspected time field position analysis is used to identify message segments that conform to time encoding format characteristics; and the suspected check bit position analysis is used to identify check fields at the end of the message or at specific positions.

[0056] S34 performs continuous verification on the above-mentioned new template candidates. When the continuous verification results meet the preset verification pass rate, the above-mentioned new template candidates are written into the above-mentioned template fingerprint database to ensure the reliability of the newly added templates.

[0057] In S35, after a target template or a new template candidate is successfully matched and verified and written into the template fingerprint database, the template fingerprint matching result is used to reverse verify the data source type determination result. When the template matching result is inconsistent with the determined data source type, the corresponding data access link is remarked as pending determination. The reverse verification mechanism enables the differential parsing result and the joint identification result to form cross-verification.

[0058] like Figure 6 As shown, by establishing a serial port message template fingerprint database and combining it with a four-dimensional scoring mechanism to select the optimal target template from the candidate template set, the parsing error rate of non-standard RS232 / RS485 serial port data is reduced, and the adaptive parsing capability of edge terminals for various vendor-defined serial port message formats is improved. Simultaneously, through dynamic generation of new templates and a reverse verification closed loop, the serial port message template fingerprint database has adaptive update capabilities, adapting to the RS232 / RS485 serial port message formats of newly added device models without manual intervention, reducing manual configuration costs, and is particularly suitable for situations where device models change frequently in field monitoring scenarios.

[0059] For PakBus data, the differential parsing engine performs message unpacking and monitoring field extraction, extracting information including device identifier, sampling time, monitoring parameter name, monitoring value, and status identifier, forming a structured time-series record. For FTP file data, the differential parsing engine performs file content parsing or file index extraction, with the file index including file name, file creation time, file size, and file type identifier. For video data, the differential parsing engine performs video signaling parsing or media stream resource index extraction, extracting information including device registration information, channel information, and media resource index. The aforementioned differential parsing engine automatically calls the corresponding parsing path based on the data source type, achieving unified differential parsing of PakBus data, RS232 / RS485 serial port data, FTP file data, and video data at the edge, improving the parsing efficiency and accuracy of multi-source heterogeneous data in field monitoring scenarios.

[0060] In another embodiment, step S4 includes: S41: Determine the priority of time fields based on the data source type, and select target time fields sequentially from the parsing results according to the priority of time fields; among them, data communication protocol data prioritizes the sampling time field within the message, serial port data prioritizes the parsed time field, file transfer data prioritizes the file content time field or filename time field, and video data prioritizes the signaling time field, stream establishment time field, or pull-stream time field; when no available time field is found in the parsing results, estimate the standard event time based on the data arrival time and device reporting cycle; S42: Generate standard event times based on the target time field, and map the parsing results from different sources to the same time base to generate standardized data containing data source type identifiers, standard event times, and standard fields.

[0061] For each parsing result, record at least three of the following: original acquisition time, device local time, standard event time, time source identifier, and time conversion validity identifier, and map time series monitoring data, file data, and video index data to the same time base.

[0062] After completing the unified event timeline mapping, the standardization processing module further performs field mapping, unit conversion, anomaly detection, and integrity verification. The anomaly detection includes at least one of the following: range threshold detection, rate of change threshold detection, and cross-source consistency detection. The integrity verification includes at least one of the following: message-level verification, file-level verification, and aggregate packet-level verification.

[0063] The working principle and beneficial effects of the above technical solution are as follows: S41 determines the priority of time fields according to the above data source types. Among them, PakBus data prioritizes the sampling time field within the message, RS232 / RS485 serial port data prioritizes the parsed time field, FTP file data prioritizes the file content time field or file name time field, and video data prioritizes the signaling time field, stream establishment time field, or pull stream time field. When there is no available time field in the parsing result, the standard event time is estimated based on the data arrival time and the corresponding device reporting cycle.

[0064] S42 generates a standard event time based on the target time field mentioned above. The parsing results from different sources are then mapped to the same time base using the standard event time, generating standardized data that includes data source type identifier, standard event time, and standard fields.

[0065] Regarding anomaly detection, the aforementioned standardized processing module performs three types of anomaly detection on the parsing results: range threshold detection is used to detect whether the monitored value exceeds the physically reasonable range, change rate threshold detection is used to detect whether the rate of change of the monitored value is abnormal, and cross-source consistency detection is used to detect the logical consistency between data from different sources within the same time period. For example, cross-source consistency detection is performed on multiple temperature observations from PakBus devices and RS232 / RS485 serial port devices within the same event time window.

[0066] Regarding integrity verification, the aforementioned standardized processing module performs message-level integrity verification, file-level integrity verification, and aggregated packet-level integrity verification. For example, it performs file-level verification on FTP file data and message-level verification on RS232 / RS485 serial port messages and PakBus data.

[0067] By setting the priority of time fields according to the data source type and selecting target time fields from the parsing results to generate standard event times, PakBus time-series monitoring data, RS232 / RS485 serial port data, FTP file data, and video index data are associated under the same time base. At the same time, the reliability of standardized data is improved through three types of anomaly discrimination mechanisms and three-level integrity verification, thereby enhancing the ability of edge processing to perform cross-source time correlation of multi-source heterogeneous data in field monitoring scenarios.

[0068] In another embodiment, step S5 includes: S51: Write the standardized data into the corresponding category cache queue according to the data type of the standardized data. The category cache queue includes the time series monitoring data cache queue, the file index cache queue, and the video index cache queue. S52: Determine the timeliness level based on the time difference between the standard event time of the standardized data and the current time. Determine the transmission priority based on at least two of the timeliness level, network status, cache occupancy rate, and remaining power. Retrieve standardized data from the classified cache queue according to the transmission priority and perform hierarchical transmission scheduling. When the network is restricted, prioritize the transmission of time-series monitoring data with a timeliness level higher than the preset threshold. After the network is restored, prioritize the transmission of key observation data within the preset event time window.

[0069] Specifically, the aforementioned timing monitoring data cache queue is used to cache timing monitoring records obtained from PakBus parsing and timing monitoring records obtained from RS232 / RS485 serial port data parsing; the aforementioned file index cache queue is used to cache file index information obtained from FTP file data parsing; and the aforementioned video index cache queue is used to cache video index information obtained from video data parsing.

[0070] In the site edge deployment scheme, the aforementioned embedded edge terminal is deployed at the data aggregation node of the field monitoring site. It connects to the surrounding monitoring equipment through PakBus data access link, RS232 / RS485 serial port data access link, FTP file access link and video data access link respectively. As a site-level edge processing node, it performs joint identification, differential parsing, time axis mapping and cache scheduling of multi-source heterogeneous data locally, and then sends the standardized data to the remote monitoring platform.

[0071] The working principle and beneficial effects of the above technical solution are as follows: S51 writes the above standardized data into the corresponding classification cache queue according to the data type of the above standardized data. The above time-series monitoring data cache queue is used to cache the time-series monitoring records obtained by PakBus parsing and the time-series monitoring records obtained by RS232 / RS485 serial port data parsing. The above file index cache queue is used to cache the file index information obtained by FTP file data parsing. The above video index cache queue is used to cache the video index information obtained by video data parsing.

[0072] S52 determines the timeliness level based on the time difference between the standard event time of the above-mentioned standardized data and the current time. The timeliness level is divided into three levels according to the corresponding standard. The division standard in this embodiment is as follows: when Δt≤60s, it is determined to be high timeliness level, indicating that the data has real-time requirements and should be sent first; when 60s<Δt≤300s, it is determined to be medium timeliness level, indicating that the data can be sent with a short delay; when Δt>300s, it is determined to be low timeliness level, indicating that the data is historical backlog data and can be resent when resources are sufficient.

[0073] After determining the timeliness level, the sending priority P is calculated by combining four factors: timeliness level, network status, buffer occupancy rate, and remaining battery power. The calculation formula is as follows:

[0074] in This refers to the timeliness rating score; Network status score; This represents the cache utilization score; a lower cache utilization score indicates less current sending pressure. The remaining battery power score is represented by a, b, c, and d, which are the weight coefficients of each factor and satisfy a+b+c+d=1. The recommended values ​​are a=0.4, b=0.3, c=0.2, and d=0.1. Timeliness level and network status are given higher weights to ensure that critical time-series data is sent first under weak network conditions.

[0075] When the network is in a restricted state, priority is given to sending time-series monitoring data with a timeliness level higher than a preset threshold. Once the network recovers from the restricted state, priority is given to sending key observation data within a preset event time window cached during the restricted period. When bandwidth permits, more complete index information or further corresponding resources are sent for FTP file data and video data.

[0076] like Figure 9 As shown, the aforementioned embedded edge terminal is deployed at the data aggregation node of the field monitoring site, and connects to surrounding monitoring equipment through four types of data access links. Through categorized caching and hierarchical transmission scheduling, the edge terminal prioritizes the transmission of key time-series monitoring data in weak network environments, and prioritizes the transmission of key observation data within a preset event time window after network recovery. This improves data availability and transmission efficiency in weak network environments, thereby enhancing the edge processing's ability to transmit key data under weak network conditions in field monitoring scenarios.

[0077] In another embodiment, a multi-source heterogeneous data edge processing device for field monitoring is characterized by comprising: The multi-link access module is used to establish multiple independent data access links and receive raw data generated by different monitoring devices at the field monitoring site in parallel based on the data access links; The joint identification module is used to extract multi-dimensional features from the raw data and determine the data source type corresponding to the raw data based on the multi-dimensional features. The differential parsing module is used to call the corresponding parsing engine according to the data source type, and generate parsing results by performing differential parsing on the original data through the parsing engine; The standardization processing module is used to determine the priority of time fields based on the data source type, select the target time field from the parsing results according to the time field priority, perform unified event timeline mapping based on the target time field (as shown in Table 7), and generate standardized data containing data source type identifiers and standard event times. The cache scheduling module is used to write standardized data into the classified cache queue (as shown in Table 8), determine the timeliness level based on the time difference between the standard event time and the current time, and perform hierarchical transmission scheduling based on the timeliness level and network status.

[0078] The working principle and beneficial effects of the above technical solution are as follows: the output data of the multi-link access module is the input of the joint identification module; the judgment result of the joint identification module controls the engine call of the differential parsing module; the parsing result of the differential parsing module is the mapping input of the standardization processing module; and the standardized data of the standardization processing module is the cache and sending object of the cache scheduling module. This forms a collaborative processing closed loop from data access, joint identification, differential parsing, time axis mapping to hierarchical transmission.

[0079] The joint identification results determine the differentiated parsing path and further influence the determination of the standard event time; the standard event time, in turn, serves as the basis for classification caching and sending priority determination. This forms a closed-loop processing mechanism from data source identification, field extraction, time normalization to weak network transmission, rather than an arbitrary parallel execution of multiple conventional processing steps. This invention establishes a collaborative processing chain for field monitoring scenarios, thereby improving the ability of a single edge terminal to access multi-source heterogeneous data, its standardization capabilities, its cross-source correlation capabilities, and its adaptability to weak networks.

[0080] like Figure 3 As shown, the aforementioned embedded edge terminal includes at least a processor, memory, communication interface, and local storage module, wherein the communication interface includes a serial port interface, a network interface, and a wireless communication interface. Figure 4 As shown, the software architecture of the aforementioned embedded edge terminal includes a multi-link access layer, a joint identification layer, a differential parsing layer, a standardized processing layer, and a cache scheduling layer. Figure 8 As shown, the above-mentioned unified and standardized data record includes three core components: data source type identifier, standard event time, and standard fields (the comparison of the solution effects is shown in Table 9).

[0081] Table 7 Sample Table of Unified Event Timeline Mapping ; Table 8 Sample Table of Categorized Cache Queues and Tiered Sending ; Table 9: Comparison of the Effects of the Schemes ; In another embodiment, a computer-readable storage medium stores computer-executable instructions that, when executed by a processor, are used to implement a multi-source heterogeneous data edge processing method for field monitoring.

[0082] The working principle and beneficial effects of the above technical solution are as follows: Based on S1, PakBus data access links, RS232 / RS485 serial port data access links, FTP file access links, and video data access links are established, and independent buffers and task queues are allocated respectively to receive raw data in parallel; based on S2, four-dimensional features—link port features, message frame structure features, session interaction features, and data arrival time sequence features—are extracted, and the data source type is determined based on these features; based on S3, the PakBus parsing engine, RS232 / RS485 serial port template fingerprint parsing engine, FTP file parsing engine, or video parsing engine are called respectively to perform differentiated parsing according to the data source type; the priority of the time field is determined according to the data source type, and PakBus time-series monitoring data, RS232 / RS485 serial port data, FTP file data, and video index data are uniformly mapped to the same time base to generate standardized data; the timeliness level is determined based on the time difference between the standard event time and the current time, and hierarchical transmission scheduling is performed in combination with the timeliness level and network status.

[0083] By embedding the above methods into computer programs and storing them in computer-readable storage media, the above multi-source heterogeneous data edge processing methods are made portable and deployable, making it easy to implement on different models of embedded edge terminals through software upgrades, thereby reducing hardware replacement costs.

[0084] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from the spirit and scope of this invention.

Claims

1. A method for edge processing of multi-source heterogeneous data for field monitoring, characterized in that, include: S1: Establish multiple independent data access links and receive raw data generated by different monitoring devices at the field monitoring site in parallel based on the data access links; S2: Extract multi-dimensional features from the original data, calculate the identification confidence of multiple candidate data source types based on the multi-dimensional features, and determine the data source type corresponding to the original data according to the identification confidence; wherein, when determining the data source type according to the identification confidence, if the maximum value of the identification confidence does not reach the preset confirmation threshold, or the difference in identification confidence between two or more candidate data source types is less than the preset ambiguity threshold, supplementary features are collected and the determination is made again. S3: Call the corresponding parsing engine according to the data source type, and use the parsing engine to perform differential parsing on the original data to generate parsing results; S4: Determine the priority of the time field based on the data source type, select the target time field from the parsing results according to the time field priority, perform unified event timeline mapping based on the target time field, and generate standardized data containing data source type identifier and standard event time; S5: Write standardized data into a categorized cache queue, determine the timeliness level based on the time difference between the standard event time and the current time, and perform hierarchical sending scheduling based on the timeliness level and network status; The differential parsing of serial port data in step S3 includes: S31: Establish a serial port message template fingerprint database, and match the received serial port messages with multiple templates in the template fingerprint database to form a candidate template set; S32: Score candidate templates in the candidate template set based on field position stability, numerical type consistency, time field resolvability, and verification pass rate. Select the candidate template with the highest score that is higher than the preset threshold as the target template. Complete field extraction based on the target template and generate serial port data parsing results.

2. The multi-source heterogeneous data edge processing method for field monitoring according to claim 1, characterized in that, Step S1 includes: S11: Establish data communication protocol data access links, serial port data access links, file transfer data access links, and video data access links, wherein the video data access links include signaling access links and media stream access links; S12: Allocate independent buffers and task queues for the data access links of the data communication protocol, serial port, file transfer, and video data access links respectively, and receive raw data generated by different monitoring devices at the field monitoring site in parallel through the independent buffers and task queues.

3. The multi-source heterogeneous data edge processing method for field monitoring according to claim 1, characterized in that, Step S2 includes: S21: Extract link port features, message frame structure features, session interaction features, and data arrival time sequence features from the raw data. The link port features and message frame structure features constitute the link features, while the session interaction features and data arrival time sequence features constitute the data features. S22: Based on link port characteristics, message frame structure characteristics, session interaction characteristics, and data arrival time sequence characteristics, calculate the identification confidence of multiple candidate data source types respectively, and determine the data source type corresponding to the original data according to the identification confidence. S23: Use the data source type as the basis for controlling the selection of parsing path and time conversion strategy.

4. The multi-source heterogeneous data edge processing method for field monitoring according to claim 3, characterized in that, The steps in step S22 that determine the data source type based on the identification confidence level include: When the maximum value in the confidence score is greater than the preset confirmation threshold, the candidate data source type corresponding to the maximum value is determined as the data source type; If the maximum value does not reach the preset confirmation threshold, continue to collect subsequent features and return to step S22 to recalculate; When the confidence difference between two or more candidate data source types is less than the preset ambiguity threshold, the corresponding data access link is marked as pending judgment. After collecting subsequent message features, session information or arrival period features, the process returns to step S22 for re-judgment.

5. The multi-source heterogeneous data edge processing method for field monitoring according to claim 1, characterized in that, The method for establishing a serial port message template fingerprint database in S31 includes: Obtain a historical serial port message dataset, and preprocess the historical serial port message dataset to obtain several initial message clusters; Select any initial message cluster as the target initial message cluster, and obtain all individual messages within the target initial message cluster; construct a single message byte position axis for each individual message, map each individual message to a position node sequence according to the byte transmission order, and determine the single message structured fingerprint element of each individual message based on the position node sequence; for each individual message within the target initial message cluster, calculate the multi-dimensional stability corresponding to each individual message; determine the single message confidence of each individual message based on the multi-dimensional stability corresponding to each individual message; obtain a preset confidence threshold and a multi-dimensional stability threshold, and filter messages in the target initial message cluster whose built-in confidence is higher than the preset confidence threshold and whose multi-dimensional stability is not lower than the corresponding multi-dimensional threshold, thus agglomerating to obtain a candidate template cluster; Obtain the start and end positions of semantically identical fields in all messages within the candidate template cluster, construct a unified field position distribution axis, map the start and end positions of all semantically identical fields onto the unified field position distribution axis, and generate a field position distribution sequence; use the field boundary position samples in the field position distribution sequence as statistical objects, classify them based on preset conditional classification rules, and obtain the first stable sample and the second stable sample; Based on the first and second stable samples, the standard field structure definition of the serial port message template corresponding to the candidate template cluster is determined. Simultaneously, the fast matching fingerprint element, field attribute label, four-dimensional scoring benchmark parameter and confidence label are encapsulated to obtain the template core fingerprint of the serial port message template corresponding to the candidate template cluster. The candidate template cluster aggregation and template core fingerprint generation steps are repeated for all initial message clusters to obtain the template core fingerprints corresponding to all serial port message templates. The template core fingerprint library is then constructed by summarizing the results. Obtain all template core fingerprints, combine them in pairs to generate template pairs, calculate the fast matching fingerprint meta-similarity of each template pair, filter template pairs with similarity higher than a preset conflict threshold to obtain potential conflict template pairs, build a template conflict resolution rule base based on the potential conflict template pairs, and call it in conjunction with the template core fingerprint base. By integrating the core template fingerprint library and the conflict resolution rule library, a serial port message template fingerprint library is obtained.

6. The multi-source heterogeneous data edge processing method for field monitoring according to claim 1, characterized in that, Step S32 is followed by: S33: When a preset number of consecutive serial port messages fail to match the target template that meets the threshold condition, a new template candidate is generated based on the position of the repeating delimiter, fixed prefixes and suffixes, distribution of numerical segments, position of the suspected time field, and position of the suspected check digit. S34: Perform continuous verification on the new template candidate. When the continuous verification result meets the preset verification pass rate, write the new template candidate into the template fingerprint database. S35: After successfully matching the target template or verifying the new template candidate and writing it into the template fingerprint database, compare the corresponding template fingerprint matching result with the determined data source type. If the two are inconsistent, re-mark the corresponding data access link as pending judgment and re-determine the data source type.

7. The multi-source heterogeneous data edge processing method for field monitoring according to claim 2, characterized in that, Step S4 includes: S41: Determine the priority of time fields based on the data source type, and select target time fields sequentially from the parsing results according to the priority of time fields; among them, data communication protocol data prioritizes the sampling time field within the message, serial port data prioritizes the parsed time field, file transfer data prioritizes the file content time field or filename time field, and video data prioritizes the signaling time field, stream establishment time field, or pull-stream time field; when no available time field is found in the parsing results, estimate the standard event time based on the data arrival time and device reporting cycle; S42: Generate standard event times based on the target time field, and map the parsing results from different sources to the same time base to generate standardized data containing data source type identifiers, standard event times, and standard fields.

8. A multi-source heterogeneous data edge processing device for field monitoring, characterized in that, include: The multi-link access module is used to establish multiple independent data access links and receive raw data generated by different monitoring devices at the field monitoring site in parallel based on the data access links; The joint identification module is used to extract multi-dimensional features from the original data, calculate the identification confidence of multiple candidate data source types based on the multi-dimensional features, and determine the data source type corresponding to the original data according to the identification confidence. When determining the data source type based on the identification confidence, if the maximum value of the identification confidence does not reach the preset confirmation threshold, or the difference in identification confidence between two or more candidate data source types is less than the preset ambiguity threshold, subsequent features are collected and the determination is made again. The differential parsing module is used to call the corresponding parsing engine according to the data source type, and generate parsing results by performing differential parsing on the original data through the parsing engine; The standardization processing module is used to determine the priority of time fields based on the data source type, select the target time field from the parsing results according to the time field priority, perform unified event timeline mapping based on the target time field, and generate standardized data containing data source type identifier and standard event time. The cache scheduling module is used to write standardized data into the classified cache queue, determine the timeliness level based on the time difference between the standard event time and the current time, and perform hierarchical sending scheduling based on the timeliness level and network status. Differential parsing of serial port data includes: Establish a serial port message template fingerprint database, and match the received serial port messages with multiple templates in the template fingerprint database to form a candidate template set; Candidate templates in the candidate template set are scored based on field position stability, numerical type consistency, time field resolvability, and verification pass rate. The candidate template with the highest score that is higher than a preset threshold is selected as the target template. Field extraction is performed based on the target template to generate serial port data parsing results.

9. A computer-readable storage medium, characterized in that, The computer storage medium stores computer execution instructions, which, when executed by a processor, are used to implement the multi-source heterogeneous data edge processing method for field monitoring as described in any one of claims 1 to 7.

Images