A front-end embedded point information edge receiving method, system and related device
By using stateless edge receiving nodes, the data types of front-end embedded information are identified and parsed, structured data is generated and asynchronously persisted, solving the flexibility and stability issues of centralized receiving systems and achieving complete data access and traceable retention.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- JIALIAN PAYMENTS CO LTD
- Filing Date
- 2026-05-22
- Publication Date
- 2026-06-19
Smart Images

Figure CN122247979A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of information technology, and in particular to an edge receiving method, system and related apparatus for front-end embedded information. Background Technology
[0002] With the continuous development of internet applications, front-end event tracking information and program runtime logs are playing an increasingly prominent role in system monitoring, fault location, and data analysis. Multi-terminal applications generate a large amount of diverse and variable-field reported data during operation. This data is typically centrally accessed, parsed, and processed by receiving programs to support log analysis and system maintenance.
[0003] In existing technologies, a centralized receiving program is typically used to uniformly receive and process information reported from the front end. This type of receiving program is generally designed based on a fixed interface structure, predefined constraints on reported fields, and performs format conversion and log output after reception. Simultaneously, to ensure system stability, some solutions also introduce master-slave switching or load-sharing mechanisms, and achieve data transmission and storage through synchronous reporting or coupled processing flows. However, when faced with multiple data sources and dynamically changing reported content, the predefined constraints on field structures actually limit the system's flexibility, making it incompatible with dynamically changing data formats. Furthermore, the centralized processing architecture is prone to chain reactions when encountering traffic surges or localized anomalies in remote networks, ultimately leading to the overload and collapse of the entire receiving system, resulting in the loss of massive amounts of front-end reported data and the untraceability of historical records. Summary of the Invention
[0004] This application provides an edge receiving method, system, and related apparatus for front-end tracking information, which enables complete reception, standardized processing, and full-cycle traceable retention of front-end tracking information.
[0005] The first aspect of this application provides an edge receiving method for front-end embedded information. The edge receiving method is applied to a plurality of stateless edge receiving nodes, which are independent of each other. The edge receiving method includes:
[0006] Based on a pre-defined Web service framework, a network communication interface capable of providing services to the outside world is built, continuously listening to and receiving data reporting requests sent from different front-end carriers, and extracting the original reporting data carried in the data reporting requests. Identify the fields to be processed in the original reported data and obtain the data type corresponding to the fields to be processed; Perform the corresponding data parsing operation on the field to be processed according to the data type to obtain the parsed structured data; Basic field information is added to the structured data according to preset formatting rules to generate standard format data; The standard format data is persistently written to the local storage medium in log file format, and the subsequent processing of the log file adopts an asynchronous processing mechanism that is decoupled from the data parsing operation and the persistent writing operation.
[0007] Optionally, identifying the fields to be processed in the original reported data and obtaining the data type corresponding to the fields to be processed includes: Iterate through all fields in the original reported data as fields to be processed, and obtain the data type corresponding to each field to be processed; Based on the obtained data types, the fields to be processed are divided into basic data type fields and composite structure type fields. The basic data type fields are string, numeric, boolean, or one-dimensional array type fields, and the composite structure type fields are key-value pair type fields containing nested subfields.
[0008] Optionally, the step of performing a corresponding data parsing operation on the field to be processed according to the data type to obtain parsed structured data includes: When the field to be processed is a basic data type field, the original value of the field to be processed is directly extracted as the parsing result; When the field to be processed is a composite structure type field, the nested subfields within the field to be processed are traversed layer by layer, the type of each nested subfield is identified and the corresponding data parsing operation is performed, triggering a recursive parsing process until all nested subfields have been parsed, and all valid data of the field to be processed is extracted and integrated as the parsing result. The parsing results of all fields to be processed are integrated to obtain the parsed structured data.
[0009] Optionally, after traversing all fields in the original reported data as fields to be processed, the edge receiving method further includes: Identify whether a preset special processing field exists in the field to be processed; The step of performing a corresponding data parsing operation on the field to be processed according to the data type to obtain the parsed structured data further includes: For the identified special processing fields, the corresponding data parsing operation is performed according to the preset special processing rules to obtain the parsed structured data.
[0010] Optionally, the step of performing corresponding data parsing operations according to preset special processing rules for the identified special processing fields includes: When the special processing field is identified as an encrypted field, a preset symmetric decryption algorithm and an agreed decryption key are invoked to perform a decryption operation on the encrypted content of the special processing field, and the decrypted plaintext data is extracted as the parsing result. When the special processing field is identified as a non-encrypted field, the corresponding processing is performed according to the preset format conversion, data desensitization or data filtering rules, and the processed valid data is extracted as the parsing result.
[0011] Optionally, the step of supplementing the structured data with basic field information according to preset formatting rules to generate standard format data includes: Based on the data reporting request, extract the reporting time information, front-end terminal type information, and reporting IP information corresponding to the original reported data; According to the preset formatting rules, the reporting time information, the front-end type information, and the reporting IP information are written into the specified field positions of the structured data to generate standard format data.
[0012] Optionally, the subsequent processing of the log file employs an asynchronous processing mechanism decoupled from the data parsing operation and the persistent write operation, including: The log file is read by a separate log processing component, which then performs the data push operation to the remote server.
[0013] Optionally, for the stateless deployed edge receiving nodes, the edge receiving method further includes the following cluster scheduling and management steps: Real-time monitoring of traffic data from front-end data reporting requests, and dynamic adjustment of the number of edge receiving nodes deployed based on fluctuations in the traffic data; The system monitors the operational status of each edge receiving node in real time. When any edge receiving node is detected to have a failure, the system automatically restarts the faulty node and takes over the processing of the data reporting request.
[0014] A second aspect of this application provides an edge receiving system for front-end embedded information. The edge receiving system is applied to a plurality of stateless edge receiving nodes, which are independent of each other. The edge receiving system includes: The receiving unit is used to build a network communication interface that can provide services to the outside world based on a preset Web service framework, continuously listen to and receive data reporting requests sent from different front-end carriers, and extract the original reporting data carried in the data reporting requests. The acquisition unit is used to identify the fields to be processed in the original reported data and to obtain the data type corresponding to the fields to be processed. The parsing unit is used to perform corresponding data parsing operations on the field to be processed according to the data type, so as to obtain the parsed structured data; The generation unit is used to supplement basic field information into the structured data according to preset formatting rules to generate standard format data; The processing unit is used to persistently write the standard format data into the local storage medium in the format of a log file, and to use an asynchronous processing mechanism that is decoupled from the data parsing operation and the persistent writing operation for subsequent processing of the log file.
[0015] A third aspect of this application provides an edge receiving device for front-end embedded point information, the device comprising: Processor, memory, input / output units, and bus; The processor is connected to the memory, the input / output unit, and the bus; The memory stores a program, which the processor calls to execute the first aspect and any optional front-end embedding information edge receiving method in the first aspect.
[0016] The fourth aspect of this application provides a computer-readable storage medium storing a program that, when executed on a computer, performs the first aspect and any optional front-end embedding information edge receiving method in the first aspect.
[0017] As can be seen from the above technical solutions, this application has the following advantages: Stateless, independently deployed edge receiving nodes receive front-end data reporting requests. A single node completes the entire closed-loop processing of reported data, with nodes operating independently and adapting to distributed access of front-end data from multiple sources. By identifying the type of the fields to be processed and parsing them differently, the system flexibly adapts to dynamically changing reported field content and extracts valid information to generate structured data. Basic field information is supplemented according to preset formatting rules to generate data in a unified standard format. The standard format data is persistently written to local storage media in the form of log files to ensure the traceability of reported data. At the same time, an asynchronous mechanism decoupled from core parsing and persistence operations is used to process subsequent log processes, making log processing independent of the core business process in terms of timing and ensuring the continuous and stable operation of the core process.
[0018] This application can flexibly adapt to the dynamically changing data formats of multiple terminals, realize reliable control over the entire process of front-end data tracking, and complete the unified and standardized formatting process while ensuring the complete access of all data, thus ensuring the full lifecycle traceability and retention of historical data. Attached Figure Description
[0019] To more clearly illustrate the technical solutions in this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0020] Figure 1 A schematic flowchart of an embodiment of the edge receiving method for front-end embedded point information provided in this application; Figure 2 A schematic diagram of an embodiment of the edge receiving method for front-end embedded information provided in this application, which involves obtaining the data type of the field to be processed and performing the corresponding data parsing operation; Figure 3 A schematic flowchart of an embodiment of the edge receiving method for front-end embedded information provided in this application for the identification and parsing of special processing fields; Figure 4 A schematic diagram of an embodiment of the edge receiving method for front-end embedded information provided in this application, which supplements basic field information into structured data; Figure 5 A schematic flowchart of an embodiment of the edge receiving method for front-end embedded point information provided in this application, concerning cluster scheduling and management; Figure 6 A schematic diagram of an embodiment of the edge receiving method for front-end embedded information provided in this application, showing the system architecture and data flow throughout the entire link. Figure 7 A schematic diagram of an embodiment of the edge receiving system for front-end embedded point information provided in this application; Figure 8 A schematic diagram of an embodiment of the edge receiving device for front-end embedded point information provided in this application. Detailed Implementation
[0021] This application provides an edge receiving method, system, and related apparatus for front-end tracking information, which enables complete reception, standardized processing, and full-cycle traceable retention of front-end tracking information.
[0022] It should be noted that the following embodiments are only used to illustrate the technical solution of this application in detail. The specific execution subjects, implementation carriers and supporting components mentioned are all preferred implementation methods and do not constitute any limitation on the protection scope of this application. The edge receiving method for front-end embedded information protected by this application is executed by a stateless edge receiving node with network communication, data processing and local storage capabilities. The specific hardware form, development language, operating framework, deployment method, supporting functional components and front-end data source disclosed in the embodiments are only specific implementation examples. Those skilled in the art can choose any equivalent solution that can achieve the same function according to actual needs, and all related substitutions and modifications fall within the protection scope of this application.
[0023] Please see Figure 1 , Figure 1 An embodiment of the edge receiving method for front-end embedded point information provided in this application includes: S101. Based on the preset Web service framework, build a network communication interface that can provide services to the outside world, continuously listen to and receive data reporting requests sent from different front-end carriers, and extract the original reporting data carried in the data reporting requests. In this embodiment, the edge receiving node builds a network communication interface based on a preset Web service framework, which supports application-layer HTTP and HTTPS protocols and can receive data reporting requests sent from different front-end carriers via POST requests. Front-end carriers include web applications, mobile applications, and mini-program applications. After completing the network handshake and connection with the front-end carrier, the edge receiving node continuously listens for and receives complete data reporting request messages sent by the front-end carrier. To ensure the data security of the edge receiving node, it triggers a request verification mechanism for the received data reporting requests, performs validity checks on the data reporting requests, determines whether the request payload is empty and whether it triggers preset malicious request interception rules, and directly discards and filters data reporting request messages identified as invalid.
[0024] For data reporting requests that pass verification, the edge receiving node performs basic protocol parsing on the data reporting request message, stripping away the protocol header information and transport layer encapsulation content to obtain the corresponding message body content. Further, the edge receiving node performs complete reading and encoding format conversion on the obtained message body content, identifying the encoding format corresponding to the message body content and uniformly converting message body content with different encoding formats into a basic data format that the edge receiving node can directly process. From the converted message body content, the complete original reporting data reported by the front-end carrier is extracted. This original reporting data includes business data generated by front-end tracking behavior and error data generated by front-end program runtime anomalies.
[0025] It should be noted that during the extraction of raw reported data, the edge receiving node must completely preserve the field structure and content information of the raw reported data. It must not delete or modify the number, names, or content of fields in the raw reported data, ensuring that the extracted raw reported data is completely consistent with the reported content sent by the front-end carrier. After extraction, the edge receiving node writes the raw reported data into a pre-allocated temporary buffer for temporary storage, completing the secure reception and caching phase of the front-end data. The raw reported data temporarily stored in the temporary buffer serves as the data input source for subsequent dynamic data parsing operations.
[0026] S102. Identify the fields to be processed in the original reported data and obtain the data type corresponding to the fields to be processed; Considering that existing conventional receiving and processing methods require the receiving program and the front end to pre-agree on the names, quantities, and format rules of all reported fields, when adjustments to the front end's business logic lead to the addition, modification, or deletion of reported fields, the field verification and recognition logic of the receiving program needs to be modified synchronously. This results in long adaptation cycles, high maintenance costs, and a high risk of data loss due to mismatched field agreements. In this embodiment, the edge receiving node adopts a flexible dynamic parsing mechanism when performing field recognition, i.e., it does not set pre-constraints on the field names and quantities reported by the front end. Specifically, after the edge receiving node completes the extraction of the original reported data and confirms its integrity, it directly identifies the fields to be processed contained in the original reported data. At this time, it is necessary to retain the original identifier and content information of each field to be processed. Subsequently, for each field to be processed, the edge receiving node calls the preset type judgment logic to identify and analyze the field content, dynamically determining the data type corresponding to each field content in the current operating environment. The edge receiving node associates and maps the obtained identifier information of the fields to be processed with the corresponding data type attributes and stores it in the running memory, providing a matching basis for subsequent data parsing operations.
[0027] The unconstrained identification method used in this embodiment can directly adapt to the dynamic adjustment of front-end reported fields. When the reported fields are expanded or deleted, the edge receiving nodes do not need to synchronously modify the core processing logic program, thereby significantly reducing the adaptation cost of the receiving program caused by front-end business changes. At the same time, this mechanism can ensure complete coverage and acceptance of all valid information in the original reported data, fundamentally avoiding the omission of valid data due to pre-defined field constraints.
[0028] S103. Perform the corresponding data parsing operation on the field to be processed according to the data type to obtain the parsed structured data; After the edge receiving node completes the association and retention of the identification information and data type of all fields to be processed, it sequentially reads the identification information and corresponding data type of each field to be processed from the associated and retained information. Based on the read data type, it calls the preset parsing function that matches the data type to parse the original content of each field to be processed and converts the parsed content into structured data that can be uniformly processed within the edge receiving node. This structured data contains field identification information and parsed standardized content information.
[0029] By adopting the above approach, appropriate parsing logic can be used for fields of different data types, avoiding parsing errors or incomplete parsing caused by a uniform parsing logic. Existing conventional receiving and processing methods may use a single text parsing logic for all fields, failing to accurately restore the original meaning of non-text fields such as numeric and Boolean values. They may also fail to adequately parse fields with complex structures, affecting the accuracy of subsequent data processing. This embodiment, using a data type-matching parsing function, can accurately restore the original meaning of various field contents, ensuring the accuracy and completeness of the structured data after parsing.
[0030] S104. Supplement basic field information into the structured data according to the preset formatting rules to generate standard format data; After receiving the parsed structured data, the edge receiving node reads the pre-configured formatting rules and retrieves the basic field information related to the current data reporting request according to the formatting rules. It then supplements the structured data with this basic field information at the designated location. After supplementing the basic field information, the edge receiving node performs a unified format conversion and data alignment process on the supplemented structured data to generate standard format data. This unified formatting method can achieve structural uniformity and basic information completion for all reported data without modifying the original reported business data. It eliminates the need to develop separate adaptation logic for different reporting scenarios, significantly reducing the adaptation cost of subsequent data processing stages. Simultaneously, by supplementing the basic field information, it ensures that each reported data has a complete traceable identifier, providing comprehensive information support for subsequent data tracing and problem investigation.
[0031] S105. Standard format data is persistently written to the local storage medium in log file format, and the subsequent processing of the log file adopts an asynchronous processing mechanism that is decoupled from the data parsing operation and the persistent writing operation.
[0032] After generating standard-format data, the edge receiving node converts the data into log file format and persists it to its local storage medium. For log files already written to local storage, subsequent data processing and flow employ an independent asynchronous mechanism. This asynchronous mechanism is independent of the data parsing and persistence operations performed by the edge receiving node; that is, the subsequent processing status of the log file will not interfere with or block the edge receiving node from receiving and processing new data reporting requests. Even if network latency or execution errors occur in subsequent log flow stages, it will not affect the operational status of the edge receiving node, ensuring its throughput and overall stability in handling high-concurrency front-end reporting scenarios.
[0033] In some specific embodiments, this asynchronous processing mechanism can read log files through a separate log processing component, which then performs the data push operation to the remote server. That is, subsequent processing of log files is completed by an independently deployed log processing component. This component runs independently of the core processing program of the edge receiving node, and their running states and execution sequences do not interfere with each other. The log processing component continuously monitors the update status of log files in the local storage medium according to preset reading rules. When a newly written log file is detected, it reads the complete content of the corresponding log file. During the reading process, it does not modify the original content or storage attributes of the log file, nor does it occupy the computational and I / O resources of the edge receiving node's core processing flow. After completing the complete reading of the log file content, the log processing component establishes a stable network connection with the remote server according to preset transmission protocols and rules, and pushes the read log file content to the remote server, completing the remote synchronization operation of the log data.
[0034] In some specific embodiments, after the edge receiving node persistently writes standard format data to the local storage medium in log file format, it can also perform log rotation management and automatic expiration cleanup strategies on the generated log files. Specifically, the edge receiving node manages the log files by using either time-based or file-size-based rotation according to preset log management rules: when the writing time of a local log file reaches a preset time threshold, or the storage capacity of the log file reaches a preset size threshold, it automatically performs an archiving and splitting operation on the currently written log file, generating independent archived log files and creating new log files to continue receiving data, thus avoiding the impact of excessively large single log file sizes on reading and transmission efficiency. Simultaneously, the edge receiving node automatically identifies and deletes old archived log files that have exceeded their retention period in the local storage medium according to preset log retention periods and expiration cleanup rules, periodically releasing storage space occupied by redundant logs, ensuring that local storage resources remain stable and available, and preventing storage resource exhaustion due to the infinite accumulation of log files. For example, edge receiving nodes can be configured to split log files hourly, automatically sealing the current log and creating a new log file to continue writing every hour; or configured to automatically perform a split operation when the log file reaches 512MB. Simultaneously, a 7-day log retention period can be configured, with the system periodically scanning the local storage medium daily and automatically deleting old log files older than 7 days. This ensures efficient utilization and stable maintenance of local storage resources while guaranteeing that recent logs can be read and pushed normally.
[0035] In this embodiment, stateless, independently deployed edge receiving nodes receive front-end data reporting requests. A single node completes the entire closed-loop processing of the reported data, and the operation between nodes does not interfere with each other, adapting to distributed access of front-end data from multiple sources. By identifying the type of the fields to be processed and parsing them differently, the system flexibly adapts to dynamically changing reported field content and extracts effective information to generate structured data. Basic field information is supplemented according to preset formatting rules to generate data in a unified standard format. The standard format data is persistently written to the local storage medium in the form of log files to ensure the traceability of the reported data. At the same time, an asynchronous mechanism decoupled from the core parsing and persistence operations is used to process the subsequent log processes, making log processing independent of the core business process in terms of timing, and ensuring the continuous and stable operation of the core process.
[0036] This application can flexibly adapt to the dynamically changing data formats of multiple terminals, realize reliable control over the entire process of front-end data tracking, and complete the unified and standardized formatting process while ensuring the complete access of all data, thus ensuring the full lifecycle traceability and retention of historical data.
[0037] Please see Figure 2According to some embodiments of the present invention, in step S102, the fields to be processed in the original reported data are identified, and the data type corresponding to the fields to be processed is obtained. In step 103, the corresponding data parsing operation is performed on the fields to be processed according to the data type to obtain the parsed structured data, including but not limited to the following: S201. Traverse all fields in the original reported data as fields to be processed, and obtain the data type corresponding to each field to be processed; The edge receiving node converts the extracted raw reported data from transmission or text format into a memory data object that can be directly manipulated by the current operating environment. During the conversion process, all field hierarchy relationships and content information of the original reported data must be completely preserved without any deletion or modification. After the conversion is completed, the edge receiving node traverses the memory data object, starting from the root node and locating each independent field node contained within the memory data object according to a preset traversal order. Upon locating each field node, the original identification information of that field node is extracted, and the field node is marked as a field to be processed and added to the set of fields to be processed.
[0038] For each field in the set of fields to be processed, the edge receiving node calls the type checking interface provided by the underlying runtime environment, or accesses the actual data value carried by the field to be processed through object attribute reading logic. It analyzes the storage characteristics, encoding characteristics, and structural characteristics of the actual data value, and matches the predefined data type identifier in the runtime environment based on the analysis results, thereby accurately obtaining the data type corresponding to each field to be processed in the current runtime environment.
[0039] For example, regarding the pending field in the raw reported data from front-end event tracking that carries the time of a user click event, the edge receiving node uses the `typeof` type check interface at the runtime environment's underlying layer to read that the actual data value of this field is stored as a double-precision floating-point number, its encoding characteristics conform to the encoding specifications of numeric types, and it has no nested structure. This matches the predefined numeric type identifier in the runtime environment, thus determining that the data type of the pending field is numeric. Similarly, regarding the pending field carrying the identifier of the user's business line, the edge receiving node reads that the actual data value of this field is stored as a UTF-8 encoded character sequence, has no nested structure, and matches the predefined string type identifier in the runtime environment, thus determining that the data type of the pending field is string. For the pending field carrying whether the user has completed login... For fields, the edge receiving node reads that the actual data value of the field has only two fixed logical states, and matches it with the predefined Boolean value type identifier in the operating environment to determine that the data type of the field to be processed is Boolean. For fields carrying the list of product IDs associated with the user's current operation, the edge receiving node reads that the actual data value of the field has the storage characteristic of a single-dimensional ordered data set, and matches it with the predefined one-dimensional array type identifier in the operating environment to determine that the data type of the field to be processed is one-dimensional array. For fields carrying user device information, the edge receiving node reads that the actual data value of the field has the storage characteristic of a key-value pair mapping structure containing multiple nested subfields, and matches it with the predefined object type identifier in the operating environment to determine that the data type of the field to be processed is a key-value pair type containing nested subfields. The edge receiving node associates and stores the original identifier information of each field to be processed with the acquired data type, providing a complete basis for subsequent classification operations.
[0040] S202. Based on the obtained data types, the fields to be processed are divided into basic data type fields and composite structure type fields. Basic data type fields are string, numeric, boolean, or one-dimensional array type fields, and composite structure type fields are key-value pair type fields containing nested subfields. After obtaining the data types of all fields to be processed, the edge receiving node classifies them into two categories based on their data types: basic data type fields and composite structure type fields. The specific classification criteria are as follows: when the data type of a field to be processed is identified as a string, number, boolean value, or one-dimensional array, it is classified as a basic data type field because this type of data exhibits a simple, flat storage structure; when the data type of a field to be processed is identified as a key-value pair type containing nested subfields, it is classified as a composite structure type field because this type of data possesses multi-level tree mapping or object nesting characteristics. After completing the classification judgment of the current field to be processed, the edge receiving node associates and stores the field's identification information, data type, and classification result, and then continues processing the next field to be processed until all fields in the set of fields to be processed have been classified.
[0041] S203. When the field to be processed is a basic data type field, directly extract the original value of the field to be processed as the parsing result; After completing the classification and routing operations for all fields to be processed, the edge receiving node iterates through the set of fields to be processed, reading the classification result of each field sequentially. For the currently read field, if the classification result is a basic data type field, the edge receiving node directly accesses the original value of that field in the memory data object, without performing any additional format conversion, content modification, or structural adjustment operations, fully preserving the storage characteristics, encoding characteristics, and content information of the original value. The edge receiving node associates the extracted original value with the original identification information of the field to be processed, using it as the parsing result of that field.
[0042] For example, for a numeric field representing the time of a user's click event, the edge receiving node directly reads the original double-precision floating-point value stored in memory without any precision adjustment or format conversion; for a string field representing the identifier of the user's business line, it directly reads the original UTF-8 encoded character sequence value stored in the field; for a boolean field representing whether the user has completed login, it directly reads the original true or false logical state value stored in the field; and for a one-dimensional array field representing the list of product IDs associated with the user's current operation, it directly reads the original value of the single-dimensional ordered data set stored in the field.
[0043] S204. When the field to be processed is a composite structure type field, traverse the nested subfields within the field to be processed layer by layer, identify the type of each nested subfield and execute the corresponding data parsing operation, trigger the recursive parsing process until all nested subfields have been parsed, extract and integrate all valid data of the field to be processed as the parsing result. For the currently read field to be processed, if the classification result is a composite structure type field, the edge receiving node initiates a recursive parsing process. The edge receiving node first accesses the original key-value pair structure carried by the composite structure type field in the memory data object. Starting from the outermost layer of the original key-value pair structure, it locates each nested sub-field node contained within the composite structure type field according to a preset traversal order. For the currently located nested sub-field node, the edge receiving node calls the runtime environment's underlying type checking interface or object attribute reading logic to identify the data type of the nested sub-field node and determine whether the nested sub-field node belongs to a basic data type field or a next-level composite structure type field.
[0044] If the data is determined to be a basic data type field, the edge receiving node directly extracts the original value of the nested sub-field node without performing any additional format conversion, content modification, or structural adjustment operations. It fully preserves the storage characteristics, encoding characteristics, and content information of the original value and associates the extracted original value with the original identification information of the nested sub-field node as the parsing result of the nested sub-field node.
[0045] If the field is determined to be a composite structure type field of the next level, the edge receiving node continues to trigger the recursive parsing process, repeatedly executing the location, type identification, classification judgment and corresponding parsing operations of the nested sub-field nodes, until all nested sub-fields in the next level composite structure type field have been parsed. For example, for a composite structure field carrying user device information, which is a key-value pair structure containing multiple nested subfields in the memory data object, the edge receiving node first accesses the outermost layer of this key-value pair structure and locates the nested subfield nodes such as device model, operating system version, and screen resolution one by one according to the preset traversal order. For the device model nested subfield node, the edge receiving node identifies its data type as string, which belongs to the basic data type field. It directly reads the original value of the UTF-8 encoded character sequence stored in this subfield and associates the original value with the original identification information of the device model as the parsing result. For the operating system version nested subfield node, it identifies its data type as string and also directly extracts the original value as the parsing result. For the screen resolution nested subfield node, it identifies its data type as a one-dimensional array and directly extracts the original value of the ordered data set containing width and height values stored in this subfield as the parsing result. If the current composite structure type's field to be processed contains nested subfields of the next level of composite structure type, such as a subfield carrying user network environment information, and this subfield further contains nested subfields such as network type and signal strength, the edge receiving node will trigger a recursive parsing process, repeatedly executing the location, type identification, classification judgment, and corresponding parsing operations of the nested subfield nodes, until all nested subfields within the network environment information subfield have been parsed.
[0046] After parsing all nested subfields within the current composite structure type field, the edge receiving node integrates and arranges the parsing results of all nested subfield nodes according to the hierarchical relationship and arrangement order of the original key-value pair structure, completely preserving the hierarchical organization form of the original composite structure type field. It then associates all the integrated valid data with the original identification information of the composite structure type field as the parsing result of the composite structure type field.
[0047] S205. Integrate the parsing results of all fields to be processed to obtain the parsed structured data.
[0048] After completing the classification and parsing of all fields in the set of fields to be processed, the edge receiving node performs an integrity check on the set of fields to be processed and the corresponding parsing result set. It verifies that the original identifier information of each field to be processed has a matching parsing result, confirming that all root-level outer fields and all nested subfields within all composite structure types in the original reported data have been parsed and corresponding parsing results have been generated, with no fields missing. After the integrity check passes, the edge receiving node uses the root node of the memory data object converted from the original reported data as a reference, and sequentially reads the original identifier information and corresponding parsing result of each field to be processed according to the inherent field arrangement order and hierarchical organization of the original reported data. For basic data type fields at the root level, the edge receiving nodes directly write the original identifier information and parsing results of the fields into the corresponding positions of the root-level nodes in the structured data. For composite structure type fields at the root level, the edge receiving nodes, according to the original nesting hierarchy of the field, write the original identifier information and parsing results of all nested subfields under the field into the corresponding hierarchical nodes of the structured data, layer by layer, thus completely preserving the field hierarchy structure and field arrangement order of the original reported data. After completing the hierarchical writing and integration of all field parsing results, the edge receiving nodes can generate complete parsed structured data that fully corresponds to the structure of the original reported data.
[0049] In this embodiment, by unconstrained traversal and identification of all fields, coupled with differentiated parsing logic for basic data types and composite structure types, various dynamic adjustments to front-end business can be adapted without pre-determining relevant rules for the reported fields. Furthermore, the processing logic in this embodiment can completely preserve the original content and hierarchical structure of the reported data, avoiding incomplete parsing of nested structures or distortion of field content. The resulting structured data provides a stable and reliable basic data source for subsequent processing.
[0050] Please see Figure 3According to some embodiments of the present invention, the edge receiving method may further include processing logic for special processing fields. That is, during the execution of step S201, which traverses all fields in the original reported data and determines them as fields to be processed, a process for identifying and parsing special processing fields is simultaneously embedded, including but not limited to the following steps: S301. Identify whether there are preset special processing fields in the fields to be processed; When the edge receiving node performs a full field traversal of the raw reported data, it simultaneously retrieves a pre-configured and permanently stored special processing field configuration list to identify whether any pre-defined special processing fields exist among the fields to be processed. This configuration list pre-registers sensitive and special business fields that require customized control in front-end tracking reporting scenarios. The edge receiving node uses field name and unique field identifier as the matching criteria to traverse each field to be processed sequentially, comparing and verifying the identifier information of the field to be processed against the pre-defined special processing field configuration list item by item. Through precise matching, it determines whether the current field to be processed belongs to the agreed special processing fields, while distinguishing and marking ordinary business fields from special processing fields, thus completely screening out all special processing fields contained in the reported data.
[0051] For example, in the business scenario of front-end tracking and reporting, the preset special processing fields may include sensitive privacy fields such as user mobile phone number, user ID card number, login account, and device unique identifier. When the edge receiving node traverses the fields to be processed one by one, once it matches the above field identifier, it is determined that the preset special processing field has been identified.
[0052] S302. For the identified special processing fields, perform the corresponding data parsing operation according to the preset special processing rules to obtain the parsed structured data.
[0053] After the edge receiving node completes the identification and marking of special processing fields, it no longer uses the conventional processing logic of directly retrieving values from basic fields and recursively parsing composite fields. Instead, it retrieves the preset special processing rules bound to the identified special processing field type. Based on the customized processing logic agreed in the rules, such as data desensitization, encryption / decryption, character truncation, and format normalization, it performs corresponding data parsing operations on the original field content of the special processing field. The processing process keeps the original field identifier unchanged, and only performs compliant conversion and standardization processing on the actual field value according to the rules.
[0054] In some specific embodiments, after identifying that the field to be processed belongs to a preset special processing field, the edge receiving node further distinguishes the special processing field into two categories: encrypted fields and unencrypted fields, and performs differentiated parsing processing accordingly. Specifically, when the special processing field is identified as an encrypted field, a preset symmetric decryption algorithm and an agreed decryption key are invoked to decrypt the encrypted content of the special processing field, and the decrypted plaintext data is extracted as the parsing result; when the special processing field is identified as an unencrypted field, corresponding processing is performed according to preset format conversion, data desensitization, or data filtering rules, and the processed valid data is extracted as the parsing result.
[0055] For example, when front-end data points report data, sensitive fields such as user unique identifiers and privacy identity numbers are transmitted in encrypted form. After the edge receiving node matches the identifier of this encrypted field, it calls a pre-defined symmetric decryption algorithm and key to decrypt it, obtaining readable plaintext information, which serves as the parsing result for that field. When the specially processed field is identified as a non-encrypted field, no decryption operation is required. Instead, corresponding customized processing is performed according to pre-configured format conversion rules, data anonymization rules, or data filtering rules. According to the rules, the original content of the field is format-normalized, sensitive characters are hidden or anonymized, or invalid and redundant field content is filtered out, retaining compliant and usable valid information. The valid data processed by the rules is extracted as the parsing result for the current non-encrypted specially processed field. For example, when matching non-encrypted privacy fields such as user mobile phone numbers and permanent addresses, the middle characters are hidden and replaced according to the data anonymization rules; when matching time format or regional code fields, they are uniformly normalized to a standard format according to the format conversion rules; when matching fields containing invalid garbled characters or redundant placeholder content, invalid content is removed according to the data filtering rules, retaining only valid business data to complete the corresponding parsing processing.
[0056] After completing the customized parsing of all special processing fields, the parsing results of the special processing fields (S302) are integrated with the regular parsing results of the other ordinary basic type fields and ordinary composite structure type fields (S203, S204) in the original hierarchical order, and finally generated well-organized and unified parsed structured data that takes into account both business availability and data security.
[0057] In this embodiment, by identifying preset special processing fields and distinguishing between encrypted and unencrypted fields using differentiated rules, encrypted fields are symmetrically decrypted to restore plaintext, while unencrypted fields are desensitized, format converted, and data filtered. This not only ensures the security of sensitive data transmission and parsing of front-end embedded points and avoids the risk of privacy information leakage, but also unifies the data format and content specifications of special fields, improving the overall compliance and usability of data parsing.
[0058] Please see Figure 4According to some embodiments of the present invention, in step S104, basic field information is added to the structured data according to preset formatting rules to generate standard format data, including but not limited to the following: S401. Based on the data reporting request, extract the reporting time information, front-end terminal type information, and reporting IP information corresponding to the original reported data; After parsing all fields to be processed and obtaining structured data, the edge receiving node extracts basic association information matching the original reported data based on the request message and network connection context information of this data reporting request, ensuring that each reported data has a complete and traceable identifier. Specifically, the edge receiving node records the system standard time (formatted as YYYY-MM-DDHH:MM:SS) of the current receiving and processing of the reporting request as the reporting time information; it identifies and extracts the device type and application affiliation information corresponding to the front-end carrier from the client identifier fields such as User-Agent and AppID carried in the request message header, as the front-end terminal type information; and it parses and extracts the network address that initiated the reporting request from the link parameters and request source address of this network connection, as the reporting IP information. All three types of information can be automatically captured from the context of the reporting request without relying on the internal fields of the original reported data, ensuring that the associated information is complete and traceable, and independent of the original business data.
[0059] For example, if a front-end user initiates a data reporting request through a shopping app on their mobile phone, the edge receiving node records the system time "2026-04-29 15:30:45" for receiving the request, as the reporting time information; it identifies the device type as "iOS phone" and the application as "a shopping app V5.8.0" from the User-Agent field in the request header, and integrates these as the front-end client information; it also resolves the IP address "112.XX.XX.89" from the request source address of the network connection, as the reporting IP information, thus completing the extraction of basic information.
[0060] S402. According to the preset formatting rules, the reporting time information, front-end terminal type information, and reporting IP information are written into the specified field positions of the structured data to generate standard format data.
[0061] After acquiring the reporting time information, front-end endpoint information, and reporting IP information, the edge receiving node reads the system's pre-configured formatting rules. These rules pre-define the fixed arrangement positions, field naming conventions, and data writing formats for three types of basic supplementary fields, ensuring that all reported data has a unified structure and is standardized and reusable. Without altering the content, hierarchical relationships, or arrangement order of the original structured data's internal business fields, the edge receiving node sequentially inserts the extracted reporting time information, front-end endpoint information, and reporting IP information into the structured data according to the fixed field positions specified by the formatting rules, ensuring that the field format matches the rule requirements during the insertion process. After completing the standardized writing of the basic fields, the overall data structure is unified and standardized, the traceability information is complete, and finally, standard format data with a well-structured structure and complete fields is generated.
[0062] In this embodiment, three types of basic traceability information—reporting time, front-end terminal type, and reporting IP—are automatically extracted from the data reporting request context and written into the specified location of the structured data according to preset formatting rules. This eliminates the need to rely on supplementary information from the internal fields of the original reported data, achieving unified formatting and organization of all data points, improving the data traceability dimension, and facilitating subsequent log retrieval, problem investigation, and statistical analysis of terminal sources.
[0063] Please see Figure 5 According to some embodiments of the present invention, for a plurality of stateless deployed edge receiving nodes, the edge receiving method further includes the following cluster scheduling and management steps, including but not limited to the following: S501: Monitor the traffic data of front-end data reporting requests in real time, and dynamically adjust the number of edge receiving nodes deployed based on the fluctuation of traffic data; The cluster management module continuously collects traffic metrics for all data reporting requests from the front end in real time, including request concurrency, request frequency, and total requests per unit time, monitoring traffic peaks, troughs, and instantaneous fluctuations. Combined with preset traffic thresholds and scaling-up / down strategies, when the traffic of front-end data reporting requests continuously increases and exceeds the current capacity limit of the edge receiving node cluster, it automatically adds several stateless edge receiving nodes to achieve horizontal scaling of the cluster and alleviate the pressure on reporting request processing. When the traffic continuously declines during off-peak periods and cluster resources are idle and redundant, it automatically reduces the number of deployed edge receiving nodes to release idle computing and storage resources, achieving elastic scaling-up of the cluster. Because the edge receiving nodes adopt a stateless deployment mode, no private session data is stored between nodes. Adding or removing nodes does not affect the normal processing of existing reporting requests, enabling seamless dynamic scheduling based on traffic fluctuations. For example, during major promotional events on e-commerce platforms, the concurrent requests for front-end user page browsing and product click tracking surged. The cluster management module detected that the traffic far exceeded the daily threshold and automatically added multiple edge receiving nodes to share the request processing. During the off-peak hours in the early morning, the traffic of tracking reports dropped significantly, and the system automatically took some idle edge receiving nodes offline to save cluster deployment resources.
[0064] S502: Monitor the operating status of each edge receiving node in real time. When any edge receiving node is detected to have an operational failure, automatically restart the faulty node and take over the processing of data reporting requests.
[0065] The cluster management module can also continuously monitor the process liveness status, CPU load, memory usage, and request processing response status of each edge receiving node within the cluster, allowing for real-time monitoring of the health and operational status of each node. When an edge receiving node is detected to have a process freeze, response timeout, or abnormal crash, it is immediately determined to be a failure. The cluster management module first performs an automatic restart and recovery operation on the failed edge receiving node. Simultaneously, it automatically reassigns data reporting requests originally allocated to the failed node to other normally functioning stateless edge receiving nodes within the cluster. These functioning nodes seamlessly take over the entire process of receiving, parsing, formatting, and persisting logs for the corresponding data reporting requests. Once the failed node restarts and returns to a normal, usable state, it is reinstated into the cluster scheduling list to participate in request load balancing. This achieves seamless reporting for the user side and uninterrupted data processing, ensuring a stable and reliable overall data reception link.
[0066] In this embodiment, for the stateless deployment of the edge receiving node cluster, the number of nodes deployed is dynamically adjusted according to the fluctuation of the reported request traffic to achieve elastic scaling and reasonable scheduling of system resources; at the same time, the node operating status is monitored in real time, and the node is automatically restarted and seamlessly takes over the business processing in case of failure, which effectively improves the load adaptability, resource utilization and fault self-healing capability of the edge receiving cluster, and ensures the continuous and stable uninterrupted front-end data reporting and processing business.
[0067] Please see Figure 6 , Figure 6 This is a schematic diagram illustrating an embodiment of the system architecture and end-to-end data flow of a front-end embedded information edge receiving method. Figure 6 The functional components disclosed herein are examples of preferred implementations of the core process of this embodiment and do not constitute any limitation on the scope of protection of this application. Figure 6 As shown, data reporting requests initiated by the frontend first pass through the nginx module, which handles load balancing, distributing received requests to the stateless edge node cluster deployed in the backend according to preset load rules. The edge node cluster contains multiple independently operating edge receiving nodes, supporting horizontal scaling to adjust the deployment number. Each edge receiving node can independently complete the entire closed-loop processing of receiving reporting requests, extracting raw reporting data, identifying fields and obtaining types, parsing data, formatting, and writing logs, with no interference between nodes. After completing the core processing, the edge receiving node writes the generated standard-format data to local storage in log text format, achieving persistent data retention. The independently deployed ilogtail component, decoupled from the core processing flow of the edge receiving nodes, continuously monitors the update status of log files in the local storage medium, reads the complete content of persistently written log files, and the reading process does not consume the computational and I / O resources of the edge receiving node's core processing flow. After reading the log content, the ilogtail component sends the read log data to the remote SLS log service, completing remote synchronization and storage of the log data.
[0068] The edge receiving system for front-end embedded data provided in this application is described in detail below. Please refer to [link / reference]. Figure 7 , Figure 7 Another embodiment of the edge receiving system for front-end data tracking provided in this application is an edge receiving system applied to several stateless edge receiving nodes, which are independent of each other. The edge receiving system includes: The receiving unit 701 is used to build a network communication interface that can provide services to the outside world based on a preset Web service framework, continuously listen to and receive data reporting requests sent from different front-end carriers, and extract the original reporting data carried in the data reporting requests. The acquisition unit 702 is used to identify the fields to be processed in the original reported data and obtain the data type corresponding to the fields to be processed. Parsing unit 703 is used to perform corresponding data parsing operations on the field to be processed according to the data type, and obtain the parsed structured data; The generation unit 704 is used to supplement basic field information into structured data according to preset formatting rules and generate standard format data. The processing unit 705 is used to persistently write standard format data into the local storage medium in log file format, and the subsequent processing of the log file adopts an asynchronous processing mechanism that is decoupled from the data parsing operation and the persistent writing operation.
[0069] In this embodiment, the functions of each unit are the same as described above. Figures 1 to 5 The steps in the method embodiments shown correspond to those in the examples, and will not be repeated here.
[0070] This application also provides an edge receiving device for front-end embedded point information; please refer to [link to relevant documentation]. Figure 8 , Figure 8 An embodiment of the edge receiving device for front-end embedded point information provided in this application includes: Processor 801, memory 802, input / output unit 803, bus 804; The processor 801 is connected to the memory 802, the input / output unit 803, and the bus 804; The memory 802 stores a program, and the processor 801 calls the program to execute any of the above-mentioned edge receiving methods for front-end embedded point information.
[0071] This application also relates to a computer-readable storage medium on which a program is stored, which, when run on a computer, causes the computer to execute any of the above-mentioned edge receiving methods for front-end embedded information.
[0072] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.
[0073] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces, or indirect coupling or communication connection between apparatuses or units, and may be electrical, mechanical, or other forms.
[0074] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.
[0075] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.
[0076] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.
Claims
1. An edge receiving method for front-end embedded point information, characterized in that, The edge receiving method is applied to several stateless edge receiving nodes, which are independent of each other. The edge receiving method includes: Based on a pre-defined Web service framework, a network communication interface capable of providing services to the outside world is built, continuously listening to and receiving data reporting requests sent from different front-end carriers, and extracting the original reporting data carried in the data reporting requests. Identify the fields to be processed in the original reported data and obtain the data type corresponding to the fields to be processed; Perform the corresponding data parsing operation on the field to be processed according to the data type to obtain the parsed structured data; Basic field information is added to the structured data according to preset formatting rules to generate standard format data; The standard format data is persistently written to the local storage medium in log file format, and the subsequent processing of the log file adopts an asynchronous processing mechanism that is decoupled from the data parsing operation and the persistent writing operation. For the stateless deployment of several edge receiving nodes, the edge receiving method further includes the following cluster scheduling and management steps: Real-time monitoring of traffic data from front-end data reporting requests, and dynamic adjustment of the number of edge receiving nodes deployed based on fluctuations in the traffic data; The system monitors the operational status of each edge receiving node in real time. When any edge receiving node is detected to have a failure, the system automatically restarts the faulty node and takes over the processing of the data reporting request.
2. The edge receiving method according to claim 1, characterized in that, The step of identifying the fields to be processed in the original reported data and obtaining the data type corresponding to the fields to be processed includes: Iterate through all fields in the original reported data as fields to be processed, and obtain the data type corresponding to each field to be processed; Based on the obtained data types, the fields to be processed are divided into basic data type fields and composite structure type fields. The basic data type fields are string, numeric, boolean, or one-dimensional array type fields, and the composite structure type fields are key-value pair type fields containing nested subfields.
3. The edge receiving method according to claim 2, characterized in that, The step of performing corresponding data parsing operations on the field to be processed according to the data type to obtain parsed structured data includes: When the field to be processed is a basic data type field, the original value of the field to be processed is directly extracted as the parsing result; When the field to be processed is a composite structure type field, the nested subfields within the field to be processed are traversed layer by layer, the type of each nested subfield is identified and the corresponding data parsing operation is performed, triggering a recursive parsing process until all nested subfields have been parsed, and all valid data of the field to be processed is extracted and integrated as the parsing result. The parsing results of all fields to be processed are integrated to obtain the parsed structured data.
4. The edge receiving method according to claim 2, characterized in that, After traversing all fields in the original reported data as fields to be processed, the edge receiving method further includes: Identify whether a preset special processing field exists in the field to be processed; The step of performing a corresponding data parsing operation on the field to be processed according to the data type to obtain the parsed structured data further includes: For the identified special processing fields, the corresponding data parsing operation is performed according to the preset special processing rules to obtain the parsed structured data.
5. The edge receiving method according to claim 4, characterized in that, The process of performing corresponding data parsing operations on the identified special processing fields according to preset special processing rules includes: When the special processing field is identified as an encrypted field, a preset symmetric decryption algorithm and an agreed decryption key are invoked to perform a decryption operation on the encrypted content of the special processing field, and the decrypted plaintext data is extracted as the parsing result. When the special processing field is identified as a non-encrypted field, the corresponding processing is performed according to the preset format conversion, data desensitization or data filtering rules, and the processed valid data is extracted as the parsing result.
6. The edge receiving method according to claim 1, characterized in that, The step of supplementing the structured data with basic field information according to preset formatting rules to generate standard format data includes: Based on the data reporting request, extract the reporting time information, front-end terminal type information, and reporting IP information corresponding to the original reported data; According to the preset formatting rules, the reporting time information, the front-end type information, and the reporting IP information are written into the specified field positions of the structured data to generate standard format data.
7. The edge receiving method according to claim 1, characterized in that, The subsequent processing of the log file employs an asynchronous processing mechanism decoupled from the data parsing operation and the persistent write operation, including: The log file is read by a separate log processing component, which then performs the data push operation to the remote server.
8. An edge receiving system for front-end embedded point information, characterized in that, The edge receiving system is applied to several stateless edge receiving nodes, which are independent of each other. The edge receiving system includes: The receiving unit is used to build a network communication interface that can provide services to the outside world based on a preset Web service framework, continuously listen to and receive data reporting requests sent from different front-end carriers, and extract the original reporting data carried in the data reporting requests. The acquisition unit is used to identify the fields to be processed in the original reported data and to obtain the data type corresponding to the fields to be processed. The parsing unit is used to perform corresponding data parsing operations on the field to be processed according to the data type, so as to obtain the parsed structured data; The generation unit is used to supplement basic field information into the structured data according to preset formatting rules to generate standard format data; The processing unit is used to persistently write the standard format data into the local storage medium in the format of a log file, and to adopt an asynchronous processing mechanism that is decoupled from the data parsing operation and the persistent writing operation for subsequent processing of the log file. For the stateless deployment of several edge receiving nodes, the edge receiving system further includes: The dynamic adjustment unit is used to monitor the traffic data of the front-end data reporting requests in real time, and dynamically adjust the number of edge receiving nodes deployed according to the fluctuation of the traffic data. The fault restart unit is used to monitor the operating status of each edge receiving node in real time. When any edge receiving node is detected to have an operating failure, it automatically restarts the faulty node and takes over the processing of the data reporting request.
9. An edge receiving device for front-end embedded point information, characterized in that, The device includes: Processor, memory, input / output units, and bus; The processor is connected to the memory, the input / output unit, and the bus; The memory stores a program, which the processor invokes to perform the method as described in any one of claims 1 to 8.