Unstructured message processing method, system, equipment and medium

Abstract

The invention discloses an unstructured message processing method, system, equipment and medium, and belongs to the technical field of data analysis processing, and the method comprises the steps: constructing a heterogeneous source data model, locking an atomization component sequence, extracting effective elements through feature cutting and cascade recursion analysis, and finally loading the effective elements to a standardized target model. The system comprises a source data model building module, a component sequence locking module, a feature recognition and cutting module, a cascade analysis model building module, a recursive circulation driving module and a structured recombination module. According to the invention, through dynamic arrangement of the atomization component and a multi-level cascade mechanism, precise recursive analysis of the complex nested message is realized. Component hot loading and strategy dynamic updating are supported, and flexible capacity expansion is achieved on the premise that high availability of services is guaranteed. The method gets rid of dependence on complex regularization, the processing efficiency of the unstructured data is remarkably improved, and the maintenance cost is reduced.

Description

Technical Field

[0001] This invention relates to the field of data parsing and processing technology, and specifically to a method, system, device, and medium for processing unstructured messages. Background Technology

[0002] With the rapid development of information technology, modern integrated information technology systems are increasingly widely used in various business scenarios, leading to an explosive growth in data access demands. In complex digital ecosystems, systems need to interact frequently from heterogeneous sources through multiple communication protocols, generating massive amounts of log messages and business files. This data typically exists in unstructured or semi-structured text formats, often with complex and varied content formats and a lack of unified standards. To effectively utilize these data resources, information collectors must perform in-depth protocol adaptation, content parsing, and information extraction for different data formats, converting raw unstructured text into standardized structured data for subsequent business analysis, statistical calculations, or decision support. Faced with a continuous influx of large-scale data traffic, especially in complex scenarios with high concurrency and multiple sources, how to achieve data reception, efficient parsing, accurate extraction, and rapid conversion has become a key challenge restricting the data processing capabilities of existing systems.

[0003] To address these needs, current mainstream text data processing systems primarily rely on regular expressions or simple optimization schemes based on them for message parsing. This pattern-matching approach has significant limitations in practical applications. First, writing regular expressions has a very high technical threshold; developers must possess strong professional backgrounds to handle complex and varied message formats. The resulting expressions are often logically obscure, making them difficult for ordinary technical personnel to quickly understand and maintain, leading to lengthy testing, debugging, and iteration cycles. Second, regular expression matching has an inherent bottleneck in runtime efficiency. To accommodate diverse data patterns, complex regular expressions generate numerous backtracking operations during runtime, significantly reducing matching efficiency. Especially when processing massive data streams, systems often have to rely on consuming more hardware resources to maintain necessary processing performance, resulting in excessive consumption of computing resources and a significant increase in system construction and operation costs. Summary of the Invention

[0004] In view of the above-mentioned problems, the present invention provides a method, system, device and medium for processing unstructured messages.

[0005] Therefore, the technical problem solved by this invention is that the existing technology relies on regular expression matching to process unstructured messages, which has the problems of high threshold for rule writing, low running efficiency, large resource consumption, and difficulty in dealing with the parsing of complex formats with multiple nesting layers.

[0006] To address the aforementioned technical problems, this invention provides the following technical solution: a method for processing unstructured messages, comprising: acquiring unstructured source data streams, encapsulating data fingerprints and meta-attributes, and constructing a heterogeneous source data object model; responding to the establishment of the heterogeneous source data object model, locking a suitable atomic processing component sequence based on a pre-defined strategy library of the data fingerprint index; traversing the atomic processing component sequence, cutting the unstructured source data stream through feature recognition logic to obtain a discrete feature set; combining the inherent structure of the discrete feature set to construct a multi-level cascaded parsing model for path planning of deeply nested information; through the multi-level cascaded parsing model, establishing a mapping channel between the current output and component input when a feature to be refined is detected, driving the data stream to iteratively flow between the atomic processing component sequences to obtain effective information elements; and based on mapping transformation logic, loading the effective information elements into a standardized target information model to complete the structured reorganization of the data.

[0007] In a preferred embodiment of the unstructured message processing method of the present invention, the atomic processing component sequence includes logically coupled feature extraction components and attribute assignment components; traversing the atomic processing component sequence includes: activating the feature extraction components, cutting the unstructured source data stream through preset feature delimitation logic to produce a temporary feature set; using the multi-level cascaded parsing model to monitor the recursive attributes of the temporary feature set; in response to the detection of recursion, constructing a secondary input stream, looping the temporary feature set back into the feature extraction components, and performing iterative parsing; in response to the absence of recursion, activating the attribute assignment components, mapping the temporary feature set to target business data, and generating the effective information elements.

[0008] In a preferred embodiment of the unstructured message processing method described in this invention, the following steps are taken: a dynamic operator-residence container is constructed to host component instances conforming to a unified contract interface protocol; in response to the triggering of a function extension instruction, an external component encapsulation package is parsed, and compliance with the unified contract interface protocol is verified; the external component encapsulation package is instantiated using a hot-loading mechanism to generate an extended component instance and register it to the dynamic operator-residence container; the index mapping of the policy library is refreshed, and the atomic processing component sequence is granted permission to call the extended component instance.

[0009] As a preferred embodiment of the unstructured message processing method of the present invention, the method includes: accessing a policy index library residing in a memory buffer, wherein the policy index library maintains the mapping relationship between policy indexes and policy entities based on a periodic refresh mechanism; extracting the data source attribution identifier from the heterogeneous source data object model as a first-level index key, and locating the corresponding parsing rule set in the policy index library; extracting the detailed rule location pointer from the heterogeneous source data object model as a second-level index key, and addressing a specific parsing logic unit encapsulating component orchestration configuration data in the parsing rule set; parsing the component orchestration configuration data, and sequentially instantiating the atomic processing component sequence.

[0010] As a preferred embodiment of the unstructured message processing method of the present invention, the method includes: executing key-value pair anchoring logic, which locates and segments the key-value pair structure based on predefined key name identifiers and connector paradigms; executing position offset delimiting logic, which extracts feature segments at fixed positions based on preset byte lengths or character index intervals; executing fence closure delimiting logic, which extracts the payload within the closed interval based on paired start and end symbols; and executing structured serialization logic, which traverses the syntax tree structure of an object markup language and maps the attribute values ​​of leaf nodes.

[0011] As a preferred embodiment of the unstructured message processing method of the present invention, the method comprises: executing timing regularization logic, which parses heterogeneous timestamp formats and converts them into a unified timing metric standard; executing semantic translation logic, which translates the original feature values ​​into business semantic values ​​according to pre-configured association mapping rules; executing encoding restoration logic, which identifies the transmission encoding protocol and restores it to plaintext characters; and executing morphological reshaping logic, which reconstructs the surface form of the data through character modification operations.

[0012] As a preferred embodiment of the unstructured message processing method described in this invention, the method includes: executing protocol curation logic, which establishes communication connections with multiple source data access points and converts heterogeneous transport protocol payloads into a standard input stream; executing routing marking logic, which injects the data fingerprint into the standard input stream based on the static configuration context of the data access task, the data fingerprint containing guiding information for indexing the policy library; and executing metadata synthesis logic, which generates a globally unique sequence number and a receiving timestamp, and binds them to the standard input stream to establish the lifecycle attributes of the data object.

[0013] To address the aforementioned technical problems, this invention also provides the following technical solution: a system for processing unstructured messages, comprising a source data model construction module for acquiring unstructured source data streams, encapsulating data fingerprints and meta-attributes, and constructing a heterogeneous source data object model; a component sequence locking module for locking an appropriate atomic processing component sequence based on a pre-defined strategy library of the data fingerprint index in response to the establishment of the heterogeneous source data object model; a feature recognition and segmentation module for traversing the atomic processing component sequence and segmenting the unstructured source data stream through feature recognition logic to obtain a discrete feature set; a cascaded parsing model construction module for constructing a multi-level cascaded parsing model based on the inherent structure of the discrete feature set for path planning of deeply nested information; a recursive flow driving module for establishing a mapping channel between the current output and component input when a feature to be refined is detected through the multi-level cascaded parsing model, driving the data stream to iteratively flow between the atomic processing component sequences to obtain effective information elements; and a structured reorganization module for loading the effective information elements into a standardized target information model based on mapping transformation logic to complete the structured reorganization of the data.

[0014] The present invention provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of the method for processing unstructured messages.

[0015] The present invention provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of the method for processing unstructured messages.

[0016] The beneficial effects of this invention are as follows: This invention replaces complex regular expressions with atomic component orchestration, significantly reducing the configuration threshold; it introduces a multi-level cascading parsing model to achieve accurate recursive extraction of deeply nested messages, improving information integrity; it supports hot loading of components, completing capability iteration without downtime, ensuring high business availability. Furthermore, this method optimizes the matching logic, greatly improving the processing efficiency of massive heterogeneous data and reducing resource consumption. Attached Figure Description

[0017] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 This is a flowchart illustrating a method for processing unstructured messages according to an embodiment of the present invention.

[0019] Figure 2 This is a block diagram of a system for processing unstructured messages according to an embodiment of the present invention.

[0020] Figure 3 This is a logical architecture diagram of a method for processing unstructured messages according to an embodiment of the present invention.

[0021] Figure 4 This is a schematic diagram of the parsing rule structure and recursive flow logic provided in one embodiment of the present invention. Detailed Implementation

[0022] To enable those skilled in the art to better understand the present invention, specific embodiments of the present invention are described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of them. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without inventive effort should fall within the protection scope of the present invention.

[0023] Example 1, referring to Figure 1 As one embodiment of the present invention, a method for processing unstructured messages is provided, comprising:

[0024] S1. Obtain unstructured source data streams, encapsulate data fingerprints and meta-attributes, and construct heterogeneous source data object models;

[0025] S2. In response to the establishment of the heterogeneous source data object model, based on the strategy library pre-set by the data fingerprint index, lock the appropriate atomic processing component sequence.

[0026] S3. Traverse the sequence of atomic processing components and cut the unstructured source data stream through feature recognition logic to obtain a discrete feature set;

[0027] S4. Combining the inherent structure of the discrete feature set, construct a multi-level cascaded analytical model for path planning of deeply nested information;

[0028] S5. Through the multi-level cascaded analytical model, when the feature to be refined is detected, a mapping channel between the current output and the component input is established, and the data flow is driven to iteratively flow between the atomic processing component sequences to obtain effective information elements.

[0029] S6. Based on the mapping transformation logic, the effective information elements are loaded into the standardized target information model to complete the structured reorganization of the data.

[0030] It should be noted that current mainstream unstructured message processing technologies generally rely on regular expression matching mechanisms. This approach reveals significant technical shortcomings when dealing with high-concurrency and heterogeneous data with varying formats in modern IT systems. On the one hand, to adapt to complex message structures containing multiple nested layers or mixed encoding, engineers often need to write extremely obscure and lengthy regular expressions. This not only significantly raises the technical threshold for rule configuration and subsequent maintenance, but also means that rigid regular expression rules often face the risk of becoming invalid or requiring complete reconstruction when the upstream data format undergoes minor changes. On the other hand, from an operational efficiency perspective, complex regular expression pattern matching triggers a large amount of backtracking computation when processing massive data streams. This computationally intensive operation leads to high server CPU resource utilization, thus becoming a performance bottleneck restricting the overall data throughput and real-time performance of the system.

[0031] Therefore, to address the aforementioned problems, this invention implements a dynamic parsing scheme based on atomic component orchestration through steps S1 to S5. This scheme first utilizes the heterogeneous source data object model in step S1 to standardize the multi-source, chaotic data, and then achieves O1-level strategy localization through the data fingerprint indexing mechanism in step S2, avoiding the performance overhead of traditional polling matching. More importantly, steps S3 to S5 introduce a multi-level cascaded parsing model to decompose the complex single full-match into an iterative sequence of atomic operations. The system can utilize the structural information of discrete feature sets for path planning, and when deeply nested features are discovered, it drives the data flow recursively between components by establishing mapping channels. This mechanism enables this invention to accurately extract the effective information elements of deeply nested features with extremely low computational cost, thus completely eliminating the dependence on high-consumption, complex regular expressions. This invention significantly reduces the difficulty of rule configuration while effectively improving the processing efficiency and accuracy of unstructured messages, greatly reducing hardware resource consumption, and achieving high efficiency and lightweighting of the entire data access process.

[0032] Example 2, refer to Figures 1 to 4 This is one embodiment of the present invention. Based on the previous embodiment, a method for processing unstructured messages is provided.

[0033] S1. Obtain unstructured source data streams, encapsulate data fingerprints and meta-attributes, and construct heterogeneous source data object models.

[0034] Specifically, in step S1, the unstructured source data stream is obtained, data fingerprints and meta-attributes are encapsulated, and a heterogeneous source data object model is constructed, including the following steps A1-A3:

[0035] A1: Executes protocol curation logic, establishes communication connections with multiple data access points, and converts heterogeneous transport protocol payloads into a standard input stream. The acquisition task, acting as an active or passive execution unit for data access, establishes connections with different platforms or heterogeneous data sources based on pre-configured access parameters. The system supports multiple standard or proprietary communication protocols, such as receiving real-time streaming data via TCP, acquiring file transfer data via SFTP, or reading database logs via JDBC. Regardless of the protocol used to transmit the raw data, the protocol curation logic strips away the underlying transport layer encapsulation, extracts the valid payload content, and converts it into a uniform string or byte stream format (standard input stream) for internal system processing, thereby eliminating interference from different communication protocols in the subsequent parsing process.

[0036] A2: Execute the routing marking logic. Based on the static configuration context of the data access task, inject a data fingerprint into the standard input stream. The data fingerprint contains guiding information used for indexing the policy library. During the task configuration phase, the administrator has specified the data source, acquisition protocol, and authentication information corresponding to the task. Based on this static configuration context, the routing marking logic automatically injects key guiding identifiers when generating objects. Specifically, the system marks the currently accessed data stream with a data source identifier (key) and a parsing identifier (mark). These two identifiers together constitute the data fingerprint. The key is used to locate the specific set of parsing rules in the global policy library, while the mark is used to further locate the specific parsing details within that set. This method of injecting fingerprints at the source ensures that subsequent processing components can quickly and accurately retrieve the appropriate parsing logic.

[0037] A3: Execute the metadata synthesis logic, generate a globally unique sequence number and a received timestamp, and bind them to the standard input stream to establish the lifecycle attributes of the data object. To ensure data traceability and uniqueness, the system automatically generates a globally unique string as the `id` attribute when constructing the object. Simultaneously, the system records the time the data was collected and received, generating a creation time (`ctime`) attribute. Furthermore, the metadata synthesis logic also encapsulates context information such as the business source `log_from`. Finally, the raw message content `raw`, along with the generated `id`, `ctime`, `key`, and `mark` metadata attributes, are assembled into a standardized heterogeneous source data object. This object serves as the unified input carrier for all subsequent parsing steps, masking the differences in the original data source.

[0038] S2. In response to the establishment of the heterogeneous source data object model, based on the strategy library pre-set by the data fingerprint index, lock the appropriate atomic processing component sequence.

[0039] In the embodiments of this application, in step S2, in response to the establishment of the heterogeneous source data object model, based on the pre-set strategy library of the data fingerprint index, the appropriate atomic processing component sequence is locked, including the following steps B1-B3:

[0040] B1: Accesses a policy index library residing in a memory buffer. This index library maintains the mapping between policy indexes and policy entities based on a periodic refresh mechanism. To ensure efficient policy matching, the device loads all parsing rule configurations into the system's internal cache during startup. This cache uses the globally unique code of the parsing rule as the key and the definition object of the parsing rule as the value, constructing a high-speed index structure based on hash mapping. Simultaneously, the system executes periodic update tasks through a built-in daemon thread, periodically detecting changes in the configuration source and reloading the latest configuration to refresh the cache. This ensures that the policy index library remains synchronized with business requirements, and this read operation is performed entirely in memory, avoiding frequent disk or database I / O overhead.

[0041] B2: Extract the data source attribution identifier from the heterogeneous source data object model as a primary index key to locate the corresponding set of parsing rules in the policy index library. The parsing program first reads the pre-injected data source identifier attribute from the heterogeneous source data object model, which indicates the business source or log type from which the data was generated. The system uses this identifier as a primary index key to perform a precise lookup directly in the in-memory policy index library, quickly locating the overall set of parsing rules corresponding to the data source in constant time complexity. This process skips the traversal and matching of irrelevant rules, achieving rapid convergence from a massive number of rules to a specific rule domain.

[0042] B3: Extract the detailed location pointer from the heterogeneous source data object model as a secondary index key. Address the specific parsing logic unit encapsulated with component orchestration configuration data within the parsing rule set, parse the component orchestration configuration data, and instantiate the atomic processing component sequence in sequence. Specifically, after locking the parsing rule set, the system further reads the parsing identifier attribute from the object model as a secondary index key. Since a business source may contain messages of various formats or versions, this identifier can accurately point to a specific parsing detail within the rule set. Based on the orchestration logic defined in this detail, the system reads the corresponding extraction and assignment configuration, and calls the corresponding Java implementation class or pre-initialized singleton object from the operator container, assembling it into an executable atomic processing component sequence according to the configuration order. This sequence clarifies the specific steps, parameters, and flow order for processing the current message, preparing for subsequent execution phases.

[0043] like Figure 4The diagram shown illustrates the logical architecture for unstructured message processing in this embodiment of the invention. The entire processing flow begins with the input of the original text object and ends with the output of the result model. The core parsing rule layer maintains runtime component instances through operator caching. This caching mechanism not only supports high-speed calls to regular operators but also allows for dynamic registration and hot reloading of new operators, thereby enabling capability expansion without stopping service. In terms of processing logic, Figure 4 The data flow branching paths are visually illustrated: after identification and matching, the features extracted from the operators can either flow directly to the assignment operators to generate the final result fields, or trigger a secondary extraction path when complex structures are detected, achieving deep parsing through multi-level cascading. This architecture ensures that the system can handle both flat data and flexibly cope with deeply nested complex messages.

[0044] In an optional implementation, in step S2, in response to the establishment of the heterogeneous source data object model, based on the pre-set strategy library of the data fingerprint index, the sequence of atomic processing components that is locked and adapted can also be differentiated through a gray-scale version routing mechanism. When parsing the data fingerprint, the system additionally extracts the version control identifier carried in the header of the source data or a random probability seed generated based on the source IP, and compares the identifier with the version control table in the strategy library in real time. Based on this, a specific proportion of the source data flow is directed to the new version component sequence in the testing and verification stage, thereby ensuring the stable operation of the main production link while realizing online verification and smooth iteration of the new strategy logic.

[0045] In another optional implementation, in step S2, in response to the establishment of the heterogeneous source data object model, based on the pre-set strategy library of the data fingerprint index, the locked and adapted atomic processing component sequence can also process unknown messages through a heuristic fuzzy matching mechanism. When the system cannot use the accurate data fingerprint to retrieve a completely matching rule key value in the strategy library, it automatically triggers the similarity calculation logic based on text feature density, scans the distribution features of frequently occurring keywords in the source data and compares them with historical strategy samples, and then selects the general component sequence with the highest confidence as a fallback processing solution to ensure that the system can still maintain basic parsing capabilities when facing non-standard or slightly mutated data formats.

[0046] S3. Traverse the sequence of atomic processing components and cut the unstructured source data stream through feature recognition logic to obtain a discrete feature set.

[0047] Specifically, in step S3, the sequence of atomic processing components is traversed, and the unstructured source data stream is segmented through feature recognition logic to obtain a discrete feature set, including the following steps C1-C3:

[0048] C1: The component traversal engine is launched, activating atomic processing components in the sequence sequentially and determining the component type based on their interface definitions. Specifically, the parser reads operator instances in the sequence sequentially, identifying whether they are extraction operators inheriting from the IExtAnalysis interface or assignment operators inheriting from the IExtAssign interface. For extraction operators, the engine prepares the input stream data and initializes the parameter context required for the extraction operation, ensuring that the operator can correctly obtain the text fragment to be processed.

[0049] C2: Executes preset feature delimitation logic, calling the character processing algorithm encapsulated in the extraction component to physically segment and recognize unstructured data. The system supports multiple dimensions of delimitation logic to adapt to different message formats. For example, when the KEY-VALUE extraction operator is called, the system identifies and segments key-value pairs according to predefined key-value connector and delimiter paradigms; when the opening / closing extraction operator is called, the system extracts the payload within the closed interval by recognizing specific start and end characters; when the JSON formatting operator is called, the system uses a serialization engine to parse the text into key-value collection objects. These operators break down the continuous text stream into independent fragments through standardized interface calls.

[0050] C3: Outputs a discrete feature set in a temporary state, encapsulating the segmented data into a list of key-value pairs and residing it in memory. Regardless of the specific extraction algorithm used, the output of the extraction operator is standardized into a List collection containing Map objects. This collection, as an intermediate state material, not only contains the specific field values ​​extracted at the current level but also retains references to the original message fragments, providing a standardized data foundation for subsequent secondary extraction or final assignment transformations.

[0051] like Figure 3 As shown in the diagram, this invention presents a hierarchical tree structure for the parsing rule structure and recursive flow logic. Each parsing rule contains one or more parsing details, and each parsing detail consists of a series of ordered operation nodes. These operation nodes are divided into two categories: extraction operations and assignment operations. When an extraction operation is executed, the system generates a temporary output value; this output value can be passed to the assignment list as final state data, or it can be used as a secondary input value to loop back into the next level of extraction operation, thus forming a closed-loop recursive call chain. This structure establishes the logical foundation of a multi-level cascaded parsing model, enabling the data flow to circulate multiple times between extraction components according to the configuration until the assignment conditions are met.

[0052] S4. Combining the inherent structure of the discrete feature set, construct a multi-level cascaded analytical model for path planning of deeply nested information.

[0053] Specifically, in step S4, a multi-level cascaded analytical model is constructed by combining the inherent structure of the discrete feature set for path planning of deeply nested information, including the following steps D1-D3:

[0054] D1: Analyze the internal structure of the discrete feature set, scan the current parsing rule configuration, and detect whether there are secondary extraction instructions for specific fields. After obtaining the initially extracted discrete feature set, the system does not immediately assign values; instead, it iterates through the current rule configuration, checking if any fields are marked as containing nested structures or requiring further parsing. For example, for an extracted JSON string field, if the configuration specifies further mining of its internal fields, the system will identify that field as a feature to be refined.

[0055] D2: Constructing the context environment for recursive parsing and establishing the association mapping between parent features and child parsing logic. Specifically, once a feature to be refined is detected, the system creates a new parsing context, using the currently extracted feature value as the new input source, and loading the corresponding secondary extraction configuration as the execution logic of the current context. This mechanism constructs a logical stack structure in memory to maintain the data flow state between parent and child levels, ensuring that the association information of the parent log is not lost when deeply parsing the substructure.

[0056] D3: Plan the execution path for deeply nested information, determining whether subsequent operations should proceed to the next level of extraction loop or jump to assignment output. The multi-level cascading parsing model is essentially a state machine that dynamically determines the direction of the data flow based on configuration. If the current node still has an extraction operator attached, the path planning logic will point the pointer to that child operator, preparing to trigger a recursive call; if the current node has reached the end of the parsing chain, the path planning logic will point the pointer to the assignment operator, preparing for the final data transformation. This path planning ensures that the parsing of messages with arbitrary depth of nesting can proceed in the expected logical topology order.

[0057] S5. Through the multi-level cascaded analytical model, when a feature to be refined is detected, a mapping channel between the current output and the component input is established, driving the data flow to iteratively flow between the atomic processing component sequences to obtain effective information elements.

[0058] Specifically, in step S5, a multi-level cascaded analytical model is used to establish a mapping channel between the current output and the component input when the feature to be refined is detected. This drives the data flow to iteratively flow between the atomic processing component sequences to obtain effective information elements, including the following steps E1-E3:

[0059] E1: Executes the mapping channel establishment logic. In response to the recursiveness detection of the multi-level cascaded parsing model, it dynamically allocates memory addresses to link the output port of the parent operator with the input port of the child operator. Specifically, when the path planning logic determines that the currently extracted data segment contains a nested structure requiring further parsing, the system does not immediately release the temporarily stored feature set, but instead marks it as secondary source data to be processed. The mapping channel establishment logic locks the memory reference of this secondary source data and points it to the input buffer of the next-level extraction component, thus logically establishing a cross-level data transmission link and eliminating the overhead of repeatedly reading and writing temporary disk files required in traditional linear parsing.

[0060] E2: Executes data flow-driven logic, activating sub-level atomic processing components using a recursive call mechanism, and pushing secondary source data into the next round of feature segmentation. The system passes the mapping channel established in the previous step as a parameter to the recursive function, driving the data flow to loop between component sequences. At this point, the extraction-type components in the atomic processing component sequence are reawakened, but their operation objects have changed from the initial unstructured source data flow to the local fragments stripped from the previous level. This driving logic supports infinite levels of deep recursion until the data flow is segmented into indivisible atomic information units.

[0061] E3: Executes the iterative flow termination logic, monitors the end status of the parsing path, and locks valid information elements when data flows to non-extractable nodes. As the recursion depth increases, the data granularity becomes increasingly refined. When the path planning logic detects that the current component node no longer has extractable attributes but points to a clear assignment definition, the system determines that the iteration process has ended. At this point, the data flowing in the channel is the core business data after layers of stripping. The system locks it as a valid information element, terminates the current recursive loop, and prepares to enter the subsequent format conversion stage.

[0062] S6. Based on the mapping transformation logic, the effective information elements are loaded into the standardized target information model to complete the structured reorganization of the data.

[0063] In the embodiments of this application, in step S6, based on the mapping transformation logic, the effective information elements are loaded into the standardized target information model to complete the structured reorganization of the data, including the following steps F1-F3:

[0064] F1: Executes business semantic translation logic, invoking attribute assignment components to reshape and standardize valid information elements. Specifically, the system performs necessary cleaning and transformation on the captured raw strings based on pre-configured assignment operators. For example, it calls time conversion operators to parse heterogeneous timestamp formats and unify them into standard time-series metrics; it calls dictionary mapping operators to translate coded enumeration values ​​into readable business meanings; or it calls de-identification operators to mask sensitive data. This logic ensures that the extracted information conforms to the semantic and formatting requirements of downstream business systems.

[0065] F2: Executes the target model loading logic, mapping the translated business data to the corresponding field slots in the normalized target information model. The normalized target information model is a predefined flat key-value pair structure encompassing both general and business-specific fields. Based on the output mapping configuration in the parsing rules, the system precisely fills each processed valid information element into target fields such as id, account, and operation_time. This process reorganizes the originally discrete and fragmented information into structured records with rigorous logical relationships.

[0066] F3: Executes the structured reorganization completion logic, verifies the integrity of the target model, and outputs the final result object. After all fields are loaded, the system performs integrity constraint verification on the result model to ensure that required fields are complete and of correct data type. Models that pass the verification are marked as the final output and serialized into standard JSON or XML format for output to downstream storage or analysis systems, thus completing the entire process of transforming unstructured messages into high-value structured data.

[0067] In an optional implementation, step S6, based on mapping and transformation logic, loads effective information elements into the standardized target information model. It can also expand the data dimensions in real time through an external knowledge base association enhancement mechanism. When assembling the target model, the system identifies key index fields such as device codes or network addresses, and uses these as keys to synchronously query the distributed cache system or master data management platform to obtain auxiliary metadata such as geographical coordinates, asset ownership departments, or detailed fault descriptions. Subsequently, these extended attributes, which are not directly reflected in the original message, are integrated into the target information model, thereby improving the semantic density and business availability of the final output data.

[0068] In another optional implementation, step S6, based on mapping and transformation logic, loads the effective information elements into the standardized target information model. Data security can also be ensured through a privacy compliance-driven desensitization filtering mechanism. Before loading, the system compares the field types of the effective information elements with the preset data security governance strategy. Once sensitive privacy attributes such as ID card numbers, mobile phone numbers, or authentication keys are detected, desensitization algorithms such as character replacement, hash encryption, or truncation masking are automatically triggered to process the feature values, ensuring that the generated standardized target information model meets the downstream business analysis needs while complying with the legal and regulatory requirements for data security and privacy protection.

[0069] It should be noted that existing technical solutions for processing unstructured messages generally employ pattern matching techniques based on regular expressions. This approach requires engineers to write complex regular expression syntax to handle varying log formats, resulting in extremely high learning and maintenance costs. Furthermore, when processing long texts or nested formats, the greedy matching and backtracking mechanisms of the regular expression engine consume significant CPU resources, leading to a decrease in system throughput. In addition, traditional solutions struggle to handle deeply nested data structures, often requiring the development of customized scripts for multiple rounds of processing, lacking versatility and flexibility.

[0070] Therefore, to address the aforementioned problems, this invention constructs an unstructured message processing system based on atomic component orchestration and recursive parsing through steps S1 to S6. This invention utilizes the fingerprint indexing mechanism in S1 and S2 to achieve rapid policy localization, avoiding the performance overhead of full rule polling; S3 decomposes the complex parsing task into reusable atomic extraction actions, lowering the configuration threshold; in particular, the multi-level cascaded parsing model introduced in S4 and S5 achieves recursive extraction of deeply nested messages by establishing memory mapping channels, accurately extracting valid information without writing complex regular expressions; finally, the standardized mapping in S6 ensures high quality and standardization of the output data. This method significantly improves parsing efficiency and accuracy while greatly reducing system hardware resource consumption and maintenance costs.

[0071] Example 3, referring to Figure 2 The above is an illustrative scheme of a method for processing unstructured messages. It should be noted that the technical solution of this unstructured message processing system and the technical solution of the aforementioned unstructured message processing method belong to the same concept. Details not described in detail in this embodiment of the unstructured message processing system can be found in the description of the aforementioned unstructured message processing method.

[0072] This embodiment also provides a system for processing unstructured messages, including:

[0073] The source data model building module is used to acquire unstructured source data streams, encapsulate data fingerprints and meta-attributes, and build heterogeneous source data object models.

[0074] The component sequence locking module is used to lock the adapted atomic processing component sequence in response to the establishment of the heterogeneous source data object model, based on the strategy library pre-set by the data fingerprint index.

[0075] The feature recognition and segmentation module is used to traverse the sequence of atomic processing components and segment the unstructured source data stream through feature recognition logic to obtain a discrete feature set;

[0076] The cascaded analytical model construction module is used to construct a multi-level cascaded analytical model by combining the inherent structure of the discrete feature set, which is used for path planning of deeply nested information.

[0077] The recursive flow driving module is used to establish a mapping channel between the current output and the component input when the feature to be refined is detected through the multi-level cascaded parsing model, and drive the data flow to iteratively flow between the atomic processing component sequences to obtain effective information elements.

[0078] The structured reorganization module is used to load the effective information elements into the standardized target information model based on mapping transformation logic, thereby completing the structured reorganization of the data.

[0079] This embodiment also provides an electronic device applicable to a method for processing unstructured messages, comprising: a memory and a processor; the memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions to implement the method for processing unstructured messages as proposed in the above embodiment.

[0080] This embodiment also provides a storage medium storing a computer program that, when executed by a processor, implements a method for processing unstructured messages as described in the above embodiments.

[0081] The storage medium proposed in this embodiment and the method for implementing unstructured message processing proposed in the above embodiments belong to the same inventive concept. Technical details not described in detail in this embodiment can be found in the above embodiments, and this embodiment has the same beneficial effects as the above embodiments.

[0082] Based on the above description of the implementation methods, those skilled in the art can clearly understand that the present invention can be implemented using software and necessary general-purpose hardware, and of course, it can also be implemented using hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as a computer floppy disk, read-only memory (ROM), random access memory (RAM), flash memory, hard disk, or optical disk, etc., including several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods of the various embodiments of the present invention.

[0083] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that the technical solutions of the present invention can be modified or equivalently replaced without departing from the spirit and scope of the technical solutions of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A method for processing unstructured messages, characterized in that, include: Acquire unstructured source data streams, encapsulate data fingerprints and meta-attributes, and construct heterogeneous source data object models; In response to the establishment of the heterogeneous source data object model, based on the strategy library pre-set by the data fingerprint index, the appropriate atomic processing component sequence is locked. By traversing the sequence of atomic processing components, the unstructured source data stream is segmented using feature recognition logic to obtain a discrete feature set; Based on the inherent structure of the discrete feature set, a multi-level cascaded analytical model is constructed for path planning of deeply nested information. Through the multi-level cascaded analytical model, when a feature to be refined is detected, a mapping channel between the current output and the component input is established, driving the data flow to iteratively flow between the atomic processing component sequences to obtain effective information elements; Based on the mapping and transformation logic, the effective information elements are loaded into the standardized target information model to complete the structured reorganization of the data.

2. The method for processing unstructured messages as described in claim 1, characterized in that, The atomication processing component sequence includes logically coupled feature extraction components and attribute assignment components; The traversal of the sequence of atomized processing components includes: Activate the feature extraction component and cut the unstructured source data stream through preset feature delimitation logic to produce a temporary feature set; The recursive properties of the temporary state feature set are monitored using the multi-level cascaded analytical model. In response to the detection of the recursion, a secondary input stream is constructed, and the temporary feature set is looped back into the feature extraction class component for iterative parsing. In response to the lack of recursion, the attribute assignment class component is activated to map the temporary feature set to the target business data and generate the effective information element.

3. The method for processing unstructured messages as described in claim 1, characterized in that, Also includes: Construct a dynamic operator-residence container to host component instances that conform to the unified contract interface protocol; In response to the triggering of the function extension instruction, the external component encapsulation package is parsed, and the compliance with the unified contract interface protocol is verified; The external component package is instantiated using a hot-loading mechanism to generate an extended component instance and register it to the dynamic operator resident container; Refresh the index mapping of the policy library and grant the atomic processing component the permission to sequentially call the extended component instance.

4. The method for processing unstructured messages as described in claim 1, characterized in that, The sequence of atomic processing components for locking adaptation includes: Access a policy index library residing in a memory buffer, wherein the policy index library maintains the mapping relationship between policy indexes and policy entities based on a periodic refresh mechanism; Extract the data source attribution identifier from the heterogeneous source data object model as a first-level index key, and locate the corresponding set of parsing rules in the strategy index library; Extract the detailed positioning pointer from the heterogeneous source data object model as a secondary index key, and address the specific parsing logic unit that encapsulates component orchestration configuration data in the parsing rule set; The component orchestration configuration data is parsed, and the atomic processing component sequence is instantiated in order.

5. The method for processing unstructured messages as described in claim 2, characterized in that, The preset feature delimitation logic includes: Execute key-value pair anchoring logic, which locates and segments the key-value pair structure based on a predefined key name identifier and connector paradigm; Execute position offset delimiting logic, which extracts feature segments at fixed positions based on a preset byte length or character index range; Execute fence closure and boundary logic, which extracts the effective load within the closed interval based on the pair of start and end symbols; Execute structured serialization logic, which is based on the syntax tree structure of the object markup language and maps the attribute values ​​of the leaf nodes.

6. The method for processing unstructured messages as described in claim 1, characterized in that, The mapping transformation logic includes: The timing warping logic is executed, which parses heterogeneous timestamp formats and converts them into a unified timing metric standard. The semantic translation logic is executed, which translates the original feature values ​​into business semantic values ​​according to the pre-configured association mapping rules; The encoding restoration logic is executed, which identifies the transmission encoding protocol and restores it to plaintext characters. The morphological reshaping logic is executed, which reconstructs the surface form of the data through character modification operations.

7. The method for processing unstructured messages as described in claim 1, characterized in that, The construction of the heterogeneous source data object model includes: The protocol curation logic is executed, which establishes a communication connection with multiple source data access points and converts heterogeneous transmission protocol payloads into standard input streams. The routing marking logic is executed, which injects the data fingerprint into the standard input stream based on the static configuration context of the data access task. The data fingerprint contains guiding information for indexing the policy library. The metadata synthesis logic is executed, which generates a globally unique sequence number and a received timestamp, and binds them to the standard input stream to establish the lifecycle attributes of the data object.

8. A system for processing unstructured messages, employing the method for processing unstructured messages as described in any one of claims 1 to 7, characterized in that, include: The source data model building module is used to acquire unstructured source data streams, encapsulate data fingerprints and meta-attributes, and build heterogeneous source data object models. The component sequence locking module is used to lock the adapted atomic processing component sequence in response to the establishment of the heterogeneous source data object model, based on the strategy library pre-set by the data fingerprint index. The feature recognition and segmentation module is used to traverse the sequence of atomic processing components and segment the unstructured source data stream through feature recognition logic to obtain a discrete feature set; The cascaded analytical model construction module is used to construct a multi-level cascaded analytical model by combining the inherent structure of the discrete feature set, which is used for path planning of deeply nested information. The recursive flow driving module is used to establish a mapping channel between the current output and the component input when the feature to be refined is detected through the multi-level cascaded parsing model, and drive the data flow to iteratively flow between the atomic processing component sequences to obtain effective information elements. The structured reorganization module is used to load the effective information elements into the standardized target information model based on mapping transformation logic, thereby completing the structured reorganization of the data.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method for processing unstructured messages according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method for processing unstructured messages according to any one of claims 1 to 7.

Images