A data processing method and device, electronic equipment and storage medium

Abstract

The application provides a data processing method and device, electronic equipment and storage medium. The original heterogeneous IT alarm text is received, and when it is determined that the original heterogeneous IT alarm text does not have a structured rule, target privacy information of a non-IT technology dimension is located from the original heterogeneous IT alarm text, the target privacy information is replaced with a mask constant that is invariant in a global state, a desensitization intermediate state text is output, a constant template representing an IT event type and an implicit key-value pair representing a dynamic parameter value are extracted from the desensitization intermediate state text, a structured data framework is determined according to the constant template, a standard slot corresponding to the dynamic parameter type under the structured data framework is determined according to a preset semantic mapping table, a parameter value corresponding to the dynamic parameter type is filled into the standard slot, a structured data file is obtained, heterogeneous data originally in a complex discrete state is reconstructed into structure-aligned data, and a semantically consistent basic observation atlas input is provided for intelligent analysis of a back end.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a data processing method, apparatus, electronic device, and storage medium. Background Technology

[0002] With the exponential expansion of modern IT infrastructure and the widespread adoption of microservice architectures and containerized orchestration technologies (such as the minimum deployable unit mechanism in Kubernetes clusters), the volume of monitoring alerts and operational logs generated by underlying IT information systems is exploding. In the production environments of large enterprises, the underlying monitoring toolchains of IT information systems are typically highly heterogeneous and fragmented. Monitoring probes from different original equipment manufacturers (OEMs), operating system kernels across different layers, and network layer devices are sending massive amounts of discrete status information and alert events to the operations and maintenance center every moment.

[0003] However, existing traditional IT monitoring systems generally suffer from a logical limitation in their underlying architecture design: prioritizing front-end visual presentation over back-end structured data representation. The core atomic information of massive amounts of IT alerts on the network is often triggered by front-end monitoring scripts and hard-coded aggregation based on static rules, resulting in extremely lengthy and unstandardized text content that is detrimental to subsequent applications. Summary of the Invention

[0004] The purpose of this application is to provide a data processing method, apparatus, electronic device, and storage medium to solve the above-mentioned technical problems.

[0005] On the one hand, a data processing method is provided, including: Receive raw heterogeneous IT alarm text, and when it is determined that the raw heterogeneous IT alarm text does not have a structured pattern, locate target privacy information that is not related to IT technology from the raw heterogeneous IT alarm text; Replace the target privacy information with a mask constant that remains unchanged in the global state, and output the de-identified intermediate text; Extract constant templates representing IT event types and implicit key-value pairs representing dynamic parameter values ​​from the de-identified intermediate text; the implicit key-value pairs include the correspondence between dynamic parameter types and parameter values; A structured data framework is determined based on the constant template, and a standard slot corresponding to the dynamic parameter type under the structured data framework is determined based on the preset semantic mapping table. The parameter values ​​corresponding to the dynamic parameter type are filled into the standard slots to obtain a structured data file. The semantic mapping table defines the correspondence between preset dynamic parameter types and preset standard slots.

[0006] In one embodiment, determining that the original heterogeneous IT alarm text does not have a structured pattern includes: Calculate the Shannon information entropy and structured feature density corresponding to the original heterogeneous IT alarm text; When the Shannon information entropy is determined to be greater than or equal to a preset entropy threshold, or when the structured feature density is determined to be less than or equal to a preset density threshold, the original heterogeneous IT alarm text is determined to have no structured regularity.

[0007] In one embodiment, the method further includes: When the Shannon information entropy is determined to be less than the preset entropy threshold and the structured feature density is greater than the preset density threshold, the original heterogeneous IT alarm text is directly deserialized.

[0008] In one embodiment, locating target privacy information that is not related to IT technology from the original heterogeneous IT alarm text includes: The main semantic awareness branch locates preliminary privacy information from the original heterogeneous IT alarm text in non-IT technical dimensions; The bias removal penalty and domain compliance verification module determines the domain semantic dependency distance of the preliminary privacy information based on its text location. When it is determined that the preceding information of the preliminary privacy information contains preset IT technical terminology features, a negative confidence penalty scalar is generated to prevent the preliminary privacy information from being marked as the target privacy information. When it is determined that the preceding information of the preliminary privacy information does not contain preset IT technical terminology features, the preliminary privacy information is directly marked as the target privacy information.

[0009] In one embodiment, extracting constant templates describing IT event types and implicit key-value pairs describing dynamic parameter values ​​from the de-identified intermediate text includes: Symbols in the de-identified intermediate text that do not possess technical semantics are marked as redundant nodes without business semantics, while symbols in the de-identified intermediate text that possess technical semantics are marked as valid business nodes. For each redundant node without business semantics in memory, the parent node of the redundant node without business semantics is redirected to the next valid business node of the redundant node without business semantics, and the redundant node without business semantics is deleted from the memory. A syntax tree is constructed using the remaining valid business nodes and the relationships between them. Semantic deduction is performed on the syntax tree, the implicit key-value pairs are obtained by stripping the dynamic parameters from the syntax tree, and placeholders are filled in the positions of the syntax tree corresponding to the dynamic parameters to obtain the constant template.

[0010] In one embodiment, extracting constant templates describing IT event types and implicit key-value pairs describing dynamic parameter values ​​from the de-identified intermediate text includes: Extract background information representing the source of the original heterogeneous IT alarm text; A hash signature is generated based on the background information, and the hash signature is used as the contextual hint boundary of the search space of the large language model solution; Based on the large language model, constant templates describing IT event types and implicit key-value pairs describing dynamic parameter values ​​are extracted from the anonymized intermediate text.

[0011] In one embodiment, determining the standard slot corresponding to the dynamic parameter type under the structured data framework according to a preset semantic mapping table includes: When the dynamic parameter type does not exist in the semantic mapping table, the semantic similarity between the word vector corresponding to the dynamic parameter type and the word vector corresponding to each preset dynamic parameter type in the semantic mapping table is calculated, and the preset standard slot corresponding to the preset dynamic parameter type with the highest semantic similarity is taken as the standard slot corresponding to the dynamic parameter type.

[0012] On the other hand, a data processing system is also provided, including: The positioning module is used to receive raw heterogeneous IT alarm texts and, when it is determined that the raw heterogeneous IT alarm texts do not have a structured pattern, locate target privacy information that is not related to IT technology from the raw heterogeneous IT alarm texts. The replacement module is used to replace the target privacy information with a mask constant that remains unchanged in the global state and output the desensitized intermediate text. The extraction module is used to extract constant templates representing IT event types and implicit key-value pairs representing dynamic parameter values ​​from the desensitized intermediate text; the implicit key-value pairs include the correspondence between dynamic parameter types and parameter values; The filling module is used to determine a structured data framework based on the constant template, and to determine the standard slots corresponding to the dynamic parameter types under the structured data framework based on a preset semantic mapping table. The module fills the standard slots with the parameter values ​​corresponding to the dynamic parameter types to obtain a structured data file. The semantic mapping table defines the correspondence between preset dynamic parameter types and preset standard slots.

[0013] On the other hand, an electronic device is also provided, including a processor and a memory, wherein the memory stores a computer program, and the processor executes the computer program to implement any of the methods described above.

[0014] On the other hand, a computer-readable storage medium is also provided, which stores a computer program that, when executed by at least one processor, implements any of the methods described above.

[0015] The data processing method, apparatus, electronic device, and storage medium provided in this application receive raw heterogeneous IT alarm texts. When it is determined that the raw heterogeneous IT alarm texts do not have a structured pattern, the method locates target privacy information in the non-IT technology dimension from the raw heterogeneous IT alarm texts, replaces the target privacy information with a mask constant that remains unchanged in the global state, outputs de-identified intermediate text, extracts constant templates representing IT event types and implicit key-value pairs representing dynamic parameter values ​​from the de-identified intermediate text, determines a structured data framework based on the constant templates, and determines standard slots corresponding to dynamic parameter types under the structured data framework according to a preset semantic mapping table. The method fills the standard slots with parameter values ​​corresponding to dynamic parameter types to obtain a structured data file. This reconstructs heterogeneous data that originally presented a complex, discrete, and coupled state into structured data, providing a semantically consistent basic observation map input for backend intelligent analysis.

[0016] Other features and advantages of this application will be set forth in the following description and will be apparent in part from the description or may be learned by practicing the application. The purposes and other advantages of this application may be realized and obtained by means of the structures particularly pointed out in the written description, claims, and drawings. It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not intended to limit this disclosure. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of this application. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0018] Figure 1 A flowchart illustrating the data processing method provided in an embodiment of this application; Figure 2 This is a schematic diagram of the structure of the data processing method system provided in the embodiments of this application; Figure 3 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0019] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0020] Please see Figure 1 As shown in the figure, this application provides a data processing method, including: S11: Receive the original heterogeneous IT alarm text, and when it is determined that the original heterogeneous IT alarm text does not have a structured pattern, locate the target privacy information in the non-IT technology dimension from the original heterogeneous IT alarm text.

[0021] S12: Replace the target privacy information with a mask constant that remains unchanged in the global state, and output the desensitized intermediate text.

[0022] S13: Extract constant templates representing IT event types and implicit key-value pairs representing dynamic parameter values ​​from the desensitized intermediate text; the implicit key-value pairs include the correspondence between dynamic parameter types and parameter values.

[0023] S14: Determine the structured data framework based on the constant template, and determine the standard slots corresponding to the dynamic parameter types under the structured data framework based on the preset semantic mapping table. Fill the standard slots with the parameter values ​​corresponding to the dynamic parameter types to obtain the structured data file. The semantic mapping table defines the correspondence between preset dynamic parameter types and preset standard slots.

[0024] Step S11 can be deployed at system edge nodes as an intelligent high-speed buffer and pre-sorting gateway, utilizing lightweight statistical features to perform dynamic routing and traffic distribution. Specifically, it can distribute the original heterogeneous IT alarm texts to different data processing links based on whether they have a structured pattern.

[0025] In one embodiment, determining that the original heterogeneous IT alarm text does not have a structured pattern includes: Calculate the Shannon information entropy and structured feature density corresponding to the original heterogeneous IT alarm text; when the Shannon information entropy is greater than or equal to a preset entropy threshold, or when the structured feature density is less than or equal to a preset density threshold, determine that the original heterogeneous IT alarm text does not have a structured pattern.

[0026] When it is determined that the original heterogeneous IT alarm text does not have a structured pattern, it can be diverted to the backbone link and the above S12 to S14 can be executed. When it is determined that the original heterogeneous IT alarm text has a structured pattern, it can be diverted to the high-speed direct link to directly deserialize the original heterogeneous IT alarm text. The deserialization operation is the process of restoring the serialized data (such as byte stream, string, JSON / XML text) into a memory object or data structure that the program can directly use.

[0027] In this embodiment, when the Shannon information entropy corresponding to the original heterogeneous IT alarm text is less than a preset entropy threshold and the structured feature density is greater than a preset density threshold, it can be determined that the original heterogeneous IT alarm text has a structured pattern, and then the original heterogeneous IT alarm text can be directly deserialized.

[0028] Specifically, the original heterogeneous IT alarm text can be a raw byte stream from different underlying IT computing resources. The unpacking engine strips the transport layer and session layer network packet headers, extracts the core character length field from the payload, and for this field, the system extracts the original character sequence stream X = [x1, x2, ..., xn], where n is the one-dimensional spatial length of the character sequence. Subsequently, the system performs basic character encoding forced alignment (such as uniformly converging to the UTF-8 standard) and cleans up invisible illegal control characters.

[0029] For the above-mentioned original character sequence stream, the discrete probability distribution P(x) of the character sequence at the basic vocabulary level can be statistically analyzed, and the Shannon information entropy H(X) representing the degree of disorder of the sequence can be calculated in an unsupervised mode. The formula for calculating the Shannon information entropy can be referred to the existing formulas and will not be elaborated in this patent.

[0030] Secondly, the engine extracts a set of specific structured delimited punctuation marks preset by the system (such as closed-loop marks of JSON or XML syntax), counts the discrete total frequency of their occurrence in the original character sequence stream X, and divides it by the total length n of the original character sequence stream to calculate the structured feature density D that represents the format standardization.

[0031] Non-IT technology-related target privacy data refers to sensitive data in the original heterogeneous IT alert text that is unrelated to the system's technical operation but is related to other things (such as "people" or "business organizations").

[0032] In one embodiment, locating target privacy information that is not related to IT technology from original heterogeneous IT alarm text includes: The main semantic awareness branch locates preliminary privacy information from the original heterogeneous IT alarm text in non-IT technical dimensions; The bias removal penalty and domain compliance verification module determines the domain semantic dependency distance of the preliminary privacy information based on its text location. When it is determined that the preceding information of the preliminary privacy information contains preset IT technical lexical features, a negative confidence penalty scalar is generated to prevent the preliminary privacy information from being marked as target privacy information. When it is determined that the preceding information of the preliminary privacy information does not contain preset IT technical lexical features, the preliminary privacy information is directly marked as target privacy information.

[0033] Given that hard-coded non-IT contact information typically exhibits an implicit structured bias that tends to cluster at the end of the sequence, this step can establish a positional weight decay model. By assigning higher "suspected unstructured noise prior penalty weights" to character intervals near the end of the sequence that meet a specific character density, the tensor sensing region to be reviewed is initially delineated.

[0034] The main semantic awareness branch can employ a pre-trained bidirectional Transformer Encoder architecture. During tensor transformation, the encoder simultaneously performs global attention scans on the sequence X = [x1, x2, ... xn] from left to right and from right to left, generating a dense context embedding vector matrix E = [e1, e2, ... en] containing rich contextual information. The mission of this branch is to keenly capture hidden pre-contextual semantic units (such as "contact person," "extension number," "Call," etc.) in the sequence and output preliminary entity classification prediction log odds, that is, to determine preliminary privacy information.

[0035] The bias reduction penalty and domain compliance verification module is the core defense against accidental deletion of underlying core code. IT underlying logs widely contain hexadecimal network card MAC addresses (e.g., 00:AA:BB:CC:DD:EE), thread stack PIDs, and other numerical matrices that are easily confused with phone numbers in character form. At the code flow level, when the backbone network forward scan encounters a continuous sequence of numbers and provides a high-confidence prediction of personal identification information, this module intervenes forcefully. It calculates the semantic dependency distance by cross-comparing the preceding context embedding vector with the built-in IT domain compliance tensor dictionary. Once it finds that the preceding terminology (i.e., the preceding information) contains a specific IT technology prefix (e.g., "PID:", "MAC:", "stack_trace:"), this module will directly generate a strong negative confidence penalty scalar at the micro-level of calculating the activation function to prevent preliminary privacy information from being marked as target privacy information. If it is determined that the preceding information of the preliminary privacy information does not contain the preset IT technology terminology features (i.e., the aforementioned IT technology prefixes), the preliminary privacy information is directly marked as the target privacy information.

[0036] Logical suppression execution: Before the final Softmax probability normalization layer, the system executes the formula: Adjusted_Logits = Original_Logits - (Penalty_Weight * Domain_Match_Score). Where, Original_Logits represents the preliminary entity classification prediction probability output by the global scan of the main semantic perception branch, i.e., the original prediction activation value before the preliminary privacy information is subject to compliance verification intervention. Domain_Match_Score represents the domain semantic dependency distance calculated in tensor space by the bias removal penalty and the domain compliance verification module, i.e., the feature matching quantization score between the pre-existing information of the preliminary privacy information and the system's built-in IT domain compliance tensor dictionary. Penalty_Weight represents the negative confidence penalty scalar constant generated by the system through micro-level forced triggering when the calculation confirms that the pre-existing information contains preset IT technical lexical features. Adjusted_Logits represents the corrected log-probability tensor generated after probability penalty. By executing the probability suppression formula, this system uses the product between the feature matching quantization score and the penalty scalar constant as a negative increment to forcibly suppress and approximate the activation probability of the underlying physical parameters being incorrectly predicted as target privacy information at the tensor flow level. This ensures that when the system outputs the desensitized intermediate text downstream, the true underlying business element parameters are absolutely preserved and retained. For example, at the tensor level, the activation probability of the underlying physical parameters being predicted as "phone number" or "employee number" can be forcibly suppressed and approximate to 0, ensuring that the true underlying business elements are absolutely not destroyed.

[0037] Once the network finally confirms the target privacy information (non-IT technology dimension) with high confidence, a high-dimensional mapping replacement engine is executed. The target sequence is physically overwritten with a preset globally state-invariant mask constant (such as "...").<PII_NAME_MASK> " or"<PHONE_MASK> This process completely removes redundant entity-sensitive information that could disrupt subsequent feature space dimensionality reduction extraction logic, outputting desensitized intermediate text.

[0038] In one embodiment, extracting constant templates describing IT event types and implicit key-value pairs describing dynamic parameter values ​​from the de-identified intermediate text includes: Symbols in the de-identified intermediate text that do not possess technical semantics are marked as redundant nodes without business semantics, while symbols in the de-identified intermediate text that possess technical semantics are marked as valid business nodes. For each redundant node without business semantics in memory, the parent node of the redundant node without business semantics is redirected to the next valid business node of the redundant node without business semantics, and the redundant node without business semantics is deleted from the memory. A syntax tree is constructed using the remaining valid business nodes and the relationships between them. Semantic deduction is performed on the syntax tree, the implicit key-value pairs are obtained by stripping the dynamic parameters from the syntax tree, and placeholders are filled in the positions of the syntax tree corresponding to the dynamic parameters to obtain the constant template.

[0039] The steps described above in this embodiment can be implemented using a large language model. The anonymized intermediate text input is first fed into a highly customized lexical analyzer. When mapping discrete characters to tokens, the lexical analyzer uses a built-in noise decision tree to dynamically evaluate the semantic weights of the symbols. For purely decorative redundant symbols that lack substantial technical semantics (such as multi-layered nested braces abused by front-end scripts, unbounded consecutive dashes, and independent semicolons without logical connection), the lexical analyzer refuses to map them as valid business nodes, instead forcibly reducing their dimensionality and labeling them as "redundant nodes without business semantics"

[0040] Subsequently, before constructing a fixed-depth prefix parse tree (i.e., syntax tree) in memory, the system triggers an asynchronous intelligent pruning algorithm. This algorithm recursively traverses the initial node linked list, and once it detects a redundant node without business semantics, it immediately redirects the routing address of its parent node directly to its child node at the memory pointer level, physically removing it from the active memory graph, flattening the deeply nested redundant text, and outputting a lightweight abstract syntax tree containing only the rigid topological skeleton of IT events.

[0041] After outputting a lightweight abstract syntax tree containing only the rigid topological skeleton of IT events in memory, the system forcibly maps and injects placeholders (concretely instantiated as constants) into the dictionary lexical space at the bottom layer of the large language model. <param> The data engineering mission of this placeholder is to accurately define and characterize the boundaries of dynamically fluctuating business digital features in the underlying alarms within the feature solution space.

[0042] To map the lightweight abstract syntax tree to the feature space of the large language model in a high-dimensional manner, while breaking through the physical limitation bottleneck of graphics processor video memory in the enterprise private deployment environment, the system can strictly adopt the Low-Rank Adaptation (LoRA) mechanism to inject local domain knowledge into the pre-trained weights. Specifically, the system receives the remaining valid business nodes of the abstract syntax tree as input tensors, and performs dimensional relationship derivation: for the huge query and value dense weight matrix W0 (whose tensor dimension is defined as d rows and k columns, i.e., d x k) of the self-attention mechanism in the pre-training architecture of the large language model, the system forcibly freezes its global gradient calculation at the physical memory level and prohibits its update; instead, the system injects two extremely small-scale low-rank decomposition matrices A and B into the feature processing bypass. The dimension of the low-rank decomposition matrix A is configured as r rows and k columns, and the dimension of the low-rank decomposition matrix B is configured as d rows and r columns. To achieve extreme compression of video memory resources, the system imposes a strict rank constraint on the bypass: the rank parameter r is forcibly constrained to satisfy r<<min(d, k). When the model performs forward propagation calculation for the valid business nodes, the linear update logic of the weights is executed with dimensionality reduction suppression. Through this extremely compressed low-rank feature representation, the system completes the logical alignment of the high-dimensional heterogeneous feature space with a physical video memory overhead less than one-thousandth of that of the original pre-trained model. This alignment mechanism directly enables the downstream multi-head attention mechanism to maximize the output using cross-entropy loss calculation in the specific IT fault context <param> and provides basic architecture support for the confidence of the placeholder, thereby successfully deconstructing the constant skeleton and implicit key-value pairs.

[0043] In one embodiment, extracting the constant template describing the IT event type and the implicit key-value pair describing the value of the dynamic parameter from the desensitized intermediate state text includes: Extracting background information characterizing the information source from the original heterogeneous IT alarm text; Generating a hash signature according to the background information, and using the hash signature as the context prompt boundary of the large language model solution search space; Extracting the constant template describing the IT event type and the implicit key-value pair describing the value of the dynamic parameter from the desensitized intermediate state text based on the large language model.

[0044] For the high-entropy sequence entering the main link, a one-way hash function can be used to perform aggregation calculation on the IP address of the source device, the network port, and the type label of the underlying application component (such as "metadata.product.vendor_name="Oracle"") to generate a globally unique traceability metadata hash signature. This signature will serve as a prior context prompt boundary that limits the subsequent solution search space of the large language model and reduces the semantic hallucination rate.

[0045] In one embodiment, determining the standard slot corresponding to the dynamic parameter type under the structured data framework according to a preset semantic mapping table includes: When the dynamic parameter type does not exist in the semantic mapping table, the semantic similarity between the word vector corresponding to the dynamic parameter type and the word vector corresponding to each preset dynamic parameter type in the semantic mapping table is calculated, and the preset standard slot corresponding to the preset dynamic parameter type with the highest semantic similarity is taken as the standard slot corresponding to the dynamic parameter type.

[0046] Of course, if the dynamic parameter type exists in the semantic mapping table, the preset standard slot corresponding to the dynamic parameter type will be directly used as the standard slot corresponding to the dynamic parameter type.

[0047] Step S14 is the terminal exit of the system's core architecture. It aims to drive the execution of forced semantic alignment mapping across the feature space using the heterogeneous dynamic variable matrix (implicit key-value pairs) and abstract skeleton (constant template) extracted after multi-level washing and high-dimensional parsing, so as to completely eliminate the semantic gaps in the synonymous terminology representation of the original probe in the underlying architecture.

[0048] The system pre-registers and loads a general data model (CDM) in kernel mode that is strictly aligned with the principles of industry-standardized network security architecture frameworks (covering, but not limited to, OCSF or ASIM specifications). This ontology dictionary allocates a series of event categories, basic dictionaries, and standard slots in memory with standardized hierarchical topology specifications.

[0049] All heterogeneous variables representing the basic physical machine or virtual instance identifier (such as "Host_Name" and "Server_ID") are forcibly upgraded and aligned to standard slots (such as "device.ip", "device.name" or "src_endpoint.ip") according to data flow and hierarchical attributes.

[0050] Metadata and component ownership convergence: Completely correct and write the self-created heterogeneous product or software version identifier into the standard slots of the specified basic metadata ownership (such as "metadata.product.vendor_name" and "metadata.version").

[0051] Abnormal status standardization: The constant abstract template of the underlying fault tendency and the basic error parameters are mapped and corrected to the standardized attribute definition "status_id" or "status_detail", and the basic alarm classification is uniformly mapped to the higher-order event category variable "class_name" slot.

[0052] Based on the general ontology specification, the mapping engine automatically derives and establishes the dependency topology semantic edges between underlying components (such as the inclusion relationship edges derived based on the containerized orchestration environment).

[0053] Finally, the standardized structured data files described above are delivered to the external bus.

[0054] To enable those skilled in the art to intuitively and deeply understand the underlying logic of the data processing method of this invention at the microscopic tensor flow level, the following will further elaborate on the data processing method provided in the embodiments of this application by combining an extremely harsh long alarm text of the virtual underlying database containing multi-source heterogeneous parameters and strongly coupled structured bias noise.

[0055] Initial dirty data feature input: Suppose that a multi-protocol gateway deployed at the system edge access layer passively receives an Oracle database performance anomaly alarm text via the Syslog protocol during a period of high business concurrency. This alarm text is forcibly concatenated from a traditional front-end monitoring script through multi-level hard-coding. This one-dimensional character sequence stream X exhibits the following highly discrete and coupled high-entropy state: [10.0.0.1] node_type=Oracle_Exadata - FATAL_ERR: ORA-04031: unable to allocate 4096 bytes of shared memory ("shared pool","select * from user_tables where rownum<100","sql area","tmp_01") !!! current_mem_usage=98.5% |threshold=90.0% ;; Thread_PID: 99999 ;; MAC_Addr: 00:AA:BB:CC:DD:EE **noise**:: ————Project Team Contact Person: On-Duty Personnel A, Phone: 13800000000; Please also notify 99999 Information entropy measurement and tensor classification: After receiving the sequence stream, the heuristic probe initiates the Shannon information entropy calculation module in memory. Due to the extreme mixing of standard IPv4 format, low-level SQL code snippets (containing numerous quotation marks and operators), hexadecimal MAC addresses, floating-point performance metrics (98.5%), logically meaningless consecutive punctuation marks ("!!!", ";;", "**noise**::"), and Chinese character strings, the probe calculates that the Shannon information entropy H(X) of the sequence reaches an extremely high discrete peak (far exceeding the system's preset H_threshold critical value). Simultaneously, the closed-loop structured feature density D approaches zero.

[0056] Based on the above two-dimensional calculation results, the routing scheduler marks the data packet as a "high-entropy discrete coupled sequence", completely rejects its entry into the bypass basic deserialization channel with a time complexity of O(n), and uses "node_type=Oracle_Exadata" to generate a priori traceable metadata hash, forcibly pushing it into the backbone deep neural network processing link.

[0057] Adversarial mask stripping with morphological obfuscation: This long text contains a highly destructive extreme boundary condition: the numeric string "99999" appears twice. The first time is as the core system thread parameter "Thread_PID: 99999"; the second time is as the human contact code "Please also notify 99999". Traditional regular expression cleaning or basic named entity recognition models are extremely prone to global erasure when faced with this pure numeric sequence, leading to a catastrophic loss of underlying code features.

[0058] During global topology scanning, the bidirectional Transformer encoder in the main semantic perception branch keenly captures leading words such as "contact," "telephone," and "notification" at the end of the sequence and outputs high entity prediction confidence. At this point, the bias removal penalty and domain compliance verification modules are forcibly involved in the calculation.

[0059] For the first "99999" with the preceding context "Thread_PID:", the compliance verification module calculates an extremely high domain semantic dependency distance, instantly generating a negative confidence penalty scalar at the micro level, forcibly suppressing the entity prediction activation probability of this fragment, ensuring the complete preservation of basic operational features. Conversely, for the second "99999" guided by "notification" and "on-duty personnel A" and "13800000000", the system successfully performs high-dimensional space mapping, overwriting them in situ as global state-invariant mask constants.

[0060] After elution, a semi-pure feature sequence is output: [10.0.0.1] node_type=Oracle_Exadata - FATAL_ERR: ORA-04031: unable toallocate 4096 bytes of shared memory ("shared pool","select * from user_tables where rownum<100","sql area","tmp_01") !!! current_mem_usage=98.5% |threshold=90.0% ;; Thread_PID: 99999 ;; MAC_Addr: 00:AA:BB:CC:DD:EE **noise**:: ————Project team contact:<PII_NAME_MASK> Telephone:<PHONE_MASK> Please also notify<PHONE_MASK> Skeleton deconstruction and implicit feature extraction: The intermediate tensor is fed into a fixed-depth prefix parse tree engine. The lexical analyzer marks purely decorative characters such as "!!!", ";;", "**noise**::", and consecutive hyphens "————" as redundant nodes without business semantics, and performs physical redirection pruning in the memory graph to flatten the sequence into a high-density business code skeleton.

[0061] The pruned, clean sequence is input into a finely tuned, compact large language model. Through multi-head attention and the computation of dimensionality reduction matrices A and B, the model deeply understands that "10.0.0.1", "4096", "shared pool"... "tmp_01", "98.5%", "90.0%", "99999", and "00:AA:BB:CC:DD:EE" are all instantaneous dynamic attributes that triggered this specific anomaly. The model calculates cross-entropy loss and extracts these discrete features into independent key-value pair matrices at the output layer, completely replacing the corresponding positions in the original sequence with the first type of custom feature blocking token. <param> ".

[0062] Output separation space: Constant template: [ <param> node_type= <param> - FATAL_ERR: ORA-04031: unable to allocate <param> bytes of shared memory <param> current_mem_usage= <param>|threshold= <param> Thread_PID: <param> MAC_Addr: <param> Project team contact person:<PII_NAME_MASK> Telephone:<PHONE_MASK> Please also notify<PHONE_MASK> (Completely converged system behavior signature) Implicit key-value pair dictionary: Contains heterogeneous variables such as source IP, number of allocated bytes, SQL statement fragments, current value, threshold, PID and MAC.

[0063] The final dimensionality reduction reconstruction to a general data model: At the end of the pipeline, the aforementioned discrete heterogeneous variables and constant skeletons are forcibly pushed into a unified ontology mold, preferably but not limited to OCSF and ASIM architectures. The mapping engine performs semantic alignment across the feature space: mounting "10.0.0.1" to the "device.ip" slot, mounting "ORA-04031" to the "status_id" slot, strictly converging the performance metrics "98.5%" and "90.0%" into the "observability.metrics" topology branch, and injecting the desensitized state into the "enrichments" validation domain.

[0064] Ultimately, this extremely malicious piece of concatenated, dirty data was reconstructed into a normalized JSON entity code block containing rich technical context and absolute semantic consistency, which was then directly provided to the downstream unified data lake for ingestion. { "event_time": "2026-02-26T07:07:46.000Z", "metadata": { "product": { "vendor_name": "Oracle", "name": "Database_Exadata }, "parser_route": "Deep_LLM_AST_Main_Pipeline", "confidence_score": 0.992 }, "class_name": "Database Performance&Fault", "device": { "ip": "10.0.0.1", "mac": "00:AA:BB:CC:DD:EE" }, "status_id": "ORA-04031", "status_detail": "unable to allocate bytes of shared memory", "observability": { "metrics": { "current_mem_usage_percent": 98.5, "threshold_percent": 90.0, "failed_allocate_bytes": 4096 }, "process": { "thread_pid": 99999 } }, "database_context": { "pool_name": "shared pool", "sql_statement_snippet": "select * from user_tables where rownum<100", "target_area": "sql area", "object_name": "tmp_01" }, "enrichments": { "pii_masked_status": true, "masked_constants": [ "<PII_NAME_MASK>", "<PHONE_MASK>" ] } } This application's embodiments, through deep reconstruction of the underlying data tensor flow chain, creatively integrate an anti-bias sequence labeling model, a fixed-depth abstract syntax tree, and a lightweight fine-tuning architecture for large language models. Without altering any existing front-end probe hard-coded splicing logic, the following technical benefits are achieved: It completely abandons traditional regular expressions with extremely high maintenance costs and heuristic shallow statistical algorithms that are prone to state machine explosion. By introducing a dual-branch anti-bias masking engine, the system forces the long-tailed unstructured artificial noise (such as thousands of different combinations of duty contacts) that causes dimensional expansion to be physically overwritten into highly convergent constant abstract symbols within microseconds. This mathematically prevents the geometrical explosion of the downstream log constant template pool, reduces the computational and storage overhead by several orders of magnitude, and completely liberates senior experts from manual maintenance work.

[0065] When faced with malicious IT corpora containing hexadecimal stack traces and network hardware addresses, this system abandons destructive physical character deletion mechanisms. Its unique domain-specific compliance checks and negative confidence penalty mechanisms ensure that the model can accurately locate and globally shield personally identifiable information while maintaining 100% high-fidelity retention of underlying technical code with extremely similar formats (such as purely numeric thread PIDs and MAC addresses). This approach satisfies the most stringent enterprise-level data compliance outbound review while guaranteeing the physical completeness of basic observation elements.

[0066] Leveraging efficient parameter fine-tuning technology injected into a compact large language model and cross-entropy semantic inference capabilities, the system breaks through the limitations of rigid character matching. The model is no longer confined to rigid spaces or specific character definitions, but instead deeply understands the essential topological differences between the system's constant operational skeleton and the instantaneous dynamic features that trigger anomalies through a contextual multi-head attention mechanism. Even after front-end business upgrades and significant semantic drift in probe assembly formats, this system can still achieve plug-and-play accurate high-dimensional parsing and extraction without manual intervention.

[0067] The internationally recognized data model and ontology mapping engine built into the terminal architecture of this invention completely eliminates naming discrepancies across heterogeneous systems and different hardware brands on the same observation dimension. Heterogeneous fields that were originally in a complex, discrete, and coupled state are forcibly reduced in dimension and reconstructed, and rigorously packaged into structurally aligned preferred, but not limited to, OCSF or ASIM-compatible entities. This not only eradicates the silo effect in enterprise-level data lakes, but also provides backend intelligent analysis algorithms with a lossless, pure, and absolutely semantically consistent basic observation map input.

[0068] It should be understood that although the steps in the flowchart above are shown sequentially as indicated by the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowchart above may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the sub-steps or stages of other steps.

[0069] In one embodiment, please refer to Figure 2 As shown, a data processing system is provided, characterized in that it includes: The positioning module 201 is used to receive the original heterogeneous IT alarm text and, when it is determined that the original heterogeneous IT alarm text does not have a structured pattern, locate the target privacy information in the non-IT technology dimension from the original heterogeneous IT alarm text. Replacement module 202 is used to replace the target privacy information with a mask constant that remains unchanged in global state and output desensitized intermediate text; The extraction module 203 is used to extract constant templates representing IT event types and implicit key-value pairs representing dynamic parameter values ​​from the desensitized intermediate text; the implicit key-value pairs include the correspondence between dynamic parameter types and parameter values; The filling module 204 is used to determine a structured data framework based on the constant template, and to determine the standard slots corresponding to the dynamic parameter types under the structured data framework based on a preset semantic mapping table, and to fill the parameter values ​​corresponding to the dynamic parameter types into the standard slots to obtain a structured data file; the semantic mapping table defines the correspondence between preset dynamic parameter types and preset standard slots.

[0070] It should be understood that, for the sake of brevity, some of the content described in the previous embodiments will not be repeated in this embodiment.

[0071] In one embodiment, please refer to Figure 3 As shown, an electronic device is provided, including a processor 301 and a memory 302. The memory 302 stores a computer program, and the processor 301 executes the computer program to implement the steps of the method described above, which will not be repeated here.

[0072] Processor 301 can be an integrated circuit chip with signal processing capabilities. The processor 301 can be a general-purpose processor, including a CPU (Central Processing Unit), NP (Network Processor), etc.; it can also be a DSP (Digital Signal Processor), ASIC (Application Specific Integrated Circuit), FPGA (Field Programmable Gate Array), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor can be a microprocessor or any conventional processor.

[0073] The memory 302 may include, but is not limited to, RAM (Random Access Memory), ROM (Read Only Memory), PROM (Programmable Read Only Memory), EPROM (Erasable Programmable Read-Only Memory), and EEPROM (Electrically Erasable Programmable Read Only Memory).

[0074] Those skilled in the art will understand that Figure 3 The structure shown in the figure is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the application of the present application. Specific electronic devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0075] Based on the same inventive concept, embodiments of this application also provide a computer-readable storage medium, such as a floppy disk, optical disk, hard disk, flash memory, USB flash drive, SD (Secure Digital) card, MMC (Multi-Media Card), etc. The computer-readable storage medium stores a computer program, which, when executed by at least one processor, implements the steps of the methods in the above embodiments, which will not be repeated here.

[0076] Based on the same inventive concept, embodiments of this application also provide a computer program product, including a computer program that, when executed by a processor, implements any of the methods described above.

[0077] The program code for executing the computer program product of this application can be written in any combination of one or more programming languages. The program code can be executed entirely on the user device, partially on the user device, as a standalone software package, partially on the user device and partially on a remote device, or entirely on a remote device.

[0078] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, optical storage, etc.) containing computer-usable program code.

[0079] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the document does not necessarily refer to the same embodiment, nor is it a mutually exclusive, independent, or alternative embodiment. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0080] As illustrated herein, unless the context clearly indicates otherwise, the words “a,” “an,” “an,” and / or “the” do not specifically refer to the singular and may also include the plural. Generally speaking, the terms “comprising” and “including” only indicate the inclusion of explicitly identified steps and elements, which do not constitute an exclusive list, and the method or apparatus may also include other steps or elements.

[0081] The definitions used herein, such as the terms “having,” “may have,” “comprising,” or “may include,” indicate the presence of the corresponding function, operation, element, etc., and do not limit the presence of one or more other functions, operations, elements, etc. Furthermore, it should be understood that the terms “comprising” or “having” as used herein indicate the presence of the features, figures, steps, operations, elements, components, or combinations thereof described in the specification, without excluding the presence or addition of one or more other features, figures, steps, operations, elements, components, or combinations thereof.

[0082] The embodiments described above are merely examples of several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application.

Claims

1. A data processing method, characterized in that, include: Receive raw heterogeneous IT alarm text, and when it is determined that the raw heterogeneous IT alarm text does not have a structured pattern, locate target privacy information that is not related to IT technology from the raw heterogeneous IT alarm text; Replace the target privacy information with a mask constant that remains unchanged in the global state, and output the de-identified intermediate text; Extract constant templates representing IT event types and implicit key-value pairs representing dynamic parameter values ​​from the de-identified intermediate text; the implicit key-value pairs include the correspondence between dynamic parameter types and parameter values; A structured data framework is determined based on the constant template, and a standard slot corresponding to the dynamic parameter type under the structured data framework is determined based on the preset semantic mapping table. The parameter values ​​corresponding to the dynamic parameter type are filled into the standard slots to obtain a structured data file. The semantic mapping table defines a correspondence between preset dynamic parameter types and preset standard slots; Extract constant templates representing IT event types and implicit key-value pairs representing dynamic parameter values ​​from the de-identified intermediate text, including: Symbols in the de-identified intermediate text that do not possess technical semantics are marked as redundant nodes without business semantics, while symbols in the de-identified intermediate text that possess technical semantics are marked as valid business nodes. For each redundant node without business semantics in memory, the parent node of the redundant node without business semantics is redirected to the next valid business node of the redundant node without business semantics, and the redundant node without business semantics is deleted from the memory. A syntax tree is constructed using the remaining valid business nodes and the relationships between them. Semantic deduction is performed on the syntax tree, the implicit key-value pairs are obtained by stripping the dynamic parameters from the syntax tree, and placeholders are filled in the positions of the syntax tree corresponding to the dynamic parameters to obtain the constant template.

2. The data processing method as described in claim 1, characterized in that, Determining that the original heterogeneous IT alarm texts do not have a structured pattern includes: Calculate the Shannon information entropy and structured feature density corresponding to the original heterogeneous IT alarm text; When the Shannon information entropy is determined to be greater than or equal to a preset entropy threshold, or when the structured feature density is determined to be less than or equal to a preset density threshold, the original heterogeneous IT alarm text is determined to have no structured regularity.

3. The data processing method as described in claim 2, characterized in that, The method further includes: When the Shannon information entropy is determined to be less than the preset entropy threshold and the structured feature density is greater than the preset density threshold, the original heterogeneous IT alarm text is directly deserialized.

4. The method as described in claim 1, characterized in that, Locating target privacy information from the original heterogeneous IT alert text that is not related to IT technology, including: The main semantic awareness branch locates preliminary privacy information from the original heterogeneous IT alarm text in non-IT technical dimensions; The bias removal penalty and domain compliance verification module determines the domain semantic dependency distance of the preliminary privacy information based on its text location. When it is determined that the preceding information of the preliminary privacy information contains preset IT technical terminology features, a negative confidence penalty scalar is generated to prevent the preliminary privacy information from being marked as the target privacy information. When it is determined that the preceding information of the preliminary privacy information does not contain preset IT technical terminology features, the preliminary privacy information is directly marked as the target privacy information.

5. The method as described in claim 1, characterized in that, Extract constant templates representing IT event types and implicit key-value pairs representing dynamic parameter values ​​from the de-identified intermediate text, including: Extract background information representing the source of the original heterogeneous IT alarm text; A hash signature is generated based on the background information, and the hash signature is used as the contextual hint boundary of the search space of the large language model solution; Based on the large language model, constant templates representing IT event types and implicit key-value pairs representing dynamic parameter values ​​are extracted from the de-identified intermediate text.

6. The data processing method as described in claim 1, characterized in that, Based on a preset semantic mapping table, the standard slots corresponding to the dynamic parameter types under the structured data framework are determined, including: When the dynamic parameter type does not exist in the semantic mapping table, the semantic similarity between the word vector corresponding to the dynamic parameter type and the word vector corresponding to each preset dynamic parameter type in the semantic mapping table is calculated, and the preset standard slot corresponding to the preset dynamic parameter type with the highest semantic similarity is taken as the standard slot corresponding to the dynamic parameter type.

7. A data processing system, characterized in that, include: The positioning module is used to receive raw heterogeneous IT alarm texts and, when it is determined that the raw heterogeneous IT alarm texts do not have a structured pattern, locate target privacy information that is not related to IT technology from the raw heterogeneous IT alarm texts. The replacement module is used to replace the target privacy information with a mask constant that remains unchanged in the global state and output the desensitized intermediate text. The extraction module is used to extract constant templates representing IT event types and implicit key-value pairs representing dynamic parameter values ​​from the desensitized intermediate text; the implicit key-value pairs include the correspondence between dynamic parameter types and parameter values; The filling module is used to determine a structured data framework based on the constant template, and to determine the standard slots corresponding to the dynamic parameter types under the structured data framework based on a preset semantic mapping table, and to fill the parameter values ​​corresponding to the dynamic parameter types into the standard slots to obtain a structured data file. The semantic mapping table defines a correspondence between preset dynamic parameter types and preset standard slots; Extract constant templates representing IT event types and implicit key-value pairs representing dynamic parameter values ​​from the de-identified intermediate text, including: Symbols in the de-identified intermediate text that do not possess technical semantics are marked as redundant nodes without business semantics, while symbols in the de-identified intermediate text that possess technical semantics are marked as valid business nodes. For each redundant node without business semantics in memory, the parent node of the redundant node without business semantics is redirected to the next valid business node of the redundant node without business semantics, and the redundant node without business semantics is deleted from the memory. A syntax tree is constructed using the remaining valid business nodes and the relationships between them. Semantic deduction is performed on the syntax tree, the implicit key-value pairs are obtained by stripping the dynamic parameters from the syntax tree, and placeholders are filled in the positions of the syntax tree corresponding to the dynamic parameters to obtain the constant template.

8. An electronic device, characterized in that, It includes a processor and a memory, wherein the memory stores a computer program, and the processor executes the computer program to implement the method as described in any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by at least one processor, implements the method as described in any one of claims 1-6.

Images