A method for intelligent writing of platform data based on big data

Abstract

This invention discloses a method for intelligent data writing on a big data platform, relating to the field of data storage and computing technology. The method includes: enhancing the context and metadata of a standardized order record set and integrating it with basic metadata to generate an enhanced order record set; mapping the enhanced order record set to field feature vectors, analyzing the implicit dependencies between these vectors using a graph neural network, calculating dependency weights in the vector space, constructing an implicit dependency graph, and generating ordered write batches using a hierarchical scheduling algorithm; performing multi-dimensional knowledge embedding processing on each standardized order record in the ordered write batch to generate an indexed order data write package; and using an RPA algorithm to perform write orchestration on the indexed order data write package according to the ordered write batch, outputting the write execution record. This invention improves the semantic consistency, processing efficiency, and system robustness of data writing.

Description

Technical Field

[0001] This invention relates to the field of data storage and computing technology, and in particular to a method for intelligent data writing to a platform based on big data. Background Technology

[0002] With the rapid development of e-commerce and fintech, efficient data writing to platforms has become a key technology supporting real-time transactions, data analysis, and business decision-making. In existing technologies, log stream-based data processing systems are widely used for order data collection and storage. For example, distributed message queues (such as Kafka or RabbitMQ) are used to asynchronously transmit transaction logs, and ETL (Extract-Transform-Load) processes are used to standardize and persist the data. These systems typically rely on predefined data patterns and rule engines to parse and map raw logs, generating structured records and writing them to the database in batches. Furthermore, some optimization solutions introduce lightweight caching mechanisms (such as Redis) or parallel computing frameworks (such as Spark Streaming) to improve data throughput and processing efficiency.

[0003] Existing solutions often generate write batches based on static rules or shallow statistical features (such as field co-occurrence frequency), neglecting the potential semantic relationships and temporal dependencies in order data. Traditional methods have limited utilization of metadata, typically using it only as auxiliary tags without fully integrating it with business context for enhanced analysis. This limitation restricts the optimization space for write scheduling strategies, especially when dealing with massive amounts of heterogeneous data, potentially leading to redundant I / O operations or wasted storage resources. Currently, industry solutions for multi-platform data synchronization mainly rely on manual operations. For example, suppliers on government procurement cloud platforms need to repeatedly enter data into multiple independent systems (such as e-commerce platforms, ERP, and financial systems), which is inefficient and prone to errors. Summary of the Invention

[0004] In view of the aforementioned existing problems, the present invention is proposed.

[0005] Therefore, this invention provides a method for intelligent data writing on a platform based on big data to solve the problem of low writing efficiency caused by reliance on static rules and shallow features.

[0006] To solve the above-mentioned technical problems, the present invention provides the following technical solution: This invention provides a method for intelligent writing of platform data based on big data, which includes accessing user transaction log data streams and attaching basic metadata to each user transaction log data stream to form order access data; The order data is parsed and standardized, and basic business identifiers are extracted to obtain a set of standardized order records. The standardized order record set is enhanced with context and metadata, and then integrated with the basic metadata to generate an enhanced order record set; The enhanced order record set is mapped to field feature vectors. The implicit dependencies between field feature vectors are analyzed by graph neural networks, and the dependency weights are calculated in the vector space to construct an implicit dependency graph. An ordered write batch is generated by a hierarchical scheduling algorithm. Perform multi-dimensional knowledge embedding processing on each standardized record of an order in the ordered write batch to generate an indexed order data write package; The RPA algorithm is used to perform write orchestration on indexed order data write packages according to ordered write batches, and the write execution record is output.

[0007] As a preferred embodiment of the platform data intelligent writing method based on big data described in this invention, the steps of attaching basic metadata to each user transaction log data stream to form order access data are as follows: The basic metadata is bound to the user transaction log data stream to form a single order data record; Individual order data records are parsed and structured in a unified format and then aggregated into order access data.

[0008] As a preferred embodiment of the platform data intelligent writing method based on big data described in this invention, the basic metadata is obtained by organizing the source identifier, access timestamp, and access channel identifier automatically recorded when accessing each user transaction log data stream.

[0009] As a preferred embodiment of the platform data intelligent writing method based on big data described in this invention, the steps of parsing and standardizing the order access data, extracting basic business identifiers, and obtaining a standardized order record set are as follows: The transaction log field and basic metadata field are read from a single order data record in the order access data, split, and converted into an order data representation structure. Perform validity checks and corrections on the order data representation structure to obtain standardized order records; Extract basic business identifiers from the standardized order records and associate them with the corresponding standardized order records to form a set of standardized order records.

[0010] As a preferred embodiment of the platform data intelligent writing method based on big data described in this invention, the steps of enhancing the context and metadata of the standardized order record set and integrating it with the basic metadata to generate an enhanced order record set are as follows: Based on the basic business identifier, the corresponding attribute is retrieved from the reference information table, and the retrieval results are appended to the standardized record set of the order to obtain the reference attribute record set; The reference attribute record set is standardized with uniform time, time field labels are generated, and time period labels and historical transaction summaries are appended to the reference attribute record set to obtain the time context record set; Parse the access channel identifier from the time context record set, obtain the geographic location information, match the valid promotional information in the business activity table, and append it to the time context record set to obtain the geographic activity context record set. The integrity and consistency of the geographic activity context record set are checked, and the record set is merged with the basic metadata at the field level to generate an enhanced order record set.

[0011] As a preferred embodiment of the platform data intelligent writing method based on big data described in this invention, the historical transaction summary is obtained by retrieving relevant transaction data within the corresponding time window based on the business identifier and time field in the reference attribute record set, and then aggregating and compressing the retrieval results.

[0012] As a preferred embodiment of the platform data intelligent writing method based on big data described in this invention, the steps of mapping the enhanced order record set into field feature vectors, analyzing the implicit dependencies between field feature vectors through a graph neural network, calculating dependency weights in the vector space, and constructing an implicit dependency graph are as follows. Extract feature values ​​for each enhanced order record in the enhanced order record set by field and map them into field feature vectors; Each field feature vector is used as a node. In the graph neural network, node features and adjacency relationships are used to transmit and aggregate information, and the implicit dependencies between field feature vectors are extracted. Based on implicit dependencies, the correlation measurement of the field feature vectors of the source node and the target node is calculated in the vector space to obtain the dependency weights; The connection directions between nodes are determined based on implicit dependencies, and the dependency weights are assigned to the corresponding edges to construct an implicit dependency graph.

[0013] As a preferred embodiment of the intelligent data writing method for a platform based on big data described in this invention, the specific steps for generating ordered write batches through a hierarchical scheduling algorithm are as follows: Based on the in-degree and dependency weight of the nodes in the implicit dependency graph, the nodes are divided into different levels, and within each level, the order of the nodes is adjusted according to the strength of the dependencies between them. The hierarchical and sorted nodes are combined in batches to form ordered write batches.

[0014] As a preferred embodiment of the platform data intelligent writing method based on big data described in this invention, the specific steps for performing multi-dimensional knowledge embedding processing on each standardized order record in the ordered writing batch to generate an indexed order data writing package are as follows: Each standardized record of an order written in the batch is sliced ​​by field in a multidimensional way to form an initial feature fragment. The initial feature fragment is then transformed into a semantic atomic fragment through semantic encoding and vectorization. Calculate the dependency strength between semantic atomic fragments, perform multi-hop dependency propagation through meta-path expansion, and obtain the set of embedded vectors after multi-dimensional interaction; Different dimensions of embedded vectors are mapped to a unified semantic space, and conflicting fields are automatically corrected through a context voting mechanism to generate a unique index anchor for each standardized record of an order. The embedded vector set is bucketed according to the unique index anchor point, and a three-layer encapsulation structure is built in the corresponding write group to form an indexed order data write package; the three-layer encapsulation structure includes a vector index table, semantic alignment labels, and original normalized fields.

[0015] As a preferred embodiment of the intelligent data writing method for a platform based on big data described in this invention, the steps of performing write orchestration on indexed order data write packages according to ordered write batches using the RPA algorithm and outputting write execution records are as follows. The indexed order data is written to the package and then sequentially input into the RPA algorithm in ordered batches to generate a queue to be written. The queue to be written is persisted using the RPA algorithm. After the write is completed, a corresponding write confirmation set is generated. At the same time, the index service is triggered to perform an index insertion operation and return an index insertion receipt. The write confirmation set is integrated with the index insert receipt and recorded as a write execution record.

[0016] The beneficial effects of this invention are as follows: By enhancing the context and metadata of standardized order records, the semantic extension and contextual reconstruction of the original transaction data are realized, significantly improving the integrity and business interpretability of the data; and by mapping the enhanced order records into field feature vectors through graph neural networks, by modeling the implicit dependencies between vectors and calculating the dependency weights, an implicit dependency graph reflecting the real business logic is constructed, realizing the automatic discovery and quantitative analysis of complex non-explicit associations, improving the semantic consistency of data writing, processing efficiency and system robustness. Attached Figure Description

[0017] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 A flowchart illustrating a method for intelligent data writing to a big data-based platform.

[0019] Figure 2 This is a flowchart for processing order data.

[0020] Figure 3 A flowchart for enhancing the generation of order record sets.

[0021] Figure 4 A flowchart for writing indexed order data. Detailed Implementation

[0022] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0023] Many specific details are set forth in the following description in order to provide a full understanding of the invention. However, the invention may also be practiced in other ways different from those described herein, and those skilled in the art can make similar extensions without departing from the spirit of the invention. Therefore, the invention is not limited to the specific embodiments disclosed below.

[0024] Secondly, the term "one embodiment" or "embodiment" as used herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation of the present invention. The phrase "in one embodiment" appearing in different places in this specification does not necessarily refer to the same embodiment, nor is it a single or selective embodiment that is mutually exclusive with other embodiments.

[0025] Reference Figures 1-4This is one embodiment of the present invention, which provides a method for intelligent data writing to a platform based on big data, including the following steps: S1. Access user transaction log data streams and attach basic metadata to each user transaction log data stream to form order access data.

[0026] It should be noted that the user transaction log data stream includes transaction time, transaction type, transaction amount, transaction status, and product information.

[0027] S1.1 Bind the basic metadata with the user transaction log data stream to form a single order data record.

[0028] Furthermore, the system receives a user transaction log and parses out the transaction fields (transaction time, transaction type, transaction amount, transaction status, and product information, etc.); it standardizes the format and validates the types of the transaction fields and basic metadata, fills in or marks missing or abnormal items, and performs idempotency / deduplication checks; it generates a unique order identifier based on a hash strategy, and merges and encapsulates the transaction fields, basic metadata, unique identifier, and validation summary in a unified structure to generate a single order data record. The basic metadata is compiled from the source identifier, access timestamp, and access channel identifier automatically recorded when accessing each user transaction log data stream.

[0029] It should be noted that the above content has been agreed to by the user and is used for legitimate purposes.

[0030] S1.2. Perform unified format parsing and structured processing on individual order data records, and summarize them into order access data.

[0031] Furthermore, the transaction fields and basic metadata are formatted and converted to unify the field names, time formats, and amount precision from different sources to a standard specification. The transaction fields and basic metadata are also classified and hierarchically organized to ensure that core business fields (such as transaction amount and product number) and auxiliary information fields (such as access channel and source identifier) ​​can be mapped in a unified structure. Field integrity and consistency are verified, redundant or abnormal data is removed, a standardized field table is generated, and the data is summarized and organized to form order access data.

[0032] S2. Perform format parsing and standardization processing on the order access data, extract basic business identifiers, and obtain a set of standardized order records.

[0033] S2.1 Read the transaction log field and basic metadata field from a single order data record in the order access data, split them, and convert them into an order data representation structure.

[0034] Furthermore, the system sequentially reads each order data record from the order access data, breaks down the transaction fields and basic metadata, and performs type conversion and standardization mapping on each field using ETL data preprocessing methods. For example, it unifies the time to a standard time format, converts the amount to a decimal form with uniform precision, and maps the status and type to normalized enumeration values. The transaction fields and basic metadata after the field conversion are then encapsulated in a unified structure to form an order data representation structure that clearly distinguishes business information from metadata information.

[0035] S2.2 Perform validity checks and corrections on the order data representation structure to obtain standardized order records.

[0036] Furthermore, the integrity of the order data representation structure fields is verified to ensure that the transaction time, transaction type, transaction amount, transaction status, product information, source identifier, access timestamp, and access channel identifier are correctly parsed. Then, a validity check is performed, such as verifying whether the time field conforms to a unified time format, whether the amount field is non-negative and has the correct decimal precision, and whether the transaction status and transaction type fall within the predefined enumeration range (e.g., transaction status can be limited to "success, failure, and processing," and transaction type can be limited to "payment, refund, and cancellation"). For missing, abnormal, or inconsistent fields, corrections are made through rule correction, default value filling, or backtracking to the original logs. All fields that have passed verification and correction are reorganized into a unified data structure, outputting standardized order records that meet the standardization requirements. It should be noted that the range of enumerated values ​​is a finite set of values ​​defined based on the actual value distribution of historical transaction data and business logic constraints.

[0037] S2.3 Extract basic business identifiers from the standardized order records and associate them with the corresponding standardized order records to form a set of standardized order records.

[0038] Furthermore, basic business identifiers, including key fields such as order number, user identifier, product number, and merchant identifier, are extracted from the standardized order records. The extracted basic business identifiers are then associated with the corresponding standardized order records for storage, ensuring that each record can be quickly located and uniquely identified through the business identifier. All associated records are then aggregated to form a complete set of standardized order records.

[0039] S3. Enhance the context and metadata of the standardized order record set, and integrate it with the basic metadata to generate an enhanced order record set.

[0040] S3.1. Based on the basic business identifier, retrieve the corresponding attribute in the reference information table, and append the retrieval results to the standardized record set of the order to obtain the reference attribute record set.

[0041] Furthermore, the corresponding attributes that match the basic business identifier are retrieved one by one in the reference information table (e.g., matching product category, brand, specifications, and supplier based on product number); the retrieved attribute information is then appended and merged with the original standardized order records one by one to form a set of reference attribute records containing basic information and reference attributes. It should be noted that the reference information table is obtained by structuring and summarizing historical transaction data and external auxiliary data (including industry standards, merchant information and public basic data) in the user transaction log data stream. It includes static attributes, classification information and extended fields related to basic business identifiers, and is stored as a searchable data table.

[0042] S3.2. Standardize the reference attribute record set for time, generate time field labels, and attach the time period labels and historical transaction summaries to the reference attribute record set to obtain the time context record set.

[0043] Furthermore, the time fields in the reference attribute record set are uniformly mapped to standard time zones and normative time formats, and the time is decomposed to generate time field labels, including date, hour, weekday, month, and time period labels (e.g., AM, PM, and PM). The generated time field labels, time period labels, and historical transaction summaries are appended one by one and the aggregated time window and data source information are saved, outputting a time context record set containing time labels and historical summaries. It should be noted that the historical transaction summary is obtained by retrieving relevant transaction data within the corresponding time window based on the business identifier and time field in the reference attribute record set, and then aggregating and compressing the retrieval results.

[0044] S3.3 Parse the access channel identifier from the time context record set, obtain the geographical location information, match the valid promotional information in the business activity table, and append it to the time context record set to obtain the geographical activity context record set.

[0045] Furthermore, the system reads the access channel identifiers from the time context record set one by one, performs standardized format conversion and uniqueness parsing on the access channel identifiers, and generates channel identifier results that can be used for geospatial analysis. Based on the channel identifier results, it performs access terminal IP location, merchant address mapping, and channel metadata reverse lookup to obtain geographical location information such as latitude and longitude, city / province, and business district. At the same time, using the order's basic business identifier, time field, and parsed geographical location as matching keys, it retrieves valid promotional information within the corresponding time window and applicable geographical area from the business activity table, including activity identifier, activity type, applicable scope, effective time, and discount details. The matching confidence score is calculated for the retrieval results, expressed as: ; In the formula, It is the matching confidence level; It is a weighted coefficient for time-match similarity. It is the time matching similarity, which represents the degree of matching between the order occurrence time and the effective time interval of the promotion activity, with a value range of [0,1]. It is a weighted coefficient for the similarity of geographical location matching. It is the geographic location matching similarity, which represents the degree of coverage between the geographic location of the order and the applicable scope of the promotional activity (such as city, business district), and the value range is [0,1]. It is a weighted coefficient for the similarity of business identifier matching. It is the business identifier matching similarity, which represents the degree of fit between the order business type (payment, refund, and cancellation, etc.) and the applicable business category of the promotional activity, with a value range of [0,1]. Time-based matching similarity is evaluated by comparing the target fields in the order record with the corresponding applicable fields in the promotional activity, based on their matching relationship. For example, time-based matching similarity can be determined by comparing the order occurrence time with the effective time range of the promotional activity to see if the order time falls within the activity range and to what extent, thus providing a matching degree value between 0 and 1. Geographic location matching similarity and business identifier matching similarity also use the above operations, comparing the order access location with the applicable scope of the promotional activity and the order business type with the activity business category, respectively, to form a quantifiable matching degree result. The geographic location information, the matched promotional information, and their source and confidence fields are appended back to the original time context record set one by one to form a geographic activity context record set that includes geographic and activity context extended fields; It should be noted that the business activity table is generated by structuring existing marketing data, promotion rules, and activity logs, and stores the activity identifier, scope of application, validity period, and discount information into a searchable data table.

[0046] S3.4 Perform integrity and consistency checks on the geographic activity context record set, and merge it with the basic metadata at the field level to generate an enhanced order record set.

[0047] Furthermore, the system checks for missing fields or null values ​​in the geographic activity context record set and supplements them by calling historical backups. It compares the values ​​of key fields such as order identifier, timestamp, geographic location, and business type (e.g., payment, refund, cancellation, and promotional participation) across different sources to ensure consistency. When the same field has inconsistent values ​​across multiple sources, it is identified as a conflicting field, and conflicting fields are prioritized and corrected. While ensuring record accuracy, the validated geographic activity context records are aligned with the basic metadata line by line and merged at the field level. The merged content includes basic information such as source identifier, access timestamp, channel identifier, and system serial number. The output is an enhanced order record set that simultaneously possesses enhanced contextual information and basic metadata.

[0048] S4. Map the enhanced order record set to field feature vectors, analyze the implicit dependencies between field feature vectors through graph neural networks, calculate dependency weights in the vector space, construct an implicit dependency graph, and generate ordered write batches through a hierarchical scheduling algorithm.

[0049] It should be noted that existing methods typically encode the fields in the enhanced order record set independently, and dependency analysis often relies on static rules or simple similarity measures. They lack comprehensive modeling of the complex implicit dependencies between field feature vectors. When generating batches, they often use a fixed order or sort by partial field dependencies, which cannot make full use of the multidimensional interaction information between fields. This may result in the order of writing to batches deviating from the actual business dependencies. This invention maps each enhanced order record in the enhanced order record set to a field feature vector, and uses a graph neural network to analyze the implicit dependencies between field feature vectors. Information is transmitted and aggregated through node features and adjacency relationships to extract complex multidimensional dependencies between fields. Dependency weights are calculated in the vector space, and hierarchical scheduling is performed by combining node in-degree and dependency strength. The hierarchical and sorted nodes are combined in batches to generate ordered write batches that closely match actual business dependencies, thereby improving the rationality of batch order and data processing efficiency.

[0050] S4.1 Extract feature values ​​for each enhanced order record in the enhanced order record set by field and map them to field feature vectors.

[0051] Furthermore, fields such as transaction time, transaction type, transaction amount, transaction status, product information, source identifier, access channel, geographical location, activity identifier, time tag, and historical transaction summary are extracted from each enhanced order record. Missing values ​​are handled for the extracted fields, and missing value flags are added to preserve complete information. Semantic processing is performed according to field type—numerical fields are uniformly scaled, categorical fields are represented by lookup table mapping or embedding, time fields are decomposed according to time granularity such as date, day of the week, and hour to generate periodic and time period features, and text fields are generated by a semantic encoder to generate vector representations. The scalar or vector features of each field are mapped to a unified vector representation space, and field identifiers and source metadata are appended to the field vectors. Finally, each enhanced order record outputs a set of field feature vectors arranged by field.

[0052] S4.2. Treat each field feature vector as a node, and use node features and adjacency relationships in the graph neural network to transmit and aggregate information, and extract the implicit dependencies between field feature vectors.

[0053] Each field feature vector is used as a node in the graph, and initial adjacency relationships are constructed. The node features and adjacency matrix are preprocessed and normalized. Multi-layer message passing and aggregation are performed in the graph neural network. In each layer, messages are first collected from the neighbors for each node and aggregated according to edge weights or attention weights. Then, the aggregated messages are fused with the node's own features through an update function to update the node representation. By stacking multiple layers and using supervised or self-supervised training, the graph neural network can gradually capture multi-hop semantics and statistical dependencies, while modifying and strengthening the edge weights. The implicit dependencies between field feature vectors are extracted from the trained edge weights, attention scores, or node representation similarity, and the output is a set of dependency edges with weights and confidence. It should be noted that adjacency relationships can be determined based on field co-occurrence frequency, mutual information, semantic similarity, and business rules, and then thresholded to generate directed or undirected edges and initial edge weights. The update function is a function in each layer of a graph neural network that fuses the features of a node itself with the features of its neighbors through parameterized weighting and a non-linear activation function. It is generally implemented through linear transformations, activation functions, and parameter matrices. Using a standardized set of historical orders and an enhanced set of orders as training samples, the field feature vectors and initial adjacency relationships in each record are input into a graph neural network. The node representations are iteratively updated through self-supervised or supervised methods. The training objectives include reconstructing field co-occurrence relationships, maximizing the similarity of neighboring field features, or minimizing prediction errors, so that the graph neural network can capture the potential dependency patterns between field feature vectors. During the training process, the real field associations and business rules in the historical order data and enhanced order records serve as supervision signals to continuously optimize node features and edge weights until the graph neural network can accurately reflect the implicit dependencies between field feature vectors.

[0054] S4.3. Based on the implicit dependency, perform correlation measurement calculation on the field feature vectors of the source node and the target node in the vector space to obtain the dependency weight.

[0055] Furthermore, each pair of source and target nodes is identified from the field feature vectors and implicit dependencies output by the graph neural network. For each pair of source and target nodes, the corresponding field feature vectors are extracted. In a unified vector space, vector similarity is used to calculate the correlation between the feature vectors of the source and target nodes, yielding the dependency weight, expressed as: ; In the formula, It is the source node and target node The dependency weights, with values ​​ranging from [0,1], are dimensionless; It is the source node The field feature vector; The target node The field feature vector; It is the source node The norm of the field feature vector is used for normalization to ensure that the result is dimensionless and the value ranges from 0 to 1. The target node The norm of the field feature vector is used for normalization to ensure that the result is dimensionless and takes values ​​between 0 and 1.

[0056] S4.4 Determine the connection direction between nodes based on implicit dependencies, assign dependency weights to the corresponding edges, and construct the implicit dependency graph.

[0057] Furthermore, the connection direction of the edges is determined based on the dependency directionality or business logic rules between the field feature vectors (such as the order of transaction time, the order of influence of transaction amount, the fact that payment behavior must precede refund behavior, and the dependency relationship between product information and activity identifier within the same order). Then, the field feature vectors of each pair of nodes are extracted, and the weight of each edge is calculated using the aforementioned dependency weight expression. The dependency weight of each pair of nodes is assigned to the corresponding edge, and all nodes and edges are integrated to construct a complete implicit dependency graph, where nodes represent field feature vectors and edges represent the dependency relationship and weight between nodes.

[0058] S4.5. Based on the in-degree and dependency weight of the nodes in the implicit dependency graph, divide the nodes into different levels. Within each level, adjust the order of the nodes according to the strength of their dependencies.

[0059] Furthermore, the in-degree of all nodes in the implicit dependency graph is counted, and nodes with lower in-degree are assigned to higher levels, while nodes with higher in-degree are assigned to lower levels, forming a preliminary hierarchical division. In each level, the dependency weights between nodes are extracted, and the node order is determined by comparing the dependency weights of each pair of nodes. Nodes with higher dependency weights are arranged in earlier positions, and nodes with lower dependency weights are arranged in later positions, thus achieving an ordered arrangement of nodes within each level. The output is a set of nodes after hierarchical division and intra-level order adjustment.

[0060] S4.6. Combine the layered and sorted nodes into batches to form ordered write batches.

[0061] Furthermore, starting from the top level, nodes are extracted sequentially, and the order of nodes in the batch is determined according to the internal sorting results of each level. After the nodes in each level are extracted, they are combined with the nodes in the lower level in sequence to form a batch set, ensuring that the order of nodes in the batch follows the hierarchy and dependency strength of the implicit dependency graph, and the output is written to the batch in an ordered manner.

[0062] S5. Perform multi-dimensional knowledge embedding processing on each standardized record of an order in the ordered write batch to generate an indexed order data write package.

[0063] It should be noted that when existing methods process each standardized record of an order in an ordered batch, they usually directly map each field to a fixed vector representation or a simple feature encoding, and then pack and write it into the database or index storage in batch order. This lacks comprehensive analysis of the semantic relationships between fields and multidimensional knowledge dependencies. The generated write package mainly reflects basic vectors or field sets, and cannot capture complex multidimensional semantic interaction information. When performing multi-dimensional knowledge embedding processing on each standardized order record in an ordered write batch, this invention first maps the transaction log fields and basic metadata fields into field feature vectors. Then, it analyzes the implicit dependencies between field feature vectors through a graph neural network, calculates dependency weights in a unified vector space, constructs an implicit dependency graph, forms an ordered node batch through a hierarchical scheduling algorithm, and performs multi-dimensional slicing, semantic encoding, and vectorization processing on each record. Finally, it constructs a three-layer encapsulation structure (vector index table, semantic alignment label, and original standardized fields) in the write group to generate an indexed order data write package, achieving high-fidelity embedding and indexed storage of multi-dimensional semantic interaction.

[0064] S5.1. Each standardized record of an order written in the batch is sliced ​​into multiple dimensions by field to form an initial feature fragment. The initial feature fragment is then transformed into a semantic atomic fragment through semantic encoding and vectorization.

[0065] Furthermore, transaction log fields are extracted from each standardized record of an order written in an ordered batch and divided into several initial feature fragments. Each initial feature fragment is semantically encoded, mapping text fields to word vectors or semantic vector representations, categorical fields to one-hot vectors or embedding vectors, and numerical fields are normalized and mapped to vector representations. After vectorizing each field, each initial feature fragment is vectorized and combined in a unified vector space to form semantic atomic fragments, ensuring that each semantic atomic fragment retains the semantic information of the original field while maintaining the independence and composability between fields.

[0066] S5.2 Calculate the dependency strength between semantic atomic fragments, perform multi-hop dependency propagation through meta-path expansion, and obtain the set of embedded vectors after multi-dimensional interaction.

[0067] Furthermore, a vector similarity method is used to calculate the dependency strength of each pair of semantic atomic fragments, and a preliminary dependency relationship is constructed based on the dependency strength. Following the predefined meta-path, the semantic atomic fragment nodes on each path are visited sequentially. Starting from the starting node, the embedding vector of the starting node is fused with the embedding vectors of adjacent nodes in the path and the dependency strength between nodes. Each fusion includes weighted merging of the source node embedding vector and the target node embedding vector according to the dependency strength, and updating the node vector with the accumulated dependency information obtained in the previous step. This process is repeated along the meta-path to realize the semantic information transfer and accumulation between multi-hop nodes, so that the final embedding vector of each semantic atomic fragment contains its own semantic representation. Combined with the semantics and dependency strength of related multi-hop nodes, a multi-dimensional interactive set of embedding vectors is finally formed, providing a rich multi-dimensional semantic relationship representation for the generation of indexed order data writing packages. It should be noted that dependency strength is obtained by calculating the vector similarity between semantic atomic fragments. Cosine similarity or dot product is usually used to quantify the relevance of two nodes in the semantic space, and weighted correction is made by combining historical interaction data or business association information. The meta-path is determined based on the business logic relationships between the features of each field in the standardized order record. For example, fields such as transaction time, transaction type, transaction amount, transaction status, and product information have sequential and dependent relationships in the transaction process. Therefore, a path is established along the relationship between these field features, and the order of frequently associated or dependent fields is defined as a multi-hop path for information propagation and multi-dimensional interaction between semantic atomic fragments.

[0068] S5.3: Embedded vectors of different dimensions are mapped to a unified semantic space, and conflicting fields are automatically corrected through a context voting mechanism to generate a unique index anchor for each standardized record of an order.

[0069] Furthermore, the embedded vectors after multi-dimensional interaction are mapped according to a unified semantic space mapping rule, so that the embedded vectors of all dimensions are comparable in the same vector space. At the same time, for field conflicts that occur during the mapping process, the frequency of occurrence and semantic similarity of the corresponding fields of each standardized order record in different contexts are analyzed through a context voting mechanism, so as to select the most representative field value for correction, and finally generate a unique index anchor for each standardized order record. It should be noted that the unified semantic space mapping rule is established based on the semantic feature distribution of embedded vectors of different dimensions and the consistency pattern in historical order data. It is used to align multi-dimensional vectors to the same semantic space to ensure comparability.

[0070] S5.4. The embedded vector set is bucketed according to the unique index anchor point, and a three-layer encapsulation structure is built in the corresponding write group to form an indexed order data write package.

[0071] Furthermore, the set of embedded vectors is organized into a vector index table according to the unique index anchor, so that the index of each standardized order record corresponds one-to-one with the corresponding embedded vector. Then, a semantic alignment label is attached to each index anchor in the vector index table, and the embedded vectors of different dimensions are mapped to a unified semantic space and labeled to achieve alignment of vectors in the semantic space. Finally, the original standardized fields are attached to the semantic alignment label, so that the original fields, index anchors and semantic vectors of each enhanced order record are integrated within the same write group, forming a complete three-layer encapsulation structure. The first layer of encapsulation contains the index anchor and the corresponding set of embedded vectors. The second layer of encapsulation combines the index anchors and their embedded vector sets within the same write group. The third layer of encapsulation integrates all write groups to form a complete indexed order data write package. It should be noted that the three-layer encapsulation structure includes a vector index table, semantic alignment labels, and raw normalized fields.

[0072] S6. Using the RPA algorithm, perform write orchestration on the indexed order data write package according to the ordered write batch, and output the write execution record.

[0073] S6.1. Input the indexed order data into the RPA algorithm in the order of writing batches to generate a queue to be written.

[0074] Furthermore, the indexed order data is written to the RPA algorithm in the order of the ordered write batches. Inside the RPA algorithm, the vector index table, semantic alignment labels and original normalized fields in the three-layer encapsulation structure are parsed according to the batch order to identify the write target and dependencies of each enhanced order record. The enhanced order records are then organized into a queue to be written according to the index order.

[0075] S6.2. Perform persistent write operation on the queue to be written using the RPA algorithm, and generate the corresponding write confirmation set after the write is completed. At the same time, trigger the index service to perform index insertion operation and return the index insertion receipt.

[0076] Furthermore, the vector index table, semantic alignment tags, and original normalized fields are parsed from the indexed order data write package. The enhanced order records are written to the corresponding tables or record sets in the target storage according to the field order. During the writing process, the start time, completion time, and status identifier of each enhanced order record are recorded. After the writing is completed, a write confirmation set is formed, and the write status and timestamp information of each enhanced order record are recorded in the write confirmation set. After the writing is completed, the index service is triggered to insert the vector index table and semantic alignment tags into the corresponding index database. The index service returns an insertion receipt to confirm that each index insertion operation is successfully completed, thereby ensuring the integrity and consistency of the persistent writing of enhanced order records and the index insertion operation.

[0077] S6.3 Integrate the write confirmation set with the index insert receipt and record it as a write execution record.

[0078] Furthermore, the write confirmation set and index insertion receipt are matched and integrated according to the unique index anchor of the enhanced order record. The write status, write timestamp, index insertion receipt status and receipt timestamp information of each enhanced order record are merged into a unified record. The integrated information is organized according to the order of the written batches, and finally a write execution record is generated.

[0079] This embodiment also provides a computer device applicable to the method of intelligent data writing on a platform based on big data, including: a memory and a processor; the memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions to realize the method of intelligent data writing on a platform based on big data as proposed in the above embodiment.

[0080] The computer device can be a terminal, comprising a processor, memory, communication interface, display screen, and input devices connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, carrier networks, NFC (Near Field Communication), or other technologies. The display screen can be an LCD screen or an e-ink screen. The input devices can be a touch layer covering the display screen, buttons, a trackball, or a touchpad on the computer device's casing, or an external keyboard, touchpad, or mouse.

[0081] This embodiment also provides a storage medium storing a computer program, which, when executed by a processor, implements the method for intelligent writing of platform data based on big data as proposed in the above embodiments. The storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as Static Random Access Memory (SRAM), Electrically Erasable Programmable Read-Only Memory (EEPROM), Erasable Programmable Read Only Memory (EPROM), Programmable Red-Only Memory (PROM), Read-Only Memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0082] In summary, this invention significantly improves data integrity and business interpretability by enhancing the context and metadata of standardized order records, thereby achieving semantic expansion and contextual reconstruction of original transaction data. Furthermore, by mapping the enhanced order records into field feature vectors through graph neural networks, and by modeling the implicit dependencies between vectors and calculating dependency weights, an implicit dependency graph reflecting the real business logic is constructed. This enables the automatic discovery and quantitative analysis of complex non-explicit associations, improving the semantic consistency of data writing, processing efficiency, and system robustness.

[0083] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A method for intelligent data writing to a platform based on big data, characterized in that: include, Access user transaction log data streams and attach basic metadata to each user transaction log data stream to form order access data; The order data is parsed and standardized, and basic business identifiers are extracted to obtain a set of standardized order records. The standardized order record set is enhanced with context and metadata, and then integrated with the basic metadata to generate an enhanced order record set; The enhanced order record set is mapped to field feature vectors. The implicit dependencies between field feature vectors are analyzed by graph neural networks, and the dependency weights are calculated in the vector space to construct an implicit dependency graph. An ordered write batch is generated by a hierarchical scheduling algorithm. Perform multi-dimensional knowledge embedding processing on each standardized record of an order in the ordered write batch to generate an indexed order data write package; The RPA algorithm is used to perform write orchestration on indexed order data write packages according to ordered write batches, and the write execution record is output.

2. The method for intelligent data writing on a platform based on big data as described in claim 1, characterized in that: The steps for attaching basic metadata to each user transaction log data stream to form order access data are as follows: The basic metadata is bound to the user transaction log data stream to form a single order data record; Individual order data records are parsed and structured in a unified format and then aggregated into order access data.

3. The method for intelligent data writing on a platform based on big data as described in claim 2, characterized in that: The basic metadata is obtained by compiling the source identifier, access timestamp, and access channel identifier automatically recorded when accessing each user transaction log data stream.

4. The method for intelligent data writing on a platform based on big data as described in claim 3, characterized in that: The specific steps for parsing and standardizing the order data, extracting basic business identifiers, and obtaining a standardized set of order records are as follows: The transaction log field and basic metadata field are read from a single order data record in the order access data, split, and converted into an order data representation structure. Perform validity checks and corrections on the order data representation structure to obtain standardized order records; Extract basic business identifiers from the standardized order records and associate them with the corresponding standardized order records to form a set of standardized order records.

5. The method for intelligent data writing on a platform based on big data as described in claim 4, characterized in that: The process of enhancing the context and metadata of the standardized order record set and integrating it with the basic metadata to generate an enhanced order record set involves the following steps: Based on the basic business identifier, the corresponding attribute is retrieved from the reference information table, and the retrieval results are appended to the standardized record set of the order to obtain the reference attribute record set; The reference attribute record set is standardized with uniform time, time field labels are generated, and time period labels and historical transaction summaries are appended to the reference attribute record set to obtain the time context record set; Parse the access channel identifier from the time context record set, obtain the geographic location information, match the valid promotional information in the business activity table, and append it to the time context record set to obtain the geographic activity context record set. The integrity and consistency of the geographic activity context record set are checked, and the record set is merged with the basic metadata at the field level to generate an enhanced order record set.

6. The method for intelligent data writing on a platform based on big data as described in claim 5, characterized in that: The historical transaction summary is obtained by retrieving relevant transaction data within the corresponding time window based on the business identifier and time field in the reference attribute record set, and then aggregating and compressing the retrieval results.

7. The method for intelligent data writing on a platform based on big data as described in claim 5, characterized in that: The process involves mapping the enhanced order record set into field feature vectors, analyzing the implicit dependencies between these vectors using a graph neural network, calculating dependency weights in the vector space, and constructing a implicit dependency graph. The specific steps are as follows: Extract feature values ​​for each enhanced order record in the enhanced order record set by field and map them into field feature vectors; Each field feature vector is used as a node. In the graph neural network, node features and adjacency relationships are used to transmit and aggregate information, and the implicit dependencies between field feature vectors are extracted. Based on implicit dependencies, the correlation measurement of the field feature vectors of the source node and the target node is calculated in the vector space to obtain the dependency weights; The connection directions between nodes are determined based on implicit dependencies, and the dependency weights are assigned to the corresponding edges to construct an implicit dependency graph.

8. The method for intelligent data writing on a platform based on big data as described in claim 1, characterized in that: The specific steps for generating ordered write batches using a hierarchical scheduling algorithm are as follows. Based on the in-degree and dependency weight of the nodes in the implicit dependency graph, the nodes are divided into different levels, and within each level, the order of the nodes is adjusted according to the strength of the dependencies between them. The hierarchical and sorted nodes are combined in batches to form ordered write batches.

9. The method for intelligent data writing on a platform based on big data as described in claim 8, characterized in that: The process of performing multi-dimensional knowledge embedding on each standardized record in the ordered write batch to generate an indexed order data write package involves the following steps: Each standardized record of an order written in the batch is sliced ​​by field in a multidimensional way to form an initial feature fragment. The initial feature fragment is then transformed into a semantic atomic fragment through semantic encoding and vectorization. Calculate the dependency strength between semantic atomic fragments, perform multi-hop dependency propagation through meta-path expansion, and obtain the set of embedded vectors after multi-dimensional interaction; Different dimensions of embedded vectors are mapped to a unified semantic space, and conflicting fields are automatically corrected through a context voting mechanism to generate a unique index anchor for each standardized record of an order. The embedded vector set is bucketed according to the unique index anchor point, and a three-layer encapsulation structure is built in the corresponding write group to form an indexed order data write package; the three-layer encapsulation structure includes a vector index table, semantic alignment labels, and original normalized fields.

10. The method for intelligent data writing on a platform based on big data as described in claim 9, characterized in that: The RPA algorithm is used to write indexed order data into packets 's' in ordered batches, performing write orchestration and outputting write execution records. The specific steps are as follows: The indexed order data is written to the package and then sequentially input into the RPA algorithm in ordered batches to generate a queue to be written. The queue to be written is persisted using the RPA algorithm. After the write is completed, a corresponding write confirmation set is generated. At the same time, the index service is triggered to perform an index insertion operation and return an index insertion receipt. The write confirmation set is integrated with the index insert receipt and recorded as a write execution record.

Images