Method for optimizing expression calculation during data mapping rule field mapping

Abstract

The invention provides an optimization method for expression calculation during data mapping rule field mapping, and belongs to the technical field of business data processing. The method comprises the following steps: based on a plurality of expressions on a mapping unit, extracting a source table and field information required by expression calculation; setting context data required by expression calculation based on the source table, the field information and the source data; on the basis of the context data, expression calculation is conducted in a multi-thread mode, a calculation result is generated, the context of source data in each thread is the same, and calculation of all expressions of one mapping unit is executed in one thread; and combining calculation results, and generating a target data set. The performance loss and initialization time of expression calculation are effectively reduced, and the execution efficiency and speed of the data mapping rule in a large-scale data processing scene are improved.

Description

Technical Field

[0001] This application belongs to the technical field of business data processing, and particularly relates to an optimization method for expression calculation during field mapping of data mapping rules. Background Art

[0002] In the current data processing field, data mapping rules are widely used in the process of generating target data from source data. Different applications and business scenarios require specific expression calculations to achieve mapping and conversion between fields, such as operations like field aggregation and character concatenation.

[0003] However, in actual applications, when faced with a large amount of data and complex mapping relationships, traditional data mapping methods exhibit significant performance bottlenecks. The main problems are concentrated in the excessive time consumption of the packet unpacking process during the initialization of the expression calculation context. Due to the large amount of data to be processed and a lot of redundant information, the overall execution efficiency of the data mapping rules is low, affecting the overall speed of the business process. Summary of the Invention

[0004] To solve at least one aspect of the technical problems in the background art, this application provides an optimization method for expression calculation during field mapping of data mapping rules, effectively reducing the performance loss and initialization time of expression calculation, and improving the execution efficiency and speed of data mapping rules in large-scale data processing scenarios.

[0005] The second aspect embodiment of this application provides an optimization device for expression calculation during field mapping of data mapping rules.

[0006] The third aspect embodiment of this application provides an electronic device.

[0007] The technical solutions adopted by this application are as follows: The first aspect embodiment of this application provides an optimization method for expression calculation during field mapping of data mapping rules, including: Based on multiple expressions on the mapping unit, extract the source table and field information required for the expression calculation; Based on the source table, the field information, and the source data, set the context data required for the expression calculation; Based on the context data, perform expression calculation in a multi-threaded manner to generate calculation results, where the context of the source data in each thread is the same, and all expressions of one mapping unit are calculated in one thread; Merge the calculation results and generate a target data set.

[0008] According to an embodiment of the present application, extracting the source table and field information required for calculating the expressions based on multiple expressions on the mapping unit specifically includes: Analyze the structure and content of each expression to identify the source table associated with the expression and the fields corresponding to the expression; Based on the identification results, construct an information set containing the source table and the fields.

[0009] According to an embodiment of the present application, setting the context data required for calculating the expressions based on the source table, the field information, and the source data specifically includes: Filter the specific field values in the source data related to the calculation of the expression; Generate simplified context data based on the field values, where the context data only includes the minimum data set directly related to the calculation of the expression.

[0010] According to an embodiment of the present application, performing the calculation of the expressions in a multi-threaded manner based on the context data to generate a calculation result, where the context of the source data in each thread is the same, and all the expressions of one mapping unit are calculated in one thread specifically includes: Allocate multiple mapping units to different calculation threads, and each thread independently processes one mapping unit; Initialize the expression calculation engine and the corresponding context data once inside each thread; Sequentially perform continuous calculations on all the expressions in the mapping unit belonging to the thread, and reuse the same context data.

[0011] According to an embodiment of the present application, performing the calculation of the expressions in a multi-threaded manner based on the context data to generate a calculation result, where the context of the source data in each thread is the same, and all the expressions of one mapping unit are calculated in one thread further includes: Coordinate the execution order of each thread through a thread scheduling mechanism to ensure data consistency in a multi-threaded environment.

[0012] According to an embodiment of the present application, the method further includes: Before performing the calculation of the expressions, perform syntax verification and field legality check on the expressions. The syntax verification includes determining whether the expression conforms to the preset syntax rules, and the field legality check includes verifying whether the fields referenced in the expression exist in the corresponding source table; If a syntax error or illegal field reference is found in the expression, generate an error prompt message and terminate the calculation process of the current mapping unit.

[0013] According to an embodiment of the present application, the method further includes: After generating the target data set, perform verification on the target data set, where the verification includes verifying whether the values of the target fields conform to a preset data type, format, or value range; If it is found that the data of the target field does not conform to the verification rule, record the exception information and mark the target field as abnormal data; Associate the abnormal data with the corresponding original source data for tracking and processing.

[0014] An embodiment of the second aspect of the present application provides an optimization device for expression calculation during data mapping rule field mapping, including: An extraction module, adapted to extract the source table and field information required for the expression calculation based on multiple expressions on the mapping unit; A parameter setting module, adapted to set the context data required for the expression calculation based on the source table, the field information, and the source data; A result generation module, adapted to perform expression calculation in a multi-threaded manner based on the context data to generate a calculation result, where the context of the source data in each thread is the same, and all expressions of a mapping unit are calculated in one thread; A result output module, adapted to merge the calculation results and generate a target data set.

[0015] An embodiment of the third aspect of the present application provides an electronic device, including a memory, a processor, and a computer program stored on the memory and executable on the processor, where when the processor executes the program, it implements the optimization method for expression calculation during data mapping rule field mapping in any embodiment of the first aspect as described above.

[0016] The present application also provides a non-volatile computer storage medium, on which computer-executable instructions are stored, and when the computer-executable instructions are executed by a processor, they implement the optimization method for expression calculation during data mapping rule field mapping in any embodiment of the first aspect as described above.

[0017] According to the optimization method for expression calculation during data mapping rule field mapping in the first aspect embodiment of the present application, first, by analyzing multiple expressions on the mapping unit, the source tables and field information required for calculation are extracted. This process streamlines the data set required for subsequent processing and avoids the impact of redundant data on performance. Next, based on the identified source tables, field information, and actual source data, the context data required for expression calculation is set, further reducing unnecessary data loading and initialization time and improving calculation efficiency. Then, a multi-threaded approach is used to perform expression calculation based on the above context data. The source data context within each thread is consistent, and all expressions of a mapping unit are calculated in one thread. This method not only improves CPU utilization but also reduces the overhead caused by frequent initialization of the expression calculation engine. Finally, the calculation results generated by each thread are merged to generate the target data set. By streamlining the data set, reducing redundant initialization processes, and using multi-threaded technology to efficiently execute expression calculations, this method significantly reduces the calculation time-consuming, greatly improves the execution efficiency and speed of data mapping rules in large-scale data processing scenarios, and realizes performance optimization. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation of the present application. In the drawings: Figure 1 is a schematic flowchart of the optimization method for expression calculation during data mapping rule field mapping provided by the embodiment of the present application; Figure 2 is a schematic structural diagram of the optimization device for expression calculation during data mapping rule field mapping provided by the embodiment of the present application; Figure 3 is a schematic structural diagram of the electronic device provided by the embodiment of the present application.

[0019] REFERENCE SIGNS: 110, extraction module; 120, parameter setting module; 130, result generation module; 140, result output module; 810, processor; 820, communication interface; 830, memory; 840, communication bus. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0020] In order to more clearly illustrate the overall concept of the present application, the following is a detailed description by way of example in conjunction with the drawings of the specification.

[0021] In the following description, many specific details are set forth in order to provide a thorough understanding of the present application. However, the present application may be implemented in other ways different from those described herein. Therefore, the scope of protection of the present application is not limited by the specific embodiments disclosed below. It should be noted that, without conflict, the embodiments of the present application and the features in each embodiment may be combined with each other.

[0022] In the present application, unless otherwise clearly defined and limited, the first feature being "on" or "under" the second feature may mean that the first and second features are in direct contact, or the first and second features are indirectly in contact through an intermediate medium. In the description of this specification, the description with reference to terms such as "one embodiment", "some embodiments", "example", "specific example", or "some examples" means that the specific features, structures, materials, or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic expressions of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described may be combined in a suitable manner in any one or more embodiments or examples.

[0023] As Figure 1 shown, an embodiment of the first aspect of the present application provides an optimization method for expression calculation during data mapping rule field mapping, including: Step 100: Based on multiple expressions on the mapping unit, extract the source tables and field information required for expression calculation.

[0024] Step 200: Based on the source tables, field information, and source data, set the context data required for expression calculation.

[0025] Step 300: Based on the context data, perform expression calculation in a multi-threaded manner to generate calculation results, where the context of the source data in each thread is the same, and all expressions of one mapping unit are calculated in one thread.

[0026] Step 400: Merge the calculation results and generate a target data set.

[0027] In step 100, first, the method analyzes the structure and content of each expression to identify the source table associated with the expression and the corresponding fields. This process not only ensures that only the data that is truly needed is included in the subsequent processing flow, but also avoids the additional performance loss caused by processing irrelevant data. By constructing an information set containing all the necessary source tables and their fields, an accurate data basis can be provided for the next steps, thereby improving the efficiency of the entire calculation process.

[0028] Secondly, this step helps reduce unnecessary computational complexity. In the traditional data mapping process, a large amount of redundant data processing may be involved, which not only wastes system resources but also may slow down the calculation speed. By accurately extracting the required field information, the data volume in the subsequent calculation stage can be significantly reduced, making the expression calculation more efficient. Especially when dealing with large-scale data sets, this optimization is particularly important.

[0029] Suppose we have a requirement to map a Sales Order (SO) to a Purchase Order (PO). The main table of the sales order contains 146 fields, and the detail table of the sales order contains 156 fields. In this process, we need to generate the values of certain fields in the purchase order according to specific expressions.

[0030] Expression examples: Expression 1: {"expr": "'Sales document: '+SO.salesOrderCode"}, which is used to generate a certain field in the header of the purchase order.

[0031] Expression 2: {"expr": " SOItem.TaxInValue * SOItem.SettleQty "}, which is used to calculate the value of a certain field in the detail of the purchase order.

[0032] The specific steps are as follows: Parse the expressions: For Expression 1, it is parsed that it needs to use the salesOrderCode field in the SO table.

[0033] For Expression 2, it is parsed that it needs to use the TaxInValue and SettleQty fields in the SOItem table.

[0034] Construct the information set: According to the above parsing results, construct the following information set: json { "SO": ["salesOrderCode"], "SOItem": ["TaxInValue", "SettleQty"] } This information set clarifies which fields are necessary to execute these two expressions.

[0035] Streamline the data set: When organizing the minimum data set, only include the necessary fields mentioned above, rather than all 146 or 156 original fields. For example, for the sales order detail table, which might originally contain a large amount of other information, now only a few fields such as TaxInValue and SettleQty need to be concerned about. In step 200, the work in the calculation preparation stage is further refined by filtering the specific field values related to the expression calculation to generate a streamlined context data. This means that only the data directly involved in the calculation will be loaded into memory, greatly reducing the initialization time and memory occupation. This method is especially suitable for processing source models containing a large amount of irrelevant data because it can effectively filter out the information that is useless for the current calculation.

[0036] In addition, using the filtered minimum data set as the context data can ensure that the calculation engine only needs to focus on those necessary inputs, thus accelerating the process of expression calculation. This is crucial for improving the overall system response speed, especially in high-concurrency or real-time data processing scenarios, where it can significantly reduce latency and increase throughput.

[0037] The implementation steps include: Determine the necessary fields: Based on the source table and field information required by the expression extracted in step 100, clarify which fields are necessary.

[0038] Filter the relevant field values: Filter out the specific field values corresponding to the above necessary fields from the source data to construct a minimized data set. This step can be achieved through database queries or data processing functions in programming languages.

[0039] Organize the context data: Organize the filtered field values in a certain structure to form the context data. This context data only contains the fields directly related to the current expression calculation, avoiding the existence of redundant data.

[0040] Optimize memory usage: By only loading the necessary field values into memory, unnecessary memory occupation is reduced, and the system response speed is improved.

[0041] Suppose we have a requirement to map a sales order (SO) to a purchase order (PO), which involves the following two expressions: Expression 1: {"expr": "'Sales document: '+SO.salesOrderCode"}, which is used to generate a certain field in the purchase order header.

[0042] Expression 2: {"expr": "SOItem.TaxInValue * SOItem.SettleQty"}, used to calculate a field value in the purchase order details.

[0043] Step description Determine necessary fields: Based on the analysis in Step 100, determine that Expression 1 requires the SO.salesOrderCode field; Expression 2 requires the SOItem.TaxInValue and SOItem.SettleQty fields.

[0044] Filter relevant field values: For each sales order record, filter out the values of the SO.salesOrderCode, SOItem.TaxInValue, and SOItem.SettleQty fields.

[0045] Assume the original data is a sales order master table with 146 fields and a sales order details table with 156 fields. Now only need to focus on the following fields: json { "SO": { "salesOrderCode": "S001" }, "SOItem": {"TaxInValue": 450, "SettleQty": 20}, {"TaxInValue": 300, "SettleQty": 10} } Organize context data: Organize the filtered field values into a format suitable for subsequent calculations. For example, for a sales order record, the following context data can be constructed: json { "SO": { "salesOrderCode": "S001" }, "SOItem": {"TaxInValue": 450, "SettleQty": 20}, {"TaxInValue": 300, "SettleQty": 10} ​​} In this way, the computing engine only needs to process these necessary fields instead of the entire original data set. In step 300, multi-threading technology is adopted for expression calculation. This approach makes full use of the multi-core processor advantages of modern computers and improves CPU utilization. Multiple mapping units are assigned to different threads, and the expression calculation engine and context data are initialized once inside a thread, and then all the expression calculations in the mapping unit are continuously executed. This not only reduces the overhead of thread startup but also maximizes the utilization of system resources and speeds up the processing speed.

[0046] At the same time, since the source data context within each thread remains consistent and all the expression calculations in a mapping unit are completed at one time, this can avoid the performance loss caused by repeated initialization of the context. For mapping tasks with a large amount of data, such a design can significantly shorten the total calculation time and make data processing more efficient and fast.

[0047] The implementation steps include: Assign mapping units: First, multiple mapping units are assigned to different threads. Each mapping unit contains a set of data to be processed and its corresponding expressions.

[0048] Initialize context data and calculation engine: Inside a thread, the expression calculation engine and context data are initialized only once. This can avoid the performance loss caused by frequent initialization.

[0049] Execute expression calculation: Inside the thread, all the expressions in the affiliated mapping unit are continuously calculated in sequence, and the same context data is reused. This not only reduces the time overhead of repeated initialization but also ensures data consistency.

[0050] Merge results: After each thread is completed, its calculation results are merged to generate the final target data set.

[0051] Thread scheduling mechanism: Through a reasonable thread scheduling mechanism (such as task queue, synchronization lock, etc.), coordinate the execution order of each thread to ensure data consistency.

[0052] Suppose we have a requirement to map a Sales Order (SO) to a Purchase Order (PO), which involves the following two mapping units: Mapping unit 1: Map the main table of the sales order to the main table of the purchase order, including 5 expressions.

[0053] Mapping unit 2: Map the detail table of the sales order to the detail table of the purchase order, including 6 expressions.

[0054] Step description Distribution mapping unit: Suppose there are 10 sales order records. We divide these records into two groups, with 5 records in each group. Each group corresponds to a mapping unit and is processed by two threads respectively.

[0055] Initialize context data and calculation engine: Inside each thread, initialize the expression calculation engine and context data once. For example, when processing mapping unit 1 in thread 1, initialize the following context data: json { "SO": { "salesOrderCode": "S001", "GIVendorID": "V001", "IsRed": false, "TaxInValue": 1000, "SettleValue": 950, "PayValue": 900 } } Similarly, when processing mapping unit 2 in thread 2, initialize the following context data: json { "SOItem": {"MaterialID": "M001", "TaxInValue": 450, "SettleQty": 20}, {"MaterialID": "M002", "TaxInValue": 300, "SettleQty": 10} } Execute expression calculation: In thread 1, calculate all expressions in mapping unit 1 in sequence: java MappingExpressionEvaluator evaluator = new MappingExpressionEvaluator(); evaluator.setContext(context); ​tarData.setValue("key1", evaluator.evaluate("{\"expr\":\"'Sales document: '+SO.salesOrderCode\",\"sexpr\":\"[]\"}")); tarData.setValue("key2", evaluator.evaluate("expression2")); / / Continue to calculate other expressions... In thread 2, calculate all the expressions in mapping unit 2 in sequence: java MappingExpressionEvaluator evaluator = new MappingExpressionEvaluator(); evaluator.setContext(context); tarData.setValue("keyA", evaluator.evaluate("{\"expr\":\"SOItem.TaxInValue * SOItem.SettleQty\",\"sexpr\":\"[]\"}")); tarData.setValue("keyB", evaluator.evaluate("expressionB")); / / Continue to calculate other expressions... Merge results: After each thread completes the calculation of all expressions in its mapping unit, the results are merged to generate the target data set. For example: json { "POHeader": { "key1": "Sales document: S001", "key2": "...", / / Other field values }, "PODetails": {"keyA": 9000, "keyB": "..."}, {"keyA": 3000, "keyB": "..."} } Thread scheduling mechanism:​ Use a task queue or synchronization lock to manage the execution order of threads and ensure data consistency. For example, use Java's ExecutorService to manage the thread pool: java ExecutorService executor = Executors.newFixedThreadPool(2); for (int i = 0; i < 2; i++) { final int threadIndex = i; executor.submit(() -> { / / Initialize context data / / Execute expression calculation / / Merge results }); } executor.shutdown(); In step 400, first, it is necessary to merge the calculation results generated by each thread to form the final target data set. This process requires efficiently and accurately integrating the results from different threads to ensure data consistency and integrity. By reasonably designing the merging strategy, it is possible to maximize the merging efficiency while ensuring quality, which is crucial for achieving fast data output.

[0056] Finally, the generated target data set is not only the product of a series of complex transformations on the original data but also a concrete manifestation of the entire data mapping rule optimization method. It not only contains the information expected by the user but also reflects high performance in processing large-scale data sets. Through the above series of optimization measures, both in terms of computational efficiency and final data accuracy, the expected effects have been achieved, realizing an effective improvement over traditional data mapping methods.

[0057] The implementation steps include: Collect the local results of each thread After each thread completes the expression calculation of its own mapping unit, a set of local results (such as target header field values, detail item values, etc.) will be generated.

[0058] These local results usually exist in a structured form (such as JSON objects, Maps, Lists, etc.).

[0059] Design a unified data structure model Define a unified target data structure in the program (such as the structure of the main table and detail table of a purchase order PO) as the basic template for the final output.

[0060] The calculation results of all threads need to be classified and filled in according to this structure.

[0061] Use a thread-safe mechanism to merge the results During the merging process, concurrent conflicts should be avoided. The following strategies can be adopted: Use thread-safe containers (such as ConcurrentHashMap, CopyOnWriteArrayList) to receive the results submitted by each thread.

[0062] Or pass the results through a blocking queue (such as BlockingQueue), and let the main thread or a dedicated merging thread be responsible for processing.

[0063] If database writing is involved, transaction control or batch insertion can be used to ensure consistency.

[0064] Perform data verification and deduplication processing Check the integrity of the merged data (such as whether required fields are missing).

[0065] If there are duplicate records (such as multiple threads processing the same primary key), perform deduplication or overwrite processing according to the business logic.

[0066] Output the final target data set Convert the merged data into the format expected by the user (such as JSON, XML, database table, etc.).

[0067] Visualize and display or call the interface to return to the upper-level system.

[0068] For example, we are processing the mapping task from Sales Order (SO) to Purchase Order (PO), which contains two mapping units: Mapping unit A: Sales order main table → Purchase order main table Mapping unit B: Sales order detail table → Purchase order detail table Each mapping unit is assigned to an independent thread for processing. Finally, the results of the two threads need to be merged into the complete purchase order data.

[0069] Detailed implementation steps: 1. Example of thread processing results Suppose thread 1 processes the main table part and gets the following results: json { "POHeader": { "POCode": "PO20250613001", "VendorID": "V001", "Description": "Sales document: SO20250613" } } Thread 2 processes the details list and gets the following results: json { "PODetails": {"MaterialID": "M001", "Qty": 20, "Amount": 9000}, {"MaterialID": "M002", "Qty": 10, "Amount": 3000} } 2. Merge strategy design Define a unified target structure (Java Bean example): java class PurchaseOrder { String POCode; String VendorID; String Description; List <podetail>details; } The main thread monitors the completion status of two threads and starts merging when both are completed: java PurchaseOrder finalPO = new PurchaseOrder(); / / Get the result of thread 1 and assign it to finalPO finalPO.setPOCode(thread1Result.get("POCode")); finalPO.setVendorID(thread1Result.get("VendorID")); finalPO.setDescription(thread1Result.get("Description")); / / Get the result of thread 2 and assign it to the details list finalPO.setDetails(thread2Result.getList("PODetails")); 3. Consistency and integrity verification Verify whether the main table fields are complete (such as whether POCode exists) Verify whether the details are empty or have an abnormal quantity If it is found that the amount of a certain detail is negative, trigger a log warning or throw an exception 4. Output the final result The output is in JSON format: json { "POCode": "PO20250613001", "VendorID": "V001", "Description": "Sales document: SO20250613", "details": {"MaterialID": "M001", "Qty": 20, "Amount": 9000}, {"MaterialID": "M002", "Qty": 10, "Amount": 3000} } Or directly write to the database:​ sql INSERT INTO PurchaseOrders (POCode, VendorID, Description) VALUES ('PO20250613001', 'V001','Sales document: SO20250613'); INSERT INTO PODetails (POCode, MaterialID, Qty, Amount) VALUES ('PO20250613001', 'M001', 20, 9000), ('PO20250613001', 'M002', 10, 3000).

[0070] According to the optimization method for expression calculation during data mapping rule field mapping provided by the embodiments of the first aspect of the present application, first, by analyzing multiple expressions on the mapping unit, the source tables and field information required for calculation are extracted. This process streamlines the data set required for subsequent processing and avoids the impact of redundant data on performance. Next, based on the identified source tables, field information, and actual source data, the context data required for expression calculation is set, further reducing unnecessary data loading and initialization time and improving calculation efficiency. Then, a multi-threaded approach is used to perform expression calculation based on the above context data. The context of the source data within each thread remains consistent, and all expressions of a mapping unit are calculated in one thread. This approach not only improves CPU utilization but also reduces the overhead caused by frequent initialization of the expression calculation engine. Finally, the calculation results generated by each thread are merged to generate the target data set. By streamlining the data set, reducing the redundant initialization process, and using multi-threaded technology to efficiently execute expression calculation, this method significantly reduces the calculation time-consuming, greatly improves the execution efficiency and speed of data mapping rules in large-scale data processing scenarios, and realizes performance optimization.

[0071] In some embodiments of the present application, based on multiple expressions on the mapping unit, the source tables and field information required for expression calculation are extracted as follows: Analyze the structure and content of each expression to identify the source tables associated with the expression and the fields corresponding to the expression; Based on the identification results, construct an information set containing the source tables and fields.

[0072] Specifically, when performing field mapping for data mapping rules, the first step of the optimization method is to extract the source tables and field information required for expression calculation based on multiple expressions on the mapping unit. Specifically, this process first involves a detailed analysis of the structure and content of each expression to identify the source tables associated with the expression and the specific fields corresponding to the expression. Through this analysis, it can be clarified which data sources and fields are necessary for the calculation of the current expression, thus providing an accurate data access path for subsequent steps.

[0073] Based on the above analysis results, next, an information set containing all necessary source tables and their fields is constructed. This information set not only simplifies the data scope required for subsequent processing, reduces unnecessary data loading and processing work, but also provides a clear data basis for expression calculation. In this way, it is possible to effectively avoid processing irrelevant redundant data, reduce computational complexity and resource consumption, and improve the overall data processing efficiency. This step lays a solid foundation for setting the context data required for expression calculation in the subsequent steps, ensuring that the entire calculation process is more efficient and accurate.

[0074] That is to say, the process of extracting the source tables and field information required for expression calculation based on multiple expressions on the mapping unit is achieved through syntactic parsing and semantic analysis of the structure and content of each expression. Specifically, the system will parse the expression strings one by one, identify the variable naming formats involved (such as "SO.salesOrderCode", "SOItem.TaxInValue"), and extract the corresponding source table names (such as SO, SOItem) and field names (such as salesOrderCode, TaxInValue) according to the naming rules. This process can be completed by constructing an expression parser module, which supports lexical and syntactic analysis of multiple expression grammars (such as JavaScript, Groovy, custom DSL, etc.), so as to accurately extract all data source fields participating in the operation.

[0075] For example, during the mapping process from a Sales Order to a Purchase Order, there are the following two expressions: {"expr": "'Sales document: '+SO.salesOrderCode"} {"expr": " SOItem.TaxInValue * SOItem.SettleQty "} By parsing the first expression, the system can identify that it depends on the field salesOrderCode in the source table SO; after parsing the second expression, it identifies that it depends on the fields TaxInValue and SettleQty in the source table SOItem. Finally, the system aggregates this information into a structured information set, such as: json { "SO": ["salesOrderCode"], "SOItem": ["TaxInValue", "SettleQty"] } This information set clearly indicates all the source data fields required for the current mapping task, avoiding the performance loss caused by loading and processing irrelevant fields, and providing accurate data support for subsequent context construction and expression calculation.

[0076] In some embodiments of the present application, based on the source table, field information, and source data, context data required for expression calculation is set, specifically: Filter specific field values in the source data related to expression calculation; Generate a streamlined context data based on the field values, where the context data only includes the minimum data set directly related to expression calculation.

[0077] Specifically, this step first involves filtering specific field values in the source data related to expression calculation. This means identifying and extracting those field values that directly participate in expression calculation from the entire source data set, while ignoring the irrelevant data. In this way, the amount of data to be processed can be significantly reduced, avoiding unnecessary calculations and memory occupation.

[0078] Based on the filtered field values, a streamlined context data set is then generated. This context data set only contains the minimum data set directly related to expression calculation, ensuring the efficiency and accuracy of the calculation process. The principle is that by precisely filtering the data and removing redundant information, the time cost of data transmission and processing is reduced. This not only speeds up the calculation speed of the expression but also reduces the consumption of system resources, especially when dealing with large-scale data, the effect is particularly obvious.

[0079] That is to say, the process of setting the context data required for expression calculation is based on the source table and field information extracted in the previous step, and specific field values directly related to the expression calculation are filtered out from the original source data. The system will perform field-level filtering on the original data source according to the constructed information set (such as {"SO": ["salesOrderCode"], "SOItem": ["TaxInValue", "SettleQty"]}), only extracting the data values of the required fields and ignoring other irrelevant fields. For example, when there are 146 fields in the sales order main table, if only the salesOrderCode field is used for expression calculation, only the value of this field will be extracted; similarly, when there are originally 156 fields in the detail table, only the values of the two fields TaxInValue and SettleQty will be retained.

[0080] Taking a specific mapping task as an example, assume the current sales order data being processed is as follows: json { "SO": { "salesOrderCode": "SO20250613", "customerName": "ABC Company", "orderDate": "2025-06-13", ... }, "SOItem": {"MaterialID": "M001", "TaxInValue": 450, "SettleQty": 20, ...}, {"MaterialID": "M002", "TaxInValue": 300, "SettleQty": 10, ...} } After field filtering, the generated streamlined context data will only include: json { "SO": { "salesOrderCode": "SO20250613" }, "SOItem": {"TaxInValue": 450, "SettleQty": 20}, ​{"TaxInValue": 300, "SettleQty": 10} [[ID=4}} This streamlined context dataset will serve as the input environment for subsequent expression calculations, not only reducing memory occupancy and data transmission overhead, but also enhancing computational efficiency and system response speed, especially suitable for efficient data mapping processing in high-concurrency or large-data-volume scenarios.

[0081] In some embodiments of the present application, based on the context data, the expression calculations are performed in a multi-threaded manner to generate calculation results. Among them, the context of the source data in each thread is the same, and all expressions of a mapping unit are calculated in one thread. Specifically: Allocate multiple mapping units to different computing threads, and each thread independently processes one mapping unit; Initialize the expression calculation engine and the corresponding context data once inside each thread; Sequentially perform continuous calculations on all expressions in the mapping unit belonging to the thread, and reuse the same context data.

[0082] Specifically, first, multiple mapping units are allocated to different computing threads, and each thread independently processes one mapping unit. This allocation strategy makes full use of the multi-core processor advantages of modern computers, allowing multiple mapping units to be processed in parallel, thereby significantly improving the overall computing speed. In this way, even in the face of large-scale data sets or complex mapping rules, the computing time can be effectively shortened.

[0083] Secondly, inside each thread, initialize the expression calculation engine and the corresponding context data once. This means that for each mapping unit, only the required computing environment and data context need to be set up once, without repeating this process for each expression. Then, sequentially perform continuous calculations on all expressions in the mapping unit belonging to the thread, and reuse the same context data. The advantage of this is to reduce the performance loss caused by frequent initialization, while also ensuring data consistency and integrity. Since all expressions of a mapping unit are calculated in one thread, this method also avoids the additional overhead caused by thread switching, further enhancing the computing efficiency.

[0084] ​By reasonably utilizing multi-threading technology and combining with streamlined context data, it is possible to maximize the utilization rate of computing resources and computing speed while ensuring the accuracy of data processing. Especially when dealing with a large amount of data or complex mapping rules, it can significantly reduce the total computing time, making the entire data mapping process more efficient and fast. In addition, by reducing unnecessary initialization steps and data transmission, the consumption of system resources is also reduced, enhancing the scalability and response speed of the system.

[0085] That is to say, based on the constructed streamlined context data, the system adopts a multi-threaded approach to execute expression calculations in parallel to improve the overall processing efficiency. The specific implementation process is as follows: First, multiple mapping units (such as the mapping tasks of the sales order master table and the detail table) are assigned to different threads for independent processing. After each thread receives a mapping unit, it only initializes the expression calculation engine and the corresponding context data once inside. For example, it loads a structure such as {"SO": {"salesOrderCode": "SO20250613"}, "SOItem": [{"TaxInValue": 450, "SettleQty":20}]} as the global context environment within the current thread. Then, the thread sequentially performs continuous calculations on all expressions in the affiliated mapping unit and reuses the initialized context data, thus avoiding the performance loss caused by repeated initialization for each expression.

[0086] For example, in an actual data mapping task, the system needs to map the sales order master table and the detail table to the purchase order master table and the detail table respectively. These two mapping tasks are divided into two independent mapping units. The system creates two threads to process them separately. Thread A is responsible for the master table mapping, containing expressions such as "expr":"'Purchase document: '+SO.salesOrderCode"; Thread B is responsible for the detail table mapping, containing expressions such as "expr":"SOItem.TaxInValue* SOItem.SettleQty". When Thread A starts, it loads the context data of the master table and initializes the calculation engine at one time, and then sequentially executes all expression calculations related to the master table; Thread B also initializes the context of the detail table and executes the expressions related to the detail table. The two threads run in parallel without interference, and the final result is collected and merged by the main thread. This method not only makes full use of the multi-core CPU resources, but also significantly improves the overall computing efficiency by reducing the thread switching overhead and reusing the context data, and is especially suitable for efficient processing scenarios of large-scale data or complex mapping logics.

[0087] In some embodiments of the present application, based on context data, expression calculation is performed in a multi-threaded manner to generate a calculation result. Among them, the context of the source data in each thread is the same, and all expressions of a mapping unit are calculated in one thread. It further includes: Coordinating the execution order of each thread through a thread scheduling mechanism to ensure data consistency in a multi-threaded environment.

[0088] Specifically, when performing expression calculation in a multi-threaded environment, in addition to allocating multiple mapping units to different threads, initializing the context once within a thread and reusing the context for continuous calculation, it is also necessary to introduce a thread scheduling mechanism to coordinate the execution order of each thread to ensure the consistency and correctness of the entire data processing process.

[0089] Since concurrent execution of multiple threads may cause problems such as resource competition and data conflicts, especially when multiple threads need to access shared resources (such as intermediate result caches, logging modules, etc.), if not controlled, it may lead to data inconsistency or program exceptions. Therefore, through a reasonable thread scheduling mechanism (such as thread priority control, synchronization locks, task queue management, etc.), the execution order and resource access permissions of each thread can be effectively managed. For example, in scenarios involving shared state updates, mutexes (Mutex) or read-write locks (ReadWriteLock) can be used to ensure atomic operations; in tasks that need to be executed in order, sequential control can be achieved through thread wait / notify mechanisms or the use of ordered task queues.

[0090] On the one hand, this mechanism ensures the data consistency and stability of expression calculation in a multi-threaded environment, avoiding incorrect results or system crashes caused by concurrent conflicts; on the other hand, by optimizing the scheduling strategy, it can also improve thread utilization, reduce thread idle or blocked time, thereby further enhancing the throughput and response speed of the overall system. This is of great significance for the high-concurrency and high-reliability requirements in large-scale data mapping tasks.

[0091] That is to say, during the process of calculating expressions in a multi-threaded manner, in order to ensure data consistency and execution stability in a multi-threaded environment, the system introduces a thread scheduling mechanism to coordinate the execution order and resource access permissions of each thread. Specific implementation methods include using thread pool management, task queues, synchronization lock mechanisms (such as ReentrantLock, ReadWriteLock), thread wait / notification mechanisms, etc., to control the concurrent behavior between threads. For example, when multiple threads need to write data to the same intermediate result cache, the system uses a locking mechanism to ensure that only one thread can perform the write operation at the same time, thus avoiding data overwrite or conflicts; another example is that when there are dependencies between certain mapping tasks (such as the processing of the detail table depends on the generation result of the main table), the thread wait mechanism can be used to ensure that the child thread is started after the main thread is completed, thus ensuring the consistency of the overall logic.

[0092] For example, in a purchase order generation system, the main sales order table and the detail table are respectively divided into two mapping units and processed in parallel by two independent threads. The main table thread is responsible for generating fields such as the purchase order number and supplier information, and the detail thread is responsible for calculating the amount and quantity of each detail item. Since the detail items need to reference the order number generated by the main table, the system uses an inter-thread communication mechanism to make the detail thread wait for the main table thread to complete the number generation before execution. At the same time, the two threads reuse the initialized context data within their respective scopes and submit their calculation results to the main thread for summarization through a thread-safe blocking queue. The main thread uses the Future mechanism of the thread pool to monitor the completion status of each sub-task and unifies the merged results after all threads are completed. This method not only achieves efficient parallel processing but also ensures data consistency and task execution order through a reasonable scheduling strategy, especially suitable for complex mapping scenarios involving shared resource access or task dependencies.

[0093] In some embodiments of the present application, the method further includes: Before performing the expression calculation, perform syntax verification and field legality check on the expression. The syntax verification includes determining whether the expression conforms to the preset syntax rules, and the field legality check includes verifying whether the fields referenced in the expression exist in the corresponding source table; If a syntax error or illegal field reference is found in the expression, generate an error prompt message and terminate the calculation process of the current mapping unit.

[0094] Specifically, the syntax verification is used to determine whether the expression conforms to the syntax rules preset by the system, such as whether the operators are used correctly, whether the parentheses match, and whether the function call format is standardized; while the field legality check is to verify whether the fields referenced in the expression actually exist in the corresponding source table, preventing runtime exceptions caused by misspelled field names or data source changes.

[0095] Through this preprocessing step, potential problems can be detected in a timely manner before the formal calculation, avoiding the waste of invalid computing resources. If a syntax error or illegal field reference is found in the expression during the verification process, the system will immediately generate the corresponding error prompt message and terminate the calculation process of the current mapping unit. This "early interception" mechanism helps to improve the robustness and fault tolerance of the system. At the same time, it also helps developers or configurators quickly locate and correct problems, thus ensuring the accuracy and stability of the entire data mapping process.

[0096] That is to say, in order to ensure the accuracy and reliability of the expression calculation, before performing the actual calculation, the system first performs a syntax check and a field legality check on each expression. The syntax check confirms whether the expression conforms to the preset syntax rules by parsing the structure of the expression, such as the correct use of operators, parentheses matching, and function call formats, etc.; the field legality check is to verify whether all the fields referenced in the expression actually exist in the corresponding source table to prevent runtime exceptions caused by misspelled field names or data source changes. For example, for the expression "expr": "SOItem.TaxInValue * SOItem.SettleQty", the system will check whether the TaxInValue and SettleQty fields actually exist in the sales order detail table (SOItem).

[0097] Suppose the system needs to process a mapping unit that contains the expression "expr": "SO.salesOrderCode + SO.nonExistingField". During the verification phase, the system first performs a syntax check and confirms that the structure of the expression is correct (such as the correct use of the addition operator). Next, the system performs a field legality check and finds that nonExistingField does not exist in the sales order main table (SO). At this time, the system generates a detailed error prompt message, such as "The field nonExistingField was not found in the source table SO", and immediately terminates the calculation process of the current mapping unit. This "early interception" mechanism can not only avoid the waste of invalid computing resources, but also help developers quickly locate and correct configuration errors, thus ensuring the accuracy and stability of the entire data mapping process.

[0098] In some embodiments of the present application, the method further includes: After generating the target data set, perform a verification on the target data set. The verification includes verifying whether the values of the target fields conform to the preset data types, formats, or value ranges; If it is found that the data of the target field does not conform to the verification rules, record the exception information and mark the target field as abnormal data; Associate the abnormal data with the corresponding original source data for tracking and processing.

[0099] Specifically, after generating the target data set, the method further introduces a target data verification mechanism to ensure the accuracy and compliance of the output results. This step first systematically verifies the generated target data set, specifically including: verifying whether the values of the target fields conform to the preset data types (such as integers, strings, dates, etc.), format requirements (such as date format, string length, regular expression matching), and value ranges (such as numerical intervals, enumeration value restrictions, etc.). By setting these rules, problem data caused by expression calculation errors, data conversion anomalies, or logic processing deviations can be effectively identified.

[0100] During the verification process, if it is found that a field in a certain target data does not conform to any of the above rules, the system will automatically record the corresponding abnormal information (such as abnormal field name, expected type, actual value, etc.), and mark this target data as "abnormal data". At the same time, the system will also associate and store these abnormal data with the corresponding original source data to ensure that each piece of abnormal data can be traced back to its original input source. This design not only improves the traceability of data quality problems but also provides convenient conditions for subsequent manual review, correction, and reprocessing.

[0101] That is to say, after generating the target data set, the system will automatically execute the target data verification process to ensure that the output results conform to the preset business rules and data specifications. This process checks each target field through a set of predefined verification rules, including verifying whether the field value meets the specified data types (such as integers, floating-point numbers, strings, etc.), format requirements (such as date format yyyy-MM-dd, mobile phone number regular expression), and value ranges (such as the numerical value must be greater than 0, the status code must be one of the preset enumeration values). For example, during the process of mapping a sales order to a purchase order, the system will perform a numerical type verification on the generated target field such as "purchase amount", a date format verification on "order placement date", and an enumeration value matching verification on "order status".

[0102] Suppose the "Purchase Amount" field in a certain purchase order data is calculated by the expression SOItem.TaxInValue * SOItem.SettleQty, and it is expected to be a positive number. However, since the SettleQty of a certain detail item in the original source data is negative, the finally calculated amount is also negative, violating the preset value range rule (the amount must be ≥ 0). At this time, after the system detects this exception during the verification phase, it will record the following exception information: "The value [-9000] of the field [Amount] does not conform to the verification rule: it must be a non - negative number", and mark this purchase order as "abnormal data". At the same time, the system will also establish an association relationship between this abnormal data and the original sales order detail data, recording the information of the original TaxInValue = 450 and SettleQty = -20 for subsequent manual verification or automatic repair. This mechanism not only improves the automation level of data quality control but also enhances the traceability and processing efficiency of abnormal data.

[0103] As Figure 2 shown, the embodiment of the second aspect of the present application provides an optimization device for expression calculation during field mapping of a data mapping rule, including: An extraction module 110, adapted to extract the source table and field information required for expression calculation based on multiple expressions on the mapping unit; A parameter setting module 120, adapted to set the context data required for expression calculation based on the source table, field information, and source data; A result generation module 130, adapted to perform expression calculation in a multi - thread manner based on the context data to generate a calculation result, where the context of the source data in each thread is the same, and all expressions of one mapping unit are calculated in one thread; A result output module 140, adapted to merge the calculation results and generate a target data set.

[0104] The optimization device for expression calculation during field mapping of the data mapping rule provided by the embodiment of the second aspect of the present application can implement the optimization method for expression calculation during field mapping of the data mapping rule in any of the above - mentioned first - aspect embodiments. Therefore, it can achieve any of the technical effects in the above - mentioned optimization method for expression calculation during field mapping of the data mapping rule, which will not be elaborated here.

[0105] The embodiment of the third aspect of the present application provides an electronic device, including a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the program, it implements the optimization method for expression calculation during field mapping of the data mapping rule in any of the above - mentioned first - aspect embodiments.

[0106] Figure 3 Illustrates a schematic diagram of the physical structure of an electronic device, as Figure 3 shown. The electronic device may include: a processor 810, a communications interface 820, a memory 830, and a communication bus 840. Among them, the processor 810, the communications interface 820, and the memory 830 communicate with each other through the communication bus 840. The processor 810 may call the logical instructions in the memory 830 to execute the optimization method for calculating the expression during the mapping of the data mapping rule field in any embodiment of the first aspect. The method includes: Step 100: Based on multiple expressions on the mapping unit, extract the source table and field information required for expression calculation.

[0107] Step 200: Based on the source table, field information, and source data, set the context data required for expression calculation.

[0108] Step 300: Based on the context data, perform expression calculation in a multi-threaded manner to generate calculation results. Among them, the context of the source data in each thread is the same, and all expressions of one mapping unit are calculated in one thread.

[0109] Step 400: Merge the calculation results and generate a target data set.

[0110] In addition, when the logical instructions in the above-mentioned memory 830 are implemented in the form of software functional units and sold or used as an independent product, they may be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or a part of this technical solution, may be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in various embodiments of the present invention. The foregoing storage medium includes: various media such as USB flash drives, mobile hard disks, read-only memories (ROMs), random access memories (RAMs), magnetic disks, or optical discs that can store program codes.

[0111] On the other hand, the present invention also provides a computer program product. The computer program product includes a computer program. The computer program can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the optimization method for calculating the expression during the mapping of the data mapping rule field provided by the above-mentioned various methods. The method includes: Step 100: Extract the source tables and field information required for expression calculation based on multiple expressions on the mapping unit.

[0112] Step 200: Set the context data required for expression calculation based on the source tables, field information, and source data.

[0113] Step 300: Perform expression calculation in a multi-threaded manner based on the context data to generate calculation results. Among them, the context of the source data in each thread is the same, and all expressions of a mapping unit are calculated in one thread.

[0114] Step 400: Merge the calculation results and generate the target data set.

[0115] On the other hand, the present invention also provides a non-transitory computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, it implements an optimization method for expression calculation during field mapping of the data mapping rules provided by the above methods. The method includes: Step 100: Extract the source tables and field information required for expression calculation based on multiple expressions on the mapping unit.

[0116] Step 200: Set the context data required for expression calculation based on the source tables, field information, and source data.

[0117] Step 300: Perform expression calculation in a multi-threaded manner based on the context data to generate calculation results. Among them, the context of the source data in each thread is the same, and all expressions of a mapping unit are calculated in one thread.

[0118] Step 400: Merge the calculation results and generate the target data set.

[0119] Finally, the present invention also provides a non-volatile computer storage medium, on which computer-executable instructions are stored. When the computer-executable instructions are executed by a processor, it implements an optimization method for expression calculation during field mapping of the data mapping rules provided by the above methods. The method includes: Step 100: Extract the source tables and field information required for expression calculation based on multiple expressions on the mapping unit.

[0120] Step 200: Set the context data required for expression calculation based on the source tables, field information, and source data.

[0121] Step 300: Perform expression calculation in a multi-threaded manner based on the context data to generate calculation results. Among them, the context of the source data in each thread is the same, and all expressions of a mapping unit are calculated in one thread.

[0122] Step 400: Combine the calculation results and generate a target data set.

[0123] In this application, the parts not described can be implemented by adopting or referring to the existing technologies.

[0124] Each embodiment in this specification is described in a progressive manner. For the same or similar parts among the embodiments, reference can be made to each other. Each embodiment focuses on the differences from other embodiments.

[0125] The above description is only for the embodiments of this application and is not intended to limit this application. For those skilled in the art, this application can have various changes and modifications. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of this application shall be included within the protection scope of this application.< / podetail> ​

Claims

1. An optimization method for expression calculation during data mapping rule field mapping, characterized in that, Including: Based on multiple expressions on the mapping unit, extract the source tables and field information required for calculating the expressions; Based on the source tables, the field information, and the source data, set the context data required for expression calculation; Based on the context data, perform expression calculation in a multi-threaded manner to generate calculation results, where the context of the source data in each thread is the same, and all expressions of one mapping unit are calculated in one thread; Merge the calculation results and generate a target data set.

2. The optimization method for expression calculation during data mapping rule field mapping according to claim 1, characterized in that The step of extracting the source tables and field information required for calculating the expressions based on multiple expressions on the mapping unit is specifically: Analyze the structure and content of each expression to identify the source tables associated with the expression and the fields corresponding to the expression; Based on the identification results, construct an information set containing the source tables and the fields.

3. The optimization method for expression calculation during data mapping rule field mapping according to claim 1, characterized in that The step of setting the context data required for expression calculation based on the source tables, the field information, and the source data is specifically: Filter the specific field values in the source data related to the expression calculation; Generate simplified context data based on the field values, where the context data only includes the minimum data set directly related to the expression calculation.

4. The optimization method for expression calculation during data mapping rule field mapping according to claim 1, characterized in that The step of performing expression calculation in a multi-threaded manner based on the context data to generate calculation results, where the context of the source data in each thread is the same, and all expressions of one mapping unit are calculated in one thread, is specifically: Allocate multiple mapping units to different calculation threads, and each thread independently processes one mapping unit; Initialize the expression calculation engine and the corresponding context data once inside each thread; Sequentially perform continuous calculations on all expressions in the mapping unit belonging to the thread, and reuse the same context data.

5. The optimization method for expression calculation during data mapping rule field mapping according to claim 4, characterized in that The step of performing expression calculation in a multi-threaded manner based on the context data to generate calculation results, where the context of the source data in each thread is the same, and all expressions of one mapping unit are calculated in one thread, further includes: Coordinate the execution order of each thread through a thread scheduling mechanism to ensure data consistency in a multi-threaded environment.

6. The optimization method for expression calculation during data mapping rule field mapping according to any one of claims 1 to 5, characterized in that The method further includes: Before performing expression calculation, perform syntax verification and field legality check on the expressions. The syntax verification includes determining whether the expressions conform to the preset syntax rules, and the field legality check includes verifying whether the fields referenced in the expressions exist in the corresponding source tables; If a syntax error or illegal field reference is found in the expression, generate an error prompt message and terminate the calculation process of the current mapping unit.

7. The optimization method for expression calculation during data mapping rule field mapping according to any one of claims 1 to 5, characterized in that The method further includes: After generating the target data set, perform verification on the target data set. The verification includes verifying whether the values of the target fields conform to the preset data types, formats, or value ranges; If it is found that the data of the target field does not conform to the verification rules, record the exception information and mark the target field as abnormal data; Associate the abnormal data with the corresponding original source data for tracking and processing.

8. An optimization device for expression calculation during data mapping rule field mapping, characterized in that Including: An extraction module, adapted to extract source tables and field information required for calculating the expressions based on multiple expressions on a mapping unit; A parameter setting module, adapted to set context data required for calculating the expressions based on the source tables, the field information, and source data; A result generation module, adapted to calculate the expressions in a multi-threaded manner based on the context data to generate calculation results, wherein the context of the source data in each thread is the same, and all expressions of a mapping unit are calculated in one thread; A result output module, adapted to merge the calculation results and generate a target data set.

9. An electronic device, comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the optimization method for calculating expressions during field mapping of the data mapping rules as described in any one of claims 1 to 7.

10. A non-volatile computer storage medium having computer-executable instructions stored thereon, characterized in that, When the computer-executable instructions are executed by the processor, it implements the optimization method for calculating expressions during field mapping of the data mapping rules as described in any one of claims 1 to 7.

Images