Data processing method and device, equipment and storage medium
By pre-creating a wide table in the logistics system, the data of the routing plan service is directly synchronized to the summary indicator query service, the problems of data IO times and operation complexity are solved, and the operation efficiency of the logistics system is improved.
Patent Information
- Application Number
- CN202310524384.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2023-05-10
- Publication Date
- 2024-11-12
AI Technical Summary
In the logistics system, when the data generated by the routing plan service is synchronized to the summary indicator query service, multiple data input and output are required, and the data generated by the detailed query service requires a large amount of summary operations during the synchronization process, which affects the system efficiency.
A wide table is created to contain primary key fields and impact fields. Through these fields, the data of the routing plan service is directly synchronized to the summary metric query service, reducing the number of data IO times and reducing the complexity of operations.
It improves the operation efficiency of the logistics system and reduces the number of data IO times and operation complexity in the data acquisition process of the summary indicator query service.
Smart Images

Figure CN118939662A_ABST
Abstract
Description
Technical Field
[0001] Embodiments of the present invention relate to the field of computer technology, and in particular, to a data processing method, apparatus, device, and storage medium. Background Art
[0002] At present, the logistics system in the supply chain field can simultaneously run multiple services such as routing planning services, detailed query services, and summary index query services. Specifically, the routing planning service can determine a detailed distribution plan for a logistics waybill (for example, it can be a complete distribution cycle including multiple routing nodes such as warehouses, sorting centers, and transshipment centers), the detailed query service can provide detailed routing information query services to business users, and the summary index query service can provide routing indicators (for example, it can be a waybill volume prediction indicator used to predict the waybill volume of routing nodes) to business users.
[0003] However, in the process of implementing the present invention, it is found that there are at least the following problems in the prior art: in the logistics system, the data generated by the route planning business can be synchronized to the detailed query business, and the data generated by the detailed query business can be synchronized to the summary index query business. In this way, the data acquisition process of the summary index query business needs to go through multiple data input and output (IO); and the data generated by the detailed query business needs to be subjected to a large number of summary operations in the process of being synchronized to the summary index query business. The above problems will affect the operating efficiency of the logistics system. Summary of the invention
[0004] The embodiments of the present invention provide a data processing method, device, equipment and storage medium, which can improve the operation efficiency of a logistics system.
[0005] In a first aspect, an embodiment of the present invention provides a data processing method, which may include: receiving a data synchronization request; wherein the data synchronization request carries routing data corresponding to at least one routing node of a target routing task; for each routing node of the target routing task, based on the routing data corresponding to the current routing node, determining the current primary key field value of the current routing node for each primary key field identifier in a pre-created wide table, and the current influence field value of the current routing node for each influence field identifier in the wide table, and based on each current primary key field value and each current influence field value, determining the indicator summary reference data corresponding to the current routing node; each influence field identifier corresponds to a different routing indicator; and updating the wide table according to the indicator summary reference data corresponding to each routing node of the target routing task.
[0006] In a second aspect, an embodiment of the present invention further provides a data processing device, which may include: a receiving module, a determining module, and an updating module;
[0007] Specifically, a receiving module is used to receive a data synchronization request; wherein the data synchronization request carries routing data corresponding to at least one routing node of a target routing task; a determining module is used to determine, for each routing node of the target routing task, the current primary key field value of each primary key field identifier in a pre-created wide table for the current routing node, and the current influence field value of each influence field identifier in the wide table for the current routing node based on the routing data corresponding to the current routing node, and determine the indicator summary reference data corresponding to the current routing node based on each current primary key field value and each current influence field value; each influence field identifier corresponds to a different routing indicator; an updating module is used to update the wide table according to the indicator summary reference data corresponding to each routing node of the target routing task.
[0008] In a third aspect, an embodiment of the present invention provides a data processing device, the data processing device comprising:
[0009] One or more processors; a memory for storing one or more programs; when the one or more programs are executed by the one or more processors, the one or more processors implement the data processing method provided by any embodiment of the present invention.
[0010] In a fourth aspect, an embodiment of the present invention further provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements a data processing method as provided in any embodiment of the present invention.
[0011] The embodiments of the above invention have the following advantages or beneficial effects:
[0012] In the technical solution provided by the embodiment of the present invention, a wide table can be created in advance, and the structure of the wide table can include multiple primary key fields and multiple influence fields. Among them, the primary key field value of each primary key field identifier of the wide table can uniquely represent a routing node in the distribution plan generated by the routing plan business, and each influence field identifier of the wide table corresponds to a different routing indicator, and the influence field value of each influence field identifier can represent the degree of influence of the routing node on each routing indicator. Based on the pre-created wide table, the embodiment of the present invention can directly synchronize the routing data generated by the routing plan business to the summary indicator query business. Specifically, after the summary indicator query service receives the routing data corresponding to each routing node of the target routing task generated by the routing plan service, for each routing node of the target routing task, the current primary key field value of the current routing node for each primary key field identifier in the wide table and the current influence field value of the current routing node for each influence field identifier in the wide table can be determined based on the routing data corresponding to the current routing node, and the indicator summary reference data corresponding to the current routing node can be determined based on each current primary key field value and each current influence field value; thereafter, the wide table can be updated according to the indicator summary reference data corresponding to each routing node of the target routing task.
[0013] It can be seen that in the embodiment of the present invention, the summary index query service can convert the routing data generated by the routing plan service into the index summary reference data based on the structure of the pre-created wide table. In this way, the data acquisition process of the summary index query service only needs to go through one data IO, so the operation efficiency of the logistics system can be improved. In addition, in the process of mapping and converting the routing data generated by the routing plan service in the embodiment of the present invention, the various routing indicators are not directly summarized, but the influence of the routing node on each routing indicator is recorded with the influence field value. Obviously, the statistical calculation process for the influence field value is much simpler than the summary calculation process for each routing indicator. Therefore, the embodiment of the present invention can reduce the computational complexity of the mapping and conversion of the routing data, thereby further improving the operation efficiency of the logistics system. In summary, the embodiment of the present invention can improve the operation efficiency of the logistics system by reducing the number of data IO times in the data acquisition process of the summary index query service and reducing the computational complexity when processing the routing data. BRIEF DESCRIPTION OF THE DRAWINGS
[0014] Figure 1 It is a structural schematic diagram of a logistics system provided by the prior art;
[0015] Figure 2 It is a flowchart of a data processing method provided by an embodiment of the present invention;
[0016] Figure 3 It is a structural schematic diagram of a logistics system provided by an embodiment of the present invention;
[0017] Figure 4 is a flow chart of another data processing method provided by an embodiment of the present invention;
[0018] Figure 5 It is a structural diagram of an indicator calculation domain model provided by an embodiment of the present invention;
[0019] Figure 6 It is a structural diagram of a storage structure of a database provided by an embodiment of the present invention;
[0020] Figure 7 is a flowchart of another data processing method provided by an embodiment of the present invention;
[0021] Figure 8 It is a schematic diagram of a process of dynamically expanding routing indicators provided by an embodiment of the present invention;
[0022] Fig. 9 It is a structural diagram of a routing plan model provided by an embodiment of the present invention;
[0023] Fig.10 is a structural schematic diagram of a data processing subsystem provided by an embodiment of the present invention;
[0024] Fig.11 is a structural schematic diagram of a data processing device provided by an embodiment of the present invention;
[0025] Fig.12 It is a structural schematic diagram of a data processing device provided by an embodiment of the present invention. DETAILED DESCRIPTION
[0026] The present invention will be further described in detail below in conjunction with the accompanying drawings and embodiments. It is to be understood that the specific embodiments described herein are only used to explain the present invention, rather than to limit the present invention. It should also be noted that, for ease of description, only parts related to the present invention, rather than all structures, are shown in the accompanying drawings.
[0027] The term "and / or" in this article is merely a description of the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone.
[0028] In addition, the terms "including" and "having" and any variations thereof mentioned in the description of the embodiments of the present invention are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device including a series of steps or units is not limited to the listed steps or units, but may optionally include other steps or units that are not listed, or may optionally include other steps or units that are inherent to these processes, methods, products or devices.
[0029] It should be noted that, in the embodiments of the present invention, words such as "exemplary" or "for example" are used to indicate examples, illustrations or explanations. Any embodiment or design scheme described as "exemplary" or "for example" in the embodiments of the present invention should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a specific way.
[0030] In the description of the embodiments of the present invention, unless otherwise specified, “plurality” or “each” means two or more than two.
[0031] At present, the logistics system in the supply chain field can run multiple businesses at the same time. Businesses with the same focus can share the same data processing subsystem. For example, the detailed query business and the summary index query business can share the same data processing subsystem, and the routing plan business can use an independent data processing subsystem. Figure 1 , is a structural diagram of a logistics system in the prior art. Figure 1 As shown, the detailed query business and the summary index query business can share the same user interface (UI), for example Figure 1 The route planning service can use an independent UI, such as Figure 1 The console UI in .
[0032] like Figure 1 As shown, during the operation of the logistics system, the routing planning business can determine a detailed distribution plan for the logistics waybill generated in real time, and can store the routing data corresponding to the distribution plan in the database corresponding to the routing planning business (i.e. Figure 1 When the source DB receives the routing data, it can synchronously update the received routing data to the database corresponding to the detailed query service (i.e. Figure 1 When receiving the routing data, the destination DB1 can map and convert the routing data, convert it into routing detail data and store it, and can synchronously update the converted routing detail data to the database corresponding to the summary index query service (ie Figure 1After receiving the routing details data, the summary index query service can re-summarize the routing indicators based on the routing details data, and can store the summarized routing indicators. Among them, the detail query service can provide a detail query service to the service user based on the routing details data stored in the destination DB1 (corresponding to Figure 1 The data IO service 3 in the target DB2 can provide the service user with a routing index query service based on the routing index stored in the target DB2 (corresponding to Figure 1 Data IO service in 3').
[0033] From the data processing flow of the logistics system in the above prior art, it can be seen that the prior art has at least the following problems: the data acquisition process of the summary index query business needs to go through two data IOs (i.e. Figure 1 In addition, after the summary index query service receives the route detail data transmitted by the detail query service, it needs to perform a large amount of summary operations on the route detail data. The above problems will affect the operating efficiency of the logistics system.
[0034] In view of the above problems existing in the prior art, an embodiment of the present invention provides a data processing method. Figure 2 A flowchart of a data processing method provided in an embodiment of the present invention. The method can be executed by a data processing device provided in an embodiment of the present invention. The device can be implemented in software and / or hardware and integrated in a data processing device.
[0035] The data processing method provided by the embodiment of the present invention can be applied to a logistics system. In a possible implementation, a data processing device executing the method can be configured at a business end of a summary index query service in a logistics system, and correspondingly, a data processing device integrated with the data processing device can be a corresponding device of the business end of the summary index query service, such as a database server corresponding to the business end of the summary index query service.
[0036] In another possible implementation, the summary index query service and the detail query service in the logistics system involved in the embodiment of the present invention can share the same service terminal and the same database, and the data processing device for executing the method can be configured on the service terminal shared by the summary index query service and the detail query service (hereinafter referred to as the shared service terminal in the following description of the embodiment of the present invention). Figure 3 , is a schematic diagram of the structure of a logistics system provided by an embodiment of the present invention. Figure 3 As shown, the detailed query business and the summary index query business can share the same business terminal (corresponding to Figure 3Detailed query / aggregate indicator query in), and can share the same database destination DB. When the source DB receives the routing data, it can synchronously update the routing data to the destination DB; when the destination DB receives the routing data, it can map and convert the routing data into indicator summary reference data, which can simultaneously represent the routing detail data and the degree of influence (contribution value) of the routing nodes corresponding to the routing detail data on each routing indicator, and can store the converted indicator summary reference data. The shared business end can provide detailed query services to business users based on the indicator summary reference data stored in the destination DB (corresponding to Figure 3 Data IO service 3) and routing indicator query service (corresponding to Figure 3 In the following description of the embodiment of the present invention, the data processing device is configured on the common service end as an example.
[0037] like Figure 2 As shown, the data processing method provided by the embodiment of the present invention specifically includes the following steps:
[0038] S210: Receive a data synchronization request.
[0039] The data synchronization request carries routing data corresponding to at least one routing node of the target routing task.
[0040] In one possible implementation, after receiving a new logistics waybill, the routing planning service can generate a target routing task for the logistics waybill to guide the staff of the logistics system to complete the distribution processing of the logistics waybill. Specifically, the task content of the target routing task may include a distribution plan for processing the logistics waybill. The distribution plan may represent the following information: which routing nodes the logistics waybill needs to pass through for processing; and the detailed processing plan of each routing node for the logistics waybill (corresponding to the routing data in the embodiment of the present invention). After the routing planning service generates the target routing task for the logistics waybill, it can initiate a data synchronization request to the shared service end.
[0041] Routing nodes may be distribution processing outlets such as warehouses, sorting centers, transshipment centers, and distribution stations. For example, the distribution plan for logistics order A may be: first, items A and B in logistics order A are packaged in warehouse A, and then the package containing items A and B may be transported to sorting center A, where item C in logistics order A may be picked and packaged. Afterwards, the package containing item C and the package containing item A and item B may be transported to distribution station A by a predetermined transportation method, and the delivery personnel at distribution station A deliver the packages.
[0042] S220. For each routing node of the target routing task, based on the routing data corresponding to the current routing node, determine the current primary key field value of the current routing node for each primary key field identifier in the pre-created wide table, and the current influence field value of the current routing node for each influence field identifier in the wide table, and based on each current primary key field value and each current influence field value, determine the indicator summary reference data corresponding to the current routing node.
[0043] In an embodiment of the present invention, a wide table may be created in advance, and the structure of the wide table may include multiple primary key fields and multiple influence fields. The primary key field value of each primary key field identifier of the wide table may uniquely represent a routing node in the distribution plan generated by the routing plan business. For example, the primary key field identifier may be a routing node field identifier, and the primary key field value may be warehouse A.
[0044] Each influence field identifier of the wide table corresponds to a different routing indicator. For each influence field value of the current routing node, the current influence field value is used to characterize the degree of influence of the current routing node on the current routing indicator, and the current routing indicator is the routing indicator represented by the influence field identifier corresponding to the current influence field value. Exemplarily, the routing indicator may be a waybill quantity prediction indicator, and the influence field identifier may be the field identifier A1 corresponding to the waybill quantity prediction indicator. If the influence field value of A1 is 1, it may indicate that the waybill quantity generated by the current routing node is 1, that is, the contribution value of the routing data of the current routing node to the waybill quantity prediction indicator is 1.
[0045] It is understandable that in practical applications, the routing index may also be other evaluation indexes, which are not limited in the embodiments of the present invention. For example, the routing index may be a waybill quantity prediction index, an item shipment quantity prediction index, and a distribution quantity prediction index.
[0046] Still taking the distribution plan for logistics order A as an example, the routing nodes may include warehouse A, sorting center A, and distribution site A. In the distribution plan, the routing data representation of warehouse A requires packaging of items A and B in logistics order A at warehouse A, the routing data representation of sorting center A requires picking of item C in logistics order A at sorting center A, and the routing data representation of distribution site A requires delivery of each item in logistics order A at distribution site A. If the primary key field identifier in the wide table includes the routing node field identifier, each influence field identifier includes: field identifier A1 corresponding to the waybill quantity prediction index, field identifier A2 corresponding to the item shipment quantity prediction index, and field identifier A3 corresponding to the distribution quantity prediction index, then for the routing data of warehouse A, the influence field value of the routing data on field identifier A1 is 1, the influence field value on field identifier A2 is 2, and the influence field value on field identifier A3 is 0. Similarly, based on the structure of the wide table, the three indicator summary reference data in the wide table provided in Table 1 (i.e., the last three rows of data in Table 1) can be obtained:
[0047] Table 1
[0048]
[0049] Optionally, an embodiment of the present invention may determine the current influence field value of the current routing node for each influence field identifier in the wide table in the following manner: obtain the influence rule expression corresponding to each influence field identifier; for each influence field identifier, based on the influence rule expression corresponding to the current influence field identifier and the routing data corresponding to the current routing node, determine the current influence field value of the current routing node for the current influence field identifier.
[0050] Among them, the influence rule expression can be a formula determined in advance for calculating the influence field value. Exemplarily, taking the item shipment prediction index as an example, the influence rule expression of the influence field identifier corresponding to the item shipment prediction index can be: N=x. N represents the influence field value, and x represents the total number of items picked at the current routing node. Taking the distribution volume prediction index as an example, the influence rule expression of the influence field identifier corresponding to the distribution volume prediction index can be: N represents the influence field value, y represents the total number of items picked up by the current routing node, and j i Represents the volume of the i-th item among y items.
[0051] In the embodiment of the present invention, not only the influence degree of routing data on simple routing indicators such as item shipment prediction indicators can be characterized by the influence degree field value, but also the influence degree of routing data on complex routing indicators such as distribution volume prediction indicators can be characterized by the influence degree field value. Based on this, the embodiment of the present invention can determine the influence degree rule expression of the influence degree field identifier corresponding to each routing indicator in advance, so as to provide query services for various routing indicators to business users. In this way, the business functions of the logistics system can be enriched, and more detailed data references of routing indicators can be provided to the business users of the logistics system, which is convenient for business users to improve the logistics system. In addition, the influence degree rule expression is a statistical calculation formula, and the calculation performance loss is very small. Therefore, the embodiment of the present invention determines the influence degree field value based on the influence degree rule expression, which can further improve the operation efficiency of the logistics system.
[0052] S230 , summarizing reference data according to the indicators corresponding to each routing node of the target routing task, and updating the wide table.
[0053] After the shared business end determines the index summary reference data corresponding to each routing node of the target routing task, it can add several blank rows in the wide table. The number of newly added blank rows is equal to the number of nodes of each routing node of the target routing task. Then, based on the field identifiers in the wide table, the field values in the index summary reference data can be inserted into the newly added blank rows in sequence. For example, after inserting the three index summary reference data of logistics order A into the wide table, a wide table as shown in Table 1 can be obtained.
[0054] In the data processing method provided by the embodiment of the present invention, a wide table can be created in advance, and the structure of the wide table may include multiple primary key fields and multiple influence fields. Among them, the primary key field value of each primary key field identifier of the wide table can uniquely represent a routing node in the distribution plan generated by the routing plan business, and each influence field identifier of the wide table corresponds to a different routing indicator, and the influence field value of each influence field identifier can represent the degree of influence of the routing node on each routing indicator. Based on the pre-created wide table, the embodiment of the present invention can directly synchronize the routing data generated by the routing plan business to the summary indicator query business. Specifically, after the summary indicator query service receives the routing data corresponding to each routing node of the target routing task generated by the routing plan service, for each routing node of the target routing task, the current primary key field value of the current routing node for each primary key field identifier in the wide table and the current influence field value of the current routing node for each influence field identifier in the wide table can be determined based on the routing data corresponding to the current routing node, and the indicator summary reference data corresponding to the current routing node can be determined based on each current primary key field value and each current influence field value; thereafter, the wide table can be updated according to the indicator summary reference data corresponding to each routing node of the target routing task.
[0055] It can be seen that in the embodiment of the present invention, the summary index query service can convert the routing data generated by the routing plan service into the index summary reference data based on the structure of the pre-created wide table. In this way, the data acquisition process of the summary index query service only needs to go through one data IO, so the operation efficiency of the logistics system can be improved. In addition, in the process of mapping and converting the routing data generated by the routing plan service in the embodiment of the present invention, the various routing indicators are not directly summarized, but the influence of the routing node on each routing indicator is recorded with the influence field value. Obviously, the statistical calculation process for the influence field value is much simpler than the summary calculation process for each routing indicator. Therefore, the embodiment of the present invention can reduce the computational complexity of the mapping and conversion of the routing data, thereby further improving the operation efficiency of the logistics system. In summary, the embodiment of the present invention can improve the operation efficiency of the logistics system by reducing the number of data IO times in the data acquisition process of the summary index query service and reducing the computational complexity when processing the routing data.
[0056] Reference Figure 4 , is a flow chart of another data processing method provided by an embodiment of the present invention. The method in this embodiment can be combined with various optional solutions in the data processing method provided by the previous embodiment to further optimize the data processing method provided by the previous embodiment. Figure 4 As shown, the specific steps include:
[0057] S410: Receive a data synchronization request.
[0058] S420. For each routing node of the target routing task, based on the routing data corresponding to the current routing node, determine the current primary key field value of the current routing node for each primary key field identifier in the pre-created wide table, and the current influence field value of the current routing node for each influence field identifier in the wide table, and based on each current primary key field value and each current influence field value, determine the indicator summary reference data corresponding to the current routing node.
[0059] Optionally, each primary key field identifier includes at least a routing node field identifier, and the routing node field identifier includes at least a node type field identifier and a node attribute field identifier.
[0060] Among them, the node type field identifier can uniquely represent a node type, for example, the node type can include warehouse type, sorting center type, and distribution station type, etc. The node attribute field identifier can uniquely represent a routing node under a certain node type. If a1 is used to represent the node type field identifier of a warehouse type node, then when the current routing node is warehouse A, the node type field value can be a1, the node attribute field value can be warehouse A, or the node attribute field value can be A.
[0061] In practical applications, a business user may have a need to collect statistics on the routing indicators of a certain type of routing node. To meet the diverse statistical needs of business users, the routing node field in the wide table structure of the embodiment of the present invention may include a node type field and a node attribute field. In this way, it is possible to quickly collect statistics on the routing indicators of a certain type of routing node based on the routing node field and the indicator summary reference data.
[0062] Optionally, each primary key field identifier may also include at least one summary dimension field identifier and / or at least one routing detail field identifier.
[0063] In the embodiment of the present invention, the summary index query service and the detail query service can share the same service end and the same database. On this basis, the embodiment of the present invention can add a routing detail field to the structure of the wide table, that is, the data of the summary index query service and the detail query service can share the same wide table. In this way, the number of tables in the database of the shared service end can be reduced, thereby saving storage resources.
[0064] by Figure 3 Taking the logistics system shown as an example, after the destination DB receives the routing data of the current routing node transmitted by the routing plan business, it can map and convert the routing data of the current routing node based on the fields in the structure of the wide table to obtain the routing details data and influence data of the current routing node. The indicator summary reference data of the current routing node may include the routing details data and influence data.
[0065] The summary dimension may be a time dimension, a transportation schedule dimension, etc. For example, the time dimension may be a day or an hour, and the transportation schedule dimension may include three transportation schedules: morning, midday, and evening.
[0066] In actual applications, when a business user collects statistics on routing indicators, he or she may not only need to collect statistics on routing indicators of a certain routing node, but also need to collect statistics on routing indicators of a certain routing node on a certain day / a certain transport shift. In order to further meet the diverse statistical needs of business users, the embodiment of the present invention can also add multiple summary dimension field identifiers to the structure of the wide table to further enrich the business functions of the logistics system.
[0067] Optionally, each routing detail field identifier includes at least a predicted field identifier and an actual field identifier; when the routing data corresponding to the current routing node is predicted data, the current predicted field value of the current routing node for the predicted field identifier is a first preset field value, and the current actual field value of the current routing node for the actual field identifier is a second preset field value; when the routing data corresponding to the current routing node is actual data, the current predicted field value of the current routing node for the predicted field identifier is a second preset field value, and the current actual field value of the current routing node for the actual field identifier is a first preset field value.
[0068] The first preset field value and the second preset field value may be two different field values determined in advance. For example, the first preset field value may be 1, and the second preset field value may be 0.
[0069] In the application process of the embodiment of the present invention, after the routing plan service determines the distribution plan for the logistics waybill, the staff of the logistics system can complete the processing of the logistics waybill according to the distribution plan. However, in the process of the logistics system staff processing the logistics waybill according to the distribution plan, the actual processing process may have a certain error with the distribution plan. For example, the distribution plan of logistics waybill A can be successfully delivered at 16:00 on xx / xx / xxxx, but the actual time of successful delivery may have a certain time difference with 16:00 on xx / xx / xxxx. Based on this, in the embodiment of the present invention, the routing plan service can not only generate a distribution plan for the logistics order, obtain the routing data (i.e., predicted data) planned for each routing node, but also monitor the actual processing process of the logistics order, and synchronize the routing data (i.e., actual data) generated by the actual processing process to the shared business end. Correspondingly, some influence fields corresponding to the routing indicators related to the actual data can be added to the structure of the wide table. In this way, it is convenient for the shared business end to count the routing indicators related to the actual data based on the index summary reference data, which can further enrich the business functions of the logistics system.
[0070] Exemplarily, the routing indicators related to the actual data may be indicators such as the amount of missed operations, the amount of timeouts and failures to deliver, the amount of timeouts and failures to obtain, the amount of delays, the amount of on-time delivery, the amount of wrong deliveries, and the amount of wrong arrivals.
[0071] In order to distinguish whether the summary reference data of each indicator recorded in the wide table is actual data or predicted data, the embodiment of the present invention can add two routing detail field identifiers, namely the prediction field identifier and the actual field identifier, in the structure of the wide table. Correspondingly, when the shared service end performs statistical calculations on the routing indicators, it can distinguish the source of the current indicator summary reference data based on the prediction field value and the actual field value in the summary reference data of each indicator.
[0072] Optionally, each routing node of the target routing task has a different current node type field value for the node type field identifier; the routing node field identifier also includes a node subtype field identifier, the current routing node includes at least one routing subnode, and each routing subnode in the current routing node has a different current node subtype field value for the node subtype field identifier.
[0073] Among them, the node type subfield identifier can represent different node subtypes. The node subtype can represent the task type of the processing subtask in the routing node. Exemplarily, if the node type is a warehouse class, the node subtype can include two subnode classes: warehouse order acceptance and order packaging; if the node type is a sorting center class, the node subtype can include four subnode classes: sorting inspection, sorting shipment, sealing, and unsealing; if the node type is a distribution site class, the node subtype can include three subnode classes: site inspection, deliveryman receipt, and successful delivery.
[0074] In practical applications, multiple subtasks may need to be processed for a logistics order at the same routing node, and different subtasks may generate different routing indicators. Therefore, the embodiment of the present invention can split the routing node and split it into multiple routing subnodes based on the task processing flow of the routing node. In this way, it is convenient to count routing indicators with smaller granularity, thereby further enriching the business functions of the logistics system.
[0075] Optionally, the routing data corresponding to the current routing node includes the routing plan sequences corresponding to each routing sub-node in the current routing node. The embodiment of the present invention can determine the indicator summary reference data corresponding to the current routing node in the following manner: for each routing sub-node in the current routing node, based on the routing plan sequence corresponding to the current routing sub-node, determine the current node type field value of the current routing sub-node for the node type field identifier, the current node sub-type field value of the current routing sub-node for the node sub-type field identifier, and the current node attribute field value of the current routing sub-node for the node attribute field identifier, and determine the indicator summary reference sequence corresponding to the current routing sub-node based on the current node type field value, the current node sub-type field value, the current node attribute field value, and the current influence field values of the current routing sub-node; determine the indicator summary reference data corresponding to the current routing node according to the indicator summary reference sequences corresponding to each routing sub-node in the current routing node.
[0076] Exemplarily, still taking the distribution plan for logistics order A as an example, the routing nodes may include warehouse A, sorting center A, and distribution site A. If the current routing node is sorting center A, then each routing sub-node of the current routing node may include sorting inspection, sorting shipment, sealing vehicles, and unsealing vehicles. Correspondingly, the routing data of the current routing node received by the shared business end may include the routing plan sequences corresponding to sorting inspection, sorting shipment, sealing vehicles, and unsealing vehicles, respectively. For these four routing plan sequences, the embodiment of the present invention may respectively determine four indicator summary reference sequences. Taking the current routing sub-node being sorting inspection as an example, the current node type field value may be a1, the current node sub-type field value may be b1 (indicating sorting inspection), and the current node attribute field value may include sorting center A and inspection station A.
[0077] Referring to Table 2, illustratively, the embodiment of the present invention further provides another wide table structure:
[0078] Table 2
[0079]
[0080] As shown in Table 2, the structure of the wide table may include two major fields: primary key fields and influence fields. Among them, the primary key field may include three fields: routing node field, summary dimension field, and routing detail field, and the routing node field may include three sub-fields: node type field, node sub-type field, and node attribute field. Exemplarily, the summary dimension field may also include multiple dimension fields such as time dimension and transport shift dimension, the routing detail field may include multiple detail fields such as waybill number, prediction field, actual field, etc., and the influence field may include fields corresponding to multiple routing indicators such as waybill quantity prediction index, non-arrival quantity, missed operation quantity, pickup quantity, overtime non-delivery quantity, overtime non-pickup quantity, late quantity, on-time quantity, wrong delivery quantity, and wrong arrival quantity.
[0081] Based on the structure of the wide table provided in the embodiment of the present invention, refer to Figure 5 , the embodiment of the present invention also provides an indicator calculation domain model. Figure 5 As shown, four objects can be abstracted from the indicator calculation domain model, namely, value concerns (corresponding to the routing node field identifier in Table 2), summary dimensions (corresponding to the summary dimension field identifier in Table 2), routing indicators, and routing details (corresponding to the routing details field identifier in Table 2).
[0082] Among them, the indicator calculation domain model can be applied to the domain layer of the common business end, and the data source of the common business end is the routing plan business, that is, the indicator calculation domain model is oriented to the routing plan business, so the abstraction of the indicator calculation domain model needs to return to the routing plan business. The routing data generated by the routing plan business can represent the responsible handover node of the package corresponding to the logistics waybill. Therefore, when abstracting the indicator calculation domain model, the routing node, the task handover sub-node of the routing node (i.e., the routing sub-node), and the routing line and other value concerns can be abstracted.
[0083] The statistical process of routing indicators depends on routing details. In order to facilitate the statistics of routing indicators and save storage resources, routing details can also be abstracted when abstracting the indicator calculation domain model. In addition, in practical applications, the granularity of routing details may be too large, so the routing details can also be decomposed into granularities that can adapt to value concerns. The statistical process of routing indicators also depends on the summary dimension, and the summary dimension also expands the granularity control of value concerns. For example, high dimensions can correspond to coarse granularity, and low dimensions can correspond to fine granularity.
[0084] In an embodiment of the present invention, the influence field value is used to record the degree of influence of the routing node of the routing task on the routing index. Based on this, the logical strategy of the aggregation service can also be abstracted in the index calculation domain model, which is used to aggregate the influence field values in the wide table based on value concerns, summary dimensions, and routing details to determine the routing index.
[0085] S430: Summarize reference data according to the indicators corresponding to the routing nodes of the target routing task, and update the wide table.
[0086] Optionally, an embodiment of the present invention may update the wide table in the following manner: for each routing node of the target routing task, the indicator summary reference data corresponding to the current routing node is searched in the wide table based on the current primary key field values of the current routing node; if it is determined that the indicator summary reference data corresponding to the current routing node is not found in the wide table, the indicator summary reference data corresponding to the current routing node is updated to the wide table.
[0087] In the existing logistics system, the data acquisition process of the summary index query business may cause the summarized routing index to be inaccurate due to network interruption or hardware problems. Figure 1Taking the logistics system shown as an example, data IO service 2 is an unstable service. After the logistics system is abnormally restarted, repeated statistics of routing details may occur, resulting in inaccurate summarized routing indicators. For example, for the routing indicator index1, its structured query language (SQL) statement can be: update t set index1 = index1 + 1. For example, the current parameter value of index1 is 100, and the impact field value of a single logistics waybill on index1 is 1. Under normal circumstances, when a new logistics waybill is received, the current parameter value of index1 will be rewritten to 101. When the logistics system is abnormally restarted, the new logistics waybill may be counted twice, and the current parameter value of index1 will be rewritten to 102, resulting in repeated counting.
[0088] In order to solve the problem of inaccurate routing indicators summarized due to system restart in the prior art, the embodiment of the present invention can use a unified large wide table to record routing details and influence field values, and can specifically design the wide table to be idempotent. For example, taking the example of each primary key field identifier including routing node field identifier, summary dimension field identifier and routing detail field identifier, the embodiment of the present invention can use routing node field value, summary dimension field value and routing detail field value as the unique idempotent key of the wide table. Based on the unique idempotent key, when inserting indicator summary reference data into the wide table, if the indicator summary reference data to be inserted already exists in the wide table, the data insertion operation is not performed on the indicator summary reference data. In this way, it can be ensured that the same indicator summary reference data is only counted once, thereby avoiding the problem of inaccurate routing indicators summarized due to network interruption or hardware problems.
[0089] Optionally, the wide table includes various candidate data table sets, and each candidate data table set includes at least one candidate data table group; each candidate data table set is stored in a different storage node, and each candidate data table group in the current candidate data table set is stored in a different storage partition in the same storage node; the embodiment of the present invention can also update the wide table in the following manner: for each routing node of the target routing task, based on the current node type field value of the current routing node for the node type field identifier, and the current node attribute field value of the current routing node for the node attribute field identifier, determine the target storage node from each storage node; according to the current summary dimension field value of the current routing node for each summary dimension field identifier, determine the target storage partition from each storage partition of the target storage node; update the indicator summary reference data corresponding to the current routing node to the target candidate data table group of the target storage partition.
[0090] In an embodiment of the present invention, the shared business end needs to have the ability to perform aggregation operations on batches of routing indicators, and needs to have the ability to check routing details (for example, it has the ability to query the routing details of a logistics order). Based on this, an embodiment of the present invention can implement distributed storage of indicator summary reference data based on routing nodes, summary dimensions, etc. Specifically, the indicator summary reference data of different routing nodes can be stored in different storage nodes, and the indicator summary reference data of different summary dimensions can be stored in different storage partitions of the same storage node. In this way, the screening speed when checking routing details can be improved, so that invalid data can be filtered as much as possible when checking routing details. At the same time, it is also possible to implement as many parallel operations as possible when performing aggregation operations on routing indicators and then summarize the results of the parallel operations, thereby improving the aggregation efficiency.
[0091] Reference Figure 6 , is a schematic diagram of a storage structure of a database provided by an embodiment of the present invention. Figure 6 As shown, the storage structure of the database includes multiple storage nodes, and each storage node includes multiple storage partitions. For example, storage node A can correspond to candidate data table set A, which is used to store the index summary reference data generated by routing node A, that is, to store the data of the same value focus in the same storage node; storage node B can correspond to candidate data table set B, which is used to store the index summary reference data generated by routing node B; storage node C can correspond to candidate data table set C, which is used to store the index summary reference data generated by routing node C. Taking storage node C as an example, storage node C may include storage partition A, storage partition B, storage partition C, and storage partition D. For example, storage partition A may correspond to summary dimension A, which is used to store the data of the same value focus under the same summary dimension in the same data table.
[0092] In addition, the database server corresponding to the shared business terminal in the embodiment of the present invention may have both storage and computing capabilities, and may fully utilize data localization through mobile computing to achieve the effect of minimizing the transmission of computing results.
[0093] In addition, the database server can also have the functions of non-update writing and automatic merging during idle time. Specifically, the wide table structure supports non-update writing of route details (append writing, no update, no deletion). It only requires a simple positive or negative value for the impact field value to obtain the update or deletion effect, which can reduce the query performance loss introduced by update and delete operations. In addition, by creating a scheduled task, data that meets the aggregation conditions within the same storage node or the same storage partition can be merged. In this way, the amount of data in the data table can be effectively reduced, and the retrieval scope when the routing indicators are counted can be narrowed, thereby further improving the operating efficiency of the logistics system.
[0094] Understandably, Figure 6 The storage structure of the database provided is only an example and does not constitute a limitation on the storage structure of the database in the embodiment of the present invention. For example, in actual applications, the storage partition can be further divided to obtain multiple storage sub-partitions, so that while quickly counting the routing indicators, the same storage sub-partition under the same storage partition under the same storage node can be aggregated by running a single machine multi-threaded operation in parallel to make full use of the parallel capability of the single machine computing power; and the division of storage sub-partitions can also further improve the high performance of the automatic merging process during idle time (compressing the amount of data, reducing storage costs, and improving aggregation performance), so as to further improve the operating efficiency of the logistics system.
[0095] S440: Receive an indicator query request.
[0096] The indicator query request may be a request initiated by an external application, carrying the target primary key field values corresponding to each primary key field identifier. Figure 3 Taking the logistics system shown as an example, the indicator query request may be an indicator query request initiated by an external application corresponding to the view UI.
[0097] In addition, the embodiment of the present invention can provide the service user with both the route index query service and the route detail query service based on the wide table structure. In order to distinguish the two query services, the index query request can also carry the service identifier corresponding to the route index query service.
[0098] S450: Based on the target primary key field values, select candidate indicator summary reference data from the indicator summary reference data of the wide table.
[0099] For example, the primary key field identifiers carried by the indicator query request include the summary dimension identifier, the node type field identifier, and the node attribute field identifier. If the target primary key field values corresponding to the primary key field identifiers are: xxxx year xx month x day; a1; warehouse A, then the indicator query request is used to count the routing indicators of warehouse A on xxxx year xx month x day. If the summary reference data of each indicator of the wide table includes 100 data, and 10 of these 100 data are the data of warehouse A on xxxx year xx month x day, then the 10 selected data are the summary reference data of each candidate indicator.
[0100] S460. For each influence field identifier, based on the aggregation rule expression corresponding to the current influence field identifier and the task attributes of the candidate routing tasks corresponding to each candidate indicator summary reference data, an aggregation operation is performed on each target influence field value corresponding to the current influence field identifier in each candidate indicator summary reference data to obtain the parameter value of the routing indicator corresponding to the current influence field identifier.
[0101] The aggregation rule expression may be a pre-determined aggregation calculation formula. For example, the aggregation calculation formula may be a summation formula.
[0102] In practical applications, different types of routing tasks have different effects on routing indicators. Some routing tasks have positive effects on routing indicators, while some routing tasks have negative effects on routing indicators. Based on this, in the embodiment of the present invention, when statistics are collected on each routing indicator, it is also necessary to refer to the task attributes of the routing task corresponding to each indicator summary reference data (used to characterize the task type of the routing task).
[0103] For example, each candidate indicator summary reference data may include candidate indicator summary reference data A, candidate indicator summary reference data B, and candidate indicator summary reference data C. The routing tasks corresponding to candidate indicator summary reference data A and candidate indicator summary reference data B are to process the user's order placement business, and the routing task corresponding to candidate indicator summary reference data C is to process the user's return order business. When it is necessary to count the routing indicator of the number of actually completed orders, if the influence field values corresponding to the number of actually completed orders in candidate indicator summary reference data A, candidate indicator summary reference data B, and candidate indicator summary reference data C are all 1, then the actual influence field values of candidate indicator summary reference data A and candidate indicator summary reference data B on the number of actually completed orders are both 1, and the actual influence field value of candidate indicator summary reference data C on the number of actually completed orders is -1, then the statistical result of the aggregation operation of the number of actually completed orders of these three candidate indicator summary reference data is 2.
[0104] In the embodiment of the present invention, based on the above-mentioned embodiment, an indicator query service can be provided to the business user based on the field values in the wide table structure. In order to meet the differentiated needs when performing statistics on routing indicators of different types of routing services, the embodiment of the present invention can combine the aggregation rule expression and the task attributes of the routing service to perform statistics on the reference data of each candidate indicator. In this way, the business functions of the logistics system can be further enriched.
[0105] It should be noted that the data processing method proposed in the embodiment of the present invention and the aforementioned embodiment belong to the same inventive concept. For technical details not fully described in this embodiment, reference can be made to the aforementioned embodiment, and the beneficial effects of the aforementioned embodiment are also applicable to this embodiment.
[0106] Reference Figure 7 , is a flow chart of another data processing method provided by an embodiment of the present invention. The method in this embodiment can be combined with various optional solutions in the data processing method provided by the previous embodiment to further optimize the data processing method provided by the previous embodiment. Figure 7 As shown, the specific steps include:
[0107] S710: Receive a data synchronization request.
[0108] S720. For each routing node of the target routing task, based on the routing data corresponding to the current routing node, determine the current primary key field value of the current routing node for each primary key field identifier in the pre-created wide table, and the current influence field value of the current routing node for each influence field identifier in the wide table, and based on each current primary key field value and each current influence field value, determine the indicator summary reference data corresponding to the current routing node.
[0109] S730 , summarizing reference data according to the indicators corresponding to each routing node of the target routing task, and updating the wide table.
[0110] S740: Receive an indicator query request.
[0111] The indicator query request carries the target primary key field values corresponding to each primary key field identifier.
[0112] S750 . Based on the values of the target primary key fields, select candidate indicator summary reference data from the indicator summary reference data of the wide table.
[0113] S760: Determine the aggregation order of the routing indicators corresponding to the respective impact field identifiers according to the aggregation levels respectively corresponding to the respective impact field identifiers, and determine the parameter value of each routing indicator based on the aggregation order and the summary reference data of each candidate indicator.
[0114] For example, if each routing indicator includes routing indicator A and routing indicator B, and routing indicator A is used to count the influence field value (such as field value a1) corresponding to routing indicator A in the reference data of all candidate indicators; and routing indicator B is used to count the field value a1 in each field value a1 that meets certain conditions (for example, the condition can be that the volume of items in the logistics waybill corresponding to field value a1 reaches a certain volume). It can be seen that routing indicator A is a statistical analysis of the reference data of all candidate indicators, and routing indicator B is a statistical analysis of the reference data of some candidate indicators. Based on this, in an embodiment of the present invention, it can be predetermined that the aggregation level of routing indicator A is higher than that of routing indicator B, and correspondingly, the aggregation order of routing indicator A is ranked before the aggregation order of routing indicator B. In this way, in the process of performing aggregation operations on each routing indicator, the search scope can be gradually narrowed, thereby reducing the number of searches, improving the efficiency of statistics on routing indicators, and further improving the business efficiency of the logistics system.
[0115] Optionally, the data processing method provided by the embodiment of the present invention may further include: receiving a configuration request; updating a pre-created routing indicator configuration table according to the impact field identifier to be configured and the configuration information; and updating the wide table based on the routing indicator configuration table when a wide table update instruction is detected.
[0116] The configuration request carries an identifier of the impact field to be configured and configuration information associated with the identifier of the impact field to be configured.
[0117] As the business needs of the business user change, the business user may have the need to add new routing indicators. In the prior art, the summary indicator query service is generally suspended first, and then the summary indicator query service is restarted after the business function of the summary indicator query service is reconfigured. This will cause the business user to be unable to use the summary indicator query service for a period of time. In an embodiment of the present invention, in order to meet the business user's demand for the expansion of routing indicators, the business user can be supported to dynamically expand the routing indicators.
[0118] Reference Figure 8 , is a schematic diagram of a process for dynamically expanding routing indicators provided by an embodiment of the present invention. Figure 8As shown, the embodiment of the present invention can add two functions, metadata management service and database proxy service (DB Proxy), to the summary index query business, wherein the metadata management service is used to manage the extension points, and the metadata management service can be associated with multiple lists (for example, a value focus list, a summary dimension list, and a routing index list). In the case of receiving a configuration request, the metadata management service can add the identifier of the impact field to be configured carried by the configuration request, as well as the configuration information, to each list associated with the metadata management service. DB Proxy can periodically access each list associated with the metadata management service (or can also access each list associated with the metadata management service when the system is restarted), and reconfigure the fields in the wide table based on the information in each list. After reconfiguration, the transit link index calculation object (used to map and convert routing data) can map and convert the routing data in the data definition language (DDL) or data manipulation language format obtained based on the reconfigured wide table, and store it in the destination DB. Among them, the destination DB can be in column storage mode.
[0119] In addition, since the routing index is determined based on the impact field value (depending on the impact rule expression), the aggregation rule expression, and the routing result type statistics, and the value focus points such as the routing node and the routing sub-node may need to be referenced in the process of calculating the routing index, when expanding the routing index, the configuration information such as the impact rule expression, the aggregation rule expression, the routing result type, and the value focus points corresponding to the routing index must also be associated with the routing index.
[0120] The routing result type may include a forward result type and a rollback result type. The forward result type indicates that the routing data has a positive impact on the routing index, and the rollback result type indicates that the routing data has a negative impact on the routing index.
[0121] In addition, since the expansion of routing indicators is a low-frequency operation and the amount of data managed by the metadata management service itself is very small, adding this function has little impact on the operating efficiency of the logistics system.
[0122] In order to meet the expansion requirements of routing indicators, the routing plan model can also be reconfigured based on the structure of the wide table provided in the embodiment of the present invention. Fig. 9 , an embodiment of the present invention provides a schematic diagram of the structure of a routing plan model. Fig. 9 As shown in the figure, the routing planning model includes warehouses, receiving outlets, and sorting centers (i.e. Fig. 9 Sorting / TC in), delivery points and other routing nodes (corresponding to Fig. 9), and each routing node can include multiple routing sub-nodes (corresponding to Fig. 9 In addition, the lines in the routing plan model can be distinguished between forward and reverse directions to distinguish different types of routing results.
[0123] In an embodiment of the present invention, based on the above-mentioned embodiment, since the aggregation conditions of different routing indicators are different, the routing indicators can be grouped based on the aggregation conditions, and each group can be sorted based on the aggregation conditions. Specifically, a group with fewer aggregation conditions can be ranked before a group with more aggregation conditions. In the process of counting each routing indicator, each routing indicator can be counted in turn based on the group sorting. In this way, the search scope can be gradually narrowed, the number of searches can be reduced, and the efficiency of counting routing indicators can be improved, thereby further improving the operating efficiency of the logistics system.
[0124] It should be noted that the data processing method proposed in the embodiment of the present invention and the aforementioned embodiment belong to the same inventive concept. For technical details not fully described in this embodiment, reference can be made to the aforementioned embodiment, and the beneficial effects of the aforementioned embodiment are also applicable to this embodiment.
[0125] Fig.10 The following is a schematic diagram of the structure of a data processing subsystem provided by an embodiment of the present invention. Fig.10 As shown, the system may include a view layer, an application layer, a domain layer, and a data layer. Among them, the view layer is used to display the functions of routing indicators and routing details, including indicator views, detail views, and intelligent control services. The application layer may include data warehouse technology (Extract-Transform-Load, ETL) services and aggregation services. The ETL service is used to map and transform the received routing data based on the indicator calculation domain model provided by the domain layer to obtain routing details (which can be displayed in the detail view of the view layer) and impact field values. The Aggregation service can aggregate the impact field values based on the routing details to obtain routing indicators (which can be displayed in the indicator view of the view layer). The intelligent control service can be used to perceive the user's query operations and receive query requests.
[0126] The data processing subsystem provided in the embodiment of the present invention and the data processing methods in the aforementioned embodiments belong to the same inventive concept. For details not fully described in the embodiment of the data processing subsystem, reference can be made to the embodiments of the aforementioned data processing method.
[0127] Fig.11 A schematic diagram of the structure of a data processing device provided in an embodiment of the present invention, the function of which is equivalent to Fig.10The data processing subsystem provided includes: a receiving module 1110 , a determining module 1120 , and an updating module 1130 .
[0128] Exemplarily, the receiving module 1110 may execute S210 in the above method embodiment, the determining module 1120 may execute S220 in the above method embodiment, and the updating module 1130 may execute S230 in the above method embodiment.
[0129] Specifically, the receiving module 1110 is used to receive a data synchronization request; wherein the data synchronization request carries routing data corresponding to at least one routing node of the target routing task;
[0130] The determination module 1120 is used to determine, for each routing node of the target routing task, the current primary key field value of each primary key field identifier in the pre-created wide table of the current routing node and the current influence field value of each influence field identifier in the wide table of the current routing node based on the routing data corresponding to the current routing node, and determine the index summary reference data corresponding to the current routing node based on each current primary key field value and each current influence field value; each influence field identifier corresponds to a different routing index;
[0131] The updating module 1130 is used to update the wide table according to the index summary reference data corresponding to each routing node of the target routing task.
[0132] Optionally, in a possible implementation manner, the determining module 1120 is specifically configured to:
[0133] Obtain the influence rule expressions corresponding to the influence field identifiers respectively; for each influence field identifier, determine the current influence field value of the current routing node for the current influence field identifier based on the influence rule expression corresponding to the current influence field identifier and the routing data corresponding to the current routing node.
[0134] Optionally, in another possible implementation, each primary key field identifier includes at least a routing node field identifier, and the routing node field identifier includes at least a node type field identifier and a node attribute field identifier.
[0135] Optionally, in another possible implementation, each primary key field identifier further includes at least one summary dimension field identifier and / or at least one routing detail field identifier.
[0136] Optionally, in another possible implementation, the wide table includes candidate data table sets, each candidate data table set includes at least one candidate data table group; each candidate data table set is stored in a different storage node, and each candidate data table group in the current candidate data table set is stored in a different storage partition in the same storage node;
[0137] The update module 1130 is specifically used for: for each routing node of the target routing task, based on the current node type field value of the current routing node for the node type field identifier, and the current node attribute field value of the current routing node for the node attribute field identifier, determining the target storage node from each storage partition of the target storage node according to the current summary dimension field value of the current routing node for each summary dimension field identifier; updating the indicator summary reference data corresponding to the current routing node to the target candidate data table group of the target storage partition.
[0138] Optionally, in another possible implementation, each routing detail field identifier includes at least a predicted field identifier and an actual field identifier; when the routing data corresponding to the current routing node is predicted data, the current predicted field value of the current routing node for the predicted field identifier is a first preset field value, and the current actual field value of the current routing node for the actual field identifier is a second preset field value; when the routing data corresponding to the current routing node is actual data, the current predicted field value of the current routing node for the predicted field identifier is a second preset field value, and the current actual field value of the current routing node for the actual field identifier is a first preset field value.
[0139] Optionally, in another possible implementation, each routing node of the target routing task has a different current node type field value for the node type field identifier; the routing node field identifier also includes a node subtype field identifier, the current routing node includes at least one routing subnode, and each routing subnode in the current routing node has a different current node subtype field value for the node subtype field identifier.
[0140] Optionally, in another possible implementation, the routing data corresponding to the current routing node includes routing plan sequences corresponding to each routing sub-node in the current routing node, and the determination module 1120 is specifically used to:
[0141] For each routing sub-node in the current routing node, based on the routing plan sequence corresponding to the current routing sub-node, determine the current node type field value of the current routing sub-node for the node type field identifier, the current node sub-type field value of the current routing sub-node for the node sub-type field identifier, and the current node attribute field value of the current routing sub-node for the node attribute field identifier, and determine the indicator summary reference sequence corresponding to the current routing sub-node based on the current node type field value, the current node sub-type field value, the current node attribute field value and the current influence field values of the current routing sub-node; determine the indicator summary reference data corresponding to the current routing node according to the indicator summary reference sequences corresponding to each routing sub-node in the current routing node.
[0142] Optionally, in another possible implementation manner, the data processing device provided by the embodiment of the present invention may further include a screening module and an aggregation operation module;
[0143] The receiving module 1110 is further configured to receive an indicator query request after the updating module 1130 updates the wide table according to the indicator summary reference data corresponding to each routing node of the target routing task; wherein the indicator query request carries the target primary key field value corresponding to each primary key field identifier;
[0144] A screening module is used to screen out candidate indicator summary reference data from the indicator summary reference data of the wide table based on the value of each target primary key field;
[0145] The aggregation operation module is used to perform aggregation operation on each target influence field value corresponding to the current influence field identifier in each candidate indicator summary reference data based on the aggregation rule expression corresponding to the current influence field identifier and the task attributes of the candidate routing tasks corresponding to each candidate indicator summary reference data, so as to obtain the parameter value of the routing indicator corresponding to the current influence field identifier.
[0146] Optionally, in another possible implementation manner, the data processing device provided by the embodiment of the present invention may further include a screening module;
[0147] The receiving module 1110 is further configured to receive an indicator query request after the updating module 1130 updates the wide table according to the indicator summary reference data corresponding to each routing node of the target routing task; wherein the indicator query request carries the target primary key field value corresponding to each primary key field identifier;
[0148] A screening module is used to screen out candidate indicator summary reference data from the indicator summary reference data of the wide table based on the value of each target primary key field;
[0149] The determination module 1120 is also used to determine the aggregation order of the routing indicators corresponding to each impact field identifier according to the aggregation level corresponding to each impact field identifier, and determine the parameter value of each routing indicator based on the aggregation order and the summary reference data of each candidate indicator.
[0150] Optionally, in another possible implementation, the receiving module 1110 is further configured to receive a configuration request; wherein the configuration request carries an identifier of the impact field to be configured and configuration information associated with the identifier of the impact field to be configured;
[0151] The updating module 1130 is further used to: update the pre-created routing indicator configuration table according to the to-be-configured impact field identifier and configuration information; and update the wide table based on the routing indicator configuration table when a wide table update instruction is detected.
[0152] Optionally, in another possible implementation, the updating module 1130 is specifically configured to:
[0153] For each routing node of the target routing task, the indicator summary reference data corresponding to the current routing node is searched in the wide table based on the current primary key field values of the current routing node; if it is determined that the indicator summary reference data corresponding to the current routing node is not found in the wide table, the indicator summary reference data corresponding to the current routing node is updated to the wide table.
[0154] The data processing device provided in the embodiment of the present invention belongs to the same inventive concept as the data processing method provided in the aforementioned embodiments. For details not fully described in the embodiment of the data processing device, reference can be made to the relevant contents of the aforementioned method embodiments, and the corresponding beneficial effects can also be referred to the beneficial effect analysis of the aforementioned method embodiments.
[0155] Fig.12 A structural schematic diagram of a data processing device provided by an embodiment of the present invention. Fig.12 A block diagram of an exemplary data processing device 12 suitable for use in implementing embodiments of the present invention is shown. Fig.12 The data processing device 12 shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.
[0156] like Fig.12 As shown, data processing device 12 is in the form of a general purpose computing device. Components of data processing device 12 may include, but are not limited to: one or more processors or processing units 16, system memory 28, and bus 18 connecting various system components (including system memory 28 and processing units 16).
[0157] Bus 18 represents one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, a processor or a local bus using any of a variety of bus architectures. By way of example, these architectures include, but are not limited to, an Industry Standard Architecture (ISA) bus, a Micro Channel Architecture (MAC) bus, an Enhanced ISA bus, a Video Electronics Standards Association (VESA) local bus, and a Peripheral Component Interconnect (PCI) bus.
[0158] The data processing device 12 typically includes a variety of computer system readable media. These media can be any available media that can be accessed by the data processing device 12, including volatile and non-volatile media, removable and non-removable media.
[0159] System memory 28 may include computer system readable media in the form of volatile memory, such as random access memory (RAM) 30 and / or cache memory (i.e., Fig.12 The data processing device 12 may further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, the storage system 34 may be used to read and write non-removable, non-volatile magnetic media ( Fig.12 not shown, usually called a "hard drive"). Although Fig.12 Not shown in the figure, a disk drive for reading and writing to a removable non-volatile disk (e.g., a "floppy disk"), and an optical disk drive for reading and writing to a removable non-volatile optical disk (e.g., a CD-ROM, a DVD-ROM, or other optical media) may be provided. In these cases, each drive may be connected to the bus 18 via one or more data medium interfaces. The system memory 28 may include at least one program product having a set (e.g., at least one) of program modules that are configured to perform the functions of the various embodiments of the present invention.
[0160] A program / utility 40 having a set (at least one) of program modules 42 may be stored, for example, in system memory 28, such program modules 42 including, but not limited to, an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment. Program modules 42 generally perform the functions and / or methods of the embodiments described herein.
[0161] The data processing device 12 may also communicate with one or more external devices 14 (e.g., keyboards, pointing devices, displays 24, etc.), one or more devices that enable a user to interact with the data processing device 12, and / or any device that enables the data processing device 12 to communicate with one or more other computing devices (e.g., network cards, modems, etc.). Such communication may be performed through an input / output (I / O) interface 22. Furthermore, the data processing device 12 may also communicate with one or more networks (e.g., local area networks (LANs), wide area networks (WANs), and / or public networks, such as the Internet) through a network adapter 20. Fig.12 As shown, network adapter 20 communicates with other modules of data processing device 12 via bus 18. It should be understood that although Fig.12Not shown, other hardware and / or software modules may be used in conjunction with the data processing device 12, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.
[0162] The processing unit 16 executes various functional applications and data processing by running the programs stored in the system memory 28, such as implementing the steps of the data processing method provided in the embodiment of the present invention.
[0163] Of course, those skilled in the art can understand that the processor can also implement the technical solution of the data processing method provided by any embodiment of the present invention.
[0164] This embodiment provides a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, the steps of a data processing method provided by the aforementioned embodiment of the present invention are implemented.
[0165] The computer storage medium of the embodiment of the present invention can adopt any combination of one or more computer-readable media. The computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. The computer-readable storage medium can be, for example, but not limited to: an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples (non-exhaustive list) of computer-readable storage media include: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this document, a computer-readable storage medium can be any tangible medium containing or storing a program, which can be used by an instruction execution system, device or device or used in combination with it.
[0166] Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, which carry computer-readable program code. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than a computer-readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.
[0167] The program code embodied on the computer readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.
[0168] Computer program code for performing the operations of the present invention may be written in one or more programming languages or a combination thereof, including object-oriented programming languages such as Java, Smalltalk, C++, and conventional procedural programming languages such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or data processing device. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).
[0169] It should be understood by those skilled in the art that the modules or steps of the present invention described above can be implemented by a general-purpose computing device, they can be concentrated on a single computing device, or distributed on a network composed of multiple computing devices, optionally, they can be implemented by a program code executable by a computer device, so that they can be stored in a storage device and executed by the computing device, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. Thus, the present invention is not limited to any specific combination of hardware and software.
[0170] Note that the above are only preferred embodiments of the present invention and the technical principles used. Those skilled in the art will understand that the present invention is not limited to the specific embodiments described herein, and that various obvious changes, readjustments and substitutions can be made by those skilled in the art without departing from the scope of protection of the present invention. Therefore, although the present invention has been described in more detail through the above embodiments, the present invention is not limited to the above embodiments, and may include more other equivalent embodiments without departing from the concept of the present invention, and the scope of the present invention is determined by the scope of the appended claims.
Claims
1. A data processing method, It is characterized in that The method comprises: Receive a data synchronization request; wherein the data synchronization request carries routing data corresponding to at least one routing node of the target routing task; For each routing node of the target routing task, based on the routing data corresponding to the current routing node, determine the current primary key field value of each primary key field identifier in the pre-created wide table of the current routing node, and the current influence field value of each influence field identifier in the wide table of the current routing node, and based on each current primary key field value and each current influence field value, determine the index summary reference data corresponding to the current routing node; each influence field identifier corresponds to a different routing index; The wide table is updated according to the indicator summary reference data respectively corresponding to each routing node of the target routing task.
2. The data processing method according to claim 1, It is characterized in that Determining, based on the routing data corresponding to the current routing node, a current impact field value of the current routing node for each impact field identifier in the wide table, includes: Obtaining influence rule expressions corresponding to the influence field identifiers respectively; For each of the influence field identifiers, based on the influence rule expression corresponding to the current influence field identifier and the routing data corresponding to the current routing node, a current influence field value of the current routing node for the current influence field identifier is determined.
3. The data processing method according to claim 1, It is characterized in that The primary key field identifiers include at least a routing node field identifier, and the routing node field identifiers include at least a node type field identifier and a node attribute field identifier.
4. The data processing method according to claim 3, It is characterized in that Each primary key field identifier also includes at least one summary dimension field identifier and / or at least one routing detail field identifier.
5. The data processing method according to claim 4, It is characterized in that The wide table includes candidate data table sets, each of which includes at least one candidate data table group; each of the candidate data table sets is stored in different storage nodes, and each of the candidate data table groups in the current candidate data table set is stored in different storage partitions in the same storage node; The updating of the wide table according to the indicator aggregation reference data respectively corresponding to each routing node of the target routing task includes: For each routing node of the target routing task, based on the current node type field value of the current routing node for the node type field identifier and the current node attribute field value of the current routing node for the node attribute field identifier, determine a target storage node from each storage node; Determine a target storage partition from each of the storage partitions of the target storage node according to the current summary dimension field value identified by the current routing node for each of the summary dimension fields; The indicator summary reference data corresponding to the current routing node is updated to the target candidate data table group of the target storage partition.
6. The data processing method according to claim 4, It is characterized in that Each of the routing detail field identifiers at least includes a predicted field identifier and an actual field identifier; In the case where the routing data corresponding to the current routing node is predicted data, the current predicted field value of the current routing node for the predicted field identifier is a first preset field value, and the current actual field value of the current routing node for the actual field identifier is a second preset field value; When the routing data corresponding to the current routing node is actual data, the current predicted field value of the current routing node for the predicted field identifier is the second preset field value, and the current actual field value of the current routing node for the actual field identifier is the first preset field value.
7. The data processing method according to claim 3, It is characterized in that The current node type field values of the node type field identifiers of the routing nodes of the target routing task are different; The routing node field identifier also includes a node subtype field identifier. The current routing node includes at least one routing subnode. The routing subnodes in the current routing node have different current node subtype field values for the node subtype field identifier.
8. The data processing method according to claim 7, It is characterized in that The routing data corresponding to the current routing node includes the routing plan sequences respectively corresponding to the routing sub-nodes in the current routing node, and the current primary key field value of the current routing node for each primary key field identifier in the pre-created wide table and the current influence field value of the current routing node for each influence field identifier in the wide table are determined based on the routing data corresponding to the current routing node, and the indicator summary reference data corresponding to the current routing node is determined based on each current primary key field value and each current influence field value, including: For each of the routing subnodes in the current routing node, based on the routing plan sequence corresponding to the current routing subnode, determine the current node type field value of the current routing subnode for the node type field identifier, the current node subtype field value of the current routing subnode for the node subtype field identifier, and the current node attribute field value of the current routing subnode for the node attribute field identifier, and determine the indicator summary reference sequence corresponding to the current routing subnode based on the current node type field value, the current node subtype field value, the current node attribute field value, and each of the current influence field values of the current routing subnode; The indicator summary reference data corresponding to the current routing node is determined according to the indicator summary reference sequences respectively corresponding to the routing sub-nodes in the current routing node.
9. The data processing method according to claim 1, It is characterized in that After the wide table is updated according to the indicator aggregation reference data corresponding to each routing node of the target routing task, the method further includes: Receive an indicator query request; wherein the indicator query request carries the target primary key field values corresponding to the primary key field identifiers; Based on each of the target primary key field values, filter out each of the candidate indicator summary reference data from each of the indicator summary reference data in the wide table; For each of the influence field identifiers, based on the aggregation rule expression corresponding to the current influence field identifier and the task attributes of the candidate routing tasks corresponding to each of the candidate indicator summary reference data, an aggregation operation is performed on each of the target influence field values corresponding to the current influence field identifier in the candidate indicator summary reference data to obtain the parameter value of the routing indicator corresponding to the current influence field identifier.
10. The data processing method according to claim 1, It is characterized in that After the wide table is updated according to the indicator aggregation reference data corresponding to each routing node of the target routing task, the method further includes: Receive an indicator query request; wherein the indicator query request carries the target primary key field values corresponding to the primary key field identifiers; Based on each of the target primary key field values, filter out each of the candidate indicator summary reference data from each of the indicator summary reference data in the wide table; According to the aggregation levels corresponding to the impact field identifiers, the aggregation order of the routing indicators corresponding to the impact field identifiers is determined, and the parameter value of each routing indicator is determined based on the aggregation order and the summary reference data of the candidate indicators.
11. The data processing method according to claim 1, It is characterized in that The method further comprises: Receive a configuration request; wherein the configuration request carries an identifier of an impact field to be configured and configuration information associated with the identifier of the impact field to be configured; According to the to-be-configured impact field identifier and the configuration information, a pre-created routing indicator configuration table is updated; When a wide table update instruction is detected, the wide table is updated based on the routing indicator configuration table.
12. The data processing method according to any one of claims 1 to 11, It is characterized in that The updating of the wide table according to the indicator aggregation reference data respectively corresponding to each routing node of the target routing task includes: For each routing node of the target routing task, searching the wide table for the index summary reference data corresponding to the current routing node based on each current primary key field value of the current routing node; When it is determined that the indicator summary reference data corresponding to the current routing node is not found in the wide table, the indicator summary reference data corresponding to the current routing node is updated to the wide table.
13. A data processing device, It is characterized in that include: A receiving module, configured to receive a data synchronization request; wherein the data synchronization request carries routing data corresponding to at least one routing node of a target routing task; A determination module, configured to determine, for each routing node of the target routing task, a current primary key field value of each primary key field identifier in a pre-created wide table for the current routing node and a current influence field value of each influence field identifier in the wide table for the current routing node based on the routing data corresponding to the current routing node, and determine the index summary reference data corresponding to the current routing node based on each current primary key field value and each current influence field value; each influence field identifier corresponds to a different routing index; An updating module is used to update the wide table according to the indicator summary reference data corresponding to each routing node of the target routing task.
14. A data processing device, It is characterized in that The data processing device comprises: one or more processors; A memory for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the data processing method according to any one of claims 1 to 12.
15. A computer-readable storage medium having a computer program stored thereon, It is characterized in that When the program is executed by a processor, the data processing method according to any one of claims 1 to 12 is implemented.
Citation Information
Patent Citations
Method and device for generating wide table and computer storage medium
CN111008521A
Data model creation method and terminal
CN112612835A
Data storage method and system and data query method and system
CN112860711A
Database system with highly denormalized database structure
US20150302058A1
