Data processing method and device, electronic equipment and storage medium

Abstract

Embodiments of the present application provide a data processing method and device, electronic equipment and storage medium. The method comprises: constructing and saving at least two categories of data tables according to source data of multiple data sources and characteristics of data objects; the categories represent categories corresponding to characteristics of the data objects; determining a target category that subscribes to a changed data source from the at least two categories according to a subscription relationship between the data sources and the categories; querying a data table of the target category according to the changed data source to obtain a corresponding query result; and updating a target wide table corresponding to the changed data source according to the query result. The embodiments of the present application can reduce the resource cost consumed by the change processing of the source data.

Description

Technical Field

[0001] This application relates to the field of computer data processing technology, and in particular to a data processing method, a data processing apparatus, an electronic device, and a storage medium. Background Technology

[0002] A data warehouse can acquire source data involved in the operation of an enterprise from data sources, and merge the acquired source data based on computing tasks to obtain multi-dimensional data such as wide tables. Through the analysis and summarization of multi-dimensional data, it can provide data support for the enterprise's decision-making process.

[0003] Currently, various wide tables subscribe to changes in the data source; thus, when the source data of the data source changes, the update chain corresponding to each wide table will be triggered.

[0004] In practical applications, the number of update chains typically corresponds to the number of wide tables. Thus, if the source data in the data source changes, multiple update chains will consume significant resources. Summary of the Invention

[0005] This application provides a data processing method that can reduce the resource costs consumed in processing changes to source data.

[0006] Correspondingly, embodiments of this application also provide a data processing device, an electronic device, and a storage medium to implement and apply the above-described methods.

[0007] To address the aforementioned problems, this application discloses a data processing method, the method comprising:

[0008] Based on the source data and characteristics of data objects from multiple data sources, construct and store data tables of at least two categories; the categories represent the categories corresponding to the characteristics of the data objects.

[0009] Based on the subscription relationship between the data source and the category, identify the target category that has subscribed to the changing data source from at least two categories;

[0010] Based on the changed data source, query the data table of the target category to obtain the corresponding query results;

[0011] Based on the query results, update the target wide table corresponding to the changed data source.

[0012] To address the aforementioned problems, this application discloses a data processing apparatus, the apparatus comprising:

[0013] A build and save module is used to build and save data tables of at least two categories based on source data from multiple data sources and the characteristics of data objects; the categories represent the categories corresponding to the characteristics of the data objects.

[0014] The target category determination module is used to determine the target category that has subscribed to the changing data source from at least two categories based on the subscription relationship between the data source and the category.

[0015] The query module is used to query the data table of the target category based on the changed data source to obtain the corresponding query results;

[0016] The update module is used to update the target wide table corresponding to the changed data source based on the query results.

[0017] Optionally, the data source for the change corresponds to a change category; the query results include: a first query result that matches the change category, and / or a second query result that does not match the change category;

[0018] The update module includes:

[0019] The first update module is used to update the first query result based on the changed source data corresponding to the changed data source, and to merge the data of the target wide table with the updated first query result; or

[0020] The second update module is used to merge the second query result with the change source data corresponding to the changed data source; or

[0021] The third update module is used to merge the second query result with the updated first query result.

[0022] Optionally, the at least two categories include at least two of the following categories: attribute category, path category, and routing plan category.

[0023] Optionally, the device further includes:

[0024] The merge module is used to merge the data tables of the at least two categories.

[0025] Optionally, the query module is specifically used to query the data table of the target category based on the data object identifier and processing identifier corresponding to the changed data source, so as to obtain the corresponding query results.

[0026] Optionally, the device further includes:

[0027] The flow control module is used to issue tokens within the execution thread scope according to preset flow control parameters, so that the execution thread processes the entry request corresponding to the data source according to the token.

[0028] Optionally, the preset flow control parameters are obtained based on inlet flow information and outlet flow threshold.

[0029] Optionally, the device further includes:

[0030] The aggregation module is used to aggregate historical and incremental data corresponding to the query target in order to obtain the updated data corresponding to the query target.

[0031] To address the aforementioned problems, this application discloses an electronic device, including: a processor; and a memory storing executable code thereon, wherein when the executable code is executed, the processor performs the method as described in any of the above embodiments.

[0032] To address the aforementioned issues, embodiments of this application disclose one or more machine-readable media storing executable code thereon, which, when executed, causes a processor to perform the method as described in any of the above embodiments.

[0033] Compared with the prior art, the embodiments of this application have the following advantages:

[0034] In the technical solution of this application embodiment, a query for the data table of the target category is triggered based on the subscription relationship between the data source and the category, and the target wide table corresponding to the changed data source is updated based on the query results. Since the update chain in this application embodiment can be a single update chain corresponding to the data table of the target category, compared with the multiple update chains corresponding to multiple wide tables in related technologies, the number of update chains can be reduced, thus reducing the resource cost consumed in processing changes to the source data.

[0035] Furthermore, this application embodiment can subscribe to all or some categories of the data source according to actual application needs. Therefore, this application embodiment can trigger queries for data tables of all or some categories. Compared to related technologies that trigger queries for wide tables across the entire chain, which typically include fields of all categories, this application embodiment, when triggering queries for data tables of some categories, can narrow the range of fields corresponding to the change processing of the source data, thus further reducing the resource costs consumed in the change processing of the source data. Attached Figure Description

[0036] Figure 1 This is a schematic diagram of the structure of a data warehouse system according to an embodiment of this application;

[0037] Figure 2 This is a schematic diagram of a data processing flow in related technologies;

[0038] Figure 3This is a schematic diagram of the data processing flow according to one embodiment of this application;

[0039] Figure 4 This is a flowchart of the steps of a data processing method according to an embodiment of this application;

[0040] Figure 5 This is a flowchart illustrating the process of constructing at least two types of data tables according to one embodiment of this application;

[0041] Figure 6 This is a flowchart of the steps of a data processing method according to an embodiment of this application;

[0042] Figure 7 This is a flowchart illustrating the processing of data tables of at least two categories according to one embodiment of this application;

[0043] Figure 8 This is a schematic diagram of the structure of a flow control device according to an embodiment of this application;

[0044] Figure 9 This is a flowchart illustrating the steps of a data processing method according to an embodiment of this application;

[0045] Figure 10 This is a flowchart illustrating the processing procedure for querying a target according to an embodiment of this application;

[0046] Figure 11 This is a schematic diagram of the structure of a data processing apparatus according to an embodiment of this application;

[0047] Figure 12 This is a schematic diagram of the structure of an exemplary device provided in one embodiment of this application. Detailed Implementation

[0048] To make the above-mentioned objectives, features and advantages of this application more apparent and understandable, the application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0049] This application's embodiments can be applied to data processing scenarios. These scenarios can include: analysis scenarios, statistical scenarios, and recommendation scenarios. Analysis scenarios can include: log analysis scenarios, traffic flow analysis scenarios, etc. Statistical scenarios can include: website access statistics scenarios, ranking scenarios, etc. Recommendation scenarios can include: advertising targeting scenarios, personalized recommendation scenarios, etc.

[0050] For example, in personalized recommendation scenarios, user search data, browsing data, and other data can be processed to obtain personalized recommendations. Similarly, in logistics scenarios, data on logistics objects such as packages can be processed to obtain information such as display data for these objects. The display data for logistics objects can be determined by those skilled in the art based on actual application needs. It is understood that the above application scenarios are merely examples, and the embodiments of this application do not limit specific application scenarios.

[0051] In logistics scenarios, this application embodiment allows for low-latency, high-accuracy data browsing of logistics nodes or links through data display. The aforementioned displayed data and its corresponding browsing can guide a company's daily operations. For example, it helps users control logistics nodes or links to ensure timely delivery of goods under control, avoiding situations like warehouse overload or missing goods. For instance, users can control material and human resources based on displayed data from logistics nodes such as warehouse nodes, transshipment nodes, and delivery nodes to prevent warehouse overload. Similarly, users can control logistics links based on displayed data representing outbound but uncollected information to prevent missing goods.

[0052] In this embodiment, the data object can be a composite information representation understood by the software. The data object may be an external entity, thing, incidental event or event, role, organizational unit, location, or structure, etc. For example, the data object may include: goods, logistics objects, etc.

[0053] Logistics objects can refer to the objects being transported. Logistics objects can include packages and services, among others. Information about a logistics object can include its name, type, or management identifier. In a logistics scenario, a logistics object may or may not have packaging; packaging can include boxes or bags. Logistics objects can be circulated together with their packaging.

[0054] According to one embodiment, the logistics process may include: warehousing, distribution, transshipment, or delivery. The warehousing process is used for centralized storage of logistics objects; the distribution process is used for loading and transporting logistics objects delivered to the distribution center according to the destination and transportation routes of the receiving end, and then implementing the plans; the transshipment process is used to realize joint (relay) transportation between different or the same modes of transportation; and the delivery process is used to deliver logistics objects to the receiving end.

[0055] A logistics node is a point in a logistics network that connects different logistics links. Logistics nodes can include: transshipment nodes, storage nodes, flow nodes, service nodes, etc.

[0056] Among them, transshipment nodes are those that connect different modes of transportation. Storage nodes are those that store goods. Circulation nodes are those that organize the movement of materials within the system; circulation warehouses, circulation centers, and distribution centers are examples of circulation nodes. Service nodes represent the service end of the logistics object; they are typically used to provide services related to logistics complaints such as lost or damaged goods, goods exceeding delivery time limits, and failure to deliver to the door.

[0057] It should be noted that the acquisition process of information, signals or data used in the embodiments of this application is carried out in compliance with the relevant data protection laws and policies of the country where the application is located, and with the authorization of the relevant owner.

[0058] In practical applications, data warehouse systems can be used to process source data from multiple data sources. (Refer to...) Figure 1 The diagram shows a structural schematic of a data warehouse system according to an embodiment of this application. The system may specifically include: a first data processing tool 101, a real-time computing engine 102, a second data processing tool 103, and a data service center 104.

[0059] The first data processing tool 101 can store source data from n data sources (n can be a positive integer), such as data source 1, data source 2, ..., data source n. The n data sources can include: data sources of a single service located in multiple regions, and / or, the n data sources can include: multiple data sources from different services.

[0060] The real-time computing engine 102 can merge source data from n data sources to obtain merged result data. The merged result data can include wide tables for the entire supply chain or specific service chains. A wide table can refer to a database table with many fields. In one implementation, the wide table can be a database table that links service-related metrics and dimensions together. In another implementation, the wide table can be a database table that links service-related metrics, dimensions, and attributes together. Different services can correspond to different wide table fields. The wide table for the entire supply chain can cover the union of the fields included in the source data from n data sources. For example, the union of the fields included in the source data from n data sources can be used to obtain the fields included in the wide table for the entire supply chain. The fields included in the wide table for specific service chains can be a subset of the fields in the wide table for the entire supply chain. In a logistics scenario, the wide tables for specific service chains can include: distribution wide tables, transportation wide tables, etc.

[0061] The second data processing tool 103 can store the merged result data. The data service center 104 can provide query services corresponding to the merged result data to display the corresponding display data.

[0062] Here, the data source can refer to the origin of the data, which is the device or original medium that provides the required data. In specific implementations, the data source can be the database or database server used by the database application. Data processing tool 101 and second data processing tool 103 can be tools such as ordinary databases and analytical databases.

[0063] In a logistics scenario, the query target corresponding to the merged result data can be a metric that the user is interested in. Examples of query targets can include at least one of the following: pickup data, outbound but not picked up data, loading data, received but not loaded data, arrival data, signature data, on-time rate data, lost logistics objects, damaged logistics objects, logistics objects exceeding delivery time, and not delivered to the door, etc.

[0064] In practical applications, source data can be incrementally collected and synchronized to a message middleware for temporary message storage. For example, a data pipeline is a type of message middleware, a piece of software that allows data to flow efficiently from one location to another. Changes in the source data can trigger a driving message, causing the first data processing tool 101 to mirror the source data in the message middleware. Correspondingly, the real-time computing engine 102 can merge source data from n data sources to obtain merged result data.

[0065] Currently, various wide tables subscribe to changes in the data source; thus, when the source data of the data source changes, the update chain corresponding to each wide table will be triggered.

[0066] In practical applications, the number of update chains typically corresponds to the number of wide tables. Thus, if the source data in the data source changes, multiple update chains will incur significant processing costs.

[0067] Reference Figure 2 The diagram illustrates a data processing flow in related technologies. A "distribution execution order" is a data source, a distribution wide table is a segmented wide table, and a full-link wide table is a full-link wide table. The first and second data processing tools can store the distribution wide table and the full-link wide table respectively.

[0068] Assuming both the distribution wide table and the end-to-end wide table subscribe to changes in the "Distribution Execution Order," when the "Distribution Execution Order" changes, the distribution wide table is updated twice in both the first and second data processing tools. The first update involves updating the distribution wide table based on the "Distribution Execution Order." The second update involves reading the corresponding data record from the end-to-end wide table based on the "Distribution Execution Order," and then updating the distribution wide table based on the "Distribution Execution Order" and that data record.

[0069] To address the technical problem of high resource costs in processing changes to source data, embodiments of this application provide a data processing solution. This solution may include: constructing and storing at least two types of data tables based on the characteristics of source data and data objects from multiple data sources; each category representing the category corresponding to the characteristics of the data object; determining the target category that has subscribed to the changed data source from at least two categories based on the subscription relationship between the data source and the category; querying the target category's data table based on the changed data source to obtain the corresponding query results; and updating the target wide table corresponding to the changed data source based on the query results.

[0070] This application embodiment constructs and stores at least two types of data tables based on source data from multiple data sources and the characteristics of data objects. Each category can represent the feature category corresponding to the characteristics of a data object. Taking a logistics object as an example, the categories can include: attribute categories, path categories, and routing plan categories. Attribute categories can include features such as weight, length, width, height, whether it is a valuable item, origin, and destination. Path categories can include the changing characteristics of the nodes corresponding to the logistics object, and can include features such as time, location, and action. Routing plan categories can represent a preset path for the logistics object, which can include a series of routing information from the origin to the destination, and this routing information can include node information between the origin and the destination. The routing plan category corresponds to the preset path of the logistics object, while the path category corresponds to the actual path of the logistics object.

[0071] The node information in the routing information can include: origin node, intermediate nodes, and destination node. In one implementation, the logistics object is shipped from a warehouse, which can be the origin node; it then goes through one or more intermediate nodes for sorting and transportation, and is finally delivered to the destination node. The distance between the destination node and the receiving location can meet preset conditions, such as the distance between the destination node and the receiving location being less than a distance threshold.

[0072] The data tables of different categories can be separate and independent of each other. For example, the data table corresponding to the attribute category can be an attribute table. Alternatively, the data table corresponding to the path category can be a path table. The data table corresponding to the routing plan category can be a routing plan table. Therefore, the at least two categories of data tables in this embodiment can achieve the separation of feature information corresponding to different feature categories of data objects.

[0073] This application embodiment represents the subscription of a data table of a certain category to a data source through a subscription relationship between the data source and the category. This subscription relationship can be used to trigger queries on the corresponding category's data table. This subscription relationship can be determined by those skilled in the art based on actual application needs. Referring to Table 1, a schematic diagram of a subscription relationship according to an embodiment of this application is shown.

[0074] Table 1

[0075] Data source A Category 1, Category 2 Data source B Category 1 Data source C Category 1, Category 2, Category 3 …… ……

[0076] In this embodiment of the application, when a change in source data is detected, the target category that has subscribed to the changed source data can first be determined from at least two categories based on the subscription relationship between the source data and the category; then, the data table of the target category is queried based on the changed source data; then, the first query result is updated based on the changed source data of the changed source data; and finally, the target wide table corresponding to the changed source data is updated based on the updated first query result.

[0077] This application embodiment triggers a query on the data table of the target category based on the subscription relationship between the data source and the category, and updates the target wide table corresponding to the changed data source based on the query results. Since the update chain in this application embodiment can be a single update chain corresponding to the data table of the target category, compared to the multiple update chains corresponding to multiple wide tables in related technologies, the number of update chains can be reduced, thus reducing the resource cost consumed in processing changes to the source data.

[0078] Furthermore, this application embodiment can subscribe to all or some categories of the data source according to actual application needs. Therefore, this application embodiment can trigger queries for data tables of all or some categories. Compared to related technologies that trigger queries for wide tables across the entire chain, which typically include fields of all categories, this application embodiment, when triggering queries for data tables of some categories, can narrow the range of fields corresponding to the change processing of the source data, thus further reducing the resource costs consumed in the change processing of the source data.

[0079] Reference Figure 3 This diagram illustrates a data processing flow chart of one embodiment of this application. Assuming an attribute category subscribes to changes in a "distribution execution order," when the "distribution execution order" changes, the query results corresponding to the attribute category can be retrieved from the first data processing tool, and the distribution wide table can be updated based on these query results. Compared to... Figure 2 The related technologies shown employ two update links. Figure 3 The embodiment shown in this application employs an update chain, which can reduce the resource costs consumed in processing changes to the source data.

[0080] Method Example 1

[0081] Reference Figure 4 The diagram illustrates a flowchart of a data processing method according to an embodiment of this application, which may specifically include the following steps:

[0082] Step 401: Based on the source data and characteristics of data objects from multiple data sources, construct and save data tables of at least two categories; these categories can represent the categories corresponding to the characteristics of the data objects.

[0083] Step 402: Based on the subscription relationship between the data source and the category, determine the target category that has subscribed to the changed data source from at least two categories;

[0084] Step 403: Based on the changed data source, query the data table of the target category to obtain the corresponding query results;

[0085] Step 404: Based on the query results, update the target wide table corresponding to the changed data source.

[0086] Figure 4 At least one step of the method shown can be executed by a real-time computing engine. It is understood that the embodiments of this application are for... Figure 4 The specific entity that performs the method shown is not restricted.

[0087] In step 401, the features of the data objects can be clustered to obtain the categories corresponding to the features of the data objects. Alternatively, the features of the data objects can be manually categorized to obtain the categories corresponding to the features of the data objects.

[0088] The source data from multiple data sources contains at least some fields that can be related to the characteristics of the data object. Therefore, data tables of at least two categories can be constructed and stored based on the categories corresponding to the fields contained in the source data from multiple data sources.

[0089] Taking logistics objects as an example, data objects can be categorized into: attribute categories, path categories, and routing plan categories. The attribute category data table can include fields related to attributes from multiple data sources, such as weight, length, width, and height, whether it's a valuable item, starting node, and destination node. The path category data table can include fields related to the actual path from multiple data sources, such as time, location, and action. The routing plan category data table can include fields related to a preset path from multiple data sources.

[0090] Reference Figure 5 The illustration shows a flowchart of constructing at least two types of data tables according to an embodiment of this application. In a logistics scenario, three types of data tables can be constructed and stored based on source data from multiple data sources and the characteristics of logistics objects: attribute table, path table, and routing plan table.

[0091] The process involves merging and formatting attribute data from data sources A1 and A2 to obtain an attribute table, which is then saved to the first data processing tool. Similarly, merging and formatting path data from data sources B1 and B2 yields a path table, which is also saved to the first data processing tool. Practical messages (actual operation messages of logistics objects) and preset path messages are obtained from the practical data of data source C1. Based on these messages, a candidate routing plan table is determined, and a routing plan table is finalized and saved to the first data processing tool. The routing information of the logistics object can be planned based on the practical messages and preset path messages to obtain a candidate routing plan table. This embodiment does not limit the specific process for determining the candidate routing plan table. The candidate routing plan table can be filtered to obtain the final routing plan table.

[0092] The attribute table, path table, and routing plan table in this application embodiment can be written to the first data processing tool. In order to control the write traffic of the first data processing tool, this application embodiment can perform traffic control at the corresponding entry points of the attribute table, path table, and routing plan table to reduce the traffic pressure on the first data processing tool and the next-level real-time computing engine.

[0093] In step 402, the subscription relationship between the data source and the category can represent the subscription of a certain category's data table to the data source. Those skilled in the art can determine the subscription relationship shown in Table 1 according to actual application requirements.

[0094] This subscription relationship can be used to trigger queries on the corresponding category's data table. Specifically, based on the subscription relationship between the data source and the category, the target category that has subscribed to the changing data source can be identified from at least two categories to trigger queries on the corresponding category's data table.

[0095] In a specific implementation, the subscription relationship may include: an identifier for the changed data source, and an identifier for the category's data table or category identifier. The target category that has subscribed to the changed data source can be found by searching the subscription relationship based on the changed data source identifier. The identifier for the changed data source can be determined by those skilled in the art based on actual application requirements; for example, the identifier for the changed data source may be the name or number of the changed data source.

[0096] In step 403, the target category's data table can be queried based on the changed data source to obtain the corresponding query results. For example, the target category's data table can be queried based on the identifier of the data object in the changed data source to obtain the corresponding query results.

[0097] In practical applications, there can be one or more target categories. Queries can be performed on tables containing all or only a subset of target categories.

[0098] For example, if a data source change corresponds to a change category, the target category can include: a first target category that matches the change category, and / or a second target category that does not match the change category. For instance, if the change category corresponding to the data source "Dispatch Execution Order" is a dispatch category or a path category, then the first target category can be a path category, and the second target category can be an attribute category or a routing plan category.

[0099] This application embodiment can query data tables of a first target category and / or a second target category based on a changed data source to obtain corresponding query results. The query results may include: a first query result matching the changed category, and / or a second query result not matching the changed category. The first query result may correspond to the first target category and originate from the data table of the first target category; that is, the first query result may be a first data record included in the data table of the first target category. The second query result may correspond to the second target category and originate from the data table of the second target category; that is, the second query result may be a second data record included in the data table of the second target category.

[0100] In one implementation, the data tables for the first target category and / or the second target category can be queried based on the identifiers of the data objects included in the changed data source to obtain the corresponding query results. The identifiers of the data objects can include: the identifiers of logistics objects, such as package IDs (identity). For example, querying the route table based on the package ID can retrieve all route data traversed by that package ID. A single data record for the route data can include: package ID, distribution ID, flow code, and time information, etc. In the case where one package ID corresponds to N (N can be a positive integer) distribution IDs, the query results can include: N data records corresponding to the N distribution IDs.

[0101] In another implementation, the target category's data table can be queried based on the data object identifier and processing identifier corresponding to the changed data source to obtain the corresponding query results. For example, querying the path table based on the package ID and distribution ID can retrieve the path data corresponding to that package ID and distribution ID. The path data corresponding to that package ID and distribution ID may include: target data records containing that package ID and distribution ID, and / or data records containing that package ID whose time information is later than the time information corresponding to the target data record.

[0102] In practical applications, a query request for the target category's data table can be generated based on the data object identifier and processing identifier corresponding to the changed data source. The query request can include the data object identifier and processing identifier corresponding to the changed data source. The data object identifier corresponding to the changed data source can be a data object identifier contained within the changed data source (such as a distribution ID).

[0103] Currently, data query dimensions are typically data objects. Taking a package as an example, if a package is transferred to distribution center ID3, triggering a package aggregation action, and the aggregation time needs to be updated in the distribution center wide table, the real-time computing engine doesn't know which distribution center ID triggered the aggregation. Therefore, it usually queries all the distribution center IDs the package passed through, ultimately outputting three data entries: data for distribution center ID1, ID2, and ID3. Querying and outputting only three data entries can easily lead to data bloat, resulting in data skew and affecting the stability of data processing.

[0104] This application's embodiments update the data query dimension from data object to data object identifier and processing identifier, which can reduce the additional read and write traffic caused by data expansion.

[0105] In step 404, based on the query result, the target wide table corresponding to the changed data source is updated, which can achieve updating the target wide table corresponding to the changed data source in one update chain. For example, the target wide table corresponding to the changed data source "distribution execution order" may include: distribution wide table. Of course, the target wide table corresponding to the changed data source "distribution execution order" may include: full-chain wide table.

[0106] This application embodiment can provide the following update method for updating the target wide table corresponding to the changed data source:

[0107] Update method 1: Update the first query result based on the changed source data corresponding to the changed data source, and merge the data of the target wide table with the updated first query result; or

[0108] Update method 2: Merge the second query result with the change source data corresponding to the changed data source; or

[0109] Update method 3: Merge the second query result with the updated first query result.

[0110] For update method 1, the first query result can be updated. Specifically, the changed fields included in the first data record corresponding to the first query result can be updated. For example, assuming the path table includes a collection time field, and the collection time in the changed data source "distribution execution order" has changed, the collection time field can be used as the changed field, and the collection time field in the path table can be updated.

[0111] It should be noted that updating the modified fields included in the first data record corresponding to the first query result can either update the first data record corresponding to the first query result, or create a new data record based on the updated first data record.

[0112] Updating the target wide table can be achieved by merging the data from the target wide table with the updated first query result. For example, matching data records that correspond to the updated first query result can be retrieved from the target wide table. The identifiers of the data objects corresponding to the matching data records and the updated first query result can be the same. Next, data fields can be matched between the updated first query result and the matching data records to obtain the matching data fields contained in the matching data records. Then, based on the updated first query result, the matching data fields contained in the matching data records can be updated. Assuming the matching data field is a data collection time field, and the content of the data collection time field in the matching data record is time 1, while the content of the data collection time field in the updated first query result is time 2, then the content of the data collection time field in the matching data record can be updated from time 1 to time 2.

[0113] For update method 2, the second query result can be merged with the change source data corresponding to the changed data source. Specifically, the second query result can correspond to a target category that has not changed. Merging the unchanged second query result with the change source data corresponding to the changed data source allows for updating the target wide table.

[0114] For example, if a first field matching the second query result can be determined from the fields of the target wide table, and a second field matching the changed source data result can be determined from the fields of the target wide table, then the field content of the first field can be determined based on the second query result, and the field content of the second field can be determined based on the changed source data. The field content of the first field and the field content of the second field can be merged to obtain a data record of the target wide table.

[0115] For update method 3, the second query result can be merged with the updated first query result. The second query result can correspond to the target category that has not changed, and the first query result can correspond to the target category that has changed. Merging the unchanged second query result with the updated first query result allows for updating the target wide table.

[0116] For example, if a first field matching the second query result can be determined from the fields of the target wide table, and a second field matching the updated first query result can be determined from the fields of the target wide table, then the field content of the first field can be determined based on the second query result, and the field content of the second field can be determined based on the updated first query result. The field content of the first field and the field content of the second field can be merged to obtain a data record of the target wide table.

[0117] This application embodiment can also merge at least two types of data tables to obtain a full-link wide table. It should be noted that the merging in this application embodiment can refer to merging fields of the same data object. This can be done by taking the union of fields from two data tables, or by selecting a target field from the union of fields from two data tables according to merging requirements. For example, merging requirements may include: fields included in the target wide table or the full-link wide table, etc.

[0118] Reference Figure 7 This illustration shows a flowchart of processing data tables of at least two categories according to an embodiment of this application. Figure 7 It can be used to construct data tables of at least two categories and merge data tables of at least two categories to obtain a full-link wide table. It should be noted that the data tables of at least two categories can be updated according to updates to the data source.

[0119] Among them, it can be according to Figure 5 The processing flow shown constructs an attribute table, a path table, and a routing plan table based on source data from multiple data sources. Further, data deduplication and merging can be performed on the attribute table, path table, and routing plan table to obtain a full-link wide table. This full-link wide table can then be saved to a second data processing tool.

[0120] The end-to-end wide table obtained in this application embodiment can be subscribed to. For example, the subscribers of the end-to-end wide table may include: distribution links, site links, or logistics services, etc. In this way, when the end-to-end wide table changes, corresponding messages can be sent to the subscribers.

[0121] In summary, the data processing method of this application embodiment triggers a query on the data table of the target category based on the subscription relationship between the data source and the category, and updates the target wide table corresponding to the changed data source based on the query results. Since the update chain in this application embodiment can be a single update chain corresponding to the data table of the target category, compared with the multiple update chains corresponding to multiple wide tables in related technologies, the number of update chains can be reduced, thus reducing the resource cost consumed in processing changes to the source data.

[0122] Furthermore, this application embodiment can subscribe to all or some categories of the data source according to actual application needs. Therefore, this application embodiment can trigger queries for data tables of all or some categories. Compared to related technologies that trigger queries for wide tables across the entire chain, which typically include fields of all categories, this application embodiment, when triggering queries for data tables of some categories, can narrow the range of fields corresponding to the change processing of the source data, thus further reducing the resource costs consumed in the change processing of the source data.

[0123] By applying the technical solution of this application embodiment, when the data in the data source changes, this application embodiment can reduce the resource costs consumed by the job corresponding to the changed data source and downstream jobs. The aforementioned resource costs may include: traffic costs, storage resources, or CPU resources, etc.

[0124] In order to reduce the resource costs of source data change processing in this application embodiment, the resources of the first data processing tool and / or the second data processing tool can be scaled down. Scale-down can be used to reduce the number of nodes used for processing. The scale-down can be a one-time scale-down or a tiered scale-down. Tiered scale-down can alleviate system failures caused by one-time scale-down to a certain extent. For example, the scale-down order of the primary and backup nodes (including primary and backup nodes) of the first data processing tool may include: 40 primary and backup nodes -> 30 primary and backup nodes -> 20 primary and backup nodes -> 15 primary and backup nodes -> 15 nodes in load balancing mode. Similarly, the scale-down order of the disk nodes of the second data processing tool may include: 64 disk nodes -> 48 disk nodes -> 36 disk nodes -> 32 disk nodes.

[0125] Method Example 2

[0126] Reference Figure 6 The diagram illustrates a flowchart of a data processing method according to an embodiment of this application, which may specifically include the following steps:

[0127] Step 601: Based on the source data and characteristics of data objects from multiple data sources, construct and save data tables of at least two categories; these categories can represent the categories corresponding to the characteristics of the data objects.

[0128] Step 602: Based on the subscription relationship between the data source and the category, determine the target category that has subscribed to the changed data source from at least two categories;

[0129] Step 603: Based on the changed data source, query the data table of the target category to obtain the corresponding query results;

[0130] Step 604: Based on the query result, update the target wide table corresponding to the changed data source;

[0131] Compared to Figure 4 The method shown in Embodiment 1 may further include:

[0132] Step 605: Based on the preset flow control parameters, issue a token within the execution thread scope so that the execution thread can process the entry request corresponding to the data source based on the token.

[0133] In practical applications, either the first or second data processing tool may experience high peak traffic. For example, the traffic sources for the second data processing tool may include writes from the real-time computing engine and data backflow from the offline computing engine. These traffic sources result in high instantaneous write traffic for the second data processing tool.

[0134] Data backflow can refer to the process of importing data from the calculation result table of the offline computing engine into the corresponding data table (second data processing tool) of the real-time computing engine.

[0135] This application embodiment utilizes thread-level token bucket technology to control the ingress traffic corresponding to the data source, achieving thread-level traffic protection. This not only alleviates thread overload to some extent but also protects the underlying infrastructure. Furthermore, the token bucket parameters can be dynamically set, enabling dynamic traffic control. With dynamic traffic control, this application embodiment can reduce the peak traffic of the first or second data processing tool.

[0136] The real-time computing engine in this embodiment can be positioned between the first data processing tool and the second data processing tool. In other words, the input of the real-time computing engine can receive source data output from the first data processing tool, and the output of the real-time computing engine can output the processing result of the source data to the second data processing tool. Therefore, the entry point in this embodiment can be the entry point of the first data processing tool, thus reducing the peak traffic of the first data processing tool. The exit point in this embodiment can correspond to the entry point of the second data processing tool; by controlling the entry traffic, this embodiment can also reduce the traffic at the exit point, i.e., the second data processing tool. Of course, the entry point of the first data processing tool is merely an example of the entry point in this embodiment; in reality, this embodiment does not limit the specific entry point.

[0137] Token bucket technology adds tokens to a bucket at a set rate. Incoming requests that receive a token are allowed to proceed; otherwise, they are discarded. The set rate can be one of the token bucket parameters. Other parameters include the set capacity. If the bucket has been idle for a long time, it may accumulate many tokens. Sudden bursts of traffic that receive tokens from the bucket can then proceed, with the flow being released based on the number of tokens already in the bucket plus the set token issuance rate. Therefore, token bucket technology can handle sudden surges in requests over a short period.

[0138] The preset flow control parameters can be an upper limit on the inbound flow. Alternatively, the preset flow control parameters can be token bucket parameters such as set speed and set capacity. In this embodiment, tokens can be issued within the execution thread scope based on the preset flow control parameters to reduce the flow peak of the execution thread within a short period.

[0139] In this embodiment, the preset flow control parameters can be obtained based on inlet flow information and outlet flow threshold. Specifically, the preset flow control parameters can be determined based on the comparison between inlet flow information and outlet flow threshold. The outlet flow threshold can be the upper limit of the outlet flow.

[0140] If the peak flow rate corresponding to the inlet flow rate information is greater than the outlet flow rate threshold, the upper limit of the inlet flow rate can be reduced, or the set speed and / or set capacity can be decreased to control the peak flow rate corresponding to the inlet flow rate information within the outlet flow rate threshold. Alternatively, if the peak flow rate corresponding to the inlet flow rate information is less than the outlet flow rate threshold, the upper limit of the inlet flow rate can be increased, or the set speed and / or set capacity can be increased to increase the peak flow rate corresponding to the inlet flow rate information.

[0141] Since the token bucket parameters can be dynamically set according to the inbound traffic information and the outbound traffic threshold in this embodiment, data processing efficiency can be improved while controlling the traffic.

[0142] like Figure 7 As shown, flow control can be performed at the entry point of the first data processing tool when the attribute table, path table, and routing plan table are written to, in order to control the write flow of the first data processing tool. Alternatively, flow control can be performed at the entry point of the second data processing tool when the end-to-end wide table is written to, in order to control the write flow of the second data processing tool.

[0143] Reference Figure 8 The diagram shows a schematic of a flow control device according to an embodiment of this application. The device is used to control the flow of inbound requests and may specifically include a scheduling module, a configuration module, a token issuance module, and a request processing module.

[0144] The scheduling module is used to schedule entry requests and distribute the corresponding entry requests to the execution threads.

[0145] The configuration module is used to set token bucket parameters and send these parameters to the execution thread. This embodiment of the application can utilize a distributed coordination service to achieve dynamic push and loading, thereby enabling dynamic traffic control; and can, to some extent, avoid the problem of the real-time computing engine reloading token bucket parameters via restart.

[0146] The token issuance module and request processing module can be set up within their respective execution threads. The token issuance module is used to issue tokens within the execution thread based on preset flow control parameters. Specifically, it can determine token bucket parameters such as set speed and set capacity based on the upper limit of inbound traffic included in the preset flow control parameters, and issue tokens within the execution thread based on the token bucket parameters.

[0147] The upper limit of the inlet flow rate can be proportional to the set speed, and the upper limit of the inlet flow rate can be proportional to the set capacity. For example, the upper limit of the inlet flow rate can include a first inlet upper limit and a second inlet upper limit. If the first inlet upper limit is greater than the second inlet upper limit, then the set speed corresponding to the first inlet upper limit can be greater than the set speed corresponding to the second inlet upper limit, or the set capacity corresponding to the first inlet upper limit can be greater than the set capacity corresponding to the second inlet upper limit.

[0148] The request processing module is used to process the entry requests corresponding to the data source based on the token. An execution thread can be used to process entry requests. Specifically, the execution thread can process entry requests that have obtained a token, while ignoring entry requests that have not obtained a token.

[0149] This application's embodiments utilize token bucket technology to block traffic at the entry point, achieving thread-level traffic protection, preventing thread overload, and thus protecting the underlying infrastructure.

[0150] Furthermore, the embodiments of this application can alleviate the problem of traffic accumulation, achieve the purpose of flow control and speed control, and avoid data loss, data disorder, and other situations.

[0151] The flow control framework of this application embodiment can be attached to the execution thread, and the execution thread can use the execution chain mechanism of the real-time computing engine to perform processes such as token issuance included in flow control; therefore, this application embodiment can save flow control costs and achieve fine-grained flow control (thread level); it can avoid the performance loss problem caused by using the performance of the underlying infrastructure for flow control to a certain extent.

[0152] In summary, the embodiments of this application can reduce the dependence on the concurrency of the real-time computing engine by controlling the flow and speed, thereby alleviating the pressure on the scheduling system and improving the overall system stability.

[0153] In one implementation, the overall rate limit for a real-time computing engine's computing jobs is equal to the number of concurrent jobs multiplied by the single-machine rate limit. The single-machine rate limit can serve as an example of a flow control parameter. This application's embodiments allow modification of the single-machine rate limit and push the modified value to the computing jobs, achieving real-time rate control even when the computing jobs are restarted. The computing jobs can process data at a uniform rate according to the predetermined rate limit, thereby improving the stability of the computing jobs.

[0154] In another implementation, the offline computing engine feeds back data from the calculation result table to the second data processing tool of the real-time computing engine. To control the data feed back flow, this embodiment can read partition data from the offline computing engine, automatically triggering data feed back upon the creation of a new partition, thus achieving controllable triggering of the data feed back. Furthermore, flow control can be performed at the entry point of the data feed back. Specifically, based on preset flow control parameters, a token is issued within the execution thread scope, enabling the execution thread to process the entry request corresponding to the data source based on the token. After flow control, the peak traffic of the second data processing tool is effectively mitigated.

[0155] The dynamic flow control in this application embodiment can be time-sharing flow control. For example, during a shopping event, a large number of orders are usually generated in the early morning, so one set of flow control parameters can be used in the early morning; while after the early morning, the number of orders decreases, so another set of flow control parameters can be used.

[0156] The traffic control method of this application embodiment can be applied to multi-tenant scenarios. In other words, the traffic control service of this application embodiment can serve multiple tenants.

[0157] Method Example 3

[0158] Reference Figure 9 The diagram illustrates a flowchart of a data processing method according to an embodiment of this application, which may specifically include the following steps:

[0159] Step 901: Based on the source data and characteristics of data objects from multiple data sources, construct and save data tables of at least two categories; these categories can represent the categories corresponding to the characteristics of the data objects.

[0160] Step 902: Based on the subscription relationship between the data source and the category, determine the target category that has subscribed to the changed data source from at least two categories;

[0161] Step 903: Based on the changed data source, query the data table of the target category to obtain the corresponding query results;

[0162] Step 904: Based on the query result, update the target wide table corresponding to the changed data source;

[0163] Compared to Figure 4 The method shown in Embodiment 1 may further include:

[0164] Step 905: Summarize the historical and incremental data corresponding to the query target to obtain the updated data corresponding to the query target.

[0165] In logistics scenarios, when the daily number of logistics objects reaches tens or hundreds of millions, the pressure on the underlying data will be significant, sometimes resulting in long query times or query failures.

[0166] This application embodiment summarizes the historical and incremental data corresponding to the query target to obtain the updated data corresponding to the query target. This allows for querying based on the updated data, thereby reducing query time and improving the efficiency of successful queries.

[0167] The query target can be a metric that the user is interested in. Examples of query targets can include at least one of the following: pickup data, outbound but not picked up data, loading data, received but not loaded data, arrival data, signature data, on-time rate data, lost logistics objects, damaged logistics objects, logistics objects exceeding the delivery time, and not delivered to the door, etc.

[0168] The aggregation of historical and incremental data corresponding to the query target in this embodiment of the application may include: processing the historical data corresponding to the query target using an offline computing engine to obtain daily-level historical processing results and daily-hourly incremental processing results; and aggregating the daily-level historical processing results and daily-hourly incremental processing results to obtain a first aggregation result.

[0169] Historical processing results can include: historical processing results for N days, where N can be a positive integer. Then, the daily-level historical processing results and the hourly-level incremental processing results for the current day can be summarized to obtain a first summary result. For example, the first summary result can include: daily-level historical processing results before day T + historical processing results at time i on day T and before time i. Here, T can represent a day of the year, and i can represent a time on day T, where i is between [1, 24]. Assuming T is 20220530 and i is 9:00, then the first summary result can include: daily-level historical processing results before 20220530 + historical processing results before time [1, 9] on day T.

[0170] The aggregation of historical and incremental data corresponding to the query target in this embodiment may further include: aggregating the first aggregation result and the change source data of the changed data source to obtain a second aggregation result. The change source data of the changed data source may include data at the minute or second level between [i, i+1]. Therefore, this embodiment can improve the timeliness of data updates.

[0171] The computations corresponding to real-time computing engines typically involve an advance observation of an infinite stream. Since it's an advance observation, modifications to the observation are possible. Referring to Table 2, an example of a change in the value of a query target according to an embodiment of this application is shown. In Table 2, the status field of package ID1 may include: arrived but not picked up, picked up, and signed for. Specifically, at X1 month y1 day z1 hour w1 minute, the value corresponding to "arrived but not picked up" is 1, while at X2 month y2 day z2 hour w2 minute, the value corresponding to "picked up" is 1, causing the value corresponding to "arrived but not picked up" to be updated to 0.

[0172] Table 2

[0173]

[0174] The processing of query targets typically involves statistical processing such as counting status fields. Assuming that the query target value lastvalue represents the number of packages with "arrived but not picked up" as 1, then if the "arrived but not picked up" status of package ID1 changes from 1 to 0, the query target value lastvalue will also change.

[0175] This application embodiment can utilize rollback technology to update the query target value lastvalue, thereby improving the accuracy of the query target value. A single rollback can update the query target value lastvalue from lastvalue(m) to lastvalue(m-1).

[0176] In one implementation, a rollback flag can be added to the query target value that needs to be rolled back, and a rollback trigger condition can be set. If the trigger condition is met, a rollback operation can be performed on the query target value with the rollback flag. For example, if the query target value corresponds to a first field, and updating the second field will change the content of the first field, then the trigger condition can be: the second field associated with the first field is updated.

[0177] Reference Figure 10 The diagram illustrates a flowchart of the processing procedure for a query target according to an embodiment of this application. The data source for the query target may include: a real-time data source, as well as daily historical processing results and hourly incremental processing results provided by an offline computing engine.

[0178] The real-time computing engine can summarize the historical processing results at the daily level and the incremental processing results at the hourly level of the current day to obtain a first summary result; it can also summarize the first summary result and the changed source data of the changed data source to obtain a second summary result.

[0179] The real-time computing engine can also query the second summary results and use rollback technology to update the query target value (lastvalue) to improve the accuracy of the query target value. The query target value in this embodiment can be used for data applications such as dashboard display and summary alerts.

[0180] It should be noted that, for the sake of simplicity, the method embodiments are all described as a series of actions. However, those skilled in the art should understand that the embodiments of this application are not limited to the described order of actions, because according to the embodiments of this application, some steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also understand that the embodiments described in the specification are all preferred embodiments, and the actions involved are not necessarily required by the embodiments of this application.

[0181] Based on the above embodiments, this application also provides a data processing apparatus, referring to... Figure 11 The device may include the following modules:

[0182] The construction and storage module 1101 is used to construct and store data tables of at least two categories based on the source data and characteristics of data objects from multiple data sources; the categories represent the categories corresponding to the characteristics of the data objects.

[0183] The target category determination module 1102 is used to determine the target category that has subscribed to the changing data source from at least two categories based on the subscription relationship between the data source and the category;

[0184] The query module 1103 is used to query the data table of the target category based on the changed data source in order to obtain the corresponding query results;

[0185] The update module 1104 is used to update the target wide table corresponding to the changed data source based on the query results.

[0186] Optionally, the change data source may correspond to a change category; the query results may include: a first query result that matches the change category, and / or, a second query result that does not match the change category;

[0187] Update module 1104 may include:

[0188] The first update module is used to update the first query result based on the changed source data corresponding to the changed data source, and to merge the data of the target wide table with the updated first query result; or

[0189] The second update module is used to merge the second query result with the change source data corresponding to the changed data source; or

[0190] The third update module is used to merge the second query result with the updated first query result.

[0191] Optionally, the at least two categories include at least two of the following categories: attribute category, path category, and routing plan category.

[0192] Optionally, the device may further include:

[0193] The merge module is used to merge the data tables of the at least two categories.

[0194] Optionally, the query module 1103 is specifically used to query the data table of the target category based on the data object identifier and processing identifier corresponding to the changed data source, so as to obtain the corresponding query results.

[0195] Optionally, the device may further include:

[0196] The flow control module is used to issue tokens within the execution thread scope according to preset flow control parameters, so that the execution thread processes the entry request corresponding to the data source according to the token.

[0197] Optionally, the preset flow control parameters are obtained based on inlet flow information and outlet flow threshold.

[0198] Optionally, the device may further include:

[0199] The aggregation module is used to aggregate historical and incremental data corresponding to the query target in order to obtain the updated data corresponding to the query target.

[0200] In summary, the data processing apparatus of this application embodiment triggers a query on the data table of the target category based on the subscription relationship between the data source and the category, and updates the target wide table corresponding to the changed data source based on the query results. Since the update chain in this application embodiment can be a single update chain corresponding to the data table of the target category, compared with the multiple update chains corresponding to multiple wide tables in related technologies, the number of update chains can be reduced, thus reducing the resource cost consumed in processing changes to the source data.

[0201] Furthermore, this application embodiment can subscribe to all or some categories of the data source according to actual application needs. Therefore, this application embodiment can trigger queries for data tables of all or some categories. Compared to related technologies that trigger queries for wide tables across the entire chain, which typically include fields of all categories, this application embodiment, when triggering queries for data tables of some categories, can narrow the range of fields corresponding to the change processing of the source data, thus further reducing the resource costs consumed in the change processing of the source data.

[0202] This application also provides a non-volatile readable storage medium storing one or more modules (programs). When these modules are applied to a device, they enable the device to execute the instructions for the method steps in this application.

[0203] This application provides one or more machine-readable media storing instructions that, when executed by one or more processors, cause an electronic device to perform one or more of the methods described in the above embodiments. In this application, the electronic device includes devices such as servers and terminal devices.

[0204] Embodiments of this disclosure can be implemented as an apparatus with any suitable hardware, firmware, software, or any combination thereof, configured as desired, and the apparatus may include electronic devices such as servers (clusters) and terminals. Figure 12 An exemplary apparatus 1300 is schematically shown that can be used to implement the various embodiments described in this application.

[0205] In one embodiment, Figure 12 An exemplary device 1300 is shown, which includes one or more processors 1302, a control module (chipset) 1304 coupled to at least one of the processors 1302, a memory 1306 coupled to the control module 1304, a non-volatile memory (NVM) / storage device 1308 coupled to the control module 1304, one or more input / output devices 1310 coupled to the control module 1304, and a network interface 1312 coupled to the control module 1304.

[0206] Processor 1302 may include one or more single-core or multi-core processors, and processor 1302 may include any combination of general-purpose processors or special-purpose processors (e.g., graphics processors, application processors, baseband processors, etc.). In some embodiments, device 1300 can serve as a server, terminal, or other device as described in the embodiments of this application.

[0207] In some embodiments, apparatus 1300 may include one or more computer-readable media (e.g., memory 1306 or NVM / storage device 1308) having instructions 1314 and one or more processors 1302 that are combined with the one or more computer-readable media and configured to execute instructions 1314 to implement modules and thus perform the actions described in this disclosure.

[0208] In one embodiment, the control module 1304 may include any suitable interface controller to provide any suitable interface to at least one of the processors 1302 and / or any suitable device or component communicating with the control module 1304.

[0209] The control module 1304 may include a memory controller module to provide an interface to the memory 1306. The memory controller module may be a hardware module, a software module, and / or a firmware module.

[0210] Memory 1306 may be used, for example, to load and store data and / or instructions 1314 for device 1300. In one embodiment, memory 1306 may include any suitable volatile memory, such as suitable DRAM. In some embodiments, memory 1306 may include double data rate type quad synchronous dynamic random access memory (DDR4 SDRAM).

[0211] In one embodiment, the control module 1304 may include one or more input / output controllers to provide interfaces to the NVM / storage device 1308 and (one or more) input / output devices 1310.

[0212] For example, NVM / storage device 1308 may be used to store data and / or instructions 1314. NVM / storage device 1308 may include any suitable non-volatile memory (e.g., flash memory) and / or may include any suitable (one or more) non-volatile storage devices (e.g., one or more hard disk drives (HDDs), one or more optical disc drives (CDs), and / or one or more digital universal optical disc (DVD) drives).

[0213] NVM / storage device 1308 may include storage resources that are part of a device on which device 1300 is mounted, or that are accessible by the device but do not necessarily have to be part of the device. For example, NVM / storage device 1308 may be accessed via a network via one or more input / output devices 1310.

[0214] One or more input / output devices 1310 may provide an interface for device 1300 to communicate with any other suitable device. Input / output devices 1310 may include communication components, audio components, sensor components, etc. Network interface 1312 may provide an interface for device 1300 to communicate via one or more networks. Device 1300 may wirelessly communicate with one or more components of a wireless network according to any of one or more wireless network standards and / or protocols, such as accessing wireless networks based on communication standards, such as WiFi, 2G, 3G, 4G, 5G, etc., or combinations thereof.

[0215] In one embodiment, at least one of the processors 1302 may be logically packaged with one or more controllers (e.g., memory controller modules) of the control module 1304. In one embodiment, at least one of the processors 1302 may be logically packaged with one or more controllers of the control module 1304 to form a system-in-package (SiP). In one embodiment, at least one of the processors 1302 may be integrated with the logic of one or more controllers of the control module 1304 on the same die. In one embodiment, at least one of the processors 1302 may be integrated with the logic of one or more controllers of the control module 1304 on the same die to form a system-on-a-chip (SoC).

[0216] In various embodiments, device 1300 may be, but is not limited to, a server, desktop computing device, or mobile computing device (e.g., laptop computing device, handheld computing device, tablet computer, netbook, etc.). In various embodiments, device 1300 may have more or fewer components and / or different architectures. For example, in some embodiments, device 1300 includes one or more cameras, a keyboard, a liquid crystal display (LCD) screen (including a touchscreen display), a non-volatile memory port, multiple antennas, a graphics chip, an application-specific integrated circuit (ASIC), and a speaker.

[0217] In the device 1300, a main control chip can be used as a processor or control module, sensor data, position information, etc. are stored in a memory or NVM / storage device, the sensor group can be used as an input / output device, and the communication interface can include a network interface.

[0218] This application also provides an electronic device, including: a processor; and a memory storing executable code thereon, which, when executed, causes the processor to perform one or more methods as described in this application.

[0219] This application also provides one or more machine-readable media having executable code stored thereon, which, when executed, causes a processor to perform one or more of the methods described in this application.

[0220] As the device embodiment is basically similar to the method embodiment, the description is relatively simple, and relevant parts can be found in the description of the method embodiment.

[0221] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.

[0222] This application describes embodiments with reference to flowchart illustrations and / or block diagrams of methods, terminal devices (systems), and computer program products according to embodiments of this application. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing terminal device to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing terminal device, generate instructions for implementing the flowchart illustrations. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0223] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing terminal device to operate in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0224] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal equipment, causing a series of operational steps to be performed on the computer or other programmable terminal equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable terminal equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0225] Although preferred embodiments of the present application have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of the embodiments of the present application.

[0226] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or terminal device that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or terminal device. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or terminal device that includes said element.

[0227] The above provides a detailed description of a data processing method, a data processing device, an electronic device, and a storage medium provided in this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A data processing method, characterized in that, The method includes: Based on the characteristics of source data and data objects from multiple data sources, construct and store at least two types of data tables; the categories represent the categories corresponding to the characteristics of the data objects; the data tables of different categories are separated and independent of each other; If a change in source data is detected, the target category that has subscribed to the changed source data is determined from at least two categories based on the subscription relationship between the data source and the category. Based on the changed data source, query the data table of the target category to obtain the corresponding query results; Based on the query results, update the target wide table corresponding to the changed data source; The change data source corresponds to a change category; the query results include: a first query result that matches the change category, and / or a second query result that does not match the change category; The step of updating the target wide table corresponding to the changed data source includes: Based on the changed source data corresponding to the changed data source, update the first query result, and merge the data of the target wide table with the updated first query result; or Merge the second query result with the change source data corresponding to the changed data source; or Merge the second query result with the updated first query result.

2. The method according to claim 1, characterized in that, The at least two categories include at least two of the following categories: attribute category, path category, and routing plan category.

3. The method according to claim 1, characterized in that, The method further includes: Merge the data tables of at least two categories.

4. The method according to claim 1, characterized in that, The querying of the target category data table includes: Based on the data object identifier and processing identifier corresponding to the changed data source, the target category data table is queried to obtain the corresponding query results.

5. The method according to any one of claims 1 to 4, characterized in that, The method further includes: Based on preset flow control parameters, tokens are issued within the execution thread scope so that the execution thread processes the entry request corresponding to the data source based on the token.

6. The method according to claim 5, characterized in that, The preset flow control parameters are obtained based on the inlet flow information and the outlet flow threshold.

7. The method according to any one of claims 1 to 4, characterized in that, The method further includes: The historical and incremental data corresponding to the query target are summarized to obtain the updated data corresponding to the query target.

8. An electronic device, characterized in that, include: processor; and A memory having executable code stored thereon, which, when executed, causes the processor to perform the method as described in any one of claims 1-7.

9. One or more machine-readable media having executable code stored thereon, which, when executed, causes a processor to perform the method as claimed in any one of claims 1-7.

Images