Data record supplementing method and device

Abstract

The invention discloses a data record supplementing method and device, and relates to the technical field of computers. A specific embodiment of the method for supplementing the data records comprises the following steps: monitoring the number of the data records of a data record set; under the condition that the number of the data records is greater than or equal to a preset number threshold value, taking the data records stored in the data record set as target data records, and obtaining dimension information of the target data records; and supplementing the target data record by using the dimension information to generate a lateral table of the target data record. According to the embodiment, after the data records are accumulated to a certain degree, the dimension information of the data records is supplemented, the dimension information can be efficiently supplemented, the read-write pressure of a storage system and a response interface is relieved, and resource consumption is reduced.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a method and device for supplementing data records. Background Art

[0002] In the field of real-time data processing, it is often necessary to supplement some dimensional information for the data records being processed. For example, when the data record includes a product ID, the data record is supplemented with product information such as product name, product inventory, product price, etc. associated with the product ID. Usually, data records flow in the system in the form of data streams. Each time a data stream is received, the corresponding dimensional information is directly supplemented according to the data records in the data stream, that is, only the dimensional information corresponding to a single data stream is supplemented each time.

[0003] In the process of implementing the present invention, the inventors found that the prior art has at least the following problems:

[0004] Frequently adding dimension information to data records will bring great read and write pressure, occupy a large amount of system resources, and the efficiency of data record addition is low. Summary of the invention

[0005] In view of this, an embodiment of the present invention provides a method and device for supplementing data records, which can efficiently supplement dimensional information, reduce reading and writing pressure, and reduce resource consumption.

[0006] To achieve the above object, according to a first aspect of an embodiment of the present invention, a method for supplementing data records is provided, comprising:

[0007] Monitor the number of data records in a data record set;

[0008] When the number of data records is greater than or equal to a preset number threshold, the data records stored in the data record set are used as target data records, and dimension information of the target data records is obtained;

[0009] The target data record is supplemented by using the dimension information to generate a lateral table of the target data record.

[0010] Optionally, when the number of data records is less than a preset number threshold, the method further includes:

[0011] Storing the received new data records in the data record set until the number of data records in the data record set is greater than or equal to a preset number threshold;

[0012] When the number of data records in the data record set is greater than or equal to a preset number threshold, the data records stored in the data record set are used as target data records, the dimension information of the target data records is supplemented, and a lateral table of the target data records is generated.

[0013] Optionally, after generating the lateral table of the target data record, the method further includes:

[0014] Clear the storage space of the target data record.

[0015] Optionally, the method further comprises:

[0016] Determine the data record with the latest receiving time among the target data records;

[0017] The data record with the latest reception time is merged with each of the complementary data records included in the lateral table.

[0018] Optionally, the lateral table is obtained by executing a preset custom table function according to Flink SQL, and the business logic of the custom table function includes: determining the number of data records, obtaining dimension information, and supplementing data records.

[0019] Optionally, the method further comprises:

[0020] In response to receiving the query request, determining a corresponding query condition;

[0021] The lateral table is searched for the completed data record that meets the query condition.

[0022] Optionally, obtaining the dimension information of the target data record includes: determining a dimension primary key included in each target data record, and obtaining the dimension information corresponding to each dimension primary key;

[0023] The target data record is supplemented by using the dimension information, including: using the dimension information corresponding to each dimension primary key to supplement the target data record including the dimension primary key.

[0024] According to a second aspect of an embodiment of the present invention, there is provided a device for supplementing data records, comprising:

[0025] A monitoring module, used for monitoring the number of data records in a data record set;

[0026] an acquisition module, configured to, when the number of data records is greater than or equal to a preset number threshold, use the data records stored in the data record set as target data records and acquire dimension information of the target data records;

[0027] A supplementation module is used to supplement the target data record using the dimension information to generate a lateral table of the target data record.

[0028] Optionally, the device further comprises:

[0029] A storage module, used for storing the received new data record into the data record set until the number of data records in the data record set is greater than or equal to a preset number threshold;

[0030] A generation module is used to use the data records stored in the data record set as target data records, supplement the dimensional information of the target data records, and generate a lateral table of the target data records when the number of data records in the data record set is greater than or equal to a preset number threshold.

[0031] Optionally, the device further comprises:

[0032] A cleaning module is used to clean up the storage space of the target data record.

[0033] Optionally, the device further comprises:

[0034] A determination module, used to determine the data record with the latest receiving time among the target data records;

[0035] A merging module is used to merge the data record with the latest receiving time with each completed data record included in the lateral table.

[0036] Optionally, the lateral table is obtained by executing a preset custom table function according to Flink SQL, and the business logic of the custom table function includes: determining the number of data records, obtaining dimension information, and supplementing data records.

[0037] Optionally, the device further comprises:

[0038] A condition acquisition module, configured to determine a corresponding query condition in response to receiving a query request;

[0039] A query module is used to query the lateral table for the completed data records that meet the query conditions.

[0040] Optionally, obtaining the dimension information of the target data record includes: determining a dimension primary key included in each target data record, and obtaining the dimension information corresponding to each dimension primary key;

[0041] The target data record is supplemented by using the dimension information, including: using the dimension information corresponding to each dimension primary key to supplement the target data record including the dimension primary key.

[0042] According to a third aspect of an embodiment of the present invention, there is provided an electronic device, including:

[0043] one or more processors;

[0044] a storage device for storing one or more programs,

[0045] When the one or more programs are executed by the one or more processors, the one or more processors implement the method described in any one of the above embodiments.

[0046] According to a fourth aspect of an embodiment of the present invention, a computer-readable medium is provided, on which a computer program is stored, and when the program is executed by a processor, the method described in any one of the above embodiments is implemented.

[0047] An embodiment of the above invention has the following advantages or beneficial effects: after the data records are accumulated to a certain extent, the dimension information of the data records is supplemented, which can efficiently supplement the dimension information, reduce the read and write pressure of the storage system and the response interface, and reduce resource consumption; when the number of data records is less than the quantity threshold, the data records are continued to be accumulated, which can avoid frequent query and supplement of dimension information, further reduce the read and write pressure, and reduce resource consumption; timely clean up the storage space of the target data records, which can facilitate the storage of newly received data records, improve the utilization rate of storage space, and effectively control the size of the storage space occupied by data records; merge the data record with the latest reception time with the side table, so as to obtain a complete data record, which is convenient for improving the query efficiency of the data record; use Flink SQL to execute a pre-set custom table function to obtain the side table, which can reduce development costs and facilitate business maintenance and business expansion; query and complete data records according to query conditions, which can improve query efficiency and query accuracy; when the target data record includes multiple dimension primary keys, batch query dimension information to improve the efficiency of dimension information acquisition.

[0048] The further effects of the above-mentioned non-conventional optional manner will be described below in conjunction with the specific implementation manner. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] The accompanying drawings are used to better understand the present invention and do not constitute an improper limitation of the present invention.

[0050] Figure 1 is a schematic diagram of the main process of the method for supplementing data records according to an embodiment of the present invention;

[0051] Figure 2 is a schematic diagram of a data record set according to a reference embodiment of the present invention;

[0052] Figure 3is a schematic diagram of data record flow according to a reference embodiment of the present invention;

[0053] Figure 4 is a schematic diagram of the overall process of data record supplementation according to a reference embodiment of the present invention;

[0054] Figure 5 is a schematic diagram of the main process of a method for supplementing data records according to a reference embodiment of the present invention;

[0055] Figure 6 is a schematic diagram of the main process of a method for supplementing data records according to another reference embodiment of the present invention;

[0056] Figure 7 is a schematic diagram of main modules of a device for data record supplementation according to an embodiment of the present invention;

[0057] Figure 8 is an exemplary system architecture diagram to which embodiments of the present invention may be applied;

[0058] Fig. 9 It is a schematic diagram of the structure of a computer system of a terminal device or a server suitable for implementing an embodiment of the present invention. DETAILED DESCRIPTION

[0059] The following is a description of exemplary embodiments of the present invention in conjunction with the accompanying drawings, including various details of the embodiments of the present invention to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present invention. Similarly, for clarity and conciseness, the description of well-known functions and structures is omitted in the following description.

[0060] It should be noted that in the technical solution of the present invention, the collection, use, storage, sharing and transfer of user personal information involved are in compliance with the provisions of relevant laws and regulations, and it is necessary to inform the user and obtain the user's consent or authorization. When applicable, the user's personal information is de-identified and / or anonymized and / or encrypted.

[0061] In the field of real-time data processing, it is often necessary to supplement some dimensional information for the data records being processed. For example, when the data record includes a product ID, the data record is supplemented with product information such as product name, product inventory, product price, etc. associated with the product ID. Usually, data records flow in the system in the form of data streams. Each time a data stream is received, the corresponding dimensional information is directly supplemented according to the data records in the data stream, that is, only the dimensional information corresponding to a single data stream is supplemented each time.

[0062] Frequently adding dimension information to data records will bring great read and write pressure, occupy a large amount of system resources, and the efficiency of data record addition is low.

[0063] In view of this, according to a first aspect of an embodiment of the present invention, a method for supplementing data records is provided.

[0064] Figure 1 FIG. 1 is a schematic diagram of the main process of the method for supplementing data records according to an embodiment of the present invention. Figure 1 As shown, the method for supplementing data records according to an embodiment of the present invention mainly includes the following steps S101 to S103.

[0065] Step S101, monitoring the number of data records in a data record set.

[0066] A data record set is a collection of data records. The data records stored in the data record set are data records that need to be supplemented with dimensional information. The number of data records stored in the data record set is the number of data records in the data record set. The number of data records in the data record set is monitored to determine the number of data records in the data record set. For example, after monitoring that a data record is added to the data record set or a data record is deleted from the data record set, the number of data records in the data record set is determined; for another example, a scheduled task is set to monitor the number of data records in the data record set at regular intervals; for another example, after receiving a new data record, the number of data records in the data record set is monitored to determine the number of data records in the data record set.

[0067] There can be one or more data record sets. For example, multiple data record sets can be set according to multiple business areas such as e-commerce, logistics, and entertainment. For another example, there are multiple links in the same business process that require additional dimensional information. Therefore, a corresponding data record set is set according to each link that requires additional dimensional information. Different data record sets classify and store data records from different business areas or different links in the same business process. Different data record sets are monitored according to monitoring needs, that is, the monitoring rules for the number of data records in different data record sets can be the same or different.

[0068] A data structure that can be referenced by a data record set is a linked list. For example, a data record set named "dataList" is represented as "List <row>dataList", where "List" indicates that the data structure of the data record set is a linked list, and "Row" indicates that the data type of the data record stored in the data record set is "Row".

[0069] Figure 2 is a schematic diagram of a data record set according to a reference embodiment of the present invention. Figure 2 As shown, a data record set for the e-commerce field and a data record set for the logistics field are pre-set, wherein the data record sets are all linked lists, and the data records in the linked lists are sorted from small to large according to the storage time of the data records. The execution subject of the embodiment of the present invention stores the received data records at the end of the linked list, and when the number of data records in the data record set is greater than or equal to the quantity threshold, the dimension primary key of each data record is determined in sequence starting from the head of the linked list, the dimension information is obtained according to the dimension primary key, and the data record is supplemented according to the obtained dimension information.

[0070] As another example, data record set A1 is used to store data records from the e-commerce field, data record set A2 is used to store data records from the logistics field, and data record set A3 is used to store data records from the entertainment field; since the data records in the e-commerce field are updated faster, the number of data records in data record set A1 is determined each time an increase in data records is monitored; since the update speed of data records in the logistics field is relatively stable, the number of data records in data record set A2 is monitored at regular intervals; since the update speed of data records in the entertainment field is related to the development of entertainment activities, the number of data records in data record set A3 is monitored at regular intervals if no activity development instruction is received, and if an activity development instruction is received, the monitoring frequency of the number of data records in data record set A3 is increased.

[0071] Monitor the data record set and determine the number of data records in the data record set to provide a data basis for subsequently determining the timing of data record supplementation; set up multiple data record sets and set up corresponding monitoring mechanisms based on data record supplementation needs, which can improve the flexibility of monitoring the number of data records and facilitate improving the efficiency of data record supplementation.

[0072] Step S102: When the number of data records is greater than or equal to a preset number threshold, the data records stored in the data record set are used as target data records, and dimension information of the target data records is obtained.

[0073] After determining the number of data records in the data record set, it is determined whether the number of data records in the data record set is greater than or equal to a preset number threshold, which is used to indicate the minimum accumulation degree of data records. When the number of data records is greater than or equal to the number threshold, it means that the data records stored in the data record set have accumulated to a degree suitable for supplementing dimensional information, that is, when the data records have accumulated to the minimum accumulation degree, supplementing the dimensional information of the data records stored in the data record set will not bring too much read and write pressure to the storage system and the response interface, and the occupied system resources are relatively small, so that the data records can be supplemented efficiently.

[0074] Specifically, when the number of data records in the data record set is greater than or equal to the quantity threshold, the data records stored in the data record set are used as target data records. Determine the dimension primary key included in each target data record. The dimension primary key is the data primary key of the data table where the dimension information is located. The dimension primary key is unique. Unique dimension information can be determined based on the dimension primary key. For example, if the dimension primary key is a user ID, the dimension information is user information. User information such as user name, user age, and user delivery address can be determined based on the user ID. Sort the target data records, determine the dimension primary key of each target data record in turn based on the sorting results, and then query the corresponding dimension information based on the dimension primary key.

[0075] It should be noted that the quantity threshold is configurable. For example, in response to receiving a quantity threshold configuration request, the request is parsed to obtain a new quantity threshold, and the new quantity threshold is used to replace the existing quantity threshold. For another example, performance indicators such as TPS (Transactions Per Second), QPS (Query Per Second), RT (Response Time), throughput, and concurrency are monitored. When the performance indicator reaches the indicator threshold, that is, the performance indicator deviates from the normal value, the quantity threshold is adjusted to restore the performance indicator to a normal value.

[0076] Exemplarily, the preset quantity threshold is 1000, that is, when the number of data records stored in the data record set is greater than or equal to 1000, the data records stored in the data record set are used as target data records; the target data records are sorted in ascending order according to the time when the target data records are added to the data record set, and the dimension primary key of each target data record is obtained in sequence according to the sorting result. The dimension primary key includes: product ID, location ID, order ID, etc., and product information such as product name, product price, and product inventory quantity is obtained according to the product ID; location information such as location name, township name, district and county name, and province and city name is obtained according to the location ID; order information such as ordering user, order time, ordered product, and delivery address is obtained according to the order ID, and each dimension information is associated with the corresponding dimension primary key, and then each dimension information is associated with the target data record containing the dimension primary key, so as to facilitate the subsequent use of dimension information to supplement the target data record.

[0077] When the number of data records in a data record set is greater than or equal to the quantity threshold, data records are supplemented to avoid frequent and repeated acquisition of dimension information, effectively reduce the read and write pressure of the storage system and response interface, reduce the occupied system resources, batch acquire dimension information, improve the efficiency of dimension information acquisition, and improve the efficiency of data record supplementation. The quantity threshold is configurable, which can improve the flexibility of dimension information acquisition and data record supplementation.

[0078] Step S103: Use the dimension information to supplement the target data record to generate a lateral table of the target data record.

[0079] After the dimension information is obtained, the dimension primary key included in the dimension information is determined, and the dimension primary key included in the dimension information is matched with the dimension primary key included in the target data record. The dimension information is used to supplement the target data record whose dimension primary key is successfully matched, and a lateral table of the target data record is generated. The lateral table is a data table of the dimension information of the target data record, and the lateral table includes the dimension information of each target data record. For example, the successfully matched dimension information is associated and stored with the target data record, and the data table including the dimension information is used as the lateral table of the target data record; for another example, a new data field is added to the target data record, and the new data field is the same as the obtained dimension information, so that the target data record after the new data field is added includes the dimension information, and the new data table obtained after the new data field is added is used as the lateral table.

[0080] Exemplarily, the lateral table includes: logistics document identification, shipper, shipping address, consignee, delivery address and other dimensional information. The LEFT JOIN keyword is used to associate the target data record with the dimensional information in the above lateral table to obtain the lateral table, which includes the target data record and the dimensional information in the above lateral table.

[0081] After the dimension information is obtained, the dimension information is associated with the target data record, or a new data table is generated, so that the dimension information of the target data record can be easily found.

[0082] According to a reference embodiment of the present invention, when obtaining the dimension information of the target data record, first determine the dimension primary key included in each target data record. When the target data record includes multiple dimension primary keys, obtain the dimension information corresponding to each dimension primary key in turn. One dimension primary key corresponds to a group of dimension information, and each target data record corresponds to one or more groups of dimension information. Exemplarily, the target data record includes: data record identifier, user identifier, order identifier, product identifier, etc., then first obtain user information such as user name and user login time according to the user identifier, then obtain order information such as order amount and order status according to the order identifier, and finally obtain product information such as product name, product quantity, product price according to the product identifier. Therefore, the dimension information corresponding to the target data record includes: user information, order information and product information.

[0083] When using dimension information to supplement the target data record, the dimension information corresponding to each dimension primary key is used to supplement the target data record including the dimension primary key. For example, the dimension information corresponding to a dimension primary key is first used to associate with the target data record, and then the dimension information corresponding to other dimension primary keys is used to associate with the previous association result, and finally, after multiple associations, the target data record including all dimension information is obtained. For another example, according to the target data record, all corresponding dimension information is stored in the same data table, and each target data record corresponds to a merged dimension information, and the merged dimension information includes: user information, order information, and product information, etc., and then the target data record is associated with the merged dimension information according to the dimension primary key, so that the corresponding dimension information can be determined according to one target data record.

[0084] When the target data record includes multiple dimensional primary keys, batch acquisition of the dimensional information of the target data record can improve the efficiency of data record supplementation, avoid long-term occupation of system resources, reduce the read and write pressure of the storage system where the dimensional information is located, and play a role in reducing costs and increasing efficiency.

[0085] According to a reference embodiment of the present invention, when the number of data records is less than a preset quantity threshold, the method further includes: storing the received new data records into a data record set until the number of data records in the data record set is greater than or equal to the preset quantity threshold. The execution subject of the embodiment of the present invention receives data records that need to be supplemented with dimensional information. After receiving the new data records, the new data records are stored into a data record set, and it is determined whether the number of data records stored in the data record set is greater than or equal to the preset data threshold. When the number of data records in the data record set is less than the quantity threshold, the new data records are continued to be received and stored until the number of data records in the data record set is greater than or equal to the quantity threshold. When the number of data records in the data record set is greater than or equal to the preset quantity threshold, the data records stored in the data record set are used as target data records, the dimensional information of the target data records is supplemented, and the lateral table of the target data records is generated. The specific implementation methods of supplementing dimensional information and generating lateral tables have been described in detail in the above steps and will not be repeated here.

[0086] For example, the preset quantity threshold is 100, and the number of data records received for the first time is 50, which is less than the quantity threshold. Therefore, instead of directly supplementing the dimension information for the received data records, the 50 received data records are stored in the data record set; then the data record received for the second time is 55, and the number of data records received twice is added to get 105, which is greater than the quantity threshold. Therefore, these 105 data records are used as target data records, and the dimension primary key of each data record is obtained, and the dimension information of the data record is supplemented according to the dimension primary key. When the number of data records is less than the quantity threshold, the data records are accumulated continuously until the number of data records is greater than or equal to the quantity threshold, and then the data records are supplemented, the dimension information is obtained, and the lateral table of the data records is generated. As another example, the number of data records used to obtain dimensional information is also determined by a quantity threshold. For example, in the above example, based on the reception time of the data records, the first 100 data records are used as target data records, that is, the dimensional information of 100 data records is supplemented each time, and the remaining 5 data records continue to be stored in the data record set, waiting for the next time when the number of data records is greater than or equal to the quantity threshold, and then the corresponding dimensional information is supplemented. In this way, the number of data records involved in each data record supplement is fixed, avoiding the supplement of too many data records at one time, so that each data record supplement is efficient, the system performance is stable, and less system resources are occupied.

[0087] When the number of data records is greater than or equal to the quantity threshold, it means that the data records have accumulated to a certain extent. At this time, obtaining dimension information based on the dimension primary key of the data record can improve the system's data processing performance, reduce resource usage, and avoid frequent acquisition of dimension information that causes excessive read and write pressure on the storage system, thereby reducing costs and increasing efficiency.

[0088] According to another reference embodiment of the present invention, after generating the lateral table of the target data record, the method further includes: clearing the storage space of the target data record. The target data record is stored in the data record set. After generating the lateral table of the target data record, it is no longer necessary to supplement the dimension information for the target data record. Therefore, the target data record is cleared from the data record set to ensure that the data records included in the data record set are data records for which the lateral table is not generated.

[0089] For example, using a linked list List <record>dataList stores data records, where Record is the object type of the data record, dataList is the name of the linked list, the dimension primary keys of the 50 target data records stored in the linked list dataList are obtained, the corresponding dimension information is obtained according to each dimension primary key, and the corresponding lateral table is generated; after the lateral table is generated, the method dataList.remove(index) is used to clear the 50 target data records in the linked list dataList, where index represents the index position of each target data record. As another example, 50 data records are stored in the linked list dataList, but the lateral table of only the first 40 data records is generated, that is, the first 40 data records in the linked list dataList are target data records, therefore, after the lateral table of the first 40 target data records is generated, only the first 40 target data records are cleared from the linked list dataList, leaving the last 10 data records, waiting for the next data record replenishment process.

[0090] After generating the lateral table of the target data record, timely clearing the target data record from the data record set can ensure that the data record set has sufficient storage space to accommodate new data records, improve the storage space utilization of the data record set, and avoid the data record set occupying too many system resources.

[0091] According to another reference embodiment of the present invention, the method further includes: determining the data record with the latest receiving time in the target data record, the execution subject of the embodiment of the present invention stores the receiving time of each data record, for example, sorting the target data records from small to large according to the receiving time, and the last data record is the data record with the latest receiving time. The data record with the latest receiving time is merged with each complementary data record included in the lateral table, the data records included in the lateral table are complementary data records, the complementary data records include the dimension information of the target data, and each target data record corresponds to a complementary data record. Specifically, after obtaining the complementary data record of each data record, each complementary data record is combined with the data record with the latest receiving time to obtain a combined data record, the number of combined data records is equal to the number of complementary data records, and each combined data record includes a complementary data record and a data record with the latest receiving time.

[0092] Figure 3 FIG. 1 is a schematic diagram of data record flow according to a reference embodiment of the present invention. Figure 3 As shown, the execution body of the embodiment of the present invention includes: multiple modules such as data receiving, data record storage, dimension information search and lateral table generation. In addition, the data receiving module receives the data record, and the data receiving module stores the received data record in the data record storage module. The preset quantity threshold is N. When the first N-1 data records are received, only the first N-1 data records are stored in the data record storage module, and the dimension information search module and the lateral table generation module have no output. After receiving the Nth data record and storing the Nth data record in the data record storage module, the dimension information search module and the lateral table generation module have no output. The information search module searches for the dimension information of the first data record (i.e., data record 1) to the Nth data record (i.e., data record N) to obtain dimension information 1 to dimension information N, and then the lateral table generation module associates data record N with each dimension information (i.e., dimension information 1 to dimension information N) to generate a lateral table of data record 1 to data record N; or, associates dimension information 1 with data record 1, associates dimension information 2 with data record 2, ..., associates dimension information N with data record N, and uses the association result as the lateral table of data record 1 to data record N.

[0093] Associating and merging the supplementary data record of the lateral table with the data record with the latest receiving time in the target data record can further supplement the information included in the lateral table, facilitate querying the information included in the target data record according to the lateral table, and meet different query requirements.

[0094] According to another embodiment of the present invention, the side table is obtained by executing a preset custom table function according to Flink SQL. Specifically, the application scenario of the execution subject of the embodiment of the present invention is: using Flink SQL to process real-time data, the developer writes a custom table function, and the business logic of the custom table function includes: judging the number of data records, obtaining dimension information, and supplementing data records. Among them, "judging the number of data records" refers to: receiving new data records, storing the data records in a data record set, judging whether the number of data records in the data record set is greater than or equal to the preset number threshold, and then executing "obtaining dimension information" when the number of data records is greater than or equal to the number threshold, otherwise, continuing to store the received data records in the data record set; "obtaining dimension information" refers to: taking each data record in the data record set as a target data record, determining the dimension primary key included in the target data record, and obtaining the corresponding dimension information according to the dimension primary key; "supplementing data records" refers to: supplementing the corresponding target data record according to the acquired dimension information, associating the dimension information with the target data record, or generating a new data record from the dimension information and the target data record, and taking the data record table including the dimension information as the side table. The execution subject of the embodiment of the present invention uses Flink SQL related syntax to execute the custom table function written by the developer, thereby generating a lateral table of target data records.

[0095] For example, the developer writes a custom UDTF (User-Defined Table-Generating Functions) function tableFunction, and then uses the syntax LATERAL TABLE(tableFunction) to execute the custom table function tableFunction to generate a lateral table of the target data records; and then performs LateralJoin on the target data records and the lateral table through Flink SQL.

[0096] Use custom table functions to define the overall process of data record supplementation, and use Flink SQL syntax LATERALTABLE to execute custom table functions, which can improve the efficiency of lateral table generation and reduce development and maintenance costs.

[0097] According to another referenceable embodiment of the present invention, the method further includes: in response to receiving a query request, parsing the query request, determining a corresponding query condition, the query condition including a query condition related to a dimension primary key or a query condition related to dimension information. Querying the side table for the completed data record that meets the query condition. Exemplarily, the query condition is: the user identifier is "uesr12", the order identifier is "order34", the product identifier is "goods56", and the side table is queried for the completed data record including the above-mentioned dimension primary key, and the user information with the user identifier "uesr12", the order information with the order identifier "order34", the product information with the product identifier "goods56", and so on are obtained. Again exemplarily, the side table includes the completed data record and the target data record. When querying the target data record, the query condition is: the data record identifier is "dataId78", and the data record with the data record identifier "dataId78" is queried in the side table, thereby avoiding determining the data record that meets the query condition in the large number of data records received.

[0098] Using the side table to query data records that meet the query conditions can improve the query efficiency of data records, narrow the query scope, further reduce the data reading pressure when querying data records, and reduce system resource usage.

[0099] Figure 4 FIG. 1 is a schematic diagram of the overall process of data record supplementation according to a reference embodiment of the present invention. Figure 4 As shown, in response to receiving a data record, the received data record is stored in a data record set, and then it is determined whether the data records in the data record set are greater than or equal to a preset quantity threshold, that is, it is determined whether the number of data records meets the search requirements of the dimension information; when the number of data records in the data record set is less than the preset quantity threshold, new data records continue to be received and stored in the data record set; when the number of data records in the data record set is greater than or equal to the preset quantity threshold, the data record in the data record set is used as the target data record, the dimension primary key of the target data record is obtained, the dimension information is searched according to the dimension primary key, the target data record is combined with the found dimension information, and a lateral table of the target data record is generated, and finally the data record set is cleaned up, and the data records in the data record set for which the lateral table has been generated are cleaned up.

[0100] Figure 5 FIG. 1 is a schematic diagram of the main process of a method for supplementing data records according to a reference embodiment of the present invention. Figure 5 As shown, the method for supplementing data records may include:

[0101] Step S501, storing the received new data record into a data record set until the number of data records in the data record set is greater than or equal to a preset number threshold;

[0102] Step S502, taking the data records stored in the data record set as target data records, and determining the dimension primary key included in each target data record;

[0103] Step S503, obtaining dimension information corresponding to each dimension primary key;

[0104] Step S504, using the dimension information corresponding to each dimension primary key, supplement the target data record including the dimension primary key to generate a lateral table of the target data record;

[0105] Step S505, determining the data record with the latest receiving time among the target data records;

[0106] Step S506: merge the data record with the latest receiving time with each of the supplementary data records included in the lateral table.

[0107] The specific implementation content of the method for supplementing data records as a reference embodiment of the present invention has been described in detail in the method for supplementing data records described above, so the repeated content will not be described here.

[0108] Figure 6 FIG. 2 is a schematic diagram of the main process of a method for supplementing data records according to another reference embodiment of the present invention. Figure 6 As shown, the method for supplementing data records may include:

[0109] Step S601, storing the received new data record into a data record set;

[0110] Step S602, determining whether the number of data records in the data record set is greater than or equal to a quantity threshold, if so, jumping to step S603, otherwise, jumping to step S601;

[0111] Step S603, determining a target data record from the data record set according to the storage time of the data record and the target number of target data records;

[0112] Step S604, determining the dimension primary key included in each target data record;

[0113] Step S605, taking the data records stored in the data record set as target data records, and determining the dimension primary key included in each target data record;

[0114] Step S606, obtaining dimension information corresponding to each dimension primary key;

[0115] Step S607, using the dimension information corresponding to each dimension primary key, supplement the target data record including the dimension primary key to generate a lateral table of the target data record;

[0116] Step S608, in response to receiving the query request, determining a corresponding query condition;

[0117] Step S609: Search the lateral table for completed data records that meet the query conditions.

[0118] The specific implementation content of the method for supplementing data records in another reference embodiment of the present invention has been described in detail in the method for supplementing data records described above, so the repeated content will not be described here.

[0119] According to a second aspect of an embodiment of the present invention, a device for supplementing data records is provided.

[0120] Figure 7 is a schematic diagram of the main modules of the device for data record supplementation according to an embodiment of the present invention, such as Figure 7 As shown, the data record supplement device 700 mainly includes:

[0121] Monitoring module 701, used to monitor the number of data records in the data record set;

[0122] An acquisition module 702 is configured to, when the number of data records is greater than or equal to a preset number threshold, use the data records stored in the data record set as target data records and acquire dimension information of the target data records;

[0123] The supplementing module 703 is used to supplement the target data record using the dimension information to generate a lateral table of the target data record.

[0124] According to a reference embodiment of the present invention, the data record supplementing device 700 further includes:

[0125] A storage module, used for storing the received new data record into the data record set until the number of data records in the data record set is greater than or equal to a preset number threshold;

[0126] A generation module is used to use the data records stored in the data record set as target data records, supplement the dimensional information of the target data records, and generate a lateral table of the target data records when the number of data records in the data record set is greater than or equal to a preset number threshold.

[0127] According to another reference embodiment of the present invention, the data record supplementing device 700 further includes:

[0128] A cleaning module is used to clean up the storage space of the target data record.

[0129] According to another reference embodiment of the present invention, the data record supplementation device 700 further includes:

[0130] A determination module, used to determine the data record with the latest receiving time among the target data records;

[0131] A merging module is used to merge the data record with the latest receiving time with each completed data record included in the lateral table.

[0132] According to another reference embodiment of the present invention, the lateral table is obtained by executing a preset custom table function according to Flink SQL, and the business logic of the custom table function includes: determining the number of data records, obtaining dimension information, and supplementing data records.

[0133] According to another reference embodiment of the present invention, the data record supplementing device 700 further includes:

[0134] A condition acquisition module, configured to determine a corresponding query condition in response to receiving a query request;

[0135] A query module is used to query the lateral table for the completed data records that meet the query conditions.

[0136] According to a reference embodiment of the present invention, obtaining dimension information of the target data record includes: determining a dimension primary key included in each target data record, and obtaining dimension information corresponding to each dimension primary key;

[0137] The target data record is supplemented by using the dimension information, including: using the dimension information corresponding to each dimension primary key to supplement the target data record including the dimension primary key.

[0138] It should be noted that the specific implementation content of the device for supplementing data records in the embodiment of the present invention has been described in detail in the method for supplementing data records described above, so the repeated content will not be described again here.

[0139] According to the technical solution of the embodiment of the present invention, after the data records are accumulated to a certain extent, the dimension information of the data records is supplemented, which can efficiently supplement the dimension information, reduce the read and write pressure of the storage system and the response interface, and reduce resource consumption; when the data records are less than the quantity threshold, the data records are continued to be accumulated, which can avoid frequent query and supplement of dimension information, further reduce the read and write pressure, and reduce resource consumption; timely clean up the storage space of the target data records, which can facilitate the storage of newly received data records, improve the utilization rate of storage space, and effectively control the size of the storage space occupied by data records; merge the data record with the latest reception time with the side table, so as to obtain a complete data record, which is convenient for improving the query efficiency of the data record; use Flink SQL to execute a pre-set custom table function to obtain the side table, which can reduce development costs and facilitate business maintenance and business expansion; query and complete data records according to query conditions, which can improve query efficiency and query accuracy; when the target data record includes multiple dimension primary keys, batch query dimension information to improve the efficiency of dimension information acquisition.

[0140] According to a third aspect of an embodiment of the present invention, there is provided an electronic device, comprising: one or more processors; a storage device for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the method provided by the first aspect of the embodiment of the present invention.

[0141] According to a fourth aspect of an embodiment of the present invention, a computer-readable medium is provided, on which a computer program is stored, and when the program is executed by a processor, the method provided by the first aspect of the embodiment of the present invention is implemented.

[0142] Figure 8 An exemplary system architecture 800 is shown to which the method for data record supplementation or the apparatus for data record supplementation according to an embodiment of the present invention can be applied.

[0143] like Figure 8 As shown, system architecture 800 may include terminal devices 801, 802, 803, a network 804, and a server 805. Network 804 is used to provide a medium for communication links between terminal devices 801, 802, 803 and server 805. Network 804 may include various connection types, such as wired, wireless communication links, or optical fiber cables, etc.

[0144] The user can use the terminal devices 801, 802, 803 to interact with the server 805 through the network 804 to receive or send messages, etc. Various communication client applications can be installed on the terminal devices 801, 802, 803, such as data record supplement applications, data record query applications, search applications, instant messaging tools, email clients, social platform software, etc. (only as examples).

[0145] The terminal devices 801 , 802 , and 803 may be various electronic devices having a display screen and supporting web browsing, including but not limited to smart phones, tablet computers, laptop computers, desktop computers, and the like.

[0146] The server 805 may be a server that provides various services, such as a background management server (only an example) that supports requests for data record supplementation sent by upstream terminal devices 801, 802, and 803. The background management server may monitor the number of data records in the data record set; when the number of data records is greater than or equal to a preset number threshold, the data record stored in the data record set is used as the target data record, and the dimension information of the target data record is obtained; the target data record is supplemented using the dimension information to generate a lateral table of the target data record; and the data record supplementation situation (only an example) is fed back to the terminal device.

[0147] It should be noted that the data record supplement method provided in the embodiment of the present invention is generally executed by the server 805, and accordingly, the data record supplement device is generally set in the server 805. The data record supplement method provided in the embodiment of the present invention can also be executed by the terminal devices 801, 802, and 803, and accordingly, the data record supplement device can be set in the terminal devices 801, 802, and 803.

[0148] It should be understood that Figure 8 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to implementation requirements.

[0149] Reference below Fig. 9 , which shows a schematic diagram of the structure of a computer system 900 of a terminal device suitable for implementing an embodiment of the present invention. Fig. 9 The terminal device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.

[0150] like Fig. 9 As shown, the computer system 900 includes a central processing unit (CPU) 901, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 902 or a program loaded from a storage part 908 into a random access memory (RAM) 903. In the RAM 903, various programs and data required for the operation of the system 900 are also stored. The CPU 901, the ROM 902, and the RAM 903 are connected to each other via a bus 904. An input / output (I / O) interface 905 is also connected to the bus 904.

[0151] The following components are connected to the I / O interface 905: an input section 906 including a keyboard, a mouse, etc.; an output section 907 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 908 including a hard disk, etc.; and a communication section 909 including a network interface card such as a LAN card, a modem, etc. The communication section 909 performs communication processing via a network such as the Internet. A drive 910 is also connected to the I / O interface 905 as needed. A removable medium 911, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 910 as needed, so that a computer program read therefrom is installed into the storage section 908 as needed.

[0152] In particular, according to the embodiments disclosed in the present invention, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present invention include a computer program, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 909, and / or installed from the removable medium 911. When the computer program is executed by the central processing unit (CPU) 901, the above-mentioned functions defined in the system of the embodiment of the present invention are executed.

[0153] It should be noted that the computer-readable medium shown in the embodiment of the present invention may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In an embodiment of the present invention, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device. In an embodiment of the present invention, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than computer-readable storage media, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer-readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, optical cable, RF, etc., or any suitable combination of the above.

[0154] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, functions and operations of the system, method and computer program according to various embodiments of the present invention. In this regard, each box in the flow chart or block diagram can represent a module, a program segment, or a part of a code, and the above-mentioned module, program segment, or a part of a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flow chart, and the combination of the boxes in the block diagram or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0155] The modules involved in the embodiments of the present invention may be implemented by software or hardware. The modules described may also be set in a processor, for example, it may be described as: a processor includes a monitoring module, an acquisition module, and a supplementary module, wherein the names of these modules do not constitute a limitation on the modules themselves under certain circumstances, for example, the monitoring module may also be described as a "module for monitoring the number of data records in a data record set".

[0156] As another aspect, an embodiment of the present invention further provides a computer-readable medium, which may be included in the device described in the above embodiment; or it may exist independently without being assembled into the device. The above computer-readable medium carries one or more programs, and when the above one or more programs are executed by a device, the device implements the following method: monitoring the number of data records in a data record set; when the number of data records is greater than or equal to a preset number threshold, taking the data records stored in the data record set as target data records, and obtaining the dimension information of the target data records; using the dimension information to supplement the target data records, and generating a lateral table of the target data records.

[0157] According to the technical solution of the embodiment of the present invention, after the data records are accumulated to a certain extent, the dimension information of the data records is supplemented, which can efficiently supplement the dimension information, reduce the read and write pressure of the storage system and the response interface, and reduce resource consumption; when the data records are less than the quantity threshold, the data records are continued to be accumulated, which can avoid frequent query and supplement of dimension information, further reduce the read and write pressure, and reduce resource consumption; timely clean up the storage space of the target data records, which can facilitate the storage of newly received data records, improve the utilization rate of storage space, and effectively control the size of the storage space occupied by data records; merge the data record with the latest reception time with the side table, so as to obtain a complete data record, which is convenient for improving the query efficiency of the data record; use Flink SQL to execute a pre-set custom table function to obtain the side table, which can reduce development costs and facilitate business maintenance and business expansion; query and complete data records according to query conditions, which can improve query efficiency and query accuracy; when the target data record includes multiple dimension primary keys, batch query dimension information to improve the efficiency of dimension information acquisition.

[0158] The above specific implementations do not constitute a limitation on the protection scope of the embodiments of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions may occur depending on design requirements and other factors. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the embodiments of the present invention should be included in the protection scope of the embodiments of the present invention.< / record> < / row>

Claims

1. A method for supplementing data records, characterized in that: include: Monitor the number of data records in a data record set; When the number of data records is greater than or equal to a preset number threshold, the data records stored in the data record set are used as target data records, and dimension information of the target data records is obtained; The target data record is supplemented by using the dimension information to generate a lateral table of the target data record.

2. The method according to claim 1, characterized in that: When the number of data records is less than a preset number threshold, the method further includes: Storing the received new data records in the data record set until the number of data records in the data record set is greater than or equal to a preset number threshold; When the number of data records in the data record set is greater than or equal to a preset number threshold, the data records stored in the data record set are used as target data records, the dimension information of the target data records is supplemented, and a lateral table of the target data records is generated.

3. The method according to claim 2, characterized in that After generating the lateral table of the target data record, the method further comprises: Clear the storage space of the target data record.

4. The method according to claim 1, characterized in that: The method further comprises: Determine the data record with the latest receiving time among the target data records; The data record with the latest reception time is merged with each of the complementary data records included in the lateral table.

5. The method according to claim 1, characterized in that The lateral table is obtained by executing a preset custom table function according to Flink SQL. The business logic of the custom table function includes: determining the number of data records, obtaining dimension information, and supplementing data records.

6. The method according to claim 1, characterized in that The method further comprises: In response to receiving the query request, determining a corresponding query condition; The lateral table is searched for the completed data record that meets the query condition.

7. The method according to claim 1, characterized in that Acquiring dimension information of the target data record, including: determining a dimension primary key included in each target data record, and acquiring dimension information corresponding to each dimension primary key; The target data record is supplemented by using the dimension information, including: using the dimension information corresponding to each dimension primary key to supplement the target data record including the dimension primary key.

8. A device for supplementing data records, characterized in that: include: A monitoring module, used for monitoring the number of data records in a data record set; an acquisition module, configured to, when the number of data records is greater than or equal to a preset number threshold, use the data records stored in the data record set as target data records and acquire dimension information of the target data records; A supplementation module is used to supplement the target data record using the dimension information to generate a lateral table of the target data record.

9. An electronic device, characterized in that: include: One or more processors; A storage device for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 7.

10. A computer readable medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images