A wide table data verification method, device and equipment and storage medium

Abstract

This specification provides a method, apparatus, device, and storage medium for wide table data verification. The method includes: obtaining source data within the same time window from multiple source data tables to obtain multiple source datasets; obtaining wide table data within the same time window from a wide table to obtain a wide table dataset; wherein the time window is a rolling time window; processing the source data in the multiple source datasets according to preset processing rules to obtain a verification dataset; and determining that the wide table data in the wide table dataset is successfully verified if it matches the verification data in the verification dataset. This specification enables rapid data verification between multiple source data tables and the wide table during the synchronization of data from multiple source data tables to a wide table, improving the accuracy of wide table data synchronization.

Description

Technical Field

[0001] The embodiments described in this specification relate to the field of computer technology, specifically to a data verification method, apparatus, device, and storage medium. Background Technology

[0002] In the medical field, doctors often need to record various physical characteristics of patients. These data often include multiple attributes and have the characteristic of "one word having multiple meanings". In order to adapt to the complex and diverse attributes of physical characteristics data, the attribute data of medical data such as physical characteristics data can be stored in an Entity-Attribute-Value (EAV) data table. Through the EAV data table, attributes of physical characteristics data can be dynamically added without changing the structure of the data table.

[0003] Currently, to improve data query performance and analysis capabilities, it is often necessary to synchronize data from multiple tables into a single wide table. When synchronizing data from multiple tables into a wide table, data validation in the wide table is typically to ensure error-free synchronization. If an error occurs during synchronization, a new synchronization command is generated for the erroneous data. However, in some cases, data loss or content tampering may occur during data transmission. These issues may not cause synchronization failure, but the actually stored data may indeed contain errors.

[0004] Therefore, the existing wide table data synchronization process suffers from the technical problem of low data synchronization accuracy. Summary of the Invention

[0005] In view of this, this specification provides a method, apparatus, device, and storage medium for wide table data verification, which can perform fast data verification between multiple source data tables and the wide table during wide table data synchronization, thereby improving the accuracy of wide table data synchronization.

[0006] This specification provides a method for validating wide table data through multiple embodiments. The method includes: obtaining source data within the same time window from multiple source data tables to obtain multiple source datasets; obtaining wide table data within the same time window from a wide table to obtain a wide table dataset; the time window is a rolling time window; processing the source data in the multiple source datasets according to preset processing rules to obtain a validation dataset; and determining that the wide table data in the wide table dataset is successfully validated if it matches the validation data in the validation dataset.

[0007] One embodiment of this specification provides a wide table data verification device. The device includes: an acquisition module, configured to acquire source data within the time window from multiple source data tables according to the same time window, to obtain multiple source datasets; and to acquire wide table data within the time window from a wide table, to obtain a wide table dataset; wherein the time window is a rolling time window; a processing module, configured to process the source data in the multiple source datasets according to preset processing rules to obtain a verification dataset; and a determination module, configured to determine that the wide table data in the wide table dataset has been successfully verified if the wide table data in the wide table dataset matches the verification data in the verification dataset.

[0008] This specification provides a computer device including a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the method described in any of the above embodiments.

[0009] This specification provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the methods described in any of the above embodiments.

[0010] The various implementation methods provided in this specification can obtain source data within a time window from multiple source data tables to obtain multiple source datasets, and obtain wide table data within a time window from a wide table to obtain a wide table dataset. According to preset processing rules, the source data in the multiple source datasets are processed to obtain a verification dataset. If the wide table data in the wide table dataset is consistent with the verification data in the verification dataset, the wide table data in the wide table dataset is considered to have been successfully verified. This enables fast data verification between multiple source data tables and the wide table during the process of synchronizing data from multiple source data tables to the wide table, thereby improving the accuracy of wide table data synchronization. Attached Figure Description

[0011] Figure 1 This is a schematic diagram of a wide table data verification system provided for one embodiment of this specification.

[0012] Figure 2 This is a schematic diagram of the flow of a wide table data verification method provided in one embodiment of this specification.

[0013] Figure 3 This is a schematic diagram of a wide table data verification device provided for one embodiment of this specification.

[0014] Figure 4 A schematic diagram of a computer device provided for one embodiment of this specification. Detailed Implementation

[0015] To enable those skilled in the art to better understand the solutions described in this specification, the technical solutions in the embodiments of this specification will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this specification, and not all of them. Based on the embodiments in this specification, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this specification.

[0016] Data often has multiple attributes. For example, vital signs data can include attributes such as weight, blood pressure, blood sugar, and blood lipids. In order to dynamically add or edit data attributes, the various attributes of the data can be stored in an Entity-Attribute-Value (EAV) type data table.

[0017] In related technologies, data from multiple tables can be synchronized to a wide table to improve data query performance and analysis capabilities. When synchronizing data from multiple tables to a wide table, data validation in the wide table is typically to ensure error-free synchronization. If an error occurs during synchronization, a new synchronization command is generated for the erroneous data. However, in some cases, data loss or content tampering may occur during data transmission. These issues may not cause synchronization failure, but the actually stored data may indeed contain errors.

[0018] Therefore, it is necessary to provide a wide table data verification method. This method can obtain multiple source datasets by retrieving source data within the same time window from multiple source data tables, and obtain wide table data within the same time window from the wide table, resulting in a wide table dataset. Based on preset processing rules, the source data in the multiple source datasets are processed to obtain a verification dataset. Then, it is determined whether the wide table data in the wide table dataset is consistent with the verification data in the verification dataset. If the wide table data in the wide table dataset is consistent with the verification data in the verification dataset, the wide table data in the wide table dataset is considered to have been successfully verified, thereby improving the accuracy of wide table data synchronization.

[0019] Please see Figure 1This specification provides a wide table data verification system. The wide table data verification system may include a client and a server. The client may be an electronic device with network access capabilities. Specifically, for example, the client may be a desktop computer, tablet computer, laptop computer, smartphone, digital assistant, smart wearable device, shopping guide terminal, television, smart speaker, microphone, etc. Smart wearable devices include, but are not limited to, smart bracelets, smartwatches, smart glasses, smart helmets, smart necklaces, etc. Alternatively, the client may be software that can run on the electronic device. The server may be an electronic device with certain computing power. It may have a network communication module, processor, and memory, etc. Of course, the server may also refer to software running on the electronic device. The server may also be a distributed server, which may be a system with multiple processors, memory, network communication modules, etc., operating collaboratively. Alternatively, the server may be a server cluster formed by several servers. Alternatively, with the development of science and technology, the server may also be a new technical means capable of realizing the corresponding functions of the embodiments described in this specification. For example, it may be a new form of "server" based on quantum computing.

[0020] In this embodiment, the client can run either a source database or a target database. The source database and the target database can run on the same client or on different clients. The source database may include multiple source data tables, and the target database may include a wide table, wherein the multiple source data tables are related through a join field.

[0021] In this embodiment, source data from multiple source data tables can be sent to the target database to generate wide table data, which is then stored in the wide table of the target database, thereby achieving data synchronization between multiple source data tables and the wide table. The server can obtain source data from multiple source data tables, as well as wide table data from the wide table, and verify the wide table data using the source data obtained from the multiple source data tables.

[0022] Please see Figure 2 This specification provides a method for validating wide table data in one embodiment. The wide table data validation method in this embodiment can be applied to a server, and specifically may include the following steps S110 to S150.

[0023] Step S110: Based on the same time window, obtain the source data within the time window from multiple source data tables to obtain multiple source datasets. Obtain the wide table data within the time window from the wide table to obtain the wide table dataset. The time window is a rolling time window.

[0024] In some cases, the synchronization process of wide table data refers to the process of sending source data from multiple source data tables to the wide table to form wide table data. During the synchronization process of wide table data, source datasets can be obtained from multiple source data tables and wide table datasets can be obtained from the wide table within the same time window. Then, the wide table data in the wide table dataset can be verified by determining whether the source dataset and the wide table dataset are consistent.

[0025] In this embodiment, the server can obtain source data within the time window from multiple source data tables, respectively, to obtain multiple source datasets; it can also obtain wide table data within the time window from a wide table, to obtain a wide table dataset. Specifically, the time window can be a scrolling time window with a fixed length, sliding in steps equal to the length of the time window. Both the source data in the source data table and the wide table data in the wide table have corresponding time information, which can refer to the generation time or modification time of the source data or wide table data. By sliding the time window, the source data in the source data table can be segmented according to the time information corresponding to the source data, thereby obtaining the source data within the time window in the source data table, and the set of source data within the time window is used as the source dataset of the source data table; similarly, the wide table data in the wide table can be segmented according to the time information corresponding to the wide table data, obtaining the wide table data within the time window in the wide table, and the set of wide table data within the time window is used as the wide table dataset of the wide table.

[0026] By sliding the time window with a step size equal to the length of the time window, the wide table data in the wide table and the source data in multiple source data tables can be accurately and comprehensively obtained. This allows for comprehensive verification of the wide table data, improving the accuracy of wide table data verification.

[0027] Step S120: According to the preset processing rules, process the source data in multiple source datasets to obtain the verification dataset.

[0028] In this embodiment, the server can obtain source datasets from multiple source data tables. After obtaining multiple source datasets, the server processes the source data in the multiple source datasets according to preset processing rules to obtain a verification dataset. Specifically, the source data in the source data tables have corresponding association information, and the source data in multiple source data tables are related through the association information. The server can process the related source data in the multiple source datasets according to preset processing rules to obtain multiple verification datasets.

[0029] Exemplarily, multiple source data tables may include an entity table, an attribute table, and a value table. As an example, please refer to Table 1, Table 2, and Table 3. Table 1 is an entity table, Table 2 is an attribute table, and Table 3 is a value table. Goods can be stored as entities in the entity table. Referring to Table 2, the attributes of goods can be stored in the attribute table, and the values of the attributes of goods can be stored in the value table.

[0030] Table 1

[0031] ID Name 1 jacket 2 Pear Mobile Phone 3 Martian Computer

[0032] Table 2

[0033] AttributeID Name 1 color 2 size 3 shape 4 motherboard 5 CPU 6 graphics card 7 Memory

[0034] Table 3

[0035]

[0036]

[0037] Among them, the association information of the source data in Table 1 may refer to ID, the association information of the source data in Table 2 may refer to AttributeID, and the association information of the source data in Table 3 may refer to ID and AttributeID. The source data in Table 1 and the source data in Table 3 are associated through the association information ID, and the source data in Table 2 and the source data in Table 3 are associated through the association information AttributeID. For example, the source data with ID "2" and Name "Pear Mobile Phone" in Table 1 is respectively associated with the source data with ID "2", AttributeID "7", and Value "256G" in Table 3, and the source data with ID "2", AttributeID "5", and Value "Snapdragon 999" in Table 3.

[0038] It should be noted that multiple source data tables can be further divided into multiple source data tables according to the data types in the value table.

[0039] Specifically, since the wide table data in the wide table is obtained by processing the source data of multiple source data tables during the synchronization process, when verifying the wide table data in the wide table, according to the preset processing rules, the associated source data in multiple source data sets can be processed respectively to obtain multiple verification data, and then a verification data set can be obtained. Among them, the preset processing rule may refer to the rule for processing the source data in multiple source data tables to obtain the wide table data during the wide table data synchronization process. After the server sends the associated multiple source data to the wide table to generate the wide table data during the wide table data synchronization process, through this preset processing rule, the associated source data in multiple source data sets can be processed asynchronously respectively to obtain multiple verification data.

[0040] This implementation method processes the source data of multiple associated source data tables according to preset processing rules to obtain a verification dataset. This avoids verifying the wide table data in the wide table separately based on the source data in multiple source data tables when verifying the wide table data in the wide table, thus improving the verification efficiency of the wide table data verification.

[0041] Step S130: If the wide table data in the wide table dataset is consistent with the verification data in the verification dataset, the wide table data in the wide table dataset is deemed to have been successfully verified.

[0042] In this embodiment, the server can determine whether the wide table data in the wide table dataset is consistent with the verification data in the verification dataset. If the wide table data in the wide table dataset is consistent with the verification data in the verification dataset, the server determines that the wide table data in the wide table dataset has been successfully verified. Specifically, the wide table data in the wide table dataset has unique identifier information, and the verification data in the verification dataset has unique identifier information. The server can determine whether the wide table data in the wide table dataset is consistent with the verification data in the verification dataset based on the identifier information of the wide table data in the wide table dataset and the verification data in the verification dataset. If the wide table data in the wide table dataset is consistent with the verification data in the verification dataset, the server determines that the wide table data in the wide table dataset has been successfully verified.

[0043] Specifically, wide table data can have unique identifiers, and validation data can have unique identifiers. The unique identifier for the target data can refer to the primary key value of the wide table data. For example, when the wide table data has multiple primary key values, the identifier can be a combination of those values. For instance, if the wide table data has three primary key values, a, b, and c, then the identifier can be "abc" or "abc". Similarly, the unique identifier for the validation data can refer to the primary key value of the validation data, or it can be a combination of multiple primary key values ​​from the source data.

[0044] For example, the server can determine whether the wide table data in the wide table dataset is consistent with the validation data in the validation dataset based on the identifier information of the wide table data in the wide table dataset and the identifier information of the validation data in the validation dataset. Whether the wide table data in the wide table dataset is consistent with the validation data in the validation dataset can refer to whether the identifier information of the validation data in the validation dataset is present in the wide table dataset; in other words, whether the validation dataset is a subset of the wide table dataset.

[0045] If the wide table data in the wide table dataset is consistent with the validation data in the validation dataset, it means that the identification information of the validation data in the validation dataset is present in the wide table dataset, i.e., the validation dataset is a subset of the wide table dataset. If the wide table data in the wide table dataset is inconsistent with the validation data in the validation dataset, it means that the identification information of the validation data in the validation dataset is not entirely present in the wide table dataset. This may be because some source data in the source table was lost or tampered with during synchronization, resulting in incorrect or missing wide table data generated in the wide table.

[0046] If the wide table data in the wide table dataset matches the verification data in the verification dataset, the server considers the wide table data in the wide table dataset to be successfully verified. This indicates that the source data in multiple source datasets has been accurately synchronized to the wide table dataset, thus achieving fast and accurate verification of the wide table data in the wide table.

[0047] In some implementations, the wide table data verification method may also include steps S140 to S160.

[0048] Step S140: If the wide table data in the wide table dataset is inconsistent with the validation data in the validation dataset, determine the missing data from the validation data in the validation dataset, and use the start time of the current time window as the start time of the next time window.

[0049] In some cases, when the wide table dataset and the validation dataset are inconsistent, the server can compensate for the wide table dataset. This involves identifying the missing data in the wide table dataset relative to the source dataset and compensating for the missing data based on the corresponding source data, thereby improving the accuracy of the wide table data synchronization process.

[0050] Specifically, when the wide table data in the wide table dataset is inconsistent with the validation data in the validation dataset, the server can determine the missing data from the validation data in the validation dataset and use the start time of the current time window as the start time of the next time window. Here, the missing data refers to the validation data in the relative complement set of the wide table dataset within the validation dataset. As an example, using the start time of the current time window as the start time of the next time window indicates that the current and next time windows completely overlap.

[0051] Step S150: Continuously calculate the number of missing data points. The number of missing data points is the number of times the wide table data in the wide table dataset is identified as missing data.

[0052] In this embodiment, the server can retrieve a wide table dataset once from the wide table and source datasets from multiple source data tables each time the time window slides once, and identify missing data once. For example, missing data may be due to untimely synchronization of wide table data caused by network latency, synchronization program back pressure, etc. After identifying missing data, the current time window overlaps with the next time window. To achieve the purpose of delayed verification of the wide table data corresponding to the missing data, after identifying missing data, missing data can be continuously identified from the verification data of the verification dataset, and the number of missing data occurrences can be continuously calculated. In this way, delayed verification of missing data caused by network latency and synchronization program back pressure can be achieved.

[0053] Step S160: If the number of missing data reaches a set threshold, add the missing data as wide table data to the wide table dataset.

[0054] In this embodiment, when adding missing data as wide table data to the wide table dataset, the missing data can be added to the compensation queue, and based on the compensation queue, the missing data can be added as wide table data to the wide table dataset.

[0055] In this implementation, to improve the throughput of wide table data verification, the wide table data corresponding to the missing data is asynchronously compensated through a compensation queue. Therefore, to prevent incorrect compensation caused by multiple source data tables having already sent the source data corresponding to the missing data to the wide table to form a wide table during the wide table data verification process, when adding the missing data as wide table data to the wide table dataset, it is necessary to determine whether there is wide table data in the wide table dataset that is the same as the missing data. At the same time, it is also necessary to determine whether the time information corresponding to the wide table data is earlier than the time information corresponding to the missing data if there is wide table data in the wide table dataset that is the same as the missing data.

[0056] Specifically, the time information corresponding to related source data in multiple source datasets is the same, and the time information corresponding to the verification data is the time information corresponding to the source data from which the verification data is obtained. The server can determine whether there is an identifier in the wide table dataset that matches the identifier information of the missing data when the number of missing data occurrences reaches a set threshold. If no such identifier exists in the wide table dataset, the missing data is added to the wide table dataset as part of the wide table data. This further improves the accuracy of wide table data verification.

[0057] Furthermore, if the server can determine whether the time information corresponding to the missing data is greater than the time information corresponding to the delayed data in the wide table dataset when the same identification information exists in the wide table dataset, and if the time information corresponding to the missing data is greater than the time information corresponding to the delayed data, the missing data is added to the wide table dataset as wide table data, so that the missing data replaces the delayed data, thereby improving the accuracy of wide table data validation. Here, delayed data refers to wide table data in the wide table dataset whose identification information is the same as that of the missing data.

[0058] Please see Figure 3 This specification provides an embodiment of a wide table data verification device. The data verification device in this embodiment may include an acquisition module, a processing module, and a determination module. The acquisition module is used to acquire source data within the same time window from multiple source data tables, obtaining multiple source datasets, and to acquire wide table data within the time window from a wide table, obtaining a wide table dataset, wherein the time window is a rolling time window. The processing module is used to process the source data from the multiple source datasets according to preset processing rules to obtain a verification dataset. The determination module is used to determine that the wide table data in the wide table dataset has been successfully verified if it matches the verification data in the verification dataset.

[0059] The specific functions and effects of the wide table data verification device can be explained by referring to other embodiments in this specification, and will not be repeated here. Each module in the wide table data verification device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device in software form, so that the processor can call and execute the operations corresponding to each module.

[0060] Please see Figure 4 In some embodiments, a computer device may be provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the wide table data verification method of the embodiments described.

[0061] This specification also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a computer, causes the computer to perform the wide table data verification method described in any of the above embodiments.

[0062] This specification also provides a computer program product containing instructions that, when executed by a computer, cause the computer to perform the wide table data verification method in any of the above embodiments.

[0063] It is understood that the specific examples in this document are only intended to help those skilled in the art better understand the embodiments described herein, and are not intended to limit the scope of the invention.

[0064] It is understood that in the various embodiments described in this specification, the sequence number of each process does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments described in this specification.

[0065] It is understood that the various implementation methods described in this specification can be implemented individually or in combination, and the implementation methods in this specification are not limited in this respect.

[0066] Unless otherwise stated, all technical and scientific terms used in the embodiments of this specification have the same meaning as commonly understood by one of ordinary skill in the art. The terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to limit the scope of this specification. The term "and / or" as used in this specification includes any and all combinations of one or more of the associated listed items. The singular forms "a," "the," and "the" as used in the embodiments of this specification and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise.

[0067] It is understood that the processor in the embodiments of this specification can be an integrated circuit chip with signal processing capabilities. In implementation, each step of the above method embodiments can be completed by integrated logic circuits in the processor's hardware or by instructions in software form. The processor can be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this specification. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this specification can be directly implemented by a hardware decoding processor, or by a combination of hardware and software modules in the decoding processor. The software modules can reside in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. This storage medium is located in memory; the processor reads information from the memory and, in conjunction with its hardware, completes the steps of the above methods.

[0068] It is understood that the memory in the embodiments of this specification may be volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. Non-volatile memory may be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. Volatile memory may be random access memory (RAM). It should be noted that the memory in the systems and methods described herein is intended to include, but is not limited to, these and any other suitable types of memory.

[0069] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this specification.

[0070] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the aforementioned method implementations, and will not be repeated here.

[0071] In the several embodiments provided in this specification, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0072] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0073] In addition, the functional units in the various embodiments of this specification can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0074] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solutions of this specification, in essence, or the parts that contribute to the prior art, or parts of the technical solutions, can be embodied in the form of software products. These computer software products are stored in a storage medium and include several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this specification. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0075] The above description is merely a specific embodiment of this specification, but the scope of protection of this invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in this specification should be included within the scope of protection of this specification. Therefore, the scope of protection of this invention should be determined by the scope of the claims.

Claims

1. A method for validating wide table data, characterized in that, The method includes: Based on the same time window, source data within the time window is obtained from multiple source data tables to obtain multiple source datasets. Wide table data within the time window is obtained from a wide table to obtain a wide table dataset. The time window is a rolling time window. The wide table data in the wide table dataset has unique identification information that identifies the wide table data. According to preset processing rules, the source data in the multiple source datasets are processed to obtain a verification dataset; the verification data in the verification dataset has unique identification information that identifies the verification data. Based on the identification information of the wide table data in the wide table dataset and the identification information of the verification data in the verification dataset, it is determined whether the wide table data in the wide table dataset is consistent with the verification data in the verification dataset; if the wide table data in the wide table dataset is consistent with the verification data in the verification dataset, the wide table data in the wide table dataset is considered to have been successfully verified. If the wide table data in the wide table dataset is inconsistent with the verification data in the verification dataset, the missing data is determined from the verification data in the verification dataset, and the start time of the current time window is used as the start time of the next time window; the missing data is the verification data in the relative complement set of the wide table dataset in the verification dataset. The number of missing data entries is continuously calculated; the number of missing data entries is the number of times that the wide table data of the wide table dataset is identified as missing data. If the number of missing data occurrences reaches a set threshold, the missing data will be added to the wide table dataset as wide table data.

2. The method according to claim 1, characterized in that, The source data in the source data table has corresponding association information, and the source data in the multiple source data tables are related through the association information; The step of processing the source data from the multiple source datasets according to preset processing rules to obtain the verification dataset includes: According to preset processing rules, the associated source data in the multiple source datasets are processed to obtain multiple verification data. The verification dataset includes the multiple verification datasets.

3. The method according to claim 1, characterized in that, The associated source data in the multiple source datasets have the same time information; the time information corresponding to the verification data is the time information corresponding to the source data from which the verification data is obtained. If the number of missing data occurrences reaches a set threshold, the missing data is added to the wide table dataset as wide table data, including: If the number of missing data reaches a set threshold, determine whether there is identification information in the wide table dataset that is the same as the identification information of the missing data; If no identifier information identical to the identifier information of the missing data exists in the wide table dataset, the missing data is added to the wide table dataset as wide table data.

4. The method according to claim 3, characterized in that, When the number of missing data occurrences reaches a set threshold, the missing data is added to the wide table dataset as wide table data, further including: If the wide table dataset contains identification information that is identical to the identification information of the missing data, determine whether the time information corresponding to the missing data is greater than the time information corresponding to the delayed data in the wide table dataset; the delayed data is wide table data in the wide table dataset whose identification information is identical to the identification information of the missing data. If the time information corresponding to the missing data is greater than the time information corresponding to the delayed data, the missing data is added to the wide table dataset as wide table data, so that the missing data replaces the delayed data.

5. The method according to claim 1, characterized in that, If the number of missing data occurrences reaches a set threshold, the missing data is added to the wide table dataset as wide table data, including: If the number of missing data occurrences reaches a set threshold, the missing data will be added to the compensation queue. Based on the compensation queue, the missing data is added to the wide table dataset as wide table data.

6. A wide table data verification device, characterized in that, The device includes: The acquisition module is used to acquire source data within the same time window from multiple source data tables to obtain multiple source datasets, and to acquire wide table data within the same time window from a wide table to obtain a wide table dataset; the time window is a rolling time window; the wide table data in the wide table dataset has unique identification information that identifies the wide table data. The processing module is used to process the source data in the multiple source datasets according to preset processing rules to obtain a verification dataset; the verification data in the verification dataset has unique identification information that identifies the verification data. The determination module is used to determine whether the wide table data in the wide table dataset is consistent with the verification data in the verification dataset based on the identification information of the wide table data in the wide table dataset and the identification information of the verification data in the verification dataset; if the wide table data in the wide table dataset is consistent with the verification data in the verification dataset, the wide table data in the wide table dataset is considered to have been successfully verified; if the wide table data in the wide table dataset is inconsistent with the verification data in the verification dataset, missing data is determined from the verification data in the verification dataset, and the start time of the current time window is used as the start time of the next time window; the missing data is the verification data in the relative complement set of the wide table dataset in the verification dataset; the number of times the missing data is missing is continuously calculated; the number of times the wide table data in the wide table dataset is determined to be the missing data; if the number of times the missing data is missing reaches a set threshold, the missing data is added to the wide table dataset as wide table data.

7. A computer device comprising a memory and a processor, the memory storing a computer program, the processor executing the computer program to implement the method of any one of claims 1 to 5.

8. A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method of any one of claims 1 to 5.

Images