Massive device data auditing method and system based on OLAP database, and medium

Abstract

This invention discloses a method, system, and medium for massive device data auditing based on an OLAP database. The method includes: constructing an OLAP database and a massive device data wide table, and synchronizing production business data to the OLAP database in real time; periodically triggering massive device auditing processing through a preset scheduling trigger, obtaining key business fields from the massive device data wide table and storing them in a Redis cache; comparing the production business data stored in the OLAP database with the key business fields stored in the Redis cache to determine the operations required on device information and recording them to obtain first operation information; and performing batch operation processing on the massive device data wide table based on the first operation information. This invention improves the efficiency of massive device data auditing while ensuring the consistency of massive device data and reduces the computing power requirements of the system, and can be widely applied in the field of Internet of Things (IoT) technology.

Description

Technical Field

[0001] This invention relates to the field of Internet of Things (IoT) technology, and in particular to a method, system, and medium for auditing massive amounts of device data based on an OLAP database. Background Technology

[0002] An IoT platform provides developers with a platform for rapid development, deployment, and application management of IoT applications. Developers do not need to consider issues such as underlying infrastructure expansion, data management and aggregation, communication protocols, and communication security, thus reducing development costs and significantly shortening development time. The IoT platform possesses general basic capabilities such as terminal access, terminal management, and data analysis and monitoring. After IoT terminals are connected to the IoT platform, data analysis monitors various device states and business behaviors, and collects massive amounts of data on device activity.

[0003] The attributes, status, and activity of massive numbers of devices are crucial operational metrics and data presentation capabilities for IoT platforms. Recording and analyzing this massive data from the platform is an essential function. Due to the sheer number of devices and their diverse forms, device behavior varies greatly, leading to diverse data analysis needs. Therefore, data collection and hierarchical management of data from all devices on the platform are indispensable. The construction of the underlying wide tables and data consistency within this hierarchical structure form the foundation for data analysis. Synchronous computation of these massive wide tables will inevitably consume significant computing and storage resources and impact the online business environment. Traditional databases are also ill-suited for handling such massive data computations.

[0004] Existing massive device auditing algorithms read the production environment device database page by page after being triggered by a scheduling service, and then synchronize it to the underlying wide table of the data analysis database. This centralized access to the production database significantly increases the pressure on the database, impacting online business operations. Due to the massive nature of device data, a large number of devices require extensive I / O and data computation operations, which are extremely time-consuming, affecting the timeliness of subsequent data analysis calculations, and the resources involved in the computation are also enormous. Because the data synchronization and auditing process is lengthy, business changes in the production environment during this period cannot be reflected in the current data synchronization and auditing process in a timely manner, resulting in poor data consistency and affecting the accuracy of subsequent data analysis. Summary of the Invention

[0005] The purpose of this invention is to at least partially solve one of the technical problems existing in the prior art.

[0006] Therefore, one objective of this invention is to provide a method for auditing massive device data based on an OLAP database. This method improves the efficiency of massive device data auditing while ensuring the consistency of massive device data and reducing the computing power requirements of the system.

[0007] Another objective of this invention is to provide a massive device data auditing system based on an OLAP database.

[0008] To achieve the above-mentioned technical objectives, the technical solutions adopted in the embodiments of the present invention include:

[0009] In a first aspect, embodiments of the present invention provide a method for auditing massive amounts of device data based on an OLAP database, comprising the following steps:

[0010] Build an OLAP database and a wide table containing massive amounts of device data, and synchronize production business data to the OLAP database in real time;

[0011] The audit process for a large number of devices is triggered periodically by a preset scheduling trigger, and the key business fields of the large number of device data wide table are obtained and stored in the Redis cache.

[0012] The production business data stored in the OLAP database is audited and compared with the key business fields stored in the Redis cache to determine the operations that need to be performed on the equipment information and record them to obtain the first operation information;

[0013] The massive device data wide table is processed in batches based on the first operation information.

[0014] Furthermore, in one embodiment of the present invention, the step of constructing an OLAP database and a large number of device data wide tables, and synchronizing production business data to the OLAP database in real time, specifically includes:

[0015] Construct an OLAP database, which includes a business data layer and a wide table data layer;

[0016] A massive device data wide table is constructed based on the original device data information, and the massive device data wide table is stored in the wide table data layer;

[0017] Read the Blog logs from the online business database and synchronize production business data to the business data layer in real time through the Blog logs.

[0018] Furthermore, in one embodiment of the present invention, the step of obtaining the key business fields of the massive device data wide table and storing them in the Redis cache specifically includes:

[0019] The key business fields in the massive device data wide table are loaded in pages and batches. The key business fields include the first device ID, the first tenant ID, the first device activation status, and the first device online status.

[0020] The first key-value pair is generated by using the first device ID as the key and the first tenant ID, the first device activation status, and the first device online status as the value, and then stored in the Redis cache.

[0021] Furthermore, in one embodiment of the present invention, the step of auditing and comparing the production business data stored in the OLAP database with the key business fields stored in the Redis cache specifically includes:

[0022] Create an add queue and an update queue in the Redis cache;

[0023] The production business data can be queried in pages to obtain key equipment information, which includes the second equipment ID, the second tenant ID, the second equipment activation status, and the second equipment online status.

[0024] Use the second device ID as the key to match in the Redis cache;

[0025] If no matching first key-value pair is found, the second device ID and the corresponding production business data are added to the new queue.

[0026] When a matching first key-value pair is found, the first tenant ID, the first device activation status, and the first device online status are obtained according to the first key-value pair and compared with the second tenant ID, the second device activation status, and the second device online status in turn.

[0027] If all comparisons match, delete the corresponding first key-value pair;

[0028] If an inconsistency is found during the comparison, the second device ID and the corresponding production business data are added to the update queue, and the corresponding first key-value pair is deleted.

[0029] Furthermore, in one embodiment of the present invention, the step of adding the second device ID and the corresponding production business data to the update queue specifically includes:

[0030] When the first tenant ID is inconsistent with the second tenant ID, the second device ID and the corresponding production business data are added to the update queue;

[0031] When the first tenant ID is the same as the second tenant ID, and the first device activation status is different from the second device activation status, the second device ID, the corresponding device basic information, and the device online information are added to the update queue.

[0032] When the first tenant ID is the same as the second tenant ID, the first device activation status is the same as the second device activation status, and the first online status is different from the second online status, the second device ID and the corresponding device online information are added to the update queue.

[0033] Furthermore, in one embodiment of the present invention, the step of determining the operation to be performed on the device information and recording it to obtain the first operation information specifically includes:

[0034] The first device information that needs to be added is determined according to the new queue, and the second device information that needs to be updated is determined according to the update queue.

[0035] The information of the third device that needs to be deleted is determined based on the remaining first key-value pairs in the Redis cache;

[0036] The first operation information is generated based on the first device information, the second device information, the third device information, and the corresponding operation type.

[0037] Furthermore, in one embodiment of the present invention, the step of performing batch operation processing on the massive device data wide table according to the first operation information specifically includes:

[0038] Based on the first device information, perform batch addition operations on the massive device data wide table;

[0039] Perform batch update operations on the massive device data wide table based on the second device information;

[0040] Perform batch deletion operations on the massive device data wide table based on the third device information.

[0041] Secondly, embodiments of the present invention provide a massive device data auditing system based on an OLAP database, comprising:

[0042] The data synchronization module is used to build an OLAP database and a wide table for massive device data, and to synchronize production business data to the OLAP database in real time.

[0043] The audit triggering module is used to periodically trigger the audit processing of massive devices through preset scheduling triggers, obtain the key business fields of the massive device data wide table and store them in the Redis cache;

[0044] The audit comparison module is used to audit and compare the production business data stored in the OLAP database with the key business fields stored in the Redis cache, determine the operations that need to be performed on the equipment information, and record them to obtain the first operation information.

[0045] The batch processing module is used to perform batch operation processing on the massive device data wide table according to the first operation information.

[0046] Thirdly, embodiments of the present invention provide a massive device data auditing device based on an OLAP database, comprising:

[0047] At least one processor;

[0048] At least one memory for storing at least one program;

[0049] When the at least one program is executed by the at least one processor, the at least one processor implements the above-described method for auditing massive device data based on an OLAP database.

[0050] Fourthly, embodiments of the present invention also provide a computer-readable storage medium storing a processor-executable program, which, when executed by a processor, is used to perform the aforementioned method for auditing massive device data based on an OLAP database.

[0051] The advantages and beneficial effects of the present invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention:

[0052] This invention synchronizes production business data to a distributed OLAP database in real time. A batch processing approach is used to initiate massive device data audits via pre-configured scheduling tasks, avoiding impact on online business operations. Key business fields cached in Redis are compared with the production business data synchronized to the OLAP database to identify and record necessary operations on device information. Finally, these operations are executed in batches, improving IO efficiency, reducing computational frequency, and lowering database pressure and system computing power requirements. By using an OLAP database to improve database statistical calculation performance and using Redis caching to reduce database IO operations, the efficiency of massive device audits is effectively improved, significantly shortening data processing time and ensuring data consistency to a certain extent. Attached Figure Description

[0053] To more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings used in the embodiments of the present invention are described below. It should be understood that the drawings described below are only for the convenience of clearly describing some embodiments of the technical solutions of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0054] Figure 1 A flowchart illustrating the steps of a method for auditing massive device data based on an OLAP database, as provided in this embodiment of the invention;

[0055] Figure 2 A detailed flowchart of a method for auditing massive device data based on an OLAP database, provided in an embodiment of the present invention;

[0056] Figure 3 A structural block diagram of a massive device data auditing system based on an OLAP database provided in an embodiment of the present invention;

[0057] Figure 4 This is a structural block diagram of a massive device data auditing device based on an OLAP database, provided as an embodiment of the present invention. Detailed Implementation

[0058] The embodiments of the present invention are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and should not be construed as limiting the present invention. The step numbers in the following embodiments are set only for ease of explanation, and there is no limitation on the order between the steps. The execution order of each step in the embodiments can be adaptively adjusted according to the understanding of those skilled in the art.

[0059] In the description of this invention, "multiple" means two or more. The use of "first" and "second" is for distinguishing technical features only and should not be construed as indicating or implying relative importance, or implicitly indicating the number of indicated technical features, or the order of the indicated technical features. Furthermore, unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art.

[0060] Reference Figure 1 This invention provides a method for auditing massive amounts of device data based on an OLAP database, specifically including the following steps:

[0061] S101. Build an OLAP database and a wide table for massive device data, and synchronize production business data to the OLAP database in real time.

[0062] Specifically, OLAP databases are primarily used for massive data analysis and processing, achieving data management and statistical calculations at different business levels through data layering. Step S101 specifically includes the following steps:

[0063] S1011. Construct an OLAP database, which includes a business data layer and a wide table data layer.

[0064] S1012. Construct a massive wide table of equipment data based on the original equipment data information, and store the massive wide table of equipment data in the wide table data layer;

[0065] S1013. Read the Blog logs from the online business database and synchronize the production business data to the business data layer in real time through the Blog logs.

[0066] Specifically, the OLAP database in this embodiment of the invention includes a business data layer and a wide table data layer. The business data layer is used to store real-time synchronized production business data, and the wide table data layer is used to store a massive wide table of device data. When constructing the massive wide table of device data, storing this wide table in column family mode can improve the performance of data reading and storage.

[0067] The data synchronization tool IDC reads the Blog logs of the online business database and achieves real-time data synchronization through the Blog logs; it also synchronizes the original device data information to the OLAP database, enabling subsequent data analysis and processing to be isolated from online business operations.

[0068] S102. The audit process of massive devices is triggered periodically by a preset scheduling trigger, and the key business fields of the massive device data wide table are obtained and stored in the Redis cache.

[0069] Specifically, configure scheduling triggers to periodically initiate audit processing for massive devices and load key business fields from the massive device data wide table into the Redis cache.

[0070] As an optional implementation, the step of obtaining key business fields from a large table of device data and storing them in a Redis cache specifically includes:

[0071] A1. Load key business fields from the massive device data wide table in paginated batches. Key business fields include the first device ID, the first tenant ID, the first device activation status, and the first device online status.

[0072] A2. Use the first device ID as the key, and the first tenant ID, the first device activation status, and the first device online status as the value to generate the first key-value pair, and store the first key-value pair in the Redis cache.

[0073] Specifically, the key business fields (including device ID, tenant ID, device activation status, and device online status) of the massive device data wide table are loaded in pages and batches, and concatenated into key device information. The device ID is used as the key, and the other key information is used as the value and stored in the cache.

[0074] S103. Audit and compare the production business data stored in the OLAP database with the key business fields stored in the Redis cache to determine the operations that need to be performed on the equipment information and record them to obtain the first operation information.

[0075] As an optional implementation, the step of auditing and comparing the production business data stored in the OLAP database with the key business fields stored in the Redis cache specifically includes:

[0076] B1. Create add queues and update queues in the Redis cache;

[0077] B2. Paged query of key equipment information in production business data. Key equipment information includes second equipment ID, second tenant ID, second equipment activation status, and second equipment online status.

[0078] B3. Use the second device ID as the key to match in the Redis cache;

[0079] B4. If no matching first key-value pair is found, add the second device ID and the corresponding production business data to the new queue.

[0080] B5. When the corresponding first key-value pair is matched, the first tenant ID, first device activation status and first device online status are obtained according to the first key-value pair and compared with the second tenant ID, second device activation status and second device online status in turn.

[0081] B6. If all comparisons match, delete the corresponding first key-value pair;

[0082] B7. When inconsistencies are found during the comparison, add the second device ID and the corresponding production business data to the update queue, and delete the corresponding first key-value pair.

[0083] As an optional implementation, the step of adding the second device ID and the corresponding production business data to the update queue specifically includes:

[0084] C1. When the first tenant ID is inconsistent with the second tenant ID, add the second device ID and the corresponding production business data to the update queue;

[0085] C2. When the first tenant ID is the same as the second tenant ID, and the first device activation status is different from the second device activation status, add the second device ID, the corresponding device basic information, and the device online information to the update queue.

[0086] C3. When the first tenant ID is the same as the second tenant ID, the first device activation status is the same as the second device activation status, and the first online status is different from the second online status, add the second device ID and the corresponding device online information to the update queue.

[0087] As a further optional implementation, the step of determining the operation that needs to be performed on the device information and recording it to obtain the first operation information specifically includes:

[0088] D1. Determine the first device information that needs to be added based on the new queue, and determine the second device information that needs to be updated based on the update queue.

[0089] D2. Determine the information of the third device that needs to be deleted based on the remaining first key-value pairs in the Redis cache;

[0090] D3. Generate first operation information based on the first device information, the second device information, the third device information, and the corresponding operation type.

[0091] Specifically, such as Figure 2 The diagram shows a detailed flowchart of a massive device data auditing method based on an OLAP database provided by an embodiment of the present invention. The specific process of auditing and comparing the business data obtained through data synchronization with the key business fields stored in the Redis cache is as follows:

[0092] 1. Create a new queue AQ and an update queue UQ in the cache;

[0093] 2. Paginated query of key equipment information in production business data: Equipment ID, Tenant ID, Equipment Activation Status, and Equipment Online Status;

[0094] 3. Use the device ID to retrieve the stored device information (i.e., the first key-value pair) from the Redis cache and compare it, as follows:

[0095] 1) If device information is retrieved from the cache, compare the tenant ID, device activation status, and device online status in sequence; if they are completely consistent, it is assumed that the device has not changed; delete the device information from the Redis cache.

[0096] 2) If device information is obtained from the cache, compare the tenant ID, device activation status, and device online status in sequence. If the tenant ID is inconsistent, obtain the full information of the device in the actual business for subsequent update operations. If the activation status is inconsistent, obtain the basic device information and device online information for subsequent update operations. If the device online status is inconsistent, only obtain the device online information for subsequent update operations. Record the device ID in the update queue UQ. Delete the device information in the cache.

[0097] 3) If device information cannot be obtained from the cache, obtain all business information of the device for subsequent new operations; record the device ID in the new queue AQ;

[0098] 4) Obtain the device ID and full device business information from AQ, concatenate the device business data, and form the first device information for subsequent batch insertion into the massive device data wide table.

[0099] 5) Obtain the device ID and the business information that needs to be updated from UQ, concatenate the device business data, and form the second device information for subsequent batch updates of the massive device data wide table;

[0100] 4. Clear AQ and UQ, and obtain the device information on the next page for auditing;

[0101] 5. Repeat steps 2-4 above until all production business data has been audited, and obtain the remaining device information in the Redis cache to form the third device information for subsequent batch deletion of massive device data wide tables.

[0102] It is understood that the first device information in this embodiment of the invention needs to include all business data corresponding to the device ID, the second device information only needs to record the device ID and the business data that needs to be updated, and the third device information only needs to record the device ID.

[0103] S104. Perform batch operation processing on the massive device data wide table based on the first operation information.

[0104] Step S104 specifically includes the following steps:

[0105] S1041. Perform batch addition operations on the massive device data wide table based on the first device information;

[0106] S1042. Perform batch update operation on the massive device data wide table based on the second device information;

[0107] S1043. Perform batch deletion operations on the massive device data wide table based on the third device information.

[0108] The method steps of the embodiments of the present invention have been described above. It can be understood that the embodiments of the present invention synchronize production business data to a distributed OLAP database in real time, and initiate massive device data audits in a batch processing manner through pre-configured scheduling tasks to avoid impacting online business: Key business fields stored in Redis cache are compared with the production business data synchronized to the OLAP database for auditing, identifying and recording operations that need to be performed on device information, and finally executing the operations in batches. This improves IO efficiency, reduces the number of calculations, and lowers the pressure on the database and the computing power requirements of the system. By using an OLAP database to improve database statistical calculation performance and using Redis caching to reduce database IO operations, the efficiency of massive device audits is effectively improved, data processing time is greatly shortened, and data consistency is guaranteed to a certain extent.

[0109] Compared to existing methods for statistical calculation of all devices, the embodiments of the present invention have the following advantages and effects:

[0110] 1) It utilizes a distributed OLAP database, which greatly increases the scale of data storage through distributed data storage. It also supports OLAP technology and allows underlying data to be stored in a key-value (K / V) manner, which significantly improves the performance of statistical calculations.

[0111] 2) The full-scale device audit algorithm has been optimized. By extracting key fields of data service changes of the devices, the algorithm can identify and compare changes in the devices without having to obtain all the information of the devices, which greatly reduces the I / O operations of the database.

[0112] 3) By utilizing caching technology, the storage and comparison of key device fields are handled by the cache, further reducing database I / O operations and greatly improving the efficiency of auditing massive numbers of devices.

[0113] 4) Update data by using batch database operations to reduce the number of times the system establishes connections with the database and further improve efficiency.

[0114] 5) After the wide table for equipment is established, subsequent data stratification and data analysis no longer require cross-database and cross-table join operations. Data can be processed directly using the data in the wide table, allowing the performance of the OLAP database to be maximized.

[0115] Reference Figure 3 This invention provides a massive device data auditing system based on an OLAP database, comprising:

[0116] The data synchronization module is used to build an OLAP database and a wide table for massive device data, and to synchronize production business data to the OLAP database in real time.

[0117] The audit triggering module is used to periodically trigger audit processing of massive devices through preset scheduling triggers, obtain key business fields of the massive device data wide table and store them in Redis cache;

[0118] The audit comparison module is used to audit and compare the production business data stored in the OLAP database with the key business fields stored in the Redis cache, determine the operations that need to be performed on the equipment information, and record them to obtain the first operation information.

[0119] The batch processing module is used to perform batch operations on a large number of device data wide tables based on the first operation information.

[0120] The content of the above method embodiments is applicable to this system embodiment. The specific functions implemented in this system embodiment are the same as those in the above method embodiments, and the beneficial effects achieved are also the same as those achieved in the above method embodiments.

[0121] Reference Figure 4 This invention provides a massive device data auditing device based on an OLAP database, comprising:

[0122] At least one processor;

[0123] At least one memory for storing at least one program;

[0124] When the above-mentioned at least one program is executed by the above-mentioned at least one processor, the above-mentioned at least one processor implements the above-mentioned method for auditing massive device data based on OLAP database.

[0125] The content of the above method embodiments is applicable to the device embodiments. The specific functions implemented by the device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0126] This invention also provides a computer-readable storage medium storing a processor-executable program, which, when executed by a processor, is used to perform the aforementioned method for auditing massive device data based on an OLAP database.

[0127] This invention provides a computer-readable storage medium that can execute a massive device data auditing method based on an OLAP database provided in the method embodiments of this invention. It can execute any combination of the implementation steps of the method embodiments and has the corresponding functions and beneficial effects of the method.

[0128] This invention also discloses a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device can read the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, causing the computer device to perform... Figure 1 The method shown.

[0129] In some alternative embodiments, the functions / operations mentioned in the block diagrams may not occur in the order shown in the operation diagrams. For example, depending on the functions / operations involved, two consecutively shown blocks may actually be executed substantially simultaneously, or the aforementioned blocks may sometimes be executed in reverse order. Furthermore, the embodiments presented and described in the flowcharts of this invention are provided by way of example to provide a more comprehensive understanding of the technology. The disclosed methods are not limited to the operations and logic flows presented herein. Alternative embodiments are contemplated in which the order of various operations is changed and sub-operations described as part of a larger operation are executed independently.

[0130] Furthermore, although the invention has been described in the context of functional modules, it should be understood that, unless otherwise stated, one or more of the aforementioned functions and / or features may be integrated into a single physical device and / or software module, or one or more functions and / or features may be implemented in a separate physical device or software module. It is also understood that a detailed discussion of the actual implementation of each module is unnecessary for understanding the invention. Rather, given the properties, functions, and internal relationships of the various functional modules in the apparatus disclosed herein, the actual implementation of the module will be understood within the scope of conventional skill of an engineer. Therefore, those skilled in the art can implement the invention as set forth in the claims using ordinary techniques without excessive experimentation. It is also understood that the specific concepts disclosed are merely illustrative and not intended to limit the scope of the invention, which is determined by the full scope of the appended claims and their equivalents.

[0131] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0132] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-included system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device.

[0133] More specific examples of computer-readable media (a non-exhaustive list) include: electrical connections (electronic devices) having one or more wires, portable computer disk drives (magnetic devices), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Furthermore, computer-readable media can even be paper or other suitable media on which the aforementioned program can be printed, because the aforementioned program can be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, interpreting, or, if necessary, processing in other suitable ways, and then stored in computer memory.

[0134] It should be understood that various parts of the present invention can be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented in software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0135] In the foregoing description of this specification, references to terms such as "one embodiment," "another embodiment," or "some embodiments" indicate that a specific feature, structure, material, or characteristic described in connection with an embodiment or example is included in at least one embodiment or example of the present invention. In this specification, illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0136] Although embodiments of the invention have been shown and described, those skilled in the art will understand that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the claims and their equivalents.

[0137] The above is a detailed description of the preferred embodiments of the present invention. However, the present invention is not limited to the above embodiments. Those skilled in the art can make various equivalent modifications or substitutions without departing from the spirit of the present invention. All such equivalent modifications or substitutions are included within the scope defined by the claims of this application.

Claims

1. A method for auditing massive amounts of device data based on an OLAP database, characterized in that, Includes the following steps: Build an OLAP database and a wide table containing massive amounts of device data, and synchronize production business data to the OLAP database in real time; The audit process for a large number of devices is triggered periodically by a preset scheduling trigger, and the key business fields of the large number of device data wide table are obtained and stored in the Redis cache. The production business data stored in the OLAP database is audited and compared with the key business fields stored in the Redis cache to determine the operations that need to be performed on the equipment information and record them to obtain the first operation information; Batch operation processing is performed on the massive device data wide table based on the first operation information; The step of auditing and comparing the production business data stored in the OLAP database with the key business fields stored in the Redis cache specifically includes: Create an add queue and an update queue in the Redis cache; The production business data can be queried in pages to obtain key equipment information, which includes the second equipment ID, the second tenant ID, the second equipment activation status, and the second equipment online status. Use the second device ID as the key to match in the Redis cache; If no matching first key-value pair is found, the second device ID and the corresponding production business data are added to the new queue. When the corresponding first key-value pair is matched, the corresponding first tenant ID, first device activation status and first device online status are obtained according to the first key-value pair and compared with the second tenant ID, second device activation status and second device online status in turn. If all comparisons match, delete the corresponding first key-value pair; If an inconsistency is found during the comparison, the second device ID and the corresponding production business data are added to the update queue, and the corresponding first key-value pair is deleted.

2. The method for auditing massive device data based on an OLAP database according to claim 1, characterized in that, The step of constructing an OLAP database and a large number of wide tables for device data, and synchronizing production business data to the OLAP database in real time, specifically includes: Construct an OLAP database, which includes a business data layer and a wide table data layer; A massive device data wide table is constructed based on the original device data information, and the massive device data wide table is stored in the wide table data layer; Read the Blog logs from the online business database and synchronize production business data to the business data layer in real time through the Blog logs.

3. The method for auditing massive device data based on an OLAP database according to claim 1, characterized in that, The step of obtaining the key business fields of the massive device data wide table and storing them in the Redis cache specifically includes: The key business fields in the massive device data wide table are loaded in pages and batches. The key business fields include the first device ID, the first tenant ID, the first device activation status, and the first device online status. The first key-value pair is generated by using the first device ID as the key and the first tenant ID, the first device activation status, and the first device online status as the value, and then stored in the Redis cache.

4. The method for auditing massive device data based on an OLAP database according to claim 1, characterized in that, The step of adding the second device ID and the corresponding production business data to the update queue specifically includes: When the first tenant ID is inconsistent with the second tenant ID, the second device ID and the corresponding production business data are added to the update queue; When the first tenant ID is the same as the second tenant ID, and the first device activation status is different from the second device activation status, the second device ID, the corresponding device basic information, and the device online information are added to the update queue. When the first tenant ID is the same as the second tenant ID, the first device activation status is the same as the second device activation status, and the first device online status is different from the second device online status, the second device ID and the corresponding device online information are added to the update queue.

5. The method for auditing massive device data based on an OLAP database according to claim 1, characterized in that, The step of determining the operation to be performed on the device information and recording it to obtain the first operation information specifically includes: The first device information that needs to be added is determined according to the new queue, and the second device information that needs to be updated is determined according to the update queue. The information of the third device that needs to be deleted is determined based on the remaining first key-value pairs in the Redis cache; The first operation information is generated based on the first device information, the second device information, the third device information, and the corresponding operation type.

6. The method for auditing massive device data based on an OLAP database according to claim 5, characterized in that, The step of performing batch operations on the massive device data wide table based on the first operation information specifically includes: Based on the first device information, perform batch addition operations on the massive device data wide table; Perform batch update operations on the massive device data wide table based on the second device information; Perform batch deletion operations on the massive device data wide table based on the third device information.

7. A massive equipment data auditing system based on an OLAP database, characterized in that, A method for auditing massive device data based on an OLAP database as described in any one of claims 1 to 6 includes: The data synchronization module is used to build an OLAP database and a large number of device data wide tables, and to synchronize production business data to the OLAP database in real time. The audit triggering module is used to periodically trigger the audit processing of massive devices through preset scheduling triggers, obtain the key business fields of the massive device data wide table and store them in the Redis cache; The audit comparison module is used to audit and compare the production business data stored in the OLAP database with the key business fields stored in the Redis cache, determine the operations that need to be performed on the equipment information, and record them to obtain the first operation information. The batch processing module is used to perform batch operation processing on the massive device data wide table according to the first operation information.

8. A massive device data auditing device based on an OLAP database, characterized in that, include: At least one processor; At least one memory for storing at least one program; When the at least one program is executed by the at least one processor, the at least one processor implements a method for auditing massive device data based on an OLAP database as described in any one of claims 1 to 6.

9. A computer-readable storage medium storing a processor-executable program, characterized in that, The processor-executable program, when executed by the processor, is used to perform a massive device data auditing method based on an OLAP database as described in any one of claims 1 to 6.

Images