Iot mass business bill processing system, method and device and storage medium

Abstract

The application discloses an Internet of Things mass service bill processing system, method and device and a storage medium, and comprises an application software service layer, a platform environment service layer and an infrastructure service layer.The application software service layer is used for collecting first service bills, and performing format analysis, deduplication, merging and distribution processing on the first service bills to obtain first bill files and first bill messages, and then uploading the first bill files and the first bill messages.The platform environment service layer is deployed with a distributed file system, a distributed coordination service, a data database and a streaming processing framework, is used for receiving the first bill files and the first bill messages, and performing full-amount synchronization on the data database according to the first bill files and incremental synchronization on the data database according to the first bill messages.The infrastructure service layer comprises a bottom cloud host, and is used for providing a computing resource pool, a storage resource pool and a network resource pool.The application improves the processing efficiency of service bills, optimizes the use perception of users, and can be widely applied to the technical field of data processing.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to a massive call detail record (CDR) processing system, method, apparatus, and storage medium for the Internet of Things (IoT). Background Technology

[0002] With the continuous development of the Industrial Internet of Things (IIoT), customer demand for IoT is constantly increasing, and the number of IoT SIM cards issued is also rising. As the number of users continues to increase, the number of call detail records (CDRs) is also growing exponentially. The original system architecture can no longer support the processing of massive amounts of user data CDRs, so a new system architecture needs to be developed to support the processing of current user CDRs, thereby enabling real-time billing and analysis of users, and providing users with real-time queries and customized usage.

[0003] Due to the unique nature of IoT services, users are often sensitive to data usage and require real-time / near real-time queries of total data usage and details. Furthermore, IoT services are subject to quota controls and frequent network outages due to data limits imposed by the Ministry of Industry and Information Technology (MIIT). This prevents the reduction of call detail records (CDRs) and the extension of CDR production time, unlike the Internet of Things (IoT), which can be achieved through mechanisms to reduce the number of CDRs and improve data usage tracking. Currently, the daily average CDR volume reaches 3.2 billion, and the HBase-based CDR processing model has reached its limits. However, the aggregation and calculation of usage continues to grow with the increasing number of connections, requiring continuous enhancement of system support capabilities. Therefore, a comprehensive optimization of the billing architecture is needed, necessitating a new technical architecture to process massive amounts of IoT user data CDRs in real-time / near real-time. Summary of the Invention

[0004] The purpose of this invention is to at least partially solve one of the technical problems existing in the prior art.

[0005] Therefore, one objective of this invention is to provide an efficient IoT massive call detail record processing system.

[0006] Another objective of this invention is to provide a method for processing massive call detail records (CDRs) of Internet of Things (IoT) services.

[0007] To achieve the above-mentioned technical objectives, the technical solutions adopted in the embodiments of the present invention include:

[0008] In a first aspect, embodiments of the present invention provide an IoT massive service call detail record processing system, comprising:

[0009] The application software service layer is used to collect the first service call detail record (CDR), and to format, analyze, deduplicatize, merge and distribute the first service CDR to obtain the first CDR file and the first CDR message, and then upload the first CDR file and the first CDR message.

[0010] The platform environment service layer is equipped with a distributed file system, a distributed coordination service, a data database, and a streaming processing framework. The platform environment service layer is used to receive the first call detail record (CDR) file and the first CDR message, and to perform full synchronization of the CDR data stored in the data database based on the first CDR file, and to perform incremental synchronization of the CDR data stored in the data database based on the first CDR message.

[0011] The infrastructure service layer includes an underlying cloud host, which provides a computing resource pool, a storage resource pool, and a network resource pool.

[0012] Furthermore, in one embodiment of the present invention, the application software service layer includes:

[0013] The acquisition engine is used to acquire the first service call detail records (CDRs) of IoT call detail record network element devices.

[0014] The full-process engine is used to format, analyze, deduplicate, merge, and distribute the first business call detail record (CDR) and the user data information stored in the data database to obtain the first CDR file and the first CDR message.

[0015] An upload engine is used to upload the first call detail record (CDR) file and the first CDR message.

[0016] Furthermore, in one embodiment of the present invention, the end-to-end engine includes:

[0017] A formatting unit is used to format the first service call detail record (CDR) to obtain a second CDR file and a second CDR message.

[0018] Analysis unit, the analysis unit is used to analyze and process the second call detail record information to obtain a third call detail record file and a third call detail record message;

[0019] A deduplication unit is used to perform deduplication processing on the third call detail record (CDR) message to obtain a fourth CDR file and a fourth CDR message.

[0020] A message distribution unit is configured to distribute the fourth call detail record (CDR) message to obtain the first CDR message, and send the first CDR message to the upload engine.

[0021] The file distribution unit is used to distribute the fourth call detail record file to obtain the first call detail record file, and send the first call detail record file to the upload engine.

[0022] Furthermore, in one embodiment of the present invention, the end-to-end engine further includes:

[0023] The merging unit is used to merge the second call detail record (CDR) file, the third CDR file, the fourth CDR file, and the first CDR message to obtain the user information, and then asynchronously store the user information to the data database through the distributed file system.

[0024] Furthermore, in one embodiment of the present invention, the platform environment service layer includes a distributed caching architecture, the distributed caching architecture comprising:

[0025] HBase unit, the HBase unit is used to store the data database;

[0026] The engine node includes a REST service unit, a Redis unit, and a masterless module. The masterless module is used to interact with the Redis unit and the HBase unit to implement the local caching mechanism of the engine node.

[0027] An interface machine is provided, which includes a data interface and a console. The console interacts with the HBase unit through the data interface and is used to control the REST service unit.

[0028] Furthermore, in one embodiment of the present invention, the platform environment service layer further includes:

[0029] A cache update service module is used to send a data change instruction to the data interface through the distributed coordination service, and to transmit the data change content to the data interface through the distributed file system, so that the data interface updates the data database.

[0030] Furthermore, in one embodiment of the present invention, the platform environment service layer further includes:

[0031] An incremental parsing module is used to parse the first business message to obtain a first parsing file, save the first parsing file and the first business message to the distributed file system, and generate a corresponding data index table.

[0032] Secondly, embodiments of the present invention provide a method for processing massive service call detail records (CDRs) in the Internet of Things (IoT), comprising the following steps:

[0033] The application software service layer collects the first service call detail record (CDR), and performs formatting, analysis, deduplication, merging, and distribution processing on the first service CDR to obtain the first CDR file and the first CDR message, and then uploads the first CDR file and the first CDR message.

[0034] The platform environment service layer receives the first call detail record (CDR) file and the first CDR message, performs full synchronization of the CDR data stored in the data database based on the first CDR file, and performs incremental synchronization of the CDR data stored in the data database based on the first CDR message.

[0035] The platform environment service layer deploys a distributed file system, a distributed coordination service, a data database, and a streaming processing framework. The platform environment service layer provides computing, storage, and network communication services through the underlying cloud hosts set up in the infrastructure service layer.

[0036] Thirdly, embodiments of the present invention provide an IoT massive service call detail record processing device, comprising:

[0037] At least one processor;

[0038] At least one memory for storing at least one program;

[0039] When the at least one program is executed by the at least one processor, the at least one processor implements the above-described method for processing massive IoT service call detail records.

[0040] Fourthly, embodiments of the present invention also provide a computer-readable storage medium storing a processor-executable program, which, when executed by a processor, is used to perform the above-described method for processing massive IoT service call detail records.

[0041] The advantages and beneficial effects of the present invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention:

[0042] This invention, through the deployment of application software service layer, platform environment service layer, and infrastructure service layer, combined with the advantages of Redis caching, can process and store business call records of massive users in real time and transmit the results downstream in real time. This avoids the performance bottleneck problem of HBase, enables timely network disconnection / usage query for larger call records, improves the processing efficiency of business call records, and optimizes the user experience. Attached Figure Description

[0043] To more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings used in the embodiments of the present invention are described below. It should be understood that the drawings described below are only for the convenience of clearly describing some embodiments of the technical solutions of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0044] Figure 1 A flowchart illustrating the steps of a method for processing massive IoT service call detail records (CDRs) according to an embodiment of the present invention;

[0045] Figure 2 A structural block diagram of an IoT massive service call detail record processing system provided in an embodiment of the present invention;

[0046] Figure 3 This is a schematic diagram of the specific structure of an IoT massive service call detail record processing system provided in an embodiment of the present invention;

[0047] Figure 4 A schematic diagram of the service call detail record (CDR) processing flow provided in an embodiment of the present invention;

[0048] Figure 5 A schematic diagram of a distributed caching architecture provided in an embodiment of the present invention;

[0049] Figure 6 This is a structural block diagram of an IoT massive service call detail record processing device provided in an embodiment of the present invention. Detailed Implementation

[0050] The embodiments of the present invention are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and should not be construed as limiting the present invention. The step numbers in the following embodiments are set only for ease of explanation, and there is no limitation on the order between the steps. The execution order of each step in the embodiments can be adaptively adjusted according to the understanding of those skilled in the art.

[0051] In the description of this invention, "multiple" means two or more. The use of "first" and "second" is for distinguishing technical features only and should not be construed as indicating or implying relative importance, or implicitly indicating the number of indicated technical features, or the order of the indicated technical features. Furthermore, unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art.

[0052] The continuous development of IoT services and the increasing number of SIM cards, terminals, and devices connecting to the network have led to a surge in call detail records (CDRs) processed by the IoT BOSS billing system. Simultaneously, the rollout of 5G and the resulting increase in data traffic have further fueled the growth in CDR volume. Furthermore, due to the unique nature of IoT services, many users require real-time / near-real-time network disconnection capabilities, making it impossible to fundamentally reduce the volume of user CDRs. The current system architecture cannot support the ever-increasing volume of CDRs. Therefore, a completely new architectural model is needed to adapt to the processing of massive user CDRs and subsequent billing.

[0053] Reference Figure 2 This invention provides an IoT massive call detail record (CDR) processing system, comprising:

[0054] The application software service layer is used to collect the first business call detail records (CDRs), and to format, analyze, deduplicatize, merge, and distribute the first business CDRs to obtain the first CDR file and the first CDR message, and then upload the first CDR file and the first CDR message.

[0055] The platform environment service layer is equipped with a distributed file system, a distributed coordination service, a data database, and a streaming processing framework. The platform environment service layer is used to receive the first call detail record (CDR) file and the first CDR message, and to perform full synchronization of the CDR data stored in the data database based on the first CDR file, and incremental synchronization of the CDR data stored in the data database based on the first CDR message.

[0056] The infrastructure service layer includes the underlying cloud hosts, which provide computing resource pools, storage resource pools, and network resource pools.

[0057] Specifically, such as Figure 3 The diagram shows a specific structural schematic of an IoT massive call detail record (CDR) processing system provided in an embodiment of the present invention. The Infrastructure as a Service (IaaS) layer deploys underlying cloud hosts, including computing, storage, and network communication functions, enabling batch file processing and massive data storage. The Platform as a Service (PaaS) layer, combined with the cloud host hardware, deploys components such as a distributed file system (HDFS), HBase, distributed coordination service (Zookeeper), distributed cache (Redis), and streaming processing framework (Wind). The Application Software as a Service (SaaS) layer deploys relevant CDR processing applications, including essential CDR processing procedures such as CDR collection, CDR formatting, enhanced CDR analysis, CDR deduplication, CDR merging, and CDR distribution.

[0058] As a further optional implementation, the application software service layer includes:

[0059] The data acquisition engine is used to collect the first service call detail records (CDRs) from IoT network element devices.

[0060] The full-process engine is used to format, analyze, deduplicate, merge, and distribute the user information stored in the first business call detail record (CDR) and data database to obtain the first CDR file and the first CDR message.

[0061] The upload engine is used to upload the first episode file and the first episode message.

[0062] As an optional implementation, the end-to-end engine includes:

[0063] The formatting unit is used to format the first service call detail record (CDR) to obtain the second CDR file and the second CDR message.

[0064] The analysis unit is used to analyze and process the second call detail record (CDR) information to obtain the third CDR file and the third CDR message.

[0065] The deduplication unit is used to deduplicatize the third call detail record (CDR) message to obtain the fourth CDR file and the fourth CDR message.

[0066] The message distribution unit is used to distribute the fourth call log message to obtain the first call log message, and then send the first call log message to the upload engine.

[0067] The file distribution unit is used to distribute the fourth episode file to obtain the first episode file, and then send the first episode file to the upload engine.

[0068] As an optional implementation, the end-to-end engine also includes:

[0069] The merging unit is used to merge the second call detail record (CDR) file, the third CDR file, the fourth CDR file, and the first CDR message to obtain user information, and then asynchronously store the user information to the data database through a distributed file system.

[0070] Specifically, such as Figure 4 The diagram shows a service call detail record (CDR) processing flow provided in an embodiment of the present invention. The application software service layer is the front end of the billing system, which connects to various IoT CDR network elements. Based on Redis and Hadoop data storage and processing streams, the entire CDR preprocessing process is carried out through the stored user information and the service CDRs collected from the source, so that the CDRs can be processed in real time and pushed to the downstream billing and pricing system in real time.

[0071] As an optional implementation, the platform environment service layer includes a distributed caching architecture, which includes:

[0072] HBase Unit: An HBase unit is used to store data in a database.

[0073] The engine node includes a REST service unit, a Redis unit, and a masterless module. The masterless module is used to interact with the Redis unit and the HBase unit to implement the local caching mechanism of the engine node.

[0074] The interface machine includes a data interface and a console. The console interacts with the HBase unit through the data interface and is used to control the REST service unit.

[0075] Specifically, various types of data are accessed during business processing. Currently, the database uses HBase as the storage layer, which cannot handle massive call detail records in a timely manner for some high-frequency data access services (e.g., ownerless). Therefore, it is necessary to reduce IO overhead by using caching.

[0076] like Figure 5 The diagram illustrates the distributed caching architecture provided in this embodiment of the invention. This embodiment implements a distributed caching structure using Redis, constructing a local caching mechanism for engine nodes. This ensures that over 80% of the call detail record (CDR) processing by the entire engine accesses the local cache, reducing network interactions and resolving the single-point-of-flight performance issue when the distributed component HDFS processes small files. During CDR processing, the local cache is queried first; if the data is not found there, the cloud cache is queried and downloaded. This ensures that the data is only accessed remotely for the first time, reduces reliance on HBase, and increases processing efficiency.

[0077] The data structure within the cache basically corresponds to the structure of the data table to be cached, and mainly includes the following fields:

[0078] key, accNbr, accServid, billingType, cityId, custId, cycleId, effDate, effdate, eventTypeId, expDate, expdate, nbr, paymentMode, productId, servId, status, ttl, updateDate, userChar, userType, zone, zoneCode.

[0079] Components required for caching: Redis cluster, unified DAO access layer, daily operation and maintenance toolchain (loading, cleaning, repairing, etc.), cache update service for communicating with data interfaces, and a WebUI for easy maintenance and querying.

[0080] As an optional implementation, the platform environment service layer further includes:

[0081] The cache update service module is used to send data change instructions to the data interface through a distributed coordination service, and to pass the data change content to the data interface through a distributed file system, so that the data interface can update the data database.

[0082] Specifically, the cache update service is a persistent service that communicates with the data interface through Zookeeper and HDFS. Zookeeper is used to notify data change information, and HDFS is used to store the data changes.

[0083] As an optional implementation, the platform environment service layer further includes:

[0084] The incremental parsing module is used to parse the first business message to obtain the first parsing file, save the first parsing file and the first business message to the distributed file system, and generate the corresponding data index table.

[0085] Specifically, during data synchronization, full data is retrieved from the first business file, and incremental data is retrieved from the first business message. This data is then imported into the data database. The source table and result table maintain a one-to-one correspondence; no records are added, modified, or deleted. The source table defines which data will be parsed; data not belonging to the user database cannot be parsed and imported. An HBase key is also set to facilitate later querying.

[0086] The incremental parsing module parses messages during the process of converting them from the source table to the result table and then storing them in the database. First, it directly saves a copy of the information obtained from the first business message to HDFS. Then, it places a copy of the parsed first parsed file into HDFS, distinguishing them by filename. Finally, it writes the filename of the first parsed file to ZooKeeper to notify the system to update the cache promptly. In addition, during the synchronization process, a number index table is generated for later use by the ownerless module when searching for devices by number.

[0087] The system structure of the embodiments of the present invention has been introduced above. It can be understood that the processing of traditional charging call records is highly dependent on the performance of HBASE. If there is a performance bottleneck or downtime in HBASE, there will be a long-term backlog of call records, resulting in the inability of users to trigger the activation of the first call record, disconnection due to reaching the limit, and query of usage amount in a timely manner, seriously affecting the user experience. In the embodiments of the present invention, by combining distributed Redis and cloudified Redis to replace the processing of the leaderless part of HBASE, and based on the original distributed file system, distributed cache, distributed message queue, distributed database, Zookeeper and other components for charging, the timely processing of a large number of call records is achieved; through the deployment of the application software service layer, the platform environment service layer, and the infrastructure service layer, combined with the advantages of Redis caching, the business call records of a large number of users can be processed and stored in real time, and the results can be transmitted downstream in real time, avoiding the performance bottleneck problem of HBASE, enabling timely disconnection / usage query for a larger number of call records, improving the processing efficiency of business call records, and optimizing the user experience.

[0088] Referring to Figure 1 , the embodiments of the present invention provide a method for processing a large number of business call records in the Internet of Things, including the following steps:

[0089] S101. Collect the first business call record through the application software service layer, and perform formatting, analysis, duplicate removal, merging, and distribution processing on the first business call record to obtain a first call record file and a first call record message, and then upload the first call record file and the first call record message;

[0090] S102. Receive the first call record file and the first call record message through the platform environment service layer, perform full synchronization on the call record data stored in the data database according to the first call record file, and perform incremental synchronization on the call record data stored in the data database according to the first call record message;

[0091] Among them, the platform environment service layer is deployed with a distributed file system, a distributed coordination service, a data database, and a streaming processing framework. The platform environment service layer provides computing, storage, and network communication services through the underlying cloud hosts set in the infrastructure service layer.

[0092] The content in the above system embodiments is applicable to the method embodiments of the present invention. The functions specifically implemented by the method embodiments of the present invention are the same as those of the above system embodiments, and the beneficial effects achieved are also the same as those of the above system embodiments.

[0093] Referring to Figure 6 , the embodiments of the present invention provide a device for processing a large number of business call records in the Internet of Things, including:

[0094] At least one processor;

[0095] At least one memory for storing at least one program;

[0096] When the above-mentioned at least one program is executed by the above-mentioned at least one processor, the above-mentioned at least one processor implements the above-mentioned method for processing massive IoT service call detail records.

[0097] The content of the above method embodiments is applicable to the device embodiments. The specific functions implemented by the device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0098] This invention also provides a computer-readable storage medium storing a processor-executable program, which, when executed by a processor, is used to perform the above-described method for processing massive IoT call detail records.

[0099] This invention provides a computer-readable storage medium that can execute a method for processing massive IoT call detail records provided in the method embodiments of this invention. It can execute any combination of the implementation steps of the method embodiments and has the corresponding functions and beneficial effects of the method.

[0100] This invention also discloses a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device can read the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, causing the computer device to perform... Figure 1 The method shown.

[0101] In some alternative embodiments, the functions / operations mentioned in the block diagrams may not occur in the order shown in the operation diagrams. For example, depending on the functions / operations involved, two consecutively shown blocks may actually be executed substantially simultaneously, or the aforementioned blocks may sometimes be executed in reverse order. Furthermore, the embodiments presented and described in the flowcharts of this invention are provided by way of example to provide a more comprehensive understanding of the technology. The disclosed methods are not limited to the operations and logic flows presented herein. Alternative embodiments are contemplated in which the order of various operations is changed and sub-operations described as part of a larger operation are executed independently.

[0102] Furthermore, although the invention has been described in the context of functional modules, it should be understood that, unless otherwise stated, one or more of the aforementioned functions and / or features may be integrated into a single physical device and / or software module, or one or more functions and / or features may be implemented in a separate physical device or software module. It is also understood that a detailed discussion of the actual implementation of each module is unnecessary for understanding the invention. Rather, given the properties, functions, and internal relationships of the various functional modules in the apparatus disclosed herein, the actual implementation of the module will be understood within the scope of conventional skill of an engineer. Therefore, those skilled in the art can implement the invention as set forth in the claims using ordinary techniques without excessive experimentation. It is also understood that the specific concepts disclosed are merely illustrative and not intended to limit the scope of the invention, which is determined by the full scope of the appended claims and their equivalents.

[0103] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0104] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-included system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device.

[0105] More specific examples of computer-readable media (a non-exhaustive list) include: electrical connections (electronic devices) having one or more wires, portable computer disk drives (magnetic devices), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Furthermore, computer-readable media can even be paper or other suitable media on which the aforementioned program can be printed, because the aforementioned program can be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, interpreting, or, if necessary, processing in other suitable ways, and then stored in computer memory.

[0106] It should be understood that various parts of the present invention can be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented in software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0107] In the foregoing description of this specification, references to terms such as "one embodiment," "another embodiment," or "some embodiments" indicate that a specific feature, structure, material, or characteristic described in connection with an embodiment or example is included in at least one embodiment or example of the present invention. In this specification, illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0108] Although embodiments of the invention have been shown and described, those skilled in the art will understand that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the claims and their equivalents.

[0109] The above is a detailed description of the preferred embodiments of the present invention. However, the present invention is not limited to the above embodiments. Those skilled in the art can make various equivalent modifications or substitutions without departing from the spirit of the present invention. All such equivalent modifications or substitutions are included within the scope defined by the claims of this application.

Claims

1. A massive call detail record (CDR) processing system for the Internet of Things (IoT), characterized in that, include: The application software service layer is used to collect the first service call detail record (CDR), and to format, analyze, deduplicatize, merge and distribute the first service CDR to obtain the first CDR file and the first CDR message, and then upload the first CDR file and the first CDR message. The platform environment service layer is equipped with a distributed file system, a distributed coordination service, a data database, and a streaming processing framework. The platform environment service layer is used to receive the first call detail record (CDR) file and the first CDR message, and to perform full synchronization of the CDR data stored in the data database based on the first CDR file, and to perform incremental synchronization of the CDR data stored in the data database based on the first CDR message. The infrastructure service layer includes an underlying cloud host, which provides a computing resource pool, a storage resource pool, and a network resource pool. The platform environment service layer includes a distributed caching architecture, which includes an HBase unit and an engine node. The HBase unit is used to store the data database. The engine node includes a REST service unit, a Redis unit, and a masterless module. The masterless module is used to interact with the Redis unit and the HBase unit to implement the local caching mechanism of the engine node. When processing call detail records, the local cache is queried first. If the local cache is not found, the cloud cache is queried and downloaded from the cloud cache. The platform environment service layer also includes an incremental parsing module, which is used to parse the first call detail record (CDR) message to obtain a first parsing file, save the first parsing file and the first CDR message to the distributed file system, and generate a corresponding data index table.

2. The IoT massive call detail record processing system according to claim 1, characterized in that, The application software service layer includes: The acquisition engine is used to acquire the first service call detail records (CDRs) of IoT call detail record network element devices. The full-process engine is used to format, analyze, deduplicate, merge, and distribute the first business call detail record (CDR) and the user data information stored in the data database to obtain the first CDR file and the first CDR message. An upload engine is used to upload the first call detail record (CDR) file and the first CDR message.

3. The IoT massive call detail record processing system according to claim 2, characterized in that, The end-to-end engine includes: A formatting unit is used to format the first service call detail record (CDR) to obtain a second CDR file and a second CDR message. Analysis unit, the analysis unit is used to analyze and process the second call detail record information to obtain a third call detail record file and a third call detail record message; A deduplication unit is used to perform deduplication processing on the third call detail record (CDR) message to obtain a fourth CDR file and a fourth CDR message. A message distribution unit is configured to distribute the fourth call detail record (CDR) message to obtain the first CDR message, and send the first CDR message to the upload engine. The file distribution unit is used to distribute the fourth call detail record file to obtain the first call detail record file, and send the first call detail record file to the upload engine.

4. The IoT massive call detail record processing system according to claim 3, characterized in that, The end-to-end engine also includes: The merging unit is used to merge the second call detail record (CDR) file, the third CDR file, the fourth CDR file, and the first CDR message to obtain the user information, and then asynchronously store the user information to the data database through the distributed file system.

5. A massive call detail record (CDR) processing system for the Internet of Things (IoT) according to any one of claims 1 to 4, characterized in that, The distributed caching architecture also includes: An interface machine is provided, which includes a data interface and a console. The console interacts with the HBase unit through the data interface and is used to control the REST service unit.

6. The IoT massive call detail record processing system according to claim 5, characterized in that, The platform environment service layer also includes: A cache update service module is used to send a data change instruction to the data interface through the distributed coordination service, and to transmit the data change content to the data interface through the distributed file system, so that the data interface updates the data database.

7. A method for processing massive call detail records (CDRs) in the Internet of Things (IoT), characterized in that, For implementation via an IoT massive call detail record processing system as described in any one of claims 1 to 6, the following steps are included: The application software service layer collects the first service call detail record (CDR), and performs formatting, analysis, deduplication, merging, and distribution processing on the first service CDR to obtain the first CDR file and the first CDR message, and then uploads the first CDR file and the first CDR message. The platform environment service layer receives the first call detail record (CDR) file and the first CDR message, performs full synchronization of the CDR data stored in the data database based on the first CDR file, and performs incremental synchronization of the CDR data stored in the data database based on the first CDR message. The platform environment service layer deploys a distributed file system, a distributed coordination service, a data database, and a streaming processing framework. The platform environment service layer provides computing, storage, and network communication services through the underlying cloud hosts set up in the infrastructure service layer.

8. A device for processing massive call detail records (CDRs) of Internet of Things (IoT) services, characterized in that: include: At least one processor; At least one memory for storing at least one program; When the at least one program is executed by the at least one processor, the at least one processor implements the IoT massive service call detail record processing method as described in claim 7.

9. A computer-readable storage medium storing a processor-executable program, characterized in that, The program executable by the processor is used to perform the IoT massive service call detail record processing method as described in claim 7 when executed by the processor.

Images