Edge computing-enabled cross-carrier IoT SIM card heterogeneous data cleaning system

Abstract

This invention discloses a cross-carrier IoT SIM card heterogeneous data cleaning system enabled by edge computing, belonging to the field of IoT data processing technology. The system includes at least one edge node deployed on the data source side and a cloud server communicating with the edge node. The edge node has a built-in cross-carrier IoT SIM card dedicated data middleware, which includes a unified access layer module, a protocol conversion, and a data normalization engine. This invention can achieve unified access, standardized protocol conversion, and data normalization processing of multi-carrier IoT SIM card data, ensuring the reliability of data extraction and the accuracy of processing results. It significantly reduces the complexity of cross-carrier IoT SIM card integration and maintenance, and is suitable for unified management scenarios of multi-carrier IoT SIM cards.

Description

Technical Field

[0001] This invention relates to the field of IoT data processing technology, specifically to a cross-carrier IoT SIM card heterogeneous data cleaning system enabled by edge computing. Background Technology

[0002] As the core carrier for network connectivity of IoT devices, IoT SIM cards have been widely used in many fields such as vehicle networking, smart metering, and security monitoring. With the rapid expansion of enterprise IoT business, the number of IoT SIM cards used often reaches thousands, tens of thousands, or even millions. Moreover, it is often necessary to purchase IoT SIM cards from multiple operators such as China Unicom, China Mobile, China Telecom, and China Broadcasting Network simultaneously to ensure network coverage and business continuity. This has brought a series of technical problems that urgently need to be solved in the management of enterprise IoT SIM cards.

[0003] At the data extraction level, under the existing management model, the IoT card management platforms of various operators are independent of each other. Enterprise maintenance personnel need to frequently switch between multiple operator systems. The operation methods of each platform are inconsistent, and the data acquisition entry points are scattered, making it impossible to achieve centralized and automated extraction of data from multiple operator cards. At the same time, IoT devices are mostly deployed in remote or unattended environments. The existing model cannot achieve real-time and reliable extraction of card status and traffic data. Events such as abnormal device traffic and downtime cannot be detected in a timely manner, which can easily lead to business losses such as device offline and data loss. For the management needs of massive card volumes, manual extraction and Excel statistics are extremely inefficient and cannot meet the needs of high concurrency and large-scale data extraction. The accuracy and timeliness of data extraction cannot be guaranteed.

[0004] At the data unification level, IoT SIM card packages from different operators vary significantly, including shared models, long-term plans, prepaid plans, non-standard billing periods, and nested sales packages. Data formats, field definitions, traffic measurement units, and pricing rules are inconsistent across operator platforms, making standardized integration of cross-operator data impossible. Under the existing manual statistical model, calculations for complex packages, tariffs, and payment cycles are prone to errors, making it impossible to achieve end-to-end traceability of billing data and hindering businesses from completing independent reconciliation and payments. Furthermore, the existing model cannot uniformly model and manage card lifecycle data, usage details, and billing data from multiple operators, failing to provide businesses with a unified and clear basis for data analysis and control. This can easily lead to losses and decreased customer satisfaction due to untimely handling of issues such as excessive card usage, unauthorized use of multiple plans, and card expiration.

[0005] At the time-series management level, IoT SIM cards undergo multi-stage time-series status changes throughout their entire lifecycle, from procurement, testing, activation, use to cancellation. Furthermore, data usage, plan changes, operation records, and alarm events all exhibit strong time-series characteristics. Existing management models cannot achieve complete traceability and sequential tracking of time-series data throughout the entire IoT SIM card lifecycle. They cannot fully record the entire chain of information, including the operator, time, object, content, and result of the operation. In the event of business disputes or billing inquiries, they cannot achieve full-chain time-series tracing from original call detail records to the final bill. Moreover, existing models cannot achieve pre-emptive warnings and post-event closed-loop processing based on time-series data on data usage and card status, failing to meet enterprises' needs for real-time monitoring, risk warnings, and full-process time-series management of massive numbers of IoT SIM cards.

[0006] Existing technical solutions for IoT SIM card management mostly only achieve basic protocol conversion and format unification. They lack dedicated processing architectures to address the three core pain points mentioned above: data extraction, data unification, and time-series control. These solutions cannot achieve centralized management of IoT SIM cards from the four major telecom operators on a single platform, are difficult to adapt to complex service plans and the management needs of massive SIM card volumes, and fail to meet enterprises' core management requirements for end-to-end traceability, real-time data synchronization, and timely risk warnings for IoT SIM cards. Therefore, how to build a system capable of reliable extraction, standardization, and end-to-end time-series control of IoT SIM card data across different operators is a pressing technical problem that needs to be solved in this field. Summary of the Invention

[0007] The purpose of this invention is to overcome the shortcomings of existing technologies and provide a cross-carrier IoT card heterogeneous data cleaning system enabled by edge computing. It addresses the technical pain points of cross-carrier IoT card management, realizes reliable extraction and standardized unification of IoT card data from multiple carriers, and completes centralized management of cards from the four major carriers on a single platform, thereby reducing enterprise operation and maintenance costs.

[0008] To address the aforementioned technical problems, this invention provides the following technical solution: a cross-carrier IoT SIM card heterogeneous data cleaning system powered by edge computing. This system includes at least one edge node deployed on the data source side and a cloud server communicatively connected to the edge node. The edge node has a built-in cross-carrier IoT SIM card data middleware, which includes a unified access layer module, a protocol conversion and data normalization engine, and an audit log module. The unified access layer module is used to connect to the independent IoT card management platforms of the four major operators: China Unicom, China Mobile, China Telecom, and China Broadcasting Network. It receives raw IoT card data from at least two operators, supports access recognition of multiple operator interface protocols such as HTTP / HTTPS, WebService, and MQTT, and adapts to the interface authentication methods of different operators. The protocol conversion and data normalization engine is connected to the unified access layer module and serves as a data bridge between heterogeneous systems. Based on a preset global unified data model, it performs interface protocol standardization conversion, field mapping, data type conversion, traffic unit MB / GB standardization and format cleaning on the original data to complete data normalization processing and generate standardized unified format data. The edge nodes are used to upload the processed, standardized, and unified formatted data to the cloud server. The cloud server is used to receive standardized and unified format data uploaded by each edge node, perform global data aggregation and storage across edge nodes, and realize centralized management of IoT card data from the four major operators (China Unicom, China Mobile, China Telecom, and China Broadcasting Network) on a single platform without switching between different operator systems.

[0009] This system enables centralized management of IoT SIM cards from the four major telecom operators on a single platform, eliminating the need for frequent switching between multiple operator systems. Data cleaning and transformation are performed locally via dedicated data middleware built into edge nodes, effectively reducing cloud computing load, significantly lowering enterprise operation and maintenance costs, and improving IoT SIM card management efficiency.

[0010] Furthermore, the unified access layer module supports two data extraction mechanisms: timed fetching and event-driven push, and has the ability to resume interrupted transmission and data compensation, ensuring the integrity, reliability, and order of the original data extracted from the IoT card.

[0011] The dual-mode data extraction mechanism balances the real-time nature and stability of data synchronization. Combined with breakpoint resume and data compensation capabilities, it can ensure the integrity, reliability, and orderliness of the extraction of raw data from massive IoT cards.

[0012] Furthermore, the protocol conversion and data normalization engine includes a multi-source data cross-validation submodule, which is used to perform consistency checks on standardized and unified format data that has completed normalization processing. The consistency checks include consistency verification between the operator of the integrated circuit card identification code segment parsing and the access source operator, logical consistency verification between card status and data usage, and range consistency verification between package type and data limit.

[0013] The multi-source data cross-validation mechanism can automatically verify the consistency of data from multiple operators, promptly identify and correct data logic conflicts, and ensure the accuracy and compliance of normalized data. This provides reliable data support for subsequent billing reconciliation and traffic control, significantly reducing the workload and error probability of manual verification.

[0014] Furthermore, the cloud server has a built-in global unified data model storage module, which is used to store a unified IoT card management data model system. The data model system includes a card object model, a usage object model, a package object model, and a bill object model, serving as a unified target model for protocol conversion and data normalization processing at each edge node.

[0015] A standardized IoT SIM card management data system was built through a globally unified data model, unifying the management standards for heterogeneous data from multiple operators. It can adapt to various complex package types such as shared and long-term packages, making cross-operator usage statistics and billing calculations clearer and more standardized, and supporting enterprises to complete independent reconciliation and payment.

[0016] Furthermore, the cloud server can provide core business support internally, enabling unified status management, usage statistics, and billing for IoT cards across different operators, thus supporting the enterprise's internal automated management needs.

[0017] With unified core business support capabilities in the cloud, centralized analysis and management of IoT SIM card data from multiple operators can be achieved, eliminating the need for manual cross-platform statistical accounting, significantly improving internal management efficiency, and adapting to the business management needs of massive SIM card volumes.

[0018] Furthermore, the protocol conversion and data normalization engine has timing control capabilities, performing serial processing according to the timestamp of data access to ensure the timing consistency of state changes and usage statistics for the same IoT card.

[0019] By controlling the timing of the data cleaning phase, card status errors and usage statistics deviations caused by disordered data can be avoided, ensuring the logical accuracy of the normalized data.

[0020] Compared with existing technologies, this edge computing-enabled cross-carrier IoT SIM card heterogeneous data cleaning system has the following advantages: I. This invention, by deploying edge nodes with built-in cross-carrier IoT SIM card dedicated data middleware on the data source side, combined with a centralized cloud management architecture, enables unified access to the IoT SIM card management platforms of the four major carriers, and completes the standardized conversion and normalization of heterogeneous data. This solves the technical problems of high complexity of cross-carrier IoT SIM card integration, inconsistent data standards, and high management and maintenance costs in existing technologies, significantly reducing the operational difficulty of cross-carrier IoT SIM card management for enterprises and improving centralized management efficiency.

[0021] Second, this invention completes data cleaning and transformation processing at edge nodes, combined with a dual-mode data extraction mechanism and a multi-source data cross-validation mechanism. This effectively reduces cloud computing load and network transmission pressure, improves data processing response speed and data accuracy, and ensures logical consistency between card status and usage data through time-series control during the data cleaning phase. Combined with unified core business support capabilities in the cloud, this enhances the system's concurrency capacity and adaptability, meets the internal automation management needs of enterprises during business expansion, and ensures system stability and business continuity.

[0022] Other advantages, objectives and features of the invention will be set forth in part in the description which follows, and in part will be apparent to those skilled in the art from the following examination or study, or may be learned from the practice of the invention. Attached Figure Description

[0023] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are merely some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without any creative effort.

[0024] Figure 1 This is a schematic diagram of the overall system architecture of the present invention; Figure 2 This is a schematic diagram of the collaborative interaction architecture between the cloud server and edge nodes of the present invention; Figure 3 This is a schematic diagram of the system workflow of the present invention. Detailed Implementation

[0025] To further illustrate the technical means and effects of the present invention in achieving its intended purpose, the following detailed description of the specific implementation methods, structures, features, and effects of the present invention, in conjunction with the accompanying drawings and preferred embodiments, is provided below.

[0026] Example This embodiment provides an edge computing-enabled cross-carrier IoT SIM card heterogeneous data cleaning system to achieve reliable extraction and standardized unification of data from multiple carriers' IoT SIM cards, solving the problems of untimely data extraction and inconsistent data formats in cross-carrier IoT SIM card management. The system described in this embodiment can be stably applied to the unified management of cross-carrier IoT SIM cards in various scenarios such as connected vehicles, smart metering, and security monitoring, and can adapt to the management scale of thousands to millions of IoT SIM cards. Figure 1As shown, the system in this embodiment includes at least one edge node deployed on the data source side, and a cloud server communicating with the edge node. The edge node has a built-in dedicated data middleware for cross-carrier IoT cards, which includes a unified access layer module, a protocol conversion and data normalization engine.

[0027] Specifically, edge nodes are deployed in enterprise-side gateways or carrier access data centers, using containerization technology and running on x86 or ARM architecture server devices. Each edge node interfaces with at least one regional carrier IoT SIM card management platform, handling the data streams from the connected carrier interfaces. This offloads data cleaning and transformation tasks from the cloud to locations closer to the data source, reducing cloud computing load and network bandwidth consumption.

[0028] Specifically, the unified access layer module is deployed at the front-end access point of the edge node, serving as the interface between the system and the IoT SIM card management platforms of the four major telecom operators. It connects to the independent IoT SIM card management platforms of China Unicom, China Mobile, China Telecom, and China Broadcasting Network, receiving raw IoT SIM card data from at least two of these operators. The unified access layer module supports access recognition for multiple operator interface protocols, including HTTP / HTTPS, WebService, and MQTT, and can adapt to the interface authentication methods of different operators, completing secure interface connection and data interaction with each operator's platform.

[0029] For example, the unified access layer module can pre-configure the interface with each operator's platform, store the corresponding interface address, authentication parameters, and protocol type information of each operator, and automatically complete the identity authentication and link connection with the corresponding operator's platform during system operation, without the need for manual reconfiguration.

[0030] It is understandable that the unified access layer module is the basic entry point for the system to achieve centralized management of cross-carrier data. Through unified access adaptation capabilities, it avoids the need for enterprise operation and maintenance personnel to frequently switch between multiple carrier systems, and achieves centralized access to IoT card data from multiple carriers.

[0031] Specifically, the unified access layer module supports two data extraction mechanisms: scheduled fetching and event-driven push. The scheduled fetching mechanism proactively sends data requests to the IoT SIM card management platforms of various operators according to a preset time period, obtaining IoT SIM card status, data usage, and billing details for the corresponding period. The event-driven push mechanism can receive real-time event data such as IoT SIM card status changes, data usage exceeding thresholds, and service outage alarms proactively pushed by operator platforms, enabling real-time data acquisition.

[0032] The unified access layer module possesses breakpoint resumption and data compensation capabilities. In the event of link interruption or interface anomalies during data transmission, the unified access layer module can record the breakpoint location. Once the link is restored, data transmission can resume from the breakpoint without retransmitting the entire dataset. For data packets lost or failing verification during transmission, the unified access layer module can automatically initiate retransmission requests to compensate for the lost data, ensuring the integrity, reliability, and order of the original data extracted from the IoT SIM card.

[0033] Specifically, the protocol conversion and data normalization engine connects to the unified access layer module and is deployed at the core processing location of the edge node. It acts as a data bridge between heterogeneous systems, standardizing the raw data acquired by the unified access layer module. Based on a pre-defined globally unified data model, the engine performs interface protocol standardization conversion, field mapping, data type conversion, traffic unit (MB / GB) standardization, and format cleaning on the raw data, completing data normalization processing and generating standardized, unified format data.

[0034] For example, the globally unified data model predefines the unified format and field specifications of core data in the IoT card management process. The protocol conversion and data normalization engine can convert and process non-standardized data returned by different operator platforms according to the specifications of the globally unified data model. For traffic-related fields defined in different operator platforms, the protocol conversion and data normalization engine can uniformly map them to the corresponding traffic fields in the globally unified data model. At the same time, it can uniformly convert the different traffic units such as MB and GB used by different operators into standard units, thereby achieving the unification of data format and standards across different operators.

[0035] It is understandable that the protocol conversion and data normalization engine is the core processing unit for achieving cross-carrier data unification. Through standardized conversion processing, it solves the problem of inconsistent data formats, field definitions, and units of measurement across different carrier platforms, providing a unified data foundation for subsequent data analysis, billing, and centralized management.

[0036] Specifically, the protocol conversion and data normalization engine includes a multi-source data cross-validation submodule, which performs consistency checks on standardized, unified format data that has undergone normalization. Consistency checks include verification of the consistency between the operator of the integrated circuit card identification number segment and the access source operator, logical consistency verification of card status and data usage, and range consistency verification of package type and data limit.

[0037] For example, the multi-source data cross-validation submodule can parse the home operator corresponding to the integrated circuit card based on the number prefix of the card's identification code, compare the parsing result with the access source operator of the data, complete the consistency verification, and identify abnormal data where the home operator and access source do not match. For IoT card data with a suspended status, the multi-source data cross-validation submodule can verify whether its data usage data conforms to the logical range of a suspended status, and identify abnormal data where there is a logical conflict between the card status and data usage. For IoT card data of a corresponding plan type, the multi-source data cross-validation submodule can verify whether its data usage is within the data limit of the corresponding plan, and identify abnormal data exceeding the plan's limit.

[0038] Specifically, the protocol conversion and data normalization engine has a built-in timing scheduling unit that sorts all raw data packets from the same IoT SIM card in ascending order according to the timestamps carried by the data packets and then performs serial processing. For data packets from different IoT SIM cards, multi-threaded parallel processing is used to improve overall processing efficiency.

[0039] The timing scheduling unit maintains the latest status identifier for each IoT card. Only after the data packet of the previous state has been processed and the status identifier has been updated will the next data packet of the card be processed, ensuring that the card status transition is strictly executed in the actual time sequence.

[0040] Specifically, after the edge nodes complete the access and normalization of the raw data, they upload the processed, standardized data to the cloud server. For example... Figure 2 As shown, the cloud server establishes bidirectional communication connections with multiple edge nodes, receives standardized and unified format data uploaded by each edge node, performs global data aggregation and storage across edge nodes, and realizes centralized management of IoT card data from the four major operators (China Unicom, China Mobile, China Telecom, and China Broadcasting Network) on a single platform without the need to switch between different operator systems.

[0041] Specifically, the cloud server has a built-in global unified data model storage module, which is used to store a unified IoT card management data model system. The data model system includes card object model, usage object model, package object model and bill object model, serving as a unified target model for protocol conversion and data normalization processing at each edge node.

[0042] For example, the card object model defines unified data specifications for IoT card basic information, status information, and lifecycle stages; the usage object model defines unified statistical specifications for IoT card data usage and usage duration; the package object model defines unified management specifications for parameters, rules, and limits of different types of packages; and the billing object model defines unified accounting specifications for billing, fee details, and payment information. The globally unified data model storage module synchronizes the above unified data model system to all connected edge nodes, ensuring that data processing on all edge nodes follows unified specifications and achieving unified data standards across operators and edge nodes.

[0043] Understandably, the globally unified data model storage module provides a unified target benchmark for standardized data processing across the entire system, ensuring data consistency throughout the system. It can adapt to the management needs of various complex package types, including shared modes, long cycles, prepaid, and non-standard billing periods, making cross-operator usage statistics and billing calculations clearer and more standardized, and supporting enterprises to complete independent reconciliation and payment.

[0044] Specifically, the cloud server can provide core business support internally, enabling unified status management, usage statistics, and billing for IoT cards across different operators.

[0045] For example, the cloud server can automatically generate cross-carrier IoT SIM card status reports, data usage statistics reports, and monthly billing reports based on standardized data from aggregated storage. It supports multi-dimensional statistical queries by card, department, and project, while also enabling batch management of card status, triggering usage alerts, and automatic billing calculation. Internal operations and maintenance personnel can complete all operations directly through the cloud management interface without switching between different carrier systems.

[0046] Understandably, the unified core business support capabilities in the cloud enable full-process automation of IoT card management across multiple operators, significantly reducing manual operations, improving internal management efficiency, and meeting the centralized management needs of massive card volumes.

[0047] As shown in Figure 3, the overall workflow of the system described in this embodiment is executed according to the following steps: The first step is for the unified access layer module to receive the raw data streams from IoT cards from multiple operators and complete the connection authentication and data access with each operator's platform.

[0048] The second step involves the unified access layer module extracting the original data from the IoT card through timed fetching or event-driven push, and ensuring the integrity and reliability of the extracted data through breakpoint resume and data compensation mechanisms.

[0049] The third step involves the protocol conversion and data normalization engine performing protocol standardization conversion, field mapping, data type conversion, unit unification, and format cleaning on the original data based on a globally unified data model, thus completing the data normalization process.

[0050] The fourth step involves the multi-source data cross-validation submodule performing a consistency check on the normalized data, identifying and marking outlier data.

[0051] The fifth step involves the protocol conversion and data normalization engine performing timing control according to the timestamp of the data access, ensuring the consistency of the timing of the state changes and usage statistics of the same IoT card.

[0052] The sixth step is for the edge nodes to upload the processed, standardized data to the cloud server.

[0053] The seventh step involves the cloud server receiving standardized data uploaded from each edge node, completing global data aggregation and storage across edge nodes, and achieving centralized management of IoT card data from the four major telecom operators on a single platform.

[0054] The edge computing-enabled cross-carrier IoT SIM card heterogeneous data cleansing system described in this embodiment achieves unified access to the IoT SIM card management platforms of the four major operators, standardized conversion and normalization of heterogeneous data, and centralized management of multi-carrier IoT SIM card data on a single platform through dedicated data middleware built into edge nodes. The system works in conjunction with a cloud server to ensure the real-time performance and reliability of data acquisition through a dual-mode data extraction mechanism, and achieves unified data standards across operators through a unified data model and standardized conversion. This system can effectively reduce the operational costs of cross-carrier IoT SIM card management for enterprises, improve management efficiency and data accuracy, and can be stably applied to various cross-carrier IoT SIM card unified management scenarios.

[0055] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention in any way. Although the present invention has been disclosed above with reference to preferred embodiments, it is not intended to limit the present invention. Any person skilled in the art can make some modifications or alterations to the above-disclosed technical content to create equivalent embodiments without departing from the scope of the present invention. Any simple modifications, equivalent changes and alterations made to the above embodiments based on the technical essence of the present invention without departing from the scope of the present invention shall still fall within the scope of the present invention.

Claims

1. A cross-operator IoT card heterogeneous data cleaning system empowered by edge computing, characterized in that, The system includes at least one edge node deployed on the data source side, and a cloud server communicating with the edge node; the edge node has a built-in cross-carrier IoT SIM card data middleware, which includes a unified access layer module and a protocol conversion and data normalization engine. The unified access layer module is used to connect to the independent IoT card management platforms of the four major operators: China Unicom, China Mobile, China Telecom, and China Broadcasting Network. It receives raw IoT card data from at least two operators, supports access recognition of multiple operator interface protocols such as HTTP / HTTPS, WebService, and MQTT, and adapts to the interface authentication methods of different operators. The protocol conversion and data normalization engine is connected to the unified access layer module and serves as a data bridge between heterogeneous systems. Based on a preset global unified data model, it performs interface protocol standardization conversion, field mapping, data type conversion, traffic unit MB / GB standardization and format cleaning on the original data to complete data normalization processing and generate standardized unified format data. The edge nodes are used to upload the processed, standardized, and unified formatted data to the cloud server. The cloud server is used to receive standardized and unified format data uploaded by each edge node, perform global data aggregation and storage across edge nodes, and realize centralized management of IoT card data from the four major operators (China Unicom, China Mobile, China Telecom, and China Broadcasting Network) on a single platform without switching between different operator systems.

2. The edge computing empowered cross-operator IoT card heterogeneous data cleaning system according to claim 1, wherein, The unified access layer module supports two data extraction mechanisms: timed fetching and event-driven push. It has the ability to resume interrupted transmission and data compensation, ensuring the integrity, reliability, and order of the original data extracted from the IoT card.

3. The edge computing empowered cross-operator IoT card heterogeneous data cleaning system according to claim 1, wherein, The protocol conversion and data normalization engine includes a multi-source data cross-validation submodule. This submodule is used to perform consistency checks on standardized and unified format data that has undergone normalization. The consistency checks include consistency verification between the operator of the integrated circuit card identification code segment and the access source operator, logical consistency verification between card status and data usage, and range consistency verification between package type and data limit.

4. The edge computing empowered cross-operator IoT card heterogeneous data cleaning system according to claim 1, wherein, The cloud server has a built-in global unified data model storage module, which is used to store a unified IoT card management data model system. The data model system includes a card object model, a usage object model, a package object model, and a bill object model, serving as a unified target model for protocol conversion and data normalization processing at each edge node.

5. The edge computing-enabled cross-carrier IoT SIM card heterogeneous data cleaning system according to claim 1, characterized in that, The cloud server can provide core business support internally, enabling unified status management, usage statistics, and billing for IoT cards across different operators, thus supporting the enterprise's internal automated management needs.

6. The edge computing-enabled cross-carrier IoT SIM card heterogeneous data cleaning system according to claim 1, characterized in that, The protocol conversion and data normalization engine has timing control capabilities, and performs serial processing according to the timestamp of data access, ensuring the timing consistency of the state changes and usage statistics of the same IoT card.

Images