A method for bulk resource import with optimized cascading import logic

Abstract

This invention discloses a method for large-scale resource entry that optimizes cascading entry logic, belonging to the field of cascading entry for resource sharing between video surveillance platforms. It solves the problems of traditional single-organization entry, which often results in device entry failures before the parent organization, and frequent and time-consuming database accesses during large-scale batch reporting of single device entries. This solution sets up three collaborative threads: initial message reception and processing, asynchronous organization message processing, and asynchronous batch processing of device messages. First, the reported messages are categorized and parsed, and device change detection is performed using Redis. Then, asynchronous organization entry is completed hierarchically from parent to child, unlocking device entry permissions. Finally, timed batch entry is performed on the devices, along with a fault-tolerance mechanism including retry and log alerts. This solution ensures the accuracy of device entry, significantly reduces the number of database accesses, improves the efficiency of large-scale device entry, and guarantees the stability of the cascading entry process.

Description

Technical Field

[0001] This invention relates to the field of big data processing and database management technology, specifically a method for large-scale resource import with optimized cascading import logic. This method is applicable to fields such as video surveillance and equipment management, particularly in scenarios where resources are shared across multiple platforms. By utilizing optimized batch processing and cascading sequence control, it improves resource import efficiency while ensuring data consistency and integrity. Background Technology

[0002] With the popularization of video surveillance technology and the large-scale deployment of IoT devices, the demand for resource sharing among multiple platforms under the national standard cascade architecture is increasing. A large number of monitoring resources, such as organizations and devices, need to be reported from lower-level platforms to higher-level platforms and stored in the database. In existing technologies, the national standard higher-level platform mainly uses a single-line sequential reporting method to process monitoring resources: organization information is entered into the database one by one, and device information is also entered sequentially. This traditional method has several technical drawbacks in practical applications:

[0003] First, the lack of hierarchical control leads to data consistency issues. Because the inbound processes for organizations and equipment lack clear sequential constraints and an effective hierarchical association mechanism, equipment may be inbound before its parent organization, resulting in incorrect equipment mounting, compromising the integrity and accuracy of resource data, and impacting the normal operation of subsequent business.

[0004] Secondly, performance bottlenecks are prominent. When the number of devices is large or device information changes frequently, the method of entering data one by one will cause frequent database access, increase the database load, significantly prolong the data entry time, and make it difficult to meet the needs of rapid data entry of large batches of resources, thus restricting the overall operating efficiency of the system.

[0005] Finally, the order of warehousing lacks strict control. The warehousing processes for organizations and equipment are independent, lacking a unified time-series scheduling mechanism, which may lead to deviations in the association between equipment and organizational data, affecting system stability.

[0006] Therefore, existing technologies cannot meet the requirements of efficient and orderly data entry for large-scale monitoring resources under the national standard cascading architecture. There is an urgent need for an optimized cascading data entry logic to solve problems such as data consistency, performance bottlenecks, and sequence control. Summary of the Invention

[0007] This invention provides a method for large-scale resource ingestion with optimized cascading ingestion logic, aiming to overcome the shortcomings of existing resource ingestion technologies, such as poor data consistency, significant performance bottlenecks, and insufficient sequential control. To this end, the invention employs an independent ingestion thread and a global ingestion device identification mechanism, clearly defines the judgment rules for parent and child organizational levels, and designs a caching and reordering strategy for child organizations when the parent organization has not completed its ingestion. By refining the response aggregation window and timeout / failure handling conditions for "organization ingestion complete," it ensures that all organizations have completed ingestion sequentially before device ingestion, thereby guaranteeing accurate device mounting.

[0008] Furthermore, this invention optimizes database access frequency by introducing a Redis change detection mechanism, clearly defining comparison fields, versions, or timestamp rules, and combining this with a merging strategy for multiple changes within a batch window. Simultaneously, it employs a timed batch data entry method, combined with refined definitions of associated table identifiers and a consistency guarantee mechanism for writing data categorized by identifier, significantly improving the efficiency of large-scale resource data entry.

[0009] To ensure system stability, this invention also includes a three-retry and compensation scheme for batch data entry failures, avoiding issues such as mis-release and inconsistent device mounting in multi-link concurrent scenarios. Ultimately, the invention achieves high efficiency, data consistency, and system stability in resource entry, meeting the needs for rapid and orderly entry of large-scale monitoring resources.

[0010] To achieve the above objectives, this invention provides a method for batch resource importing with optimized cascading import logic. This method is applied to cascading scenarios where resources are shared across multiple platforms. Resource import is achieved through batch processing and cascading order control, and includes the following steps:

[0011] (1) Message classification and caching steps: receive resource messages reported by multiple platforms, parse them and distinguish between organizational resources and device resources, cache them into the corresponding resource cache queues, and lock the device entry permissions through the global control identifier;

[0012] (2) Organization hierarchical storage steps: Start an independent organization storage thread, schedule the organization resources in the cache queue in the order of parent and child cascading to complete the storage, and after all organization resources have been stored, update the global control flag to unlock the device storage permission.

[0013] (3) Equipment change detection step: perform change verification on cached equipment resources, and only include equipment resources with valid changes into the set to be added to the database;

[0014] (4) Batch equipment entry step: Start an independent equipment entry thread, schedule the equipment resources to be entered in batches based on a preset time interval, merge the related database tables after entry, and realize the batch processing of equipment resources;

[0015] (5) Inbound fault tolerance process: If the batch inbound of equipment fails, a retry mechanism of preset number of times is executed. After the retry fails, the fault information is recorded and an alarm is triggered.

[0016] Furthermore, the message classification and caching steps specifically include:

[0017] (a) The received resource message is a Kafka message or a SIP protocol message;

[0018] (b) If the data is resolved to an organizational resource, store the single piece of organizational data in the organizational cache queue and set the global control flag to the unlocked state;

[0019] (c) If the device is parsed as a device resource, the key fields of the device resource are compared with the historical information stored in Redis. The key fields include online status, device name, device parent organization ID, device longitude, device latitude, and device channel type.

[0020] (d) If the comparison result is a new device or a change in field information, the device resource and the associated table identifier field are converted into JSON data and the structure entity PU to be entered into the database and then stored in the device cache queue; if there is no change in the comparison, it is not included in the cache.

[0021] Furthermore, in the hierarchical organization entry process, the organization entry thread schedules the cache queue as follows:

[0022] (a) Check if there is any data to be added to the organization queue. If so, extract a preset number of organization data in each round and add them to the database serially according to the parent and child levels. The child organization is attached to the corresponding parent organization that has been added to the database through the parent organization code it carries. The child organization without a parent organization is attached to the root organization.

[0023] (b) If the organization’s pending data queue is empty, transfer all data in the organization’s data receiving queue to the organization’s pending data queue and repeat step (a).

[0024] Furthermore, the method for determining when all organizational resources have been entered into the database is as follows: a multi-link shared organization entry count variable is set, the count is incremented by 1 when a single organization entry request is initiated, and the count is decremented by 1 when the entry process ends (including success, failure, and timeout); when the count variable continuously returns to zero within the preset response aggregation window, and no new organizational data enters the cache queue, the organization entry is determined to be complete.

[0025] Furthermore, when the parent organization has not completed the data entry, the corresponding child organization data is temporarily stored in the organization data receiving queue. After the parent organization completes the data entry, it is rescheduled to the organization waiting to be entered into the data entry queue and entered into the data in the order of parent and child levels.

[0026] Furthermore, the specific steps for bulk equipment warehousing include:

[0027] (a) If the equipment waiting queue is empty, first transfer all data in the equipment data receiving queue to the equipment waiting queue;

[0028] (b) Extract equipment resources from the equipment waiting queue at preset time intervals, extract a preset number of equipment data in each batch, aggregate and merge them into a temporary table, and update the related database table in batches in the form of tuple data;

[0029] (c) The association of the related database tables is based on the association table identifier, which is a national standard equipment code, used to ensure the association of data of the same equipment and the independence of data of different equipment.

[0030] Furthermore, the scope of the global control identifier is global, based on cascaded links, or based on organizational tree branches, in order to avoid misunderstandings in device entry permissions during multi-link concurrent scenarios.

[0031] Furthermore, the key field comparison also incorporates version number or timestamp rules; if the same device resource undergoes multiple changes within the batch entry window, only the device data of the latest change is retained and included in the set to be entered into the database.

[0032] Furthermore, during batch data entry, equipment data is categorized according to the associated table identifier. Equipment data with the same associated table identifier is grouped into the same batch for data entry, ensuring data consistency between database tables.

[0033] Furthermore, the preset retry mechanism allows for three retries. If all three retries fail, the device data, error code, and cause of failure are recorded in the system log, and a Kafka alarm notification is sent to the management terminal to achieve real-time alerts for failed data entry and subsequent troubleshooting.

[0034] The present invention, by adopting the above technical solution, has at least the following beneficial effects:

[0035] This invention addresses the cascading scenario of shared resources across multiple platforms under a national standard cascading architecture. Through a technical solution combining optimized cascading sequence control and batch processing mechanisms, it precisely solves the technical defects of existing technologies, such as poor data consistency, significant performance bottlenecks, and insufficient sequence control in resource storage. Simultaneously, it adapts to the practical application requirements of multi-link concurrency, achieving significant improvements in data consistency, storage efficiency, and system stability. It possesses both strong practicality and scalability, with the following specific beneficial effects:

[0036] This invention employs an independent organization entry thread and a resource scheduling strategy based on the parent-child cascading order. Combined with a caching and reordering mechanism for child organizations when the parent organization has not completed its entry, it ensures that organizational resources are entered into the database in a hierarchical and orderly manner. Simultaneously, relying on the timing locking function of the global control identifier, and combining multi-link shared counting variables and response aggregation windows, it achieves precise determination of the organization entry completion status. Device entry permissions are only unlocked after all organizations have completed their entry, fundamentally avoiding the phenomenon in existing technologies where devices are entered before their parent organizations, resulting in devices having no parent organization attached. This ensures the accuracy of the association between organization and device data and guarantees the data integrity and consistency of the entire cascading resource database.

[0037] This invention utilizes a Redis key field change detection mechanism to precisely filter device resources, including only newly added or modified device resources in the database entry set. This effectively filters out duplicate data without changes, reducing invalid entry operations. Simultaneously, it employs a timed batch entry processing method, aggregating scattered device data into a temporary table and updating the associated database table in batches as tuple data. This replaces the existing single-row entry mode, significantly reducing database access frequency and lowering database load. Furthermore, the preset time interval and data volume for batch entry can be flexibly adjusted, perfectly adapting to application scenarios involving large-scale device reporting. This significantly shortens the overall entry time, greatly improves the speed of reporting lower-level device information, and solves the performance bottleneck of long entry times when dealing with a large number of devices in existing technologies.

[0038] This invention, through its multi-dimensional scope design of global control identifiers, can be set to global, cascaded link, or organizational tree branch according to actual needs. This effectively avoids the problem of misunderstanding and locking of device entry permissions in multi-link concurrent scenarios, and achieves strict control over the entry order. At the same time, a fault-tolerant mechanism with three retries is designed for batch entry. The system logs the error code and fault reason for retry failure and sends real-time alarm notifications to the management end through Kafka, realizing real-time reminders and fault traceability for entry failure. Combined with the consistency guarantee mechanism of writing according to the association table identifier, it avoids the problem of inconsistent organization and device mounting relationships caused by batch entry failure. This allows the entire resource entry process to run stably in complex scenarios with multi-link concurrency and frequent changes in device information, improving the robustness of the system.

[0039] The core optimization logic of this invention is based on the national standard cascading architecture design, which can be directly applied to multi-platform resource sharing scenarios in fields such as video surveillance and equipment management. It is compatible with resource message reception using Kafka and SIP protocols, and the time interval for batch data entry, the amount of data per batch, the comparison fields for Redis change detection, and the scope of global control identifiers can all be flexibly adjusted according to actual business needs without modifying the core architecture. At the same time, the entire technical solution adopts a modular thread design and queue scheduling, which can be seamlessly integrated with the resource management system of existing cascading platforms. It has low transformation costs, high practicality, and is suitable for multi-platform cascading resource entry needs of different scales.

[0040] This invention, through the collaborative optimization of cascading sequence control and batch processing, combined with supporting mechanisms such as change detection, fault tolerance, and concurrency control, forms a complete solution for cascading large-scale resource storage. It achieves a balance between high efficiency, data consistency, and system stability in resource storage, perfectly meeting the actual needs of rapid and orderly storage of large-scale monitored resources under the national standard cascading architecture. Compared with existing technologies, it has outstanding substantive features and significant progress. Attached Figure Description

[0041] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0042] Figure 1 This is a flowchart of the method for batch resource entry into the database, which optimizes the cascading entry logic of this invention.

[0043] Figure 2 This is a flowchart of the initial message receiving and processing thread of the present invention.

[0044] Figure 3 This is a flowchart of the asynchronous message loading thread of this invention.

[0045] Figure 4 This is a flowchart of the asynchronous batch data entry thread for device messages in this invention. Detailed Implementation

[0046] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numerals in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present invention. Rather, they are merely examples of apparatuses and methods consistent with some aspects of the invention as detailed in the appended claims.

[0047] Example 1

[0048] like Figure 1 As shown, this embodiment provides a method for batch resource importing with optimized cascading import logic, applicable to cascading scenarios where resources are shared across multiple platforms. Resource import is achieved through batch processing and cascading order control, including the following steps:

[0049] (1) Message classification and caching step: Receive resource messages reported by multiple platforms, parse them to distinguish between organizational resources and device resources, cache them in the corresponding resource cache queues, and lock device entry permissions through a global control identifier. This step, as the core pre-process of the entire entry process, realizes unified reception and classification control of multi-platform cascading messages, locks device entry permissions in advance through a global control identifier, establishes the priority entry principle of organizations from the source of the process, and lays the foundation for subsequent cascading sequence control. In the existing technology, organizational and device data are stored together without isolation, and there is no dedicated entry permission locking mechanism. This step realizes strict isolation between organizational and device data through a dedicated queue, and realizes pre-process permission control of device entry based on a global control identifier. It is the core pre-process design to realize organizational entry before device entry, and also provides a technical basis for entry control in subsequent multi-link concurrent scenarios. It solves the problem of chaotic entry sequence caused by mixed storage of organizational and device data in traditional solutions.

[0050] (2) The hierarchical organization entry step starts an independent organization entry thread, schedules the organization resources in the cache queue in the parent-child cascade order to complete the entry, and updates the global control flag to unlock the device entry permission after all organization resources have been entered. The asynchronous processing of organization entry is realized through an independent thread to avoid mutual blocking with the device entry process. The parent-child cascade order scheduling ensures the accuracy of the organizational hierarchy association. The status update of the global control flag realizes the precise timing linkage between the organization and the device entry process, fundamentally avoiding the problem of the device being mounted without a parent organization. This step is the core implementation link of the key innovation of this invention. In the prior art, there is no strict hierarchical order requirement for the entry of a single organization, nor is there a unified standard for judging the completion of entry. This application realizes the individual entry association of the national standard cascaded reporting organization by using an independent entry thread in conjunction with the parent-child hierarchical scheduling method and the multi-link shared entry completion judgment mechanism, ensuring that all organizations have been successfully entered in the hierarchy before the device is entered, and completely solving the problem of occasional mounting failure caused by the device entering the database before the parent organization in the prior art.

[0051] (3) The equipment change detection step verifies the changes of cached equipment resources and only includes equipment resources with valid changes into the set to be entered into the database. This step is a preliminary data screening for batch entry. By accurately verifying changes, duplicate equipment data without changes is filtered out, which greatly reduces the amount of data to be entered into the database in subsequent batches, avoids invalid database operations, and further improves the overall entry efficiency. In the prior art, no change detection step is set for single entry of equipment, resulting in a large number of repetitive and time-consuming entry operations. This application relies on Redis to complete the dual change detection of key fields combined with version or timestamp, and follows the merging strategy of retaining only the latest data for multiple changes within the batch window. This achieves accurate retrieval of equipment data for the first and second time, reduces invalid entry operations from the data source, and provides a data foundation for the efficient entry of large batches of equipment.

[0052] (4) Batch equipment entry step: Start an independent equipment entry thread, schedule the equipment resources to be entered in batches based on a preset time interval, merge them into the database table, and realize batch processing of equipment resources; the independent thread ensures the efficient execution of batch equipment entry, and the combination of preset time interval and batch scheduling transforms the traditional single entry into batch integrated entry, greatly reducing the frequency of database access and the database load, and adapting to the scenario requirements of large-scale equipment cascading reporting. This step is the core implementation link of another key innovation of this invention. In the prior art, equipment is entered into the database one by one. When there are too many equipment, there will be frequent access to the database table, and the entry time will increase significantly with the increase of the number of equipment. This application uses a combination of timed batch scheduling and temporary table merging, and writes the equipment into the database according to the national standard equipment code as the identifier of the related table, realizing the transformation from single and repeated timed entry to timed batch fast entry, greatly accelerating the entry and reporting speed of lower-level equipment information, while effectively reducing the number of database table accesses and ensuring data consistency between related database tables.

[0053] (5) Fault-tolerant processing steps for inbound storage: If batch inbound storage of equipment fails, a retry mechanism with a preset number of attempts is executed. After a retry failure, fault information is recorded and an alarm is triggered. This step provides a fallback guarantee for the batch inbound process. The retry mechanism can solve temporary faults such as network jitter and temporary database busyness. Fault information recording and alarms can realize the real-time discovery and traceability of problems, and avoid the accumulation of fault data from affecting the stability of the overall cascading inbound process. In the existing technology, there is no dedicated fault-tolerant mechanism for equipment inbound storage, and there are no compensation processing measures after inbound storage failure, which can easily lead to inconsistencies between the organization and the equipment mounting relationship. This application achieves full-process control of inbound storage failure through a compensation processing scheme of multiple retries, combined with complete fault log recording and real-time Kafka alarms, effectively avoiding the problem of equipment and organization mounting association caused by batch inbound storage failure, and ensuring the stability of the entire cascading inbound process.

[0054] As one implementation method, the message classification and caching step in this embodiment specifically includes:

[0055] (a) The received resource messages are Kafka messages or SIP protocol messages; this message type is compatible with the mainstream resource reporting transmission protocol of the national standard cascading platform, and can be compatible with heterogeneous message interaction between multiple platforms, thereby improving the scenario adaptability of the technical solution.

[0056] (b) If the data is parsed as an organizational resource, the single piece of organizational data is stored in the organizational cache queue and the global control flag is set to the unlocked state. The global control flag is initially in the unlocked state, which can lock the device's access permission from the source and prevent the device from initiating an access request in advance when the organization has not been accessed. This ensures the core logic of prioritizing access to the organization in terms of timing.

[0057] (c) If the resolution is for equipment resources, the key fields of the equipment resources are compared with the historical information stored in Redis. The key fields include online status, equipment name, equipment parent organization ID, equipment longitude, equipment latitude, and equipment channel type. Redis is a high-performance cache database that can quickly complete the comparison of key fields. The equipment channel type specifically includes national standard monitoring equipment types such as bullet cameras and PTZ cameras. The comparison fields comprehensively cover the core attributes of the equipment to ensure the accuracy of change detection.

[0058] (d) If the comparison result indicates the addition of a new device or a change in field information, the device resource and associated table identifier fields are converted into JSON data and the PU structure entity to be imported into the database before being stored in the device cache queue; if the comparison shows no changes, the data is not included in the cache. JSON data is compatible with cross-platform data transmission, and the PU structure entity to be imported into the database is compatible with the local database import format. The dual format conversion can take into account both data transmission compatibility and import adaptability. Direct filtering of unchanged data can significantly reduce invalid import operations and reduce database access pressure.

[0059] As one implementation method, in the hierarchical organization entry step of this embodiment, the organization entry thread schedules the cache queue in the following manner:

[0060] (a) Check if there is any data to be added to the organization queue. If so, extract a preset number of organization data in each round and add them to the database sequentially according to the parent and child levels. The child organization is attached to the corresponding parent organization that has been added to the database through the parent organization code it carries. The child organization without a parent organization is attached to the root organization. In this embodiment, the preset number is 1,000, which is suitable for the efficiency of adding a large number of organization resources. The sequential addition of data from parent to child levels is a hard requirement of the national standard cascading architecture. The accurate attachment through the parent organization code can ensure the accuracy of the organization hierarchy association. The attachment to the root organization without a parent organization can ensure the integrity of the organization tree.

[0061] (b) If the organization's pending data entry queue is empty, transfer all data in the organization's data receiving queue to the organization's pending data entry queue and repeat step (a). Using the whole queue transfer method instead of transferring individual data can reduce the frequency of queue operations, improve the efficiency of organization data scheduling, and ensure that the order of organization data entry is consistent with the order of reporting.

[0062] As one implementation method, the determination method for all organizational resources being fully entered into the database in this embodiment is as follows:

[0063] A shared organization entry count variable is set up across multiple links. The count is incremented by 1 when a single organization entry request is initiated and decremented by 1 when the entry process ends (including success, failure, and timeout). When the count variable continuously reaches zero within a preset response aggregation window, and no new organization data enters the cache queue, the organization entry is considered complete. This shared count variable adapts to multi-platform cascading concurrent reporting scenarios, avoiding the limitations of single-link judgments. The response aggregation window filters out temporary zero-value states of the count variable, ensuring the accuracy of the organization entry completion determination. The count deduction for timeout and failure scenarios prevents the count variable from becoming stuck, ensuring the robustness of the determination mechanism.

[0064] As one implementation method, in this embodiment, when the parent organization has not completed its data entry, the corresponding child organization data is temporarily stored in the organization data receiving queue. After the parent organization completes its data entry, it is rescheduled to the organization waiting-to-be-entered queue and entered into the database in parent-child hierarchical order. Temporarily storing child organizations can avoid the problem of mounting failure caused by the parent organization not being entered into the database. Rescheduling after the parent organization is entered into the database can ensure the hierarchical entry order of the organization tree and ensure the hierarchical consistency of the entire organization's resources.

[0065] As one implementation method, the batch warehousing step in this embodiment specifically includes:

[0066] (a) If the equipment waiting queue is empty, all data in the equipment data receiving queue is transferred to the equipment waiting queue. The overall transfer of the equipment queue can ensure that the order of equipment data entering the warehouse is consistent with the order of reporting, while reducing the number of queue scheduling operations and improving the efficiency of batch processing of equipment data.

[0067] (b) Extract equipment resources from the equipment waiting queue at preset time intervals, extract a preset number of equipment data in each batch, aggregate and merge them into a temporary table, and update the associated database table in batches in the form of tuple data; in this embodiment, the preset time interval is 2 seconds and the preset number is 1000 records. These parameters are adapted to the efficiency requirements of a large number of equipment reports. The temporary table merging can integrate scattered single equipment data into structured data, and the batch update in the form of tuple data can simplify multiple single database accesses into one batch access, greatly reducing the frequency of database access and reducing the database load;

[0068] (c) The association of the related database tables is based on the association table identifier, which is the national standard equipment code. This identifier is used to ensure the association of data for the same device and the independence of data for different devices. The national standard equipment code is a unique identifier for devices in the national standard cascading platform. Using this as the association basis can ensure the association of multi-dimensional data of the same device in different database tables, while ensuring the independence between data of different devices and avoiding data confusion.

[0069] As one implementation method, the scope of the global control identifier in this embodiment is global, per cascaded link, or per organizational tree branch, used to avoid misunderstandings in device entry permissions during multi-link concurrent scenarios. The global scope adapts to the overall entry control of cascaded systems across the entire platform, while the scopes of cascaded link and organizational tree branch-based scopes adapt to hierarchical cascaded entry control across multiple regions and branches. This allows for flexible selection based on the actual cascaded deployment scenario, accurately avoiding misunderstandings in device entry permissions during multi-link concurrent scenarios.

[0070] As one implementation method, this embodiment also combines version number or timestamp rules for key field comparison; if the same device resource undergoes multiple changes within the batch entry window, only the device data of the latest change is retained and included in the pending entry set. Version number and timestamp rules can accurately determine the change sequence of device data. Retaining only the latest data after multiple changes within the batch window can avoid duplicate entry and data overwriting issues, reduce invalid device entry operations, and further improve batch entry efficiency.

[0071] As one implementation method, in this embodiment, during batch data entry, device data is categorized according to the identifier of the associated table. Device data with the same associated table identifier is grouped into the same batch for data entry, ensuring data consistency between database tables. Ensuring that device data with the same associated table identifier is entered into the same batch avoids inconsistencies between database tables caused by entering multi-dimensional data of the same device across batches, thus ensuring the synchronization and integrity of device data in the associated database tables.

[0072] As one implementation method, the retry mechanism in this embodiment presets three retries. If all three retries fail, the device data, error code, and cause of failure are recorded in the system log, and a Kafka alarm notification is sent to the management terminal to achieve real-time alerts and subsequent troubleshooting for data entry failures. Three retries represent the optimal number of retries that balances fault tolerance and processing efficiency, resolving temporary data entry failures caused by network jitter or temporary database busyness. The system log can fully retain failure information for subsequent traceability and analysis, and the Kafka alarm notification enables real-time alerts from the management terminal, significantly improving the efficiency of handling data entry failures and preventing the accumulation of faulty data.

[0073] Example 2

[0074] like Figure 2, Figure 3 and Figure 4 As shown, this embodiment provides a method for bulk resource entry into the database by optimizing the cascading entry logic, applied to cascading scenarios where video surveillance resources are shared between video surveillance platforms. Existing technologies suffer from two major problems: first, when an organization enters the database individually, if a device occasionally completes its entry before its parent organization, the device may lack a parent organization to mount; second, when a device enters the database individually, if the number of devices is too large, frequent access to database tables will occur, increasing the time consumption. To solve these problems, this embodiment uses a three-thread collaborative scheduling mechanism—a message initial reception and processing thread, an organization message asynchronous entry thread, and a device message asynchronous batch entry thread—to achieve hierarchical priority entry of organizations and efficient, timed batch entry of devices. The specific implementation process is as follows:

[0075] like Figure 2 The following is a flowchart of thread 1: initial message reception and processing thread;

[0076] This thread is the pre-processing stage for cascading resource import, responsible for message reception, parsing and classification, change detection, and data caching. The specific steps are as follows:

[0077] 1.1 Received device information message: Received device information message (Kafka message and SIP protocol message); This message type is compatible with the mainstream signaling transmission specifications of the national standard cascading platform, ensuring that heterogeneous messages between multiple platforms can interact normally;

[0078] 1.2 Parsing Cascading Organization and Device Messages: Parsing cascading organization and device message messages; performing structured parsing on the messages to extract relevant information on organization resources and device resources, laying the foundation for subsequent classification and storage;

[0079] 1.3 Determine resource type and cache organization data: Determine whether the resource is an organization or a device. If it is an organization, store the single data entry of the organization to be entered into the organization data receiving queue, and set the identifier of the device to be entered to false. The scope of this identifier is global. The identifier of the device to be entered can also be flexibly configured to be based on cascading links or organizational tree branches, which can be selected according to the actual scenario to avoid accidental permission granting in multi-link concurrent scenarios.

[0080] 1.4 Device Resource Change Detection: If it is a device, the received device information is compared with the key fields of the existing information in Redis. The key fields include online status, device name, device parent organization ID, device longitude, device latitude, and device channel type (bullet camera, PTZ camera). The comparison process is combined with version number or timestamp rules to further improve the accuracy of change detection and avoid misjudgment caused by repeated field comparisons. 1.5 Change Judgment and Data Conversion: Determine whether the device information has changed. If it is a newly added device, there must be a change. If true, the device information to be entered into the database and the related table identifier field are converted into JSON data and the structure entity PU to be entered into the database. If there is no change, it is not entered into the database. JSON data is adapted for cross-platform transmission, and the structure entity PU to be entered into the database is adapted to the local database format. The dual format conversion takes into account compatibility and database adaptation. 1.6 Caching Device Data: Store the single data to be entered into the database into the device data receiving queue to complete the entire process of initial message reception and processing.

[0081] like Figure 3 The following is a flowchart of thread 2: asynchronous message storage.

[0082] This thread is the core step in organizing resources and is responsible for hierarchically scheduling and organizing data and determining the completion status of the data insertion. The specific steps are as follows:

[0083] 2.1 Organization pending entry queue scheduling: Determine if the organization pending entry queue is empty. If not empty, extract 1000 organization information entries from the queue in each round. The national standard stipulates that parent organizations are reported first, followed by child organizations, and they are entered into the database sequentially. Based on the parent organization code field carried by each child organization, select an existing matching parent organization code to attach to it. If there is no parent organization, attach to the root organization. If the queue is empty, proceed to the next step 2.2. If the parent organization has not completed its entry, the corresponding child organization data is temporarily stored in the organization data receiving queue. After the parent organization completes its entry, it is rescheduled to the organization pending entry queue to ensure the hierarchical entry order.

[0084] 2.2 Transfer of data from the organization receiving queue: Determine if the organization data receiving queue is empty. If not empty, push the entire organization data receiving queue into the organization waiting-to-be-in queue. If empty, proceed to step 2.3. Return to step 2.1 and repeat the queue scheduling to ensure that all organization data can be processed.

[0085] 2.3 Organization Inbound Completion Determination: When a single organization inbound request is completed, the count of the organization inbound variable shared by multiple links is incremented by 1. When a single organization inbound process ends, the inbound response interface is called (regardless of success, failure, or timeout). At this interface, the count of the organization inbound variable is decremented by 1. When the organization inbound variable returns to zero within a certain period of time, that is, all organization inbound processes are completed and no more are added, it indicates that both the organization reporting and organization inbound processes have been completed in stages, and the organization inbound is determined to be complete, which means that the identifier of the inbound device turns to true. In this embodiment, the preset response aggregation window of 5 seconds can filter the temporary state of the count variable returning to zero, ensuring the accuracy of the determination result.

[0086] like Figure 4 The following is a flowchart of thread 3: asynchronous batch data entry of device messages.

[0087] This thread is the core step in the device resource ingestion process. It is responsible for batch scheduling of device data, ensuring consistency during ingestion, and handling fault tolerance. The specific steps are as follows:

[0088] 3.1 Device entry permission verification: Determine whether the identifier of the device to be entered is true. If it is false, continue to the next scheduling loop to start this thread. If it is true, proceed to the next step 3.2. In this embodiment, the scheduling loop interval is set to 1 second to balance waiting efficiency and resource consumption.

[0089] 3.2 Scheduled Batch Data Scheduling and Merging: Batch data entry is performed every 2 seconds. If the device data entry queue is empty, the entire device data receiving queue is pushed into the device data entry queue first. Each batch of data entry consists of 1000 records (1000 individual JSON records are concatenated into one record). For each batch change, multiple device data records and multiple fields under each device data record are aggregated and merged into a temporary table. The table is used as tuple data to batch update the related database table, realizing rapid merging and entry of batch device data. If the same device undergoes multiple changes within the batch entry window, only the latest changed data is retained to avoid duplicate entry and data overwriting.

[0090] 3.3 Categorized Data Entry and Consistency Assurance: The batch data entry function enters batch data into the associated tables based on their associated table identifiers. The unique identifier for national standard equipment is the national standard equipment code. The correlation between database information tables for the same equipment and the independence between data from different equipment are based on this code, i.e., the associated table identifier. Similarly, the national standard codes for key equipment information reported by lower-level units also use the same identifier. This identifier is used to accurately link and enter the equipment information into the database's associated table. Data is written according to the associated table identifier to ensure data consistency between database tables and avoid association anomalies caused by cross-batch data entry.

[0091] 3.4 Inbound Fault Tolerance and Retry: If the batch inbound is successful, the process ends; if the inbound fails, the inbound will be retried three times; three retries can resolve temporary faults such as network jitter and temporary database busy, and improve the inbound success rate.

[0092] 3.5 Fault Recording and Alarms: If the retry is successful, the process ends; otherwise, the error code and reason are printed in the log, and a Kafka notification of the failed data entry is sent to the management terminal to remind the user in real time. The device information and reason for the failure are also recorded in the system log, allowing users to periodically check the log records of failed data entry. The fault information is fully retained for subsequent traceability and analysis, avoiding the accumulation of fault data.

[0093] This embodiment offers at least the following benefits: It associates organizations reported via national standard cascading into a separate database, using independent threads and device identifiers to ensure successful database entry before device entry, thus preventing devices from being without a parent organization. It optimizes time-consuming, repetitive database entry into scheduled, batch-based, fast entry, processing 1000 records every 2 seconds. Data is merged using temporary tables, significantly reducing database access frequency and accelerating the reporting speed of lower-level device information. It clarifies the scope of device identifiers (global, by cascading link, by organizational tree branch), supplements organizational hierarchy (parent to child) and caching and reordering strategies for child organizations when the parent organization is incomplete, avoiding timing chaos in multi-link concurrent scenarios. It clarifies the comparison fields, version, or timestamp rules for Redis change detection, as well as the merging strategy for multiple changes within a batch window. It refines the meaning and classification of associated table identifiers, and supplements compensation schemes for three retries, log recording, and Kafka alerts to ensure data consistency.

[0094] Compared to the advantages of traditional video cascading platforms, this embodiment solves the problems of organization entry order and data isolation: In traditional technical solutions, organization entry does not have strict order and strict isolation of device entry data; This embodiment achieves strict organization entry process control by using independent organization entry threads, hierarchical scheduling from parent to child, and data isolation between organization data receiving queues and device data receiving queues, ensuring that devices are accurately attached to parent organizations;

[0095] Improving the efficiency of batch device data entry: Traditional technical solutions involve many repetitive and time-consuming operations when entering data for a single device, and the slow entry process becomes apparent when entering data for a large number of devices. This embodiment uses Redis change detection to achieve the first and second retrieval of device data (first comparison with historical information, second filtering of data without changes), combined with the modification of scheduled batch data entry, making data entry more efficient and faster in scenarios with large-scale device reporting.

[0096] Although embodiments of the present invention have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting the present invention. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of the present invention.

Claims

1. A method for bulk resource importing with optimized cascading import logic, characterized in that: In cascading scenarios where resources are shared across multiple platforms, resource import is achieved through batch processing and cascading order control, including the following steps: (1) Message classification and caching steps: receive resource messages reported by multiple platforms, parse them and distinguish between organizational resources and device resources, cache them into the corresponding resource cache queues, and lock the device entry permissions through the global control identifier; (2) Organization hierarchical storage steps: Start an independent organization storage thread, schedule the organization resources in the cache queue in the order of parent and child cascading to complete the storage, and after all organization resources have been stored, update the global control flag to unlock the device storage permission. (3) Equipment change detection step: perform change verification on cached equipment resources, and only include equipment resources with valid changes into the set to be added to the database; (4) Batch equipment entry step: Start an independent equipment entry thread, schedule the equipment resources to be entered in batches based on a preset time interval, merge the related database tables after entry, and realize the batch processing of equipment resources; (5) Inbound fault tolerance process: If the batch inbound of equipment fails, a retry mechanism of preset number of times shall be executed. If the retry fails, the fault information shall be recorded and an alarm shall be triggered. During the hierarchical organization entry process, the organization entry thread schedules the cache queue as follows: (a) Check if there is any data to be added to the organization queue. If so, extract a preset number of organization data in each round and add them to the database serially according to the parent and child levels. The child organization is attached to the corresponding parent organization that has been added to the database through the parent organization code it carries. The child organization without a parent organization is attached to the root organization. (b) If the organization’s pending data queue is empty, transfer all data in the organization’s data receiving queue to the organization’s pending data queue and repeat step (a). The method for determining when all organizational resources have been entered into the database is as follows: a multi-link shared organization entry count variable is set. The count is incremented by 1 when a single organization entry request is initiated and decremented by 1 when the entry process ends. When the count variable continuously returns to zero within the preset response aggregation window and no new organizational data enters the cache queue, the organization entry is determined to be complete. When the parent organization has not completed the data entry, the corresponding child organization data is temporarily stored in the organization data receiving queue. After the parent organization completes the data entry, it is rescheduled to the organization waiting to be entered queue and entered in the order of parent and child levels. The specific steps for bulk equipment warehousing include: (a) If the equipment waiting queue is empty, first transfer all data in the equipment data receiving queue to the equipment waiting queue; (b) Extract equipment resources from the equipment waiting queue at preset time intervals, extract a preset number of equipment data in each batch, aggregate and merge them into a temporary table, and update the related database table in batches in the form of tuple data; (c) The association of the related database tables is based on the association table identifier, which is a national standard equipment code, used to ensure the association of data of the same equipment and the independence of data of different equipment.

2. The method according to claim 1, characterized in that: The message classification and caching steps specifically include: (a) The received resource message is a Kafka message or a SIP protocol message; (b) If the data is resolved to an organizational resource, store the single piece of organizational data in the organizational cache queue and set the global control flag to the unlocked state; (c) If the device is parsed as a device resource, the key fields of the device resource are compared with the historical information stored in Redis. The key fields include online status, device name, device parent organization ID, device longitude, device latitude, and device channel type. (d) If the comparison result is a new device or a change in field information, the device resource and the associated table identifier field are converted into JSON data and the structure entity PU to be entered into the database and then stored in the device cache queue; if there is no change in the comparison, it is not included in the cache.

3. The method according to claim 1, characterized in that: The scope of the global control identifier is global, based on cascaded links, or based on organizational tree branches. It is used to avoid misunderstandings in device entry permissions in multi-link concurrent scenarios.

4. The method according to claim 2, characterized in that: Key field comparisons also incorporate version number or timestamp rules; if the same device resource undergoes multiple changes within the batch import window, only the device data of the latest change is retained and included in the import set.

5. The method according to claim 4, characterized in that: When performing batch inbound data entry, the equipment data is categorized according to the associated table identifier. Equipment data with the same associated table identifier is grouped into the same batch for inbound data entry, ensuring data consistency between database tables.

6. The method according to claim 1, characterized in that: The preset retry mechanism is three retries. If all three retries fail, the device data, error code, and cause of failure are recorded in the system log, and a Kafka alarm notification is sent to the management terminal to realize real-time reminders of data entry failures and subsequent troubleshooting.

Images