Low-delay processing and dynamic capacity expansion and contraction method for mass structured data of view library

Abstract

The invention belongs to the technical field of security and protection video monitoring and multimedia data processing, and discloses a low-delay processing and dynamic capacity expansion and contraction method for mass structured data of a view library. Acquisition side access service and data analysis storage service are decoupled, message-oriented middleware is introduced to realize asynchronous transmission of structured data, structured field storage and picture uploading are completed in a centralized manner by processing service, and structured data and picture resources are managed in a layered manner by adopting sub-library sub-table and object storage. Multi-instance deployment and elastic capacity expansion and shrinkage are performed on the processing service according to the number of the acquisition devices and the data push amount, the access pressure and database load are reduced in a business high-concurrency scene, the structured data processing and result return time delay is shortened, and the stability and expandability of a view library system under large-scale deployment are improved.

Description

Technical Field

[0001] This invention belongs to the field of security video surveillance and multimedia data processing technology, specifically involving a method for low-latency processing and dynamic scaling of massive structured data in a view library. Background Technology

[0002] In the field of security video surveillance and multimedia data processing, view libraries typically serve as the core hub connecting acquisition devices / systems, application platforms, and analysis devices / systems. They are responsible for functions such as multi-device access, video image and structured data management, and external services. Industry standards often use the GA / T1400 series standards as the basic basis, specifying the interaction rules and network relationships between the four main modules—acquisition devices / systems, view libraries, application platforms, and analysis devices / systems—based on acquisition interfaces, cascading interfaces, data service interfaces, and analysis interfaces. In engineering practice, view libraries often also include 1400DAS modules for accessing acquisition devices / systems, GAC / GAS modules for cascading upper and lower level view libraries, and interface service modules for connecting to third-party acquisition systems and application platforms, to support interoperability with different platforms and systems.

[0003] As video surveillance continues to expand, front-end cameras, acquisition systems, and intelligent analysis systems continuously report structured data and associated images containing attribute information of objects such as vehicles, faces, and human bodies to the view library. A single device can generate a large volume of structured records and image data during daily operation. When the number of devices reaches a certain level and access nodes are distributed across different areas, the concurrent write and query volume of structured data faced by the view library will surge dramatically in a short period. This places higher demands on the storage capacity of massive amounts of structured data and images, the ability to handle high-concurrency requests, and the ability to process and return results with low latency. Simultaneously, application platforms and analysis systems will initiate registration, query, subscription, and data upload operations to the view library through general interfaces, data service interfaces, and analysis interfaces, placing significant pressure on the view library's flexibility, stability, and scalability in terms of external interaction.

[0004] To address these needs, the industry has proposed various view library implementation solutions. For example, some solutions adopt a direct data processing approach: the data acquisition device access module simultaneously handles device registration, keep-alive operations, and the parsing, format conversion, and writing of structured data and images. This type of solution can work when the workload is small, but when structured data and images are frequently reported and the concurrency is high, the access module has to maintain interaction with a large number of devices while executing complex parsing and storage logic. This can easily lead to the processing pressure being concentrated on a single service, resulting in problems such as excessive resource consumption, slow response, or even service anomalies.

[0005] Some solutions attempt to break down modules according to interfaces or business functions, deploying functions such as upper-level services, lower-level services, alarm collection, access, and querying in a distributed manner as microservices. They also introduce components such as message queues, caches, and intermediate databases for data transmission, information synchronization, and service management, thus achieving layering and distribution in form. However, in these solutions, strong dependencies often remain between modules. Structured data flows multiple times between various services, sharing some storage resources or cache space, resulting in long links, high coupling, and requiring coordinated adjustments to multiple modules during expansion. Furthermore, flexibility is limited when integrating with different external systems.

[0006] In terms of data storage, traditional implementations often employ the approach of deploying business processing services and databases on the same machine, and uniformly storing structured data and images in the database. All reported structured data and its associated images are centrally written to a relational database or database-like system. During peak collection periods or centralized deployment scenarios, a large number of write requests can consume disk and memory resources, easily causing database write delays, slower query responses, and even affecting the availability and stability of other core services within the view library that depend on the database (such as query services, cascading services, etc.).

[0007] Furthermore, existing technologies have proposed view library architectures based on big data platforms or combinations of various middleware. For example, some solutions build view library systems based on the Hadoop architecture, introducing distributed file storage, real-time computing, and full-text search components to support the storage, analysis, and retrieval of massive amounts of video images and structured data; other solutions, while meeting the GA / T1400 industry standard, use Kafka and Redis as intermediate data processing units, combined with columnar databases to achieve data ingestion and querying under high concurrency; still other solutions, in scenarios such as geographic video, utilize in-memory databases and distributed file systems to organize and manage video data, thereby improving real-time analysis and data scheduling capabilities.

[0008] However, the existing technical solutions mentioned above still have some prominent problems in scenarios with massive structured data: First, the boundaries of responsibility between the access layer and the data processing and storage layers are not clear enough. The access module is often responsible for both device access control and some data parsing and database writing tasks, resulting in high pressure on a single service. When faced with large-scale concurrent access and structured data reporting, stability and latency performance are limited. Second, the coupling between modules is high. Structured data is repeatedly transferred or shares resources between multiple services, resulting in insufficient scalability and on-demand tailoring capabilities, and high costs for connecting to different platforms or analysis systems. Third, structured data and images are directly co-stored in databases or fixed underlying storage, lacking a layered management mechanism for the characteristics of structured fields and image resources, which can easily lead to resource contention and performance degradation during peak periods. Fourth, the elastic scaling capability for dynamic business loads is limited. There is a lack of fine-grained scaling and load-sharing strategies for the structured data processing link, making it difficult to adapt to significant fluctuations in the number of devices and the amount of data pushed in a timely and stable manner.

[0009] Against this backdrop, how to collaboratively design the access, data transmission, processing, and storage processes in the view library while adhering to the GA / T1400 standard, so as to alleviate the pressure on single-point services, reduce module coupling, optimize the hierarchical storage of structured data and images, and improve low-latency processing capabilities and scalability under high concurrency conditions, has become one of the urgent technical problems to be solved in this field. Summary of the Invention

[0010] The purpose of this invention is to address the problems faced by existing view libraries in scenarios involving the access service pressure, high module coupling, severe resource contention due to the co-storage of structured data and images, and the lack of fine-grained elastic scaling mechanisms for dynamic business loads. This invention provides a method for low-latency processing and dynamic scaling of massive structured data in view libraries. By coordinating the design of access, data transmission, processing, and storage processes within the view library, and adhering to the GA / T1400 series standards, this method achieves low-latency processing of multi-source structured data, hierarchical management of massive image resources, and on-demand expansion of processing capabilities, thereby improving the stability and scalability of the view library in large-scale deployment scenarios.

[0011] One objective of this invention is to decouple the responsibilities of the acquisition device / system access module, the upper and lower level view library cascading module, and the module for connecting to third-party acquisition systems and application platforms. This allows the access layer to primarily handle control and channel management functions such as device registration, keep-alive, deregistration, time synchronization, and data forwarding. Computationally intensive tasks such as parsing, storing, and image storage of structured data are offloaded from the access layer to a dedicated data processing module. This alleviates the pressure on the access service under high-concurrency scenarios from the source, and improves the stability of device access and the reliability of external interactions.

[0012] Another objective of this invention is to construct a structured data processing link with a message middleware as the link. The structured data reported by the acquisition device / system, the upper and lower level view libraries, and the analysis device / system are uniformly written into the middleware. A dedicated processing service retrieves the data from the middleware and performs operations such as parsing, field extraction, core field storage, and writing image resources into object storage. This forms a pipelined processing architecture of access—message transmission—centralized processing—layered storage, which improves the overall throughput of the system and shortens the time from receiving to processing a single piece of structured data.

[0013] Another objective of this invention is to address the differences in capacity, access patterns, and performance requirements between structured data records and image resources by employing a layered storage approach that combines structured data sharding and image object storage. This approach allows the database to primarily manage structured fields and image identification information, while the original image data is stored in a storage system designed for massive objects. This avoids storage bloat and I / O contention issues caused by directly storing structured data and images in the same database, thereby improving the overall scalability and long-term operational stability of the storage system while ensuring query and retrieval efficiency.

[0014] Another objective of this invention is to dynamically scale up and down the structured data processing service by combining operational metrics such as the number of devices connected to the view library, the amount of structured data reported, and the scale of data to be processed in the message middleware. When the business load increases, the processing instances are expanded on demand, and when the business load decreases, the processing instances are appropriately recycled. During the scaling up and down process, the data processing load is reasonably allocated and migrated, so that the view library can maintain relatively stable processing latency and resource utilization under different business scales, and realize elastic processing capabilities for massive structured data scenarios.

[0015] By achieving the above objectives, this invention can effectively reduce the pressure on single-point services of the view library, weaken the coupling between modules, and improve the storage and access efficiency of structured data and image resources, while complying with the GA / T1400 series standards. It can also maintain the stable operation of the view library system and low processing latency even when the number of devices and the amount of data pushed change significantly.

[0016] To achieve the above objectives, the technical solution adopted by this invention is as follows: a method for low-latency processing and dynamic scaling of massive structured data in a view library, applied to processing structured data and its associated image data reported by acquisition devices / systems, comprising the following steps: (1) Decoupling of access and processing: The data acquisition-side access service only performs registration, keep-alive, time synchronization, and access and transparent transmission of structured data for the acquisition devices / systems. It does not perform parsing, storage, or retrieval calculations of structured data and images on the access side. When there is no need to directly push the data to a third-party system according to preset rules, it performs integrity verification on the received structured data and writes the verified structured data and its image identifier into the message channel of the message middleware, so that the data acquisition-side access service mainly undertakes connection control and channel management functions. (2) Asynchronous forwarding and centralized processing: The processing service consumes the structured data from the message channel, parses the structured data according to the predetermined protocol specifications, extracts the core fields for business queries, writes the core fields into the database, and uploads the corresponding image data to the cloud / object storage. The database records the mapping relationship between image identifiers and cloud / object storage addresses, so that the access process and the data processing and storage process are asynchronously decoupled through message middleware.

[0017] (3) Tiered storage: Structured data and regular business data are managed in separate databases, and structured data is divided into tables according to object type. The database only stores the core fields and the mapping information between image identifiers and cloud / object storage addresses. The original image data is stored independently in the cloud / object storage to avoid mixed storage of structured data and image data in the same database, thereby reducing the input / output and storage pressure of the database.

[0018] (4) Dynamic expansion and contraction: Based on the number of data collection devices and the amount of structured data pushed, the processing service is deployed in multiple instances and scaled up or down. When scaling up, processing service instances are added according to the observed business load, and subscription partitions in the message channel are allocated to the new instances. When scaling down, the messages to be processed are migrated to the subscription partitions corresponding to the retained instances before the processing service instances are shut down. The message consumption allocation relationship is adjusted after the addition or removal of instances so as to ensure that message consumption is not interrupted and the resource utilization of each instance is basically balanced when the business load changes.

[0019] (5) External interaction decoupling and pipelined processing: Independent services that interface with upper and lower level view libraries and third-party platforms provide registration, subscription or push, query and data upload capabilities. Within the view library, these independent services only interact with the message middleware and database or cloud / object storage, without handling structured data parsing and database entry calculations. This decouples external interaction from the access, processing and storage processes, forming a horizontally scalable pipeline processing channel consisting of access services, message middleware, processing services and hierarchical storage, for processing massive amounts of structured data and their associated images.

[0020] Furthermore, the independent services that interface with upper and lower level view libraries and third-party platforms include: a 1400DAS module for device / system registration, keep-alive, deregistration, time synchronization, and structured data reporting; a GAC ​​module for data push, registration, keep-alive, deregistration, and time synchronization to third-party view libraries; a GAS module for subscription and registration processing to third-party view libraries; and interface acquisition system services and interface application platform services for connecting to third-party acquisition systems and application platforms and providing registration, keep-alive, deregistration, time synchronization, data query, and / or data upload functions, respectively. The above modules are deployed decoupled from the processing services.

[0021] Furthermore, the registration and keep-alive interaction process between GAS and GAC includes: (1) A cascading channel is established by one party sending a registration request to the other party, the receiving party completing the registration process and returning a registration response; (2) After successful registration, the data push side sends a keep-alive message according to the preset keep-alive period, and the receiving side returns a keep-alive response to maintain the channel validity; (3) Time synchronization should be performed as needed during cascaded operation to maintain consistency of time base between the two parties; (4) When it is necessary to cancel the cascade channel, initiate a cancellation request. After the receiving side completes the cancellation process, it returns a cancellation response and closes the cascade channel.

[0022] Furthermore, the acquisition-side access service is a 1400DAS module, and its specific process for executing structured data access and transparent transmission includes: (1) Receive access requests initiated by the data acquisition device / system and parse them into registration, keep-alive, or time synchronization requests; (2) After verifying the identity of the registration request and completing the entry of the device information into the database, return a registration success response; for the keep-alive request, update the online status of the device and return a keep-alive confirmation response; for the time synchronization request, push the standard time to the acquisition device / system to complete the time synchronization. (3) Receive the structured data of vehicles, faces or bodies and related image data reported by the data collection equipment / system, and perform basic integrity verification; (4) Determine whether the structured data needs to be pushed to a third-party system: if needed, push to a preset third-party storage address; if not needed, perform pass-through and write the structured data and associated image identifiers into the message channel of the message middleware.

[0023] Furthermore, the processing flow for data sent or pushed from the parent view library or third-party systems includes: (1) The message channel of the message middleware is used to register, verify the permissions and format of the data to be sent or pushed; (2) When the data object is device information, perform device information entry into the database; (3) When the data object is structured data and associated image data, the data is pushed to the processing service for unified parsing and storage, and the image is uploaded to the cloud / object storage and the image identifier is mapped to the storage address.

[0024] Furthermore, the processing service is the AlarmCollect service, whose process for consuming and processing structured data includes: (1) During initialization, subscribe to the topic corresponding to the structured data in the message middleware and configure a message consumption thread or thread pool to concurrently pull and consume messages; (2) Pull structured data from the message channel and parse it according to the GA / T1400 protocol specification to extract the core fields of objects such as vehicles, faces, and bodies; (3) Validate the core fields and write the validated data into the structured data database after database sharding and table partitioning; (4) Extract the image identifier from the structured data, call the upload interface of the cloud / object storage to upload the image data, obtain the storage address after uploading, and write the mapping relationship between the image identifier and the storage address into the database; (5) When it is necessary to trigger the synchronization of the upper and lower level view libraries, send a data push trigger signal to the GAC module or GAS module.

[0025] Furthermore, the implementation of the hierarchical storage includes: managing the 1400 structured data and regular business data in separate databases, and dividing the 1400 structured data into tables according to object type; the database stores the mapping information between the core structured fields used for querying and retrieval, as well as image identifiers and cloud / object storage addresses, to support retrieval based on structured fields and on-demand retrieval of image data from the origin cloud / object storage.

[0026] Furthermore, the message middleware and cloud / object storage are configured as follows: the message middleware uses Kafka, a distributed message queue that supports topic and partitioning mechanisms, and its message channels are configured with topics and partitions according to business needs to achieve parallel consumption and load balancing of structured data; the cloud / object storage uses OBS object storage server to store image resources and their feature or attribute data files associated with the structured data.

[0027] Furthermore, the dynamic expansion and contraction includes: (1) Set expansion and reduction trigger conditions based on observation indicators such as the number of online acquisition devices and the amount of structured data pushed; (2) When any observation metric meets the expansion trigger condition, a processing service instance is automatically added, and a corresponding Kafka topic partition subscription is assigned to the new instance to share the structured data to be processed; (3) When the observed index meets the scaling down trigger condition, the number of processing service instances is automatically reduced. Before scaling down, the messages to be processed in the partition corresponding to the instance to be scaled down are migrated to the subscription partition corresponding to the retained instance. After the migration is completed, the message subscription of the instance to be scaled down is closed. (4) During the scaling up and down process, message consumption is kept uninterrupted, and the message allocation relationship of each instance is automatically updated after the instance is added or removed, so as to achieve dynamic balance of message consumption load.

[0028] Furthermore, the third-party platform integration includes integration with the data collection system service and integration with the application platform service. Specifically: the data collection system service receives and verifies registration requests from the analysis or data collection system to complete system information entry into the database; receives data query requests from the analysis or data collection system and transmits them to the data query service to obtain query results before returning them; and receives data upload requests from the analysis or data collection system and forwards the data to the message channel of the message middleware. The application platform service receives registration requests and query conditions from the application platform, transmits the query conditions to the data query service and returns query results; receives alarm subscription requests from the application platform, configures alarm push rules to the processing service, and triggers alarm pushes accordingly; and when image content needs to be returned, retrieves the image from the source cloud / object storage based on the image identifier in the database and the cloud / object storage address mapping, and then returns it.

[0029] The present invention adopts the above technical solution, and the specific beneficial effects are as follows: In this invention, the registration, keep-alive, time synchronization, and structured data reporting of the acquisition devices / systems are centrally handled by the acquisition-side access service. Furthermore, the access layer explicitly avoids parsing, storing, or retrieving structured data and images. It only performs integrity verification on the received structured data and writes it to the message middleware when it doesn't need to be pushed to a third-party system according to preset rules. This allows the access layer to primarily handle connection control and channel management. This reduces the computational and storage burden on the access service, avoiding the excessive pressure on a single service caused by the access module simultaneously handling access and complex data processing tasks in existing technologies. It also helps maintain the stability of device access and external interaction in scenarios with a large number of concurrent devices accessing and continuously reporting structured data.

[0030] In this invention, a message middleware is introduced as a unified channel for transmitting structured data. Structured data reported by the acquisition-side access service, upper and lower level view libraries, and third-party systems are uniformly written into the message middleware. An independently deployed processing service retrieves data from the message middleware and completes operations such as protocol parsing, core field extraction, core field storage, and image data upload and object storage. This decouples the access process from the data processing and storage processes. This processing chain makes the flow path of structured data within the system clearer, reduces repeated transmission between multiple business modules, lowers the dependency between modules, improves overall processing throughput, and shortens the time from receiving to storing structured data, thereby improving the response time of the view library in scenarios with massive amounts of structured data.

[0031] In this invention, structured data and business data are managed in separate databases, and the structured data is further divided into tables according to object type. The database primarily stores core fields used for querying and retrieval, as well as the mapping information between image identifiers and object storage addresses. Simultaneously, the original image data is stored independently in an object storage system, achieving hierarchical management of structured data and image resources. This storage method avoids the storage space expansion and input / output resource contention problems caused by directly storing structured data and large volumes of images in the same database. It reduces the pressure on the database during high-concurrency writes and queries, and leverages the scalability advantages of object storage in terms of capacity and number of objects while ensuring the performance of structured queries. This supports the long-term stable storage of large-scale image resources and on-demand retrieval from the source.

[0032] This invention combines operational metrics such as the number of online acquisition devices and the volume of structured data pushed to implement multi-instance deployment and dynamic scaling control for the structured data processing service. When business load increases, processing service instances are added and assigned subscription partitions. When business load decreases, pending messages are migrated and instances are reclaimed before shutdown. Coupled with an architecture design where independent services interfacing with upper- and lower-level view libraries and third-party platforms interact only with message middleware and databases or object storage, scaling operations are largely transparent to the access layer and external interfaces. This mechanism allows for flexible adjustment of processing capacity based on changes in business scale, ensuring relatively stable processing latency and resource utilization under different load levels. It also reduces the difficulty of on-demand expansion, partitioned deployment, and optimized resource allocation during engineering deployment, improving the overall scalability and ease of operation and maintenance of the view library system. Attached Figure Description

[0033] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0034] Figure 1 This is a flowchart of the low-latency processing and dynamic scaling of massive structured data in the view library of this invention.

[0035] Figure 2 This is a schematic diagram of the multi-terminal interaction and data processing architecture of the view library of this invention.

[0036] Figure 3 This is a diagram illustrating the interaction architecture between the multi-terminal system and the view library of this invention.

[0037] Figure 4 This is a flowchart illustrating the registration process between GAS and GAC in this invention. Detailed Implementation

[0038] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numerals in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present invention. Rather, they are merely examples of apparatuses and methods consistent with some aspects of the invention as detailed in the appended claims.

[0039] Example 1

[0040] like Figure 1 As shown, this embodiment provides a method for low-latency processing and dynamic scaling of massive structured data in a view library, applied to processing structured data and its associated image data reported by acquisition devices / systems, including the following steps: (1) Decoupling of access and processing: The access service on the acquisition side only performs the registration, keep-alive, time synchronization and access and transparent transmission of structured data of the acquisition device / system, and does not perform the parsing, storage or retrieval calculation of structured data and images on the access side; when there is no need to push directly to the third-party system according to the preset rules, the integrity of the received structured data is verified, and the verified structured data and its image identifier are written into the message channel of the message middleware, so that the access service on the acquisition side mainly undertakes the connection control and channel management functions; (2) Asynchronous forwarding and centralized processing: The processing service consumes the structured data from the message channel, parses the structured data according to the predetermined protocol specification, extracts the core fields for business query, writes the core fields into the database, and uploads the corresponding image data to the cloud / object storage. The mapping relationship between the image identifier and the cloud / object storage address is recorded in the database, so that the access process and the data processing and storage process are asynchronously decoupled through the message middleware.

[0041] (3) Layered storage: Structured data and regular business data are managed in separate databases, and structured data is divided into tables according to object type. The database only stores the mapping information of the core fields and image identifiers to the cloud / object storage address. The original image data is stored independently in the cloud / object storage to avoid mixed storage of structured data and image data in the same database, thereby reducing the input / output and storage pressure of the database.

[0042] (4) Dynamic scaling up and down: Based on the number of collection devices and the amount of structured data pushed, the processing service is deployed in multiple instances and scaled up or down. When scaling up, the processing service instance is added according to the observed business load, and the subscription partition in the message channel is allocated to the new instance. When scaling down, the pending messages of the processing service instance are migrated to the subscription partition corresponding to the retained instance before the processing service instance is shut down. After the instance is added or removed, the message consumption allocation relationship is adjusted so as to keep message consumption uninterrupted and the resource utilization of each instance basically balanced when the business load changes.

[0043] (5) Decoupling of external interaction and pipelined processing: The independent service that interfaces with the upper and lower level view libraries and third-party platforms provides registration, subscription or push, query and data upload capabilities. The independent service only interacts with the message middleware and database or cloud / object storage within the view library, without carrying out structured data parsing and database calculation. This decouples the external interaction from the access, processing and storage processes, forming a horizontally scalable pipelined processing channel consisting of access service, message middleware, processing service and hierarchical storage, which is used to process massive structured data and its associated images.

[0044] This embodiment connects the acquisition side access, message transmission, structured data parsing, and hierarchical storage sequentially through the above steps, and introduces dynamic scaling capability into the processing link. This allows the view library to maintain the stability of access and external interaction when facing the expansion of device scale and changes in the amount of structured data pushed, while also being able to flexibly adjust its processing capacity and storage architecture, thus making it suitable for various scales and scenarios in engineering deployment.

[0045] As one implementation method, the independent services that interface with upper and lower level view libraries and third-party platforms in this embodiment include: a 1400DAS module for device / system registration, keep-alive, deregistration, time synchronization, and structured data reporting; a GAC ​​module for data push, registration, keep-alive, deregistration, and time synchronization to third-party view libraries; a GAS module for subscription and registration processing to third-party view libraries; and interface acquisition system services and interface application platform services for interfacing with third-party acquisition systems and application platforms, respectively, and providing registration, keep-alive, deregistration, time synchronization, data query, and / or data upload functions. The above modules are deployed decoupled from the processing services.

[0046] In this implementation, by deploying 1400DAS, GAC, GAS, and the interfacing data acquisition system service and application platform service as independent interfacing modules, different types of external systems can complete registration, keep-alive, and data interaction through their respective modules. Each module only needs to focus on its corresponding interface protocol and business logic, reducing the need for a single module to handle multiple interfaces simultaneously. On the one hand, this decomposition approach facilitates customized adaptation for different external platforms, reducing the modification costs when expanding to new platforms or systems. On the other hand, changes and maintenance needs of external systems do not need to directly affect the structured data processing service, which helps maintain the stability of the data processing chain.

[0047] As one implementation method, the registration and keep-alive interaction process between GAS and GAC in this embodiment includes: (1) A cascading channel is established by one party sending a registration request to the other party, the receiving party completing the registration process and returning a registration response; (2) After successful registration, the data push side sends a keep-alive message according to the preset keep-alive period, and the receiving side returns a keep-alive response to maintain the channel validity; (3) Time synchronization should be performed as needed during cascaded operation to maintain consistency of time base between the two parties; (4) When it is necessary to cancel the cascade channel, initiate a cancellation request. After the receiving side completes the cancellation process, it returns a cancellation response and closes the cascade channel.

[0048] In this implementation, by clearly defining the registration, keep-alive, time synchronization, and deregistration interaction processes between GAS and GAC, the cascading relationship between upper and lower level view libraries has clear message exchanges and processing steps at each stage of establishment, maintenance, and release. This allows for timely detection and handling of connection anomalies when network conditions change or system maintenance occurs. Periodic keep-alive and on-demand time synchronization reduce the risk of seemingly online but actually invalid cascading links after prolonged periods, ensuring good continuity and time consistency of structured data and images during transmission between upper and lower level view libraries.

[0049] As one implementation method, the acquisition-side access service in this embodiment is a 1400DAS module, and its specific process for performing structured data access and transparent transmission includes: (1) Receive access requests initiated by the data acquisition device / system and parse them into registration, keep-alive, or time synchronization requests; (2) After verifying the identity of the registration request and completing the entry of the device information into the database, return a registration success response; for the keep-alive request, update the online status of the device and return a keep-alive confirmation response; for the time synchronization request, push the standard time to the acquisition device / system to complete the time synchronization. (3) Receive the structured data of vehicles, faces or bodies and related image data reported by the data collection equipment / system, and perform basic integrity verification; (4) Determine whether the structured data needs to be pushed to a third-party system: if needed, push to a preset third-party storage address; if not needed, perform pass-through and write the structured data and associated image identifiers into the message channel of the message middleware.

[0050] In this embodiment, the 1400DAS module integrates device access control with the integrity verification, push, or pass-through of structured data into its processing flow. This allows acquisition devices / systems to complete registration, keep-alive, time synchronization, and data reporting simply by maintaining a stable connection with the 1400DAS. By performing basic integrity verification on the access side, data with incorrect formatting or incomplete content can be filtered out before entering the message middleware, reducing the burden on subsequent processing services and storage systems. Furthermore, by distinguishing between push to third-party systems and pass-through to the message middleware, the data flow direction can be flexibly selected according to actual business needs, improving the flexibility of data interaction.

[0051] As one implementation method, the data processing flow for data sent or pushed from a higher-level view library or a third-party system in this embodiment includes: (1) The message channel of the message middleware is used to register, verify the permissions and format of the data to be sent or pushed; (2) When the data object is device information, perform device information entry into the database; (3) When the data object is structured data and associated image data, the data is pushed to the processing service for unified parsing and storage, and the image is uploaded to the cloud / object storage and the image identifier is mapped to the storage address.

[0052] In this implementation, data from the upper-level view library or third-party systems undergoes unified registration, permission, and format verification. This prevents data from unknown sources or with insufficient permissions from directly entering the system's core processes, enhancing the overall data security and reliability of the view library. Separate processing steps are defined for device information and structured data. This approach helps maintain consistent management of device information and data content. Furthermore, by uniformly entrusting structured data and images to the processing service for parsing and database storage, a unified data standard and storage specification are established within the system, reducing inconsistencies caused by repeated parsing and database writing.

[0053] As one implementation method, the processing service described in this embodiment is the AlarmCollect service, whose process for consuming and processing structured data includes: (1) During initialization, subscribe to the topic corresponding to the structured data in the message middleware and configure a message consumption thread or thread pool to concurrently pull and consume messages; (2) Pull structured data from the message channel and parse it according to the GA / T1400 protocol specification to extract the core fields of objects such as vehicles, faces, and bodies; (3) Validate the core fields and write the validated data into the structured data database after database sharding and table partitioning; (4) Extract the image identifier from the structured data, call the upload interface of the cloud / object storage to upload the image data, obtain the storage address after uploading, and write the mapping relationship between the image identifier and the storage address into the database; (5) When it is necessary to trigger the synchronization of the upper and lower level view libraries, send a data push trigger signal to the GAC module or GAS module.

[0054] In this implementation, the AlarmCollect service achieves continuous processing capabilities for large amounts of structured data records through subscription and concurrent consumption of structured data topics. It utilizes parsing logic compliant with the GA / T1400 standard to ensure that the extracted core fields conform to industry standards. The core fields are validated before being written to the sharded structured database, helping to guarantee the quality of the imported data and the accuracy of subsequent queries. Image data is mapped to storage addresses while being uploaded to object storage, allowing subsequent application platforms to quickly locate the corresponding images using structured fields, achieving linkage between structured retrieval and image source retrieval. Furthermore, when synchronization between upper and lower level view libraries is required, the service initiates a trigger signal, coordinating data processing and cascading synchronization operations within the same processing chain.

[0055] As one implementation method, the hierarchical storage described in this embodiment includes: managing 1400 structured data and regular business data in separate databases, and dividing the 1400 structured data into tables according to object type; the database stores the core structured fields used for querying and retrieval, as well as image identifiers and cloud / object storage address mapping information, to support retrieval based on structured fields and on-demand retrieval of image data from the origin cloud / object storage.

[0056] In this implementation, by placing the 1400 structured data and other business data in separate databases, and further partitioning the 1400 data based on object type, different types of data can be optimized in terms of index design, table structure adjustment, and resource allocation. This avoids the management complexity and performance interference caused by mixing multiple types of business data in the same data table. The database only stores the mapping information between structured fields and image identifiers and storage addresses, so that the storage burden of the database is mainly concentrated on the structured records themselves. Transferring the large volume of image data to object storage enhances the overall performance of the system in structured retrieval and image access.

[0057] As one implementation method, the message middleware and cloud / object storage configuration in this embodiment are as follows: the message middleware uses Kafka, a distributed message queue that supports topic and partitioning mechanisms, and its message channels are configured with topics and partitions according to business needs to achieve parallel consumption and load sharing of structured data; the cloud / object storage uses OBS object storage server to store image resources and their feature or attribute data files associated with structured data.

[0058] In this implementation, a distributed message queue supporting topics and partitioning mechanisms is used to handle the transmission of structured data. This allows data from different categories or sources to be isolated through different topics, and load balancing among multiple instances is achieved through partitioning, thereby improving overall message processing capabilities and reducing the pressure on a single instance. The object storage server is responsible for hosting image resources and related files associated with the structured data. Leveraging its capabilities in large-capacity storage and object expansion, it can long-term store large-scale image data and provide stable access services to upper-layer applications without significantly impacting the main database of the view library.

[0059] As one implementation method, the dynamic expansion and contraction described in this embodiment includes: (1) Set expansion and reduction trigger conditions based on observation indicators such as the number of online acquisition devices and the amount of structured data pushed; (2) When any observation metric meets the expansion trigger condition, a processing service instance is automatically added, and a corresponding Kafka topic partition subscription is assigned to the new instance to share the structured data to be processed; (3) When the observed index meets the scaling down trigger condition, the number of processing service instances is automatically reduced. Before scaling down, the messages to be processed in the partition corresponding to the instance to be scaled down are migrated to the subscription partition corresponding to the retained instance. After the migration is completed, the message subscription of the instance to be scaled down is closed. (4) During the scaling up and down process, message consumption is kept uninterrupted, and the message allocation relationship of each instance is automatically updated after the instance is added or removed, so as to achieve dynamic balance of message consumption load.

[0060] In this implementation, by using operational metrics such as the online status of acquisition devices and the volume of structured data pushed as the basis for scaling up and down decisions, the system can promptly add processing service instances when the workload increases significantly and reclaim some instances when the workload decreases, avoiding resource waste and latency fluctuations caused by long-term overcapacity or undercapacity. By migrating pending messages before scaling down and updating message allocation relationships after instance additions or removals, the impact of scaling up and down operations on current processing tasks can be reduced, allowing the view library to maintain the continuity of message consumption and the integrity of processing results while dynamically adjusting processing resources.

[0061] As one implementation method, the third-party platform integration in this embodiment includes integration with a data collection system service and integration with an application platform service. Specifically: the data collection system service receives and verifies registration requests from the analysis or collection system to complete system information entry into the database; receives data query requests from the analysis or collection system and transmits them to the data query service to obtain query results before returning them; and receives data upload requests from the analysis or collection system and forwards the data to the message channel of the message middleware. The application platform service receives registration requests and query conditions from the application platform, transmits the query conditions to the data query service and returns query results; receives alarm subscription requests from the application platform, configures alarm push rules to the processing service, and triggers alarm pushes accordingly; and when image content needs to be returned, retrieves the image from the source cloud / object storage based on the image identifier and cloud / object storage address mapping in the database and returns it.

[0062] In this implementation, the data acquisition system service and the application platform service are designed for different types of external systems. The former primarily handles data uploads and query requests, while the latter primarily handles business queries and alarm subscriptions. By uniformly passing external requests to the data query and processing services within the view library, the consistency of the view library's internal data structure and processing flow can be maintained, while providing relatively fixed and clear interface capabilities to the outside world. When the application platform needs to return image content, it retrieves the image from the source using the mapping relationship between image identifiers and object storage addresses in the database. This allows for the provision of complete text and image data to the upper-layer system without exposing the underlying storage details, satisfying business display needs while also protecting the underlying storage system.

[0063] Based on relevant industry standards, this embodiment provides specific implementation methods for key aspects such as data collection and access, message transmission, centralized processing, hierarchical storage, and dynamic scaling. Through module decoupling and division of responsibilities, the view library can maintain low processing latency and high system stability under the concurrent reporting of massive amounts of structured data and images. At the same time, it retains the adaptability to different external platforms and business scenarios, and has good engineering application value.

[0064] Example 2

[0065] This embodiment provides a low-latency processing and dynamic scaling system for massive structured data in a view library, used to implement the method described in Embodiment 1. Deployed in a server cluster, the system is suitable for processing structured data and its associated image data reported by acquisition devices or systems. The system comprises: an acquisition-side access service module, a message middleware module, a structured data processing module, a hierarchical storage module, an external interaction module, a dynamic scaling control module, and a data query service module. These modules communicate and cooperate with each other through a network and internal interfaces to complete the access, transmission, processing, and storage of massive structured data.

[0066] In the system, the acquisition-side access service module establishes connections with front-end cameras, acquisition systems, or analysis systems, and is responsible for device registration, keep-alive, time synchronization, and access control for structured data reporting. The message middleware module receives structured data messages written from the acquisition-side access service module and the external interaction module, providing buffering and asynchronous forwarding capabilities for structured data processing. The structured data processing module retrieves structured data from the message middleware module and performs operations such as protocol parsing, core field extraction, and image uploading. The hierarchical storage module includes a structured data database and object storage, used to store structured fields and image resources respectively. The external interaction module is used for registration, subscription, push, query, and data upload interactions with upper and lower level view libraries, third-party acquisition systems, and application platforms. The dynamic scaling control module adjusts the number of instances of the structured data processing module according to operating indicators. The data query service module provides a unified query interface based on structured fields and image resources for application platforms and analysis systems.

[0067] In this embodiment, the cooperation relationship between the various modules of the system corresponds to the method steps described in Embodiment 1: the acquisition-side access service module undertakes the responsibility of access and transparent transmission, the message middleware module undertakes the responsibility of asynchronous forwarding, the structured data processing module undertakes the responsibility of centralized processing, the hierarchical storage module realizes the hierarchical management of structured data and image resources, the external interaction module provides a unified interface with external systems, the dynamic expansion and contraction control module flexibly adjusts the processing capacity, and the data query service module provides a unified data access entry for upper-layer businesses.

[0068] In one implementation, the acquisition-side access service module may include a 1400 data access service submodule for registration, keep-alive, deregistration, time synchronization, and structured data reporting of acquisition devices or systems. This submodule receives access requests initiated by acquisition devices or systems and parses them into registration, keep-alive, or time synchronization types. In the registration scenario, it verifies the identity of the access entity and writes device information into the device information database; in the keep-alive scenario, it updates the online status; and in the time synchronization scenario, it returns the standard time to the acquisition device or system. In the structured data reporting scenario, it performs basic integrity checks on the reported structured data and its associated image data. Data that needs to be directly pushed to third-party systems is pushed through the external interaction module; data that needs to be processed within the system is written to the message middleware module along with its image identifier. Through this design, the acquisition-side access service module primarily focuses on connection management and data entry control, avoiding complex data parsing and database entry operations at the access layer, which helps reduce the load on the access service when there is a large number of concurrent device accesses.

[0069] As one implementation method, the message middleware module can employ a distributed message queue system that supports topics and partitioning mechanisms. This module configures multiple topics to respectively carry structured data from different sources or different business types, and distributes data of the same type across multiple partitions through a partitioning mechanism for concurrent consumption by multiple structured data processing instances. The message middleware module provides data write interfaces to the acquisition-side access service module and the external interaction module, and provides data pull interfaces to the structured data processing module. By introducing the message middleware module into the system, a buffer layer can be formed between the acquisition-side access service module and the structured data processing module, allowing the access rate and processing rate to be decoupled within a certain range. This improves the overall system's adaptability to short-term peak traffic and reduces service anomalies caused by instantaneous load fluctuations.

[0070] In one implementation, the structured data processing module may include an alarm processing service submodule and a corresponding thread management unit. During initialization, this module subscribes to one or more topics corresponding to the structured data from the message middleware module and allocates a corresponding number of consumption threads to each topic or group partition according to the configuration. After pulling structured data from the message middleware module, the structured data processing module parses the structured data according to industry standards, extracts core fields of objects such as vehicles, faces, and bodies, and performs basic verification on the parsing results. For data that passes verification, the core fields are written to the structured data database, and image identifiers are extracted from the structured data. The image data is then written by calling the object storage upload interface, while simultaneously recording the correspondence between the image identifier and the object storage address. When data synchronization with upper and lower level view libraries is required, the structured data processing module can also send a trigger signal to the external interaction module, notifying the external interaction module to push the corresponding data to the target system. Through this centralized processing method, the structured data processing module can provide services for structured data from different sources under a unified parsing and storage logic, reducing redundant implementation within the system and improving data processing throughput by combining a multi-threaded consumption mechanism.

[0071] As one implementation method, the hierarchical storage module may include a structured data database submodule and an object storage submodule. The structured data database submodule stores the core fields of 1400 structured data entries, as well as the mapping information between image identifiers and object storage addresses, and is deployed separately from other business data databases. Within the structured data database, structured records are stored in different data tables according to object types such as vehicles, faces, and bodies, facilitating subsequent indexing and query optimization for different object types. The object storage submodule stores image files associated with the structured data and their feature or attribute data files, and provides interfaces for writing and reading based on image identifiers. By storing structured records and image resources in the structured data database and object storage respectively, the storage bloat and input / output bottlenecks caused by directly storing large amounts of image data in relational databases can be avoided. Simultaneously, the advantages of object storage in terms of capacity expansion support the long-term storage of massive amounts of images.

[0072] As one implementation method, the external interaction module may include a sub-module for interfacing with acquisition systems and acquisition devices, a sub-module for interfacing with application platforms, and a sub-module for cascading with upper and lower level view libraries. The cascading service sub-module may internally include service units for pushing data to, registering, keeping alive, and synchronizing time with upper or lower level view libraries, as well as service units for initiating subscription and registration processes with the other party. The sub-module for interfacing with acquisition systems receives registration requests from analysis systems or acquisition systems and completes system information entry after verification; receives data query requests and forwards query conditions to the data query service module; receives data upload requests and writes the uploaded content to the message middleware module. The sub-module for interfacing with application platforms receives registration requests and query conditions from application platforms, forwards query conditions to the data query service module and returns query results to the application platform; and writes alarm configurations to the structured data processing module or related configuration storage when the application platform initiates an alarm subscription request. By assigning the interaction needs of different external entities to dedicated sub-modules, this system can flexibly provide interface capabilities for different upper-layer applications and platforms while maintaining the stability of the internal processing chain.

[0073] As one implementation method, the dynamic scaling control module can be used to monitor indicators such as the number of online acquisition devices, the volume of structured data pushed, the scale of data to be processed in the message middleware, and the operating status of the structured data processing module. This module can adjust the number of instances of the structured data processing module during operation according to preset scaling strategies: when the volume of structured data pushed is continuously increasing and the current processing capacity is approaching a predetermined limit, the dynamic scaling control module can increase the number of instances of the structured data processing module and allocate corresponding message partitions for consumption to the new instances; when the load is continuously decreasing over a certain period, the number of instances of the structured data processing module can be gradually reduced, and the data to be processed in the message partitions of the retained instances can be migrated to the message partitions corresponding to the retained instances before the instances are reclaimed. Through this dynamic scaling control mechanism, the system can control the scale of processing resource usage while ensuring continuous processing of structured data, enabling the system to maintain reasonable resource utilization and processing latency under both high and low load conditions.

[0074] As one implementation method, the data query service module can be deployed as a service independent of the structured data processing module. It receives query requests forwarded from the data acquisition system service submodule or the application platform service submodule, parses the query conditions, and retrieves records that meet the conditions from the structured data database. When image content needs to be returned, the data query service module can retrieve the corresponding image file from the object storage submodule based on the image identifier in the structured data record, and then return the combined structured fields and image content to the requester. By separating the query service from the structured data processing module, the data query service can focus on query and result encapsulation logic, avoiding system performance fluctuations caused by the mutual influence of write and query loads. It also facilitates optimization for different upper-layer application query patterns.

[0075] The system described in this embodiment can be used in conjunction with the method described in Embodiment 1. Specifically, it uses a data acquisition-side access service module to handle the access and transparent transmission of structured data, a message middleware module to handle asynchronous forwarding, a structured data processing module to handle centralized parsing and data storage, a hierarchical storage module to manage structured records and image resources hierarchically, an external interaction module to connect with upper and lower level view libraries and various third-party systems, a dynamic scaling control module to dynamically adjust the number of processing instances, and a data query service module to provide unified query capabilities for upper-layer applications. Those skilled in the art will understand that adjustments to the module deployment method, calling order, and some implementation details, without changing the basic functions and interrelationships of the above modules, should be considered as falling within the system solution defined in this embodiment.

[0076] The low-latency processing and dynamic scaling system for massive structured data in the view library provided in Embodiment 2, through the collaborative design of modules such as acquisition-side access, message middleware, structured data processing, hierarchical storage, external interaction, dynamic scaling control, and data query, enables the system to ensure the stability of device access and external interfaces while improving the throughput of the data processing link and the scalability of the storage system under concurrent reporting of massive structured data and image resources. It also has the ability to elastically scale up and down in response to changes in business load. It is suitable as a system implementation scheme for the method described in Embodiment 1 and has the value of being protected as an independent system scheme.

[0077] To further illustrate Embodiment 1 and Embodiment 2, specific implementation examples are now provided. Those skilled in the art can read the following content as a separate embodiment, or consider it as a further supplement to the above embodiments.

[0078] In this embodiment, Figure 2This paper presents a complete architecture for the view library in scenarios involving local data collection, hierarchical cascading, and interaction with third-party systems. Logically, the system can be divided into a local data collection and processing area, and a hierarchical cascading and third-party interaction area. The two parts work together through message middleware and cascading services.

[0079] In the local acquisition and processing area, cameras, acting as front-end devices, are used to acquire live video streams and generate structured data and associated images of objects such as vehicles, faces, and bodies in the front-end or supporting analysis unit. The cameras report video and structured data to the video access module according to the agreed-upon protocol. The video access module can be implemented based on GB28181, Onvif, or a vendor's SDK. In this embodiment, it only handles control functions such as device registration, keep-alive, time synchronization, and data access and pass-through; it does not perform parsing, storage, or retrieval calculations of structured data and images on the access side. After performing basic integrity checks on the reported structured data, the video access module writes it to the message middleware Kafka, thereby decoupling the access side from subsequent processing. Media streams and related information from lower-level view libraries can also be accessed to the video access module through the GB28181 lower-level platform for unified management.

[0080] In this embodiment, Kafka is used as a massive message middleware for caching and forwarding structured data and alarm information. Structured data written by the video access module first enters Kafka and is then consumed asynchronously by the backend alarm processing service, AlarmCollect. Since the access layer is only responsible for writing to the message queue and does not directly call the processing service, a buffer is formed between the access layer and the processing layer. This effectively avoids link blocking and single-point performance bottlenecks caused by synchronous calls, enabling the system to withstand large-scale concurrent access.

[0081] The AlarmCollect service is the core data processing module within the view library. After consuming structured data from Kafka, this service parses the data according to the GA / T1400 protocol, extracting core fields for business queries such as vehicle license plate numbers, vehicle identification numbers, facial feature values, and human feature values, and writes these fields into the structured data database. Simultaneously, AlarmCollect extracts image identifiers and corresponding image content from the data, calls the object storage interface to upload the images to OBS, and records the mapping relationship between image identifiers and object storage addresses in the database. In this way, structured fields and image resources are stored in the database and object storage respectively, forming a hierarchical storage structure. Depending on business needs, AlarmCollect can also generate alarm information and send alarm trigger signals to upper-layer applications or cascading services.

[0082] The structured data database can be implemented using relational databases such as PostgreSQL. Through a database sharding strategy, the 1400 structured data entries are managed separately from regular business data, and stored in separate tables according to object types such as vehicles, faces, and bodies. The database primarily stores core fields used for retrieval and image mapping information, reducing the storage pressure on a single table while improving read and write performance. Object storage (OBS) is used to store image data associated with the structured data and optional feature files, providing large-capacity object-level storage capabilities suitable for long-term storage of massive amounts of images. The database and OBS are linked through a mapping relationship between image identifiers and storage addresses. During business queries, data can be retrieved first based on structured fields, and then the source image can be retrieved from OBS as needed.

[0083] On the local side, streaming media services and the SouthAPI interface service are also included. When AlarmCollect generates an alarm or requires linked video, it can send trigger information to the streaming media service, which is responsible for media resource management and distribution. SouthAPI, as the southbound interface, provides a unified access channel for the video conferencing application. The video conferencing application is deployed on the local application platform, providing on-duty personnel with functions such as alarm display, target retrieval, and video playback. It obtains alarm results and related videos and images through SouthAPI, realizing the visualization display and business operations of local view library data.

[0084] exist Figure 2 The upper-level cascading and third-party interaction areas shown include GB1400 access service, dedicated 1400 data processing AlarmCollect service, file service module, independent storage media, view analysis system, data interface service, view library application, 1400 upper-level platform, and GAC and GAS cascading services, which are used to complete the interconnection with the upper-level view library and third-party systems.

[0085] The GB1400 access service handles access requests from external systems such as the 1400 upper-level platform and view analysis systems, including registration, subscription, data upload, and query requests. Upon receiving structured data or alarm data, the service performs protocol parsing and basic permission verification before writing the data to Kafka, where it is consumed uniformly by the backend 1400 data processing AlarmCollect. When receiving query or subscription requests, the GB1400 access service forwards the query conditions or subscription rules to the backend data processing and query service based on the request type, maintaining protocol compatibility while avoiding the accumulation of complex business logic on the access end.

[0086] The 1400 Data Processing AlarmCollect service consumes request data from the GB1400 access service, further checks the data format and permissions, and triggers data queries, data synchronization, or alarm push operations based on the request type. When image resources are needed, the 1400 Data Processing AlarmCollect service calls the file service module based on the image identifier in the database. The file service parses the storage address in OBS, retrieves the image content as needed, and then returns the structured fields and image data to the upper-level platform or view analysis system. To alleviate the pressure on the local main database, this embodiment configures an independent storage medium on the right side to persist the structured data generated by the 1400 Data Processing AlarmCollect, thus distributing the read / write load across multiple devices.

[0087] The view analysis system exists as a third-party analysis platform, capable of sending data query and analysis requests to the GB1400 access service. Structured data is processed by the GB1400 data processing AlarmCollect and the data interface service before being returned, thus supporting statistical analysis, model training, and other advanced applications based on the view library data. The data interface service provides unified data output capabilities, offering standardized interfaces to external systems such as the upper-level view library and view analysis system, and returning structured fields and image mapping information as needed.

[0088] On the upper-level view library side, the view library application is deployed on the upper-level platform. It can receive cascaded data from this view library through the 1400 upper-level platform, enabling unified viewing and retrieval of alarms, target information, and image resources. The 1400 upper-level platform establishes a cascade relationship with this view library through the GAC service, receiving data pushed by this view library or sending subscription requests to this view library. The GAC service is mainly responsible for pushing data to the upper-level platform, while handling registration, keep-alive, and deregistration processes between the upper and lower levels. Correspondingly, the GAS service completes subscription and registration processing on the lower-level platform side. Through interaction with the 1400 lower-level platform, it realizes the subscription and management of lower-level acquisition systems or view library resources. The 1400 lower-level platform is used to host lower-level view libraries or acquisition systems. It can receive subscription instructions initiated by the GAS service and also report locally acquired data to this view library.

[0089] pass Figure 2 As can be seen from the overall architecture shown, this embodiment clearly separates the access, message passing, centralized processing, and hierarchical storage links, and completes the interconnection with the upper and lower level view libraries through cascading services. From the system level, it reflects the technical solutions of access decoupling, asynchronous processing, hierarchical storage, and external decoupling in Embodiment 1 and Embodiment 2.

[0090] Figure 3This demonstrates an interactive logic architecture between a multi-terminal system and a view library. The architecture details the signaling and data flow relationships between the application platform, analysis system, acquisition devices, and view library from both the external system layer and the internal service layer of the view library.

[0091] At the external system layer, the application platform resides in the third-party application layer, typically providing general interfaces and data service interfaces. The application platform initiates registration, deregistration, keep-alive, and time synchronization requests to the view library through the general interfaces. After completing identity verification, the internal application platform service of the view library writes the platform information to the database and maintains the platform status. When it needs to query structured data or alarm records, the application platform submits query conditions through the data service interface. The connecting application platform service forwards these conditions to the internal data query service, which retrieves the results from the database and returns them. The application platform then displays and interacts with these results accordingly. Figure 3 The module marked as New Service 1: Connecting to the application platform corresponds to the aforementioned application platform connection service.

[0092] The analysis system resides in the third-party analysis or data acquisition layer and typically possesses general interfaces, analysis interfaces, data acquisition interfaces, and data service interfaces. The analysis system performs registration, deregistration, keep-alive, and time synchronization processes with the view library through the general or analysis interfaces, and interfaces with the data acquisition system services to store system information and maintain its operational status. Figure 3 The module labeled as New Service 2 is the module for connecting to the data acquisition system, corresponding to the data acquisition system service in this embodiment. When the analysis system needs to upload structured data, it sends data to the view library through the acquisition interface. The data acquisition system service writes the received data into Kafka, where the alarm processing service parses and stores it in the database. When the analysis system needs to query structured data in the view library, it submits query conditions through the data service interface. The data acquisition system service forwards the conditions to the data query service, which retrieves the results from the database and returns them, thus realizing bidirectional interaction between data query and upload.

[0093] The on / offline data acquisition devices belong to the front-end acquisition layer and communicate with 1400DAS through acquisition interfaces and general interfaces. When a device comes online, it sends a registration request to 1400DAS. After the view library completes authentication, it writes the device information to the database. During operation, the device sends keep-alive messages periodically and can synchronize its time as needed to ensure the consistency of event times. Structured data is uploaded by the device through the acquisition interface. 1400DAS can, according to the deployment strategy, directly push the data to an external storage address or write it to Kafka for unified processing by the internal alarm processing service. Figure 3 The 1400DAS module corresponds to the data acquisition side access service in this embodiment.

[0094] Within the internal service layer of the view library, the application platform service and the data acquisition system service are responsible for the access and pass-through of the application platform and analysis system, respectively. This is the specific implementation of the application platform service and the data acquisition system service in Example 2. To unify the query logic, a data query service is set up internally within the view library as a unified query entry point. This service receives query conditions forwarded by the application platform service and the data acquisition system service, retrieves structured data and image mapping information from the database, and returns the results to the caller. Figure 3 The module marked as New Service 3: Data Query Service corresponds to this function. The database is used to store device information, system information, core fields of structured data, and image mapping relationships. The web management interface provides maintenance personnel with an entry point for device and system configuration management.

[0095] In this embodiment, Redis is used to cache some status data and alarm-related information, which can improve read and write efficiency in the alarm processing flow and support fast marking and reading of data that needs to be pushed. Kafka undertakes the message relay task of massive structured data and accepts write requests from 1400DAS and the data acquisition system service. OBS object storage is responsible for storing image resources and feature files associated with structured data, forming a hierarchical storage system together with the database.

[0096] Regarding alarm and cascading services, multiple AlarmCollect instances can be deployed within the view library based on workload. Figure 3 The module labeled AlarmCollect (handles general alarms) is primarily used to process general alarm data, consuming corresponding messages from Kafka and completing alarm generation and distribution. The module labeled AlarmCollect (handles 1400 structured alarms) focuses on processing 1400 structured alarm data, responsible for extracting core fields and storing them in the database, and performing operations such as image uploading and object storage. AlarmCollect instances can be horizontally scaled as needed, improving the ability to handle high-concurrency data through parallel consumption by multiple instances. GAC, as a subordinate service module of GB1400, works in conjunction with Redis and AlarmCollect, responsible for sending the data and alarms to be pushed to the upper-level view database, and maintaining the registration, keep-alive, and deregistration processes with the upper-level platform.

[0097] pass Figure 3 As can be seen from the interactive architecture shown, the 1400DAS, the application platform service, and the data acquisition system service mainly undertake the functions of external system access and data transmission. Data parsing, field extraction, and alarm generation are completed by a dedicated AlarmCollect service. The message middleware and object storage undertake the basic work of data flow and hierarchical storage. Overall, it reflects the design idea of ​​decoupling access and processing, as well as decoupling external interaction and internal logic.

[0098] Figure 4 This illustrates a registration and keep-alive interaction process between the GAS service and the GAC service. In this embodiment, the GAS initiates subscription and registration with the upper-level view library, while the GAC pushes data to the upper-level view library and maintains the connection state. The two establish and maintain a cascading channel through a standard registration, keep-alive, time synchronization, and deregistration mechanism.

[0099] During the initial channel establishment phase, GAS sends a registration request to GAC, carrying parameters such as the cascade account, device identifier, and authentication password. After GAC completes identity verification, it returns a registration success message. The two parties then synchronize their time to confirm that their system clocks are consistent, and the cascade channel enters the connection state.

[0100] During the channel maintenance phase, GAS sends keep-alive messages to GAC at preset intervals. GAC returns a keep-alive response after verifying that the parameters are correct. As long as a normal response is received within the set time window, the connection status remains connected, and GAC will continue to receive data from this view library and forward it to the upper-level platform. If necessary, both parties can also re-synchronize their time periodically or as needed to avoid time offsets caused by long-term operation.

[0101] Figure 4 The document also illustrates the disconnection and reconnection process in case of configuration changes or network anomalies. For example, if the upstream side modifies the cascade username or password but fails to synchronize it to the peer in a timely manner, password verification will fail. The service will mark the current connection status as disconnected locally to indicate the need for configuration correction. After the parameters are restored to correct, GAS can re-initiate registration and keep-alive requests. GAC verifies the changes successfully and returns a response, restoring the channel to a connected state. The downstream side will also switch its connection status to disconnected when it detects that it has not received a keep-alive message for an extended period, to avoid continuing to push data to invalid channels. When the network recovers and a valid keep-alive message is received from the upstream side, the downstream side will update its status to connected, and data push will resume.

[0102] Through the above-mentioned registration, keep-alive, disconnection detection and reconnection mechanisms, GAS and GAC can automatically restore the cascading channel in the event of account parameter adjustments, network jitter or device restart, ensuring the long-term stability and reliability of the data link between upper and lower level view libraries, and providing a guarantee for alarm linkage and data synchronization under the multi-level view library architecture.

[0103] like Figures 2 to 4The architecture shown in this embodiment decomposes and combines functions such as data acquisition and access, message passing, data processing, hierarchical storage, and cascading between upper and lower levels in a microservice manner. It achieves asynchronous processing and horizontal scaling of massive structured data through Kafka message middleware and multi-instance AlarmCollect service. It achieves hierarchical management of structured fields and image resources through the cooperation of database sharding and object storage OBS. It ensures stable cascading connections between multi-level view libraries through the registration and keep-alive mechanism of GAS and GAC. Thus, in engineering practice, it specifically implements the low-latency processing and dynamic scaling scheme for massive structured data of view libraries described in Embodiments 1 and 2.

[0104] Although embodiments of the present invention have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting the present invention. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of the present invention.

Claims

1. A method for low-latency processing and dynamic scaling of massive structured data in a view library, characterized in that: Includes the following steps: The access service on the acquisition side only performs registration, keep-alive, time synchronization, and access and transparent transmission of structured data for acquisition devices / systems. It does not perform parsing, storage, or retrieval of structured data and images on the access side. When there is no need to push the data to a third-party system according to preset rules, it performs integrity verification on the received structured data and writes the verified structured data and its image identifier into the message middleware. The processing service consumes the structured data from the message middleware, parses the structured data according to a predetermined protocol, extracts the core fields, writes the core fields into the database, and uploads the corresponding image data to the cloud / object storage. The database records the mapping relationship between the image identifier and the cloud / object storage address, thus decoupling the access process from the data processing and storage process. Structured data and business data are separated into separate databases, and structured data is divided into tables according to object type. The database only stores the mapping information between the core fields and image identifiers and cloud / object storage addresses, while the original image data is stored independently in the cloud / object storage. Based on the number of devices and the amount of structured data pushed, the processing service is deployed and scaled up or down in multiple instances. When scaling up, instances are added and subscription partitions are assigned to them. When scaling down, the messages to be processed are migrated to the subscription partitions corresponding to the retained instances before the instances are shut down. The message distribution relationship is adjusted after instances are added or removed. The independent service, which interfaces with the upper and lower level view libraries and third-party platforms, provides registration, subscription or push, query and data upload capabilities. The independent service only interacts with the message middleware and database or cloud / object storage within the view library, and does not carry out structured data parsing and database entry calculation. This decouples external interaction from the access, processing and storage processes, forming a horizontally scalable pipeline processing channel consisting of access services, message middleware, processing services and hierarchical storage.

2. The method according to claim 1, characterized in that: The independent services that interface with upper and lower level view libraries and third-party platforms include: a 1400DAS module for device / system registration, keep-alive, deregistration, time synchronization, and structured data reporting; a GAC ​​module for data push, registration, keep-alive, deregistration, and time synchronization to third-party view libraries; a GAS module for subscription and registration processing to third-party view libraries; and interface services for data acquisition systems and application platforms, respectively, for connecting to third-party acquisition systems and application platforms and providing registration, keep-alive, deregistration, time synchronization, data query, and / or data upload functions. All of the above modules are deployed decoupled from the processing services.

3. The method according to claim 2, characterized in that: The registration and keep-alive interaction process between GAS and GAC includes: (1) A cascading channel is established by one party sending a registration request to the other party, the receiving party completing the registration process and returning a registration response; (2) After successful registration, the data push side sends a keep-alive message according to the preset keep-alive period, and the receiving side returns a keep-alive response to maintain the channel validity; (3) Time synchronization should be performed as needed during cascaded operation to maintain consistency of time base between the two parties; (4) When it is necessary to cancel the cascade channel, initiate a cancellation request. After the receiving side completes the cancellation process, it returns a cancellation response and closes the cascade channel.

4. The method according to claim 2, characterized in that: The data acquisition side access service is a 1400DAS module, and its specific process for performing structured data access and transparent transmission includes: (1) Receive access requests initiated by the data acquisition device / system and parse them into registration, keep-alive, or time synchronization requests; (2) After verifying the identity of the registration request and completing the entry of the device information into the database, return a registration success response; for the keep-alive request, update the online status of the device and return a keep-alive confirmation response; for the time synchronization request, push the standard time to the acquisition device / system to complete the time synchronization. (3) Receive the structured data of vehicles, faces or bodies and related image data reported by the data collection equipment / system, and perform basic integrity verification; (4) Determine whether the structured data needs to be pushed to a third-party system: if needed, push to a preset third-party storage address; if not needed, perform pass-through and write the structured data and associated image identifiers into the message channel of the message middleware.

5. The method according to claim 1, characterized in that: The processing flow for data sent or pushed from a higher-level view library or a third-party system includes: (1) The message channel of the message middleware is used to register, verify the permissions and format of the data to be sent or pushed; (2) When the data object is device information, perform device information entry into the database; (3) When the data object is structured data and associated image data, the data is pushed to the processing service for unified parsing and storage, and the image is uploaded to the cloud / object storage and the image identifier is mapped to the storage address.

6. The method according to claim 1, characterized in that: The processing service is the AlarmCollect service, whose process for consuming and processing structured data includes: (1) During initialization, subscribe to the topic corresponding to the structured data in the message middleware and configure a message consumption thread or thread pool to concurrently pull and consume messages; (2) Pull structured data from the message channel and parse it according to the GA / T1400 protocol specification to extract the core fields of multiple objects in vehicles, faces and bodies; (3) Validate the core fields and write the validated data into the structured data database after database sharding and table partitioning; (4) Extract the image identifier from the structured data, call the upload interface of the cloud / object storage to upload the image data, obtain the storage address after uploading, and write the mapping relationship between the image identifier and the storage address into the database; (5) When it is necessary to trigger the synchronization of the upper and lower level view libraries, send a data push trigger signal to the GAC module or GAS module.

7. The method according to claim 1, characterized in that: The implementation of the hierarchical storage includes: managing 1400 structured data and regular business data in separate databases, and dividing the 1400 structured data into tables according to object type; the database stores the mapping information between the core structured fields used for querying and retrieval, as well as image identifiers and cloud / object storage addresses, to support retrieval based on structured fields and on-demand retrieval of image data from the origin cloud / object storage.

8. The method according to claim 1, characterized in that: The message middleware and cloud / object storage are configured as follows: the message middleware uses Kafka, a distributed message queue that supports topic and partitioning mechanisms. Its message channels are configured with topics and partitions according to business needs to achieve parallel consumption and load balancing of structured data. The cloud / object storage uses OBS object storage server to store image resources associated with structured data and their feature or attribute data files.

9. The method according to claim 1, characterized in that: The dynamic expansion and contraction includes: (1) Set expansion and reduction trigger conditions based on the observation indicators of the number of online acquisition devices and the amount of structured data pushed; (2) When any observation metric meets the expansion trigger condition, a processing service instance is automatically added, and a corresponding Kafka topic partition subscription is assigned to the new instance to share the structured data to be processed; (3) When the observed index meets the scaling down trigger condition, the number of processing service instances is automatically reduced. Before scaling down, the messages to be processed in the partition corresponding to the instance to be scaled down are migrated to the subscription partition corresponding to the retained instance. After the migration is completed, the message subscription of the instance to be scaled down is closed. (4) During the scaling up and down process, message consumption is kept uninterrupted, and the message allocation relationship of each instance is automatically updated after the instance is added or removed, so as to achieve dynamic balance of message consumption load.

10. The method according to any one of claims 1 to 9, characterized in that: The third-party platform integration includes integration with the data collection system service and integration with the application platform service. Specifically: the data collection system service receives and verifies registration requests from the analysis or data collection system, completes system information entry into the database, receives data query requests from the analysis or data collection system, forwards them to the data query service to obtain query results, and returns the results; it also receives data upload requests from the analysis or data collection system and forwards the data to the message channel of the message middleware. The application platform service receives registration requests and query conditions from the application platform, forwards the query conditions to the data query service and returns the query results, receives alarm subscription requests from the application platform, configures alarm push rules to the processing service, and triggers alarm pushes accordingly. When image content needs to be returned, it retrieves the image from the source cloud / object storage based on the image identifier and cloud / object storage address mapping in the database and returns the image.

Images