A government affair data security exchange and multi-modal information extraction system and method

Abstract

The application relates to a government affair data security exchange and multi-modal information extraction system and method, and belongs to the technical field of computer networks.The system comprises a data integration scheduling module, a data security exchange module and a multi-modal information extraction and intelligent analysis module.The data integration scheduling module is arranged at a network center node and serves as a central hub for data exchange between various exchange nodes; the data security exchange module is arranged at each exchange node, receives scheduling instructions from the data integration scheduling module and actually executes data exchange tasks; and the multi-modal information extraction and intelligent analysis module is arranged at the network center node and each exchange node, and the former is used for overall network data aggregation analysis, and the latter is used for data analysis under the corresponding exchange node.The application realizes the collaborative optimization of data security exchange and privacy protection, the fusion of multi-modal data intelligent extraction and analysis technology, and intelligent analysis and system optimization of cross-system non-business data.

Description

Technical Field

[0001] This invention belongs to the field of computer network technology, specifically relating to a system and method for secure exchange of government data and multimodal information extraction. Background Technology

[0002] With the continuous improvement of informatization and digitalization, the government sector has accumulated a large amount of heterogeneous data, covering various modalities such as text, images, audio, and video. In the process of cross-departmental collaborative work and the integration of multiple systems in large-scale government informatization projects, the data between different systems often have differences in format, structure, and storage methods, as well as a lack of sufficient security and privacy protection measures. This seriously restricts the secure and efficient flow and sharing of data, and affects the accuracy and timeliness of government decision-making.

[0003] Although there are some privacy protection technologies for data exchange (such as homomorphic encryption and differential privacy) and some basic system integration methods, these technologies still have certain limitations.

[0004] 1. In terms of data security, although homomorphic encryption can process encrypted data, its computational complexity is high and its performance is poor, making it unsuitable for large-scale data exchange scenarios. While differential privacy can protect data privacy to a certain extent, the existing privacy budget management mechanism is relatively static and cannot cope with dynamic and diverse data exchange needs.

[0005] 2. On the other hand, most existing data integration methods focus on basic data transmission between systems, lacking sufficient intelligent support, making it difficult to efficiently extract, analyze, and integrate complex multimodal data.

[0006] Therefore, existing technologies have not yet been able to effectively balance data security and the intelligent analysis needs of multimodal information. How to achieve cross-system and cross-departmental data integration and intelligent analysis while ensuring data privacy remains an important problem that urgently needs to be solved. Summary of the Invention

[0007] (a) Technical problems to be solved

[0008] The technical problem this invention aims to solve is how to provide a system and method for secure exchange of government data and extraction of multimodal information, in order to address the fact that existing technologies have not been able to effectively balance the needs of data security and intelligent analysis of multimodal information, and how to achieve cross-system and cross-departmental data integration and intelligent analysis while ensuring data privacy.

[0009] (II) Technical Solution

[0010] To address the aforementioned technical problems, this invention proposes a system for secure exchange and multimodal information extraction of government data. This system comprises three main modules: a data integration and scheduling module, a secure data exchange module, and a multimodal information extraction and intelligent analysis module.

[0011] The data integration and scheduling module is deployed at the network center node and acts as the "central hub" when exchanging data between various exchange nodes. It is responsible for managing the data flow across systems and departments, ensuring that data between systems can be exchanged efficiently and reliably according to predetermined paths, protocols and formats.

[0012] The data security exchange module is deployed at each exchange node. It receives scheduling instructions from the data integration and scheduling module, and actually executes data exchange tasks to ensure privacy protection, data integrity and transmission security during the data exchange process.

[0013] The multimodal information extraction and intelligent analysis modules are deployed at the network central node and each switching node, respectively. The former aims to perform overall network data aggregation and analysis, while the latter aims to perform data analysis at the corresponding switching nodes.

[0014] This invention also provides a method for secure exchange of government data and extraction of multimodal information, the method comprising the following steps:

[0015] Step S1: Initialize the data exchange task

[0016] Establish a basic framework for data exchange, including clarifying data sources and objectives, defining task parameters, generating metadata, and determining privacy protection requirements;

[0017] Step S2: Data Privacy Protection Processing

[0018] First, based on the privacy budget and data types set in the task initialization phase, differential privacy protection operations are performed on different modalities of data. This process reduces the impact of individual data records on the overall analysis results by adding noise, blurring, or de-identifying data, thus ensuring data privacy and security.

[0019] Step S3: Data Transmission and Scheduling

[0020] The data is transmitted securely and efficiently from the source system to the target system after privacy protection and encryption. In this process, the data integration and scheduling module is responsible for dynamically planning the transmission path, optimizing resource scheduling, and monitoring the status and log information in real time to ensure the stability, efficiency and security of the data flow.

[0021] Step S4: Data reception and multimodal information extraction

[0022] The received data is decrypted, parsed, and multimodal information is extracted and analyzed. This stage transforms the multimodal data transmitted from the source system into structured information and provides valuable business decisions and insights through intelligent analysis.

[0023] Step S5: Analysis Results Integration and Feedback Optimization

[0024] The analysis results of the multimodal information extraction module are integrated with non-business data generated during system operation to generate business insights and optimization suggestions. At the same time, this step enables dynamic optimization of privacy budget allocation and transmission scheduling strategies, ensuring the continuous and efficient operation of the entire government affairs integration platform.

[0025] (III) Beneficial Effects

[0026] This invention proposes a system and method for secure exchange of government data and multimodal information extraction. The beneficial effects of this invention include:

[0027] 1. Synergistic Optimization of Secure Data Exchange and Privacy Protection: Traditional data security technologies (such as homomorphic encryption and static privacy budget allocation) often face performance bottlenecks or insufficient flexibility when protecting privacy. This invention achieves intelligent adjustment of the privacy budget through dynamic privacy budget optimization, combined with feedback results from metadata analysis and multimodal intelligent analysis during the scheduling process. The system can automatically allocate a higher privacy budget for highly sensitive data (such as personal information involved in a case) while reducing the protection intensity for non-sensitive data (such as statistical information), thus balancing the effectiveness of privacy protection and data analysis.

[0028] 2. Integration of Intelligent Extraction and Analysis Technologies for Multimodal Data: Compared to traditional single-modal processing methods, this invention designs a unified multimodal learning framework for multimodal data such as text, images, audio, and video. It utilizes deep learning models (such as the Vision-and-Language Transformer) to perform joint representation learning and semantic understanding of each modality. Furthermore, through the fusion analysis of multimodal data, a unified knowledge graph is generated, providing comprehensive and intuitive decision support.

[0029] 3. Intelligent Analysis and System Optimization of Non-Business Data Across Systems: Traditional systems focus more on the exchange of business data, neglecting the potential value of metadata and system logs during the scheduling process. This invention introduces graph neural networks and anomaly detection technology to perform real-time analysis of metadata and logs, identifying inefficient paths, bottleneck nodes, and insufficient privacy budgets in data exchange, thereby proposing dynamic optimization suggestions. For example, for high-load exchange nodes, the system can automatically adjust the scheduling strategy to achieve load balancing. Attached Figure Description

[0030] Figure 1 This is a system architecture diagram of the system involved in this invention;

[0031] Figure 2 Technical architecture diagram of the data integration and scheduling module;

[0032] Figure 3 Technical architecture diagram of the data security exchange module;

[0033] Figure 4 Technical architecture diagram of the multimodal information extraction and intelligent analysis module;

[0034] Figure 5 This is a typical usage flow of the method involved in this invention. Detailed Implementation

[0035] To make the objectives, contents, and advantages of the present invention clearer, the specific embodiments of the present invention will be described in further detail below with reference to the accompanying drawings and examples.

[0036] To address the aforementioned issues, this invention provides a data security exchange and multimodal information extraction system and method for government integration platforms. While ensuring data security and privacy by utilizing dynamic differential privacy technology, it effectively integrates and intelligently analyzes different types of data (such as text, images, audio, and video) through multimodal information extraction technology, providing accurate and timely support for government decision-making.

[0037] The system architecture involved in this invention is as follows: Figure 1 As shown.

[0038] The system of this invention comprises three main modules: a data integration and scheduling module, a data security exchange module, and a multimodal information extraction and intelligent analysis module. Among them,

[0039] The data integration and scheduling module is deployed at the network center node and acts as the "central hub" when exchanging data between various exchange nodes. It is responsible for managing the data flow across systems and departments, ensuring that data between systems can be exchanged efficiently and reliably according to predetermined paths, protocols and formats.

[0040] The data security exchange module is deployed at each exchange node. It receives scheduling instructions from the data integration and scheduling module, and actually executes data exchange tasks to ensure privacy protection, data integrity and transmission security during the data exchange process.

[0041] The multimodal information extraction and intelligent analysis modules are deployed at the network central node and each switching node, respectively. The former aims to perform overall network data aggregation and analysis, while the latter aims to perform data analysis at the corresponding switching nodes.

[0042] The modules are described below.

[0043] 1. Data Integration and Scheduling Module

[0044] The data integration and scheduling module is the core scheduling unit of this system. It is primarily responsible for managing data flow across systems and departments, ensuring efficient and reliable data exchange between systems according to predetermined paths, protocols, and formats. In terms of technical implementation, the data integration and scheduling module's architecture is as follows: Figure 2 As shown, it includes: API gateway management submodule, data exchange scheduling submodule, task flow orchestration submodule, authentication and authorization management submodule, data exchange reliability assurance submodule, and monitoring and auditing submodule.

[0045] (1) API Gateway Management Submodule: Provides a unified data access interface, supports the conversion of multiple protocols and data formats, and realizes interconnection between systems.

[0046] (2) Data exchange scheduling submodule: responsible for scheduling and coordinating data exchange tasks, including flow control, priority management, load balancing, etc.

[0047] (3) Task process orchestration submodule: Based on business needs, dynamically orchestrate the data exchange process and ensure that the data exchange process meets security and compliance requirements.

[0048] (4) Authentication and Authorization Management Submodule: By integrating the authentication system, it ensures permission management and user authentication during the data exchange process.

[0049] (5) Data exchange reliability assurance submodule: Provides fault tolerance, breakpoint resume, message queue and other mechanisms to ensure high reliability in the data exchange process.

[0050] (6) Monitoring and Auditing Submodule: Real-time monitoring and log recording of the data exchange process, supporting auditing and compliance checks.

[0051] (1) API Gateway Management Submodule

[0052] The data integration and scheduling module provides a standardized interface for data exchange between systems through a unified API gateway, while the API gateway management submodule manages the API gateway. This API gateway supports multiple protocols (such as RESTful, SOAP, and GraphQL) and data formats (such as JSON, XML, and CSV). The API gateway can be deployed based on a microservice architecture, using open-source API gateway tools such as Kong and Zuul, or custom-developed. Its main features include:

[0053] 1) Protocol adaptation: The API gateway can automatically convert protocols and data formats according to the needs of different target systems.

[0054] 2) API Routing and Load Balancing: The API gateway performs intelligent routing and load balancing based on the target address of the request and the system status to ensure a balanced system load and avoid bottlenecks during data exchange.

[0055] (2) Data exchange scheduling submodule

[0056] The data exchange scheduling submodule involves how to allocate data transmission between different systems and nodes. In government multi-system integration scenarios, there are typically multiple data sources, each system potentially with different business logic and technology stacks. Therefore, data exchange scheduling needs to be highly flexible and configurable to ensure that each data exchange task can be executed on demand. The data exchange scheduling submodule primarily relies on the following mechanisms in its technical implementation:

[0057] 1) Task scheduling engine: Uses task scheduling frameworks such as Quartz and Apache Airflow to trigger and schedule data exchange tasks on a regular basis.

[0058] 2) Asynchronous Data Exchange: For requests that do not require immediate feedback, asynchronous processing is supported. The data exchange scheduling submodule can queue data exchange requests for asynchronous background processing, ensuring efficient request execution.

[0059] (3) Task Flow Arrangement Submodule

[0060] Task orchestration refers to the intelligent planning and sequential control of data flow according to business needs during data exchange between multiple systems. The technical implementation of the task orchestration submodule includes:

[0061] 1) BPMN (Business Process Modeling) Engine: Using the BPMN standard, it defines the process model for data exchange between different systems. For example, in a cross-unit case data exchange process, case text data is first obtained from the public security system, then audio and video evidence is obtained from the procuratorate system, and finally the results are aggregated and sent to the court system. The BPMN engine can ensure that these operations are executed automatically in sequence, reducing human intervention.

[0062] 2) Visual orchestration tools: To facilitate the configuration and modification of data exchange processes by administrators and developers, visual process orchestration tools are provided, such as process designers based on engines like Camunda or Activiti, which support drag-and-drop process design.

[0063] (4) Authentication and Authorization Management Submodule

[0064] During data exchange, the government sector has extremely high requirements for data security, thus necessitating ensuring that only authorized users or systems can initiate data exchange operations. The data integration and scheduling module, through the integrated authentication and authorization management submodule, ensures the legality and compliance of each operation. The technical implementation of the authentication and authorization management submodule includes:

[0065] 1) OAuth 2.0 or JWT: As a unified authentication mechanism, they support authorization and token verification for cross-system data access. OAuth 2.0 can ensure authentication during data access through security tokens, while JWT supports user authentication across multiple services.

[0066] 2) Role-Based Access Control (RBAC): RBAC enables fine-grained access control, ensuring that different users can only access the data and functions they are authorized to in different business processes.

[0067] (5) Data exchange reliability assurance submodule

[0068] To ensure reliability and high availability during data exchange, the data exchange reliability assurance submodule employs multiple mechanisms for fault tolerance and recovery. These mechanisms include:

[0069] 1) Resumable Download: For large-scale data exchange, resumable download technology is used. When the network is interrupted or the transmission fails, it can resume from the last transmission point, avoiding duplicate data transmission. Commonly used technologies and tools include HTTP RangeRequests, Resumable.js, and Tus Protocol.

[0070] 2) Message Queues: Use message queue systems such as Kafka and RabbitMQ to achieve asynchronous data transmission and message persistence, ensuring that data can be successfully transmitted to the target system.

[0071] 3) Data Redundancy: A data redundancy mechanism is employed to ensure that data can be recovered from the backup system in the event of a node or system failure, guaranteeing the continuity and stability of data exchange. Commonly used technologies include RAID, Hadoop HDFS, Ceph, and GlusterFS.

[0072] (6) Monitoring and Auditing Submodule

[0073] The monitoring and auditing submodule integrates real-time monitoring and logging systems to provide comprehensive monitoring, logging, and auditing capabilities for the data exchange process. Technical implementation includes:

[0074] 1) Log Management System: Use log management tools such as ELK Stack (Elasticsearch, Logstash, Kibana) to record all operation logs of data exchange in real time, which facilitates post-event auditing and analysis.

[0075] 2) Monitoring Platform: Use open-source monitoring tools such as Prometheus and Grafana to monitor system performance, network status, and task execution in real time during data exchange to ensure the healthy operation of the system.

[0076] 2. Data Security Exchange Module

[0077] The data security exchange module is the core module of the system, ensuring privacy protection, data integrity, and secure transmission during data exchange. It implements strict security measures at every stage of data exchange, ensuring that sensitive data is not leaked or tampered with during the exchange process, especially in scenarios involving the integration of multiple systems in the government sector. In terms of technical implementation, the technical architecture of the data security exchange module is as follows: Figure 3 As shown, it includes: a differential privacy protection submodule and a secure transmission management submodule.

[0078] (1) Differential privacy protection submodule: Perform differential privacy processing on sensitive statistical data during data exchange to prevent data leakage of personal privacy information.

[0079] (2) Secure transmission management submodule: It adopts encryption and authentication protocols to ensure the secure transmission of data during the exchange process and prevents it from being stolen or tampered with.

[0080] (1) Differential privacy protection submodule

[0081] Differential privacy protection is one of the core technologies of the data security exchange module in this invention. It aims to ensure that sensitive data involved in government information exchange does not leak individual privacy information, while maintaining the effectiveness and accuracy of data analysis. Unlike traditional encryption methods, the differential privacy protection submodule processes data by adding noise, making it impossible or extremely difficult to infer from individual data records. This method is particularly suitable for scenarios in the government sector that require the analysis and exchange of large amounts of statistical data, enabling data sharing and utilization while ensuring privacy.

[0082] The key technical points of differential privacy protection in this invention are as follows:

[0083] 1) Noise generation mechanism

[0084] The core idea of ​​differential privacy is to obscure individual data by introducing noise. Depending on the specific application requirements and the level of data privacy, this system employs two noise generation methods: Laplace and Gaussian.

[0085] a) Laplace Mechanism: For most government scenarios, the Laplace mechanism is used to add noise. This mechanism generates noise based on the sensitivity of the data and the privacy budget, ensuring that even if the data is leaked, individual information cannot be accurately reconstructed. Specifically, the module adds a noise source from a Laplace distribution to each data point (e.g., population statistics for a region). The intensity of the noise is closely related to the privacy budget; the level of the privacy budget determines the magnitude of the noise.

[0086] b) Gaussian Mechanism: For more complex statistical data or scenarios requiring higher privacy protection, the Gaussian mechanism is used. By adding Gaussian noise to the data, the statistical information remains largely valid, but individual information is effectively masked, preventing leakage.

[0087] 2) Privacy Budget Management and Optimization

[0088] Privacy budget management is a core component of differential privacy mechanisms, determining the intensity of noise introduced during data exchange and analysis. A smaller privacy budget introduces more noise and stronger privacy protection, but may lead to the loss of useful information; conversely, a larger privacy budget results in less noise and higher accuracy in data analysis, but weaker privacy protection. Therefore, how to reasonably set and dynamically adjust the privacy budget to ensure both data privacy and accuracy in data analysis is crucial in the design of differential privacy mechanisms.

[0089] In traditional differential privacy applications, privacy budgets are typically static, meaning they are fixed during the design phase. However, this invention addresses the practical needs of the government sector by proposing a dynamic privacy budget management and optimization mechanism. This mechanism allows the privacy budget to be adjusted in real time according to the requirements of different data exchange scenarios, maximizing the balance between privacy protection and data analysis accuracy.

[0090] a) Privacy Budget Initialization: At the start of each data exchange, the data integration scheduling module initializes the privacy budget based on the sensitivity of the data source and the privacy protection requirements of the target system. This budget value is set according to the predefined policies of each system and is initially adjusted before the exchange. For highly sensitive data (such as case data containing personally identifiable information), the privacy budget is set to a lower value to ensure sufficient noise to mask private information during the data exchange process. For less sensitive data (such as public statistics), a higher privacy budget value can be set to improve the efficiency of data exchange and analysis.

[0091] b) Privacy Budget Optimization Based on Historical Analysis: The effectiveness of differential privacy depends not only on the current privacy budget value but also on the processing of historical data. This invention uses a feedback mechanism in the AI ​​model to gradually accumulate the privacy protection effectiveness and actual performance of historical data analysis tasks. After each data exchange, the system evaluates the actual effect of the privacy budget setting and noise intensity. Based on this feedback, the system can intelligently optimize the privacy budget setting, thereby gradually improving the system's privacy protection capabilities for different types of data. For example, when processing highly sensitive data such as images and videos, the system learns from historical data feedback that these data have a high demand for privacy protection, thus reducing the privacy budget and increasing noise interference.

[0092] 3) Combining Differential Privacy with Multimodal Data

[0093] In the government sector, the types of data exchanged are diverse, including not only traditional text data but also multimodal data such as images, audio, and video. Differential privacy protection requires differentiated processing based on the characteristics of each data type.

[0094] a) Text data: For text data containing sensitive information (such as identity information, case information, etc.), differential privacy protection introduces noise to perturb the data statistical results, ensuring that after the data is exchanged, attackers cannot reconstruct the individual's specific information based on the query results.

[0095] b) Image Data: When dealing with sensitive data, especially image data involving facial recognition and geographic information, differential privacy ensures that personal privacy is not leaked by blurring or de-identifying the image. Image privacy protection also includes perturbing the detailed features in the image to prevent it from being used to reconstruct specific personal information.

[0096] c) Audio Data: Privacy protection for audio data is not limited to the voice content itself, but also includes background noise, audio length, and other aspects. During voice data analysis, the system processes the pitch, intonation, and speaker characteristics of the voice by adding noise perturbations to ensure that an individual's identity cannot be accurately reconstructed through audio analysis.

[0097] d) Video Data: When processing video data, differential privacy protection not only protects the privacy of each frame but also interferes with noise in the temporal dimension of the video. Specifically, for video data involving video surveillance, court recordings, etc., the system uses inter-frame noise injection technology to disturb the detailed information of individual key frames while ensuring that the video content does not affect the overall analysis.

[0098] (2) Secure transmission management submodule

[0099] The secure transmission management submodule ensures the security of data transmission during the exchange process, preventing data from being tampered with, stolen, or lost during network transmission. This function mainly relies on the following technologies.

[0100] 1) TLS / SSL Encryption Protocol: To ensure the confidentiality and integrity of data during transmission, the data security exchange module uses TLS (Transport Layer Security) or SSL (Secure Sockets Layer) to encrypt the data transmission channel. Through TLS / SSL protocols, data is encrypted during transmission, preventing eavesdropping or tampering. Specifically, during each data exchange, the data security exchange module encrypts the communication channel using the TLS protocol and performs two-way authentication before data exchange, ensuring that data can only be transmitted between authorized sources and destinations.

[0101] 2) Authentication and Access Control: To ensure that only legitimate users and systems can access the data, the data security exchange module also uses OAuth 2.0 and JWT (JSON Web Tokens) for authentication and authorization. OAuth 2.0 is used for authentication to ensure that the system or user requesting the data has legitimate access rights; JWT carries verification information in each data exchange to ensure that the transmitted data was indeed initiated by a legitimate identity.

[0102] 3) VPN Tunneling Technology: When data exchange needs to cross multiple network areas or between government agencies, the data security exchange module can also use VPN tunneling technology to establish a private, encrypted channel. Through a VPN tunnel, the data exchange process is less susceptible to interference from external network attacks, and the security and integrity of the data during transmission are further guaranteed.

[0103] 3. Multimodal Information Extraction and Intelligent Analysis Module

[0104] The multimodal information extraction and intelligent analysis module is one of the core functional modules of this invention, responsible for extracting and intelligently analyzing heterogeneous data from multiple sources and multiple modalities. This module can not only perform in-depth analysis of exchanged business data, providing decision support and business insights, but also analyze non-business data generated during scheduling, providing optimization strategies from the perspectives of system operation and privacy protection. This module is deployed at the network central node and various exchange nodes, forming a bidirectional collaborative mode. The main functions of this module include:

[0105] (1) Multimodal information extraction and analysis of business data: mainly targeting business data involved in the exchange process, using multimodal information extraction technology to extract valuable knowledge, help improve business processing efficiency, predict trends, and discover potential risks.

[0106] (2) Extraction and analysis of non-business data: Analyze the metadata, system logs and data flow information in the scheduling process, and provide strategies such as privacy budget optimization and system operation efficiency evaluation to achieve system optimization and resource scheduling improvement.

[0107] (3) Unified data analysis and integration: Intelligent analysis results of business data and system data are integrated to generate decision reports, provide optimization suggestions, and support government decision-making, in-depth insights, privacy protection strategies, etc.

[0108] In terms of technical implementation, the technical architecture of the multimodal information extraction and intelligent analysis module is as follows: Figure 4 As shown, it includes three sub-modules: multimodal information extraction and analysis of business data, extraction and analysis of non-business data, and unified data analysis and fusion.

[0109] (1) Multimodal information extraction and analysis submodule of business data

[0110] In government service scenarios, the exchanged business data involves multimodal unstructured data. To effectively integrate this heterogeneous data, this system uses a multimodal pre-trained model (such as Vision-and-Language Transformer or VisualBERT) to perform unified representation learning and information extraction on data of different modalities.

[0111] 1) Text Data Processing: Pre-trained language models based on the Transformer architecture (such as BERT or GPT) are used for semantic understanding and information extraction. Specific processing steps include Named Entity Recognition (NER), Relation Extraction (RE), and Sentiment Analysis. Through analysis of case texts, the system automatically identifies key entities and further mines potential relationships between them.

[0112] 2) Image Data Analysis: A Vision-and-Language Transformer model is employed, combining an image convolutional neural network (CNN) with a text processing module for cross-modal joint training. The system first uses a deep convolutional neural network (such as ResNet or EfficientNet) to extract image features, and then combines this with OCR technology to extract text from the image. Subsequently, through joint training, the model fuses image features with text information, thereby achieving a better understanding of the content within the image.

[0113] 3) Audio Data Processing: The audio data is transcribed into text using an Automatic Speech Recognition (ASR) system, and then combined with a text processing module for further natural language processing analysis. Based on Wav2Vec2 or a similar speech recognition model, the system can efficiently convert speech into text and perform sentiment analysis, keyword extraction, and other processing to capture the emotional information conveyed in the audio (such as changes in the speaker's tone of voice).

[0114] 4) Video Data Analysis: By using Transformer-based Video Recognition Models, the system can extract key information such as people, actions, and scenes from videos and combine it with case text data. Specifically, after extracting image features from video frames using CNNs, the data is input into a Transformer-based temporal model (such as TimeSformer) for temporal modeling to capture important moments in the video.

[0115] (2) Submodule for extraction and analysis of non-business data

[0116] Non-business data mainly refers to metadata, system logs, and data flow information generated during data integration and scheduling. By analyzing this data, the system can perform system optimization, resource scheduling, and privacy budget adjustments.

[0117] 1) Scheduling Metadata Analysis: Metadata during the scheduling process (such as data source, target node, data transmission latency, data size, etc.) is typically recorded in the form of logs or real-time metrics. To analyze this information, the system uses a Graph Neural Network (GNN) to model the data flow process, identifying and optimizing data exchange paths and nodes. By constructing a data flow graph, GNN can help identify potential bottlenecks and inefficient paths, and propose dynamic scheduling strategies based on factors such as data traffic and transmission latency to optimize resource allocation.

[0118] 2) System Log Analysis: System logs contain operational information during data exchange, such as error messages, system latency, and resource usage. Through deep autoencoders or anomaly detection algorithms (such as Isolation Forest), the system can monitor abnormal behavior in the logs in real time and trigger alerts. Anomaly detection algorithms can promptly identify potential system faults or performance bottlenecks and trigger automated processing mechanisms, such as automatically switching to backup nodes or retransmitting erroneous data. Log analysis can also be combined with reinforcement learning algorithms to optimize scheduling strategies and automatically adjust system operating parameters to improve overall efficiency.

[0119] 3) Privacy Budget Optimization: By analyzing metadata and system logs during the scheduling process, the system can identify which data exchange operations involve sensitive data and dynamically adjust the privacy budget based on the data's sensitivity. Through dynamic privacy budget algorithms (e.g., dynamically allocating the privacy budget based on data traffic, transmission duration, etc.), the system can intelligently optimize privacy protection measures, ensuring that data availability is maximized while protecting privacy.

[0120] (3) Data Unified Analysis and Integration Submodule

[0121] In this module, the goal of unified data analysis and fusion is to integrate data from different sources (such as business data and non-business data) to generate comprehensive analytical results that support government decision-making and system optimization.

[0122] 1) Multimodal Data Fusion: By employing multimodal learning frameworks such as CLIP (Contrastive Language-Image Pretraining), the system can fuse data from different modalities. This model connects image and text representations through a shared embedding space, thereby effectively integrating knowledge from multiple data sources.

[0123] 2) Knowledge Graph Construction and Analysis: Based on the extraction results of multimodal data, the system constructs a unified knowledge graph and uses graph analysis technology to perform in-depth data mining. The knowledge graph, by turning information such as events and people into nodes, forms a relationship network, helping government departments to gain a deeper understanding of the data connections and complex relationships between various systems.

[0124] 3) Decision Support and Optimization Suggestions: Based on the fused data, the system generates decision reports using reinforcement learning and intelligent decision-making algorithms. The system can provide data analysis results, risk assessments, and resource optimization suggestions to assist government decision-makers in formulating effective solutions.

[0125] Example 1:

[0126] This invention relates to a system and method for secure data exchange and multimodal information extraction in a government affairs integration platform. The typical usage flow of the method is as follows: Figure 5 As shown.

[0127] Step S1: Initialize the data exchange task

[0128] Data exchange task initialization is the starting point of this invention's method, responsible for establishing the basic framework for data exchange, including clarifying the data source and target, defining task parameters, generating metadata, and determining privacy protection requirements. This step provides clear context and boundary conditions for subsequent data processing, transmission, and analysis. This step involves the following key aspects.

[0129] (1) Determine the source system and target system for data exchange.

[0130] First, the data integration and scheduling module receives the task request and identifies the source and target systems of the data. For example, in a government affairs scenario, a local procuratorate needs to send data related to a case (such as case files and evidence) to the court to assist in the trial process. The source system is the procuratorate's data management system, and the target system is the court's data receiving system.

[0131] 1) Task Triggering: Task requests may be triggered manually by the user or automatically generated by events (such as when a case reaches the trial stage).

[0132] 2) System verification: The scheduling module first verifies the legitimacy of the source system and the target system, including checking whether the target system has been registered with the platform and whether it has the authority to receive data, to ensure that the exchange occurs in a trusted environment.

[0133] (2) Determine the data type and range

[0134] After determining the data source and target, the system needs to identify the data types and ranges to be exchanged in order to formulate clear task rules for subsequent processing and transmission.

[0135] 1) Data types include text (such as case descriptions), images (such as scanned copies of case files and photographic evidence), videos (such as court recordings), and audio (such as recorded testimonies).

[0136] 2) Data scope refers to the amount and range of data to be exchanged. For example, this exchange may only involve a portion of the case file for case number 12345, rather than all data from all cases.

[0137] By precisely defining data types and ranges, unnecessary redundant data transmission can be avoided, improving efficiency and reducing the risk of privacy leaks.

[0138] (3) Generate task metadata

[0139] The system automatically generates metadata related to this data exchange, recording detailed task information to ensure accurate execution of operations by each module in subsequent processes. This metadata includes, but is not limited to:

[0140] 1) Task ID: Uniquely identifies each data exchange task, facilitating subsequent tracking and auditing.

[0141] 2) Data attributes: such as data modality type (text, image, video, etc.) and data sensitivity level (normal, sensitive, highly sensitive).

[0142] 3) Target path: Defines the data transmission path from the source system to the target system, such as the order of intermediate nodes.

[0143] 4) Time requirements: including the deadline for data transmission and transmission priority (e.g., some tasks may require expedited processing).

[0144] For example, in a scenario where the prosecutor's office sends case files to the court, the task metadata might record the following:

[0145] 1) Task ID: TASK-20250123001

[0146] 2) Data Attributes: Image (scanned file, 10 pages), sensitivity level: "Highly Sensitive"

[0147] 3) Target path: Procuratorate → Central dispatch node → Court

[0148] 4) Time requirement: The deadline is January 24, 2025, and the priority is "high".

[0149] (4) Initialize privacy protection parameters

[0150] Based on the sensitivity level of the data, the system needs to set privacy protection parameters for data exchange tasks, especially the privacy budget ε and protection strategy:

[0151] 1) Privacy Budget Allocation: The system allocates an initial privacy budget to different types of data based on the sensitivity level of the task's metadata. For example, a lower privacy budget value (ε = 0.5) is allocated to highly sensitive data (such as information about persons involved in a case) to ensure the strength of privacy protection; a higher privacy budget value (ε = 2.0) is allocated to less sensitive data (such as statistics on the total number of cases) to ensure data availability.

[0152] 2) Differential privacy strategy: Define specific privacy protection methods, such as the intensity of noise introduced into statistical data and the range of de-identification of image data.

[0153] (5) Permission authentication and task confirmation

[0154] Data exchange tasks must be performed within clearly defined authorized scopes to prevent data leakage or mistransmission.

[0155] 1) Permission verification: The scheduling module calls the permission control mechanism to check whether the task initiator (such as the procuratorate) has the right to access or share data, and whether the target party (such as the court) has the right to receive relevant data.

[0156] 2) Task Confirmation: After authorization verification, the system notifies both parties to confirm task information, including data type, scope, transmission path, and deadline. Only after both parties confirm can the task proceed to the next step.

[0157] (6) Task storage and queuing

[0158] After task initialization is complete, the system stores the task metadata in the task queue, awaiting further processing by the scheduling module.

[0159] 1) Task storage: Task data is recorded in a secure database for subsequent process tracking and management.

[0160] 2) Task queuing: The scheduling module sorts tasks according to their priority and deadline to ensure that high-priority tasks (such as emergency case data exchange) can be processed first.

[0161] Step S2: Data Privacy Protection Processing

[0162] Data privacy protection is a key aspect of the method in this invention. The secure data exchange module first performs differential privacy protection operations on different modalities of data based on the privacy budget and data type set during the task initialization phase. This process reduces the impact of individual data records on the overall analysis results by adding noise, obfuscating, or de-identifying data, thus ensuring data privacy and security. The privacy protection methods for each modality are as follows.

[0163] (1) Privacy protection of text data

[0164] 1) Named Entity Recognition (NER) and De-identification: The system identifies sensitive entities (such as names, addresses, and case numbers) in the text using language models such as BERT, and de-identifies these entities, for example by replacing them with placeholders (such as "***") or introducing randomization for the specific values ​​of sensitive information.

[0165] 2) Differential privacy processing of statistical data: For statistical data (such as the distribution of case types), the system adds noise to the statistical results through the Laplace mechanism. For example, in the original statistics: the number of theft cases is 100; after adding noise: the number of theft cases becomes 103 (adjusted according to the noise intensity).

[0166] (2) Privacy protection of image data

[0167] 1) Blurring of key regions: Using target detection technology (such as YOLO or Faster R-CNN), the system identifies sensitive regions (such as names, ID numbers, facial features, etc.) in the image and blurs or occludes these regions.

[0168] 2) De-identification processing: For evidence photos containing faces, the system uses a face recognition model (such as Facenet) to locate the facial region and performs de-identification operations, such as blurring or replacing with other facial features.

[0169] 3) Adding noise: Random noise is added to the pixel-level content of the image to ensure that the image content cannot be easily restored. For example, the license plate number "Jing A12345" in the evidence photo may be processed as "Jing A12***".

[0170] (3) Privacy protection of audio data

[0171] 1) Automatic Speech Recognition (ASR) and text processing: Transcribe audio content into text, and process sensitive information according to the text privacy protection method.

[0172] 2) Speech feature de-identification: Process the speech features (such as intonation and speech rate) in the audio to prevent personal identity from being recognized through the voice. For example, by changing the speech frequency or adding background noise, the original voice "Zhang San" is converted into blurred speech that loses specific features.

[0173] (4) Privacy protection of video data

[0174] 1) Intra-frame protection: Perform image processing on each frame of the video, such as blurring key areas (faces, license plates) or deleting sensitive information in the frame.

[0175] 2) Inter-frame protection: Process the temporal dimension of the video, for example, reduce the video frame rate or randomly delete some frames to prevent behavior patterns from being inferred through video trajectories.

[0176] 3) Synchronized audio and video protection: If the video contains speech, the system will synchronously apply the privacy protection method for audio.

[0177] After the privacy protection is completed, the data security exchange module updates the task status to "to be transmitted", and notifies the data integration scheduling module to enter the next transmission step.

[0178] Step S3: Data transmission and scheduling

[0179] The main task of data transmission and scheduling is to securely and efficiently transmit the privacy-protected and encrypted data from the source system to the target system. In this process, the data integration scheduling module is responsible for dynamically planning transmission paths, optimizing resource scheduling, and monitoring the status and log information during the transmission process in real time, so as to ensure the stability, efficiency and security of data flow. This step involves the following key links.

[0180] (1) Data transmission path planning

[0181] The first step of data transmission is to determine the path of data flow, which is dynamically planned by the data integration scheduling module, and the optimal transmission mode is selected based on the real-time system status.

[0182] 1) Basis for path planning

[0183] Node load: The system will monitor the load status of each node in real time (such as bandwidth utilization, CPU load, and storage utilization) to avoid transmission delays caused by high-load nodes.

[0184] Transmission cost: The system comprehensively considers the transmission latency and communication cost of the path (such as the transmission cost across network domains) and selects the path with the highest cost performance.

[0185] Reliability: In government scenarios, reliability is a top priority. For example, the system will prioritize paths with high redundancy or backup nodes to improve the fault tolerance of transmission.

[0186] 2) Dynamic adjustment of transmission path

[0187] The system selects the optimal path at the start of transmission, but dynamically adjusts the path if node performance degradation or link interruption is detected during transmission. For example, when the load on a transmission node surges from 60% to 90%, the system will switch subsequent data streams to a backup node to prevent congestion.

[0188] (2) Selection of data transmission protocol

[0189] During data transmission, transport layer security (TLS) protocols ensure data integrity and security. Specifically, all data packets are encrypted using TLS to establish a secure channel and prevent man-in-the-middle attacks or data leaks. Additionally, digital certificate verification is performed between the source and target systems to ensure the legitimacy of both communicating parties.

[0190] 1) Encryption algorithm: Use a symmetric encryption algorithm (such as AES-256) to encrypt the data.

[0191] 2) Key distribution: Securely distribute symmetric encryption keys using an asymmetric encryption algorithm (such as RSA). The source system encrypts the key using the target system's public key, and the target system decrypts it using its private key to obtain the key.

[0192] 3) Encrypted data encapsulation: The encrypted data is encapsulated into standardized data packets and additional metadata (such as task ID, data modality type, encryption algorithm identifier, etc.) is attached for parsing during transmission.

[0193] (3) Real-time monitoring and anomaly detection

[0194] During data transmission, the system task status is "transmitting". The system ensures the stability and security of transmission through monitoring and anomaly detection mechanisms.

[0195] 1) Real-time monitoring

[0196] The system collects the following indicators in real time:

[0197] a) Transmission status: the sending and receiving status of data packets, as well as packet loss rate, number of retransmissions, etc.

[0198] b) Node performance: Real-time load, bandwidth usage, memory utilization, etc. of each node.

[0199] c) Transmission delay: The actual transmission delay from the source node to the target node.

[0200] 2) Anomaly Detection and Handling

[0201] The system identifies abnormal situations through anomaly detection algorithms (such as Isolation Forest or prediction models based on historical data), such as: node crashes or connection interruptions; packet loss rate exceeding the threshold (e.g., exceeding 5%); and abnormal increase in transmission latency (e.g., transmission delay exceeding 50% of the expected value).

[0202] When an exception occurs, the system will automatically perform the following operations:

[0203] a) Retransmission mechanism: Lost data packets will be retransmitted at the source node to ensure data integrity.

[0204] b) Switch to an alternate path: If a performance bottleneck is found on the primary path, the system switches to an alternate path to complete the transmission.

[0205] c) Alarm notification: The system records the abnormal log and notifies the administrator.

[0206] Once the data is successfully transmitted to the target system, the data integration and scheduling module will update the task status from "transmitting" to "transmission complete" and record a complete task log for subsequent tracking and auditing.

[0207] Step S4: Data reception and multimodal information extraction

[0208] Data reception and multimodal information extraction are responsible for decrypting, parsing, and extracting and analyzing multimodal information from the received data. This stage transforms the multimodal data transmitted from the source system into structured information and provides valuable business decisions and insights through intelligent analysis. This step involves the following key stages.

[0209] (1) Data reception and decryption

[0210] After the target system receives the data packet, it first needs to perform integrity verification, decryption, and parsing operations on the data.

[0211] 1) Integrity verification

[0212] When receiving data packets, the system verifies the integrity of the data packets using a message digest algorithm (such as SHA-256): the source system generates a hash value for the data packet when sending data and appends it to the metadata; after receiving the data packet, the target system recalculates the hash value. If the calculation result is consistent with the appended hash value, it means that the data has not been tampered with; otherwise, the system will trigger an alarm and request the source system to retransmit.

[0213] 2) Data Decryption

[0214] The data security exchange module decrypts the received data packets: the system first decrypts the symmetric key (AES key) used during transmission using the target system's private key. Then, it uses this symmetric key to decrypt the data packet content and recover the original data.

[0215] Task metadata parsing: After decryption, the system extracts task metadata (such as task ID, data type, privacy budget consumption records, etc.) from the data packet to provide input parameters for multimodal information extraction.

[0216] (2) Data classification and modality distribution

[0217] The data integration and scheduling module distributes different types of decrypted data to the corresponding processing modules based on the metadata identifiers in the data packets.

[0218] The system supports parallel processing of data of different modalities. For example, when case file data contains both text and images, the text and image processing modules can run simultaneously, improving the efficiency of information extraction.

[0219] (3) Multimodal information extraction

[0220] The multimodal information extraction module performs modality-specific analysis on the classified data, extracts key information, and generates structured results. The following is the specific processing procedure for each modality of data.

[0221] 1) Text data extraction

[0222] The text data extraction module uses pre-trained language models (such as BERT or GPT) combined with a dedicated task model for semantic understanding and information extraction. Specifically, Named Entity Recognition (NER) extracts entity information from the text (such as case number, persons involved, date, location, etc.); Relation Extraction (RE) analyzes the relationships between entities (such as the connection between persons involved and the case); and Text Sentiment Analysis identifies the sentiment in the text to assess the potential risks of the case.

[0223] 2) Image data extraction

[0224] The image data extraction module extracts information from images using computer vision technology. Specifically, Optical Character Recognition (OCR) extracts textual information from images, such as case numbers and signatures in scanned case files; Object detection and recognition uses Convolutional Neural Network (CNN) models (such as YOLO or Faster R-CNN) to detect target objects in images (such as vehicles and items in evidence photos); and de-identification checks verify whether the source system has blurred sensitive areas (such as ID numbers and faces), and performs further occlusion or blurring if necessary.

[0225] 3) Audio data extraction

[0226] The audio data extraction module processes speech content using automatic speech recognition (ASR) and semantic analysis techniques. Specifically, speech-to-text conversion uses models such as Wav2Vec2 to transform audio content into text; sentiment analysis assesses the emotional tendencies in the speech, such as identifying emotions like tension or anger in a witness's speech; and speech segmentation and speaker identification divide the audio into multiple segments and label the speaker's identity.

[0227] 4) Video data extraction

[0228] The video data extraction module combines time series analysis and image processing techniques to extract key events from the video. Specifically, scene recognition uses time series models (such as TimeSformer) to identify important scenes in the video (e.g., court hearings, witness statements); action recognition detects the actions and behaviors of people in the video, such as a witness raising their hand to speak; and event timeline construction timestamps the video content to generate an event sequence.

[0229] (4) Structuring and storing information extraction results

[0230] The multimodal information extraction module transforms the processing results into structured data, facilitating subsequent analysis and use.

[0231] 1) Structured storage: Extracted results are stored in a database in a uniform format, such as JSON or knowledge graph.

[0232] 2) Modal fusion: This involves linking data from different modalities to generate complete case information. For example:

[0233] {

[0234] Case Number: TASK-20250123001

[0235] Case Date: January 20, 2025

[0236] Location of the case: Dongcheng District, Beijing

[0237] Witness statement: "I saw Zhang San commit theft."

[0238] "Evidence Images": ["image_url_1", "image_url_2"],

[0239] "Court Hearing Video": ["video_url_1"]

[0240] }

[0241] Step S5: Analysis Results Integration and Feedback Optimization

[0242] The analysis result fusion and feedback optimization is the final step of this invention. It aims to integrate the analysis results from the multimodal information extraction module with non-business data (such as scheduling logs and metadata) generated during system operation to generate business insights and optimization suggestions. Simultaneously, this step enables dynamic optimization of privacy budget allocation and transmission scheduling strategies, ensuring the continuous and efficient operation of the entire government affairs integration platform. This step involves the following key aspects.

[0243] (1) Fusion of multimodal analysis results

[0244] Multimodal analysis result fusion involves semantic association and comprehensive integration of data extraction results from different modalities, ultimately generating unified structured information or knowledge graphs.

[0245] 1) Cross-modal data fusion

[0246] Text and image fusion: Semantic alignment is performed by extracting case description information (such as the persons involved and the time of the case) from the text and combining it with structured data (such as the case file number and signature extracted by OCR) from the image. For example, the case number "TASK-12345" in the case file image is matched and associated with the "case number TASK-12345" mentioned in the case description text.

[0247] Video and audio fusion: Combining scene analysis from the video (such as a witness raising their hand to speak) with audio content (such as the specific content of the witness's speech) to generate a complete event description. For example: "At 10:15:45, witness Zhang San stated that he witnessed the theft."

[0248] 2) Generate structured information

[0249] The merged analysis results are converted into a unified structured format for storage, such as JSON or knowledge graph.

[0250] 3) Knowledge Graph Construction

[0251] Based on the fused analysis results, the system constructs a knowledge graph with an "entity-relationship-entity" structure. For example:

[0252] Key elements: case number, persons involved, case location, evidence, etc.

[0253] Relationships: the connection between the individuals involved and the case, the connection between the evidence and the case, the time and place where the case occurred, etc.

[0254] Through knowledge graph analysis, the system can uncover potential patterns and relationships behind the data. For example, some cases may involve the same suspects or similar modus operandi, providing investigators with in-depth insights.

[0255] (2) Generate business decision support

[0256] Based on the integrated analysis results, the system generates business decision support information to help government departments handle complex matters efficiently.

[0257] 1) Business report generation

[0258] The system generates detailed business reports based on multimodal analysis results, including core business information, event chains, and risk assessments. For example, when a court is hearing a case, the system will provide an automatically generated report containing the following:

[0259] Case Summary: Case number, time, location, and persons involved.

[0260] Chain of evidence: a list of evidence and its sources (such as case files, videos).

[0261] Risk assessment: Potential legal risks or data quality issues in the case (such as incomplete evidence or insufficient data anonymization).

[0262] 2) Risk Warning and Recommendations

[0263] The system will intelligently evaluate the case analysis results, identify potential risks, and propose improvement suggestions. For example:

[0264] Privacy risk: If a file contains sensitive information that has not been fully de-identified, the system will suggest re-de-identification.

[0265] Completeness of evidence: If the video or audio related to the case is incomplete, the system will prompt you to retrieve the relevant data again.

[0266] (3) Optimization of non-business data feedback

[0267] In addition to business data analysis, the system also combines non-business data (such as metadata and scheduling logs) to generate feedback suggestions for optimizing privacy protection and transmission scheduling.

[0268] 1) Privacy Budget Optimization

[0269] The system analyzes the use of privacy budget during task execution. For example, if the privacy budget for a certain task is allocated too high, leading to a decrease in the accuracy of data analysis, the system will record relevant parameters and suggest reducing noise intensity and optimizing the allocation of privacy budget in future tasks.

[0270] By combining the results from the "multimodal information extraction and intelligent analysis module," the system can also learn and adjust future budget allocation strategies. For example, for tasks involving highly sensitive videos, the system will prioritize increasing the privacy budget.

[0271] 2) Scheduling strategy optimization

[0272] The system optimizes transmission paths and resource allocation by analyzing metadata and logs during the scheduling process. For example, if the system identifies certain nodes that frequently experience load bottlenecks, it can suggest load balancing of task traffic or plan alternative paths in advance.

[0273] The beneficial effects of this invention include:

[0274] 1. Synergistic Optimization of Secure Data Exchange and Privacy Protection: Traditional data security technologies (such as homomorphic encryption and static privacy budget allocation) often face performance bottlenecks or insufficient flexibility when protecting privacy. This invention achieves intelligent adjustment of the privacy budget through dynamic privacy budget optimization, combined with feedback results from metadata analysis and multimodal intelligent analysis during the scheduling process. The system can automatically allocate a higher privacy budget for highly sensitive data (such as personal information involved in a case) while reducing the protection intensity for non-sensitive data (such as statistical information), thus balancing the effectiveness of privacy protection and data analysis.

[0275] 2. Integration of Intelligent Extraction and Analysis Technologies for Multimodal Data: Compared to traditional single-modal processing methods, this invention designs a unified multimodal learning framework for multimodal data such as text, images, audio, and video. It utilizes deep learning models (such as the Vision-and-Language Transformer) to perform joint representation learning and semantic understanding of each modality. Furthermore, through the fusion analysis of multimodal data, a unified knowledge graph is generated, providing comprehensive and intuitive decision support.

[0276] 3. Intelligent Analysis and System Optimization of Non-Business Data Across Systems: Traditional systems focus more on the exchange of business data, neglecting the potential value of metadata and system logs during the scheduling process. This invention introduces graph neural networks and anomaly detection technology to perform real-time analysis of metadata and logs, identifying inefficient paths, bottleneck nodes, and insufficient privacy budgets in data exchange, thereby proposing dynamic optimization suggestions. For example, for high-load exchange nodes, the system can automatically adjust the scheduling strategy to achieve load balancing.

[0277] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the technical principles of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A system for secure exchange of government data and multimodal information extraction, characterized in that, The system comprises three main modules: a data integration and scheduling module, a secure data exchange module, and a multimodal information extraction and intelligent analysis module. The data integration and scheduling module is deployed at the network center node and acts as the "central hub" when exchanging data between various exchange nodes. It is responsible for managing the data flow across systems and departments, ensuring that data between systems can be exchanged efficiently and reliably according to predetermined paths, protocols and formats. The data security exchange module is deployed at each exchange node. It receives scheduling instructions from the data integration and scheduling module, and actually executes data exchange tasks to ensure privacy protection, data integrity and transmission security during the data exchange process. The multimodal information extraction and intelligent analysis modules are deployed at the network central node and each switching node, respectively. The former aims to perform overall network data aggregation and analysis, while the latter aims to perform data analysis at the corresponding switching nodes. in, The data security exchange module implements strict security measures at every stage of data exchange to ensure that sensitive data is not leaked or tampered with during the exchange process in scenarios involving the integration of multiple systems in the government sector; it includes: a differential privacy protection submodule and a secure transmission management submodule; Differential Privacy Protection Submodule: Performs differential privacy processing on sensitive statistical data during data exchange to prevent data leakage of personal privacy information; Secure transmission management submodule: Employs encryption and authentication protocols to ensure secure data transmission during the exchange process, preventing theft or tampering during transit; The differential privacy protection submodule implements privacy budget management and optimization, specifically including: Privacy budget initialization: At the start of each data exchange, the data integration scheduling module initializes the privacy budget based on the sensitivity of the data source and the privacy protection requirements of the target system. Privacy budget optimization based on historical analysis: Through the feedback mechanism of the AI ​​model, the privacy protection effect and actual performance of analysis tasks of historical data are gradually accumulated; after each data exchange, the system evaluates the actual effect of privacy budget settings and noise intensity; based on this feedback, the system intelligently optimizes the privacy budget settings. The multimodal information extraction and intelligent analysis module specifically includes: The multimodal information extraction and analysis submodule for business data: For business data involved in the exchange process, it uses multimodal information extraction technology to extract valuable knowledge, and selects a multimodal pre-trained model to perform unified representation learning and information extraction on data of different modalities; The non-business data extraction and analysis submodule analyzes metadata, system logs, and data flow information during the scheduling process, and provides privacy budget optimization and system operation efficiency evaluation strategies to achieve system optimization and resource scheduling improvement.

2. The government data secure exchange and multimodal information extraction system as described in claim 1, characterized in that, The data integration and scheduling module includes: an API gateway management submodule, a data exchange scheduling submodule, a task workflow orchestration submodule, an authentication and authorization management submodule, a data exchange reliability assurance submodule, and a monitoring and auditing submodule. API Gateway Management Submodule: Provides a unified data access interface, supports the conversion of multiple protocols and data formats, and enables interconnection between systems; Data exchange scheduling submodule: responsible for scheduling and coordinating data exchange tasks, including flow control, priority management, and load balancing; Task workflow orchestration submodule: Based on business needs, dynamically orchestrate the data exchange process and ensure that the data exchange process meets security and compliance requirements; Authentication and Authorization Management Submodule: By integrating an authentication system, it ensures access control and user authentication during data exchange. Data exchange reliability assurance submodule: Provides fault tolerance, breakpoint resumption, and message queue mechanisms to ensure high reliability during data exchange; Monitoring and Auditing Submodule: Enables real-time monitoring and logging of the data exchange process, supporting auditing and compliance checks.

3. The government data secure exchange and multimodal information extraction system as described in claim 2, characterized in that, API Gateway Management Submodule: The data integration and scheduling module provides a standardized interface for data exchange between systems through a unified API gateway. The API Gateway Management Submodule is used to manage the API gateway. The API gateway is deployed based on a microservice architecture and realizes protocol adaptation, API routing and load balancing. The data exchange scheduling submodule is used to allocate different systems and nodes for data transmission, ensuring that each data exchange task can be executed on demand, and realizing the task scheduling engine and asynchronous data exchange; The task flow orchestration submodule is used to intelligently plan and control the flow of data according to business needs when exchanging data between multiple systems, and to realize the BPMN engine and visual orchestration tools. The authentication and authorization management submodule is used to ensure the legality and compliance of each operation, implementing OAuth 2.0 or JWT, or role-based access control (RBAC). The data exchange reliability assurance submodule is used to achieve fault tolerance and recovery through breakpoint resumption, message queues, and data redundancy. The monitoring and auditing submodule provides comprehensive monitoring, logging, and auditing functions for the data exchange process by integrating real-time monitoring and logging systems. Implement a log management system and monitoring platform.

4. The government data secure exchange and multimodal information extraction system as described in claim 1, characterized in that, The differential privacy protection submodule implements a noise generation mechanism to obscure individual data by introducing noise. Depending on the specific application requirements and the data privacy level, it employs two noise generation methods: Laplace and Gaussian mechanisms. Laplace mechanism: This mechanism generates noise based on the sensitivity of the data and the privacy budget, ensuring that even if the data is leaked, individual information cannot be accurately reconstructed; by adding a noise from a Laplace distribution to each data point, the intensity of the noise is closely related to the privacy budget, and the level of the privacy budget determines the magnitude of the noise. Gaussian mechanism: By adding Gaussian noise to the data, the statistical information of the data still has general validity, but individual information is effectively masked.

5. The government data secure exchange and multimodal information extraction system as described in claim 4, characterized in that, Privacy budgets are set according to predefined policies for each system and are initially adjusted before the exchange. For highly sensitive data, the privacy budget is set to a lower value to ensure that there is enough noise to mask privacy information during the data exchange process. For less sensitive data, a higher privacy budget value is set.

6. The government data secure exchange and multimodal information extraction system as described in claim 5, characterized in that, The differential privacy protection submodule combines differential privacy with multimodal data, specifically including: Differential privacy protection requires differentiated processing based on the characteristics of different types of data. Text data: For text data containing sensitive information, differential privacy protection introduces noise to perturb the statistical results of the data, ensuring that after the data is exchanged, attackers cannot reconstruct the specific information of an individual based on the query results; Image data: For image data involving facial recognition and geographic information, differential privacy ensures that personal privacy is not leaked by blurring or de-identifying the image; image privacy protection also includes perturbing the detailed features in the image to prevent it from being used to reconstruct specific personal information; Audio data: Privacy protection for audio data is not limited to the voice content, but also includes background noise and audio length; when analyzing voice data, the system will process the pitch, intonation and speaker characteristics of the voice by adding noise perturbation to ensure that personal identity cannot be accurately reconstructed through audio analysis; Video data: When processing video data, differential privacy protection not only protects the privacy of each frame of the image, but also needs to interfere with the time dimension of the video with noise. Specifically, for video surveillance and court recording video data, the system will use inter-frame noise injection technology to disturb the detailed information of individual key frames while ensuring that the video content does not affect the overall analysis.

7. The government data secure exchange and multimodal information extraction system as described in claim 1, characterized in that, The secure transmission management submodule ensures the security of data transmission during the exchange process, specifically implemented as follows: TLS / SSL Encryption Protocol: The data security exchange module uses TLS or SSL to encrypt the data transmission channel; through the TLS / SSL protocol, data is encrypted during transmission; during each data exchange, the data security exchange module encrypts and protects the communication channel through the TLS protocol and performs two-way authentication before data exchange to ensure that data can only be transmitted between authorized sources and destinations; Authentication and access control: OAuth 2.0 and JWT are used for authentication and authorization; OAuth 2.0 is used for authentication to ensure that the system or user requesting data has legitimate access rights; JWT carries verification information in each data exchange to ensure that the transmitted data was indeed initiated by a legitimate identity. VPN tunneling technology: When data exchange needs to be conducted across multiple network areas or between government agencies, the data security exchange module also uses VPN tunneling technology to establish a private, encrypted channel; through VPN tunneling, the data exchange process is less susceptible to interference from external network attacks.

8. The government data secure exchange and multimodal information extraction system as described in claim 1, characterized in that, The multimodal information extraction and intelligent analysis module performs in-depth analysis of the exchanged business data, providing decision support and business insights. It can also analyze non-business data generated during the scheduling process and provide optimization strategies from the perspective of system operation and privacy protection. This module is deployed at the network center node and various switching nodes, forming a two-way collaborative mode; the multimodal information extraction and intelligent analysis module also includes: a data unified analysis and fusion sub-module: which integrates the intelligent analysis results of business data and system data, generates decision reports, provides optimization suggestions, and supports government decision-making, in-depth insights, and privacy protection strategies; The multimodal information extraction and analysis submodule for business data includes: Text data processing: Semantic understanding and information extraction are performed using a pre-trained language model based on the Transformer architecture; key entities in the case are automatically identified through analysis of the case text, and potential relationships between entities are further explored; Image data analysis: The Vision-and-Language Transformer model is adopted, which combines an image convolutional neural network (CNN) with a text processing module for cross-modal joint training. First, deep convolutional neural networks are used to extract image features, and OCR technology is used to extract text from the image. Then, through joint training, the model fuses image features and text information. Audio data processing: The audio is transcribed into text using an automatic speech recognition (ASR) system, and then combined with a text processing module for further natural language processing analysis; based on the Wav2Vec2 speech recognition model, the speech is converted into text, and sentiment analysis and keyword extraction are performed to capture the emotional information conveyed in the audio. Video data analysis: By using Transformer-based Video Recognition Models, key information about people, actions, and scenes in the video is extracted and combined with case text data; The non-business data extraction and analysis submodule optimizes the system, schedules resources, and adjusts the privacy budget through the analysis of non-business data; specifically, it includes: Scheduling metadata analysis: Metadata during the scheduling process is recorded in the form of logs or real-time metrics; Graph Neural Networks (GNNs) are used to model the data flow process, identify and optimize data exchange paths and nodes; By constructing a data flow graph, GNNs discover potential bottlenecks and inefficient paths, and propose dynamic scheduling strategies based on data traffic and transmission latency factors to optimize resource allocation; System log analysis: Real-time monitoring of abnormal behavior in logs and alarm processing; anomaly detection algorithms can promptly identify potential system faults or performance bottlenecks and trigger automated processing mechanisms. Combined with reinforcement learning algorithms, scheduling strategies are optimized and system operating parameters are automatically adjusted. Privacy Budget Optimization: By analyzing metadata and system logs during the scheduling process, we identify which data exchange operations involve sensitive data and dynamically adjust the privacy budget based on the sensitivity of the data. Through a dynamic privacy budget algorithm, we intelligently optimize privacy protection measures to ensure that data availability is maximized while protecting privacy. The unified data analysis and fusion submodule integrates data from different sources to generate comprehensive analysis results, supporting government decision-making and system optimization; specifically including: Multimodal data fusion: By adopting the CLIP multimodal learning framework, data from different modalities are fused; by linking image and text representations through a shared embedding space, knowledge from multiple data sources is effectively integrated. Knowledge Graph Construction and Analysis: Based on the extraction results of multimodal data, a unified knowledge graph is constructed, and graph analysis technology is used to perform in-depth data mining; the knowledge graph forms a relationship network by turning event and person information into nodes; Decision support and optimization suggestions: Based on the fused data, the system uses reinforcement learning and intelligent decision-making algorithms to generate decision reports; it provides data analysis results, risk assessments, and resource optimization suggestions to assist government decision-makers in formulating effective solutions.

9. A method for secure exchange of government data and extraction of multimodal information based on the system described in any one of claims 1-8, characterized in that, The method includes the following steps: Step S1: Initialize the data exchange task Establish the basic framework for data exchange, including clarifying the data source and target, defining task parameters, generating metadata, and determining privacy protection requirements; Step S2: Data Privacy Protection Processing First, based on the privacy budget and data types set in the task initialization phase, differential privacy protection operations are performed on different modalities of data. This process reduces the impact of individual data records on the overall analysis results by adding noise, blurring, or de-identifying data, thus ensuring data privacy and security. Step S3: Data Transmission and Scheduling The data is transmitted securely and efficiently from the source system to the target system after privacy protection and encryption. In this process, the data integration and scheduling module is responsible for dynamically planning the transmission path, optimizing resource scheduling, and monitoring the status and log information in real time to ensure the stability, efficiency and security of the data flow. Step S4: Data reception and multimodal information extraction The received data is decrypted, parsed, and multimodal information is extracted and analyzed. This stage transforms the multimodal data transmitted from the source system into structured information and provides valuable business decisions and insights through intelligent analysis. Step S5: Analysis Results Integration and Feedback Optimization The analysis results from the multimodal information extraction module are integrated with non-business data generated during system operation to generate business insights and optimization suggestions. At the same time, this step enables dynamic optimization of privacy budget allocation and transmission scheduling strategies, ensuring the continuous and efficient operation of the entire government affairs integration platform.

Images