Data acquisition method and semi-closed network data acquisition system

Abstract

The invention provides a data acquisition method and a semi-closed network data acquisition system, and the method comprises the steps: a server drives a client to execute mixed data acquisition by sending an SM2 encryption instruction containing a structured data source definition, an unstructured file path and a dynamic fragmentation threshold; the client converts the structured data into an ORC column format, after an unstructured file is fragmented according to a threshold value, an SM4 encryption memory is adopted to process and generate ciphertext fragments, and an SM3 abstract and an SM2 digital signature are added to form an encrypted data packet to be transmitted back; and the server decrypts the SM2 digital signature through the public key to verify the identity legality of the client, recalculates the decrypted fragmented SM3 abstract to perform integrity verification, and finally recombines the ORC data according to the fragmented number and decrypts and stores the ORC data. According to the method, through end-to-end data security protection and in combination with a dynamic fragmentation mechanism of ORC column storage, the transmission efficiency is improved while the data integrity is ensured, and unified and efficient processing of structured and unstructured data is supported.

Description

Technical Field

[0001] This application relates to the field of network security technology. Specifically, it relates to a data collection method and a semi-closed network data collection system. Background Art

[0002] In the wave of digital transformation, data collection technology, as a core part of information systems, faces multiple challenges such as security, efficiency, and the ability to process heterogeneous data.

[0003] Traditional collection systems are difficult to efficiently process structured data (such as database tables) and unstructured data (such as log files, images) simultaneously. Structured data is usually directly obtained through SQL queries, while unstructured data requires independent shard transmission, resulting in fragmented collection processes and wasted resources. In addition, existing sharding strategies mostly use fixed thresholds and cannot be dynamically adjusted according to the network environment, easily causing transmission delays or bandwidth waste. Summary of the Invention

[0004] The purpose of the embodiments of this application is to provide a data collection method and a semi-closed network data collection system, which can achieve unified scheduling of dual-modal data by defining collection instructions for structured data sources (databases / tables / fields) and unstructured paths (file system regular expressions).

[0005] A data collection method provided by the embodiments of this application, which is applied to a server, includes: Sending an encrypted collection task instruction to the client. The encrypted collection task instruction includes a structured data source definition, an unstructured file path, and a sharding threshold, so that the client collects data according to a preset rule; Receiving an encrypted shard data packet sent by the client. The encrypted shard data packet includes: a client ID, an SM2 digital signature, an SM4-encrypted ORC format data shard, and an SM3 digest; Performing client matching: decrypting the SM2 digital signature using a pre-stored public key and verifying the matching between the client ID and the task authorization list; After client matching, performing data integrity verification: calculating the SM3 digest for the decrypted ORC format data shard and comparing it with the SM3 digest submitted by the client; If the data integrity verification passes, reorganize and decrypt the data according to the shard number, generate a complete data set, and write it into the target storage.

[0006] Among them, the client ID: uniquely identifies the collection terminal.

[0007] The SM2 digital signature: The client uses the private key to sign "ID + timestamp + task serial number" to prevent replay attacks.

[0008] SM4-encrypted ORC shards: ciphertext data units.

[0009] SM3 digest: hash value of the original data shards.

[0010] In the above technical solution, the server drives the client to perform hybrid data collection by sending an SM2 encryption instruction containing the structured data source definition, unstructured file path, and dynamic sharding threshold; after the client converts the structured data into the ORC columnar format and shards the unstructured files according to the threshold, it uses SM4 encryption in-memory processing to generate ciphertext shards, and attaches the SM3 digest and SM2 digital signature to form an encrypted data packet for return; the server decrypts the SM2 digital signature with the public key to verify the legitimacy of the client's identity, and recalculates the SM3 digest of the decrypted shards for integrity verification, and finally reorganizes the ORC data according to the shard number and decrypts it for storage. This method realizes end-to-end data security protection through the full-stack use of national cryptography algorithms (SM2 authentication, SM4 encryption, SM3 verification). Combining with the dynamic sharding mechanism of ORC columnar storage, while ensuring data integrity, it improves the transmission efficiency by more than 40%, and supports the unified and efficient processing of structured and unstructured data, solving the technical bottlenecks of insufficient encryption strength, sharding redundancy, and fragmented hybrid data processing in traditional solutions, and is especially suitable for secure data aggregation in high-sensitivity scenarios such as government affairs and finance.

[0011] Specifically, the task instruction includes structured data sources (such as database name, table name, time range), unstructured file paths (supporting regular expression matching), and dynamic sharding thresholds (such as 64MB - 512MB). The server encrypts the instruction using the SM2 public key pre-stored by the client and sends it through the HTTPS channel to ensure the security of instruction transmission. The sharding threshold is dynamically adjusted according to the client's historical network performance (small shards are used for low bandwidth to reduce packet loss, and large shards are used for high bandwidth to increase throughput). This embodiment supports the unified scheduling of hybrid data types, avoiding the resource waste of separate processing of structured and unstructured data in traditional solutions. Moreover, the dynamic sharding strategy adapts to network fluctuations and reduces the transmission failure rate.

[0012] For structured data: Execute an SQL query with a time range, convert the result set to the ORC format, and cut it into multiple Stripe units according to a preset threshold. For unstructured data: Cut the binary file according to the sharding threshold and append the ORC file header (including metadata such as shard number and total number of shards). Encrypt the ORC shards using the SM4-CTR mode. The initialization vector (IV) is derived from the task ID and shard number through the KDF algorithm, and the in-memory plaintext is immediately destroyed after encryption. Calculate the SM3 hash digest for the original data shards to ensure data integrity. The ORC columnar storage in this embodiment significantly improves the compression ratio (on average reducing the storage space by 30% - 50%). Moreover, in-memory encryption achieves "zero plaintext on disk" to prevent data leakage caused by physical attacks or memory dumps.

[0013] The transmission of the encrypted shard data packet is through the HTTPS channel, with SM2 business layer encryption superimposed to form double protection of "channel + content".

[0014] In some alternative embodiments, the steps for generating the encrypted collection task instruction include: Receive externally input spatio-temporal constraint parameters, including the data collection time window, database table row filtering conditions, and file modification time range; Predict the optimal shard size based on the historical transmission rate. Structured data is sharded according to the ORC Stripe threshold, and unstructured files are cut according to the specified sharding threshold; After being encrypted using the target client's public key by SM2, it is sent down through the HTTPS channel.

[0015] In the above technical solution, by integrating spatio-temporal constraints, dynamic sharding, and national cryptography encryption technologies, a secure and efficient data collection process is achieved. Specifically: The server first receives externally input spatio-temporal constraint parameters (such as the data collection time window, database table row filtering conditions, and file modification time range) to accurately limit the data collection scope to reduce redundancy; Subsequently, based on the historical transmission rate, dynamically predict the optimal sharding threshold. For structured data, adopt the ORC Stripe sharding strategy (default 64MB - 256MB), and use the compression advantage of columnar storage to reduce the data volume by 40% - 60%. For unstructured files, cut them according to the specified threshold, and adaptively adjust the shard size in combination with the network bandwidth to improve the transmission throughput; Finally, encrypt the instruction through SM2 public key and send it down through the HTTPS channel, forming double encryption protection of "transport layer + business layer" to resist the risks of man-in-the-middle attacks and data tampering. The technical effects of this solution are remarkable: The dynamic sharding strategy improves the utilization rate of network bandwidth. The ORC format compression and SM2 encryption work together to reduce the storage overhead. The spatio-temporal constraint filters invalid data to reduce the server processing pressure. At the same time, the SM2 digital signature mechanism ensures the integrity of the instruction and the legitimacy of the client, achieving a balance between security and efficiency.

[0016] In some alternative embodiments, the method for generating an SM2 digital signature includes: The client uses the private key to perform an SM2 digital signature on the client ID, timestamp, and task serial number. After decryption, the server verifies the validity of the timestamp and checks whether the task serial number is executed repeatedly.

[0017] In the above technical solution, the client combines the client ID, timestamp, and task serial number into a signature data body and uses the private key to perform an SM2 digital signature. After receiving it, the server decrypts it with the public key and performs a dual verification: First, it checks whether the deviation between the timestamp and the current time is within a preset window (such as ±5 minutes) to prevent expired requests from being replayed; Second, it verifies the uniqueness of the task serial number in the server database to avoid repeated submission and execution of the same signature. This method combines dynamic aging control and uniqueness identification verification, and further enhances the anti-replay attack ability on the basis of the native security of the national cryptographic algorithm. In terms of technical effects, the timestamp mechanism can resist the risk of old data tampering caused by network latency, and the uniqueness check of the task serial number completely eliminates the forgery operation of malicious attackers by intercepting and repeatedly sending the same signature data packet.

[0018] In some alternative embodiments, the encryption process of ORC format data sharding includes: The client completes data assembly in memory, generates Stripe units after compressing in ORC format. Encrypt the Stripe data using the SM4-CTR mode, and the initialization vector is derived from the task ID and shard number; immediately destroy the plaintext in memory after encryption.

[0019] In the above technical solution, the client adopts memory-level security processing and dynamic key derivation technology: First, assemble structured or unstructured data into Stripe units (the smallest data storage unit of ORC, default 64MB - 256MB) in memory according to the ORC format, and use the columnar compression characteristics of ORC (such as Zlib or ZSTD algorithm) to compress the data to 40% - 60% of the original volume; Subsequently, use the SM4-CTR mode to perform streaming encryption on the Stripe data, where the initialization vector (IV) is generated through the task ID and shard number by the key derivation function (KDF) to ensure that the encryption key stream of each shard is unique and unpredictable; Immediately clear the plaintext data in memory after encryption, and only retain the encrypted binary stream to write to the disk or transmission channel.

[0020] In some alternative embodiments, when the client detects a transmission failure, it automatically switches to an alternative port for retransmission, and the retry interval increases according to the exponential backoff algorithm; The server records the shard reception status, generates incremental task instructions for the missing shards, and preferentially retransmits the missing parts.

[0021] In the above technical solution, the client and the server adopt a collaborative intelligent retransmission strategy: when the client detects a fragment transmission failure (such as continuous timeout or ACK unconfirmed), it automatically switches from the default port (such as 443) to the preset backup port queue (such as 8443 / 9443) to initiate retransmission, and dynamically adjusts the retry interval based on the exponential backoff algorithm (initial interval 1 second, maximum interval 64 seconds) to avoid network congestion; the server maintains the fragment reception status matrix in real time (using a bitmap to mark the received fragments), generates incremental retransmission instructions for missing fragments (only contains a list of missing fragment numbers), and prioritizes scheduling high-priority transmission queues.

[0022] A data collection method provided in an embodiment of the present application is applied to a client, including: Poll the server task queue through a one-way channel to obtain the encrypted collection task instructions encrypted by SM2; After decrypting the command, execute the acquisition task: For structured data, connect to the specified database, execute SQL queries with time ranges, and convert the result set into ORC format; for unstructured data, traverse the files under the specified path and cut them into binary blocks according to the sharding threshold to obtain ORC format data shards; Use SM4 to encrypt ORC format data shards and generate SM3 digests for the shards; According to the client ID, SM2 digital signature, SM4 encrypted ORC format data fragment and SM3 summary, an encrypted fragment data packet is obtained; the encrypted fragment data packet is sent to the server.

[0023] In the above technical solution, safe and efficient data collection is achieved through a hybrid data unified processing framework and a full-stack encryption technology of the national secret algorithm: the client actively polls the server through a one-way channel to obtain the SM2 encrypted collection instructions (avoiding the risk of server port exposure), and performs tasks in a modal manner after decryption - for structured data, SQL queries are executed based on the time window and converted into the ORC column storage format, and unstructured files are cut into binary blocks according to the preset threshold and encapsulated into ORC fragments; then the ORC fragments are encrypted in memory using the SM4-CTR mode (the key is derived from the task ID), and the SM3 summary is generated to ensure data integrity, and finally the client ID, SM2 digital signature (including timestamp anti-replay), encrypted fragments and summary are encapsulated as data packets and transmitted back to the server. The technical effect of this solution is significant: structured and unstructured data are uniformly processed through the ORC format to reduce metadata redundancy; SM4 memory encryption achieves zero plaintext disk, combined with SM2 instruction encryption to form end-to-end protection to resist man-in-the-middle attacks and data leakage risks; the SM3 summary mechanism ensures transmission integrity and reduces the error retransmission rate.

[0024] In some alternative embodiments, for structured data, connect to the specified database, execute an SQL query with a time range, and convert the result set to the ORC format, including: Adopt incremental collection for large data tables: record the last collection position, and the next task only queries WHERE update_time > last_sync; When converting to ORC in memory, perform SM3 hashing on predefined sensitive columns.

[0025] In the above technical solution, in the structured data processing method, the client optimizes data collection efficiency and security through a technology that combines incremental collection and desensitization of sensitive data: Incremental collection mechanism: By recording the last timestamp of the previous collection (such as last_sync), the client only executes an SQL query of WHERE update_time > last_sync, and only extracts newly added or updated data records, reducing the data transmission volume; ORC columnar conversion: Compress the query result set into Stripe units in ORC format in memory, and utilize the efficient compression characteristics of columnar storage (such as the Zlib or Snappy algorithm) to reduce the data volume, while columnar storage optimizes subsequent query performance; Desensitization of sensitive fields: In the memory processing stage, perform SM3 hashing on predefined sensitive columns (such as ID card numbers and mobile phone numbers) to generate irreversible digest values to replace the original data and prevent reverse cracking after data leakage.

[0026] A semi-closed network data collection system provided by an embodiment of the present application includes a server and a client; The server is used for: Send an encrypted collection task instruction to the client. The encrypted collection task instruction includes a structured data source definition, an unstructured file path, and a sharding threshold, so that the client collects data according to a preset rule; Receive the encrypted sharded data packet sent by the client; the encrypted sharded data packet includes: client ID, SM2 digital signature, SM4-encrypted ORC format data shard, and SM3 digest; Perform client matching: Decrypt the SM2 digital signature using the pre-stored public key, and verify the matching of the client ID with the task authorization list; After client matching, perform data integrity verification: Calculate the SM3 digest for the decrypted ORC format data shard and compare it with the SM3 digest submitted by the client; If the data integrity verification passes, reorganize and decrypt the data according to the shard number, and generate a complete data set and write it into the target storage.

[0027] The client is used for: Poll the server task queue through a unidirectional channel to obtain the encrypted acquisition task instruction encrypted by SM2; After decrypting the instruction, execute the acquisition task: For structured data, connect to the specified database, execute a SQL query with a time range, and convert the result set to the ORC format; for unstructured data, traverse the files under the specified path and cut them into binary blocks according to the sharding threshold; obtain the ORC format data shards; Encrypt the ORC format data shards using SM4 and generate the shard SM3 digest; According to the client ID, SM2 digital signature, SM4-encrypted ORC format data shards, and SM3 digest, obtain the encrypted shard data packet; send the encrypted shard data packet to the server.

[0028] An electronic device provided by an embodiment of the present application includes: a processor and a memory, the memory stores machine-readable instructions executable by the processor, and when the machine-readable instructions are executed by the processor, the method described in any one of the above is executed.

[0029] A computer program product provided by an embodiment of the present application includes a computer program / instructions, and when the computer program / instructions are executed by a processor, the steps of the method described in any one of the above are implemented. Description of the Drawings

[0030] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following will briefly introduce the drawings required to be used in the embodiments of the present application. It should be understood that the following drawings only show some embodiments of the present application, and therefore should not be regarded as limiting the scope. For those of ordinary skill in the art, without creative efforts, other related drawings can also be obtained based on these drawings.

[0031] Figure 1 It is a flow chart of the steps of a data acquisition method provided by an embodiment of the present application applied to the server; Figure 2 It is a flow chart of the steps of a data acquisition method provided by an embodiment of the present application applied to the client; Figure 3 It is a possible structural schematic diagram of an electronic device provided by an embodiment of the present application. Detailed Embodiments

[0032] The following will describe the technical solutions in the embodiments of the present application with reference to the drawings in the embodiments of the present application.

[0033] Please refer to Figure 1 , a data acquisition method provided by an embodiment of the present application, applied to the server, includes: Step S1: Send an encrypted data collection task instruction to the client. The encrypted data collection task instruction includes the structured data source definition, the unstructured file path, and the sharding threshold, so that the client can collect data according to the preset rules; Step S2: Receive the encrypted sharded data packets sent by the client. The encrypted sharded data packets include: client ID, SM2 digital signature, SM4-encrypted ORC format data shards, and SM3 digest; Step S3: Perform client matching: Use the pre-stored public key to decrypt the SM2 digital signature and verify the matching between the client ID and the task authorization list; Step S4: After client matching, perform data integrity verification: Calculate the SM3 digest for the decrypted ORC format data shards and compare it with the SM3 digest submitted by the client; Step S5: If the data integrity verification passes, reorganize and decrypt the data according to the shard number, generate a complete data set, and write it to the target storage.

[0034] Among them, the client ID: uniquely identifies the collection terminal.

[0035] SM2 digital signature: The client uses the private key to sign "ID + timestamp + task serial number" to prevent replay attacks.

[0036] SM4-encrypted ORC shards: Ciphertext data units.

[0037] SM3 digest: The hash value of the original data shards.

[0038] In the embodiments of the present application, the server drives the client to perform hybrid data collection by sending an SM2 encrypted instruction including the structured data source definition, the unstructured file path, and the dynamic sharding threshold; after the client converts the structured data into the ORC columnar format and shards the unstructured file according to the threshold, it uses SM4 encryption for in-memory processing to generate ciphertext shards, and attaches the SM3 digest and the SM2 digital signature to form an encrypted data packet for backhaul; the server decrypts the SM2 digital signature with the public key to verify the legitimacy of the client identity, and recalculates the SM3 digest of the decrypted shards for integrity verification, and finally reorganizes and decrypts the ORC data according to the shard number for storage. This method realizes end-to-end data security protection through the full-stack application of national cryptographic algorithms (SM2 authentication, SM4 encryption, SM3 verification). Combining with the dynamic sharding mechanism of ORC columnar storage, while ensuring data integrity, it improves the transmission efficiency by more than 40%, and supports the unified and efficient processing of structured and unstructured data, solving the technical bottlenecks of insufficient encryption strength, sharding redundancy, and fragmented processing of hybrid data in traditional solutions, and is especially suitable for secure data aggregation in highly sensitive scenarios such as government affairs and finance.

[0039] Specifically, the task instruction includes a structured data source (such as database name, table name, time range), an unstructured file path (supporting regular expression matching), and a dynamic sharding threshold (such as 64MB - 512MB). The server encrypts the instruction using the SM2 public key pre-stored by the client and distributes it through the HTTPS channel to ensure the security of instruction transmission. The sharding threshold is dynamically adjusted according to the client's historical network performance (small shards are used for low bandwidth to reduce packet loss, and large shards are used for high bandwidth to improve throughput). This embodiment supports the unified scheduling of mixed data types, avoiding the resource waste caused by the separate processing of structured and unstructured data in traditional solutions. Moreover, the dynamic sharding strategy adapts to network fluctuations and reduces the transmission failure rate.

[0040] For structured data: Execute an SQL query with a time range, convert the result set to the ORC format, and cut it into multiple Stripe units according to a preset threshold. For unstructured data: Cut the binary file according to the sharding threshold and append an ORC file header (including metadata such as shard number and total number of shards). Encrypt the ORC shards using the SM4-CTR mode. The initialization vector (IV) is derived from the task ID and shard number through the KDF algorithm, and the in-memory plaintext is immediately destroyed after encryption. Calculate the SM3 hash digest for the original data shards to ensure data integrity. The ORC columnar storage in this embodiment significantly improves the compression ratio (on average reducing the storage space by 30% - 50%). Moreover, in-memory encryption achieves "zero plaintext on disk", preventing data leakage caused by physical attacks or memory dumps.

[0041] The transmission of the encrypted shard data packet is through the HTTPS channel, with SM2 business layer encryption superimposed to form double protection of "channel + content".

[0042] In some alternative embodiments, the steps for generating the encrypted collection task instruction include: Receiving externally input spatio-temporal constraint parameters, including the data collection time window, the database table row filtering condition, and the file modification time range; Predicting the optimal shard size based on the historical transmission rate, sharding the structured data according to the ORC Stripe threshold, and cutting the unstructured file according to the specified sharding threshold; After performing SM2 encryption using the public key of the target client, it is distributed through the HTTPS channel.

[0043] In the embodiments of the present application, by integrating spatio-temporal constraints, dynamic sharding, and national cryptography encryption technologies, a secure and efficient data collection process is achieved. Specifically: The server first receives externally input spatio-temporal constraint parameters (such as data collection time window, database table row filtering conditions, file modification time range) to accurately limit the data collection scope and reduce redundancy; subsequently, based on the historical transmission rate, the optimal sharding threshold is dynamically predicted, and for structured data, the ORC Stripe sharding strategy (default 64MB - 256MB) is adopted, using the compression advantage of columnar storage to reduce the data volume by 40% - 60%. For unstructured files, they are cut according to the specified threshold, and the sharding size is adaptively adjusted in combination with the network bandwidth to improve the transmission throughput; finally, the SM2 public key encryption instruction is issued and superimposed on the HTTPS channel to form a double encryption protection of "transport layer + service layer" to resist man-in-the-middle attacks and data tampering risks. The technical effects of this solution are remarkable: The dynamic sharding strategy improves the network bandwidth utilization rate, the ORC format compression and SM2 encryption cooperate to reduce the storage overhead, the spatio-temporal constraint filters invalid data and reduces the server processing pressure. At the same time, the SM2 digital signature mechanism ensures the integrity of the instruction and the legitimacy of the client, achieving a balance between security and efficiency.

[0044] In some alternative embodiments, the method for generating an SM2 digital signature includes: The client uses the private key to perform an SM2 digital signature on the client ID, timestamp, and task serial number. After the server decrypts it, it verifies the validity of the timestamp and checks whether the task serial number is executed repeatedly.

[0045] In the embodiments of the present application, the client combines the client ID, timestamp, and task serial number into a signature data body, performs an SM2 digital signature using the private key, and after the server receives it, decrypts it with the public key and performs a double verification: First, check whether the deviation between the timestamp and the current time is within a preset window (such as ±5 minutes) to prevent expired requests from being replayed; second, verify the uniqueness of the task serial number in the server database to avoid the same signature being submitted and executed repeatedly. This method combines dynamic aging control and uniqueness identification verification, and further enhances the anti-replay attack ability on the basis of the native security of the national cryptography algorithm. In terms of technical effects, the timestamp mechanism can resist the risk of old data tampering caused by network latency, and the uniqueness check of the task serial number completely prevents malicious attackers from forging operations by intercepting and repeatedly sending the same signature data packet.

[0046] In some alternative embodiments, the encryption process of ORC format data sharding includes: The client completes data assembly in memory, generates Stripe units after compressing in ORC format. Encrypt Stripe data using the SM4-CTR mode. The initialization vector is derived from the task ID and shard number, and the plaintext in memory is destroyed immediately after encryption.

[0047] In the embodiments of this application, the client adopts in-memory security processing and dynamic key derivation technology: First, in memory, structured or unstructured data is assembled into Stripe units in ORC format (the smallest data storage unit of ORC, default 64MB - 256MB). The data is compressed to 40% - 60% of the original volume using the columnar compression feature of ORC (such as the Zlib or ZSTD algorithm). Subsequently, the SM4-CTR mode is used to perform streaming encryption on the Stripe data. The initialization vector (IV) is generated through the task ID and shard number by a key derivation function (KDF) to ensure that the encryption key stream for each shard is unique and unpredictable. After encryption, the plaintext data in memory is cleared immediately, and only the encrypted binary stream is retained and written to disk or the transmission channel.

[0048] In some optional embodiments, when the client detects a transmission failure, it automatically switches to an alternate port for retransmission, and the retry interval increases according to the exponential backoff algorithm. The server records the shard reception status, generates incremental task instructions for missing shards, and preferentially retransmits the missing parts.

[0049] In the embodiments of this application, the client and the server adopt a collaborative intelligent retransmission strategy: When the client detects a shard transmission failure (such as consecutive timeouts or unacknowledged ACKs), it automatically switches from the default port (such as 443) to the pre-configured alternate port queue (such as 8443 / 9443) to initiate retransmission, and dynamically adjusts the retry interval based on the exponential backoff algorithm (initial interval 1 second, maximum interval 64 seconds) to avoid network congestion. The server maintains a shard reception status matrix in real-time (using a bitmap to mark received shards), generates incremental retransmission instructions for missing shards (only including the list of missing shard numbers), and preferentially schedules the high-priority transmission queue.

[0050] Please refer to Figure 2 , a data acquisition method provided by the embodiments of this application, which is applied to the client and includes: Step S6: Poll the server task queue through a one-way channel to obtain the encrypted acquisition task instructions encrypted by SM2. Step S7: After decrypting the instructions, execute the acquisition task: For structured data, connect to the specified database, execute a SQL query with a time range, and convert the result set to ORC format; for unstructured data, traverse the files under the specified path and cut them into binary blocks according to the shard threshold; obtain the ORC format data shards. Step S8: Encrypt the ORC format data shards using SM4 and generate a shard SM3 digest. Step S9: Obtain an encrypted shard data packet based on the client ID, SM2 digital signature, SM4-encrypted ORC format data shards, and SM3 digest; send the encrypted shard data packet to the server.

[0051] In the embodiments of the present application, secure and efficient data collection is achieved through a hybrid data unified processing framework and a full-stack encryption technology of national cryptography algorithms: The client actively polls the server through a one-way channel to obtain the SM2-encrypted collection instructions (to avoid the risk of server port exposure), decrypts them and executes tasks in different modes - for structured data, perform a SQL query based on a time window and convert it to the ORC columnar storage format, and cut unstructured files into binary blocks according to a preset threshold and encapsulate them as ORC shards; then encrypt the ORC shards in the SM4-CTR mode in memory (the key is derived from the task ID), generate an SM3 digest to ensure data integrity, and finally encapsulate the client ID, SM2 digital signature (including a timestamp to prevent replay), encrypted shards, and digest into a data packet and send it back to the server. The technical effects of this solution are remarkable: Structured and unstructured data are uniformly processed in the ORC format, reducing metadata redundancy; SM4 in-memory encryption achieves zero plaintext disk write, combined with SM2 instruction encryption to form end-to-end protection, resisting the risks of man-in-the-middle attacks and data leakage; the SM3 digest mechanism ensures transmission integrity and reduces the error retransmission rate.

[0052] In some alternative embodiments, for structured data, connect to a specified database, execute a SQL query with a time range, and convert the result set to the ORC format, including: Adopt incremental collection for large data tables: Record the last collection position, and in the next task, only query WHERE update_time > last_sync; When converting to ORC in memory, perform SM3 hashing on predefined sensitive columns.

[0053] In the embodiments of the present application, in the structured data processing method, the client optimizes data collection efficiency and security through a technology that combines incremental collection and sensitive data desensitization: Incremental collection mechanism: By recording the last timestamp of the previous collection (such as last_sync), the client only executes a SQL query of WHERE update_time > last_sync, and only extracts newly added or updated data records, reducing the data transmission volume; ORC columnar conversion: Compress the query result set into Stripe units in the ORC format in memory, and utilize the efficient compression characteristics of columnar storage (such as the Zlib or Snappy algorithm) to reduce the data volume, while columnar storage optimizes subsequent query performance; Sensitive field desensitization: During the in-memory processing stage, perform SM3 hashing operations on predefined sensitive columns (such as ID card numbers and mobile phone numbers) to generate irreversible digest values to replace the original data and prevent reverse cracking after data leakage.

[0054] A semi-closed network data acquisition system provided by an embodiment of the present application includes a server and a client. The server is used for: Send an encrypted acquisition task instruction to the client. The encrypted acquisition task instruction includes a structured data source definition, an unstructured file path, and a sharding threshold, so that the client acquires data according to a preset rule. Receive the encrypted sharded data packet sent by the client. The encrypted sharded data packet includes: client ID, SM2 digital signature, SM4-encrypted ORC format data shard, and SM3 digest. Perform client matching: Use the pre-stored public key to decrypt the SM2 digital signature and verify the matching of the client ID with the task authorization list. After client matching, perform data integrity verification: Calculate the SM3 digest for the decrypted ORC format data shard and compare it with the SM3 digest submitted by the client. If the data integrity verification passes, reorganize and decrypt the data according to the shard number, generate a complete data set, and write it to the target storage.

[0055] The client is used for: Poll the server task queue through a one-way channel to obtain the SM2-encrypted encrypted acquisition task instruction. After decrypting the instruction, execute the acquisition task: For structured data, connect to the specified database, execute an SQL query with a time range, and convert the result set to the ORC format; for unstructured data, traverse the files under the specified path and cut them into binary blocks according to the sharding threshold; obtain the ORC format data shards. Encrypt the ORC format data shard using SM4 and generate a shard SM3 digest. According to the client ID, SM2 digital signature, SM4-encrypted ORC format data shard, and SM3 digest, obtain the encrypted sharded data packet; send the encrypted sharded data packet to the server.

[0056] Figure 3 Shows a possible structure of the electronic device provided by an embodiment of the present application. Refer to Figure 3 , the electronic device includes: a processor, a memory, and a communication interface. These components are interconnected and communicate with each other through a communication bus and / or other forms of connection mechanisms (not shown).

[0057] Among them, the memory includes one or more (only one is shown in the figure), which can be, but is not limited to, random access memory (RAM), read only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), etc. The processor and other possible components can access the memory, read and / or write the data therein.

[0058] The processor includes one or more (only one is shown in the figure), which can be an integrated circuit chip with signal processing capabilities. The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a microcontroller unit (MCU), a network processor (NP), or other conventional processors; it can also be a dedicated processor, including a neural-network processing unit (NPU), a graphics processing unit (GPU), a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components. And when there are multiple processors, a part of them can be general-purpose processors and another part can be dedicated processors.

[0059] The communication interface includes one or more (only one is shown in the figure), which can be used to communicate directly or indirectly with other devices for data interaction. The communication interface can include interfaces for wired and / or wireless communication.

[0060] One or more computer program instructions can be stored in the memory, and the processor can read and run these computer program instructions to implement the method provided in the embodiments of the present application.

[0061] It can be understood that Figure 3 the structure shown is only illustrative, and the electronic device may further include more or fewer components than those shown in Figure 3 or have a structure different from that shown in Figure 3 . Figure 3 Each component shown in can be implemented by hardware, software, or a combination thereof. The electronic device may be a physical device, such as a PC, laptop, tablet, mobile phone, server, embedded device, etc., or a virtual device, such as a virtual machine, virtualization container, etc. Moreover, the electronic device is not limited to a single device, and may also be a combination of multiple devices or a cluster composed of a large number of devices.

[0062] A computer program product provided by an embodiment of the present application includes a computer program / instructions, and when the computer program / instructions are executed by a processor, the steps of any of the above methods are implemented.

[0063] In the embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are merely illustrative. For example, the division of the units is only a logical function division, and there may be other division methods in actual implementation. For another example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the displayed or discussed coupling or direct coupling or communication connection between each other can be through some communication interfaces, and the indirect coupling or communication connection of the devices or units can be in electrical, mechanical or other forms.

[0064] In addition, the units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place, or may be distributed to multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0065] Furthermore, in each embodiment of the present application, the various functional modules can be integrated together to form an independent part, or each module can exist alone, or two or more modules can be integrated to form an independent part.

[0066] In this article, relational terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations.

[0067] The above are only embodiments of the present application and are not intended to limit the protection scope of the present application. For those skilled in the art, various changes and modifications can be made to the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included within the protection scope of the present application.

Claims

1. A data collection method, characterized in that: Applied to the server, including: Sending an encrypted collection task instruction to the client; the encrypted collection task instruction includes: a structured data source definition, an unstructured file path and a sharding threshold, so that the client collects data according to a preset rule; Receive an encrypted fragmented data packet sent by the client; the encrypted fragmented data packet includes: client ID, SM2 digital signature, SM4 encrypted ORC format data fragment and SM3 summary; Perform client matching: use the pre-stored public key to decrypt the SM2 digital signature and verify the matching of the client ID with the task authorization list; After the client matches, a data integrity check is performed: the SM3 digest is calculated for the decrypted ORC format data fragments and compared with the SM3 digest submitted by the client; If the data integrity check passes, the data is reassembled and decrypted according to the shard number to generate a complete data set and write it to the target storage.

2. The method according to claim 1, characterized in that The step of generating the encrypted acquisition task instruction comprises: Receive external input of time and space constraint parameters, including data collection time window, database table row filtering conditions, and file modification time range; The optimal shard size is predicted based on the historical transfer rate. Structured data is sharded according to the ORC Stripe threshold, and unstructured files are cut according to the specified shard threshold. After SM2 encryption using the target client public key, it is delivered through the HTTPS channel.

3. The method according to claim 1, characterized in that The method for generating the SM2 digital signature includes: The client uses the private key to perform SM2 digital signature on the client ID, timestamp and task serial number; After decryption, the server verifies the validity of the timestamp and checks whether the task serial number is executed repeatedly.

4. The method according to claim 1, characterized in that: The encryption process of the ORC format data fragments includes: The client completes data assembly in memory and generates Stripe units after compression in ORC format; Stripe data is encrypted using SM4-CTR mode, with the initialization vector derived from the task ID and shard number; the plaintext in memory is destroyed immediately after encryption.

5. The method according to any one of claims 1 to 4, characterized in that: When the client detects a transmission failure, it automatically switches to the backup port for retransmission, and the retry interval increases according to the exponential backoff algorithm; the server records the fragment reception status, generates incremental task instructions for the missing fragments, and gives priority to retransmitting the missing parts.

6. A data collection method, characterized in that: Applied to the client, including: Poll the server task queue through a one-way channel to obtain the encrypted collection task instructions encrypted by SM2; After decrypting the command, execute the acquisition task: For structured data, connect to the specified database, execute SQL queries with time ranges, and convert the result set into ORC format; for unstructured data, traverse the files under the specified path and cut them into binary blocks according to the sharding threshold to obtain ORC format data shards; Encrypt the ORC format data fragment using SM4 and generate a fragment SM3 digest; According to the client ID, SM2 digital signature, SM4 encrypted ORC format data fragment and SM3 summary, an encrypted fragment data packet is obtained; and the encrypted fragment data packet is sent to the server.

7. The method according to claim 6, characterized in that For structured data, the specified database is connected, an SQL query with a time range is executed, and the result set is converted to ORC format, including: Use incremental collection for large data tables: record the last collection location, and the next task only queries WHERE update_time >last_sync; When performing in-memory conversion of ORC, SM3 hashing is performed on predefined sensitive columns.

8. A semi-closed network data acquisition system, characterized in that: Including server and client; The server is used to: Sending an encrypted collection task instruction to the client, wherein the encrypted collection task instruction includes: a structured data source definition, an unstructured file path, and a sharding threshold, so that the client collects data according to a preset rule; Receive an encrypted fragmented data packet sent by the client; the encrypted fragmented data packet includes: client ID, SM2 digital signature, SM4 encrypted ORC format data fragment and SM3 summary; Perform client matching: use the pre-stored public key to decrypt the SM2 digital signature and verify the matching of the client ID with the task authorization list; After the client matches, a data integrity check is performed: the SM3 digest is calculated for the decrypted ORC format data fragments and compared with the SM3 digest submitted by the client; If the data integrity check passes, the data is reassembled and decrypted according to the shard number to generate a complete data set and write it to the target storage; The client is used to: Poll the server task queue through a one-way channel to obtain the encrypted collection task instructions encrypted by SM2; After decrypting the command, execute the acquisition task: For structured data, connect to the specified database, execute SQL queries with time ranges, and convert the result set into ORC format; for unstructured data, traverse the files under the specified path and cut them into binary blocks according to the sharding threshold to obtain ORC format data shards; Encrypt the ORC format data fragment using SM4 and generate a fragment SM3 digest; According to the client ID, SM2 digital signature, SM4 encrypted ORC format data fragment and SM3 summary, an encrypted fragment data packet is obtained; and the encrypted fragment data packet is sent to the server.

9. An electronic device, characterized in that: include: A processor and a memory, wherein the memory stores machine-readable instructions executable by the processor, and when the machine-readable instructions are executed by the processor, the method according to any one of claims 1 to 7 is performed.

10. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instructions are executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images