Data processing method and device, electronic equipment and storage medium

Abstract

The invention provides a data processing method and device, electronic equipment and a storage medium, and the method comprises the steps: carrying out the data division of to-be-processed data according to a set rule in response to the obtained to-be-processed data; wherein the set rule comprises the steps of dividing the to-be-processed data according to a time dimension and a space dimension, and identifying the to-be-processed data according to a source of the to-be-processed data; performing data storage on the divided data to form storage data; in response to a transmission request for the storage data, determining to-be-transmitted data corresponding to the transmission request in the storage data; and performing data transmission on the to-be-transmitted data according to a set transmission rule.

Description

Technical Field

[0001] This disclosure relates to the field of electronic computer data processing technology, and in particular to a data processing method, apparatus, electronic device and storage medium. Background Technology

[0002] With the deepening of intelligent network construction, different fields have been impacted to varying degrees during application. Taking the power system as an example, the spatiotemporal data generated during the production and operation of the power system is experiencing explosive growth. This data mainly includes: drone inspection images, remote sensing satellite data, power grid equipment vector data, three-dimensional power facility models, and various equipment ledger data, etc.

[0003] It can be seen that these data are characterized by multiple sources, heterogeneity, and massive volume, posing a huge challenge to traditional power GIS (Geographic Information System) systems when processing such data.

[0004] It should be noted that the information disclosed in the background section above is only used to enhance the understanding of the background of this disclosure, and therefore may include information that does not constitute prior art known to those skilled in the art. Summary of the Invention

[0005] In view of this, the present disclosure provides a data processing method, apparatus, electronic device, and storage medium to solve or partially solve the above-mentioned problems.

[0006] For the purposes described above, this disclosure provides a data processing method, including: In response to the acquisition of data to be processed, the data to be processed is divided according to the set rules; wherein the set rules include dividing the data to be processed according to the time dimension and the spatial dimension, and identifying the data to be processed according to the source of the data to be processed; The partitioned data is then stored to form stored data. In response to a transmission request for the stored data, determine the data to be transmitted corresponding to the transmission request from the stored data; The data to be transmitted is transmitted according to the set transmission rules.

[0007] In some exemplary embodiments, the step of dividing the data to be processed according to a set rule includes: According to the time division rules, the data to be processed is divided into at least two time dimensions; Determine the data type of the data to be processed, determine the indexing algorithm for the spatial dimensions based on the data type, and divide the data to be processed according to the indexing algorithm; The data to be processed is divided into time dimension, spatial dimension, and then into the identifier, and an index of at least three levels is established.

[0008] In some exemplary embodiments, after forming the stored data, the method further includes: In response to an update operation on the stored data, determine the data to be updated; The metadata and indexes at each level corresponding to the updated data are updated.

[0009] In some exemplary embodiments, the storage of the partitioned data includes: Determine the data type and size of the partitioned data; The storage method of the partitioned data is determined based on the data type and / or the data size.

[0010] In some exemplary embodiments, determining the storage method of the partitioned data includes: Since the data type of the partitioned data is structured data, a relational database is used to store the partitioned data; Since the data type of the partitioned data is metadata, the partitioned data is stored using a set metadata cache pool; In response to the fact that the data after division is fragmented, a merging tool is used to merge the divided data, and the merged data is cached locally. If the size of the partitioned data exceeds a first threshold, the partitioned data is stored using a distributed file system. If the size of the partitioned data is less than a second threshold, the partitioned data is stored in a key-value pair database.

[0011] In some exemplary embodiments, the step of transmitting the data to be transmitted according to a set transmission rule includes: The data to be transmitted is divided into blocks and encrypted to form at least one data block and a transmission list; A transmission link is determined to transmit the transmission list. The transmission link transmits the at least one data block at a time, so that the receiving end can perform statistics and decryption on the received at least one data block according to the transmission list.

[0012] In some exemplary embodiments, the transmission link includes at least one primary link and at least one backup link. When the packet loss rate of any primary link exceeds a set threshold, a switch is performed between the primary link and the at least one backup link.

[0013] Based on the same concept, this disclosure also provides a data processing apparatus, including: The first module, in response to acquiring data to be processed, divides the data to be processed according to a set rule; wherein the set rule includes dividing the data to be processed according to the time dimension and the spatial dimension, and identifying the data to be processed according to the source of the data to be processed; The second module stores the partitioned data to form stored data. The third module, in response to a transmission request for the stored data, determines the data to be transmitted corresponding to the transmission request in the stored data; The fourth module transmits the data to be transmitted according to the set transmission rules.

[0014] Based on the same concept, this disclosure also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the computer program, implements the method as described in any of the preceding claims.

[0015] Based on the same concept, this disclosure also provides a non-transitory computer-readable storage medium storing computer instructions for causing a computer to perform the method as described in any of the preceding claims.

[0016] As can be seen from the above, this disclosure provides a data processing method, apparatus, electronic device, and storage medium. The method includes: in response to acquiring data to be processed, dividing the data to be processed according to a set rule; wherein the set rule includes dividing the data to be processed according to a time dimension and a spatial dimension, and identifying the data to be processed according to its source; storing the divided data to form stored data; in response to a transmission request for the stored data, determining the data to be transmitted corresponding to the transmission request in the stored data; and transmitting the data to be transmitted according to the set transmission rule. This disclosure establishes a unified, deep, composite spatiotemporal index for all types of data to be processed, specifically by associating the time and spatial attributes of the data, and further classifying and dividing it according to the data type identifier, thereby establishing a multi-level index structure to shorten data retrieval time and transforming complex multi-condition joint queries into efficient index key-value queries and set operations. The query response time is improved from minutes in traditional solutions to seconds or even sub-seconds, improving retrieval efficiency by 10-100 times, making real-time interactive big data analysis possible, which was previously impossible. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in the embodiments or related technologies of this disclosure, the accompanying drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the accompanying drawings described below are only embodiments of this disclosure. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 A flowchart illustrating an exemplary method provided in an embodiment of this disclosure.

[0019] Figure 2 A schematic diagram of the structure of an exemplary device provided in an embodiment of this disclosure.

[0020] Figure 3 This is a schematic diagram of the electronic device structure provided in an embodiment of this disclosure. Detailed Implementation

[0021] To make the objectives, technical solutions, and advantages of this specification clearer, the following detailed description is provided in conjunction with specific embodiments and the accompanying drawings.

[0022] It should be noted that, unless otherwise defined, the technical or scientific terms used in the embodiments of this disclosure should have the ordinary meaning understood by one of ordinary skill in the art to which this disclosure pertains. The terms "first," "second," and similar terms used in the embodiments of this disclosure do not indicate any order, quantity, or importance, but are merely used to distinguish different components. Terms such as "comprising" or "including" mean that the element, object, or method step preceding the term covers the element, object, or method step listed after the term and its equivalents, but does not exclude other elements, objects, or method steps. Terms such as "connected" or "linked" are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. Terms such as "upper," "lower," "left," and "right" are only used to indicate relative positional relationships; when the absolute position of the described object changes, the relative positional relationship may also change accordingly.

[0023] As described in the background section, taking the power system as an example, in the power industry, the management and application of spatiotemporal data are directly related to the safe operation and intelligent level of the power grid. How to efficiently manage and retrieve this multi-source heterogeneous data, and achieve its secure and efficient transmission between the Internet and the power information network, has become a key technical challenge in the current digital transformation of the power industry.

[0024] In some embodiments, data partitioning and indexing are often designed for single-type sensing data. For example, the method of indexing remote sensing images in time and space only applies to remote sensing image data. It extracts the image capture timestamp and latitude / longitude range, and adopts a two-level index structure of "time slice + R-tree"—first dividing the time slices by quarter, and then constructing an R-tree spatial index for the images within each time slice according to latitude and longitude. This scheme only supports one data type of remote sensing image and cannot adapt to the indexing requirements of vector data (which requires associating with a set of coordinate points) and 3D model data (which requires associating with spatial bounding boxes).

[0025] As can be seen, in the above embodiments, only an effective spatial index (R-tree) was established for vector data, while a deep index was lacking for the file data (images, models) that constitute the vast majority of the data volume. The index is actually built on the file's metadata, not the file content itself. When the number of files reaches tens of millions, performing spatiotemporal filtering on massive amounts of metadata in the database becomes a performance bottleneck. More seriously, because the indexing methods for different types of data are fragmented (R-tree for vectors, database queries for files), when performing joint queries on multi-source data, the system needs to execute multiple queries and aggregate the results at the application layer, generating significant network I / O and computational overhead. This results in extremely slow query response times, making it difficult to meet the real-time requirements of business scenarios (such as emergency command).

[0026] In light of the above-mentioned practical situation, this disclosure provides a data processing method. This disclosure establishes a unified, deep, composite spatiotemporal index for all types of data to be processed. Specifically, it associates the temporal and spatial attributes of the data and further classifies it according to data type identifiers, thereby establishing a multi-level index structure to shorten data retrieval time. Complex multi-condition joint queries are transformed into efficient index key-value queries and set operations. Query response time is improved from minutes in traditional solutions to seconds or even sub-seconds, increasing retrieval efficiency by 10-100 times, making real-time interactive big data analysis possible—previously impossible.

[0027] Figure 1 A flowchart illustrating an exemplary method provided by an embodiment of this disclosure is shown.

[0028] like Figure 1 As shown in the embodiments of this disclosure, the data processing method specifically includes the following steps.

[0029] Step 102: In response to obtaining the data to be processed, the data to be processed is divided according to the set rules; wherein the set rules include dividing the data to be processed according to the time dimension and the spatial dimension, and identifying the data to be processed according to the source of the data to be processed.

[0030] In this step, the data to be processed can be relevant data that requires data classification, storage, and transmission. In specific power system scenarios, this data can be remote sensing data, UAV data, vector data, 3D model data, etc. This data is collected and acquired through corresponding devices in the perception layer. Specifically, remote sensing / UAV data can include: acquisition timestamps (accurate to the second), spatial latitude and longitude ranges (e.g., 30°-31°N, 120°-121°E), resolution, etc.; vector data can include: coordinate point sets (e.g., power pole coordinates (x1, y1, z1), (x2, y2, z2)), update timestamps, etc.; 3D model data can include: spatial bounding boxes (minimum / maximum x / y / z coordinates), modeling timestamps, etc. Finally, the acquired metadata can be formatted according to subsequent classification rules, such as standardizing the metadata into a "time-space-data type" field format according to set rules, and then storing it in the metadata database.

[0031] After identifying the data to be processed, it can be divided, filtered, and indexed according to the established rules to facilitate subsequent storage and querying. In this embodiment, the data is divided at least from three levels: time, space, and data source. For example, the time dimension can be layered by "day-hour," such as "20250916-14" (14:00 on September 16, 2025), with metadata from the same time layer grouped into the same time bucket. Of course, in other embodiments, it can also be layered by "week-day" (suitable for low-frequency updated data, such as monthly remote sensing imagery), or by "hour-minute" (suitable for high-frequency updated data, such as real-time UAV inspection data), etc. The granularity of the layering can be dynamically adjusted according to the data update frequency. Then, the spatial dimension can be indexed according to the data type—R-tree indexing is used for remote sensing / UAV data (area spatial range), quadtree indexing is used for vector data (point / line coordinates), and Octree indexing is used for 3D model data (spatial bounding boxes), etc. Alternatively, a KD-tree (a k-dimensional tree, a data structure for partitioning k-dimensional data space) could be used instead of a quadtree (suitable for low-dimensional coordinate indexing of vector data), or... A tree (an improved variant of the R-tree, designed to optimize spatial indexing performance, especially in high-dimensional data scenarios) replaces the R-tree (suitable for high-dimensional spatial range indexing of remote sensing data). Both can achieve spatial positioning of multiple types of data, requiring only adjustments to the dimensional partitioning rules during index construction. Finally, for data type identifiers, a unique code can be assigned to each type of data (e.g., remote sensing = R, UAV = U, vector = V, 3D model = M, etc.) as an index. Of course, the above embodiments are only examples of one or several scenarios. In other implementation scenarios, specific adjustments can be made to the time dimension, spatial dimension, type dimension, etc., depending on the specific scenario. That is, in some embodiments, the data partitioning of the data to be processed according to the set rules includes: partitioning the data to be processed according to the time partitioning rules, according to at least two levels of time dimensions; determining the data type of the data to be processed, determining the spatial dimension indexing algorithm according to the data type, partitioning the data to be processed according to the indexing algorithm; and establishing at least three levels of indexes for the data to be processed, from time dimension partitioning to spatial dimension partitioning and then to the identifier. The time segmentation rules can be based on the aforementioned hierarchical structure such as "day-hour" or "week-day," with at least two time dimensions corresponding to days and hours, and weeks and days, respectively. Subsequently, at least three levels of indexes are established, which divide the data according to the aforementioned three dimensions to form corresponding multi-level lookup tables. During retrieval, filtering can be performed first based on the provided time information, then based on spatial information, and finally based on type information, etc. The three-level index information can be stored in a distributed index library (such as Elasticsearch), establishing a mapping relationship between "index and original data."

[0032] Step 104: Store the partitioned data to form stored data.

[0033] In this step, after the data to be processed is divided, the data can be stored. The stored data is called stored data. In some embodiments, the data can be cached by dividing the data into "large files (>500MB)" and "small files (≤500MB)" according to the file size. Large files use distributed file system (HDFS) block caching, and small files use local disk directory caching. This does not distinguish the storage differences of structured data (which needs to be associated with field indexes), fragmented files (<1MB, which are generated frequently and easily occupy disk fragments), metadata (which needs to be read frequently), etc., and there is no unified cache eviction and update standard.

[0034] It can be seen that the caching mechanisms in the above embodiments (such as Redis caching query results) are general-purpose and fail to fully consider the specific characteristics of data access patterns in power business. For massive amounts of small files (such as millions of images generated by drone inspections), frequent metadata queries and file read operations will generate huge disk IOPS pressure, and general-purpose caching cannot effectively alleviate this problem. For extremely large files (such as single remote sensing images of tens of gigabytes), general-purpose caching strategies either cache the entire file (wasting memory) or do not cache it (slow access), failing to achieve efficient "partial reading" and caching, resulting in an extremely low cache hit rate. As a result, the IO performance of the underlying storage system cannot meet the high-concurrency access requirements of the upper-layer applications, becoming the performance bottleneck of the entire system.

[0035] Furthermore, in some embodiments, during data storage, the data can be further divided according to data type and data size, and then the storage method for different data can be determined based on the division. Finally, different types and sizes of data are stored according to different storage methods. Specifically, the data division can be as shown in Table 1.

[0036]

[0037] Table 1. Data storage requirements for different data types and sizes (reference) It can be seen that different responses can be specifically set according to different data types and data sizes, and these data have various different needs, so their corresponding storage methods can also be set accordingly. That is, in some embodiments, the storage of the partitioned data includes: determining the data type and data size of the partitioned data; and determining the storage method of the partitioned data based on the data type and / or the data size.

[0038] Specifically, in some embodiments, for example, for structured data: a relational database cache (such as Redis Cluster) can be used, with field indexes established, supporting SQL queries, and the cache validity period synchronized with the database update frequency (e.g., every 10 minutes). For large files: a distributed file system (HDFS) cache can be used, divided into blocks of 1GB each, with 3 block replicas (to ensure reliability), and the cache eviction policy is "lowest access frequency priority". For small-sized unstructured data: a key-value database (Redis) cache can be used, with the key being "data ID + timestamp" and the value being the file's binary stream, and the cache validity period set to 24 hours. For fragmented files: multiple fragmented files can be merged into a 100MB "merged file" using a merging tool (such as Logstash), and then stored in a local SSD cache, named with "date + device ID", and the cache validity period set to 7 days. For metadata: it can be stored in a dedicated metadata cache pool (based on the in-memory database Memcached), set with the highest access priority, and never actively evicted (only replaced when metadata is updated). That is, in some embodiments, determining the storage method of the partitioned data includes: storing the partitioned data using a relational database in response to the data type being structured data; storing the partitioned data using a metadata cache pool in response to the data type being metadata; merging the partitioned data using a merging tool in response to the data type being fragmented data, and caching the merged data locally; storing the partitioned data using a distributed file system in response to the data size being greater than a first threshold; and storing the partitioned data using a key-value pair database in response to the data size being less than a second threshold. The first and second thresholds can be set according to specific application scenarios, and these two thresholds are used to distinguish between large and small data. Furthermore, for fragmented data, in addition to the aforementioned merging methods, the HadoopArchive (HAR) tool can be used instead of Logstash to package fragmented files into archive files for caching, which is suitable for the Hadoop ecosystem and does not require the development of additional merging tools.

[0039] Furthermore, for the stored data that has been stored, dynamic optimization can be performed. For example, a "access frequency statistics + dynamic priority adjustment" mechanism can be adopted: the access count of each cached file is counted every hour, and files accessed ≥10 times are marked as "high-frequency files," increasing their cache priority (e.g., increasing the number of large file block replicas to 4). Then, an improved LRU (Least Recently Used) algorithm can be used: when cache space is insufficient, "low-frequency + least recently used" files are evicted first, while high-frequency files are retained, increasing the cache hit rate to over 90%. Of course, for the eviction algorithm, the LFU (Least Frequently Used) algorithm can be used instead of the improved LRU algorithm, suitable for scenarios with "large differences in access frequency" (e.g., some fragmented files have an average daily access count >100 times), requiring only the addition of an access count counter in the cache management module.

[0040] In practical applications, the storage methods described above, with their adaptive caching strategies, perfectly suit the characteristics of power data. By temporarily storing frequently accessed data on high-speed storage media, the number of accesses to the original storage device is reduced, adapting to different storage formats such as structured data (e.g., database tables), large files (e.g., remote sensing images), and fragmented files (e.g., logs). The IOPS pressure on massive amounts of small files is alleviated through a "packaging" strategy, while caching large files becomes highly efficient through a "blocking" strategy. The cache hit rate increases from less than 30% to over 80%, significantly reducing the load on the underlying storage system. The overall system throughput and concurrency processing capabilities are increased by 5-10 times, while simultaneously reducing hardware costs.

[0041] In some embodiments, after the stored data is formed, situations may arise where it needs to be updated, modified, or added. Therefore, based on the aforementioned embodiments regarding the partitioning and storage of data to be processed, when updating the data, the corresponding time index, spatial index, and data type identifier can be synchronously adjusted and updated in the corresponding hierarchical lookup table and other structural parameter tables. That is, when data in the perception layer is added / deleted, the metadata database is automatically updated, synchronously updating the corresponding time layer, spatial index, and data type identifier in the index library. In other words, in some embodiments, after the stored data is formed, the method further includes: in response to an update operation on the stored data, determining the data to be updated; and updating the metadata and hierarchical indexes corresponding to the updated data. The updated data is the stored data targeted by the update operation, which may be adjusted through the update operation.

[0042] Step 106: In response to the transmission request for the stored data, determine the data to be transmitted corresponding to the transmission request in the stored data.

[0043] In this step, after data storage is completed, requests for data queries and transmissions can be received. Generally, requests require locating the corresponding data before further processing; that is, data queries are usually performed first. According to the aforementioned embodiment, because the stored data is divided accordingly, retrieval queries are performed hierarchically based on this hierarchical division. For example, if a user inputs search criteria (such as "20250916-14:00, 30°-31° North latitude, vector data"), the index module first filters time buckets, then locates the spatial range using the corresponding spatial indexing algorithm, and finally filters by data type identifier, directly returning the matching original data address. The retrieval time can be shortened to less than 50ms.

[0044] Next, the transmission request is a request to transmit the corresponding data. Based on this request, the specific data needs to be determined first. According to the aforementioned query method, the data to be transmitted corresponding to the transmission request in the stored data can be determined.

[0045] Step 108: Transmit the data to be transmitted according to the set transmission rules.

[0046] In this step, once the data to be transmitted is determined, data transmission operations can be performed according to the corresponding transmission rules. In some embodiments, data transmission, especially cross-network data transmission, can be carried out using a general protocol, such as the FTP protocol based on the RFC959 standard. Specifically, an FTP client and server are deployed on the Internet and the power information network respectively, and cross-network connection is achieved through port mapping to directly transmit the original file. In some scenarios, SSL encryption (i.e., FTPS) is superimposed, but no chunked transmission and breakpoint resumption mechanism is designed for large files (>10GB), and there is no access control policy for dedicated transmission channels. It is easy for transmission to be interrupted due to network fluctuations, and there is also the risk of unauthorized access.

[0047] As can be seen, the FTP and network gateway-based transfer methods described in the above embodiments are essentially "full" transfers. Even if only a small portion of the file content changes (such as modifying the metadata tags of an image), the entire file still needs to be transferred, resulting in a significant waste of network bandwidth and low transmission efficiency. This transfer process lacks effective fault tolerance and verification mechanisms. In unstable network conditions, large file transfers are easily interrupted and must be retransmitted from the beginning, rather than resuming from the breakpoint, making success rates unreliable. Furthermore, the transfer tasks lack intelligent scheduling, making it difficult to manage complex dependencies and priorities, and failing to meet the stringent timeliness requirements of power services.

[0048] Furthermore, in some embodiments, before transmitting data, the data to be transmitted can first undergo preprocessing such as block segmentation and encryption. Specific preprocessing processes may include: Block segmentation: Large files (>1GB) to be transmitted can be divided into 100MB blocks, generating block numbers (e.g., File001_Block01), and calculating an MD5 checksum for each block (for integrity verification). Encryption: Each file block can be encrypted using the AES-256 symmetric encryption algorithm. The key is dynamically generated by the power information network and transmitted to the internet via asymmetric encryption (RSA) (to prevent key leakage). Transmission list generation: A transmission list (XML format) containing "file ID, number of blocks, MD5 of each block, and encryption key ID" can be generated. The list itself is digitally signed (SHA256 + private key) to ensure immutability. Of course, in other embodiments, the SM4 algorithm can be used instead of AES-256 for encryption, requiring only the algorithm library of the encryption / decryption unit to be replaced, while maintaining the key transmission process unchanged.

[0049] Afterwards, the corresponding transmission link can be determined, such as establishing a dedicated IPsec VPN channel for dedicated data transmission. During transmission, a transmission list can be sent first. The receiving end can use the transmission list to determine the encryption method, transmission method, and number of blocks of the data to be transmitted, facilitating subsequent statistics and decryption. The receiving end can perform data statistics, decryption, and concatenation after receiving each data block, or it can perform decryption and concatenation operations only after all data blocks have been received. That is, in some embodiments, the data transmission according to the set transmission rules includes: dividing the data to be transmitted into blocks and encrypting it to form at least one data block and a transmission list; determining the transmission link to transmit the transmission list, wherein the transmission link transmits the at least one data block at a time, so that the receiving end can perform statistics and decryption on the received at least one data block according to the transmission list.

[0050] Furthermore, to ensure transmission stability, the transmission link can be configured as at least a dual-link backup channel, i.e., it includes at least one primary link and one backup link (the primary link bandwidth can be set to 100Mbps, and the backup link bandwidth to 50Mbps, etc.). When the primary link cannot meet the transmission requirements (e.g., the packet loss rate exceeds a certain threshold), the backup link is switched in a timely manner for transmission. That is, in some embodiments, the transmission link includes at least one primary link and at least one backup link. When the packet loss rate of any primary link exceeds a set threshold, the primary link is switched to the at least one backup link. Of course, in some embodiments, there can be more than one primary link, and multiple primary links can be used for parallel transmission of data blocks. Multiple backup links can also be used to facilitate timely replacement when the primary or backup link fails.

[0051] In more specific embodiments, access control can also be implemented for transmission, such as configuring IP whitelists (allowing only specified internet IPs to access) and port restrictions (only opening VPN-dedicated ports 500 / 4500), while deploying an intrusion detection system (IDS) to intercept abnormal access requests (such as high-frequency connection attempts). During the transmission process, a "sliding window protocol + breakpoint resumption" mechanism can be used for control. The sliding window can be set to a window size of 5 (transmitting 5 file blocks simultaneously). The internet end sends blocks sequentially by block number, and the power information network end returns an "acknowledgment frame" (containing the block number and MD5 checksum result) after receiving each block. Breakpoint resumption: if a block transmission fails (e.g., checksum mismatch, timeout), the power information network end returns a "retransmission request," and the internet end only retransmits that block (without retransmitting the entire file). Progress feedback can also be provided: the number of transmitted blocks and the total number of blocks can be counted in real time, and the transmission progress can be displayed through a visual interface (e.g., "50 / 100 blocks, 50%)," and an SMS notification can be sent to the administrator upon completion of the transmission. After transmission is complete, the receiving end can perform integrity verification: After receiving all blocks, the power information network end (receiving end) can merge them into a complete file according to the block number, recalculate the file MD5, and compare it with the file MD5 in the transmission list. If they match, the file is confirmed to be complete. Decryption and storage are also possible: The complete file can be decrypted using a dynamic key and stored in a designated storage device of the power information network (such as SAN storage). Simultaneously, the transmission list and decryption log are stored in an audit database (retained for 6 months for security auditing). A cleanup mechanism is also possible: After transmission is complete, the internet end (sender end) can automatically delete the local encrypted block file, and the power information network end (receiving end) can delete the temporary key to prevent data residue. In some other embodiments, the QUIC protocol can be used instead of the sliding window protocol. QUIC is based on UDP, supports 0-RTT connection establishment, is suitable for networks with high packet loss rates (such as internet in remote areas), and can further reduce transmission latency by 10%.

[0052] In practical applications, the block-level differential synchronization mechanism, through the transmission methods described in the above embodiments, ensures that only changed data is transmitted between networks. For updates to large remote sensing images or model files, this can typically reduce data transmission volume by more than 90%, and transmission time by several to tens of times. Combined with breakpoint resume and strong verification mechanisms, the reliability of the transmission task reaches over 99.99%, fully guaranteeing the continuity of services and the timeliness of data.

[0053] Finally, overall, the standardized metadata model and indexing interface make adding new data types simple. The tiered storage architecture allows the system to seamlessly expand storage capacity based on cost-effectiveness principles. The entire system architecture is clear, with low coupling between modules, making it easy to maintain and upgrade.

[0054] As can be seen from the above embodiments, this disclosure provides a data processing method, the method comprising: in response to acquiring data to be processed, dividing the data to be processed according to a set rule; wherein the set rule includes dividing the data to be processed according to a time dimension and a spatial dimension, and identifying the data to be processed according to its source; storing the divided data to form stored data; in response to a transmission request for the stored data, determining the data to be transmitted corresponding to the transmission request in the stored data; and transmitting the data to be transmitted according to the set transmission rule. This disclosure establishes a unified, deep composite spatiotemporal index for all types of data to be processed, specifically by associating the time and spatial attributes of the data, and further classifying and dividing it according to the data type identifier, thereby establishing a multi-level index structure to shorten data retrieval time, and transforming complex multi-condition joint queries into efficient index key-value queries and set operations. The query response time is improved from minutes in traditional solutions to seconds or even sub-seconds, improving retrieval efficiency by 10-100 times, making real-time interactive big data analysis possible, which was previously impossible.

[0055] In specific applications, the above-described solutions, in addition to power system scenarios, can be widely applied to other fields that require processing massive amounts of multi-source heterogeneous spatiotemporal data, including but not limited to: (1) Smart cities and digital twins: used to manage city-level oblique photography models, BIM models, monitoring video streams, and IoT sensor data to achieve real-time visualization and comprehensive analysis of urban operation status. (2) Autonomous driving and high-precision maps: used for differential updates of high-precision map data (DeltaUpdate), where vehicles only need to download the changed road network information blocks, greatly saving update traffic and time. (3) Natural resources and environmental monitoring: used to manage multi-period remote sensing images and ecological monitoring data in forestry, water conservancy, meteorology, oceanography, and other fields, supporting rapid detection and analysis of environmental changes. (4) Emergency command and public safety: in emergency scenarios, rapidly integrate multi-source data such as satellite images, UAV aerial photography, personnel positioning, and emergency plan documents to provide unified and efficient data support for command and decision-making. (5) Military and national defense: used for battlefield environmental information fusion, integrating multi-source intelligence data such as satellite reconnaissance, radar, and UAV reconnaissance to establish a unified battlefield situation map.

[0056] In specific scenarios, for data partitioning, the core implementation technology of unified spatiotemporal indexing for multiple data types is adopted in this embodiment. Firstly, metadata standardization technology is used: for the four core data types in the perception layer (remote sensing / UAV data, vector data, and 3D model data), differentiated metadata fields (such as "resolution" for remote sensing data, "coordinate point set" for vector data, and "spatial bounding box" for 3D model data) are extracted and uniformly mapped to a "time-space-data type" standard field format. This solves the problem of incompatible metadata formats for multiple data types, which leads to inconsistent indexing and forms the basis for subsequent three-level index construction. Next, dynamic time stratification technology is used: a "day-hour" basic stratification granularity is adopted, while supporting dynamic adjustment based on data update frequency (e.g., high-frequency updated UAV real-time inspection data can be refined to "hour-minute" stratification, while low-frequency updated monthly remote sensing imagery can be expanded to "week-day" stratification). This avoids index redundancy or insufficient retrieval accuracy caused by fixed stratification, ensuring the flexibility and efficiency of the time-dimensional index. Next, we selected spatial indexing algorithms to adapt to different data types: Based on the differences in spatial data morphology, we matched the optimal indexing algorithm—using R-tree indexes for areal spatial ranges (remote sensing / UAV imagery) (efficiently handling multi-dimensional rectangular range queries), using quadtree indexes for point / line coordinates (vector data) (fastly locating discrete coordinate points), and using octree indexes for 3D spatial bounding boxes (3D models) (adapting to hierarchical partitioning of 3D space), thus solving the problem of poor adaptability of a single spatial indexing algorithm to multiple data types. Finally, we implemented index-data linkage update technology: establishing a real-time linkage mechanism between the metadata database and a distributed index library (such as Elasticsearch). When data is added / deleted in the sensing layer, it automatically triggers updates to the metadata database fields, synchronously driving updates to the corresponding time buckets, spatial index structures, and data type identifiers in the index library, ensuring consistency between the index and the original data, and avoiding retrieval errors caused by index failure.

[0057] For data storage, the embodiment adopts a unified caching optimization technology for multiple storage formats. The first is the intelligent file type classification technology: based on the three dimensions of "data structure + file size + access frequency", the input data is accurately divided into five categories: structured data, large files, small-sized unstructured data, fragmented files, and metadata. Fragmented files are defined as files with a single file size of <1MB and an average daily generation of >100 files, and metadata is defined as data with a single data entry of <1KB and an average daily access frequency of >50 times. This solves the caching strategy mismatch problem caused by the traditional classification based on a single "file size". Next, a differentiated caching storage strategy was designed: dedicated storage solutions were designed for five types of files—structured data using Redis Cluster caching (supporting SQL queries and transaction consistency), large files using HDFS block caching (1GB / block + 3 replicas), small-sized unstructured data using Redis key-value pair caching (key is "data ID + timestamp"), fragmented files using "Logstash merging + SSD caching" (merged into 100MB / file + named by "date + device ID"), and metadata using Memcached memory caching (highest priority + never actively evicted), maximizing the caching performance of each type of file. Finally, a dynamic cache priority control technology was implemented: the number of cached file accesses is counted every hour, and files with "accesses ≥ 10 times" are marked as high-frequency files, triggering priority increases (e.g., increasing the number of replicas for large files from 3 to 4, extending the cache validity period for small files from 24 hours to 48 hours); at the same time, an improved LRU eviction algorithm is used, prioritizing the eviction of files with "access frequency < 3 times / hour and the longest unused" when cache space is insufficient, ensuring that the cache hit rate of high-frequency files remains stable above 90%.

[0058] For data transmission, the embodiment employs a secure and efficient technology for cross-network transmission of large files between the Internet and the local area network. Firstly, it utilizes block-level encryption and secure key transmission technology: large files (>1GB) are divided into 100MB blocks, and each block is encrypted using the AES-256 symmetric encryption algorithm (reducing overall encryption time). Simultaneously, a symmetric key is dynamically generated at the power information network end and transmitted to the Internet end via the RSA asymmetric encryption algorithm (preventing key theft during transmission), resolving the contradiction between low encryption efficiency for large files and secure key transmission. Next, a dedicated transmission channel technology with dual-link backup is used: a dedicated IPsec VPN channel is established between the Internet and the power information network, configured with a dual-link architecture of "100Mbps primary link + 50Mbps backup link." Automatic switching is achieved by real-time monitoring of the primary link packet loss rate (threshold >5%). Simultaneously, triple access control is implemented using IP whitelisting (allowing only specified Internet IPs to access), dedicated port restrictions (only opening ports 500 / 4500), and IDS intrusion detection to ensure channel security and transmission stability. Next is the block-level breakpoint resumption and integrity verification technology: A "sliding window (window size = 5) + block-level confirmation" mechanism is adopted. Upon receiving a single block, the power information network immediately verifies its MD5 hash. If the verification fails, only the block is retransmitted (no need to retransmit the entire file). After transmission, a dual verification process of "single-block MD5 cumulative verification + complete file MD5 secondary verification" ensures file integrity, solving the problems of "retransmission of the entire file upon interruption" and "integrity cannot be confirmed" in traditional FTP transfers of large files. Finally, there is the full-process transmission audit technology: An XML-formatted transmission list containing "file ID, number of blocks, MD5 hash of each block, and key ID" is generated. The list is tamper-proofed using SHA256 + private key digital signature. After transmission, the list, decryption logs, and channel switching records are stored in the audit database and retained for 6 months, meeting the power industry's security audit compliance requirements (such as Level 3 requirements of Information Security Protection 2.0).

[0059] It should be noted that the method of this disclosure embodiment can be executed by a single device, such as a computer or server. The method of this disclosure embodiment can also be applied in a distributed scenario, where multiple devices cooperate to complete the task. In such a distributed scenario, one of these devices may execute only one or more steps of the method of this disclosure embodiment, and the multiple devices will interact with each other to complete the method described.

[0060] It should be noted that the above description describes specific embodiments of this disclosure. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in a different order than that shown in the above embodiments and still achieve the desired results. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0061] Based on the same concept, corresponding to the methods of any of the above embodiments, this disclosure also provides a data processing apparatus.

[0062] refer to Figure 2 The data processing device includes: The first module 210, in response to acquiring the data to be processed, divides the data to be processed according to a set rule; wherein the set rule includes dividing the data to be processed according to the time dimension and the spatial dimension, and identifying the data to be processed according to the source of the data to be processed.

[0063] The second module 220 stores the partitioned data to form stored data.

[0064] The third module 230, in response to a transmission request for the stored data, determines the data to be transmitted corresponding to the transmission request in the stored data.

[0065] The fourth module 240 transmits the data to be transmitted according to the set transmission rules.

[0066] In some exemplary embodiments, the first module 210 is further configured to: According to the time division rules, the data to be processed is divided into at least two time dimensions; Determine the data type of the data to be processed, determine the indexing algorithm for the spatial dimensions based on the data type, and divide the data to be processed according to the indexing algorithm; The data to be processed is divided into time dimension, spatial dimension, and then into the identifier, and an index of at least three levels is established.

[0067] In some exemplary embodiments, the second module 220 is further configured to: In response to an update operation on the stored data, determine the data to be updated; The metadata and indexes at each level corresponding to the updated data are updated.

[0068] In some exemplary embodiments, the second module 220 is further configured to: Determine the data type and size of the partitioned data; The storage method of the partitioned data is determined based on the data type and / or the data size.

[0069] In some exemplary embodiments, the second module 220 is further configured to: Since the data type of the partitioned data is structured data, a relational database is used to store the partitioned data; Since the data type of the partitioned data is metadata, the partitioned data is stored using a set metadata cache pool; In response to the fact that the data after division is fragmented, a merging tool is used to merge the divided data, and the merged data is cached locally. If the size of the partitioned data exceeds a first threshold, the partitioned data is stored using a distributed file system. If the size of the partitioned data is less than a second threshold, the partitioned data is stored in a key-value pair database.

[0070] In some exemplary embodiments, the fourth module 240 is further configured to: The data to be transmitted is divided into blocks and encrypted to form at least one data block and a transmission list; A transmission link is determined to transmit the transmission list. The transmission link transmits the at least one data block at a time, so that the receiving end can perform statistics and decryption on the received at least one data block according to the transmission list.

[0071] In some exemplary embodiments, the transmission link includes at least one primary link and at least one backup link. When the packet loss rate of any primary link exceeds a set threshold, a switch is performed between the primary link and the at least one backup link.

[0072] For ease of description, the above apparatus is described in terms of its functions, divided into various modules. Of course, in implementing the embodiments of this disclosure, the functions of each module can be implemented in one or more software and / or hardware.

[0073] The apparatus described above is used to implement the corresponding data processing methods in the foregoing embodiments and has the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0074] Based on the same concept, corresponding to the methods of any of the above embodiments, this disclosure also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the data processing method as described in any of the above embodiments.

[0075] Figure 3 This embodiment illustrates a more specific hardware structure of an electronic device, which may include a processor 1010, a memory 1020, an input / output interface 1030, a communication interface 1040, and a bus 1050. The processor 1010, memory 1020, input / output interface 1030, and communication interface 1040 are interconnected internally via the bus 1050.

[0076] The processor 1010 can be implemented using a general-purpose CPU (Central Processing Unit), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this specification.

[0077] The memory 1020 can be implemented in the form of ROM (Read Only Memory), RAM (Random Access Memory), static storage device, dynamic storage device, etc. The memory 1020 can store the operating system and other applications. When the technical solutions provided in the embodiments of this specification are implemented by software or firmware, the relevant program code is stored in the memory 1020 and is called and executed by the processor 1010.

[0078] The input / output interface 1030 is used to connect input / output modules to realize information input and output. The input / output modules can be configured as components in the device (not shown in the figure) or externally connected to the device to provide corresponding functions. Input devices may include keyboards, mice, touch screens, microphones, various sensors, etc., and output devices may include displays, speakers, vibrators, indicator lights, etc.

[0079] The communication interface 1040 is used to connect a communication module (not shown in the figure) to enable communication between this device and other devices. The communication module can communicate via wired means (such as USB, Ethernet cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.).

[0080] Bus 1050 includes a pathway for transmitting information between various components of the device, such as processor 1010, memory 1020, input / output interface 1030, and communication interface 1040.

[0081] It should be noted that although the above-described device only shows the processor 1010, memory 1020, input / output interface 1030, communication interface 1040, and bus 1050, in specific implementations, the device may also include other components necessary for normal operation. Furthermore, those skilled in the art will understand that the above-described device may only include the components necessary for implementing the embodiments of this specification, and not necessarily all the components shown in the figures.

[0082] The electronic devices described above are used to implement the corresponding data processing methods in any of the foregoing embodiments and have the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0083] Based on the same concept, corresponding to the methods of any of the above embodiments, this disclosure also provides a non-transitory computer-readable storage medium that stores computer instructions for causing the computer to perform the data processing method as described in any of the above embodiments.

[0084] The computer-readable medium of this embodiment includes permanent and non-permanent, removable and non-removable media, which can be used to store information by any method or technology. Information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transfer medium that can be used to store information accessible by a computing device.

[0085] The computer instructions stored in the storage medium of the above embodiments are used to cause the computer to execute the data processing method as described in any of the above embodiments, and have the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0086] Based on the same concept, corresponding to the methods of any of the above embodiments, this disclosure also provides a computer program product, which includes computer program instructions. In some embodiments, the computer program instructions can be executed by one or more processors of a computer to cause the computer and / or the processors to perform the data processing method. Corresponding to the execution entity for each step in each embodiment of the data processing method, the processor executing the corresponding step may belong to the corresponding execution entity.

[0087] The computer program products of the above embodiments are used to cause the computer and / or the processor to perform the data processing method as described in any of the above embodiments, and have the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0088] Those skilled in the art should understand that the discussion of any of the above embodiments is merely exemplary and is not intended to imply that the scope of this disclosure (including the claims) is limited to these examples; within the framework of this disclosure, the technical features of the above embodiments or different embodiments can also be combined, the steps can be implemented in any order, and there are many other variations of different aspects of the embodiments of this disclosure as described above, which are not provided in detail for the sake of brevity.

[0089] Additionally, to simplify the description and discussion, and to avoid obscuring the embodiments of this disclosure, the provided drawings may or may not show well-known power / ground connections to integrated circuit (IC) chips and other components. Furthermore, the apparatus may be shown in block diagram form to avoid obscuring the embodiments of this disclosure, and this also takes into account the fact that the details of implementation of these block diagram apparatuses are highly dependent on the platform on which the embodiments of this disclosure will be implemented (i.e., these details should be fully understood by those skilled in the art). While specific details (e.g., circuits) have been set forth to describe exemplary embodiments of this disclosure, it will be apparent to those skilled in the art that the embodiments of this disclosure can be implemented without these specific details or with variations thereof. Therefore, these descriptions should be considered illustrative rather than restrictive.

[0090] Although this disclosure has been described in conjunction with specific embodiments thereof, many substitutions, modifications, and variations of these embodiments will be apparent to those skilled in the art from the foregoing description. For example, other memory architectures (e.g., dynamic RAM (DRAM)) may be used with the embodiments discussed.

[0091] This disclosure is intended to cover all such substitutions, modifications, and variations that fall within the broad scope of the appended claims. Therefore, any omissions, modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A data processing method, characterized in that, include: In response to the acquisition of data to be processed, the data to be processed is divided according to the set rules; wherein the set rules include dividing the data to be processed according to the time dimension and the spatial dimension, and identifying the data to be processed according to the source of the data to be processed; The partitioned data is then stored to form stored data. In response to a transmission request for the stored data, determine the data to be transmitted corresponding to the transmission request from the stored data; The data to be transmitted is transmitted according to the set transmission rules.

2. The method according to claim 1, characterized in that, The process of dividing the data to be processed according to the set rules includes: According to the time division rules, the data to be processed is divided into at least two time dimensions; Determine the data type of the data to be processed, determine the indexing algorithm for the spatial dimensions based on the data type, and divide the data to be processed according to the indexing algorithm; The data to be processed is divided into time dimension, spatial dimension, and then into the identifier, and an index of at least three levels is established.

3. The method according to claim 2, characterized in that, After the storage data is formed, the method further includes: In response to an update operation on the stored data, determine the data to be updated; Update the metadata and indexes at each level corresponding to the updated data.

4. The method according to claim 1, characterized in that, The process of storing the partitioned data includes: Determine the data type and size of the partitioned data; The storage method of the partitioned data is determined based on the data type and / or the data size.

5. The method according to claim 4, characterized in that, Determining the storage method for the partitioned data includes: Since the data type of the partitioned data is structured data, a relational database is used to store the partitioned data; Since the data type of the partitioned data is metadata, the partitioned data is stored using a set metadata cache pool; In response to the fact that the data after division is fragmented, a merging tool is used to merge the divided data, and the merged data is cached locally. If the size of the partitioned data exceeds a first threshold, the partitioned data is stored using a distributed file system. In response to the fact that the size of the partitioned data is less than a second threshold, the partitioned data is stored in a key-value pair database.

6. The method according to claim 1, characterized in that, The step of transmitting the data to be transmitted according to the set transmission rules includes: The data to be transmitted is divided into blocks and encrypted to form at least one data block and a transmission list; A transmission link is determined to transmit the transmission list. The transmission link transmits the at least one data block at a time, so that the receiving end can perform statistics and decryption on the received at least one data block according to the transmission list.

7. The method according to claim 6, characterized in that, The transmission link includes at least one main link and at least one backup link. When the packet loss rate of any main link exceeds a set threshold, the main link is switched to the at least one backup link.

8. A data processing apparatus, characterized in that, include: The first module, in response to acquiring data to be processed, divides the data to be processed according to a set rule; wherein the set rule includes dividing the data to be processed according to the time dimension and the spatial dimension, and identifying the data to be processed according to the source of the data to be processed; The second module stores the partitioned data to form stored data. The third module, in response to a transmission request for the stored data, determines the data to be transmitted corresponding to the transmission request in the stored data; The fourth module transmits the data to be transmitted according to the set transmission rules.

9. An electronic device, characterized in that, It includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the computer program, implements the method as described in any one of claims 1 to 7.

10. A non-transitory computer-readable storage medium, characterized in that, The non-transitory computer-readable storage medium stores computer instructions for causing a computer to perform the method as described in any one of claims 1 to 7.

Images