Data processor method and device, electronic equipment and readable storage medium

Abstract

The invention discloses a data processor method and device, electronic equipment and a readable storage medium. The data processing method is applied to a first data processor, remote procedure call information sent by a first target data processor is received under the condition that a first subscription relation exists between the first data processor and the first target data processor, and the first subscription relation comprises subscribing a first data category published by the first target data processor. The remote procedure call information comprises subscription data corresponding to a first data category pushed by the first target data processor; analyzing the remote procedure call information to obtain subscription data; and converting the storage structure of the subscription data into a column type storage structure, and writing the subscription data of the column type storage structure into a storage space corresponding to the first data processor. According to the embodiment of the invention, the data access can be quickly responded, and the business service with high stability and high speed can be conveniently provided.

Description

Technical Field

[0001] This application belongs to the field of data storage technology, and in particular relates to a data processor method, apparatus, electronic device and readable storage medium. Background Technology

[0002] With the development of data storage technology, cloud and network infrastructures are highly integrated, and network infrastructure is gradually moving towards the cloud. Enterprise application services typically involve large amounts of data; therefore, cloud deployment of enterprise application services based on distributed application service architectures has become a trend. In the process of cloud deployment of enterprise application services, data processing units (DPUs) based on smart network interface cards (NICs) are increasingly widely used. These DPUs have functions such as data storage, reading and writing, encryption and decryption, and compression and decompression, which can effectively reduce the load on the central processing unit (CPU) of electronic devices.

[0003] While deploying enterprise application services across clouds, regions, and clusters has become easier, in distributed application service architectures, accessing data from other nodes across clouds, regions, and clusters via remote networks is a common practical need due to business requirements, resource costs, management requirements, and objective environmental factors. Since network bandwidth across clouds, regions, and clusters remains limited, and providing high-speed, stable services based on various access needs is highly dependent on the physical environment, this leads to significant uncertainty in responding to access requests, especially remote access requests, resulting in slow service response times and low stability. Summary of the Invention

[0004] This application provides a data processor method, apparatus, electronic device, and readable storage medium, which facilitates rapid response to data access and enables the provision of highly stable and fast business services.

[0005] In a first aspect, embodiments of this application provide a data processing method. The method is applied to a first data processor, which is communicatively connected to at least one second data processor. The at least one second data processor includes a first target data processor. The first target data processor uses an interface description language to define the data structure of the data categories to be published. The method includes:

[0006] A first data processor is configured to receive remote procedure call information sent by a first target data processor when a first subscription relationship exists with the first target data processor, wherein the first subscription relationship includes subscribing to a first data category published by the first target data processor, and the remote procedure call information includes subscription data corresponding to the first data category pushed by the first target data processor;

[0007] Parse the remote procedure call information to obtain the subscription data;

[0008] The storage structure of the subscription data is converted into a columnar storage structure, and the subscription data of the columnar storage structure is written into the storage space corresponding to the first data processor. The storage space includes at least one data category and the subscription data corresponding to each data category.

[0009] Among the possible implementations of the first aspect, the method also includes:

[0010] Receive data access requests, whereby the data access requests include target data category access information;

[0011] In response to a data access request, the subscribed data corresponding to the target data class target identifier is obtained from the storage space corresponding to the first data processor based on the target data class target identifier;

[0012] Output the subscription data corresponding to the target class target identifier.

[0013] In some possible implementations of the first aspect, the storage space corresponding to the first data processor includes solid-state drives and hard disk drives;

[0014] Converting the storage structure of the subscription data into a columnar storage structure and writing the subscription data in the columnar storage structure into the storage space corresponding to the first data processor includes:

[0015] Based on the log structure merge tree storage engine, the lowest-level row-ordered key-value pair set corresponding to the subscribed data is determined, and the lowest-level row-ordered key-value pair set is converted into column-ordered storage data.

[0016] Store columnar data onto a hard disk drive; and,

[0017] If the columnar storage data is identified to include text data fields, a storage dictionary for the text data fields and a storage index corresponding to the storage dictionary are generated. The storage dictionary is then stored on a solid-state drive (SSD), and the storage index is stored on a hard disk drive (HDD).

[0018] In some possible implementations of the first aspect, the method further includes, before receiving the remote procedure call information sent by the first target data processor:

[0019] Receive first configuration information, wherein the first configuration information includes first subscription information, and the first subscription information includes the identifier of the first target data processor and the identifier of the first data category;

[0020] Based on the first subscription information, a subscription request is sent to the first target data processor to establish a first subscription relationship with the first target data processor.

[0021] Among the possible implementations of the first aspect, the method also includes:

[0022] Receive second configuration information, wherein the second configuration information includes the second data category to be published;

[0023] In response to the second configuration information, publishing information is generated based on the second data category to be published. The publishing information includes the identifier of the second data category, the name of the second data category, and the publishing status of the second data category.

[0024] Send the release information to each second data processor;

[0025] Upon receiving a subscription request from a second target data processor, a second subscription relationship is established with the second target data processor, wherein at least one first data processor includes the second target data processor, and the second subscription relationship includes subscribing to a second data category published by the first data processor.

[0026] Among the possible implementations of the first aspect, the method also includes:

[0027] Upon receiving the subscription data corresponding to the second data category, the subscription data corresponding to the second data category is sent to the second target data processor so that the second target data processor stores the subscription data corresponding to the second data category in the storage space corresponding to the second target data processor.

[0028] Secondly, embodiments of this application provide a data processing method applied to a first target data processor. The first target data processor uses an interface description language to define the data structure of data categories to be published. The first target data processor is communicatively connected to at least one first data processor. A first subscription relationship exists between the first data processor and the first target data processor, including subscribing to a first data category published by the first target data processor. The method includes:

[0029] The first target data processor is used to obtain the subscription data corresponding to the first data category;

[0030] Remote procedure call information is pushed to each first data processor so that the first data processor can parse the remote procedure call information to obtain subscription data, convert the storage structure of the subscription data into a columnar storage structure, and write the subscription data of the columnar storage structure into the storage space corresponding to the first data processor. The storage space includes at least one data category and subscription data corresponding to each data category.

[0031] Thirdly, embodiments of this application provide a first data processing apparatus, which is communicatively connected to at least one second data processor, wherein the at least one second data processor includes a first target data processor; the first data processing apparatus includes:

[0032] The data configuration management module is used to receive remote procedure call information sent by the first target data processor when there is a first subscription relationship with the first target data processor. The first subscription relationship includes subscribing to a first data category published by the first target data processor, and the remote procedure call information includes the subscription data corresponding to the first data category pushed by the first target data processor.

[0033] The write module is used to parse remote procedure call information and obtain subscription data;

[0034] The writing module is further configured to convert the storage structure of the subscription data into a columnar storage structure and write the subscription data of the columnar storage structure into the storage space corresponding to the first data processor, wherein the storage space includes at least one data category and subscription data corresponding to each data category.

[0035] Fourthly, embodiments of this application provide a first target data processing device. The first target data processor uses an interface description language to define the data structure of the data categories to be published. The first target data processing device is communicatively connected to at least one first data processor. A first subscription relationship exists between the first data processor and the first target data processing device, wherein the first subscription relationship includes subscribing to a first data category published by the first target data processing device. The first target data processing device includes:

[0036] The data configuration management module is used to obtain the subscription data corresponding to the first data category;

[0037] The transmission module is used to push remote procedure call information to each first data processor, so that the first data processor can parse the remote procedure call information to obtain subscription data, convert the storage structure of the subscription data into a columnar storage structure, and write the subscription data of the columnar storage structure into the storage space corresponding to the first data processor, wherein the storage space includes at least one data category and subscription data corresponding to each data category.

[0038] In some possible implementations of the first aspect, the apparatus further includes:

[0039] The reading module is used to receive data access requests, which include target data category access information.

[0040] The reading module is also used to respond to a data access request, obtain the subscription data corresponding to the target data class identifier in the storage space corresponding to the first data processor, based on the target data class identifier, and output the subscription data corresponding to the target data class identifier.

[0041] Fifthly, this application provides an electronic device comprising: a processor and a memory storing computer program instructions; wherein the processor, when executing the computer program instructions, implements the first aspect, any implementable method of the first aspect, or the data processing method described in the second aspect.

[0042] In a sixth aspect, this application provides a computer-readable storage medium storing computer program instructions, which, when executed by a processor, implement the first aspect, any implementable mode of the first aspect, or the data processor method described in the second aspect.

[0043] In a seventh aspect, embodiments of this application provide a computer program product in which instructions, when executed by a processor of an electronic device, cause the electronic device to perform the data processor method as described in the first aspect, any implementable of the first aspect, or the second aspect.

[0044] The data processor method, data processor, electronic device, and readable storage medium of this application embodiment include a first data processor that can communicate with one or more data processors outside the local machine and establish a subscription relationship with the data processors outside the local machine. The subscription relationship may include subscribing to a first data category published by the first target data processor. Based on this, each time the first target data processor obtains the subscription data corresponding to the first data category, it can promptly send the subscription data corresponding to the first data category to the first data processor based on the Remote Procedure Call (RFC) transmission method. Based on this, the first data processor can save the subscription data corresponding to the first data category in the local storage space, thereby realizing data synchronization with the data processor of the remote node. Thus, when the first data processor is about to be accessed for the data corresponding to the first data category, it can find the accessed data in the local storage space, which is conducive to realizing a highly stable and fast business service. Meanwhile, since the first data processor converts the storage structure of the subscribed data into a columnar storage structure and writes the subscribed data in the columnar storage structure into the storage space corresponding to the first data processor, the first data processor can improve the speed of finding the corresponding data of the first data category in the storage space when it is about to be accessed, thereby effectively saving the access response time. Attached Figure Description

[0045] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments of this application will be briefly introduced below. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0046] Figure 1 This is a schematic diagram of the structure of a data processing system provided in an embodiment of this application;

[0047] Figure 2 This is a flowchart illustrating a data processing method provided in an embodiment of this application;

[0048] Figure 3 This is a schematic diagram of the structure of a data processor provided in an embodiment of this application;

[0049] Figure 4 This is a schematic diagram of the structure of a storage space provided in an embodiment of this application;

[0050] Figure 5 This is a flowchart illustrating another data processing method provided in an embodiment of this application;

[0051] Figure 6 This is a schematic diagram of a data push provided in an embodiment of this application;

[0052] Figure 7 This is a flowchart illustrating another data processing method provided in an embodiment of this application;

[0053] Figure 8 This is a schematic diagram of the structure of a first data processing device provided in an embodiment of this application;

[0054] Figure 9 This is a schematic diagram of the structure of a first target data processing device provided in an embodiment of this application;

[0055] Figure 10 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0056] The features and exemplary embodiments of various aspects of this application will be described in detail below. To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only intended to explain this application and not to limit it. For those skilled in the art, this application can be implemented without some of these specific details. The following description of the embodiments is merely to provide a better understanding of this application by illustrating examples.

[0057] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising..." does not exclude the presence of additional identical elements in the process, method, article, or apparatus that includes said element.

[0058] With the development of data storage technology, cloud and network infrastructures are highly integrated, and network infrastructure is gradually moving towards the cloud. Enterprise application services typically involve large amounts of data; therefore, cloud deployment of enterprise application services based on distributed application service architectures has become a trend. In the process of cloud deployment of enterprise application services, data processing units (DPUs) based on smart network interface cards (NICs) are increasingly widely used. These DPUs have functions such as data storage, reading and writing, encryption and decryption, and compression and decompression, which can effectively reduce the load on the central processing unit (CPU) of electronic devices.

[0059] While deploying enterprise application services across clouds, regions, and clusters has become easier, in distributed application service architectures, accessing data from other nodes across clouds, regions, and clusters via remote networks is a common practical need due to business requirements, resource costs, management requirements, and objective environmental factors. Since network bandwidth across clouds, regions, and clusters remains limited, and providing high-speed, stable services based on various access needs is highly dependent on the physical environment, this leads to significant uncertainty in responding to access requests, especially remote access requests, resulting in slow service response times and low stability.

[0060] In response to this, embodiments of this application provide a data processor method, a data processor, an electronic device, and a readable storage medium, which can quickly respond to data access and facilitate the provision of highly stable and fast business services.

[0061] It should be noted that the acquisition, storage, use, and processing of data in this application embodiment all comply with the relevant provisions of national laws and regulations.

[0062] The data processor method, data processor, electronic device, and readable storage medium provided in the embodiments of this application are described below with reference to the accompanying drawings.

[0063] Figure 1 A schematic diagram of the structure of a data processing system provided in one embodiment of this application is shown. Figure 1 As shown, the data processing system includes multiple data processors, and each data node may include one or more data processors. These data nodes can be deployed in different regions or in the same region, and they can communicate with each other, for example, through remote procedure call (RPC) data communication.

[0064] For ease of description, this application embodiment uses a data node including a data processor as an example.

[0065] In a data processor system, each data node can act as either a data publisher or a data subscriber. For example, the first data node 11 can publish the first data category, and the second data node 21 and the third data node 22 can subscribe to the first data category from the first data node 11. Therefore, the first data node 11 is a data publisher, and the second and third data nodes 21 and 22 are data subscribers. After the first data node 11 updates the subscription data corresponding to the first data category, it can send the subscription data to the second data node 21 and the third data node 22 respectively. This allows the second data node 21 and the third data node 22 to save the subscription data to their respective data storage spaces, thus achieving remote data localization. This enables the retrieval of accessed data in local storage space, facilitating highly stable and fast business services and effectively saving access response time.

[0066] Figure 2 This is a flowchart illustrating a data processing method provided in an embodiment of this application. Specifically, the data processing method can be applied to a first data processor, which is communicatively connected to at least one second data processor. The at least one second data processor includes a first target data processor, wherein the first target data processor uses an interface description language to define the data structure of the data categories to be published.

[0067] Combination Figure 2 As shown, the data processing method may include steps 201 to 203.

[0068] Step 201, the first data processor is used to receive remote procedure call information sent by the first target data processor when there is a first subscription relationship with the first target data processor, wherein the first subscription relationship includes subscribing to a first data category published by the first target data processor, and the remote procedure call information includes the subscription data corresponding to the first data category pushed by the first target data processor;

[0069] Step 202: Parse the remote procedure call information to obtain the subscription data;

[0070] Step 203: Convert the storage structure of the subscription data into a columnar storage structure, and write the subscription data of the columnar storage structure into the storage space corresponding to the first data processor, wherein the storage space includes at least one data category and subscription data corresponding to each data category.

[0071] The steps described above are explained in detail below.

[0072] First, regarding step 201 above, the first data processor can be as follows: Figure 1 The data processor 11 shown, the second data processor may include, for example, Figure 1 The data processors 21 and 22 are shown. The first target data processor can be either data processor 21 or data processor 22. For example, the first target data processor can specifically be as follows: Figure 1 The data processor 21 shown.

[0073] A first subscription relationship exists between the first data processor and the first target data processor, meaning the first data processor subscribes to a first data category published by the first target data processor. Based on this, the first data processor can conveniently receive the subscribed data corresponding to the first data category pushed by the first target data processor in real time.

[0074] The first target data processor can push subscription data to the first data processor based on remote procedure call information. The remote subscription data includes subscription data corresponding to the first data category pushed by the first target data processor. Based on this, after the first target data processor obtains the subscription data corresponding to the first data category, it will push the subscription data corresponding to the first data category to the first data processor in real time.

[0075] In this embodiment of the application, by utilizing the RPC component, an RPC connection pool for the data processing system can be established. By setting the maximum and minimum number of connections and setting network connection hold and disconnection policies, short connections can be dynamically established between nodes, improving network resource utilization. Compared with communication methods such as TCP, this can effectively reduce the problem of long-term occupation of network resources.

[0076] Next, in step 202 above, the first data processor receives remote procedure call information and can obtain the subscription data.

[0077] For example, remote procedure call information has a predetermined data structure. Through the interface description language toolset, specified language code can be automatically generated to parse the remote procedure call information and obtain subscription data.

[0078] Next, in step 203 above, the first data processor can convert the storage structure of the subscription data into a columnar storage structure and write the subscription data of the columnar storage structure into the storage space corresponding to the first data processor.

[0079] It is understandable that the storage space corresponding to the first data processor may include one or more other data categories, as well as the subscription data corresponding to each data category.

[0080] According to the embodiments of this application, data synchronization with the data processor of the remote node is achieved. Therefore, when the first data processor is about to receive access to data corresponding to the first data category, it can locate the accessed data in its local storage space, which is beneficial for achieving highly stable and fast business services. Simultaneously, since the first data processor converts the storage structure of the subscribed data into a columnar storage structure and writes the subscribed data of the columnar storage structure into the storage space corresponding to the first data processor, when the first data processor is about to receive access to data corresponding to the first data category, it is beneficial to improve the speed of locating the corresponding data in the storage space, thereby effectively saving access response time.

[0081] In some embodiments, the storage space corresponding to the first data processor includes a solid-state drive and a hard disk drive;

[0082] In step 203 above, the storage structure of the subscription data is converted into a columnar storage structure, and the subscription data in the columnar storage structure is written into the storage space corresponding to the first data processor. Specifically, this may include:

[0083] Based on the log structure merge tree storage engine, the lowest-level row-ordered key-value pair set corresponding to the subscribed data is determined, and the lowest-level row-ordered key-value pair set is converted into column-ordered storage data.

[0084] Store columnar data onto a hard disk drive; and,

[0085] If the columnar storage data is identified to include text data fields, a storage dictionary for the text data fields and a storage index corresponding to the storage dictionary are generated. The storage dictionary is then stored on a solid-state drive (SSD), and the storage index is stored on a hard disk drive (HDD).

[0086] Specifically, a storage engine based on a log-structured merge tree, such as but not limited to RocksDB (hereinafter referred to as the storage engine), can be configured and installed on the first data processor. Its functionality can be extended. The storage engine writes data in key-value pairs as entire rows, where the values ​​can be complex data structures. Sequential writes improve disk write efficiency. The storage engine's data storage path is set to a mechanical hard disk (HDD). Optionally, a disk array can be used to accelerate data read and write operations. HDDs handle frequent write and update operations, reducing data storage costs compared to the limited write cycles of solid-state drives (SSDs) and ensuring higher data security.

[0087] The data written by the first data processor through the log structure merge tree storage engine, from memory to disk, is an ordered set of key-value pairs. After multiple layers of merging, the data at the bottom layer changes very infrequently. Therefore, the bottom-level data can be converted from row-based key-value pair storage to column-based storage (such as, but not limited to, Parquet), and the data processing operations can be transformed from a dynamic mode to a static mode. Optionally, during the row-to-column conversion process, query frequencies can be statistically analyzed. For queries whose query time exceeds a set threshold, aggregate tables can be constructed from row-based data based on the relationships between one or more tables in their FROM clauses, and the data can be converted into column-based data.

[0088] For example, Figure 3 This is a schematic diagram of the structure of a data processor provided in an embodiment of this application.

[0089] To achieve data acceleration, the first data processor's data acceleration device consists of specific hardware and software, with special configurations for network connectivity and disk storage hardware. Combined with... Figure 3 As shown, the first data processor is an external hardware component on the physical machine. It consists of a central processing unit (CPU), memory, a solid-state drive (SSD), a hard disk drive (HDD), a smart network interface card (NIC), and a data bus. The first data processor provides a driver for the physical host and hosted virtual machines, mapping it to an interface (such as, but not limited to, providing an SDK / API). User processes can directly call the driver interface to read and write data from the storage devices of the first data processor. The software architecture of the first data processor uses the Linux kernel, integrating and managing all external devices, and meeting requirements for high performance, low power consumption, and small size through proprietary customized programs.

[0090] Optionally, a solid-state drive (SSD) dictionary area can be set up. When converting the ordered key-value pair set at the lowest level of the module's storage engine from row-based to column-based storage, the structural information of the data can be used to determine whether the field's attribute is text type. Text data can then be dictionary-encoded to accelerate subsequent data reading. Dictionary encoding technology is used for text data fields, with a basic unit including, but not limited to, a data block. Long strings are mapped to a shortest possible code (also called an index). The SSD stores the dictionary (i.e., the mapping between the index and the original string), while the hard disk drive (HDD) stores the index. The dictionary, stored on the SSD, has a large value length, resulting in faster read speeds. Since the dictionary is shared data, the number of reads can be reduced. The actual index is stored on the HDD, with a smaller index length, reducing the amount of read / write I / O data stored on the HDD. Raw data, especially text data, occupies a large amount of storage space and contains a lot of repetitive content. Optionally, for long text data fields, word segmentation can be performed first using a word segmentation library, followed by dictionary encoding of the segmented words. For non-text data fields, inference mechanisms can be used to determine data characteristics and apply different encodings, such as, but not limited to, RLE encoding for consecutively repeating data and differential encoding for columns with partially identical data. The encoded actual index data is stored on a hard disk in a columnar file format (such as, but not limited to, Parquet). Optionally, the data can be further compressed using compression algorithms (such as, but not limited to, Snappy, Gzip, and LZO) before storage. Encoding / decoding algorithms and compression algorithms can be integrated into the DPU's FPGA to improve data decompression efficiency.

[0091] The first data processor uses an interface description language-related toolset and RPC components to define and transmit data, such as, but not limited to, ProtoBuf / gRPC. All data in the first data processor has its data structure defined using the interface description language within the data configuration management module. The first data processor exposes a data configuration management port, through which other nodes can configure and obtain metadata information for data categories. In the transmission module, the publishing node pushes data to the subscribing nodes via RPC.

[0092] The first data processor uses a hybrid storage method combining solid-state drives (SSDs) and hard disk drives (HDDs). The data writing module uses HDDs and employs a log-structured merging tree-based storage engine to handle data writing, transforming the lowest-level ordered key-value pair set from row-based to column-based storage. The first data processor divides the SSD into a buffer and a dictionary encoding area, while the read module leverages the SSD's higher read efficiency to accelerate data retrieval.

[0093] In some embodiments of this application, the first data processor may also receive data access requests. For example, receiving a data access request may include the following steps:

[0094] Receive a data access request, wherein the data access request includes target data category access information;

[0095] In response to the data access request, in the storage space corresponding to the first data processor, the subscribed data corresponding to the target data class target identifier is obtained based on the target data class target identifier;

[0096] Output the subscription data corresponding to the target class target identification.

[0097] For example, the metadata information of data categories in the first data processor driver interface and the data configuration management module provides standard SQL for structured data reading operations to upper-layer applications. Columnar storage is read-friendly and can reduce disk I / O using techniques such as column pruning and predicate pushdown. When reading data written in dictionary encoding, the index is read from the mechanical hard drive and the dictionary is read from the solid-state drive, and dictionary decoding is performed to restore the data. Other encoded data is decoded and restored according to the corresponding decoder. Optionally, secondary indexes can be created for columnar storage data to improve data retrieval and reading efficiency.

[0098] Optionally, using a buffer within the solid-state drive (SSD) can facilitate faster data read operations. For example, Figure 4 This is a schematic diagram of a storage space structure provided in an embodiment of this application, combined with... Figure 4 As shown, the storage engine allocates a certain amount of memory space as a memory buffer acceleration area and a certain amount of solid-state drive (SSD) storage space as an SSD buffer acceleration area. It utilizes the relatively high read speed of both to accelerate data reading and uses the LRU algorithm to evict data, improving the read hit rate. The SSD buffer acceleration area is configured with a query frequency threshold that allows data to be added and a data eviction rate threshold. The query frequency of data evicted from the memory acceleration area is compared with the threshold set in the SSD acceleration area to determine whether to add it (if both are greater than or equal to the threshold, addition is allowed). When the data eviction rate in the SSD acceleration area exceeds the set threshold, the program will automatically increase the allowed data addition query frequency threshold and decrease the write frequency of the SSD acceleration area. A Bloom filter is added to the SSD acceleration area to reduce unnecessary reads.

[0099] According to the embodiments of this application, data access requests can use SQL to read columnar stored data through the driver interface of the first data processor, and data reading is accelerated through technologies such as columnar storage, memory buffer, solid-state drive buffer, solid-state drive dictionary area, and secondary index.

[0100] In some embodiments, before receiving remote procedure call information sent by the first target data processor, the method further includes:

[0101] Receive first configuration information, wherein the first configuration information includes subscription information, and the first subscription information includes the identifier of the first target data processor and the identifier of the first data category.

[0102] Based on the first subscription information, a subscription request is sent to the first target data processor to establish a first subscription relationship with the first target data processor.

[0103] Specifically, the first target data processor can publish first configuration information. After receiving the first configuration information, if the first data processor needs to subscribe to the first data category included in the first configuration information, it can send a subscription request to the first target data processor, thereby establishing a first subscription relationship with the first target data processor.

[0104] For example, when an upper-layer application publishes a new data category through the driver interface of the first target data processor, the first target data processor first generates a unique data category number. The data configuration management module of the first target data processor has a built-in toolkit related to the interface description language, such as using but not limited to ProtoBuf. Then, it defines the data structure through the interface description language and generates interface description language structure information. All data of the DPU is defined through the interface description language. Finally, the specified language code is automatically generated through the interface description language toolkit for interface calls and data parsing, realizing programming language independence between nodes and improving the flexibility of data transmission and interface calls between nodes.

[0105] The first target data processor records information such as the naming of the new data category and the interface description language structure to the data configuration management module of this node. The data configuration management module manages and configures the data category information uniformly both internally and externally. If it is necessary to cancel the publication, the data category publication status will be updated to cancel publication.

[0106] When the first data processor needs to subscribe to the first data category, it can also add an identifier for the subscription node (such as, but not limited to, IP address / node hostname) to its data category in the data configuration management module, ensuring consistency in the subscription information of the publishing and subscribing nodes. After the first data processor completes the subscription, it becomes a subscribing node. The first target data processor can then push and distribute data for the corresponding data category to the subscribing node. Conversely, when unsubscribing from a data category, the subscription information needs to be deleted from both the publishing and subscribing nodes while maintaining consistency; the unsubscription is then complete.

[0107] To more clearly illustrate the embodiments of this application, Figure 5 This is a flowchart illustrating another data processing method provided in an embodiment of this application, combined with... Figure 5 As shown, the method may include steps 501 to 502.

[0108] Step 501: The first target data processor is used to obtain the subscription data corresponding to the first data category;

[0109] Step 502: Push remote procedure call information to each first data processor so that the first data processor can parse the remote procedure call information to obtain subscription data, convert the storage structure of the subscription data into a columnar storage structure, and write the subscription data of the columnar storage structure into the storage space corresponding to the first data processor, wherein the storage space includes at least one data category and subscription data corresponding to each data category.

[0110] For example, after the first target data processor establishes a subscription relationship with the first data processor, the data transmission module of the first target data processor can be directly connected to the smart network card of the first data processor through the driver interface, such as... Figure 6 As shown, the programmable configuration capability and hardware performance of the smart network card are used to encrypt and decrypt data, transmit data, and control and forward data, thereby improving data transmission efficiency.

[0111] In some embodiments, the first data processor may also act as a publishing node, publishing configuration information to data processors communicatively connected to it. Exemplarily, the data processing method further includes:

[0112] Receive second configuration information, wherein the second configuration information includes a second data category to be published;

[0113] In response to the second configuration information, publishing information is generated according to the second data category to be published. The publishing information includes the identifier of the second data category, the name of the second data category, and the publishing status of the second data category.

[0114] Send the publishing information to each of the second data processors;

[0115] Upon receiving a subscription request from a second target data processor, a second subscription relationship is established with the second target data processor, wherein the at least one first data processor includes the second target data processor, and the second subscription relationship includes subscribing to a second data category published by the first data processor.

[0116] For example, the first data processor may be configured with the network address of the registration center. The registration center can form a cluster of multiple first data processors and can uniformly manage, view and configure the data by connecting to the data configuration management port of the first data processor.

[0117] The data configuration management module of the first data processor includes functions for data publishing, data subscription, and data unsubscription, which can be configured through the driver interface or management port. All data in the first data processor node is stored in a database table format, with each table employing a partitioning and bucketing strategy to reduce the data scanning range during random read / write operations. The basic unit of data configuration management is the data category, with the data category name serving as the table name. Upper-level applications can read and write data using SQL (supporting standard SELECT, FROM, WHERE, etc. statements) through the first data processor driver interface.

[0118] The configuration information may include the following fields: data category number, data category name, data category status, data category publishing node, data category subscription node list, and data structure.

[0119] For example, the data category number can be automatically assigned and generated by the first data processor and serve as a unique identifier for the data category.

[0120] The data category name can be defined by the data category publisher by the first data processor. It cannot be repeated and will be used as the table name for data read and write operations.

[0121] Data category status indicates the publication status of a data category, such as published / cancelled.

[0122] The data category publishing node is the identifier of the data source node. This field contains information such as the IP address or node hostname that can be connected to the network.

[0123] The data category subscription node list can include data category subscription nodes, i.e., the identifiers of the target nodes for data synchronization. The structure type is an array, and each item in the array is the node identifier for data subscription. This field information is the IP address or node hostname that can achieve network connection.

[0124] Data structure, specifically, can be defined through an interface description language to define the data categories and their data structures.

[0125] Based on this, after the first data processor establishes a subscription relationship with the second target data processor, the data processing method executed by the first data processor may further include the following steps: upon receiving the subscription data corresponding to the second data category, sending the subscription data corresponding to the second data category to the second target data processor, so that the second target data processor stores the subscription data corresponding to the second data category in the storage space corresponding to the second target data processor.

[0126] The first data processor pushes data to the second target data processor via RPC. After receiving the data, the second target data processor uses the data category and data structure information from the data configuration management module and the specified language code automatically generated by the interface description language toolset to parse the data. The parsed data is then transferred to the writing module, and the second data processor writes the parsed data into its corresponding storage space.

[0127] To more clearly illustrate the embodiments of this application, Figure 7 This is a flowchart illustrating another data processing method provided in an embodiment of this application. (In conjunction with...) Figure 7 As shown, the data processing method may include steps 701 to 707.

[0128] Step 701: The publishing node defines the data structure of the data category using the interface description language through the driver interface and publishes it to the data configuration management module;

[0129] For example, the publishing node can be the first target data processor, and the subscribing node can be the first data processor.

[0130] Step 702: The subscription node obtains the published data category metadata information from the data configuration management module of the publishing node, configures the subscription information, and synchronizes and updates the subscription information to the data configuration management modules of the publishing node and the subscription node.

[0131] Step 703: The publishing node pushes newly written data for published data categories to the transport layer;

[0132] Step 704: The publishing node's transport layer pushes the data to the subscribing node via RPC, enabling the subscribing node to achieve remote data localization.

[0133] Step 705: After the subscription node transmission module receives the data through the smart network card, it parses the data through the corresponding data category metadata information of the data configuration management module.

[0134] Step 706: The subscription node writing module merges the parsed data through the log structure merging tree storage engine. After merging through multi-level ordered key-value pair sets, the bottom-level data is converted from row-based to column-based storage. The text data fields are then encoded using a solid-state drive dictionary to complete the data disk writing.

[0135] Step 707: The upper-layer application of the subscription node uses SQL through the driver interface to perform read operations on the columnar storage data in the read module. Data reading is accelerated through technologies such as columnar storage, memory buffer, solid-state drive buffer, solid-state drive dictionary area, and secondary index.

[0136] According to embodiments of this application, the first data processor can store the subscription data corresponding to the first data category in local storage space, thereby achieving data synchronization with the data processor of the remote node. Thus, when the first data processor is about to receive access to the data corresponding to the first data category, it can locate the accessed data in local storage space, which is beneficial for achieving highly stable and fast business services. Simultaneously, since the first data processor converts the storage structure of the subscription data into a columnar storage structure and writes the subscription data of the columnar storage structure into the storage space corresponding to the first data processor, when the first data processor is about to receive access to the data corresponding to the first data category, it helps to improve the speed of locating the data corresponding to the first data category in storage space, thereby effectively saving access response time.

[0137] Based on the same inventive concept, this application also provides a data processing apparatus 800 corresponding to the above-described data processing method. (Specifically combined with...) Figure 8 Please provide a detailed explanation.

[0138] Figure 8 This is a schematic diagram of the structure of a first data processing device provided in an embodiment of this application. The first data processing device is communicatively connected to at least one second data processor. The at least one second data processor includes a first target data processor. The first target data processor uses an interface description language to define the data structure of the data categories to be published.

[0139] like Figure 8 As shown, the first data processing device includes:

[0140] The data configuration management module 801 is used to receive remote procedure call information sent by the first target data processor when there is a first subscription relationship with the first target data processor. The first subscription relationship includes subscribing to a first data category published by the first target data processor, and the remote procedure call information includes the subscription data corresponding to the first data category pushed by the first target data processor.

[0141] The write module 802 is used to parse remote procedure call information and obtain subscription data;

[0142] The writing module 802 is further configured to convert the storage structure of the subscription data into a columnar storage structure and write the subscription data of the columnar storage structure into the storage space corresponding to the first data processor, wherein the storage space includes at least one data category and subscription data corresponding to each data category.

[0143] In some embodiments, the first data processing apparatus further includes:

[0144] The driver interface is used to receive data access requests, which include target data category access information.

[0145] The reading module is used to respond to data access requests and retrieve the subscribed data corresponding to the target data class target identifier from the storage space corresponding to the first data processor, based on the target data class target identifier.

[0146] The reading module is also used to output the subscription data corresponding to the target class target identification.

[0147] In some embodiments, the storage space corresponding to the first data processor includes a solid-state drive and a hard disk drive;

[0148] The write module is also used to determine the lowest-level row-ordered key-value pair set corresponding to the subscribed data based on the log structure merge tree storage engine, and to convert the lowest-level row-ordered key-value pair set into column-ordered storage data.

[0149] The write module is also used to store columnar storage data into the hard disk drive; and,

[0150] The write module is also used to generate a storage dictionary for text data fields and a storage index corresponding to the storage dictionary when it is identified that the columnar storage data includes text data fields, and to store the storage dictionary in the solid-state drive and the storage index in the hard disk drive.

[0151] In some embodiments, the driver interface is further configured to receive first configuration information, wherein the first configuration information includes first subscription information, which includes an identifier of a first target data processor and an identifier of a first data category.

[0152] The data configuration management module 801 is also used to send a subscription request to the first target data processor based on the first subscription information, so as to establish a first subscription relationship with the first target data processor.

[0153] In some embodiments, the driver interface is further configured to receive second configuration information, wherein the second configuration information includes a second data category to be published;

[0154] The data configuration management module 801 is also used to respond to the second configuration information and generate publishing information based on the second data category to be published. The publishing information includes the identifier of the second data category, the name of the second data category, and the publishing status of the second data category.

[0155] The data configuration management module 801 is also used to send publishing information to each second data processor;

[0156] The data configuration management module 801 is further configured to establish a second subscription relationship with the second target data processor upon receiving a subscription request from the second target data processor, wherein at least one of the first data processors includes the second target data processor, and the second subscription relationship includes subscribing to a second data category published by the first data processor.

[0157] In some embodiments, the data configuration management module 801 is further configured to send the subscription data corresponding to the second data category to the second target data processor upon receiving the subscription data corresponding to the second data category, so that the second target data processor stores the subscription data corresponding to the second data category in the storage space corresponding to the second target data processor.

[0158] It is understood that the first data processing device in the embodiments of this application can correspond to the execution subject of the data processor method provided in the embodiments of this application. The specific details of the operation and / or function of each module / unit of the first data processing device can be found in the description of the corresponding part in the data processor method provided in the embodiments of this application above. For the sake of brevity, they will not be repeated here.

[0159] In this embodiment, the first data processor can store the subscription data corresponding to the first data category in its local storage space, thereby achieving data synchronization with the data processor of the remote node. Thus, when the first data processor is about to access the data corresponding to the first data category, it can locate the accessed data in its local storage space, which is beneficial for achieving highly stable and fast business services. Simultaneously, since the first data processor converts the storage structure of the subscription data into a columnar storage structure and writes the subscription data of the columnar storage structure into the storage space corresponding to the first data processor, the speed of locating the data corresponding to the first data category in the storage space is improved when the first data processor is about to access the data corresponding to the first data category, thereby effectively saving access response time.

[0160] Figure 9 This is a schematic diagram of the structure of a first target data processing device provided in an embodiment of this application. The first target data processor uses an interface description language to define the data structure of the data categories to be published. The first target data processing device is communicatively connected to at least one first data processor. A first subscription relationship exists between the first data processor and the first target data processing device. The first subscription relationship includes subscribing to the first data categories published by the first target data processing device.

[0161] like Figure 9 As shown, the first target data processing device includes:

[0162] Data configuration management module 901 is used to obtain the subscription data corresponding to the first data category;

[0163] The transmission module 902 is used to push remote procedure call information to each first data processor, so that the first data processor can parse the remote procedure call information to obtain subscription data, convert the storage structure of the subscription data into a columnar storage structure, and write the subscription data of the columnar storage structure into the storage space corresponding to the first data processor, wherein the storage space includes at least one data category and subscription data corresponding to each data category.

[0164] In some embodiments, the first target data processing apparatus further includes:

[0165] The reading module is used to receive data access requests, which include target data category access information.

[0166] The reading module is also used to respond to a data access request, obtain the subscription data corresponding to the target data class identifier in the storage space corresponding to the first data processor, based on the target data class identifier, and output the subscription data corresponding to the target data class identifier.

[0167] It is understood that the first target data processing device in the embodiments of this application can correspond to the execution subject of the data processor method provided in the embodiments of this application. The specific details of the operation and / or function of each module / unit of the first target data processing device can be found in the description of the corresponding part in the data processor method provided in the embodiments of this application above. For the sake of brevity, they will not be repeated here.

[0168] The first target data processing device in this application embodiment can instantly send the subscription data corresponding to the first data category to the first data processor, enabling the first data processor to store the subscription data corresponding to the first data category in its local storage space. This achieves data synchronization with the data processor of the remote node. Consequently, when the first data processor is about to access the data corresponding to the first data category, it can locate the accessed data in its local storage space, which is beneficial for achieving highly stable and fast business services. Simultaneously, since the first data processor converts the storage structure of the subscription data into a columnar storage structure and writes the subscription data of the columnar storage structure into the storage space corresponding to the first data processor, the speed of locating the data corresponding to the first data category in the storage space is improved when the first data processor is about to access the data corresponding to the first data category, thereby effectively saving access response time.

[0169] Figure 10 A schematic diagram of the structure of an electronic device according to an embodiment of this application is shown. Figure 10As shown, the device may include a processor 1001 and a memory 1002 storing computer program instructions.

[0170] Specifically, the processor 1001 may include a central processing unit (CPU), an application specific integrated circuit (ASIC), or one or more integrated circuits that can be configured to implement the embodiments of this application.

[0171] Memory 1002 may include mass storage for information or instructions. For example, and not limitingly, memory 1002 may include a hard disk drive (HDD), floppy disk drive, flash memory, optical disk, magneto-optical disk, magnetic tape, or Universal Serial Bus (USB) drive, or a combination of two or more of these. In one instance, memory 1002 may include removable or non-removable (or fixed) media, or memory 1002 may be non-volatile solid-state memory. Memory 1002 may be internal or external to an electronic device.

[0172] Memory may include read-only memory (ROM), random access memory (RAM), disk storage media devices, optical storage media devices, flash memory devices, and electrical, optical, or other physical / tangible memory storage devices. Therefore, typically, memory includes one or more tangible (non-transitory) computer-readable storage media (e.g., memory devices) encoded with software including computer-executable instructions, and when the software is executed (e.g., by one or more processors), it is operable to perform the operations described with reference to the methods according to one aspect of this disclosure.

[0173] The processor 1001 reads and executes computer program instructions stored in the memory 1002 to implement the method described in the embodiments of this application and achieve the corresponding technical effects achieved by executing the method in the embodiments of this application. For the sake of brevity, it will not be described in detail here.

[0174] In one example, the electronic device may also include a communication interface 1003 and a bus 1004. Wherein, for example... Figure 10 As shown, the processor 1001, memory 1002, and communication interface 1003 are connected through bus 1004 and complete communication with each other.

[0175] The communication interface 1003 is mainly used to realize communication between various modules, devices, units and / or equipment in the embodiments of this application.

[0176] Bus 1004 includes hardware, software, or both, that couples components of an online information flow metering device together. For example, and not limitingly, the bus may include an Accelerated Graphics Port (AGP) or other graphics bus, an Extended Industry Standard Architecture (EISA) bus, a Front Side Bus (FSB), a Hyper Transport (HT) interconnect, an Industry Standard Architecture (ISA) bus, an Infinite Bandwidth Interconnect, a Low Pin Count (LPC) bus, a memory bus, a Microchannel Architecture (MCA) bus, a Peripheral Component Interconnect (PCI) bus, a PCI-Express (PCI-X) bus, a Serial Advanced Technology Attachment (SATA) bus, a Video Electronics Standards Association Local (VLB) bus, or other suitable buses, or combinations of two or more of these. Where appropriate, bus 1004 may include one or more buses. Although specific buses are described and illustrated in embodiments of this application, this application contemplates any suitable bus or interconnect.

[0177] The electronic device can execute the data processor method in the embodiments of this application, thereby achieving the corresponding technical effects of the data processor method described in the embodiments of this application.

[0178] Furthermore, in conjunction with the data processor methods in the above embodiments, this application embodiment can provide a readable storage medium for implementation. This readable storage medium stores computer program instructions; when these computer program instructions are executed by a processor, they implement any of the data processor methods in the above embodiments. Examples of readable storage media can be non-transitory machine-readable media, such as electronic circuits, semiconductor memory devices, read-only memory (ROM), floppy disks, compact disc read-only memory (CD-ROM), optical disks, hard disks, etc.

[0179] It should be clarified that this application is not limited to the specific configurations and processes described above and shown in the figures. For the sake of brevity, detailed descriptions of known methods are omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of this application is not limited to the specific steps described and shown. Those skilled in the art can make various changes, modifications, and additions, or change the order of steps, after understanding the spirit of this application.

[0180] The functional blocks shown in the above-described block diagram can be implemented as hardware, software, firmware, or a combination thereof. When implemented in hardware, they can be, for example, electronic circuits, application-specific integrated circuits (ASICs), appropriate firmware, plug-ins, function cards, etc. When implemented in software, the elements of this application are programs or code segments used to perform the required tasks. Programs or code segments can be stored on a machine-readable medium or transmitted over a transmission medium or communication link via data signals carried on a carrier wave. "Machine-readable medium" can include any medium capable of storing or transmitting information. Examples of machine-readable media include electronic circuits, semiconductor memory devices, read-only memory (ROM), flash memory, erasable read-only memory (EROM), floppy disks, compact disc read-only memory (CD-ROM), optical disks, hard disks, fiber optic media, radio frequency (RF) links, etc. Code segments can be downloaded via computer networks such as the Internet, intranets, etc.

[0181] It should also be noted that the exemplary embodiments mentioned in this application describe methods or systems based on a series of steps or apparatus. However, this application is not limited to the order of the above steps; that is, the steps can be performed in the order mentioned in the embodiments, or in a different order, or several steps can be performed simultaneously.

[0182] This application also provides a computer-readable storage medium storing computer program instructions; when executed by a processor, the computer program instructions implement the data processor method provided in this application.

[0183] Furthermore, in conjunction with the data processor method, apparatus, and readable storage medium described in the above embodiments, this application can provide a computer program product for implementation. When the instructions in the computer program product are executed by the processor of an electronic device, the electronic device causes the electronic device to perform any of the data processor methods described in the above embodiments.

[0184] The aspects of this disclosure have been described above with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this disclosure. It should be understood that each block in the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that these instructions, executable via the processor of the computer or other programmable data processing apparatus, enable the implementation of the functions / actions specified in one or more blocks of the flowchart illustrations and / or block diagrams. Such a processor can be, but is not limited to, a general-purpose processor, a special-purpose processor, a special application processor, or a field-programmable logic circuit. It is also understood that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can also be implemented by special-purpose hardware performing the specified functions or actions, or can be implemented by a combination of special-purpose hardware and computer instructions.

[0185] The above description is merely a specific implementation of this application. Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, modules, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here. It should be understood that the protection scope of this application is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in this application, and these modifications or substitutions should all be covered within the protection scope of this application.

Claims

1. A data processing method, characterized in that, The method is applied to a first data processor, which is communicatively connected to at least one second data processor, wherein the at least one second data processor includes a first target data processor, wherein the first target data processor uses an interface description language to define the data structure of the data categories to be published; the method includes: The first data processor is configured to receive remote procedure call information sent by the first target data processor when a first subscription relationship exists with the first target data processor, wherein the first subscription relationship includes subscribing to a first data category published by the first target data processor, and the remote procedure call information includes subscription data corresponding to the first data category pushed by the first target data processor; The remote procedure call information is parsed to obtain the subscription data; The storage structure of the subscription data is converted into a columnar storage structure, and the subscription data of the columnar storage structure is written into the storage space corresponding to the first data processor. The storage space includes at least one data category and subscription data corresponding to each data category.

2. The method according to claim 1, characterized in that, The method further includes: Receive a data access request, wherein the data access request includes target data category access information; In response to the data access request, in the storage space corresponding to the first data processor, the subscribed data corresponding to the target data class target identifier is obtained based on the target data class target identifier; Output the subscription data corresponding to the target class target identification.

3. The method according to claim 1, characterized in that, The storage space corresponding to the first data processor includes solid-state drives and hard disk drives; The step of converting the storage structure of the subscription data into a columnar storage structure and writing the subscription data of the columnar storage structure into the storage space corresponding to the first data processor includes: Based on the log structure merging tree storage engine, the lowest-level row-ordered key-value pair set corresponding to the subscription data is determined, and the lowest-level row-ordered key-value pair set is converted into column-ordered storage data. The columnar storage data is stored in the hard disk drive; and... If the columnar storage data is identified to include text data fields, a storage dictionary for the text data fields and a storage index corresponding to the storage dictionary are generated, and the storage dictionary is stored in the solid-state drive and the storage index is stored in the hard disk drive.

4. The method according to claim 1, characterized in that, Before receiving the remote procedure call information sent by the first target data processor, the method further includes: Receive first configuration information, wherein the first configuration information includes first subscription information, and the first subscription information includes the identifier of the first target data processor and the identifier of the first data category; Based on the first subscription information, a subscription request is sent to the first target data processor to establish the first subscription relationship with the first target data processor.

5. The method according to claim 1, characterized in that, The method further includes: Receive second configuration information, wherein the second configuration information includes a second data category to be published; In response to the second configuration information, publishing information is generated according to the second data category to be published. The publishing information includes the identifier of the second data category, the name of the second data category, and the publishing status of the second data category. Send the publishing information to each of the second data processors; Upon receiving a subscription request from a second target data processor, a second subscription relationship is established with the second target data processor, wherein the at least one first data processor includes the second target data processor, and the second subscription relationship includes subscribing to a second data category published by the first data processor.

6. The method according to claim 5, characterized in that, The method further includes: Upon receiving the subscription data corresponding to the second data category, the subscription data corresponding to the second data category is sent to the second target data processor so that the second target data processor stores the subscription data corresponding to the second data category in the storage space corresponding to the second target data processor.

7. A data processing method, characterized in that, The method is applied to a first target data processor, which uses an interface description language to define the data structure of the data categories to be published. The first target data processor is communicatively connected to at least one first data processor, wherein a first subscription relationship exists between the first data processor and the first target data processor, and the first subscription relationship includes subscribing to a first data category published by the first target data processor; the method includes: The first target data processor is used to acquire subscription data corresponding to the first data category; Remote procedure call information is pushed to each of the first data processors, so that the first data processors parse the remote procedure call information to obtain the subscription data, convert the storage structure of the subscription data into a columnar storage structure, and write the subscription data in the columnar storage structure into the storage space corresponding to the first data processor, wherein the storage space includes at least one data category and subscription data corresponding to each data category.

8. A first data processing apparatus, characterized in that, The first data processing device is communicatively connected to at least one second data processor, wherein the at least one second data processor includes a first target data processor, wherein the first target data processor uses an interface description language to define the data structure of the data categories to be published; the device includes: The data configuration management module is used to receive remote procedure call information sent by the first target data processor when there is a first subscription relationship with the first target data processor, wherein the first subscription relationship includes subscribing to a first data category published by the first target data processor, and the remote procedure call information includes subscription data corresponding to the first data category pushed by the first target data processor; The writing module is used to parse the remote procedure call information to obtain the subscription data; The writing module is further configured to convert the storage structure of the subscription data into a columnar storage structure, and write the subscription data of the columnar storage structure into the storage space corresponding to the first data processor, wherein the storage space includes at least one data category and subscription data corresponding to each data category.

9. A first target data processing device, characterized in that, The first target data processing device uses an interface description language to define the data structure of the data categories to be published. The first target data processing device is communicatively connected to at least one first data processor. A first subscription relationship exists between the first data processor and the first target data processing device. The first subscription relationship includes subscribing to the first data categories published by the first target data processing device. The device includes: The data configuration management module is used to obtain the subscription data corresponding to the first data category; The transmission module is configured to push remote procedure call information to each of the first data processors, so that the first data processors parse the remote procedure call information to obtain the subscription data, convert the storage structure of the subscription data into a columnar storage structure, and write the subscription data in the columnar storage structure into the storage space corresponding to the first data processor, wherein the storage space includes at least one data category and subscription data corresponding to each data category.

10. The first target data processing apparatus according to claim 9, characterized in that, The device further includes: A reading module is used to receive data access requests, wherein the data access requests include target data category access information; The reading module is further configured to, in response to the data access request, obtain the subscription data corresponding to the target data class identifier in the storage space corresponding to the first data processor, based on the target data class identifier, and output the subscription data corresponding to the target data class identifier.

11. An electronic device, characterized in that, The device includes: a processor, and a memory storing computer program instructions; The processor reads and executes the computer program instructions to implement the data processing method as described in any one of claims 1-7.

12. A readable storage medium, characterized in that, The readable storage medium stores computer program instructions, which, when executed by a processor, implement the data processing method as described in any one of claims 1-7.

13. A computer program product, characterized in that, When the instructions in the computer program product are executed by the processor of the electronic device, the electronic device performs the data processing method as described in any one of claims 1-7.

Images