Data processing method and device

Abstract

The present application provides a data processing method and device, wherein the data processing method comprises: obtaining data to be processed and the data type of the data to be processed; determining the target node and associated node corresponding to the data type from a pre-established record tree, wherein the record tree is established according to the parent-child relationship between different data types, and the associated node includes the parent node and the sibling node of the target node; and obtaining the associated data of the data to be processed based on the target node and the associated node. The present solution can improve the data processing efficiency.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a data processing method. The present application also relates to a data processing device, a computing device, and a computer-readable storage medium. Background Art

[0002] With the development of information technology, the amount and type of data involved in data processing have exploded. In order to improve the efficiency of determining target data from a large amount of data, candidate data is usually determined first, and then the target data is determined from the candidate data. Especially in scenarios such as retrieval and recommendation, in order to ensure the richness and personalization of the target data to improve the user experience, it is necessary to obtain additional associated data that is different from the data type of the candidate data and has an associated relationship with the candidate data. On this basis, the target data can be determined from the candidate data based on the associated data. For example, the candidate data is the product identifier that reaches the specified keyword, and the associated data is the store name, etc.

[0003] In the related art, in order to reduce the index depth and improve the data processing efficiency, different types of data usually correspond to different indexes. In addition, in order to ensure the richness and personalization of the target data, the above-mentioned related data are often of multiple types. Therefore, it is necessary to use different types of indexes to determine the related data respectively, and it is necessary to process the indexes multiple times, which increases the number of data processing times and reduces the data processing efficiency. Summary of the invention

[0004] In view of this, an embodiment of the present application provides a data processing method. The present application also relates to a data processing device, a computing device, and a computer-readable storage medium to solve the problem of reduced data processing efficiency in the prior art.

[0005] According to a first aspect of an embodiment of the present application, a data processing method is provided, including:

[0006] Acquire data to be processed and the data type of the data to be processed;

[0007] Determine a target node and associated nodes corresponding to the data type from a pre-established record tree, wherein the record tree is established according to the parent-child relationship between different data types, and the associated nodes include the parent node and sibling nodes of the target node;

[0008] Based on the target node and the associated node, associated data of the data to be processed is obtained.

[0009] According to a second aspect of an embodiment of the present application, there is provided a data processing device, including:

[0010] An acquisition module is configured to acquire data to be processed and a data type of the data to be processed;

[0011] A node determination module is configured to determine a target node and associated nodes corresponding to the data type from a pre-established record tree, wherein the record tree is established according to a parent-child relationship between different data types, and the associated nodes include a parent node and a sibling node of the target node;

[0012] The associated data determination module is configured to obtain the associated data of the data to be processed based on the target node and the associated node.

[0013] According to a third aspect of an embodiment of the present application, a computing device is provided, comprising a memory, a processor, and computer instructions stored in the memory and executable on the processor, wherein the processor implements the steps of the data processing method when executing the instructions.

[0014] According to a fourth aspect of an embodiment of the present application, a computer-readable storage medium is provided, which stores computer instructions, and when the instructions are executed by a processor, the steps of the data processing method are implemented.

[0015] An embodiment of the present application realizes obtaining the data to be processed and the data type of the data to be processed; from a pre-established record tree, the target node and the associated node corresponding to the data type are determined, and based on the target node and the associated node, the associated data of the data to be processed is obtained. Among them, the record tree is established according to the parent-child relationship between the data types of different data, and the associated nodes include the parent node and the brother node of the target node. Therefore, based on the target node and the associated node, the obtained data is in an associated relationship with the data to be processed and is different from the data type of the data to be processed, and therefore can be used as the associated data of the data to be processed. In this way, it is equivalent to using the parent-child relationship between different data types to process an object: record tree once to obtain the associated data, without the need to process multiple objects corresponding to different data types: different indexes multiple times. Therefore, this solution can reduce the number of data processing times in the determination of associated data and improve data processing efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 is a flow chart of a data processing method provided by an embodiment of the present application;

[0017] Figure 2 It is an example diagram of a record tree in a data processing method provided in an embodiment of the present application;

[0018] Figure 3 It is an example diagram of a relationship between data in a record tree in a data processing method provided in an embodiment of the present application;

[0019] Figure 4 is a flow chart of a data processing method provided by another embodiment of the present application;

[0020] Figure 5 It is an example diagram of a data table stored in a record tree in a data processing method provided in another embodiment of the present application;

[0021] Figure 6 is a structural schematic diagram of a data processing device provided by an embodiment of the present application;

[0022] Figure 7 It is a structural block diagram of a computing device provided in one embodiment of the present application. DETAILED DESCRIPTION

[0023] Many specific details are described in the following description to facilitate a full understanding of the present application. However, the present application can be implemented in many other ways than those described herein, and those skilled in the art can make similar generalizations without violating the connotation of the present application, so the present application is not limited by the specific implementation disclosed below.

[0024] The terms used in one or more embodiments of the present application are only for the purpose of describing specific embodiments, and are not intended to limit one or more embodiments of the present application. The singular forms of "a", "said" and "the" used in one or more embodiments of the present application and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used in one or more embodiments of the present application refers to and includes any or all possible combinations of one or more associated listed items.

[0025] It should be understood that, although the terms first, second, etc. may be used to describe various information in one or more embodiments of the present application, these information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of one or more embodiments of the present application, the first may also be referred to as the second, and similarly, the second may also be referred to as the first. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0026] First, the terms involved in one or more embodiments of the present application are explained.

[0027] Structured data: refers to data that strictly complies with the data format Schema definition and length definition. Structured data is easy to be hosted in the memory pool for management.

[0028] Slab memory allocation: A technique that can efficiently support the application and management of small blocks of memory. In this application, it is used to store variable-length data in data, namely fields.

[0029] Retrieval Trigger: refers to the initial processing of the retrieval: based on the query statement or basic directional conditions, a certain amount of result sets are retrieved from the corresponding manuscript library for subsequent multiple rounds of screening.

[0030] In the present application, a data processing method is provided. The present application also relates to a data processing apparatus, a computing device, and a computer-readable storage medium, which are described in detail one by one in the following embodiments.

[0031] Figure 1 A flowchart of a data processing method provided according to an embodiment of the present application is shown, which specifically includes the following steps:

[0032] S102, obtaining data to be processed and the data type of the data to be processed.

[0033] Among them, the data to be processed can be data for which associated data needs to be obtained, for example, candidate data that meets specified conditions. There can be multiple ways to obtain the data to be processed. Exemplarily, the data to be processed can be received directly; or, by retrieving the Trigger, the data set that meets the specified query information or target conditions is queried as the data to be processed. For example, in the scenario recommended by the dress-up mall, candidate package data that meets the query information can be obtained from the dress-up package data based on query information such as region, gender, and age. The candidate package data is the data to be processed. On this basis, in order to improve the richness and personalization of the recommended dress-up packages, it is necessary to obtain the associated data of the candidate package data: creator data, dress-up components, dress-up tags, control strategies and other data.

[0034] Furthermore, the data type can be divided according to the differences in standards such as the business process stage to which the data belongs and the physical meaning represented by the data, and can be set according to specific needs, which is not limited in this embodiment. For example, the data type of the above-mentioned dressing package data is a business output type, the data types of the creator and the dressing label are both data attribute types, the data type of the control strategy is a business scheduling type, and the data type of the dressing component is a scheduling tool type. There can be many ways to obtain the data type of the data to be processed. Exemplarily, when the data to be processed contains a data type, the data type in the data to be processed can be extracted; or, the data identifier of the data to be processed is determined, and the data type corresponding to the data identifier of the data to be processed is found from the correspondence between the pre-stored data identifier and the data type to obtain the data type of the data to be processed.

[0035] S104, determining the target node and associated nodes corresponding to the data type of the to-be-processed data from a pre-established record tree, wherein the record tree is established according to the parent-child relationship between different data types, and the associated nodes include the parent node and sibling nodes of the target node.

[0036] In a specific application, a pre-established record tree is established according to the parent-child relationship between different data types. Therefore, different data can be stored in the corresponding nodes of the record tree according to the parent-child relationship between different data types. For example, the data type of the shop information and the data type of the product information are in a parent-child relationship, then the shop information is stored in the parent node, and the product information is stored in the child node of the parent node. In addition, the storage form of the data in the nodes of the record tree can be various. Exemplarily, the data can be stored directly in the nodes of the record tree, or the data can be stored in the nodes of the record tree in the form of structured data, such as a data table. In order to facilitate understanding and reasonable layout, the second example case will be specifically described in the form of an optional embodiment later. Among them, the brother node of the target node refers to a node that has the same parent node as the target node.

[0037] Furthermore, the target nodes and associated nodes corresponding to the data type of the data to be processed can be determined from the pre-established record tree, which can be multiple. Exemplarily, if the data to be processed is stored in the record tree, the record tree and the data to be processed can be matched, and the node with successful matching can be determined as the target node corresponding to the data type of the data to be processed, and the parent node and brother node of the target node can be determined as the above-mentioned associated nodes. Alternatively, exemplarily, if the data to be processed is not stored in the record tree, and the nodes of the record tree are marked with corresponding data types, the data type of the data to be processed can be matched with the record tree, and the node with successful matching can be determined as the target node corresponding to the data type of the data to be processed, and the parent node and brother node of the target node can be determined as the above-mentioned associated nodes.

[0038] Any method that can determine the target node and associated node corresponding to the data type of the to-be-processed data from the pre-established record tree can be used in the present invention, and this embodiment does not limit this.

[0039] S106, obtaining associated data of the data to be processed based on the target node and the associated nodes.

[0040] In a specific application, the above-mentioned record tree is established according to the parent-child relationship between different data types, so the data of the parent data type is the parent data, and the data of the child data type is the child data. In addition, based on the target node and the associated node, there are multiple ways to obtain the associated data of the data to be processed. Exemplarily, if the data to be processed is not stored in the record tree, the associated data of the data to be processed can be obtained according to the data stored in the target node and the associated node. Or, exemplary, if the data to be processed is stored in the record tree, the data different from the data to be processed can be obtained according to the data stored in the target node and the associated node as the above-mentioned associated data. Wherein, if the data itself is stored in any node, the data in the node is directly read as the acquired data. Or, if the index of the data is stored in any node, the data is acquired according to the index. In addition, if the above-mentioned parent data and child data are one-to-one, and the data itself is stored in the node, the data in the child node is the associated data of the data in the parent node. In this way, when the record tree does not store the data to be processed, the data stored in the target node and the associated node can be directly determined as the associated data. In order to facilitate understanding and reasonable layout, the case of storing indexes and the case of one-to-many between parent data and child data will be specifically described in the form of optional embodiments later.

[0041] In addition, in one case, if the data of the above-mentioned different data types belong to the same business, and the record tree is established according to the parent-child relationship between different data types. For example, a dress set recommendation business corresponds to a record tree, a product search business corresponds to a record tree, a video recommendation business corresponds to a record tree, and so on. In this way, for any record tree, there is an association relationship between the data determined based on each node in the record tree. Therefore, in order to further improve the richness of the data and the accuracy of personalization, this step can also expand more nodes for determining related data based on the target node and the associated node. How to expand the node can be set according to the specific needs of the associated data, and this embodiment does not limit this. The following is an exemplary description of the nodes that can be expanded.

[0042] Exemplarily, the node for determining the associated data may also include: at least one of the child nodes of the target node, the child nodes of the sibling nodes of the target node, the parent node of the parent node of the target node, etc. Moreover, the nodes of each of the above extensions also conform to the above-mentioned parent-child relationship. Therefore, the method for determining the associated data is similar to the method provided in the embodiment of the present application, except that the specific nodes are different. In this regard, the specific nodes can be substituted in the steps provided in the embodiment of the present application according to the above-mentioned parent-child relationship for replacement. For the same content, it can be seen that the data processing method provided in the embodiment of the present application is not repeated here.

[0043] In one embodiment of the present application, a record tree is established based on the parent-child relationship between data types of different data, and the associated nodes include the parent node and sibling nodes of the target node. Therefore, based on the target node and the associated node, the obtained data is in an associated relationship with the data to be processed and is different from the data type of the data to be processed, and therefore can be used as associated data of the data to be processed. In this way, it is equivalent to using the parent-child relationship between different data types to process an object: a record tree once to obtain associated data, without the need to process multiple objects: different indexes corresponding to different data types multiple times. Therefore, this solution can reduce the number of times data is processed in determining associated data and improve data processing efficiency.

[0044] In an optional implementation, before obtaining the data to be processed and the data type of the data to be processed, the data processing method provided in the embodiment of the present application may further include the following steps:

[0045] For each data type, multiple data having the data type are stored as a data table;

[0046] According to the parent-child relationship between different data types, multiple data tables are stored in corresponding nodes of the record tree respectively.

[0047] In specific applications, for various data types, the specific way of storing multiple data with the data type as a data table can be similar to the design of a relational database table. Specifically, different data tables can be defined according to the data abstraction of various data types in the business targeted by data processing. Among them, each data table has at most one parent data table, and the root table is the one without a data parent table. Each data table can have multiple child tables, and a tree structure is formed between different data tables. In addition, each data table can define the schema of the data table itself, and the records stored in the table can be structured data. In addition, in order to further improve the data processing efficiency, the above-mentioned data table can be a data table that can be stored in memory, so that the record tree can be loaded into the memory, which can be effectively applied to the scene where there is a demand for data processing efficiency such as retrieval. In addition, in order to further improve the data processing efficiency, the fixed-length data with a fixed field length in the stored data can be stored in the memory pool, and the index of the variable-length data with a non-fixed field length can be stored in the memory pool. Therefore, in the process of defining the data table, the fields of the above-mentioned fixed-length data and variable-length data can also be defined.

[0048] Exemplarily, the above data table can be defined using the following code:

[0049]

[0050]

[0051] The table_id is set for the data table, the parent_table_id is set for the parent data table in the data table, and the children_table_ids is set for the child data table in the data table. In addition, the size of the child data table, children_table_size, is set, and the field definition schema for fixed-length data <payloadtype>schema, and set up the memory pool RecordPool for storing fixed-length data <payloadtype>record_pool.

[0052] For example, Figure 2 In a data processing method provided by an embodiment of the present application, an example diagram of a record tree is shown; the node where the data table TableA is located is the root node, which is also the parent node. TableA is the root table, and there may be multiple child tables: data tables TableA to TableD, which are respectively stored in the nodes of the root node. Similarly, data table TableB has multiple child tables: data tables Table E to TableF, which are respectively stored in the child nodes of the node where data table TableB is located. The child table of data table TableC is data table TableG, which is stored in the child node of the node where data table TableC is located. In this way, through this embodiment, data tables of various data types can be constructed into a multi-level tree, that is, a record tree, according to the parent-child relationship between different data types.

[0053] This embodiment can ensure that the stored data is structured data by storing multiple data of the same data type as a data table, which is conducive to the convenience of data processing. In addition, storing multiple data in a data table can further improve the richness of the data.

[0054] In an optional implementation, the above data table also stores the index value of the child data; the associated data includes the target parent data and the target child data;

[0055] Accordingly, the above-mentioned obtaining the associated data of the data to be processed based on the target node and the associated node may specifically include the following steps:

[0056] Obtain the target index value of the data to be processed, and according to the target index value, obtain the index value of the target parent data of the data to be processed and the child data of the target parent data from the data table stored in the parent node of the target node;

[0057] According to the index value of the child data of the target parent data, the target child data is obtained from the data table stored in the target node and the sibling nodes of the target node.

[0058] In a specific application, when the node of the record tree stores a data table, there are multiple data in the target node and the associated node. And, in order to further improve the richness of the data, the parent data and the child data are in a one-to-many relationship. In this way, the associated data may include the target parent data: the target data in the parent node of the target node; and the target child data; the target data in the target node and the target data in the brother node of the target node. Among them, there are multiple ways to obtain the target index value of the data to be processed. Exemplarily, if the data to be processed is stored in the record tree, the data to be processed can be input into a preset index value model to obtain the target index value of the data to be processed. The preset index value model can specify a hash algorithm, a correspondence table between a pointer and a data identifier, and the like. Alternatively, if the data to be processed is not stored in the record tree, the index value of the specified data in the data table stored in the target node can be determined as the target index value, which is reasonable.

[0059] Furthermore, the above-mentioned data table may also store the index value of the child data, so as to determine the child data of the parent data. The index value is used to indicate the storage location of the data, and may specifically be a hash value, a pointer, a data identifier, and the like. Therefore, this embodiment can take into account both the richness and accuracy of the data. Furthermore, the index value of the stored child data may be at least one index value of the child data. Accordingly, in this embodiment, according to the target index value, the target parent data of the data to be processed and the index value of the child data of the target parent data are obtained from the data table stored in the parent node of the target node, and specifically, the index value may be multiple. This is specifically described below in the form of an optional embodiment.

[0060] In an optional implementation, the above target parent data has multiple child data;

[0061] Correspondingly, the above method of obtaining the target parent data of the data to be processed from the data table stored in the parent node of the target node according to the target index value may specifically include the following steps:

[0062] According to the target index value, determine the index value of the sibling data having the same data type as the data to be processed;

[0063] The index value stored in the data table of the parent node of the target node is matched with the specified index value, the successfully matched data is determined to be the target parent data of the data to be processed, the index value of the child data of the target parent data is read, and the specified index value is at least one of the target index value and the index value of the sibling data.

[0064] In a specific application, the row where the parent data is located can store the index value of at least one child data of the parent data. In this regard, if in the process of constructing the record tree, the memory, number of child data tables and the size of the storage space occupied by each row of records in the child data table have been allocated in the definition of the data table, then the number of child records in the child data table, that is, the number of child data, is uncertain. Therefore, in order to further save storage space and improve efficiency, the parent record, that is, the parent data, can only store the specified child data in each child data table, such as the first child data. The other child data of the parent data except the specified child data can be traversed and accessed through the sibling chain corresponding to the specified child data. In this way, in this embodiment, the specified index value: at least one of the target index value and the index value of the sibling data is the index value of the specified child data.

[0065] The brother link can be a brother relationship between sub-data, or the brother data can be stored in a chain structure in the sub-data table. Figure 3 In a data processing method provided in an embodiment of the present application, an example diagram of the relationship between data in a record tree is provided; parent data 0 is stored in table A, and the first child data of parent data 0 in child data table B is 0. Other child records in child data table B, such as child data 1 and child data 2, are maintained through the internal sibling chain in child data table B. Similarly, parent data 0 of table A has child data 0 and child data 1 in child data table C. All child data point to the only parent data of the child data, such as the parent-child relationship between data 0 in child data table B and data 0 in child data table E. In this way, a multi-level parent-child relationship between tables is formed.

[0066] In an optional implementation, data of any data type may include fixed-length data and variable-length data;

[0067] Accordingly, before obtaining the data to be processed and the data type of the data to be processed, the method provided in the embodiment of the present application may further include the following steps:

[0068] For each data type, store the fixed-length data of the data type and the index value of the variable-length data of the data type into the data table of the data type, and store the variable-length data of the data type into the first storage pool;

[0069] Based on the parent-child relationship between each data type, the data table of each data type is constructed as a record tree; a fixed second storage pool is divided from the storage space, and the record tree is stored in the second storage pool.

[0070] Among them, fixed-length data refers to data with a fixed field length, such as integers, floating-point data, character arrays of specified length, etc. Variable-length data refers to data with variable field lengths, such as strings, maps, lists, etc. with indefinite field lengths. In specific applications, the second storage pool may be a storage space in a memory pool, such as Payload, and the first storage pool may be a storage space determined by SlabMemoryPool. In order to facilitate understanding and reasonable layout, the specific code for implementing the storage space division in this embodiment and the specific structure of the data table in the above embodiment will be exemplified in conjunction with the update of the record tree.

[0071] This embodiment stores fixed-length data and variable-length data differently, thereby reducing multiple processes for dividing storage space for data, thereby further improving data processing efficiency.

[0072] For ease of understanding, some of the above embodiments are integrated and described below in the form of exemplary illustrations. Figure 4 As shown in a flowchart of a data processing method provided by another embodiment of the present application, the data processing method may include the following steps:

[0073] S402, obtaining data to be processed and the data type of the data to be processed.

[0074] S404: Determine the target node and associated node corresponding to the data type of the data to be processed from the pre-established record tree.

[0075] S406, obtaining a target index value of the data to be processed, and determining, according to the target index value, an index value of sibling data having the same data type as the data to be processed.

[0076] S408, matching the index value stored in the data table of the parent node of the target node with the specified index value, determining that the successfully matched data is the target parent data of the data to be processed, and the specified index value is at least one of the target index value and the index value of the sibling data.

[0077] S410, obtaining target child data from a data table stored in a target node and a sibling node of the target node according to an index value of the child data of the target parent data.

[0078] S412: Determine the target parent data and the target child data as associated data.

[0079] The steps in this embodiment are similar to those in the above Figure 1 The same steps described in the embodiment and the optional embodiment are the same steps, and the details are as follows. Figure 1 Description of embodiments and optional embodiments. For memory-intensive application scenarios such as online retrieval, it is usually necessary to load associated data into the memory. The loading of associated data increases the occupation of memory resources and easily affects data processing efficiency. In this regard, this embodiment can relatively reduce the memory resource occupation caused by loading associated data in the form of index values ​​of sub-data, and further improve data processing efficiency.

[0080] Moreover, in one case, a space-for-time method is adopted: by splicing the data to be processed and the associated data, the time spent on searching the associated data is reduced. However, data of different data types often have different update frequencies, and the above splicing easily leads to frequent data splicing, resulting in a waste of resources. However, the present embodiment stores data of different data types in different data tables or different nodes of the record tree, and can independently update the data of the data type for different data types, thereby meeting different update requirements and reducing resource waste. Moreover, in the above record tree, a mode can be set for nodes to store structured data. Therefore, through the data processing method provided by the embodiment of the present application, there is no need to perform structured transformation on the acquired associated data, thereby further improving the data processing efficiency.

[0081] In a specific application, the data method provided by the embodiment of the present application may also include updating the current record tree, and the update may support the row-level lock of the root data table instead of the global lock of the entire record tree for data update, and may also support the lock-free update in the form of double buffer replacement of the record tree. The following is a specific description in the form of an optional embodiment.

[0082] In an optional implementation manner, the data processing method provided in the embodiment of the present application may further include the following steps:

[0083] Receive update information for the current record tree;

[0084] According to the update information, the current record tree is updated using the tree structure of the current record tree.

[0085] Among them, the tree structure of the current record tree can support the read-write lock of the root node instead of the global lock of the entire record tree, and can also support the lock-free update of the record tree in the form of double buffer replacement, so that the update efficiency can be improved. In addition, in the present application, the occupancy status of the current record tree can be recorded in the form of a bidirectional linked list in combination with the tree structure of the current record tree for the update of the current record tree. The following is a specific description in the form of an optional embodiment.

[0086] In an optional implementation manner, the above-mentioned updating of the current record tree according to the update information and using the tree structure of the current record tree may specifically include the following steps:

[0087] Copy the current record tree, update the copied record tree according to the update information, obtain an updated record tree, and replace the current record tree with the updated record tree;

[0088] When a read / write request for the current record tree is received, it is determined whether the update of the copied record tree has been completed. If not, the read / write request is executed on the current record tree.

[0089] In a specific application, the version information (version) of the record tree can be used to execute this embodiment. Among them, replacing the current record tree with the updated record tree can specifically be deleting the current record tree and using the updated record tree. Exemplarily, the version information of the current record tree is version 1.0, and the version information of the copied record tree is version 1.1. When the update of the record tree of version 1.1 has been completed, the record tree of version 1.0 is deleted, that is, the delayed release of the record tree of the current version. In this way, if a read and write request for the record tree of version 1.0 is received during the data update process, that is, when the update of the copied record tree is not completed, the read and write operation can be directly performed on the record tree of version 1.0. Correspondingly, if the update of the record tree of version 1.1 has been completed, the read and write request is executed on the record tree of version 1.1. In one case, if the above update is deletion, and the data targeted by the read and write request is the deleted data, it is possible to wait for the record tree of version 1.1 to complete the update.

[0090] In this way, this embodiment is a lock-free update, which takes into account both the correctness and efficiency of data reading and writing by updating the record tree of the new version and delaying the release of the record tree of the current version.

[0091] In an optional implementation manner, the above-mentioned updating of the current record tree according to the update information and using the tree structure of the current record tree may specifically include the following steps:

[0092] Set a read-write lock on the root node of the current record tree;

[0093] The current record tree is updated according to the update information, and the read-write lock is released when the update of the current record tree is completed.

[0094] This embodiment is updated by setting a read-write lock of the root node. In a specific application, if a data table is stored in a node of the record tree, the read-write lock of the root node is a row-level lock of the root data table. Accordingly, setting a read-write lock for the root node of the current record tree may specifically include: setting a read-write lock for each row of the data table stored in the root node of the current record tree. A read-write lock means that when multiple threads issue a read request for the same data, multiple threads can execute simultaneously; when multiple threads issue a write request for the same data, multiple threads can only execute serially. In addition, for the same data, if the thread that issues a read request is executing, the thread that issues a write request must wait for the thread that issues the read request to finish executing before it can start executing; if the thread that issues a write request is executing, the thread that issues the read request must also wait for the former to finish executing before it can start executing. In this way, the correctness of data reading in a multi-threaded scenario can be guaranteed.

[0095] In addition, if the update is determined to modify the fields of the stored data according to the update information, the above-mentioned method of setting a read-write lock can be used to update in-place in the corresponding memory. If the update is determined to add or delete data according to the update information, the above-mentioned update method of copying the record tree can be used. In this way, the data copy cost and data reading cost can be balanced.

[0096] In an optional implementation, the update information may include: the data to be added and the data type of the data to be added;

[0097] Furthermore, before obtaining the data to be processed and the data type of the data to be processed, the data processing method provided in the embodiment of the present application may further include the following steps:

[0098] According to the occupation status of the data table in the current record tree, a free linked list for recording the idle status of the node and an occupied linked list for recording the occupation status of the node are created;

[0099] Correspondingly, the above-mentioned updating of the current record tree according to the update information and using the tree structure of the current record tree may specifically include the following steps:

[0100] Determine the storage location of the data to be added according to the data type and the free linked list of the data to be added, and store the data to be added in the storage location of the data table stored in the current record tree;

[0101] The idle state record corresponding to the storage location of the data to be added is deleted from the idle linked list, and the occupied state record corresponding to the storage location of the data to be added is added to the occupied linked list.

[0102] In specific applications, both the free linked list and the occupied linked list can be bidirectional linked lists. A bidirectional linked list means that the data in the linked list can be searched from the front to the back, or from the back to the front. In addition, the linked list type of the above-mentioned bidirectional linked list can represent the occupied state and the idle state. For example, the linked list type of the free linked list is "free", which represents the idle state, and the linked list type of the occupied linked list is "occupied", which represents the occupied state. Specifically, the bidirectional linked list can include a linked list type identifier, such as the identifier "free" for the "free" type, and the identifier "used" for the "occupied" type. In this way, the free linked list and the occupied linked list can store the position information of each row in each data table of the record tree. Among them, the position information of each row can specifically be the identifier or index value of the data in each row, which is used to point to the storage location of the data. On this basis, the storage location of the data to be added is the position information of the node where the data table is located in the current record tree, and the position information of the row in the data table used to store the data to be added.

[0103] On the basis of the above, the present embodiment determines the storage location of the data to be added according to the data type of the data to be added and the free linked list, which may specifically include: determining the storage node of the data to be added in the record tree according to the data type of the data to be added, searching the free linked list for the location information of the free row of the data table stored by the storage node, and obtaining the storage location of the data to be added. In addition, deleting the idle state record corresponding to the storage location of the data to be added from the free linked list, and adding the occupied state record corresponding to the storage location of the data to be added to the occupied linked list, which may include: deleting the storage location of the data to be added from the free linked list, and adding the storage location of the data to be added to the occupied linked list. In this way, the storage location combined with the linked list type can realize the role of the occupied state information.

[0104] In addition, the occupied linked list and the free linked list record the position of all rows in the data table and the occupation status information of the position. For example, if a row is occupied, the occupied status information corresponding to the row in the occupied linked list is 1, and if a row is free, the free status information corresponding to the row in the free linked list is 1. In this way, when deleting or adding status information, the status information in the corresponding linked list can be directly processed.

[0105] This embodiment can add data through the free linked list and the occupied linked list without reallocating the storage space occupied by the data table, thereby further improving the efficiency of data processing.

[0106] In an optional implementation, the update information may include: data to be deleted and the data type of the data to be deleted;

[0107] Correspondingly, the above-mentioned updating of the current record tree according to the update information and using the tree structure of the current record tree may specifically include the following steps:

[0108] Determine the storage location of the data to be deleted according to the data type and the occupied linked list of the data to be deleted, and delete the data to be deleted from the storage location of the data table in the pre-established record tree;

[0109] The occupied state record corresponding to the storage location of the data to be deleted is deleted from the occupied linked list, and the idle state record corresponding to the storage location of the data to be deleted is added to the idle linked list.

[0110] The steps of this embodiment are similar to the above-mentioned embodiment of adding data to be added in the record tree, except that the data processed is data to be deleted, and the linked list targeted is an occupied linked list. For the same parts, reference can be made to the steps of determining the storage location of the data to be added in the above-mentioned embodiment, which will not be repeated here. This embodiment accurately deletes the data in the record tree through the occupied linked list and the free linked list, and the storage space of the deleted data can be directly used to store the newly added data without repeatedly dividing the new storage space, so the data processing efficiency can be further improved.

[0111] Exemplarily, each data table can maintain the data table through the above-mentioned occupied linked list (free_list) and free linked list (used_list), that is, manage the storage space of the data table, such as memory. On this basis, the entire memory can be divided at one time during the process of building the record tree, without multiple memory divisions, that is, memory applications. Specifically, a bidirectional linked list for managing storage space can be set through the following code:

[0112]

[0113]

[0114] Among them, RecordPool represents the memory pool for storing the bidirectional linked list. The structure of the bidirectional linked list includes: the head of the free linked list free_list_head, the tail of the free linked list free_list_tail, the head of the occupied linked list used_list_head, and the tail of the occupied linked list used_list_tail. <payloadtype>records[MAX_RECORD_LIMIT_OF_TABLE] represents the memory allocation of the data table managed by the doubly linked list. In this way, the complete allocation of memory can be completed in the process of building the record tree. The memory size required for each row of records is determined. The number of sub-data tables is already in the table definition, and there is no need to allocate memory multiple times. In addition, the Payload part can store data in the row records of the data table that can be used as associated data. The length of the data in the Payload can be determined based on the Schema information in the data table.

[0115] Furthermore, combined with the above settings, the data in the bidirectional linked list can be stored according to the structure specified by the following code:

[0116]

[0117] Among them, for each row in the data table, Record includes a metadata part and a payload part. The metadata part stores the occupancy status of the row and maintains the data stored in the above-mentioned bidirectional linked list. For example, the occupied linked list stores: the index value prev_used_idx of the data in the previous row and the index value next_used_idx of the data in the next row. The free linked list stores: the index value prev_free_idx of the data in the previous row and the index value next_free_idx of the data in the next row. In addition, the metadata part can also store hierarchical relationship data between data, including: the index value parent_record_idx of the parent data, the sibling relationship recorded in the sibling chain: the index value prev_brother_idx of the previous brother data and the index value next_brother_idx of the next brother data, and the index value first_child_idx of the first child record in each child data table.

[0118] On the basis of the above, different data tables have a cascading relationship, and the records between different data tables, that is, the stored data, also have a hierarchical relationship. Multiple data in a data table can be a data in the parent data table of the data table at the same time, that is, the child data of the parent data, that is, the parent data and the child data can be a one-to-many relationship. The multiple child data of the parent data under a certain child data table are called brother data, that is, brother data of the same data type. For example, Figure 5 In a data processing method provided by another embodiment of the present application, an example diagram of a data table stored in a record tree is shown. For ease of description, the index value prev_brother_idx of the previous brother data is not shown in the figure, data table A is the parent data table stored in the root node, and the parent data PayloadA0 does not have parent data and brother data. Therefore, the 0th row of data table A stores: the index value parent_record_-1 of the parent node, the index value next_brother_-1 of the next brother data, the index value first_child_[B]_0 of the first child record in the child data table of data table A, that is, data table B, and the index value first_child_[C]_0 of the first child record in the child data table of data table A, that is, data table C. Similarly, data is stored in each row in data table B and data table C in the above manner. Figure 5 In the formula, n, g, and k are the number of rows in data table A, data table B, and data table C respectively.

[0119] In addition, in one case, in order to further improve the richness and personalization of data, a record tree corresponding to the business can be created for different businesses. For example, a record tree corresponding to the dress set recommendation business, a record tree corresponding to the product search business, a record tree corresponding to the video recommendation business, and so on. In this way, when the data to be recommended is obtained, the business identifier to which the data to be recommended belongs can be obtained, and the record tree corresponding to the business identifier can be obtained. On this basis, the associated data of the data to be recommended can be determined based on the record tree. And, based on the record tree, determining the associated data of the data to be recommended can specifically include the steps of the data processing method provided in the embodiment of the present application; or, using specified filtering conditions to determine the associated data of the data to be recommended from the record tree; or, determining the data stored in the record tree as the associated data of the data to be recommended, which are all reasonable.

[0120] Corresponding to the above method embodiment, the present application also provides a data processing device embodiment, Figure 6 FIG. 1 is a schematic diagram showing the structure of a data processing device provided by an embodiment of the present application. Figure 6 As shown, the device comprises:

[0121] The acquisition module 602 is configured to acquire the data to be processed and the data type of the data to be processed;

[0122] The node determination module 604 is configured to determine the target node and associated nodes corresponding to the data type from a pre-established record tree, wherein the record tree is established according to the parent-child relationship between different data types, and the associated nodes include the parent node and sibling nodes of the target node;

[0123] The associated data determination module 606 is configured to obtain associated data of the data to be processed based on the target node and the associated node.

[0124] In one embodiment of the present application, a record tree is established based on the parent-child relationship between data types of different data, and the associated nodes include the parent node and sibling nodes of the target node. Therefore, based on the target node and the associated node, the obtained data is in an associated relationship with the data to be processed and is different from the data type of the data to be processed, and therefore can be used as associated data of the data to be processed. In this way, it is equivalent to using the parent-child relationship between different data types to process an object: a record tree once to obtain associated data, without the need to process multiple objects: different indexes corresponding to different data types multiple times. Therefore, this solution can reduce the number of times data is processed in determining associated data and improve data processing efficiency.

[0125] In an optional implementation, the device further includes a storage module configured to:

[0126] For each data type, multiple data having the data type are stored as a data table;

[0127] According to the parent-child relationship between different data types, the plurality of data tables are respectively stored in corresponding nodes of the record tree.

[0128] In an optional implementation, the data table further stores index values ​​of child data; the associated data includes target parent data and target child data;

[0129] The associated data determination module 606 is further configured to:

[0130] Obtaining a target index value of the data to be processed, and obtaining, according to the target index value, from a data table stored in a parent node of the target node, index values ​​of target parent data of the data to be processed and child data of the target parent data;

[0131] The target child data is obtained from the data table stored in the target node and the sibling nodes of the target node according to the index value of the child data of the target parent data.

[0132] In an optional implementation manner, the target parent data has multiple child data;

[0133] The associated data determination module 606 is further configured to:

[0134] Determine, according to the target index value, the index value of the sibling data having the same data type as the data to be processed;

[0135] The index value stored in the data table of the parent node of the target node is matched with the specified index value, and the successfully matched data is determined to be the target parent data of the data to be processed, and the specified index value is at least one of the target index value and the index value of the sibling data.

[0136] In an optional implementation, the data of any data type includes fixed-length data and variable-length data;

[0137] The device also includes a storage module configured to:

[0138] For each data type, store the fixed-length data of the data type and the index value of the variable-length data of the data type into the data table of the data type, and store the variable-length data of the data type into the first storage pool;

[0139] Based on the parent-child relationship between each data type, the data table of each data type is constructed as a record tree; a fixed second storage pool is divided from the storage space, and the record tree is stored in the second storage pool.

[0140] In an optional implementation, the device further includes an updating module configured to:

[0141] Receive update information for the current record tree;

[0142] According to the update information, the current record tree is updated using the tree structure of the current record tree.

[0143] In an optional implementation, the update module is further configured to:

[0144] Copying the current record tree, updating the copied record tree according to the update information to obtain an updated record tree, and replacing the current record tree with the updated record tree;

[0145] When a read / write request for the current record tree is received, it is determined whether the update of the copied record tree has been completed, and if not, the read / write request is executed on the current record tree.

[0146] In an optional implementation, the update module is further configured to:

[0147] Setting a read-write lock on the root node of the current record tree;

[0148] The current record tree is updated according to the update information, and the read-write lock is released when the update of the current record tree is completed.

[0149] In an optional implementation, the update information includes: the data to be added and the data type of the data to be added;

[0150] The device also includes a storage module configured to:

[0151] According to the occupation status of the data table in the current record tree, a free linked list for recording the idle status of the node and an occupied linked list for recording the occupation status of the node are created;

[0152] The update module is further configured to:

[0153] Determine the storage location of the data to be added according to the data type of the data to be added and the free linked list, and store the data to be added to the storage location of the data to be added in the data table stored in the current record tree;

[0154] The idle state record corresponding to the storage location of the to-be-added data is deleted from the idle linked list, and the occupied state record corresponding to the storage location of the to-be-added data is added to the occupied linked list.

[0155] In an optional implementation, the update information includes: data to be deleted and a data type of the data to be deleted;

[0156] The update module is further configured to:

[0157] Determine the storage location of the data to be deleted according to the data type of the data to be deleted and the occupied linked list, and delete the data to be deleted from the storage location of the data table in the pre-established record tree;

[0158] The occupied state record corresponding to the storage location of the data to be deleted is deleted from the occupied linked list, and the idle state record corresponding to the storage location of the data to be deleted is added to the idle linked list.

[0159] The above is a schematic scheme of a data processing device of this embodiment. It should be noted that the technical scheme of the data processing device and the technical scheme of the above data processing method belong to the same concept, and the details of the technical scheme of the data processing device that are not described in detail can be referred to the description of the technical scheme of the above data processing method.

[0160] Figure 7 The structure block diagram of a computing device 700 provided according to an embodiment of the present application is shown. The components of the computing device 700 include but are not limited to a memory 710 and a processor 720. The processor 720 is connected to the memory 710 via a bus 730, and the database 750 is used to store data.

[0161] The computing device 700 also includes an access device 740 that enables the computing device 700 to communicate via one or more networks 760. Examples of these networks include a public switched telephone network (PSTN), a local area network (LAN), a wide area network (WAN), a personal area network (PAN), or a combination of communication networks such as the Internet. The access device 740 may include one or more of any type of network interface (e.g., a network interface card (NIC)) of wired or wireless, such as an IEEE 802.11 wireless local area network (WLAN) wireless interface, a world-wide interoperability for microwave access (Wi-MAX) interface, an Ethernet interface, a universal serial bus (USB) interface, a cellular network interface, a Bluetooth interface, a near field communication (NFC) interface, and the like.

[0162] In one embodiment of the present application, the above components of the computing device 700 and Figure 7 Other components not shown in the figure may also be connected to each other, for example, via a bus. It should be understood that Figure 7 The computing device structure block diagram shown is only for the purpose of illustration, and is not intended to limit the scope of the present application. Those skilled in the art may add or replace other components as needed.

[0163] The computing device 700 may be any type of stationary or mobile computing device, including a mobile computer or mobile computing device (e.g., a tablet computer, a personal digital assistant, a laptop computer, a notebook computer, a netbook, etc.), a mobile phone (e.g., a smart phone), a wearable computing device (e.g., a smart watch, smart glasses, etc.), or other types of mobile devices, or a stationary computing device such as a desktop computer or PC. The computing device 700 may also be a mobile or stationary server.

[0164] Among them, when the processor 720 executes the instructions, the steps of the data processing method are implemented.

[0165] The above is a schematic scheme of a computing device of this embodiment. It should be noted that the technical scheme of the computing device and the technical scheme of the above data processing method belong to the same concept, and the details not described in detail in the technical scheme of the computing device can be referred to the description of the technical scheme of the above data processing method.

[0166] An embodiment of the present application further provides a computer-readable storage medium storing computer instructions, which, when executed by a processor, implement the steps of the data processing method as described above.

[0167] The above is a schematic scheme of a computer-readable storage medium of this embodiment. It should be noted that the technical scheme of the storage medium and the technical scheme of the above data processing method belong to the same concept, and the details not described in detail in the technical scheme of the storage medium can be referred to the description of the technical scheme of the above data processing method.

[0168] The above describes specific embodiments of the present application. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the accompanying drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0169] The computer instructions include computer program codes, which may be in source code form, object code form, executable files or some intermediate forms, etc. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium, etc. It should be noted that the content contained in the computer-readable medium may be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media do not include electric carrier signals and telecommunication signals.

[0170] It should be noted that, for the above-mentioned method embodiments, for the sake of simplicity of description, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the present application is not limited by the described action sequence, because according to the present application, certain steps can be performed in other sequences or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present application.

[0171] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0172] The preferred embodiments of the present application disclosed above are only used to help explain the present application. The optional embodiments do not describe all the details in detail, nor do they limit the invention to the specific implementation methods described. Obviously, many modifications and changes can be made according to the content of the present application. The present application selects and specifically describes these embodiments in order to better explain the principles and practical applications of the present application, so that those skilled in the art can understand and use the present application well. The present application is only limited by the claims and their full scope and equivalents.< / payloadtype> < / payloadtype> < / payloadtype>

Claims

1. A data processing method, characterized in that: include: Acquire data to be processed and the data type of the data to be processed; Determine a target node and associated nodes corresponding to the data type from a pre-established record tree, wherein the record tree is established according to the parent-child relationship between different data types, and the associated nodes include the parent node and sibling nodes of the target node; Based on the target node and the associated node, obtaining associated data of the data to be processed; Wherein, before obtaining the data to be processed and the data type of the data to be processed, the method further includes: For each data type, multiple data having the data type are stored as a data table; According to the parent-child relationship between different data types, the plurality of data tables are stored in corresponding nodes of the record tree respectively; Wherein, the data table also stores the index value of the child data; the associated data includes the target parent data and the target child data; The obtaining, based on the target node and the associated node, associated data of the data to be processed includes: Obtaining a target index value of the data to be processed, and obtaining, according to the target index value, from a data table stored in a parent node of the target node, index values ​​of target parent data of the data to be processed and child data of the target parent data; The target child data is obtained from the data table stored in the target node and the sibling nodes of the target node according to the index value of the child data of the target parent data.

2. The method according to claim 1, characterized in that The target parent data has multiple child data; obtaining the target parent data of the data to be processed from the data table stored in the parent node of the target node according to the target index value includes: Determine, according to the target index value, the index value of the sibling data having the same data type as the data to be processed; The index value stored in the data table of the parent node of the target node is matched with the specified index value, and the successfully matched data is determined to be the target parent data of the data to be processed, and the specified index value is at least one of the target index value and the index value of the sibling data.

3. The method according to any one of claims 1 to 2, characterized in that: Data of any data type includes fixed-length data and variable-length data; before obtaining the data to be processed and the data type of the data to be processed, the method further includes: For each data type, store the fixed-length data of the data type and the index value of the variable-length data of the data type into the data table of the data type, and store the variable-length data of the data type into the first storage pool; Based on the parent-child relationship between each data type, the data table of each data type is constructed as a record tree; a fixed second storage pool is divided from the storage space, and the record tree is stored in the second storage pool.

4. The method according to any one of claims 1 to 2, characterized in that: The method further comprises: Receive update information for the current record tree; According to the update information, the current record tree is updated using the tree structure of the current record tree.

5. The method according to claim 4, characterized in that The updating of the current record tree according to the update information and using the tree structure of the current record tree comprises: copying the current record tree, and updating the copied record tree according to the update information to obtain an updated record. tree, replacing the current record tree with the updated record tree; When a read / write request for the current record tree is received, it is determined whether the update of the copied record tree has been completed, and if not, the read / write request is executed on the current record tree.

6. The method according to claim 4, characterized in that The updating of the current record tree according to the update information and using the tree structure of the current record tree includes: Setting a read-write lock on the root node of the current record tree; The current record tree is updated according to the update information, and the read-write lock is released when the update of the current record tree is completed.

7. The method according to claim 4, characterized in that The update information includes: the data to be added and the data type of the data to be added; before obtaining the data to be processed and the data type of the data to be processed, the method further includes: According to the occupation status of the data table in the current record tree, a free linked list for recording the idle status of the node and an occupied linked list for recording the occupation status of the node are created; The updating of the current record tree according to the update information and using the tree structure of the current record tree includes: Determine the storage location of the data to be added according to the data type of the data to be added and the free linked list, and store the data to be added to the storage location of the data to be added in the data table stored in the current record tree; The idle state record corresponding to the storage location of the to-be-added data is deleted from the idle linked list, and the occupied state record corresponding to the storage location of the to-be-added data is added to the occupied linked list.

8. The method according to claim 7, characterized in that The update information includes: data to be deleted and the data type of the data to be deleted; The updating of the current record tree according to the update information and using the tree structure of the current record tree includes: Determine the storage location of the data to be deleted according to the data type of the data to be deleted and the occupied linked list, and delete the data to be deleted from the storage location of the data table in the pre-established record tree; The occupied state record corresponding to the storage location of the data to be deleted is deleted from the occupied linked list, and the idle state record corresponding to the storage location of the data to be deleted is added to the idle linked list.

9. A data processing device, characterized in that: include: An acquisition module is configured to acquire data to be processed and a data type of the data to be processed; A node determination module is configured to determine a target node and associated nodes corresponding to the data type from a pre-established record tree, wherein the record tree is established according to a parent-child relationship between different data types, and the associated nodes include a parent node and a sibling node of the target node; An associated data determination module, configured to obtain associated data of the data to be processed based on the target node and the associated node; Wherein, before obtaining the data to be processed and the data type of the data to be processed, the method further includes: For each data type, multiple data having the data type are stored as a data table; According to the parent-child relationship between different data types, the plurality of data tables are stored in corresponding nodes of the record tree respectively; Wherein, the data table also stores the index value of the child data; the associated data includes the target parent data and the target child data; The obtaining, based on the target node and the associated node, associated data of the data to be processed includes: Obtaining a target index value of the data to be processed, and obtaining, according to the target index value, from a data table stored in a parent node of the target node, index values ​​of target parent data of the data to be processed and child data of the target parent data; The target child data is obtained from the data table stored in the target node and the sibling nodes of the target node according to the index value of the child data of the target parent data.

10. A computing device comprising a memory, a processor, and computer instructions stored in the memory and executable on the processor, characterized in that: When the processor executes the instructions, the steps of the method according to any one of claims 1 to 8 are implemented.

11. A computer-readable storage medium storing computer instructions, characterized in that: When the instruction is executed by a processor, the steps of the method described in any one of claims 1 to 8 are implemented.

12. A computer program product comprising computer instructions, characterized in that: When the computer instructions are executed by a processor, the steps of the method described in any one of claims 1 to 8 are implemented.

Images