A data processing method, apparatus, device, medium, and program product

Abstract

This application provides a data processing method, apparatus, device, medium, and program product for achieving efficient data management and ensuring data consistency and integrity. This application can be applied to fields such as real-time computer data processing, big data analysis, and cloud computing. It includes: obtaining an original data structure and a node to be updated, the node to be updated being contained within the original data structure; creating an updated data structure based on the node to be updated, the updated data structure being different from the original data structure; writing the updated data of the node to be updated into the updated data structure; and storing the updated data structure.

Description

Technical Field

[0001] This application relates to the field of computing, and more particularly to a data processing method, apparatus, device, medium, and program product. Background Technology

[0002] With the development of computer technology, various applications are widely used, and as these applications run and are upgraded over a long period of time, their data also needs to be updated and managed in real time.

[0003] Therefore, how to achieve efficient data management is also an urgent problem to be solved. Summary of the Invention

[0004] This application provides a data processing method, apparatus, device, medium, and program product for achieving efficient data management and ensuring data consistency and integrity.

[0005] In view of this, this application provides a data processing method, comprising: obtaining an original data structure and a node to be updated, the node to be updated being contained in the original data structure; creating an updated data structure based on the node to be updated, the updated data structure being a different data structure from the original data structure; writing the updated data of the node to be updated into the updated data structure; and storing the updated data structure.

[0006] Another aspect of this application provides a data processing apparatus, including: an acquisition module, configured to acquire an original data structure and a node to be updated, wherein the node to be updated is contained in the original data structure;

[0007] The processing module is used to create an update data structure based on the node to be updated, the update data structure being a different data structure from the original data structure; and to write the update data of the node to be updated into the update data structure.

[0008] The storage module is used to store the updated data structure.

[0009] In one possible design, in another implementation of another aspect of the embodiments of this application, the processing module is used to call a first function to create the updated data structure based on the node to be updated, wherein the first function sets the path of the node to be updated in the original data structure and the updated data of the node to be updated.

[0010] In one possible design, in another implementation of another aspect of the embodiments of this application, the processing module is used to locate the node to be updated in the original data structure based on the path;

[0011] When the path indication is 0, identify the node to be updated and create an update node for that node.

[0012] The updated data structure is generated based on the update node and the replication node, which is generated by replicating the unupdated nodes in the original data structure.

[0013] In one possible design, in another implementation of another aspect of the embodiments of this application, the processing module is used to find the update node in the update data structure based on the path;

[0014] When the path indication is 0, the update node is determined and the update data is written to the update node.

[0015] In one possible design, in another implementation of another aspect of the embodiments of this application, the processing module is used to set reference counts and reference relationships for each node in the original data structure. The reference count is used to record the number of times the current node is shared, and the reference relationship is used to indicate the data sharing relationship between each node.

[0016] In one possible design, in another implementation of another aspect of the embodiments of this application, the processing module is used to locate the node to be updated in the original data structure based on the path;

[0017] When the path indication is 0, identify the node to be updated and obtain the reference count of the node to be updated;

[0018] When the reference count is greater than 1, create an update node for the node to be updated and modify the reference count of the node to be updated.

[0019] The updated data structure is generated based on the updated node, and a reference relationship is set for the updated data structure. This reference relationship is used to indicate that the updated data structure shares the unupdated nodes in the original data structure.

[0020] In one possible design, in another implementation of another aspect of the embodiments of this application, the storage module is used to store the updated data structure and the original data structure in a historical version array, and the updated data structure and the original data structure have corresponding indices.

[0021] In one possible design, in another implementation of another aspect of the embodiments of this application, the acquisition module is used to acquire the first index of the updated data structure;

[0022] The processing module is used to access the updated data structure from the historical version array based on the first index.

[0023] In one possible design, in another implementation of another aspect of the embodiments of this application, the processing module is used to create a proxy object;

[0024] Based on the proxy object, obtain the original data structure and the update operation of the updated data structure.

[0025] In one possible design, in another implementation of another aspect of the embodiments of this application, the processing module is used to call a second function to create a proxy object, the second function accepting the original data structure and the updated data structure;

[0026] The proxy object is used to call a third function to obtain the original data structure and the update operation of the updated data structure. The third function is used to set the capture object of the proxy object.

[0027] In one possible design, in another implementation of another aspect of the embodiments of this application, the processing module is used to record the update operations of the original data structure and the updated data structure to generate a change log; and to store the change log.

[0028] In one possible design, in another implementation of another aspect of the embodiments of this application, the processing module is used to call a fourth function to record the update operations of the original data structure and the updated data structure to generate a change log.

[0029] In one possible design, in another implementation of another aspect of the embodiments of this application, the processing module is configured to set a dirty flag for the original data structure and initialize the value of the dirty flag; put the updated data structure into a delayed processing queue; update the dirty flag value of the original data structure to a target value, the dirty flag being used to indicate whether there is an unapplied changed data structure, the target value being used to indicate that there is an unapplied changed data structure in the delayed processing queue; after triggering the execution operation, execute the updated data structure in the delayed processing queue to write the updated data into the updated data structure.

[0030] In one possible design, in another implementation of another aspect of the embodiments of this application, the triggering operation is a periodic triggering operation or the triggering operation is when the number of unapplied modified data structures in the delayed processing queue reaches a preset threshold.

[0031] In one possible design, in another implementation of another aspect of the embodiments of this application, the processing module is used to perform sharding processing on the original data structure and the updated data structure according to a sharding strategy to obtain multiple shard sets. The sharding strategy includes sharding processing according to the depth of the data structure, or sharding processing according to the number of nodes in the data structure, or sharding processing according to the business logic of the data in the data structure. Each shard set in the multiple shard sets includes multiple nodes.

[0032] In one possible design, in another implementation of another aspect of the embodiments of this application, the processing module determines the fragmentation boundary according to the fragmentation strategy;

[0033] Based on the partition boundary, the original data structure and the updated data structure are partitioned to obtain multiple partitioned sets. The multiple partitioned sets also include partition identifiers, which are used to find and identify the partitioned sets.

[0034] Establish the association relationship between the multiple shard sets, which is used to indicate the data dependency relationship between the multiple shard sets and the dependency relationship between the node and the multiple shard sets;

[0035] Based on this relationship, the loading strategy for the multiple shard sets is determined. This loading strategy is used to implement on-demand loading of the multiple shard sets and to maintain the loading status of the multiple shard sets.

[0036] In one possible design, in another implementation of another aspect of the embodiments of this application, the acquisition module acquires the activity information and memory storage usage information of each shard set in the plurality of shard sets. The activity information is used to indicate the number of times the shard set is loaded, and the memory storage usage information is used to indicate the amount of memory occupied by the original data structure and the updated data structure.

[0037] The processing module is used to release the memory resources of the first target shard set when the activity information of the first target shard set is lower than a first threshold, wherein the data of the first target shard set is persisted to a target database, which is a local database or a remote database;

[0038] or,

[0039] When the amount of memory occupied, as indicated by the memory storage usage information, reaches a second threshold, the memory resources of the second target shard set, which are ranked Nth from low to high based on the activity information, are selected, and the data of the second target shard set is persisted to the target database.

[0040] In one possible design, in another implementation of another aspect of the embodiments of this application, the acquisition module is used to acquire the storage state information of each shard set in the plurality of shard sets;

[0041] The processing module is used to merge the overlapping parts of the at least two shard sets into one data set and update the association relationship of the at least two shard sets when the storage status information indicates that at least two shard sets overlap.

[0042] This application also provides a computer device, including: a memory, a processor, and a bus system;

[0043] The memory is used to store programs;

[0044] The processor is used to execute programs in memory, and the processor is used to execute the methods mentioned above according to the instructions in the program code;

[0045] Bus systems are used to connect memory and processor to enable communication between them.

[0046] Another aspect of this application provides a computer-readable storage medium storing instructions that, when executed on a computer, cause the computer to perform the methods described above.

[0047] Another aspect of this application provides a computer program product or computer program including computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the methods provided in the above aspects.

[0048] As can be seen from the above technical solutions, the embodiments of this application have the following advantages: when modifying the node to be updated, the original data structure remains unchanged; only a new data structure is created for the node to be updated. This avoids modifying existing data, thus ensuring data consistency and integrity. Furthermore, each data update generates a new data structure, while the old data structure remains unchanged, making debugging and backtracking easier. Attached Figure Description

[0049] Figure 1 This is an exemplary structural diagram of a tree structure in an embodiment of this application;

[0050] Figure 2 This is a schematic diagram illustrating an application scenario of the data processing method in the embodiments of this application;

[0051] Figure 3 This is a schematic diagram of one embodiment of the data processing method in this application;

[0052] Figure 4 This is a schematic diagram of the workflow of the proxy object in the embodiments of this application;

[0053] Figure 5 This is a flowchart illustrating a data processing method in an embodiment of this application;

[0054] Figure 6 This is a schematic diagram of one embodiment of the data processing device in this application.

[0055] Figure 7This is a schematic diagram of one embodiment of the server in this application;

[0056] Figure 8 This is a schematic diagram of one embodiment of the terminal in this application. Detailed Implementation

[0057] This application provides a data processing method, apparatus, device, medium, and program product for achieving efficient data management and ensuring data consistency and integrity.

[0058] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a particular order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented, for example, in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “corresponding to,” and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0059] In this application embodiment, the terms "module" or "unit" refer to a computer program or part of a computer program that has a predetermined function and works with other related parts to achieve a predetermined goal, and can be implemented wholly or partially using software, hardware (such as processing circuitry or memory), or a combination thereof. Similarly, a processor (or multiple processors or memory) can be used to implement one or more modules or units. Furthermore, each module or unit can be part of an overall module or unit that includes the functionality of that module or unit.

[0060] With the development of computer technology, various applications are widely used. As these applications run and are upgraded, their data requires real-time updates and management. For example, in big data analytics scenarios, the explosive growth of data volume necessitates efficient methods for collecting and integrating data from diverse sources. This includes structured data (such as tabular data in databases) and unstructured data (such as social media content and sensor data), while simultaneously managing the data. For instance, after data collection, it needs to be cleaned and preprocessed, requiring modifications to existing data. Therefore, achieving efficient data management and ensuring data consistency and integrity is a pressing issue that needs to be addressed.

[0061] To address this technical problem, this application provides the following technical solution: Obtaining the original data structure and the node to be updated, the node to be updated being contained within the original data structure; creating an updated data structure based on the node to be updated, the updated data structure being different from the original data structure; writing the updated data of the node to be updated into the updated data structure; and storing the updated data structure. When modifying the node to be updated, the original data structure remains unchanged; only a new data structure is created for the node to be updated. This avoids modifying existing data, thus ensuring data consistency and integrity. Furthermore, each data update generates a new data structure, while the old data structure remains unchanged, making debugging and backtracking easier.

[0062] It should be understood that this application is primarily applied to scenarios where data is stored in a tree structure. That is, both the original and updated data structures involved in this application can be understood as tree structures or tree instances. A tree structure is a very important data structure in computer science; it simulates hierarchical systems and is widely used in various fields and applications. A tree is a non-linear data structure composed of nodes. Each node can have zero or more child nodes, but can only have one parent node (except for the root node, which has no parent node). The following example illustrates this. Figure 1 Taking the tree structure shown (which includes nodes A to F) as an example, let's explain the various terms in the tree structure:

[0063] Node: Every element in a tree structure is called a node. In this application, each node contains a value and a reference to its child nodes. The value of a node is used to store specific data content. In practical applications, it can be any data that needs to be stored. For example, Figure 1 Nodes A through F shown in the diagram all store specific data content.

[0064] Root node: The top-level node of the tree, it has no parent node. For example... Figure 1 The node A shown can be called the root node.

[0065] Child node: A node's direct descendant node. For example... Figure 1 Node B can be called a child node of node A, and node D can be called a child node of node B.

[0066] Parent node: The direct parent node of a given node. For example... Figure 1 Node A can be called the parent node of node B, and node B can be called the parent node of node D.

[0067] Sibling nodes: Nodes that share the same parent node. For example... Figure 1 The nodes D and E shown have the same parent node, namely node B. In this case, nodes D and E can be called sibling nodes.

[0068] Leaf node: A node that has no children. For example... Figure 1 As shown, nodes D and E have no child nodes, which means they can be called leaf nodes.

[0069] Path: A sequence of nodes from the root node to any other node. For example... Figure 1 As shown, the path from node A to node D can be represented as ABD.

[0070] Depth: The length of the path from the root node to a given node. For example... Figure 1 As shown, the path from node A to node D can be represented as ABD, with a depth of 2.

[0071] Height: The length of the path from a given node to its farthest leaf node.

[0072] Ancestor node: All nodes on the path from the root to this node.

[0073] Descendant nodes: All nodes on the path from this node to the leaf node.

[0074] The technical solutions of this application and their effects are described below through several exemplary embodiments. It should be noted that the following embodiments can be referenced, borrowed from, or combined with each other. Identical terms, similar features, and similar implementation steps in different embodiments will not be repeated.

[0075] Based on the aforementioned technical principles or related theoretical foundations, the data processing method proposed in this application can be applied to any computer device with data processing and computing capabilities, and this computer device can be various types of terminals or servers. When the computer device in the embodiment is a server, the server can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud service content delivery networks (CDNs), and big data and artificial intelligence platforms.

[0076] It should be further noted that the terminals involved in the embodiments of this application include, but are not limited to, smartphones, computers, intelligent voice interaction devices, smart home appliances, vehicle terminals, and aircraft. The embodiments of this application can be applied to various scenarios, including but not limited to cloud technology, artificial intelligence, smart transportation, and assisted driving.

[0077] In some possible implementations, embodiments of the present invention provide a computer program for a data processing method that can be deployed and executed on a single computer device, or on multiple computer devices located in one location; or, on multiple computer devices distributed across multiple locations and interconnected via a communication network, wherein the multiple computer devices distributed across multiple locations and interconnected via a communication network can form a blockchain system.

[0078] Since multiple computer devices can form a blockchain system, the data processing method provided in this application can be executed and completed by a node in the blockchain; and the node used to execute the data processing method can be any mobile terminal that can provide a front-end page, such as a smartphone, tablet computer, or personal computer (PC).

[0079] The embodiments of this application can be applied to scenarios such as big data analysis, cloud computing, and real-time computer data processing. The following examples illustrate this. Figure 2 An exemplary architecture is illustrated below, comprising a terminal 100, a server 200, a database 300, and a network 400. The terminal 100, the server 200, and the database 300 are connected via the network 400. The database 300 is used to store original data structures and updated data structures (which can also be understood as different data versions). Figure 2 The number of terminals 100, servers 200 and databases 300 in the system shown is only an example. For example, there may be multiple terminals 100, servers 200 and databases 300. This application does not limit the number of terminals 100, servers 200 and databases 300.

[0080] In this system, terminal 100 communicates with server 200 via a network. Database 300 can be integrated onto server 200 or hosted on a cloud or other server. During data processing, interaction can occur between terminal 100 and server 200. For example, a user uploads updated data through terminal 100, which then sends the updated data to server 200. Server 200 determines the node to be updated in the original data structure based on the updated data, creates the updated data structure based on the node to be updated, and finally writes the updated data into the updated node of the updated data structure.

[0081] Terminal 100 can be a smartphone, tablet, laptop, desktop computer, smart speaker, smartwatch, smart voice interaction device, smart home appliance, or in-vehicle terminal, but is not limited to these. Server 200 can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery network (CDN), big data, and artificial intelligence platforms.

[0082] In short, a database can be viewed as an electronic filing cabinet—a place to store electronic files, where users can perform operations such as adding, querying, updating, and deleting data. A "database" is a collection of data stored together in a certain way, shared by multiple users, with minimal redundancy, and independent of application programs. A Database Management System (DBMS) is a computer software system designed to manage databases, generally possessing basic functions such as storage, retrieval, security, and backup. DBMSs can be classified according to the database model they support, such as relational or Extensible Markup Language (XML); or according to the type of computer they support, such as server clusters or mobile phones; or according to the query language used, such as Structured Query Language (SQL) or XQuery; or according to performance priorities, such as maximum scale or maximum operating speed; or other classification methods. Regardless of the classification method used, some DBMSs can cross categories, for example, supporting multiple query languages ​​simultaneously.

[0083] It is understood that in the specific implementation of this application, data such as input data and output data are involved. When the above embodiments of this application are applied to specific products or technologies, user permission or consent is required, and the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0084] Based on the above introduction, the data processing method in this application will be described below with the server as the execution entity. Please refer to [link / reference]. Figure 3 One embodiment of the data processing method in this application includes:

[0085] 301. Obtain the original data structure and the node to be updated, which is contained in the original data structure.

[0086] In this embodiment, after receiving the raw data, the server stores the raw data in a tree structure to obtain the raw data structure. During data management, the server can receive data operation requests, which carry information about the node to be updated and the update data for that node. The raw data structure can be understood as an immutable data model; that is, in this embodiment, the raw data structure will not change.

[0087] It should be understood that the creation process of this original data structure can be as follows:

[0088] class TreeNode{

[0089] constructor(value,left=null,right=null){

[0090] this.value = value; / / The value of the node

[0091] this.left = left; / / Left child node

[0092] this.right = right; / / right child node

[0093] }

[0094] }”

[0095] The TreeNode class is used to represent nodes in a tree. Each node contains a value and two child nodes (left and right). Assume the original data structure created is as follows: Figure 1 The tree structure shown represents the original data structure. Figure 1 As shown, when the server receives an operation request to update the data content of node B, the server determines node B as the node to be updated.

[0096] 302. Create an updated data structure based on the node to be updated. This updated data structure is different from the original data structure.

[0097] In this embodiment, after obtaining the original data structure and the node to be updated, the server can call a first function to create an updated data structure based on the node to be updated. The first function sets the path of the node to be updated within the original data structure and the updated data for the node to be updated (wherein, the path indicates the sequence of nodes from the root node to the node to be updated in the original data structure, and the updated data is carried as the value set by the first function). It should be understood that, to maintain data consistency and integrity, the updated data structure is a different data structure from the original data structure.

[0098] In one exemplary scheme, the first function could be `setValue(path, value)`, used to set the value of a node in the tree. It accepts a path (representing a directional sequence from the root node to the target node) and a new value. This method calls the internal `_setValue` method to create a new tree containing the changes. `_setValue(node, path, value)` is used to recursively set the values ​​of nodes. It accepts the current node, the path, and the new value. If the path length is 0, it indicates that the target node has been reached; a new node is created and returned. Otherwise, the values ​​of child nodes are recursively set according to the path direction ('left' or 'right'). The following example... Figure 1 The original data structure shown illustrates the settings for setValue(path, value):

[0099] For example, to modify node E, the setValue(path, value) method can be set as follows:

[0100] `setValue(['left','right'],newValue)` / / From A through B to E

[0101] The newValue is used to indicate the data to be updated.

[0102] For example, to modify node F, the setValue(path, value) method can be set as follows:

[0103] `setValue(['right','left'],newValue)` / / From A through C to F

[0104] The newValue is used to indicate the data to be updated.

[0105] For example, to modify node B, the setValue(path, value) method can be set as follows:

[0106] `setValue(['left'],newValue)` / / Directly navigate from A to B

[0107] An example code file could be as follows:

[0108]

[0109] During data updates, to ensure both traceability of data operations and maintain data consistency, a proxy server can be set up as an intermediate layer to uniformly manage all data change requests and ensure that change operations are correctly recorded and processed. Specifically, during application operation, this server can create proxy objects; based on these proxy objects, it retrieves the update operations of the original and updated data structures. In terms of code, the server can call a second function to create the proxy object; then, the proxy object calls a third function to retrieve the update operations of the original and updated data structures. Based on this proxy object's method of retrieving update operations, the server can also record these update operations to generate a change log. This allows for more operations to be performed based on the change log. For example, the change log can provide users with transparency regarding data modifications, allowing them to understand which data has been updated. This provides a way to trace the history of data modifications, helping to understand how specific functions or problems have evolved over time. Furthermore, when problems arise during data usage, the change log helps determine whether the problem was caused by a data update and whether a rollback to a previous version is necessary. Moreover, if there are projects that depend on this data, the change log helps understand whether updates to dependencies will affect those projects.

[0110] In one exemplary scenario, the server can use the `createProxyTree(tree)` function to create a proxy object for the original data structure. This proxy object then calls `set(target, property, value)` to retrieve modifications to the original data structure and to update the data structure, and records the change log. An exemplary code file for the `createProxyTree(tree)` function can be represented as follows:

[0111]

[0112] It should be understood that this Proxy object is used to create a proxy for an object, thereby intercepting and customizing operations when accessing the object's properties or methods. `set(target, property, value)` can be understood as a setter catcher of the proxy object, used to intercept setting operations on tree properties. If the root property is set, a change log is recorded. Then, the property value is set and `true` is returned to indicate successful operation. An example code file could be as follows:

[0113]

[0114] tree=tree.setValue(['right'],3);

[0115] / / Output change log

[0116] console.log(tree.getChangeLog());

[0117] / / Output: [{path:['left',value:2},{path:['right'],value:3}]"

[0118] When setting up this proxy object, for a given original data structure, a global proxy object can be set. This proxy object can continuously monitor the entire operation process of the data structure; that is, operations based on the original data structure and operations based on updating the data structure can both be monitored through this single proxy object. An example code file could be as follows:

[0119] TypeScript

[0120] / / Initial creation of tree and agent

[0121] let tree=createProxyTree(new ImmutableTree(new TreeNode(1)));

[0122] / / When performing modification operations

[0123] tree.setValue(['left'], 2); / / This will create a new tree instance.

[0124] / / But there's no need to create a new proxy object, because setValue returns a new tree instance.

[0125] / / This will be handled through the existing proxy object.

[0126] tree.setValue(['right'], 3); / / Also using the existing proxy object.

[0127] When this proxy object is created, its workflow can be as follows: Figure 4 As shown:

[0128] After creating the initial tree structure, a proxy object for that initial tree structure is created. Then, the proxy object is used to monitor all modification operations on all tree structures based on the initial tree structure. If a modification operation exists, it is obtained and recorded to generate a change log. A new tree structure is created. If no modification operation exists, the monitoring of all tree structure modification operations continues.

[0129] To reduce data redundancy and memory consumption, when creating the updated data structure, you can try to create only copies of the nodes that need to be modified, and share the unupdated nodes.

[0130] To implement the above method, reference counting and reference relationship settings need to be added to each node in the original data structure. The reference count indicates the number of times each node is shared across different data versions. For example, for... Figure 1 The tree structure shown is assumed to have been updated 5 times, storing a total of 6 data versions. If the data of node B is not updated during these 5 updates, then the reference count of node B can be determined to be 6, meaning that the initial version, the first data version, the second data version, the third data version, the fourth data version, and the fifth data version all share node B.

[0131] When a node's reference count is greater than 1, it means that the node is shared by multiple versions; it can also be called a shared node. When modifying a shared node, a copy of this shared node needs to be created; this is also called a replicated node.

[0132] Based on the above description, the process for sharing unupdated nodes can be as follows: The server sets reference counts and reference relationships for each node in the original data structure. The reference count records the number of times a node is shared, and the reference relationship indicates the data sharing relationship between nodes. During the creation of the updated data structure, the server searches for the node to be updated in the original data structure based on the path. When the path indication is 0, the node to be updated is determined and its reference count is obtained. When the reference count is greater than 1, an updated node is created for the node to be updated, and its reference count is modified. The updated data structure is generated based on the updated node, and the reference relationship of the updated data structure is set. This reference relationship indicates that the updated data structure shares the unupdated nodes in the original data structure.

[0133] An example code file could look like this:

[0134]

[0135] The `constructor(value, left=null, right=null)` function is used to initialize the node's value, left child node, right child node, and reference count. `addRef()` increments the reference count. `release()` decrements the reference count; if the reference count is 0, it recursively releases the child nodes.

[0136] The above process can be incorporated into the creation and updating of data structures, and an example code file can be provided as follows:

[0137]

[0138]

[0139] In this code file, `constructor(root=null)` is used to construct the tree and initialize the root node. `setValue(path, value)` is used to set the value of a node in the tree. It accepts a path (representing the directional sequence from the root node to the target node) and a new value. This method calls the internal `_setValue` method to create a new tree containing the changes. `_setValue(node, path, value)` is used to recursively set the value of a node. It accepts the current node, the path, and the new value. If the path length is 0, it means the target node has been reached: if the node's reference count is greater than 1, a new node is created and returned. Otherwise, the node's value is modified and returned. Based on the path's direction ('left' or 'right'), the values ​​of child nodes are recursively set: if the node's reference count is greater than 1, a new node is created and returned. Otherwise, the child nodes are modified and returned.

[0140] In the scheme implemented in the above code file, any modification operation will not change the original data structure, but will return a new data structure. This ensures that the historical state of the data is not destroyed, facilitating the implementation of undo and redo functions, thus guaranteeing the immutability of the original data structure. At the same time, to avoid copying the entire data structure with each modification, structure sharing allows the old and new data structures to share the unmodified parts, thereby saving memory and improving efficiency. Furthermore, reference counting can be used to manage the lifecycle of nodes, ensuring that memory is released when there are no other references, avoiding memory leaks.

[0141] The above description can be understood as creating an updated data structure for each modification. In practical applications, data modification operations can be very frequent. Therefore, to reduce unnecessary computational consumption and memory usage, a delayed processing mechanism can be implemented. The implementation process can be as follows: The server sets a dirty flag for the original data structure and initializes the value of the dirty flag; the updated data structure is placed in the delayed processing queue; the dirty flag of the original data structure is updated to a target value. The dirty flag indicates whether there are any unapplied modified data structures, and the target value indicates that there are unapplied modified data structures in the delayed processing queue; after triggering the execution operation, the updated data structure in the delayed processing queue is executed to write the updated data into the updated data structure.

[0142] An exemplary code file for setting the dirty flag and initializing the delayed processing queue for this original data structure can be represented as follows:

[0143] “constructor(root=null){

[0144] this.root = root; / / Root node of the tree

[0145] this.dirty = false; / / Dirty flag, initially set to false.

[0146] this.operations = []; / / Operation queue, initially empty.

[0147] }”

[0148] This code file instructs that the `dirty` flag be set to `false` when initializing the `ImmutableTree` object, indicating that there are no unapplied changes to the current tree. Initializing the `operations` queue to empty indicates that there are no pending modification operations.

[0149] After initializing the dirty marking and delayed processing queue, when calling the setValue method during the creation of the update data structure, the code file can be modified as follows:

[0150] setValue(path, value){

[0151] this.operations.push({path,value}); / / Add the operation to the operation queue

[0152] this.dirty = true; / / Set the dirty flag to true.

[0153] `true` returns `this`; / / Returns the current tree object.

[0154] }”

[0155] This code file instructs that each call to the `setValue` method add the modifications to the created updated data structure to the operations queue (i.e., the deferred processing queue). Setting the `dirty` flag to `true` indicates that there are unapplied changes. It returns the current tree structure to support chained calls.

[0156] After this execution operation is triggered, such as by calling a function to execute the delayed processing queue, modification operations are performed on each updated data structure in the delayed processing queue. An example code file can be represented as follows:

[0157] “applyChanges(){

[0158] if (!this.dirty) return this; / / If there is no dirty flag, return the current tree directly.

[0159] let newRoot = this.root; / / Start from the root node

[0160] for(const{path,value}of this.operations){

[0161] newRoot = this._setValue(newRoot, path, value); / / Apply the operation one by one.

[0162] }

[0163] this.operations = []; / / Clear the operation queue

[0164] this.dirty = false; / / Reset the dirty flag.

[0165] return new ImmutableTree(newRoot); / / Returns a new tree object

[0166] }”

[0167] This code file instructs the execution process to first check the "dirty" flag. If it's false, it means there are no unapplied changes, and the current tree is returned directly. If the "dirty" flag is true, the operations in the operations queue are applied one by one, generating a new tree. The operations queue is then cleared, and the "dirty" flag is reset to false. A new `ImmutableTree` object is returned, containing the tree with all changes applied.

[0168] By using a dirty flag and an operations queue, the server records the operation when the `setValue` method is called without executing it immediately. This deferred processing allows all changes to be applied at once when needed, instead of updating the tree immediately with each `setValue` call. Simultaneously, all operations in the operations queue can be processed at once within the `applyChanges` method, merging multiple operations into a single process, reducing the overhead of multiple tree updates and improving efficiency.

[0169] Based on the above description, the method of delayed processing described above can be incorporated into the creation and updating of data structures. An example code file for this can be as follows:

[0170] / / Create a new immutable tree with a root node value of 1

[0171] let tree=new ImmutableTree(new TreeNode(1));

[0172] / / Set the value of the left child node to 2

[0173] tree=tree.setValue(['left'],2);

[0174] / / Set the value of the right child node to 3

[0175] tree=tree.setValue(['right'],3);

[0176] / / Practical application changes

[0177] tree = tree.applyChanges();

[0178] / / Output the structure of the current tree

[0179] console.log(JSON.stringify(tree.root,null,2));

[0180] / / Set the value of the left child node of the left child node to 4

[0181] tree=tree.setValue(['left','Left'],4);

[0182] / / Practical application changes

[0183] tree = tree.applyChanges();

[0184] / / Output the structure of the current tree

[0185] console.log(JSON.stringify(tree.root,null,2));”

[0186] 303. Write the update data of the node to be updated into the update data structure.

[0187] In this embodiment, after the updated data structure is created, the server can call the first function to traverse the updated data structure along the path specified by path, and then only update the data of the copies of the nodes to be updated (i.e., the updated nodes) on the path. All modifications are recorded in changeLog, which makes it easy to trace the change history.

[0188] 304. Store the updated data structure.

[0189] In this embodiment, after the data update of the updated data structure is completed, the updated data structure is stored. To ensure data consistency and integrity, the server can store both the original data structure and the updated data structure simultaneously.

[0190] In one exemplary scheme, the server can store the original data structure and the updated data structure in a historical version array, where each data structure has a corresponding index. An exemplary code file can be represented as follows:

[0191] / / Create a new immutable tree with a root node value of 1

[0192] let tree=new ImmutableTree(new TreeNode(1));

[0193] / / Use an agent to capture changes

[0194] tree=createProxyTree(tree);

[0195] / / An array that saves historical versions

[0196] Let history = [tree];

[0197] / / Set the value of the left child node to 2

[0198] tree=tree.setValue(['left'],2);

[0199] history.push(tree); / / Save the new version

[0200] / / Set the value of the right child node to 3

[0201] tree=tree.setValue(['right'],3);

[0202] history.push(tree); / / Save the new version

[0203] / / Output change log

[0204] console.log(tree.getChangeLog());

[0205] / / Output: [{path:['left',value:2},{path:['right'],value:3}]

[0206] / / Revert to the first version

[0207] let firstVersion=history[0];

[0208] console.log(firstVersion.root);

[0209] / / Output: TreeNode{value:1,left:null,right:null}

[0210] / / Revert to version 2

[0211] let secondVersion=history[1];

[0212] console.Log(secondVersion.root);

[0213] / / Output: TreeNode{value:1,left:TreeNode{value:2,left:null,right:null},right:null}"

[0214] This code file instructs that a new immutable tree be created first, with the root node value set to 1. Then, a proxy object is created using the `createProxyTree` function to capture changes. During the saving of the tree instance, an array `history` is used to store the tree instance for each version. That is, after each call to the `setValue` method, the new tree instance is saved to the `history` array. In an example scenario, the `setValue` method is called to set the left child node's value to 2, and the new tree instance is saved to the `history` array. The `setValue` method is then called to set the right child node's value to 3, and the new tree instance is saved to the `history` array. Simultaneously, the `getChangeLog` method is called to output the change log, displaying all change operations. When reverting to a specific version, accessing a specific index in the `history` array allows reverting to that version of the tree instance. For example, accessing `history[0]` reverts to the first version, and accessing `history[1]` reverts to the second version.

[0215] It should be understood that the indices of the history version array are automatically set, generally based on the array order. For example, `history[0]` is the initial version. `history[1]` is the version after the first modification. `history[2]` is the version after the second modification. Each time a new data structure is pushed into the history version array, the index is automatically incremented. By accessing these indices, one can trace back to a specific version.

[0216] Based on the above description, after the original data structure has been updated for a long time, its data storage capacity will increase significantly. At this point, the difference between the original and updated data structures will lead to a significant decrease in memory usage and operational performance. Therefore, the following solution can be provided:

[0217] The server performs sharding on the original data structure and the updated data structure according to a sharding strategy to obtain multiple shard sets. The sharding strategy includes sharding according to the depth of the data structure, sharding according to the number of nodes in the data structure, or sharding according to the business logic of the data in the data structure. Each of the multiple shard sets includes multiple nodes.

[0218] In one exemplary scheme, assuming both the original data structure and the updated data structure are tree structures, and the depth of the tree structure (i.e., the number of node levels) is 12, the tree structure can be divided into shard sets of 3 levels each. It should be understood that each node in the updated data structure can be partitioned according to its depth in the original data structure. For example, assuming the updated tree structure includes 2 levels, where the first level is the third level in the original tree structure and the second level is the fourth level in the original tree structure, the first level of the updated tree structure can be partitioned into one shard set, and the second level into another shard set.

[0219] In another exemplary scheme, assuming that both the original data structure and the updated data structure are tree structures, but the data nodes at each level are evenly distributed, the data can be partitioned according to the number of nodes. For example, if the original tree structure and the updated tree structure contain 100 nodes, then each set of 20 nodes can be used as a partition set.

[0220] In another exemplary scheme, assuming that the original data structure and the updated data structure are used to store data for different functions in the application, the nodes of the original data structure and the updated data structure can be partitioned according to the different functions of the application.

[0221] The server's implementation of sharding can be illustrated by the following specific process:

[0222] First, the server obtains the sharding boundaries of each shard set according to the sharding strategy. For example, if the data structure is divided by depth, with each shard set consisting of 3 levels, the sharding boundaries of the first shard set can be determined as the 3rd and 4th levels of the tree structure. If the data structure is divided by the number of nodes, with each shard set consisting of the 20th node, then the boundaries can be the 20th and 21st nodes of the tree structure. If the data structure is divided according to the business logic of each data element, then the nodes can be divided according to the business logic to obtain the corresponding shard sets.

[0223] Then, the server shards the original data structure and the updated data structure based on the sharding boundary to obtain the aforementioned multiple shard sets. During this sharding process, to facilitate subsequent access to data within the shard sets, the server can establish unique identifiers for each shard set and establish associations between shard sets and nodes. In one exemplary scheme, the server can create a shard index for each shard set, which serves as the unique identifier for that shard set. The server can then create a shard mapping table to store the associations between shard sets and between shard sets and nodes. Specific details are not limited here. It should be understood that these associations can be used by the server to access data in a node later. For example, the server can determine the shard set where the node belongs based on these associations; then, it can search for the node only within that shard set, thereby reducing data lookup operations and improving data access efficiency. These associations can also be used to determine the loading strategy for multiple shard sets. For example, when the server determines the shard set in which the node belongs based on the association, and then searches for the node only in that shard set, the shard set can be loaded.

[0224] Meanwhile, if a shard set is constantly in a loading state, it will also consume a significant amount of memory resources. Therefore, to reduce memory usage, in this embodiment, the server can also determine whether memory release is necessary based on the activity level of the shard sets and overall memory usage information. In an exemplary scheme, the server can obtain the activity information of each shard set (which can be understood as the number of times each shard set is loaded, i.e., the number of times each shard set is accessed) and the overall data storage memory usage information. When the activity level of a shard set is lower than a first threshold, the memory resources of that shard set can be released directly. Alternatively, when the memory usage information indicates that the memory occupancy rate has reached the memory limit, the shard sets with lower activity levels can be selected for memory release in ascending order of activity level. Furthermore, to ensure data consistency and storage validity, the shard set needs to be persisted in a local or remote database before releasing its memory resources.

[0225] As described above, during sharding, if shard boundaries overlap, duplicate data may exist in each shard set. Therefore, to further reduce memory usage, the shard sets can be merged (this merging process can be understood as storing overlapping parts only once). In an exemplary solution, the server can obtain the storage status information of each shard set. This storage status information can be used to indicate each node and its stored data within the shard set. If the storage status information indicates that at least two shard sets store the same data, this same data can be stored only once, and the association between the at least two shard sets can be updated (e.g., establishing an association between two unrelated shard sets). When storing the same data, any shard set can be selected for storage, or the same data can be stored separately, as long as the at least two shard sets can read the same data when accessed subsequently. Specific limitations are not specified here.

[0226] When the technical solution provided in this application is applied to multi-user collaborative data modification, multiple users may make different modifications to the same node. If a new data structure is created for each user's modification, efficient data merging will not be possible. In this case, semantic awareness technology can be used. During this process, semantic analysis can be used to parse the user's changes and intent to obtain semantic analysis results. Then, based on the semantic analysis results, a merging scheme is determined when different users make different modifications to the same node. For example, if the semantic analysis results indicate that multiple users' changes and meanings are different (e.g., one corrects a grammatical error, and another adds new content), these two updated data can be directly merged to generate a single updated data structure. If the semantic analysis structure indicates that multiple users have semantic conflicts during the modification process (e.g., the modifications to the same text are semantically contradictory), feedback can be sent to multiple users to confirm whether to merge or update separately.

[0227] Based on the above description, the data processing method of this application will be illustrated below with an exemplary workflow. Please refer to [link / reference needed] for details. Figure 5 As shown:

[0228] The server receives data modification requests and then determines whether a proxy object exists. If a proxy object exists, it monitors the modification operation corresponding to the data modification request (this includes creating a new data version) and generates a change log based on the monitoring results. If no proxy object exists, the server directly executes the data modification operation and creates a new data version. During the creation of the new data version, the server checks whether each node in the data structure has a reference count and determines the creation rules for the updated data structure based on the reference count. The server checks if the reference count is greater than 1. If it is greater than 1, a replica is created for the node to be updated, generating a new node; if the reference count is less than or equal to 1, the node to be updated is reused directly. When creating the updated data structure, the server can also determine whether to enable delayed processing; if delayed processing is enabled, the updated data structure is added to the operation queue and a dirty flag is set; then it waits for the `applyChanges` function to be called. Finally, when the `applyChanges` function is called, the server merges the modification operations of the updated data structure in the queue and executes the actual update to complete the update. If the delay is not enabled, the modification operation of the updated data structure will be executed directly to complete the update.

[0229] The data processing apparatus in this application is described in detail below. Please refer to [link / reference]. Figure 6 , Figure 6 This is a schematic diagram of one embodiment of the data processing apparatus in this application. The data processing apparatus 20 includes:

[0230] The acquisition module 201 is used to acquire the original data structure and the node to be updated, wherein the node to be updated is contained in the original data structure;

[0231] Processing module 202 is used to create an update data structure based on the node to be updated, the update data structure being a different data structure from the original data structure; and to write the update data of the node to be updated into the update data structure.

[0232] Storage module 203 is used to store the updated data structure.

[0233] This application provides a data processing apparatus. Using this apparatus, when modifying a node to be updated, the original data structure remains unchanged; only a new data structure is created for the node to be updated. This avoids modifying existing data, thus ensuring data consistency and integrity. Furthermore, each data update generates a new data structure while the old data structure remains unchanged, making debugging and backtracking easier.

[0234] Optionally, in the above Figure 6Based on the corresponding embodiments, in another embodiment of the data processing apparatus 20 provided in this application,

[0235] The processing module 202 is used to call a first function to create the update data structure based on the node to be updated, wherein the first function sets the path of the node to be updated in the original data structure and the update data of the node to be updated.

[0236] This application provides a data processing apparatus. Using this apparatus, new data structures are created by calling functions with paths and updated data. This simplifies operations on complex objects, thereby improving overall processing efficiency and enhancing code readability and maintainability.

[0237] Optionally, in the above Figure 6 Based on the corresponding embodiment, in another embodiment of the data processing device 20 provided in this application, the processing module 202 is used to find the node to be updated in the original data structure based on the path;

[0238] When the path indication is 0, identify the node to be updated and create an update node for that node.

[0239] The updated data structure is generated based on the update node and the replication node, which is generated by replicating the unupdated nodes in the original data structure.

[0240] This application provides a data processing apparatus. Using this apparatus, the node to be updated is determined according to the path indication in the function, and then other nodes are directly copied to generate a new updated data structure. This ensures the consistency and integrity of the overall data while modifying the data of the node to be updated.

[0241] Optionally, in the above Figure 6 Based on the corresponding embodiments, in another embodiment of the data processing apparatus 20 provided in this application,

[0242] The processing module 202 is used to locate the update node in the update data structure based on the path.

[0243] When the path indication is 0, the update node is determined and the update data is written to the update node.

[0244] This application provides a data processing apparatus. Using this apparatus, the node to be updated is determined by path indication, and then the data of the node to be updated is directly modified according to the update data carried in the function. This allows for efficient data modification while ensuring the consistency and integrity of the overall data.

[0245] Optionally, in the above Figure 6 Based on the corresponding embodiments, in another embodiment of the data processing apparatus 20 provided in this application,

[0246] The processing module 202 is used to set reference counts and reference relationships for each node in the original data structure. The reference count is used to record the number of times the current node is shared, and the reference relationship is used to indicate the data sharing relationship between each node.

[0247] This application provides a data processing apparatus. Using this apparatus, after adding reference counts to nodes, when creating and updating data structures, only replica nodes of the updated node are created. Other unupdated nodes can share data through reference counts and reference relationships, thus reducing data redundancy and memory usage issues during data modification. Simultaneously, reference counting can be used to manage the lifecycle of nodes, ensuring memory is released when there are no other references, preventing memory leaks.

[0248] Optionally, in the above Figure 6 Based on the corresponding embodiment, in another embodiment of the data processing device 20 provided in this application, the processing module 202 is used to find the node to be updated in the original data structure based on the path;

[0249] When the path indication is 0, identify the node to be updated and obtain the reference count of the node to be updated;

[0250] When the reference count is greater than 1, create an update node for the node to be updated and modify the reference count of the node to be updated.

[0251] The updated data structure is generated based on the updated node, and a reference relationship is set for the updated data structure. This reference relationship is used to indicate that the updated data structure shares the unupdated nodes in the original data structure.

[0252] In this application embodiment, a data processing apparatus is provided. Using this apparatus, after adding reference counts to nodes, when creating an updated data structure, only a copy of the updated node is created. Other unupdated nodes can share data through reference counts and reference relationships. This reduces data redundancy during data modification and significantly reduces memory usage.

[0253] Optionally, in the above Figure 6 Based on the corresponding embodiments, in another embodiment of the data processing device 20 provided in this application, the storage module 203 is used to store the updated data structure and the original data structure in a historical version array, and the updated data structure and the original data structure have corresponding indexes.

[0254] In this application embodiment, a data processing apparatus is provided. Using this apparatus, the updated data structure and the original data structure are stored in an array, and a specific version of the data structure is accessed through an index, thereby enabling rollback to a certain data structure. This ensures data consistency and integrity.

[0255] Optionally, in the above Figure 6 Based on the corresponding embodiments, in another embodiment of the data processing apparatus 20 provided in this application,

[0256] The acquisition module 201 is used to acquire the first index of the updated data structure;

[0257] The processing module 202 is used to access the updated data structure from the historical version array based on the first index.

[0258] In this application embodiment, a data processing apparatus is provided. Using this apparatus, the updated data structure and the original data structure are stored in an array, and a specific version of the data structure is accessed through an index, thereby enabling rollback to a certain data structure. This ensures data consistency and integrity.

[0259] Optionally, in the above Figure 6 Based on the corresponding embodiments, in another embodiment of the data processing apparatus 20 provided in this application,

[0260] This processing module 202 is used to create proxy objects;

[0261] Based on the proxy object, obtain the original data structure and the update operation of the updated data structure.

[0262] In this application embodiment, a data processing apparatus is provided. Using this apparatus, data update operations are recorded, thereby enabling predictable management of these records and facilitating debugging and backtracking.

[0263] Optionally, in the above Figure 6 Based on the corresponding embodiments, in another embodiment of the data processing apparatus 20 provided in this application,

[0264] The processing module 202 is used to call a second function to create a proxy object, the second function accepting the original data structure and the updated data structure;

[0265] The proxy object is used to call a third function to obtain the original data structure and the update operation of the updated data structure. The third function is used to set the capture object of the proxy object.

[0266] This application provides a data processing apparatus. Using this apparatus, data update operations are recorded, thereby enabling predictable management of these records and facilitating debugging and backtracking. Furthermore, the use of corresponding functions to implement proxy objects and set capture objects for these proxy objects simplifies operations on complex objects, improving overall processing efficiency and enhancing code readability and maintainability.

[0267] Optionally, in the above Figure 6 Based on the corresponding embodiments, in another embodiment of the data processing apparatus 20 provided in this application,

[0268] The processing module 202 is used to record the update operations of the original data structure and the updated data structure to generate a change log; and to store the change log.

[0269] In this application embodiment, a data processing apparatus is provided. Using this apparatus, data update operations are recorded, thereby enabling predictable management of these records and facilitating debugging and backtracking.

[0270] Optionally, in the above Figure 6 Based on the corresponding embodiments, in another embodiment of the data processing apparatus 20 provided in this application,

[0271] The processing module 202 is used to call the fourth function to record the update operations of the original data structure and the updated data structure in order to generate a change log.

[0272] This application provides a data processing apparatus. Using this apparatus, data update operations are recorded, thereby enabling predictable management of these records and facilitating debugging and backtracking. Furthermore, calling functions to record update operations simplifies operations on complex objects, improving overall processing efficiency and enhancing code readability and maintainability.

[0273] Optionally, in the above Figure 6 Based on the corresponding embodiments, in another embodiment of the data processing apparatus 20 provided in this application, the processing module 202 is used to set a dirty flag for the original data structure and initialize the value of the dirty flag; put the updated data structure into a delayed processing queue; update the dirty flag value of the original data structure to a target value, the dirty flag being used to indicate whether there is an unapplied changed data structure, and the target value being used to indicate that there is an unapplied changed data structure in the delayed processing queue; after triggering the execution operation, execute the updated data structure in the delayed processing queue to write the updated data into the updated data structure.

[0274] In this application embodiment, a data processing apparatus is provided. Using this apparatus, a delayed processing mechanism is constructed to merge multiple updated data structures into a single operation, reducing the overhead of multiple data structure updates, improving efficiency, and thereby reducing unnecessary computation and memory allocation.

[0275] Optionally, in the above Figure 6 Based on the corresponding embodiments, in another embodiment of the data processing apparatus 20 provided in this application, the processing module 202 is used to perform sharding processing on the original data structure and the updated data structure according to a sharding strategy to obtain multiple shard sets. The sharding strategy includes sharding processing according to the depth of the data structure, or sharding processing according to the number of nodes in the data structure, or sharding processing according to the business logic of the data in the data structure. Each shard set in the multiple shard sets includes multiple nodes.

[0276] This application provides a data processing apparatus. Using this apparatus, the original data structure and the updated data structure are stored in shards. Different sharding strategies can be applied to different data structures, thereby increasing the application scenarios of the solution. For example, for deeply nested data structures, sharding based on the depth of the data structure can effectively control the size of each shard and facilitate hierarchical data management. For scenarios with relatively even data distribution, sharding can be performed according to the number of nodes in the data structure, ensuring a relatively balanced data volume in each shard and promoting even distribution of memory usage. When the data stored in the data structure has functional or module-related relationships, the data structure can be sharded according to business logic. This organizes related data in the same shard, improving the locality of data access and facilitating independent expansion of business modules.

[0277] Optionally, in the above Figure 6 Based on the corresponding embodiments, in another embodiment of the data processing apparatus 20 provided in this application, the processing module 202 determines the fragmentation boundary according to the fragmentation strategy;

[0278] The original data structure and the updated data structure are sharded based on the sharding boundary to obtain multiple sharding sets. The multiple sharding sets also include sharding identifiers, which are used to find and identify the sharding sets.

[0279] Establish the association relationship between the multiple shard sets, the association relationship being used to indicate the data dependency relationship between the multiple shard sets and the dependency relationship between the node and the multiple shard sets;

[0280] Based on the aforementioned relationship, a loading strategy for the multiple shard sets is determined. The loading strategy is used to implement on-demand loading of the multiple shard sets and to maintain the loading status of the multiple shard sets.

[0281] This application provides a data processing apparatus. Using this apparatus, the original data structure and the updated data structure are stored in shards. During shard construction, the relationships between each shard and its nodes, as well as the shard identification information, are established. This allows the loading strategy of the shard set to be determined based on the relationships, thereby enabling on-demand loading of data stored in nodes within each shard set through the sharding mechanism. Furthermore, based on the relationships between shards and nodes, during data reading, only the data of a single shard can be read directly through the relationships between shards and nodes, thus providing a more efficient state access mechanism.

[0282] Optionally, in the above Figure 6 Based on the corresponding embodiments, in another embodiment of the data processing device 20 provided in this application, the acquisition module 201 acquires the activity information and memory storage usage information of each shard set in the plurality of shard sets. The activity information is used to indicate the number of times the shard set is loaded, and the memory storage usage information is used to indicate the amount of memory occupied by the original data structure and the updated data structure.

[0283] The processing module 202 is used to release the memory resources of the first target shard set when the activity information of the first target shard set is lower than a first threshold, wherein the data of the first target shard set is persisted to a target database, which is a local database or a remote database;

[0284] or,

[0285] When the amount of memory occupied, as indicated by the memory storage usage information, reaches a second threshold, the memory resources of the second target shard set, which are ranked Nth from low to high based on the activity information, are selected, and the data of the second target shard set is persisted to the target database.

[0286] In this embodiment, a data processing apparatus is provided. Using this apparatus, the original data structure and the updated data structure are stored in fragments, and during fragment construction, the relationships between each fragment and the fragment identification information are established. Simultaneously, the loading strategy of the fragment set is determined based on the relationships. This fragmentation mechanism enables the automatic release of memory resources from inactive fragment sets, achieving dynamic adjustment of memory usage.

[0287] Optionally, in the above Figure 6Based on the corresponding embodiments, in another embodiment of the data processing device 20 provided in this application, the acquisition module 201 is used to acquire the storage status information of each shard set in the plurality of shard sets;

[0288] The processing module 202 is used to merge the overlapping parts of the at least two shard sets into one data when the storage status information indicates that at least two shard sets overlap, and update the association relationship of the at least two shard sets.

[0289] This application provides a data processing apparatus. Using this apparatus, the original data structure and the updated data structure are stored in fragments, and the relationships between fragments and the identification information of each fragment are established during fragment construction. Simultaneously, the fragment sets are merged based on their storage states, thus reducing unnecessary memory usage of state data.

[0290] The data processing apparatus provided in this application can be used on a server; please refer to [link / reference]. Figure 7 , Figure 7 This is a schematic diagram of a server structure provided in an embodiment of this application. The server 300 can vary significantly due to different configurations or performance. It may include one or more central processing units (CPUs) 322 (e.g., one or more processors) and memory 332, and one or more storage media 330 (e.g., one or more mass storage devices) for storing application programs 342 or data 344. The memory 332 and storage media 330 can be temporary or persistent storage. The program stored in the storage media 330 may include one or more modules (not shown in the diagram), each module may include a series of instruction operations on the server. Furthermore, the CPU 322 may be configured to communicate with the storage media 330 and execute the series of instruction operations stored in the storage media 330 on the server 300.

[0291] Server 300 may also include one or more power supplies 326, one or more wired or wireless network interfaces 350, one or more input / output interfaces 358, and / or one or more operating systems 341, such as Windows Server. TM Mac OS X TM Unix TM Linux TM FreeBSD TM etc.

[0292] The steps performed by the server in the above embodiments can be based on this Figure 7 The server structure shown.

[0293] The data processing apparatus provided in this application can be used in terminal devices; please refer to [link / reference]. Figure 8 For ease of explanation, only the parts relevant to the embodiments of this application are shown. For specific technical details not disclosed, please refer to the method section of the embodiments of this application. In the embodiments of this application, a smartphone is used as an example for illustration:

[0294] Figure 8 This is a block diagram illustrating a portion of the structure of a smartphone related to the terminal device provided in the embodiments of this application. (Reference) Figure 8 The smartphone includes components such as a radio frequency (RF) circuit 410, a memory 420, an input unit 430, a display unit 440, a sensor 450, an audio circuit 460, a wireless fidelity (WiFi) module 470, a processor 480, and a power supply 490. Those skilled in the art will understand that... Figure 8 The smartphone structure shown does not constitute a limitation on smartphones and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0295] The following is combined Figure 8 A detailed introduction to the various components of a smartphone:

[0296] RF circuit 410 can be used for receiving and transmitting signals during information transmission or calls. Specifically, it receives downlink information from the base station and processes it with processor 480; additionally, it transmits uplink data to the base station. Typically, RF circuit 410 includes, but is not limited to, an antenna, at least one amplifier, a transceiver, a coupler, a low-noise amplifier (LNA), and a duplexer. Furthermore, RF circuit 410 can also communicate wirelessly with networks and other devices. The aforementioned wireless communication can use any communication standard or protocol, including but not limited to Global System for Mobile Communication (GSM), General Packet Radio Service (GPRS), Code Division Multiple Access (CDMA), Wideband Code Division Multiple Access (WCDMA), Long Term Evolution (LTE), email, and Short Message Service (SMS).

[0297] The memory 420 can be used to store software programs and modules. The processor 480 executes various functions and data processing of the smartphone by running the software programs and modules stored in the memory 420. The memory 420 may mainly include a program storage area and a data storage area. The program storage area may store the operating system, applications required for at least one function (such as sound playback function, image playback function, etc.), etc.; the data storage area may store data created according to the use of the smartphone (such as audio data, phonebook, etc.). In addition, the memory 420 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other volatile solid-state storage device.

[0298] The input unit 430 can be used to receive input numerical or character information, and to generate key signal inputs related to user settings and function control of the smartphone. Specifically, the input unit 430 may include a touch panel 431 and other input devices 432. The touch panel 431, also known as a touch screen, can collect touch operations performed by the user on or near it (such as operations performed by the user using a finger, stylus, or any suitable object or accessory on or near the touch panel 431), and drive the corresponding connected devices according to a pre-set program. Optionally, the touch panel 431 may include two parts: a touch detection device and a touch controller. The touch detection device detects the user's touch position and the signal generated by the touch operation, and transmits the signal to the touch controller; the touch controller receives touch information from the touch detection device, converts it into touch point coordinates, and sends it to the processor 480, and can also receive and execute commands sent by the processor 480. In addition, the touch panel 431 can be implemented using various types such as resistive, capacitive, infrared, and surface acoustic wave. In addition to the touch panel 431, the input unit 430 may also include other input devices 432. Specifically, other input devices 432 may include, but are not limited to, one or more of the following: physical keyboard, function keys (such as volume control buttons, power buttons, etc.), trackball, mouse, joystick, etc.

[0299] Display unit 440 can be used to display information input by the user or information provided to the user, as well as various menus of the smartphone. Display unit 440 may include display panel 441, optionally configured as a liquid crystal display (LCD), organic light-emitting diode (OLED), or similar form. Further, touch panel 431 may cover display panel 441. When touch panel 431 detects a touch operation on or near it, it transmits the information to processor 480 to determine the type of touch event. Subsequently, processor 480 provides corresponding visual output on display panel 441 based on the type of touch event. Although in Figure 8 In this embodiment, the touch panel 431 and the display panel 441 are two separate components to realize the input and output functions of the smartphone. However, in some embodiments, the touch panel 431 and the display panel 441 can be integrated to realize the input and output functions of the smartphone.

[0300] The smartphone may also include at least one sensor 450, such as a light sensor, a motion sensor, and other sensors. Specifically, the light sensor may include an ambient light sensor and a proximity sensor, wherein the ambient light sensor can adjust the brightness of the display panel 441 according to the ambient light level, and the proximity sensor can turn off the display panel 441 and / or the backlight when the smartphone is moved to the ear. As a type of motion sensor, an accelerometer sensor can detect the magnitude of acceleration in various directions (generally three axes), and can detect the magnitude and direction of gravity when stationary. It can be used for applications that recognize the smartphone's posture (such as landscape / portrait switching, related games, magnetometer posture calibration), vibration recognition-related functions (such as pedometer, tapping), etc. Other sensors that may be configured in the smartphone, such as gyroscopes, barometers, hygrometers, thermometers, and infrared sensors, will not be described in detail here.

[0301] Audio circuit 460, speaker 461, and microphone 462 provide an audio interface between the user and the smartphone. Audio circuit 460 converts received audio data into electrical signals and transmits them to speaker 461, where speaker 461 converts them into sound signals for output. On the other hand, microphone 462 converts collected sound signals into electrical signals, which are received by audio circuit 460, converted into audio data, and then processed by processor 480 before being transmitted via RF circuit 410 to, for example, another smartphone, or the audio data can be output to memory 420 for further processing.

[0302] WiFi is a short-range wireless transmission technology. Smartphones, through their WiFi modules (470), can help users send and receive emails, browse web pages, and access streaming media, providing wireless broadband internet access. Although Figure 8 WiFi module 470 is shown, but it is understood that it is not an essential component of a smartphone and can be omitted as needed without changing the nature of the invention.

[0303] The processor 480 is the control center of the smartphone, connecting various parts of the smartphone through various interfaces and lines. It performs various functions and processes data by running or executing software programs and / or modules stored in the memory 420, and by calling data stored in the memory 420, thereby providing overall monitoring of the smartphone. Optionally, the processor 480 may include one or more processing units; optionally, the processor 480 may integrate an application processor and a modem processor, wherein the application processor mainly handles the operating system, user interface, and applications, and the modem processor mainly handles wireless communication. It is understood that the aforementioned modem processor may also not be integrated into the processor 480.

[0304] The smartphone also includes a power supply 490 (such as a battery) that supplies power to various components. Optionally, the power supply can be logically connected to the processor 480 through a power management system, thereby enabling functions such as charging, discharging, and power consumption management through the power management system.

[0305] Although not shown, smartphones may also include a camera, Bluetooth module, etc., which will not be described in detail here.

[0306] The steps performed by the terminal device in the above embodiments can be based on this Figure 8 The terminal device structure is shown.

[0307] This application also provides a computer-readable storage medium storing a computer program that, when run on a computer, causes the computer to perform the methods described in the foregoing embodiments.

[0308] This application also provides a computer program product including a program, which, when run on a computer, causes the computer to perform the methods described in the foregoing embodiments.

[0309] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0310] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection between apparatuses or units through some interfaces, and may be electrical, mechanical, or other forms.

[0311] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0312] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0313] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0314] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A data processing method, characterized by, include: Obtain the original data structure and the node to be updated, wherein the node to be updated is contained in the original data structure; An updated data structure is created based on the node to be updated, and the updated data structure is a different data structure from the original data structure. Write the update data of the node to be updated into the update data structure; Store the updated data structure.

2. The method of claim 1, wherein, The process of creating an update data structure based on the node to be updated includes: The first function is invoked to create the updated data structure based on the node to be updated, wherein the first function sets the path of the node to be updated in the original data structure and the updated data of the node to be updated.

3. The method of claim 2, wherein, The step of calling the first function to create the updated data structure based on the node to be updated includes: Based on the path, the node to be updated is located in the original data structure; When the path indication is 0, the node to be updated is determined, and an update node for the node to be updated is created; The updated data structure is generated based on the updated node and the replicated node, wherein the replicated node is generated by replicating the unupdated nodes in the original data structure.

4. The method of claim 3, wherein, The step of writing the update data of the node to be updated into the update data structure includes: Based on the path, the update node is located in the updated data structure; When the path indication is 0, the update node is determined, and the update data is written to the update node.

5. The method of claim 2, wherein, The method further includes: A reference count and a reference relationship are set for each node in the original data structure. The reference count is used to record the number of times the current node is shared, and the reference relationship is used to indicate the data sharing relationship between each node.

6. The method of claim 5, wherein, The step of calling the first function to create the updated data structure based on the node to be updated includes: Based on the path, the node to be updated is located in the original data structure; When the path indication is 0, determine the node to be updated and obtain the reference count of the node to be updated; When the reference count is greater than 1, create an update node for the node to be updated and modify the reference count of the node to be updated. The updated data structure is generated based on the updated node, and a reference relationship is set for the updated data structure. The reference relationship is used to indicate that the updated data structure shares the unupdated nodes in the original data structure.

7. The method according to any one of claims 1 to 6, characterized in that, The structure for storing the updated data includes: The updated data structure and the original data structure are stored in a historical version array, and the updated data structure and the original data structure have corresponding indices; Obtain the first index of the updated data structure; The updated data structure is accessed from the historical version array based on the first index.

8. The method according to any one of claims 1 to 6, characterized in that, The method further includes: Create a proxy object; The update operations for the original data structure and the updated data structure are obtained based on the proxy object.

9. The method of claim 8, wherein, The method further includes: Record the update operations of the original data structure and the updated data structure to generate a change log; Store the change log.

10. The method according to any one of claims 1 to 6, characterized in that, Before writing the update data of the node to be updated into the update data structure, the method further includes: Set a dirty flag for the original data structure and initialize the value of the dirty flag; Place the updated data structure into the delayed processing queue; The dirty flag of the original data structure is updated to a target value. The dirty flag is used to indicate whether there are any unapplied modified data structures, and the target value is used to indicate that there are unapplied modified data structures in the delayed processing queue. After the execution operation is triggered, the update data structure in the delayed processing queue is executed to write the update data into the update data structure.

11. The method according to any one of claims 1 to 6, characterized in that, The method further includes: The original data structure and the updated data structure are sharded according to a sharding strategy to obtain multiple shard sets. The sharding strategy includes sharding according to the depth of the data structure, sharding according to the number of nodes in the data structure, or sharding according to the business logic of the data in the data structure. Each shard set in the multiple shard sets includes multiple nodes.

12. The method of claim 11, wherein, The original data structure and the updated data structure are sharded according to a sharding strategy to obtain multiple shard sets, including: Determine the fragmentation boundaries according to the fragmentation strategy described above; The original data structure and the updated data structure are sharded based on the sharding boundary to obtain multiple sharding sets. The multiple sharding sets also include sharding identifiers, which are used to find and identify the sharding sets. Establish the association relationship between the multiple shard sets, the association relationship being used to indicate the data dependency relationship between the multiple shard sets and the dependency relationship between the node and the multiple shard sets; Based on the aforementioned relationship, a loading strategy for the multiple shard sets is determined. The loading strategy is used to implement on-demand loading of the multiple shard sets and to maintain the loading status of the multiple shard sets.

13. The method of claim 12, wherein, The method further includes: Obtain the activity information and memory storage usage information of each shard set in the plurality of shard sets. The activity information is used to indicate the number of times the shard set is loaded, and the memory storage usage information is used to indicate the amount of memory occupied by the original data structure and the updated data structure. If the activity information of the first target shard set is lower than a first threshold, the memory resources of the first target shard set are released, wherein the data of the first target shard set is persisted to the target database, and the target database is a local database or a remote database; or, When the amount of memory occupied, as indicated by the memory storage usage information, reaches a second threshold, the memory resources of the second target shard set, which are ranked Nth from low to high based on the activity information, are selected, and the data of the second target shard set is persisted to the target database.

14. The method of claim 12, wherein, The method further includes: Obtain the storage status information of each shard set in the plurality of shard sets; When the storage status information indicates that at least two shard sets overlap, the overlapping portions of the at least two shard sets are merged into one data set, and the association between the at least two shard sets is updated.

15. A data processing apparatus, characterized by include: An acquisition module is used to acquire the original data structure and the node to be updated, wherein the node to be updated is contained in the original data structure; The processing module is used to create an updated data structure based on the node to be updated, wherein the updated data structure is a different data structure from the original data structure; Write the update data of the node to be updated into the update data structure; A storage module is used to store the updated data structure.

16. A computer device, characterized in that, include: Memory, processor, and bus system; The memory is used to store programs; The processor is configured to execute a program in the memory, and the processor is configured to execute the method of any one of claims 1 to 14 according to instructions in the program code; The bus system is used to connect the memory and the processor to enable communication between the memory and the processor.

17. A computer-readable storage medium comprising instructions that, when executed on a computer, cause the computer to perform the method as claimed in any one of claims 1 to 14.

18. A computer program product, comprising a computer program, characterized in that, The computer program is executed by a processor using the method as described in any one of claims 1 to 14.

Images