Method, system, device, medium and product for managing dynamic data

Abstract

The embodiment of the application discloses a kind of dynamic data management method, system, equipment, medium and product, comprising: receiving the creation request of dynamic structure table, corresponding metadata record is created based on the creation request;In response to receiving data write request, a data write batch identification is allocated and returned for the client, and the batch state corresponding to the data write batch identification is marked as unqueryable state;In response to receiving the data to be written sent by the client, the field information in the data to be written is identified, and the field information is compared with the initial field information, and the metadata record is updated according to the comparison result;In response to determining that the data to be written is completed after writing, the batch state is marked as queryable state;In response to data query request, based on the data write batch identification and the metadata record, retrieve and return the target data corresponding to the data query request.

Description

Technical Field

[0001] The application generally relates to the field of data processing technology, and in particular to a method, system, device, medium and product for managing dynamic data. Background Technology

[0002] With the widespread application of cloud computing and big data technologies, computing network platforms need to process massive amounts of dynamically changing indicator data in real time. Currently, related technologies typically employ solutions such as document databases, wide column storage, or key-value storage to adapt to the storage needs of schema-less data.

[0003] However, when faced with high-concurrency, large-scale data writing scenarios, these solutions often struggle to balance strong data consistency and efficient query performance, leading to significant contradictions between data atomic visibility, real-time query response, and dynamic structural expansion. Therefore, how to support dynamic data structures while achieving consistent writing of massive amounts of data and high-performance querying has become a pressing technical problem to be solved. Summary of the Invention In view of the above, this application provides a method, system, device, medium and product for managing dynamic data, addressing some of the deficiencies mentioned in the background art.

[0004] In a first aspect, embodiments of this application provide a method for managing dynamic data, including: The system receives a request to create a dynamically structured table from a client, and creates a corresponding metadata record based on the request; wherein the metadata record includes resource type and initial field information. In response to receiving a data write request from the client, a data write batch identifier is allocated and returned to the client, and the batch status corresponding to the data write batch identifier is marked as unqueryable. In response to receiving the data to be written sent by the client, the system identifies the field information in the data to be written, compares the field information with the initial field information, and updates the metadata record based on the comparison result. In response to determining that the data to be written has been completed, the batch status is marked as queryable. In response to a data query request, based on the data write batch identifier and the metadata record, the target data corresponding to the data query request is retrieved and returned.

[0005] In a second aspect, embodiments of this application provide a dynamic data management system, including: The metadata record creation module is configured to: receive a creation request for a dynamically structured table sent by a client, and create a corresponding metadata record based on the creation request; wherein, the metadata record includes resource type and initial field information; The write request receiving module is configured to: in response to receiving a data write request sent by the client, allocate and return a data write batch identifier to the client, and mark the batch status corresponding to the data write batch identifier as unqueryable; The metadata record update module is configured to: in response to receiving data to be written sent by the client, identify field information in the data to be written, compare the field information with the initial field information, and update the metadata record according to the comparison result; The data query module is configured to: in response to a data query request, retrieve and return the target data corresponding to the data query request based on the data write batch identifier and the metadata record.

[0006] In a third aspect, this application provides a computer device including one or more processors, a memory, and one or more programs, wherein the one or more programs are stored in the memory and executed by the one or more processors, and the one or more programs include instructions for performing the method described in the first aspect.

[0007] In a fourth aspect, this application provides a non-volatile computer-readable storage medium containing a computer program that, when executed by one or more processors, causes the one or more processors to perform the method described in the first aspect.

[0008] In a fifth aspect, this application provides a computer program product comprising one or more computer programs that, when executed by one or more processors, implement the method described in the first aspect.

[0009] The dynamic data management method, system, device, medium, and product provided in this disclosure achieve adaptive storage and consistency management of dynamic structured data by constructing a hierarchical storage architecture that includes metadata records and resource instance data tables, and introducing a batch-based status control mechanism. Specifically, new fields are automatically identified and metadata is updated during data writing; atomic visibility is achieved through status flags during data submission; and efficient retrieval is achieved based on batch status and metadata during data querying. This ensures consistency, availability, and query performance for massive data operations while supporting dynamic changes in data structure. Attached Figure Description

[0010] Figure 1A flowchart illustrating a dynamic data management method provided in an embodiment of this application is shown. Figure 2 This illustration shows a flowchart of a storage attribute record provided in an embodiment of this application; Figure 3 This paper shows a schematic diagram of the structure of a dynamic data management system provided in an embodiment of this application; Figure 4 A schematic diagram of a more specific electronic device hardware structure provided in this embodiment is shown. Detailed Implementation

[0011] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with specific embodiments and the accompanying drawings.

[0012] It should be noted that, unless otherwise defined, the technical or scientific terms used in the embodiments of this application should have the ordinary meaning understood by one of ordinary skill in the art to which this application pertains. The terms "first," "second," and similar terms used in the embodiments of this application do not indicate any order, quantity, or importance, but are merely used to distinguish different components. Terms such as "comprising" or "including" mean that the element or object preceding the word encompasses the elements or objects listed after the word and their equivalents, without excluding other elements or objects. Terms such as "connected" or "linked" are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. Terms such as "upper," "lower," "left," and "right" are only used to indicate relative positional relationships; when the absolute position of the described object changes, the relative positional relationship may also change accordingly.

[0013] The term "in response to" as used herein refers to a state in which a corresponding event occurs or a condition is satisfied. It will be understood that the timing of subsequent actions performed in response to such event or condition is not necessarily strongly correlated with the time when the event occurs or the condition is met. For example, in some cases, subsequent actions may be performed immediately upon the occurrence of the event or the fulfillment of the condition; while in others, they may be performed some time after the occurrence of the event or the fulfillment of the condition.

[0014] For clarity, before describing the specific technical solutions of the embodiments of this application, the technical terms involved in the embodiments of this application will be explained first.

[0015] As described in the background section, with the rapid development of cloud computing and big data technologies, cloud-native applications, IoT devices, and intelligent operation and maintenance systems continuously generate massive amounts of monitoring indicator data. This schema-less data does not follow a fixed pattern or format when stored or represented. Its data fields may be dynamically added or removed according to business needs, which is fundamentally different from the fixed table structure relied upon by traditional relational databases. This requires database administrators to frequently perform Data Definition Language (DDL) operations to manually modify the table structure, resulting in high maintenance costs.

[0016] To address the storage needs arising from frequent changes in data structures, various technical solutions have been proposed, such as: using document-oriented databases to adaptively store dynamic fields in a semi-structured format; using key-value pair storage to serialize dynamic structured data into JSON strings as values ​​for index-free storage; using wide column storage to dynamically specify column families when inserting data; or using text search engines to automatically infer the type of new fields and build inverted indexes based on dynamic mapping.

[0017] However, the above solutions still have significant shortcomings when dealing with the storage and consistency management of large-scale dynamic structured data, specifically: 1. While document-oriented databases can adapt to scenarios with frequent changes in data structure, they are prone to performance bottlenecks or even system unavailability when maintaining consistency at the scale of hundreds of millions of data points. 2. The lack of an effective secondary index mechanism in key-value pair storage leads to poor performance in cross-field conditional queries; wide column storage does not support complex structured query operations (such as Join, Group By) and cross-row transactions, and cannot meet the atomicity requirements of large-scale data. 3. Text search engines do not support transactions and cannot provide data consistency guarantees.

[0018] In addition, related technologies typically employ database locking mechanisms to ensure data consistency. When faced with batch writing of hundreds of millions of data entries, these mechanisms can easily lead to high memory consumption, lock contention, reduced system availability, and a significant decrease in concurrency performance. Furthermore, they are difficult to isolate uncommitted data during the writing process, resulting in dirty reads where the intermediate state of the data can be accessed externally in micro-batch writing mode.

[0019] In view of this, this application provides a method, system, device, medium and product for managing dynamic data, which can solve or partially solve the above problems to a certain extent.

[0020] First, the dynamic data management method disclosed herein can be executed by a server or cloud-based data service platform, and its interaction objects can be various types of clients, such as data acquisition and computing programs, business applications, IoT device agents, etc. The core of this application's method lies in constructing a layered metadata management and data storage architecture and introducing a batch-based state control mechanism to achieve consistent management of dynamic structured data throughout its entire lifecycle.

[0021] Figure 1 The diagram shows a flowchart of a dynamic data management method provided in an embodiment of this application.

[0022] like Figure 1 As shown, the method may include the following steps: S102: Receive a request to create a dynamic structure table sent by the client, and create a corresponding metadata record based on the creation request; wherein, the metadata record includes resource type and initial field information.

[0023] In the initial stage, a basic framework needs to be defined for a certain type of dynamic data (or "resource"). Specifically, a client (such as a data collection and calculation program) can initiate a request to create a dynamically structured table based on its business needs. This request can include a description of this type of resource, which must at least include the resource type and initial field information. The resource type is used to uniquely identify a type of data; for example, "pool" represents a resource pool metric. The initial field information describes the data structure known at the time of creation, and may include, for example, field names (code), field descriptions (desc), and expected field types (type).

[0024] In some optional embodiments, the operation of creating metadata records can be specified as establishing a series of logically related management tables in the underlying database. For example, this may include a resource table, a resource field table, and a resource sharding table. Specifically, the resource type, resource description, and configurable historical batch quantity information from the creation request are stored in the resource table; the initial field information (field name, description, type) from the creation request is stored in the resource field table; simultaneously, based on a preset or dynamically calculated number of shards, multiple physical resource instance data tables (such as pool_1, pool_2, etc.) are pre-created, and the correspondence between resource types and each instance data table is stored in the resource sharding table, laying the foundation for subsequent data sharding storage. It is understood that the number of shards can be dynamically adjusted based on data volume estimates or system load.

[0025] In some optional embodiments, the aforementioned resource table may be as shown in Table 1: Table 1 Resource Table

[0026] In some optional embodiments, the aforementioned resource field table (resource_field) can be as shown in Table 2: Table 2 Resource Fields Table

[0027] In some optional embodiments, the aforementioned resource sharding table (resource_table) can be as shown in Table 3: Table 3 Resource Fragmentation Table

[0028] In this embodiment, by pre-creating and maintaining this set of metadata records, a unified data dictionary and data navigation are established for subsequent dynamic data writing and querying. Regardless of changes in the data structure, the system can understand the semantics and storage location of the data based on this metadata record, which is the foundation for achieving structural adaptation.

[0029] S104: In response to receiving a data write request sent by the client, allocate and return a data write batch identifier to the client, and mark the batch status corresponding to the data write batch identifier as unqueryable.

[0030] When a client is ready to write a batch of data, it first sends a data write request to the server. The server responds to this request by assigning a globally unique batch identifier (batch) to this batch of data, such as a high-precision timestamp or sequence number. Simultaneously, the server creates a record corresponding to this batch identifier in the metadata record (e.g., in the newly added resource batch table `resource_batch`) and marks its batch status as "unqueryable" (e.g., `collecting`).

[0031] In some alternative embodiments, the batch identifier generation algorithm may include a timestamp-based algorithm, a UUID-based algorithm, or a distributed sequence generation algorithm. Batch status management can be implemented in a separate batch status table or integrated into a resource table via status bits. Regardless of the implementation, the core objective is to establish a logical transaction boundary for this batch of data.

[0032] In this embodiment, by introducing batch identification and a status mechanism, large-scale data write operations are divided into multiple logically atomic batches. While a batch is marked as "unqueryable," its data is invisible to external queryers, providing a core control mechanism for subsequent atomic commits and consistent visibility of data.

[0033] S106: In response to receiving the data to be written sent by the client, identify the field information in the data to be written, compare the field information with the initial field information, and update the metadata record according to the comparison result.

[0034] After obtaining the batch identifier, the client can begin sending the actual data to be written. Upon receiving this data, the server first identifies the actual fields contained within the data. Then, it compares the identified set of fields with the initial field information created in step S102.

[0035] Furthermore, the server can update the metadata record based on the comparison results. Specifically, this can include the following process: The server determines whether the data to be written contains any new fields not defined in the initial field information. If such new fields exist, a dynamic metadata expansion process is triggered. The server automatically infers the data type of the new field based on its actual value format characteristics. For example, if the field value can be parsed as a floating-point number, it is inferred to be numeric; if it conforms to a date / time format, it is inferred to be date / time; otherwise, it is inferred to be string by default. It is understood that the data type inference rules can be configured and extended, and are not limited to string, numeric, and date / time types, but can also include boolean, geospatial, etc.

[0036] After inferring the data type, the server can add the field name of the new field and its inferred data type as a new field definition to the metadata record (for example, inserting a new record into the resource_field table).

[0037] In this embodiment, by performing field comparison and type inference in real time during the data writing process, the automatic and dynamic expansion of metadata records is achieved. This enables the data storage structure to adapt to changes in business data without requiring manual intervention from developers or DBAs in the database table structure, greatly improving the system's flexibility and maintainability.

[0038] In some optional embodiments, the server adds the field name of the new field and its inferred data type as the new field definition to the metadata record. It can also convert the entity record in the data to be written into multiple attribute records and store the multiple attribute records in the target resource instance data table.

[0039] Figure 2 This illustration shows a flowchart of a storage attribute record provided in an embodiment of this application.

[0040] like Figure 2 As shown, this step specifically includes: S1062: Convert the entity record in the data to be written into multiple attribute records; wherein each attribute record contains multiple typed value fields.

[0041] S1064: Based on the data type in the metadata record, store the attribute values ​​in the attribute record into the corresponding typed value field to obtain the processed attribute record.

[0042] S1066: Determine the target resource instance data table from the plurality of resource instance data tables based on the load balancing strategy; wherein, the load balancing strategy includes a round-robin strategy, a hash modulo strategy, or a least-of-data-volume strategy.

[0043] S1068: Store the processed attribute records in the target resource instance data table.

[0044] After updating the metadata records, the data needs to be transformed for persistence in the predefined storage structure. This step first involves disassembling and transforming each entity record (i.e., a complete JSON object or a single row of a wide table record) in the data to be written.

[0045] In some optional embodiments, the target format for the conversion can be an Entity-Attribute-Value (EAV) model. Specifically, one entity record is converted into multiple attribute records. Each attribute record corresponds to a field in the entity record. An attribute record contains multiple typed value fields; for example, it can be designed to contain at least one string field, one number field, and one datetime field. During the conversion, the system can determine the data type of the field corresponding to the current attribute record based on the updated metadata record (i.e., the latest metadata record) in S103. Then, the attribute value of that field is stored in the corresponding typed value field in the attribute record according to its data type. For example, if the use_ratio field is a number, its value 0.8 is stored in the number field; if the area field is a string, its value "Suzhou" is stored in the string field.

[0046] After the conversion is complete, these attribute records need to be stored in the physical storage layer. Since multiple resource instance data tables have already been created in S101, the system needs to decide which table to store the data for the current batch in.

[0047] In some optional embodiments, this process can be implemented using a load balancing strategy. Specifically, the system can determine a target resource instance data table from multiple resource instance data tables based on the batch identifier written to the data, combined with a round-robin strategy, a hash modulo strategy, or a least-priority strategy based on the amount of data in each table. Subsequently, the processed attribute records are stored in the target table. It is understood that the choice of load balancing strategy can be configured according to different requirements such as data distribution uniformity and query hotspot dispersion.

[0048] In this embodiment, by converting dynamically structured entity records into standardized EAV attribute records and utilizing an optimized design that stores different types of values ​​in multiple columns, data can be stored efficiently in a structured manner. Combining this with a load balancing strategy to distribute data across multiple physical tables not only improves the concurrent throughput of writes but also enables parallel queries, which is key to achieving high-performance storage and querying.

[0049] S108: In response to determining that the data to be written has been completed, the batch status is marked as queryable.

[0050] After the client has sent all the data to be written in batches, it will send a data commit request to the server. The server responds to this request by updating the batch status in the metadata record corresponding to the batch created in S102 from "unqueryable status" (such as collecting) to "queryable status" (such as ready or completed).

[0051] In this embodiment, this step is the core of achieving atomic visibility of large-scale data. The state transition is an atomic operation completed instantaneously. Before this operation, regardless of whether the data has been physically written to storage, it is invisible to external queryers; after this operation, all data in that batch becomes a complete snapshot, immediately visible to external queryers. This design avoids the performance bottleneck caused by initiating massive transactions at the database level, while strictly guaranteeing data consistency semantics.

[0052] S110: In response to a data query request, based on the data writing batch identifier and the metadata record, retrieve and return the target data corresponding to the data query request.

[0053] When an external user or system needs to query data, it sends a data query request to the server. The server's query process is strictly controlled by metadata records and batch status to ensure the consistency of query results.

[0054] Specifically, retrieving and returning the target data may include: First, the system filters all data in a queryable batch state from the metadata records and writes them into batch identifiers. This ensures that the query will not touch any data in a writing or uncommitted state, thus avoiding dirty reads at the source. Then, based on these queryable batch identifiers, the system uses the mapping relationship established in S101 and S102 (such as finding the corresponding table name through the resource_batch table) to locate the resource instance data table that stores these batch data.

[0055] Finally, the system parses the query conditions in the query request (e.g., use_ratio<0.5) and understands the meaning and type of the fields based on the latest metadata records (which contain all dynamically expanded fields). Based on this, the system constructs and executes the query in the located resource instance data table. For example, for numeric conditions, filtering is applied to the number field. The query results (multiple attribute records) are reassembled by the server into an entity record format that is easy for the client to understand and then returned.

[0056] In this embodiment, the query process is tightly integrated with state control during writing, metadata management, and storage routing mechanisms. Strong consistency is ensured by relying on "queryable state" for filtering; dynamically changing fields can be correctly queried by relying on the latest metadata; and efficient targeted retrieval is achieved by locating specific data tables using batch identifiers. This allows this solution to maintain excellent query performance while providing flexible query capabilities.

[0057] Based on the same inventive concept, corresponding to any of the above-described embodiments, this application also provides a dynamic data management system.

[0058] Figure 3 This is a schematic diagram of the structure of a dynamic data management system provided in an embodiment of this application.

[0059] refer to Figure 3 As shown, the dynamic data management system includes: The metadata record creation module 202 is configured to: receive a creation request for a dynamic structure table sent by the client, and create a corresponding metadata record based on the creation request; wherein the metadata record includes resource type and initial field information.

[0060] In some optional embodiments, the dynamic structure table includes a resource table, a resource field table, and a resource sharding table; The metadata record creation module 202 further performs the following steps: Store the resource type, resource description, and batch quantity information from the creation request into the resource table; Store the initial field information from the creation request into the resource field table; Based on a preset number of shards, multiple resource instance data tables are created, and the correspondence between the resource type and each resource instance data table is stored in the resource shard table.

[0061] The write request receiving module 204 is configured to: in response to receiving a data write request sent by the client, allocate and return a data write batch identifier to the client, and mark the batch status corresponding to the data write batch identifier as unqueryable.

[0062] The metadata record update module 206 is configured to: in response to receiving data to be written sent by the client, identify field information in the data to be written, compare the field information with the initial field information, and update the metadata record according to the comparison result.

[0063] In some optional embodiments, the metadata record update module 206 further performs the following steps: In response to determining that the data to be written contains a new field not defined in the initial field information, the data type is determined according to the field value format characteristics of the new field; wherein, the data type includes string, numeric, and time types; Add the field name and data type of the new field to the metadata record; The entity records in the data to be written are converted into multiple attribute records; wherein each attribute record contains multiple typed value fields; Based on the data type in the metadata record, the attribute values ​​in the attribute record are stored in the corresponding typed value field to obtain the processed attribute record; Storing the processed attribute records into the resource instance data table further includes: The target resource instance data table is determined from the plurality of resource instance data tables based on a load balancing strategy; wherein, the load balancing strategy includes a round-robin strategy, a hash modulo strategy, or a least-first strategy based on data volume; The processed attribute records are stored in the target resource instance data table.

[0064] The data status update module 208 is configured to mark the batch status as queryable after determining that the data to be written has been completed.

[0065] The data query module 210 is configured to: in response to a data query request, retrieve and return target data corresponding to the data query request based on the data write batch identifier and the metadata record.

[0066] In some optional embodiments, the data query module 210 further performs the following steps: From the metadata record, determine the write batch identifier of the target data whose batch status is queryable; Based on the write batch identifier, locate the resource instance data table that stores the target data; Based on the query conditions in the data query request and the field information in the metadata record, a query is performed in the resource instance data table, and the query results are returned.

[0067] For ease of description, the above devices are described in terms of function, divided into various modules. Of course, in implementing this application, the functions of each module can be implemented in one or more software and / or hardware.

[0068] The apparatus described above can be used to implement the corresponding dynamic data management method in any of the foregoing embodiments, and has the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0069] Based on the same inventive concept, corresponding to the methods of any of the above embodiments, this application also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the dynamic data management method described in any of the above embodiments.

[0070] Figure 4 This embodiment illustrates a more specific hardware structure of a computer device, which may include a processor 810, a memory 820, an input / output interface 830, a communication interface 840, and a bus 850. The processor 810, memory 820, input / output interface 830, and communication interface 840 are interconnected internally via the bus 850. The processor 810 can be implemented using a general-purpose CPU (Central Processing Unit), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the dynamic data management method provided in any of the foregoing embodiments of this specification.

[0071] The memory 820 can be implemented in the form of ROM (Read Only Memory), RAM (Random Access Memory), static storage device, dynamic storage device, etc. The memory 820 can store operating devices and other application programs. When the technical solutions provided in the embodiments of this specification are implemented by software or firmware, the relevant program code is stored in the memory 820 and is called and executed by the processor 810.

[0072] The input / output interface 830 is used to connect input / output modules to enable information input and output. Input / output modules can be configured as components within the device (not shown in the figure) or externally connected to the device to provide corresponding functions. Input devices may include keyboards, mice, touchscreens, microphones, various sensors, etc., while output devices may include displays, speakers, vibrators, indicator lights, etc.

[0073] The communication interface 840 is used to connect the communication module (not shown in the figure) to enable communication between this device and other devices. The communication module can communicate via wired means (such as USB, Ethernet cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.). Bus 850 includes a pathway for transmitting information between various components of the device, such as processor 810, memory 820, input / output interface 830, and communication interface 840.

[0074] It should be noted that although the above-described device only shows the processor 810, memory 820, input / output interface 830, communication interface 840, and bus 850, in specific implementations, the device may also include other components necessary for normal operation. Furthermore, those skilled in the art will understand that the above-described device may only include the components necessary for implementing the embodiments of this specification, and not necessarily all the components shown in the figures. The electronic devices described above are used to implement the corresponding dynamic data management methods in any of the foregoing embodiments, and have the beneficial effects of the corresponding method embodiments, which will not be repeated here. Based on the same inventive concept, corresponding to the methods of any of the above embodiments, this disclosure also provides a non-transitory computer-readable storage medium that stores computer instructions for causing the computer to execute the dynamic data management method as described in any of the above embodiments.

[0075] The computer-readable medium of this embodiment includes permanent and non-permanent, removable and non-removable media, and information storage can be implemented by any method or technology. Information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transfer medium that can be used to store information accessible by a computing device. The aforementioned non-transitory computer-readable storage media can be any available medium or data storage device that a computer can access, including but not limited to magnetic storage (e.g., floppy disks, hard disks, magnetic tapes, magneto-optical disks (MOs), etc.), optical storage (e.g., CDs, DVDs, BDs, HVDs, etc.), and semiconductor storage (e.g., ROMs, EPROMs, EEPROMs, non-volatile memory (NAND flash), solid-state drives (SSDs)). The computer instructions stored in the storage medium of the above embodiments are used to cause the computer to execute the dynamic data management method as described in any of the embodiments in the exemplary method section above, and have the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0076] Those skilled in the art should understand that the discussion of any of the above embodiments is merely exemplary and is not intended to imply that the scope of this application (including the claims) is limited to these examples; within the framework of this application, the technical features of the above embodiments or different embodiments can also be combined, the steps can be implemented in any order, and there are many other variations of different aspects of the embodiments of this application as described above, which are not provided in the details for the sake of brevity.

[0077] Additionally, to simplify the description and discussion, and to avoid obscuring the embodiments of this application, the well-known power / ground connections to integrated circuit (IC) chips and other components may or may not be shown in the provided drawings. Furthermore, the apparatus may be shown in block diagram form to avoid obscuring the embodiments of this application, and this also takes into account the fact that the details of the implementation of these block diagram apparatuses are highly dependent on the platform on which the embodiments of this application will be implemented (i.e., these details should be fully understood by those skilled in the art). While specific details (e.g., circuits) have been set forth to describe exemplary embodiments of this application, it will be apparent to those skilled in the art that the embodiments of this application can be implemented without these specific details or with variations thereof. Therefore, these descriptions should be considered illustrative rather than restrictive.

[0078] Although this application has been described in conjunction with specific embodiments thereof, many substitutions, modifications, and variations of these embodiments will be apparent to those skilled in the art from the foregoing description. For example, other memory architectures (e.g., dynamic RAM (DRAM)) may be used with the embodiments discussed.

[0079] The embodiments of this application are intended to cover all such substitutions, modifications, and variations that fall within the broad scope of the appended claims. Therefore, any omissions, modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the embodiments of this application should be included within the protection scope of this application.

Claims

1. A method for managing dynamic data, characterized in that, include: The system receives a request to create a dynamically structured table from a client, and creates a corresponding metadata record based on the request; wherein the metadata record includes resource type and initial field information. In response to receiving a data write request from the client, a data write batch identifier is allocated and returned to the client, and the batch status corresponding to the data write batch identifier is marked as unqueryable. In response to receiving the data to be written sent by the client, the system identifies the field information in the data to be written, compares the field information with the initial field information, and updates the metadata record based on the comparison result. In response to determining that the data to be written has been completed, the batch status is marked as queryable. In response to a data query request, based on the data write batch identifier and the metadata record, the target data corresponding to the data query request is retrieved and returned.

2. The dynamic data management method according to claim 1, characterized in that, The dynamic structure table includes a resource table, a resource field table, and a resource sharding table; The process of receiving a dynamic structure table creation request from the client and creating corresponding metadata records based on the creation request includes: Store the resource type, resource description, and batch quantity information from the creation request into the resource table; Store the initial field information from the creation request into the resource field table; Based on a preset number of shards, multiple resource instance data tables are created, and the correspondence between the resource type and each resource instance data table is stored in the resource shard table.

3. The dynamic data management method according to claim 1, characterized in that, The step of updating the metadata record based on the comparison result includes: In response to determining that the data to be written contains a new field not defined in the initial field information, the data type is determined according to the field value format characteristics of the new field; wherein, the data type includes string, numeric, and time types; Add the field name and data type of the new field to the metadata record.

4. The method for managing dynamic data according to claim 2, characterized in that, After updating the metadata record based on the comparison result, the method further includes: The entity records in the data to be written are converted into multiple attribute records; wherein each attribute record contains multiple typed value fields; Based on the data type in the metadata record, the attribute values ​​in the attribute record are stored in the corresponding typed value field to obtain the processed attribute record; The processed attribute records are stored in the resource instance data table.

5. The management method according to claim 4, characterized in that, The step of storing the processed attribute records into the resource instance data table includes: The target resource instance data table is determined from the plurality of resource instance data tables based on a load balancing strategy; wherein, the load balancing strategy includes a round-robin strategy, a hash modulo strategy, or a least-first strategy based on data volume; The processed attribute records are stored in the target resource instance data table.

6. The method for managing dynamic data according to claim 1, characterized in that, The step of retrieving and returning the target data corresponding to the data query request based on the data writing batch identifier and the metadata record includes: From the metadata record, determine the write batch identifier of the target data whose batch status is queryable; Based on the write batch identifier, locate the resource instance data table that stores the target data; Based on the query conditions in the data query request and the field information in the metadata record, a query is performed in the resource instance data table, and the query results are returned.

7. A dynamic data management system, characterized in that, include: The metadata record creation module is configured to: receive a creation request for a dynamically structured table sent by a client, and create a corresponding metadata record based on the creation request; wherein, the metadata record includes resource type and initial field information; The write request receiving module is configured to: in response to receiving a data write request sent by the client, allocate and return a data write batch identifier to the client, and mark the batch status corresponding to the data write batch identifier as unqueryable; The metadata record update module is configured to: in response to receiving data to be written sent by the client, identify field information in the data to be written, compare the field information with the initial field information, and update the metadata record according to the comparison result; The data status update module is configured to: mark the batch status as queryable after determining that the data to be written has been completed; The data query module is configured to: in response to a data query request, retrieve and return the target data corresponding to the data query request based on the data write batch identifier and the metadata record.

8. A computer device, characterized in that, It includes one or more processors, memory; and one or more programs, wherein the one or more programs are stored in the memory and executed by the one or more processors, and the one or more programs include instructions for performing the method of any one of claims 1 to 6.

9. A non-volatile computer-readable storage medium containing a computer program, characterized in that, When the computer program is executed by one or more processors, the one or more processors perform the method according to any one of claims 1 to 6.

10. A computer program product, characterized in that, It includes one or more computer programs that, when executed by one or more processors, implement the method as described in any one of claims 1 to 6.

Images