A method for inserting semi-structured data into a database and related products

Abstract

This invention provides a method and apparatus for inserting semi-structured data into a database, comprising: acquiring an insertion request; parsing the semi-structured data to be processed; determining whether the semi-structured data to be processed contains a predefined identifier field; if it contains a predefined identifier field, using the field value as a target identifier; if it does not contain a predefined identifier field, generating a target identifier; and performing an insertion operation of the target identifier and the semi-structured data to be processed based on pre-set index rules. The solution of this invention has dual logic of identifier fields and the generation of unique identifiers, ensuring that each piece of semi-structured data to be processed corresponds to a unique target identifier. Combined with pre-set index rules, it can prevent the writing of duplicate identifiers from the insertion stage. Performing the insertion operation based on preset index rules avoids the cumbersome process of additional full data traversal verification required when inserting semi-structured data.

Description

Technical Field

[0001] This invention relates to database technology, and in particular to a method for inserting semi-structured data into a database and related products. Background Technology

[0002] Semi-structured data is a flexible data format that does not require a predefined fixed structure. Data fields can be flexibly adjusted according to actual business needs, thus adapting to diverse data storage requirements. However, this lack of a fixed structure contradicts the fundamental requirements of "unique identification and consistency verification" in data management, resulting in poor compatibility between semi-structured data and data storage systems. Summary of the Invention

[0003] One objective of this invention is to improve the overall performance of semi-structured data in adapting to diverse data storage needs.

[0004] A further objective of this invention is to improve the compatibility of semi-structured data with data storage systems.

[0005] Specifically, according to one aspect of the present invention, the present invention provides a method for inserting semi-structured data into a database, comprising: In response to receiving an insertion request for semi-structured data to be processed, the semi-structured data to be processed is parsed, and it is determined whether the semi-structured data to be processed contains a predefined identifier field. If the semi-structured data to be processed contains the predefined identifier field, the value of the identifier field shall be used as the target identifier. If the semi-structured data to be processed does not contain the predefined identifier field, generate a unique identifier associated with the semi-structured data to be processed, and use the unique identifier as the target identifier. The insertion operation of the target identifier and the semi-structured data to be processed is performed based on the pre-set indexing rules.

[0006] Optionally, the structure of the predefined identifier field is a nested path expression; Furthermore, the step of parsing the semi-structured data to be processed includes: traversing the nested path expression to parse the internal hierarchy of the semi-structured data to be processed.

[0007] Optionally, the response prior to receiving the insertion request for the semi-structured data to be processed further includes: Create a data table containing a first storage structure and a second storage structure, wherein the first storage structure is used to store semi-structured data, and the second storage structure is used to store the target identifier of the semi-structured data, wherein the second storage structure has a unique index built based on the target identifier.

[0008] Optionally, the method for generating a unique identifier associated with the semi-structured data to be processed includes: One or more of the following generation methods can be used: sequence-based generation, random number-based generation, timestamp-based generation, and generation based on a universally unique identifier algorithm.

[0009] Optionally, generating a unique identifier associated with the semi-structured data to be processed includes: Determine the hash value based on part or all of the content of the semi-structured data to be processed; The hash value is used as the unique identifier.

[0010] Optionally, the insertion operation of the target identifier and the semi-structured data to be processed is performed based on a pre-set indexing rule, including: The global uniqueness of the target identifier is verified across multiple database nodes using a consensus mechanism; In response to the fact that the target identifier is globally unique, an insertion operation is performed on the target identifier and the semi-structured data to be processed; In response to the fact that the target identifier is not globally unique, a rollback operation is performed.

[0011] Optionally, the insertion operation of the target identifier and the semi-structured data to be processed is performed based on a pre-set indexing rule, including: The validity of the target identifier is determined by the predefined time interval to which the generation time belongs and the physical database partition.

[0012] Optionally, the first storage structure is used to store semi-structured data, and the second storage structure is used to store the target identifier of the semi-structured data, including: A copy-on-write mechanism is configured when storing semi-structured data and the target identifier of the semi-structured data; The copy-on-write mechanism includes: writing the target identifier and the semi-structured data to be processed into a temporary storage structure; after verifying that the target identifier meets the pre-set indexing rules, submitting the semi-structured data to the first storage structure; and submitting the target identifier to the second storage structure.

[0013] Optionally, the insertion operation of the target identifier and the semi-structured data to be processed is performed based on a pre-set indexing rule, including: If the insertion operation of the semi-structured data to be processed fails and the reason for the failure is that the target identifier conflicts with an existing historical target identifier, the target identifier is regenerated through a preset identifier generation strategy. The preset identifier generation strategy can be any one of the following: switching algorithm, injection of random factors, or Snowflake algorithm.

[0014] According to another aspect of the invention, a computer program product is also provided, comprising a computer program that, when executed by a processor, implements the steps of any of the above-described methods for inserting semi-structured data into a database.

[0015] According to another aspect of the present invention, a computer device is also provided, including a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the method for inserting semi-structured data into a database as described above.

[0016] The method for inserting semi-structured data into a database according to this invention ensures that each piece of semi-structured data corresponds to a unique target identifier through a dual logic of extracting existing identifier fields and generating exclusive identifiers. Combined with pre-set index rules, it can prevent the writing of duplicate identifiers during the insertion process, providing a precise and unique identification basis for subsequent data querying, updating, deduplication, and other operations, thereby improving the accuracy of data management. Executing the insertion operation based on preset index rules avoids the cumbersome process of performing full data traversal verification required for existing semi-structured data insertion. The index mechanism quickly completes the verification of the legality and uniqueness of the target identifier, shortening the data insertion time. Especially in large-scale semi-structured data storage scenarios, it can significantly reduce operation latency and improve data processing throughput. The target identifier forms a strong association with the semi-structured data to be processed. Whether the identifier is user-defined or system-generated, it uniquely points to the corresponding semi-structured data. The target data can be directly located through this identifier without parsing the complete semi-structured data content, greatly improving data retrieval and traceability efficiency and providing convenient support for data lifecycle management.

[0017] Furthermore, the method for inserting semi-structured data into a database according to the present invention achieves unified verification among multiple database nodes through a consensus mechanism, thereby avoiding problems such as dirty data generation and data association errors caused by duplicate identifiers at the data layer, and improving the accuracy and consistency of cross-node data storage.

[0018] Furthermore, the method for inserting semi-structured data into a database according to the present invention establishes a "verification-execution / rollback" insertion process, performs differentiated operations on the global uniqueness verification results of identifiers, avoids the continuous execution of invalid insertion operations, reduces the waste of computing resources, storage resources and network interaction resources of database nodes, and improves the data writing efficiency of multi-node database systems.

[0019] The above and other objects, advantages and features of the present invention will become more apparent to those skilled in the art from the following detailed description of specific embodiments of the invention in conjunction with the accompanying drawings. Attached Figure Description

[0020] The following sections will describe some specific embodiments of the invention in detail by way of example and not limitation, with reference to the accompanying drawings. The same reference numerals in the drawings denote the same or similar parts or portions. Those skilled in the art should understand that these drawings are not necessarily drawn to scale. In the drawings: Figure 1 This is a flowchart illustrating a method for inserting semi-structured data into a database according to an embodiment of the present invention. Figure 2 This is a flowchart illustrating another method for inserting semi-structured data into a database, as provided in an embodiment of the present invention. Figure 3 This is a schematic diagram of the table structure of the data table provided according to an embodiment of the present invention; Figure 4 This is a flowchart illustrating the insertion operation provided according to an embodiment of the present invention; Figure 5 This is a timing diagram of a concurrent insertion operation provided according to an embodiment of the present invention; Figure 6 This is a schematic diagram of a computer program product according to an embodiment of the present invention; Figure 7 This is a schematic diagram of a computer-readable storage medium according to an embodiment of the present invention; Figure 8 This is a schematic diagram of the structure of a computer device according to an embodiment of the present invention. Detailed Implementation

[0021] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with those of the present invention. Rather, they are merely examples of methods consistent with some aspects of the embodiments of the present invention as detailed in the appended claims.

[0022] Semi-structured data, with its flexible representation characteristics, has become a key format for adapting to the diverse data storage needs of today. This type of data does not require a predefined fixed structural framework. It does not need to follow the strict schema constraints of structured data, nor does it need to determine the field types, quantities, and relationships at the initial stage of data generation or storage. Instead, it can flexibly adjust the addition or removal of data fields, the expansion of dimensions, or the change of field attributes according to the dynamic changes in the actual business scenario.

[0023] In e-commerce scenarios, descriptions of different product categories can be customized with dedicated fields (such as parameter configurations for electronic products, size and color information for clothing, and shelf-life information for food), eliminating the need for separate storage structures for each product category. In IoT data acquisition scenarios, sensor data from different devices (such as temperature, humidity, and operating parameters) can be flexibly integrated into the same dataset, without being limited by uniform field definitions. This flexibility gives semi-structured data an irreplaceable advantage in scenarios involving frequent data structure iterations, multi-source heterogeneous data fusion, and personalized business data storage.

[0024] However, this lack of a fixed structure contradicts the fundamental requirements of "unique identification and consistency verification" in data management. This contradiction directly leads to relatively poor compatibility between semi-structured data and existing data storage systems, severely restricting the reliable application and large-scale deployment of semi-structured data.

[0025] In some scenarios, a core prerequisite for data management is establishing a globally unique identifier for each piece of data. This identifier enables precise data querying, updating, deletion, deduplication, and other critical operations, ensuring the traceability and uniqueness of data throughout its entire lifecycle. However, the unstructured nature of semi-structured data makes this fundamental requirement difficult to meet using existing methods. Existing data storage systems rely heavily on fixed field constraints (such as primary key fields in structured databases) for unique identification mechanisms. However, semi-structured data does not have pre-defined fixed fields and cannot directly reuse such mechanisms. If users are allowed to define their own identifier fields, the lack of unified structural constraints may lead to inconsistent field naming across different data sources (e.g., identifiers with the same meaning may be named "id", "uid", "_id", etc.) and inconsistent field formats (e.g., some are string types and some are numeric types), resulting in the inability to form a globally unified mapping relationship between identifiers. If the data storage system forcibly generates identifiers, it may conflict with the user's defined identifier requirements, disrupting the original logic of the data. At the same time, the dynamic association problem between identifiers and semi-structured data still needs to be solved.

[0026] In some scenarios, from the perspective of consistency verification, the reliability of data storage depends on the effective verification of data integrity, uniqueness, and validity. Existing data storage systems can implement verification logic through pre-defined field type constraints, unique indexes, foreign key associations, and other mechanisms. However, the flexibility of semi-structured data makes it difficult to directly adapt these verification mechanisms. Since semi-structured data does not have a fixed identifier field, the unique index mechanism based on fixed fields in existing data storage systems fails. If the full data traversal verification method is adopted, the verification time complexity increases linearly with the increase of data volume, resulting in a sharp decrease in data insertion and update efficiency, which cannot meet the needs of large-scale data scenarios. Semi-structured data fields can be added or removed flexibly, different data may lack key related information, and the format of field values ​​(such as date, number, string) has no uniform constraints. The verification rules of existing data storage systems are difficult to cover all dynamically changing scenarios, which can easily lead to problems such as chaotic data format and missing key information, affecting data consistency. The lack of an atomic "verification-operation" coordination mechanism can easily lead to problems such as identifier conflicts and duplicate data writing when multiple entities operate simultaneously. Furthermore, the concurrency control mechanisms of existing data storage systems are mostly designed for structured data and cannot effectively solve the dynamic conflict problem of semi-structured data.

[0027] The following describes, with reference to the accompanying drawings, a method for inserting semi-structured data into a database and related products according to embodiments of the present invention.

[0028] Figure 1 This is a flowchart illustrating a method for inserting semi-structured data into a database, as provided in an embodiment of the present invention.

[0029] like Figure 1 As shown, the method for inserting semi-structured data into the database includes, but is not limited to, the following steps: S101, in response to receiving an insertion request for the semi-structured data to be processed, parse the semi-structured data to be processed and determine whether the semi-structured data to be processed contains a predefined identifier field.

[0030] S102, If the semi-structured data to be processed contains a predefined identifier field, use the value of the identifier field as the target identifier.

[0031] S103. If the semi-structured data to be processed does not contain a predefined identifier field, generate a unique identifier associated with the semi-structured data to be processed, and use the unique identifier as the target identifier.

[0032] S103, perform the insertion operation of the target identifier and the semi-structured data to be processed based on the pre-set indexing rules.

[0033] In the implementation of step S101, an externally initiated request for inserting semi-structured data to be processed is received through a preset interface (such as an HTTP / HTTPS interface, a database native interface, a message queue subscription interface, etc.). The request carries information such as the complete data body of the semi-structured data to be processed, the request identifier, and the data source identifier.

[0034] In the implementation of step S101, the data storage system performs a validity check on the request, including checking whether the request format conforms to preset specifications (e.g., completeness of request header fields, correctness of data body transmission encoding format), whether the data source has legitimate access permissions, and whether the request frequency is within a preset threshold range. If the check is successful, the subsequent processing flow is executed; if the check fails, the corresponding error message (e.g., insufficient permissions, incorrect request format) is returned and the processing is terminated.

[0035] In the implementation of step S101, a parsing tool corresponding to the request (such as a JSON parser, XML parser, user-defined semi-structured data parsing component, etc.) is used to parse the valid insertion request, extract the semi-structured data body to be processed, and remove non-data information such as request headers and request parameters. Simultaneously, metadata information such as request reception time, request ID, and data source is recorded for subsequent data traceability and anomaly investigation.

[0036] In some scenarios, for large-volume semi-structured data (i.e., the data volume exceeds the preset capacity), streaming parsing can be used to extract the data body, avoiding the problem of excessive memory consumption caused by loading all at once.

[0037] In the implementation of step S101, a corresponding parsing strategy is matched according to the format type of the semi-structured data to be processed.

[0038] In some scenarios, if the semi-structured data to be processed is in JSON format, the JSON parsing component is called to traverse the key-value pair structure of the semi-structured data to be processed, and parse the top-level field and nested fields layer by layer to form a mapping relationship between field paths and field values.

[0039] In some scenarios, if the semi-structured data to be processed is in XML format, the node structure can be traversed using DOM / SAX parsing to extract all element node names and corresponding text values, and the hierarchical relationship of the nodes can be recorded.

[0040] In some scenarios, if the semi-structured data to be processed is a set of key-value pairs or data in a custom delimiter format, the data is split according to the preset splitting rules to obtain a set of corresponding field names and field values.

[0041] It should be noted that during the parsing process, the integrity of the semi-structured data to be processed is simultaneously verified to check for issues such as syntax errors, null pointer exceptions, and missing business data fragments. If an integrity exception is detected, the parsing process is paused, a data format error message is returned, and the error location and type are logged.

[0042] In the implementation of step S101, based on the field traversal list generated after parsing, the traversal order of top-level fields first and then nested fields is adopted to determine whether the semi-structured data to be processed contains a predefined identifier field.

[0043] Step S101 of the method for inserting semi-structured data into a database provided in this embodiment of the invention actively parses and judges the received original semi-structured data to be processed, which can identify and process data that does not meet the preset format requirements at an early stage, and avoid invalid or erroneous data from entering the subsequent processing flow.

[0044] In the implementation of step S102, the field position that matches the predefined identifier field in the semi-structured data to be processed is located based on the judgment result details (including the matching field path and field name) generated in the "identifier field existence judgment" step in step S101.

[0045] In some scenarios, if the matched field is a top-level field (such as the top-level id field in JSON data or the uid element under the root node of XML data), the corresponding value is extracted by the field name, and the extracted value is used as the target identifier.

[0046] In some scenarios, if the matched field is a nested field (such as info.user._id in JSON data, or nested fields in XML data), it may be necessary to perform a more precise matching operation. <info> <user> <uid> 123< / uid> < / user> < / info> If the record is located, the data structure is traversed layer by layer according to the field path of the record, the nested field is located and the value is extracted, and the extracted value is used as the target identifier.

[0047] In some scenarios, if the semi-structured data to be processed contains multiple fields that match a predefined identifier field (e.g., both 'id' and '_id' are included in the predefined configuration), extraction is performed according to the predefined field priority rules. For example, sorting by field name or by field hierarchy: top-level fields have higher priority than nested fields. In some scenarios, if the fields matching the predefined identifier field do not have a configured priority, the first matching field value is extracted by default.

[0048] In the implementation of step S103, if it is determined that the semi-structured data to be processed does not contain a predefined identifier field, the exclusive identifier generation process is triggered according to the pre-configured exclusive identifier generation rules, and the generation operation is bound to the context of the current insertion request (such as request ID, data source, receiving time, data feature hash value, etc.) to ensure the unique association between the exclusive identifier and the semi-structured data to be processed.

[0049] In some scenarios, pre-configured exclusive identifier generation rules include, but are not limited to, UUID / UUIDv4 rules, Snowflake algorithm rules, and custom combination rules (such as timestamp + random string + data source identifier).

[0050] For example, the UUID generation component is called to generate a UUID string based on a random number algorithm, and then a unique identifier associated with the semi-structured data to be processed is obtained from the UUID string.

[0051] As an example, the snowflake algorithm generator generates an integer ID based on the current timestamp, while simultaneously checking for time rollback issues (if server time rollback is detected, generation is paused and time synchronization is waited for, or a backup sequence number is enabled), thereby obtaining a unique identifier associated with the semi-structured data to be processed.

[0052] For example, dimensions are generated by splitting according to the template corresponding to the custom combination rules: extracting the current timestamp, generating a random string of a specified length, and obtaining the data source identifier. A unique identifier is then generated by concatenating the template to associate the semi-structured data to be processed.

[0053] Steps S102 and S103 of the method for inserting semi-structured data into a database provided in this embodiment of the invention ensure that each piece of semi-structured data to be processed corresponds to a unique target identifier through the dual logic of extracting existing identifier fields and generating exclusive identifiers. Combined with pre-set index rules, the writing of duplicate identifiers can be blocked from the insertion stage, providing an accurate and unique identification basis for subsequent data query, update, deduplication and other operations, thereby improving the accuracy of data management.

[0054] In the implementation of step S104, the pre-set indexing rules include: index type, index associated fields, index activation rules, and distributed scenario extension configuration.

[0055] In some scenarios, index types include uniqueness constraint indexes (such as UNIQUE indexes in relational databases and unique indexes in MongoDB), B+ tree indexes (suitable for range queries), and hash indexes (suitable for equal interpolation).

[0056] In some scenarios, the associated field of the index is used to explicitly bind the index only to the "target identifier storage column" and to the semi-structured data storage column, thereby avoiding the performance loss caused by the index covering the semi-structured data column.

[0057] In some scenarios, index activation rules include configuring index verification timing (real-time verification during insertion operations), conflict handling strategies (directly blocking insertion / triggering retry), and index update mechanisms (automatically synchronizing index data after successful insertion).

[0058] In some scenarios, extended configuration for distributed scenarios refers to: if deployed on a distributed database, additional distributed index synchronization rules are configured (such as cross-node index consistency verification cycle, master node index priority principle).

[0059] In the implementation of step S104, the target identifier and the semi-structured data to be processed are mapped to preset independent storage columns, and the mapped two columns of data are encapsulated into a structured request packet conforming to the database insertion protocol. When performing the insertion operation, the pre-configured index rules are triggered, including: reading the uniqueness constraint index of the target identifier column and checking whether the current target identifier already exists in the index library; in a distributed scenario, the local node index is first verified, and then other nodes are verified through the cross-node index synchronization interface to ensure global uniqueness; the verification process relies on the index structure to achieve efficient retrieval at the O(log n) level, avoiding full table scans.

[0060] In the implementation of step S104, the entire process of "index verification - data insertion - index update" is encapsulated as an atomic transaction to ensure that all three either succeed or are rolled back.

[0061] It should be noted that if problems such as database connection interruption, index synchronization failure, or data write exception occur during the insertion process, a transaction rollback will be triggered immediately, including: undoing some of the data that has been written (such as only writing the identifier column but not the semi-structured data column); restoring the index to its state before the operation to avoid inconsistency between the index and the data; recording the exception log (including the exception type, failure stage, and target identifier value), and returning an "insertion failed" message.

[0062] In the implementation of step S104, if the insertion is successful, the consistency check between the index and the data is performed asynchronously, the actual value of the target identifier column is queried and compared with the identifier stored in the index database to ensure that the index is synchronized correctly; if an inconsistency is found, the index repair process (resynchronizing the index data) is triggered and recorded in the index audit log.

[0063] In the implementation of step S104, if the consistency check is successful, the content of the semi-structured data column is read to check whether the data is complete (such as whether the format is valid and whether the data size is consistent with that before insertion). If there is data corruption, the data is marked as abnormal and the operation and maintenance personnel are notified through a preset alarm mechanism.

[0064] In some scenarios, for high-frequency insertion scenarios, multi-level caching (e.g., local cache + distributed cache) is set up for the index of the target identifier. Newly inserted identifiers are first written to the cache and then synchronized to the persistent index, thereby reducing the pressure of index retrieval.

[0065] In some scenarios, if a batch of semi-structured data insertion requests are received, the process is executed according to the logic of "batch index verification - batch data insertion - batch index update" to reduce the number of index verifications and improve insertion efficiency.

[0066] In some scenarios, in a read-write separation architecture, insert operations only perform index verification and updates on the master database, while the slave database obtains index data through a master-slave synchronization mechanism, thus avoiding performance bottlenecks caused by slave database index verification.

[0067] Step S104 of the method for inserting semi-structured data into a database provided in this embodiment of the invention performs the insertion operation based on preset index rules. This avoids the cumbersome process of performing full data traversal verification required when inserting semi-structured data in existing systems. The index mechanism quickly verifies the legality and uniqueness of the target identifier, shortening the data insertion time. Especially in large-scale semi-structured data storage scenarios, it can significantly reduce operation latency and improve data processing throughput. The target identifier forms a strong association with the semi-structured data to be processed. Whether the identifier is user-defined or system-generated, it uniquely points to the corresponding semi-structured data. The target data can be directly located using this identifier without parsing the complete semi-structured data content, greatly improving data retrieval and tracing efficiency and providing convenient support for data lifecycle management. No modification to the original structure of the semi-structured data is required; data insertion management is achieved solely through identifier extraction / generation and index rule adaptation, ensuring seamless compatibility with existing semi-structured data storage systems. Furthermore, the predefined identifier fields can be flexibly configured, and the index rules can be adjusted as needed, adapting to the storage requirements of different types of semi-structured data and reducing the cost of system upgrades and business expansion.

[0068] Figure 2 This is a flowchart illustrating another method for inserting semi-structured data into a database according to an embodiment of the present invention. Figure 3 This is a schematic diagram of the table structure of the data table provided according to an embodiment of the present invention. Figure 4 This is a flowchart illustrating the insertion operation provided according to an embodiment of the present invention.

[0069] In some embodiments of the present invention, such as Figure 2 As shown, the method for inserting semi-structured data into the database includes, but is not limited to, the following steps: S201, Create a data table containing a first storage structure and a second storage structure. The first storage structure is used to store semi-structured data, and the second storage structure is used to store the target identifier of the semi-structured data. The second storage structure has a unique index built based on the target identifier.

[0070] S202, in response to receiving an insertion request for the semi-structured data to be processed, parse the semi-structured data to be processed and determine whether the semi-structured data to be processed contains a predefined identifier field.

[0071] S203, If the semi-structured data to be processed contains a predefined identifier field, use the value of the identifier field as the target identifier.

[0072] S204. If the semi-structured data to be processed does not contain a predefined identifier field, generate a unique identifier associated with the semi-structured data to be processed, and use the unique identifier as the target identifier.

[0073] S205, based on the pre-set indexing rules, perform the insertion operation of the target identifier and the semi-structured data to be processed.

[0074] In the implementation of step S201, the basic attributes of the data table are predefined, including the table name, storage engine, character encoding, and data lifecycle rules. If the database is a relational database, InnoDB / PostgreSQL is selected as the storage engine; if the database is NoSQL, MongoDB is selected as the storage engine. The data lifecycle rules include data retention duration and archiving strategy.

[0075] In the implementation of step S201, the data table adopts an architecture of "multi-storage structure separation + association binding". The multi-storage structure includes a first storage structure used as a semi-structured data storage layer and a second storage structure used as a target identifier storage layer. The two layers are physically separated but logically strongly associated, ensuring that the target identifier corresponds one-to-one with the semi-structured data.

[0076] In some scenarios, semi-structured data includes JSON, XML, key-value pair collections, etc.; the first storage structure does not have a fixed field structure and supports dynamic addition or deletion of data fields and adjustment of nesting levels, thus adapting to the storage needs of diverse business data.

[0077] In some scenarios, the target identifier serves as the retrieval anchor for semi-structured data, independent of the semi-structured data storage, facilitating uniqueness verification and data location.

[0078] In the implementation of step S201, the first storage structure is used to store semi-structured data, and the second storage structure is used to store the target identifier of the semi-structured data, including: Configure a copy-on-write mechanism when storing semi-structured data and the target identifier of the semi-structured data; The copy-on-write mechanism includes: writing the target identifier and the semi-structured data to be processed into a temporary storage structure; after verifying that the target identifier meets the pre-set indexing rules, submitting the semi-structured data to the first storage structure; and submitting the target identifier to the second storage structure.

[0079] In some scenarios, copy-on-write mechanisms are implemented at the database engine's transaction processing layer. Temporary storage structures are temporary transaction areas independent of primary storage, supporting fast writes and transaction isolation. Their forms include, but are not limited to: memory-mapped files, transaction log buffers, temporary tablespaces, and undo / rollback segments.

[0080] For example, when an insertion begins, a unique transaction ID and a corresponding temporary storage structure are assigned to the insertion. The semi-structured data to be processed and the target identifier are written to the temporary storage structure, forming a transaction snapshot. A mapping relationship between this transaction snapshot and the transaction ID is established in the temporary storage structure, and the status is marked as "pending verification".

[0081] For example, the process involves reading the target identifier from the temporary storage structure and performing verification according to pre-set indexing rules, including: checking for conflicts in the secondary storage structure index of the main storage; triggering cross-node consensus checks if distributed uniqueness verification is configured; and verifying the spatiotemporal validity of the identifier if spatiotemporal constraints are configured. The process also includes reading the semi-structured data to be processed from the temporary storage structure, including performing syntax checks and size limit checks.

[0082] For example, if all validations pass, the target identifier is inserted / updated from the transaction snapshot to the second storage structure of the main storage, and the semi-structured data to be processed is inserted / updated from the transaction snapshot to the first storage structure of the main storage. The target identifier is atomically set to the "occupied" state in the index of the second storage structure. If any validation fails, an atomic rollback operation is performed, including: releasing all resources occupied by the transaction in temporary storage, and returning the specific error code and conflict information.

[0083] In the implementation of step S201, a dedicated unique index is created for the second storage structure to ensure that the identifier is globally unique. If the database is a relational database, UNIQUE is used as the unique index, and the underlying structure is a B+ tree; if the database is NoSQL, unique: true is used as the unique index and bound to the field of the second storage structure.

[0084] In some embodiments of the present invention, such as Figure 3 As shown, the data table stores the semi-structured data to be processed into the first storage structure and the target identifier into the second storage structure through pre-set indexing rules for identification processing.

[0085] Step S201 of the method for inserting semi-structured data into a database provided in this embodiment of the invention separates the semi-structured data body and its target identifier into different storage structures, making the data model design clearer and more flexible. The first storage structure can focus on efficiently storing and retrieving unstructured data content, while the second storage structure serves as a lightweight index layer, specifically responsible for global target identifier management and uniqueness assurance. This decoupling facilitates independent optimization, expansion, and maintenance of the two parts.

[0086] In the implementation of step S202, the structure of the predefined identifier field is a nested path expression; and the step of parsing the semi-structured data to be processed includes: traversing the nested path expression to parse the internal hierarchy of the semi-structured data to be processed.

[0087] In some scenarios, if the semi-structured data is in JSON format, a "dot separator" is used to indicate nested levels, such as info.user._id representing the _id field in the user sub-object under the info object in the JSON data; and array index positioning is supported, such as items[0].order_id representing the order_id field in the first element of the items array.

[0088] In some scenarios, if the semi-structured data is in XML format, the " / " separator is used to indicate node nesting, such as info / user / uid representing nested nodes in the XML data. <info> <user> <uid> ...< / uid> < / user> < / info> The uid node under the path.

[0089] In some scenarios, if the semi-structured data is user-defined data, it can be nested in a hierarchical manner using "case-sensitive / case-insensitive + wildcard".

[0090] For example, in JSON format, if the expression is info.user._id, it is split into ["info", "user", "_id"] and traversed according to the elements.

[0091] For example, in XML format, if the expression is info / user / uid, it is split into ["info", "user", "uid"] and traversed according to the elements.

[0092] For example, for user-defined data, if the expression is items[0].order_id, it is split into ["items", 0, "order_id"] and traversed according to the elements.

[0093] Step S202 of the method for inserting semi-structured data into a database provided in this embodiment of the invention actively parses and judges the received original semi-structured data to be processed, which can identify and process data that does not meet the preset format requirements at an early stage, and avoid invalid or erroneous data from entering the subsequent processing flow.

[0094] For a further detailed description of the implementation method of step S203, please refer to the relevant content in the above embodiments, which will not be repeated here.

[0095] In the implementation of step S204, the method of generating a unique identifier associated with the semi-structured data to be processed includes one or more of the following: sequence-based generation, random number-based generation, timestamp-based generation, and generation based on a universally unique identification code algorithm.

[0096] In some scenarios, a sequence generator is used to implement a sequence generation method, including: reading the starting value and step size configured in the sequence generator; allocating node-specific sequence segments through a distributed lock in a distributed scenario; obtaining the next incrementing sequence value from the sequence generator each time a sequence is generated; and converting the sequence value into a unique identifier in a preset format.

[0097] In some scenarios, a random number generator is used to implement a random number generation method, including: specifying the random number length and character set; generating a random string; and obtaining a unique identifier after deduplication and verification of the string.

[0098] In some scenarios, the timestamp-based generation method includes: obtaining the timestamp; if the current time is "2026-01-01 15:30:20", its timestamp is 1740703820123456; converting the timestamp format from decimal to hexadecimal to obtain "67e8a9b2c3d4e5f6"; to prevent duplication, appending 1-3 random numbers (such as "67e8a9b2c3d4e5f68") to the timestamp to obtain a unique identifier.

[0099] In some scenarios, a universally unique identifier algorithm component is used to generate a standard 36-bit universally unique identifier, such as "98e2bd40-fb2a-11f0-9d44-c3baeb3e6323"; after removing the separator, the result is "98e2bd40fb2a11f09d44c3baeb3e6323"; the universally unique identifier with the separator removed is used as the exclusive identifier.

[0100] In another embodiment of step S204, generating a unique identifier associated with the semi-structured data to be processed includes: determining a hash value based on part or all of the content of the semi-structured data to be processed; and using the hash value as the unique identifier.

[0101] In some scenarios, if the size of the semi-structured data to be processed is small (e.g., less than 1MB) and the content is highly unique, a one-way hash operation is performed on all the content of the semi-structured data to be processed to obtain a hash value.

[0102] In some scenarios, if the size of the semi-structured data to be processed is large (e.g., greater than 1MB) and some data can be used to identify data uniqueness, a one-way hash operation is performed on part of the semi-structured data to obtain a hash value.

[0103] Steps S203 and S204 of the method for inserting semi-structured data into a database provided in this embodiment of the invention ensure that each piece of semi-structured data to be processed corresponds to a unique target identifier through the dual logic of extracting existing identifier fields and generating exclusive identifiers. Combined with pre-set index rules, the writing of duplicate identifiers can be blocked from the insertion stage, providing an accurate and unique identification basis for subsequent data query, update, deduplication and other operations, thereby improving the accuracy of data management.

[0104] In some embodiments of the present invention, such as Figure 4 As shown, the insertion operation of the target identifier and the semi-structured data to be processed is performed based on the pre-set indexing rules, including: S2051 verifies the global uniqueness of the target identifier across multiple database nodes through a consensus mechanism; S2052, in response to the global uniqueness of the target identifier, performs the insertion operation of the target identifier and the semi-structured data to be processed; S2052, in response to the target identifier not being globally unique, performs a rollback operation.

[0105] In the scenario of step S2051, multiple database nodes form a distributed architecture, and distributed uniqueness verification is implemented in a cluster environment containing multiple database nodes. Each database node contains a local first storage structure and a second storage structure. A coordination service component is set up in the cluster to manage the distributed consensus process. Each node communicates and synchronizes its state through the coordination service component.

[0106] In the scenario of step S2051, the consensus mechanism includes, but is not limited to, majority rule, quorum, and timestamp-based sequence exchange. A majority rule means that global uniqueness is determined when more than half of the available nodes return a "grantable" verification result. A quorum means that all nodes from a specified sub-cluster (such as a node group containing primary replicas) are required to return a "grantable" verification result. A timestamp-based sequence exchange protocol combines logical clocks or physical timestamps to ensure that requests across nodes have a global order.

[0107] In the scenarios described in steps S2052 and S2053, under the consensus mechanism, the global uniqueness of the target identifier across multiple database nodes is confirmed, and the final insertion operation is performed. This insertion operation stores the semi-structured data to be processed and the target identifier as a complete record in the database. If the target identifier does not possess global uniqueness, or if the insertion operation fails again, a rollback operation is performed.

[0108] The method for inserting semi-structured data into a database provided in the embodiments of the invention achieves unified verification among multiple database nodes through a consensus mechanism, thereby avoiding problems such as dirty data generation and data association errors caused by duplicate identifiers at the data layer, and improving the accuracy and consistency of cross-node data storage.

[0109] The method for inserting semi-structured data into a database provided in the embodiments of the invention establishes a "verification-execution / rollback" insertion process, performs differentiated operations on the global uniqueness verification results of identifiers, avoids the continuous execution of invalid insertion operations, reduces the waste of computing resources, storage resources and network interaction resources of database nodes, and improves the data writing efficiency of multi-node database systems.

[0110] In the implementation of step S205, the insertion operation of the target identifier and the semi-structured data to be processed is performed based on the pre-set index rules, including: the validity of the target identifier is determined by the predefined time interval to which the generation time belongs and the physical database partition.

[0111] In some scenarios, pre-defined indexing rules follow spatiotemporal constraints, meaning the generation, verification, and validity of target identifiers are bound to both time and spatial dimensions. The time dimension uses a predefined time interval as the smallest scheduling unit. This time interval can be a fixed duration (e.g., 10 minutes, 1 hour) or a logical period (e.g., daily batches, transaction log sequence intervals). The spatial dimension uses physical database partitions as the smallest storage structure. Partitions can be divided according to strategies such as hash values, physical ranges, or list items, and each partition has a unique partition code.

[0112] In the implementation of step S205, the insertion operation of the target identifier and the semi-structured data to be processed is performed based on the preset indexing rules, including: If the insertion operation of the semi-structured data to be processed fails and the reason for the failure is that the target identifier conflicts with the existing historical target identifier, the target identifier is regenerated through the preset identifier generation strategy. The preset identifier generation strategy is any one of the following: switching algorithm, injection of random factors, or Snowflake algorithm.

[0113] In some scenarios, when an insertion operation fails due to the target identifier violating the unique index constraint, a conflict task unit is constructed by capturing the original data, the conflict identifier, and the conflict context. Based on the conflict type and preset priority (such as business type or data urgency), the semi-structured data containing the task unit is stored in a processing queue, and the target identifier is regenerated using a preset identifier generation strategy. This preset identifier generation strategy includes any one of the following: switching algorithm, injecting a random factor, or the Snowflake algorithm.

[0114] It should be noted that the method for inserting semi-structured data into the database also includes rollback mechanisms, operations after successful insertion, high-frequency insertion, and batch insertion of semi-structured data. These contents can be found in the relevant descriptions in the above embodiments, and will not be repeated here.

[0115] Step S205 of the method for inserting semi-structured data into a database provided in this embodiment of the invention performs the insertion operation based on preset index rules. This avoids the cumbersome process of performing full data traversal verification required for inserting semi-structured data in existing systems. The index mechanism quickly verifies the legality and uniqueness of the target identifier, shortening the data insertion time. Especially in large-scale semi-structured data storage scenarios, it significantly reduces operation latency and improves data processing throughput. The target identifier forms a strong association with the semi-structured data to be processed. Whether the identifier is user-defined or system-generated, it uniquely points to the corresponding semi-structured data. The target data can be directly located using this identifier without parsing the complete semi-structured data content, greatly improving data retrieval and tracing efficiency and providing convenient support for data lifecycle management. No modification to the original structure of the semi-structured data is required; data insertion management is achieved solely through identifier extraction / generation and index rule adaptation, ensuring seamless compatibility with existing semi-structured data storage systems. Furthermore, the predefined identifier fields can be flexibly configured, and the index rules can be adjusted as needed, adapting to the storage requirements of different types of semi-structured data and reducing the cost of system upgrades and business expansion.

[0116] Exemplary illustration, Figure 4 This is a timing diagram of a concurrent insertion operation provided according to an embodiment of the present invention.

[0117] like Figure 4 As shown, the first client initiates a semi-structured data insertion operation and starts the first transaction; the database creates the first transaction context and performs the following operations within the first transaction context: checks whether the target identifier of the semi-structured data to be processed violates the constraint of the unique index; if no conflict is found, the database performs the insertion operation and waits for instructions from the first client; the first client commits the first transaction; the database completes the transaction commit and returns a success message to the first client.

[0118] like Figure 4 As shown, later, the second client initiates an insert operation for the same target identifier (or an identifier that violates the uniqueness constraint) and starts a second transaction; the database creates the context of the second transaction and attempts to perform the same uniqueness constraint check; at this time, since the first transaction has been committed, the database detects a uniqueness conflict and immediately blocks the insert operation; the second client attempts to commit the second transaction; the database rejects the commit request, performs a rollback of the second transaction, and returns a uniqueness constraint conflict error message to the second client.

[0119] like Figure 4 As shown, by encapsulating the insertion operation of the semi-structured data to be processed within a database transaction (such as the first transaction and the second transaction), the atomicity and isolation of transactions are utilized to ensure that concurrent operations on the same unique index are executed serially. This prevents data inconsistency issues caused by "dirty writes" or "phantom reads" that may occur when there is no transaction protection or insufficient isolation level. The database checks the unique index within the transaction context. When the first transaction successfully inserts the data, the conflict check of the second transaction immediately fails, and the database actively executes a transaction rollback. This mechanism makes conflict detection and handling real-time and built-in. Any write operation that violates the uniqueness constraint cannot be persisted, ensuring the absolute enforceability of business rules.

[0120] The flowcharts provided in this embodiment are not intended to indicate that the operations of the method will be performed in any particular order, or that all operations of the method are included in every case. Furthermore, the method may include additional operations. Within the scope of the technical concept provided by the method in this embodiment, additional variations can be made to the above method.

[0121] It should be understood that in some embodiments, the components may be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods may be implemented using software or firmware stored in memory and executed by a suitable instruction execution system.

[0122] This invention also provides a computer program product 10, a computer-readable storage medium 20, and a computer device 30. For example... Figure 6As shown, the computer program product 10 includes a computer program 11, which, when executed by the processor 32, implements the steps of any of the above-described methods for inserting semi-structured data into a database. Figure 7 As shown, a computer-readable storage medium 20 stores the aforementioned computer program 11, which, when executed by the processor 32, implements the steps of the method for inserting semi-structured data into a database according to any of the above embodiments. Figure 8 As shown, the computer device 30 may include a memory 31, a processor 32, and a computer program 11 stored on the memory 31 and running on the processor 32.

[0123] The computer program 11 used to perform the operations of this invention may be assembly instructions, Instruction Set Architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, integrated circuit configuration data, or source code or object code written in any combination of one or more programming languages ​​and procedural programming languages. The computer program 11 may execute entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the latter case, the remote computer may be connected to the user's computer via any type of network, including a Local Area Network (LAN) or Wide Area Network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, to perform aspects of this invention, electronic circuits, including, for example, programmable logic circuits, Field-Programmable Gate Arrays (FPGAs), or Programmable Logic Arrays (PLAs), may execute computer-readable program instructions to personalize the electronic circuits by utilizing state information from computer-readable program instructions.

[0124] For the purposes of this embodiment, computer program product 10 is a related product that includes computer program 11.

[0125] For the purposes of this embodiment, the computer-readable storage medium 20 is a tangible device capable of holding and storing a computer program 11. It can be any device capable of containing, storing, communicating, propagating, or transmitting the computer program 11 for use by or in conjunction with an instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of the computer-readable storage medium 20 include: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable optical disc read-only memory (CD-ROM), digital versatile disc (DVD), memory stick, floppy disk, mechanical encoding device, and any suitable combination thereof.

[0126] Computer device 30 can be, for example, a server, desktop computer, laptop computer, tablet computer, or smartphone. In some examples, computer device 30 can be a cloud computing node. Computer device 30 can be described in the general context of computer system executable instructions (such as program modules) executed by a computer system. Typically, program modules can include routines, programs, object programs, components, logic, data structures, etc., that perform specific tasks or implement specific abstract data types. Computer device 30 can be implemented in a distributed cloud computing environment where tasks are performed by remote processing devices linked through a communication network. In a distributed cloud computing environment, program modules can reside on local or remote computing system storage media, including storage devices.

[0127] Computer device 30 may include a processor 32 adapted to execute stored instructions and a memory 31 that provides temporary storage space for the operation of said instructions during operation. The processor 32 may be a single-core processor, a multi-core processor, a computing cluster, or any other configuration. The memory 31 may include random access memory (RAM), read-only memory, flash memory, or any other suitable storage system.

[0128] Computer device 30 may also include a network adapter / interface and an input / output (I / O) interface. The I / O interface allows external devices that can be connected to the computer device to input and output data. The network adapter / interface provides communication between the computer device and a network, typically represented as a communication network.

[0129] Therefore, those skilled in the art should recognize that although numerous exemplary embodiments of the present invention have been shown and described in detail herein, many other variations or modifications conforming to the principles of the present invention can be directly determined or derived from the disclosure of the present invention without departing from the spirit and scope of the invention. Thus, the scope of the present invention should be understood and construed as covering all such other variations or modifications.

Claims

1. A method for inserting semi-structured data into a database, characterized in that, include: In response to receiving an insertion request for semi-structured data to be processed, the semi-structured data to be processed is parsed, and it is determined whether the semi-structured data to be processed contains a predefined identifier field. If the semi-structured data to be processed contains the predefined identifier field, the value of the identifier field shall be used as the target identifier. If the semi-structured data to be processed does not contain the predefined identifier field, generate a unique identifier associated with the semi-structured data to be processed, and use the unique identifier as the target identifier. The insertion operation of the target identifier and the semi-structured data to be processed is performed based on the pre-set indexing rules.

2. The method according to claim 1, characterized in that, The structure of the predefined identifier field is a nested path expression; Furthermore, the step of parsing the semi-structured data to be processed includes: traversing the nested path expression to parse the internal hierarchy of the semi-structured data to be processed.

3. The method according to claim 1, characterized in that, The response prior to receiving the insertion request for the semi-structured data to be processed also includes: Create a data table containing a first storage structure and a second storage structure, wherein the first storage structure is used to store semi-structured data, and the second storage structure is used to store the target identifier of the semi-structured data, wherein the second storage structure has a unique index built based on the target identifier.

4. The method according to claim 1, characterized in that, The method for generating a unique identifier associated with the semi-structured data to be processed includes: One or more of the following generation methods can be used: sequence-based generation, random number-based generation, timestamp-based generation, and generation based on a universally unique identifier algorithm.

5. The method according to claim 1, characterized in that, The process of generating a unique identifier associated with the semi-structured data to be processed includes: Determine the hash value based on part or all of the content of the semi-structured data to be processed; The hash value is used as the unique identifier.

6. The method according to claim 1, characterized in that, Based on pre-set indexing rules, the insertion operation of the target identifier and the semi-structured data to be processed is performed, including: The global uniqueness of the target identifier is verified across multiple database nodes using a consensus mechanism; In response to the fact that the target identifier is globally unique, an insertion operation is performed on the target identifier and the semi-structured data to be processed; In response to the fact that the target identifier is not globally unique, a rollback operation is performed.

7. The method according to claim 1, characterized in that, Based on pre-set indexing rules, the insertion operation of the target identifier and the semi-structured data to be processed is performed, including: The validity of the target identifier is determined by the predefined time interval to which the generation time belongs and the physical database partition.

8. The method according to claim 3, characterized in that, The first storage structure is used to store semi-structured data, and the second storage structure is used to store the target identifier of the semi-structured data, including: A copy-on-write mechanism is configured when storing semi-structured data and the target identifier of the semi-structured data; The copy-on-write mechanism includes: writing the target identifier and the semi-structured data to be processed into a temporary storage structure; after verifying that the target identifier meets the preset indexing rules, submitting the semi-structured data to the first storage structure; and submitting the target identifier to the second storage structure.

9. The method according to claim 1, characterized in that, Based on pre-set indexing rules, the insertion operation of the target identifier and the semi-structured data to be processed is performed, including: If the insertion operation of the semi-structured data to be processed fails and the reason for the failure is that the target identifier conflicts with an existing historical target identifier, the target identifier is regenerated through a preset identifier generation strategy. The preset identifier generation strategy can be any one of the following: switching algorithm, injection of random factors, or Snowflake algorithm.

10. A computer device, characterized in that, The method includes a memory, a processor, and a machine-executable program stored on the memory and running on the processor, wherein the processor executes the machine-executable program to implement the steps of the method for inserting semi-structured data into a database according to any one of claims 1 to 9.

Images