Unstructured data storage and retrieval method and system for large-scale complex object

Abstract

An unstructured data storage and retrieval method for a large-scale complex object belongs to the field of databases, and comprises the step of abstractly dividing unstructured data of heterogeneous objects into three types for storage. Extracting object retrievable information from the object native data based on a meta-information model, and performing data conversion and encoding to generate object retrieval data; carrying out database and table division according to object categories, dividing into a common attribute retrieval group, a plurality of hidden attribute retrieval groups, a keyword retrieval group and an auxiliary retrieval column, and establishing a corresponding index structure; semantic extension is carried out based on a standard SQL grammatical structure, and a mixed retrieval mode is achieved. According to the method, in-database integrated storage and retrieval based on the database are realized, unnecessary data migration and synchronization overhead are avoided, data consistency is improved, system deployment complexity and operation and maintenance cost are greatly reduced, meanwhile, efficient sharing and exchange of large-scale heterogeneous diversified data are supported, and the method is suitable for popularization and application. And the requirements of AI, digital twinning and other intelligent applications on a data surface are met.

Description

Technical Field

[0001] This application relates to the field of databases, specifically to a method and system for storing and retrieving unstructured data of large-scale complex objects. Background Technology

[0002] With the development of digitalization, networking, and intelligence in various industries, large-scale, multi-source, and heterogeneous data in software application systems have become the norm. There is an increasing amount of data of different types, sources, and formats, a significant portion of which is unstructured or semi-structured heterogeneous data. This data is often complex in structure and has different uses and processing methods. It usually requires the use of object-oriented methods for data organization and management, that is, to encapsulate related data by defining a series of classes and objects, and to express more complex data volumes based on the generalization and composition between objects.

[0003] While this object-oriented data organization method brings convenience in data reading, writing and management, complex objects are difficult to model using entity relationships. The usual practice is to serialize the entire object directly into a binary or JSON text format data body and then save it directly in the BLOB or TEXT field of the database to simplify the application layer's reading and writing of data.

[0004] However, this simple unstructured storage makes subsequent object data retrieval, data sharing, and exchange very inconvenient. Although current database systems also provide JSON retrieval capabilities, the retrieval performance is not very high, and it is not suitable for use in large-scale data scenarios. If other middleware such as key / value caches and search engines are used to transfer data from the database and transform it before retrieval, there will be a lot of data migration and synchronization work, as well as complex middleware coding, which not only increases the system's performance and resource overhead, but also significantly increases the complexity of system deployment and operation and maintenance costs.

[0005] Therefore, how to efficiently store and retrieve such large-scale, object-oriented, unstructured data in a low-cost, simple, and reliable manner, while meeting the application layer's need for integrated management of heterogeneous data from different sources, and enabling convenient retrieval and rapid sharing of public data assets, is a major technical challenge and problem faced by industries in the process of digital and intelligent transformation. Summary of the Invention

[0006] This application provides a method and system for storing and retrieving unstructured data of large-scale complex objects, which can solve the technical problem in the prior art that it is difficult to achieve low-cost, simple and reliable integrated storage and efficient and easy retrieval of unstructured data of large-scale complex objects based on a database.

[0007] In a first aspect, embodiments of this application provide a method for storing and retrieving unstructured data of large-scale complex objects, the method comprising: The unstructured data of heterogeneous objects is abstracted into three categories of storage: object management data, object native data, and object retrieval data. Object management data includes common management attributes such as object type and version number. Object native data contains complete object information and supports serialized data reading and writing in an object-oriented manner. Based on the meta-information model, features are extracted and encoded from the original data of objects to generate object retrieval data, which includes equivalent terms of object feature values ​​and auxiliary retrieval data for efficient retrieval. The database is partitioned and tables are split according to object category, and the object retrieval data is divided into common attribute retrieval group, multiple hidden attribute retrieval group, keyword retrieval group and auxiliary retrieval column according to retrieval needs; Establish the corresponding index structure, including a composite full-text index of regular search groups and hidden search groups, a separate full-text index of keyword search groups, and a regular index of auxiliary search columns; Based on the standard SQL syntax structure, semantic extensions are performed to map the conditional clauses in the user query to the corresponding search groups, index columns, and index structures according to the search requirements, thereby realizing a hybrid search mode.

[0008] In conjunction with the first aspect, in one implementation, the object management data includes object type, version number, creator, modifier, creation date, modification date, and status.

[0009] In conjunction with the first aspect, in one implementation, the object's native data is stored in binary BLOB, JSON, or other proprietary formats, supporting serialized data reading and writing in an object-oriented manner.

[0010] In conjunction with the first aspect, in one implementation, the feature extraction and encoding includes replacing the positive and negative signs of feature values, unifying date formats, and processing special characters, and generating equivalent terms for object feature values ​​by combining prefix codes and feature terms.

[0011] In conjunction with the first aspect, in one implementation, the commonly used attribute retrieval group is used for basic information of frequently used objects that require synchronized data return; The multiple hidden attribute retrieval groups are feature data that are only used as query conditions and do not require data echoing. The keyword retrieval group is used for long text attribute retrieval; The auxiliary search column is used for special search methods, including interval statistics and function queries.

[0012] In conjunction with the first aspect, in one implementation, the establishment of the index structure includes: A composite full-text index is built using the regular search group and the hidden search group, and the Simple word segmenter is used. Keyword search groups are indexed separately using an Ngram tokenizer. Create regular indexes for auxiliary retrieval columns as needed.

[0013] In conjunction with the first aspect, in one implementation, the hybrid retrieval mode includes: Full-text encoded search mode, used for both regular and hidden attribute searches; Full-text character search mode is used for searching attributes in long text. The auxiliary SQL retrieval mode is used to assist in retrieving data from columns.

[0014] In conjunction with the first aspect, in one implementation, the process of generating the object retrieval data includes: Filter out inefficient and low-value information from long text data.

[0015] Secondly, embodiments of this application provide a system for storing and retrieving unstructured data of large-scale complex objects, which implements the aforementioned method for storing and retrieving such data. The system includes: The object data abstraction module is used to abstract the unstructured data of heterogeneous objects into three categories for storage: object management data, object native data, and object retrieval data. Object management data includes common management attributes such as object type and version number. Object native data contains complete object information and supports serialized data reading and writing in an object-oriented manner. The feature extraction and encoding module extracts feature attribute values ​​from the object's original data based on the meta-information model, performs data transformation and encoding, and generates object retrieval data, which includes object feature value equivalent terms and auxiliary retrieval data for efficient retrieval. The data grouping and index management module is used to partition databases and tables according to object categories, divide object retrieval data into common attribute retrieval groups, multiple hidden attribute retrieval groups, keyword retrieval groups and auxiliary retrieval columns according to retrieval needs, and establish corresponding index structures, including composite full-text indexes of regular retrieval groups and hidden retrieval groups, separate full-text indexes of keyword retrieval groups and ordinary indexes of auxiliary retrieval columns; The query processing and execution module is used to semantically extend the standard SQL syntax structure, mapping the conditional clauses in the user query to the corresponding search groups, index columns and index structures according to the search requirements, so as to realize the hybrid search mode. The data storage module is for relational databases and enables integrated storage.

[0016] In conjunction with the second aspect, in one implementation, the object management data includes object type, version number, creator, modifier, creation date, modification date, and status; The object's native data is stored in binary BLOB, JSON, or other proprietary formats, and supports serialized data reading and writing in an object-oriented manner.

[0017] The beneficial effects of the technical solutions provided in this application include: This invention abstracts unstructured data of heterogeneous objects into three categories for storage (object management data, object native data, and object retrieval data). Based on a meta-information model, it extracts and encodes features from the object native data, shards and tables according to object categories, and establishes targeted index structures. This enables a hybrid retrieval mode based on standard SQL syntax, effectively solving the problem of efficient storage and retrieval of large-scale, complex unstructured object data. Specifically, this invention leverages the existing capabilities of the database to achieve integrated storage and retrieval within the database, avoiding the data migration and synchronization performance overhead associated with migrating data to middleware such as key-value caches and search engines in traditional methods. Furthermore, by completing all data updates within a single transaction, data consistency is guaranteed. In addition, this invention eliminates the need for additional middleware systems, significantly reducing system deployment complexity and maintenance costs. More importantly, through a unified data storage and retrieval mechanism, this invention achieves seamless sharing and exchange of large-scale, heterogeneous, and diverse data, providing efficient and reliable data support for intelligent applications such as AI and digital twins, meeting the high data requirements of intelligent applications. Attached Figure Description

[0018] Figure 1 This is a flowchart illustrating an embodiment of the method for storing and retrieving unstructured data of large-scale complex objects according to this application. Figure 2 This is a schematic diagram of an embodiment of the unstructured storage model of the subject of this application; Figure 3 A schematic diagram of an embodiment of the metadata retrieval model for the object of this application; Figure 4 This is a flowchart illustrating an embodiment of feature extraction and storage for the object of this application. Figure 5 A flowchart illustrating an embodiment of the object retrieval process for this application; Figure 6 This is a schematic diagram of an embodiment of the object retrieval pattern mapping rule of this application; Figure 7 This is a schematic diagram of the functional modules of an embodiment of the unstructured data storage and retrieval system for large-scale complex objects according to this application; Figure 8This is a schematic diagram illustrating the principle of an embodiment of the unstructured data storage and retrieval system for large-scale complex objects according to this application. Detailed Implementation

[0019] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present application.

[0020] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings.

[0021] In a first aspect, embodiments of this application provide a method for storing and retrieving unstructured data of large-scale complex objects.

[0022] In one embodiment, reference is made to Figure 1 , Figure 1 This is a flowchart illustrating an embodiment of the method for storing and retrieving unstructured data of large-scale complex objects according to this application. Figure 1 As shown, methods for storing and retrieving unstructured data of large-scale complex objects include: Step S1: Abstract the unstructured data of heterogeneous objects into three categories for storage: object management data, object native data, and object retrieval data. Object management data includes common management attributes such as object type and version number. Object native data contains complete object information and supports serialized data reading and writing in an object-oriented manner.

[0023] Step S2: Based on the meta-information model, feature extraction and encoding are performed on the original object data to generate object retrieval data, which includes equivalent terms of object feature values ​​and auxiliary retrieval data for efficient retrieval.

[0024] Step S3: Divide the database and tables according to object categories, and divide the object retrieval data into common attribute retrieval groups, multiple hidden attribute retrieval groups, keyword retrieval groups and auxiliary retrieval columns according to retrieval needs.

[0025] Step S4: Establish the corresponding index structure, including a composite full-text index of regular search groups and hidden search groups, a separate full-text index of keyword search groups, and a regular index of auxiliary search columns.

[0026] Step 5: Based on the standard SQL syntax structure, semantic extension is performed to map the conditional clauses in the user query to the corresponding search groups, index columns and index structures according to the search requirements, so as to realize the hybrid search mode.

[0027] The main problem this invention aims to solve is how to achieve low-cost, simple, reliable, and efficient retrieval of complex unstructured object data in large-scale scenarios using relational databases. Specifically, traditional methods directly serialize the entire object into binary or JSON format and store it in BLOB / TEXT fields, leading to inconvenient retrieval. Current database systems have low JSON retrieval performance, making them unsuitable for large-scale scenarios. Using middleware such as key-value caches and search engines for data transfer and retrieval introduces data migration, synchronization, and system complexity. It is impossible to simultaneously achieve the convenience of heterogeneous data object management and the simplicity of relational database-style SQL retrieval. Maintaining good retrieval performance and scalability is difficult in large-scale scenarios.

[0028] This invention abstracts unstructured data of heterogeneous objects into three storage categories (object management data, object native data, and object retrieval data). Based on a meta-information model, it extracts and encodes features from the object native data, shards and tables according to object categories, and establishes targeted index structures. This enables a hybrid retrieval mode based on standard SQL syntax, effectively solving the problem of efficient storage and retrieval of large-scale, complex unstructured object data. Specifically, this invention leverages the existing capabilities of the database to achieve integrated storage and retrieval within the database, avoiding the data migration and synchronization performance overhead associated with migrating data to middleware such as key-value caches and search engines in traditional methods. Furthermore, by completing all data updates within a single transaction, data consistency is guaranteed. In addition, this invention eliminates the need for additional middleware systems, significantly reducing system deployment complexity and operational costs. More importantly, through a unified data storage and retrieval mechanism, this invention achieves seamless sharing and exchange of large-scale, heterogeneous, and diverse data, providing efficient and reliable data support for intelligent applications such as AI and digital twins, meeting the high data requirements of intelligent applications.

[0029] Specifically, a data layered storage mechanism is adopted to abstract the unstructured data of heterogeneous objects into three categories of storage: object management data (common management attributes such as object type, version number, creator, modifier, creation date, modification date, and status), object native data (stored in binary BLOB, JSON, or other private formats, supporting serialized data reading and writing in an object-oriented manner), and object retrieval data (retrievable feature information of processed objects).

[0030] This paper extracts searchable information about an object from its native data based on a meta-information model. Data transformation is performed on feature values ​​(sign replacement, date format standardization, special character handling, etc.). A combination of prefix codes and feature terms is used to express the object's searchable feature attributes (2-character encoding supports 1296 feature attributes). The object retrieval meta-information (i.e., the meta-information model) is a set of rules and models guiding feature extraction, storage, and retrieval, defining which information can be extracted, how to transform it, and how to store it. Searchable information about the object is the raw feature information extracted from the object's native data (such as attribute values, text content, etc.). Object retrieval data is the processed searchable information, including object feature value equivalent terms and auxiliary retrieval data.

[0031] The database is sharded and partitioned according to object categories. Object retrieval data is divided into: common attribute retrieval groups (basic object information that needs to be returned synchronously); multiple hidden attribute retrieval groups (feature data used only as query conditions without data display); keyword retrieval groups (for long text attribute retrieval); and auxiliary retrieval columns (for special retrieval methods, such as range statistics and function queries). Targeted indexes are created: a composite full-text index (Simple tokenizer) for the regular and hidden retrieval groups; a separate full-text index (N-gram tokenizer, typically N=2 / 3 / 4) for the keyword retrieval groups; and regular indexes for auxiliary retrieval columns as needed.

[0032] This system extends the semantics of standard SQL syntax to enable hybrid search modes. It supports SELECT [ID|SUMMARY:JSON|RAW]|COUNT, FROM object category, WHERE clause [AND|OR clause], [LIMIT m][OFFSET n]. Hybrid search modes include: full-text encoded search mode (for regular and hidden attribute searches), full-text character search mode (for long text attribute searches), and auxiliary SQL search mode (for retrieving data from auxiliary search columns).

[0033] This invention leverages the existing capabilities of databases to achieve integrated storage and retrieval within the database, avoiding the performance overhead of unnecessary data migration and synchronization. All data updates are completed within the same database transaction, ensuring data consistency and resolving the inconsistency issues caused by data synchronization in traditional methods. Different indexing technologies are employed for different retrieval needs, ensuring retrieval performance in large-scale scenarios. The system architecture is simplified, reducing deployment and maintenance costs, and supports database sharding by object category, guaranteeing system scalability. Seamless sharing and exchange of large-scale heterogeneous and diverse data is achieved, providing efficient and reliable data support for intelligent applications such as AI and digital twins, meeting the high data requirements of intelligent applications.

[0034] In summary, this invention separates the raw data of objects from their feature attributes and combines intelligent encoding and indexing strategies to achieve efficient storage and retrieval of large-scale complex unstructured object data. This solves the pain points of traditional methods in data storage and retrieval, while maintaining the system's simplicity, reliability, and low cost.

[0035] Furthermore, in one embodiment, the aforementioned object management data includes common administrative attributes such as object type, version number, creator, modifier, creation date, modification date, and status.

[0036] In this embodiment, object management data provides basic management information for objects, enabling the system to effectively classify, track, and version control objects.

[0037] Object types are used to distinguish different object categories (such as users, orders, and products) and are the basis for data sharding, table partitioning, and retrieval grouping.

[0038] Version numbers ensure that data changes are traceable and support version rollback and consistency management.

[0039] Creators / modifiers can audit and trace the responsibility for data operations.

[0040] Creation date / modification date supports time range queries and data timeliness management.

[0041] State is used for object lifecycle management, such as creation, activity, locking, and deletion, and is an important component of query conditions.

[0042] These attributes are components of the meta-information model, providing fundamental information for feature extraction and processing, and data retrieval. Among them, object type is used to determine the rules for data sharding and table partitioning.

[0043] The system automatically categorizes and manages objects based on attributes such as object type and status, achieving automatic object grouping without additional configuration. Attributes such as creator and modifier support operation auditing, meeting security and compliance requirements. Time attributes such as creation date and modification date support querying by time range, while status attributes support conditional querying based on the current state of object data, eliminating the need to traverse the entire dataset. The introduction of version numbers enables the system to track data change history, ensuring consistency during data updates. All management attributes, along with the object's native data and object retrieval data, are stored uniformly, avoiding the data inconsistency issues caused by the separation of management information and business data in traditional methods. The system manages all objects through unified management attributes, eliminating the need to design separate management attributes for each object type.

[0044] Furthermore, in one embodiment, the original data of the aforementioned object is stored in binary BLOB, JSON, or other proprietary formats, supporting serialized data reading and writing in an object-oriented manner.

[0045] In this embodiment, binary BLOBs preserve the original binary representation of the object, avoiding data loss caused by format conversion, making them suitable for storing complex objects. JSON provides a structured data representation, facilitating cross-platform interaction and direct processing at the application layer. Other proprietary formats offer customized storage methods for specific application needs, such as proprietary binary formats for specific business objects.

[0046] Furthermore, in one embodiment, the above feature extraction and encoding includes data conversion such as replacing positive and negative signs of feature values, unifying date formats, and processing special characters. The object's retrievable feature attributes are expressed by a combination of prefix code and feature terms, wherein the 2-character encoding can support 1296 feature attributes.

[0047] In this embodiment, standardized feature values ​​improve retrieval efficiency. Unified data transformation rules ensure feature value consistency. The prefix encoding mechanism supports the addition of new feature attributes in the future without modifying the system architecture. The 2-character encoding supports 1296 feature attributes, meeting the needs of most business scenarios.

[0048] Furthermore, in one embodiment, the object retrieval data includes object feature value equivalent terms for regular retrieval and auxiliary retrieval data (for special queries).

[0049] In this embodiment, regular queries use feature value equivalent terms, while special queries use auxiliary retrieval data.

[0050] Furthermore, in one embodiment, the aforementioned commonly used attribute retrieval group is used for frequently used basic information of objects that require synchronized data return. The aforementioned multiple hidden attribute retrieval groups are feature data used only as query conditions and not for data display. The aforementioned keyword retrieval group is used for long text attribute retrieval. The aforementioned auxiliary retrieval columns are used for special retrieval methods, including interval statistics and function queries.

[0051] In this embodiment, different storage structures are used for different retrieval needs, making the retrieval logic clearer and improving retrieval speed. This avoids creating the same index for all attributes, saving storage space and computing resources.

[0052] Furthermore, in one embodiment, the establishment of the above index structure includes: building a composite full-text index for the regular search group and the hidden search group using the Simple tokenizer; building a separate full-text index for the keyword search group using the N-gram tokenizer, where N=2 / 3 / 4; and building a regular index for the auxiliary search columns as needed.

[0053] In this embodiment, the best indexing technology is flexibly selected for different data types to avoid using the same index for all data and improve retrieval speed.

[0054] Furthermore, in one embodiment, the above-mentioned hybrid retrieval modes include: a full-text encoded retrieval mode for regular and hidden attribute retrieval; a full-text character retrieval mode for long text attribute retrieval; and an auxiliary SQL retrieval mode for retrieval of data in auxiliary retrieval columns.

[0055] The process of generating the above-mentioned object retrieval data includes: filtering out inefficient and low-value information from long text data.

[0056] In this embodiment, the optimal retrieval method is selected for different query types. Users only need to use standard SQL syntax, and the system will automatically select the optimal retrieval mode, making user operation more convenient and retrieval efficiency higher.

[0057] In one specific embodiment, refer to Figure 2 The unstructured object storage model shown abstracts the unstructured data of heterogeneous objects into three categories for separate storage: object management data, object native data, and object retrieval data. These three types of data are linked by internal and external keys using unique object IDs to ensure strong data consistency. Specifically, object management data and native data are stored in the database through the object persistence module, while object retrieval data is stored in the database through the data update module.

[0058] Object management data includes common management attributes that heterogeneous objects generally possess, but does not involve specific object business attributes or specific data. Typical common management attributes include: object type, version number, creator, modifier, creation date, modification date, status, etc. Object retrieval based on common management attributes can be performed using conventional retrieval methods.

[0059] The object's native data contains all the information necessary for object persistence and is stored in binary BLOB, JSON, or other proprietary formats to support serialized data reading and writing in an object-oriented manner. To improve the concurrent update capability of the object's native data, it can be further subdivided into several subclasses, with each subclass corresponding to the data display and editing of a specific sub-scenario.

[0060] The core information of raw data is defined as follows: Object native data::={object ID, sub-data category, sub-data body, internal version number}.

[0061] The object retrieval data storage and processing includes retrieveable feature information of the objects, such as equivalent terms for object feature values ​​and auxiliary retrieval data. To improve the performance of object retrieval at large scales, objects can be partitioned into databases and tables based on business similarity (e.g., by object business category). The retrieval data table structure for each type of object is basically the same, including: one regular attribute retrieval group, multiple hidden attribute retrieval groups, one keyword retrieval group, and multiple auxiliary retrieval columns. The auxiliary retrieval columns can be indexed as needed using ordinary indexes. The regular retrieval group and the hidden retrieval group require a composite full-text index and the use of the Simple tokenizer. The keyword retrieval group requires a separate full-text index and the use of the N-gram tokenizer, where N is typically 2 / 3 / 4, with 3 being sufficient in most scenarios.

[0062] The core information of object retrieval data is defined as follows: Object retrieval data::={object ID, common attribute retrieval group, hidden attribute retrieval group (*), keyword retrieval group, auxiliary retrieval column (*)}.

[0063] Among them, the auxiliary retrieval columns are named uniformly using the prefix code of the feature data, so as to facilitate automatic mapping of the data source during storage and retrieval.

[0064] Reference Figure 3 The object retrieval metadata model shown defines the basic rules to be followed in the process of feature extraction, preprocessing, data storage, and data retrieval of complex objects. The metadata defined in this patent is divided into two categories: data distribution metadata and object retrieval metadata. The core information includes the following: Data distribution metadata::={Object category ID, Object category name, Search database name, Search table name}.

[0065] Object retrieval metadata::=Object category ID, Object feature Key name, Prefix encoding, Raw data type, Retrieval group identifier, Auxiliary retrieval marker, Index type}.

[0066] The object retrieval metadata is stored in the same way as the object management data and the object's original data, and its content is shared by all processing modules of this patented device.

[0067] Reference Figure 4 The diagram shown illustrates the object feature extraction and storage process, which includes: The output is a formatted feature text of the object's searchable information. The basic content of the object feature text must include: the search category (Class) to which the object belongs and the set of object feature attributes. Each object feature attribute consists of a feature access identifier (Key) and a corresponding attribute value (Value).

[0068] Based on the object retrieval meta-information model, feature matching is performed on the output object feature text using Class+Key, and feature values ​​are extracted from the successfully matched object feature attributes.

[0069] The extracted feature values ​​are transformed. The purpose of data transformation is to improve data storage efficiency and prevent retrieval conflicts. Transformation processes include: replacing the positive and negative signs of negative numbers and exponents, standardizing the format of time and date data, clearing or escaping special characters such as punctuation marks, and atomically splitting numeric data.

[0070] The converted data is uniformly attribute-encoded, and the searchable feature attributes of an object are expressed by a combination of prefix code and feature terms. The prefix code only needs to be unique within the same type of object, and a fixed N characters are used for encoding. In most cases, N=2 is sufficient. The effective character set of the prefix code is 0~9 and A~Z. Taking a 2-character encoding as an example, the range of feature attributes that a single type of object can express can reach 36x36=1296.

[0071] For long text data in object feature data, inefficient and low-value information is filtered out by removing special symbols, stop words, etc. that are not of great search significance. The filtered long text data does not need to be encoded and can be directly stored in the keyword search group.

[0072] The encoded object feature data is grouped according to the retrieval method and stored in different database table fields, with unified storage and updates based on database transactions. Specifically: frequently used basic object information that requires synchronized data return during queries can be grouped into the common attribute retrieval group. Other object feature data used only as query conditions and not requiring data display can be further subdivided into multiple hidden attribute retrieval groups as needed. For object feature attributes requiring special retrieval methods (such as interval statistics, function queries, etc.), auxiliary retrieval columns are constructed corresponding to the attributes and uniformly named using attribute prefix codes. The data stored in the auxiliary retrieval columns can be native attribute values ​​or statistically calculated values.

[0073] Reference Figure 5 The diagram shown illustrates the object query and retrieval process. The object query and retrieval process includes: Based on the object query syntax defined in this patent, the externally issued object query request statements are syntax-checked and analyzed, and query requests that conform to the syntax rules are converted into internal structures to facilitate further analysis and processing. The internal structure is a structured expression of the object query statement, typically including: a Select structure, multiple Condition structures, etc.

[0074] The object query syntax defined in this patent adopts a syntax extension similar to standard SQL, with the following basic format: Object query syntax::=SELECT[ID]|[SUMMARY:JSON|RAW]|COUNT FROM object category.

[0075] WHERE conditional clause [AND|OR conditional clause][LIMIT m][OFFSET n].

[0076] Conditional clause::=[object property][=,>,<,LIKE,BETWEEN,CONTAIN][target value].

[0077] Where: ID represents the unique ID of the returned object, SUMMARY represents the summary of the object's commonly used attributes, the return format provides both JSON and RAW, and COUNT returns the number of objects that meet the conditions.

[0078] The internal structure of the object query generated in the previous step is optimized as follows: data transformation, attribute name replacement, and data terminology encoding are performed on the object attributes and target values ​​in the query conditions to match the actual database table field names and the encoding method of the stored data. At the same time, query conditions with the same object attribute are merged.

[0079] Further analysis is conducted on the object categories, object attributes, and search types in the query conditions. Based on the meta-information model, the corresponding search data source for each object attribute is determined, and then... Figure 6 The retrieval mode mapping rules shown determine the retrieval modes to be adopted for different object attribute searches, including: full-text encoded retrieval mode, full-text character retrieval mode, and auxiliary SQL retrieval mode. Based on different retrieval modes, object query conditions are decomposed into multiple sub-retrievals. Specifically, the full-text encoded sub-retrieval is used for searching both regular and hidden object attributes, the full-text character sub-retrieval is used for searching long text attributes of objects, and the auxiliary SQL sub-retrieval is used for searching data in auxiliary search columns.

[0080] Each sub-retrieval decomposed in the previous step is converted into a corresponding standard SQL clause for relational databases. Multiple clauses are then combined into a single mixed SQL retrieval statement that includes both full-text search and regular search based on the logical relationship between the query conditions using AND or OR operations.

[0081] Establish a connection to the database and send the mixed search SQL statement generated in the previous step to the database for execution. To prevent database overload, implement maximum concurrency control and task queue scheduling for multiple mixed search tasks sent simultaneously. After each search result is returned, perform relevant data reverse transformation and de-encoding on the returned results according to the object query statement requirements, and combine the result set into JSON data as needed before returning it.

[0082] Secondly, embodiments of this application also provide an unstructured data storage and retrieval device for large-scale complex objects.

[0083] In one embodiment, reference is made to Figure 7 and Figure 8 , Figure 7 This is a functional module diagram of an embodiment of the unstructured data storage and retrieval system for large-scale complex objects according to this application. Figure 8 This is a schematic diagram illustrating the principle of an embodiment of the unstructured data storage and retrieval system for large-scale complex objects according to this application. Figure 7 and Figure 8 As shown, a large-scale, complex object unstructured data storage and retrieval system includes: Object data abstraction module 1 is used to abstract unstructured data of heterogeneous objects into three categories for storage: object management data, object native data, and object retrieval data.

[0084] Feature extraction and encoding module 2 extracts feature attribute values ​​from the object's original data based on the meta-information model, performs data transformation and encoding, and generates object retrieval data.

[0085] The data grouping and index management module 3 is used to shard the database and tables according to object categories, divide the object retrieval data into common attribute retrieval groups, multiple hidden attribute retrieval groups, keyword retrieval groups and auxiliary retrieval columns according to retrieval needs, and establish corresponding index structures, including a composite full-text index of the regular retrieval group and the hidden retrieval group, a separate full-text index of the keyword retrieval group and a regular index of the auxiliary retrieval column.

[0086] The query processing and execution module 4 is used to perform semantic extension based on the standard SQL syntax structure, mapping the conditional clauses in the user query to the corresponding search groups, index columns and index structures according to the search requirements, so as to realize the hybrid search mode.

[0087] Data storage module 5 is a relational database that enables integrated storage.

[0088] In this embodiment, the object data abstraction module 1 abstracts the unstructured data of heterogeneous objects into three categories for storage: object management data, object native data, and object retrieval data. The object native data is stored in binary BLOB, JSON, or other proprietary formats, supporting serialized data reading and writing in an object-oriented manner. The object retrieval data is output as formatted feature text (such as XML / JSON / YAML), providing input for subsequent feature extraction.

[0089] Feature extraction and encoding module 2, based on the retrieval metadata model, extracts feature attribute values ​​from the object's original data. During feature extraction, data transformations are performed on relevant data, including sign replacement, date format standardization, and special character processing. Long text data is filtered for inefficient and low-value information. A combination of prefix codes and feature terms is used to express the object's searchable feature attributes, generating atomic object feature terms and auxiliary retrieval data, ultimately forming the object retrieval data.

[0090] The data grouping and indexing management module 3 categorizes object retrieval data into four types based on retrieval needs: common attribute retrieval groups (for frequently used basic object information that requires synchronized data return), multiple hidden attribute retrieval groups (feature data used only as query conditions without data display), keyword retrieval groups (for long text attribute retrieval), and auxiliary retrieval columns (for special retrieval methods, such as interval statistics and function queries). This module partitions the database and tables according to object categories and establishes corresponding index structures: a composite full-text index (using the Simple tokenizer) is created for the regular and hidden retrieval groups, a separate full-text index is created for the keyword retrieval groups (using the Ngram tokenizer, N=2 / 3 / 4), and ordinary indexes are created for the auxiliary retrieval columns as needed. Object feature terms are updated incrementally and then stored in the database. Simultaneously, auxiliary retrieval data is saved to the corresponding auxiliary columns in the database tables (based on attribute encoding matching). All data updates are completed within the same database transaction, ensuring data consistency.

[0091] The query processing and execution module 4 performs syntactic analysis and checks on externally issued object query requests based on the object query syntax defined in this patent (semantic extension based on standard SQL syntax structure). It maps the conditional clauses in the user query to the corresponding retrieval groups and index structures according to the retrieval requirements. This module converts the object query request into standard relational database SQL statements, including a mixed retrieval of full-text search for different attribute retrieval groups and regular SQL for auxiliary retrieval columns. Simultaneously, the query processing and execution module 4 establishes and manages the connection with the database, issues and executes the converted mixed retrieval SQL statements, and controls and schedules retrieval tasks based on maximum concurrency to prevent database overload. When the retrieval results are returned, data conversion and assembly processing is performed on the returned results according to syntactic requirements.

[0092] Data storage module 5 uses relational databases (such as MySQL, PostgreSQL, SQL Server, etc.) as the data storage medium to achieve integrated storage of object native data, object retrieval data, and retrieval metadata. Among them, the retrieval information storage is based on object category and partitioned into databases and tables, and different database indexes are established according to different data retrieval methods to ensure data retrieval performance in large-scale scenarios.

[0093] The functional implementation of each module in the above-mentioned unstructured data storage and retrieval system for large-scale complex objects corresponds to each step in the above-mentioned embodiment of the unstructured data storage and retrieval method for large-scale complex objects. Their functions and implementation processes will not be described in detail here.

[0094] It should be noted that the sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0095] The terms "comprising" and "having," and any variations thereof, in the specification, claims, and accompanying drawings of this application are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or apparatus that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to such process, method, product, or apparatus. The terms "first," "second," and "third," etc., are used to distinguish different objects, etc., and do not indicate a sequence, nor do they limit "first," "second," and "third" to different types.

[0096] In the description of the embodiments of this application, terms such as "exemplary," "for example," or "for instance" are used to indicate examples, illustrations, or explanations. Any embodiment or design described as "exemplary," "for example," or "for instance" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or designs. Specifically, the use of terms such as "exemplary," "for example," or "for instance" is intended to present the relevant concepts in a concrete manner.

[0097] In the description of the embodiments of this application, unless otherwise stated, " / " means "or". For example, A / B can mean A or B. The "and / or" in the text is merely a description of the relationship between related objects, indicating that there can be three relationships. For example, A and / or B can mean: A exists alone, A and B exist simultaneously, and B exists alone. In addition, in the description of the embodiments of this application, "multiple" means two or more.

[0098] In some processes described in the embodiments of this application, multiple operations or steps are included in a specific order. However, it should be understood that these operations or steps may not be executed in the order they appear in the embodiments of this application, or they may be executed in parallel. The sequence number of the operation is only used to distinguish different operations, and the sequence number itself does not represent any execution order. In addition, these processes may include more or fewer operations, and these operations or steps may be executed sequentially or in parallel, and these operations or steps may be combined.

[0099] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) as described above, and includes several instructions to cause a terminal device to execute the methods described in the various embodiments of this application.

[0100] The above are merely preferred embodiments of this application and do not limit the patent scope of this application. Any equivalent structural or procedural transformations made using the content of this application's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of this application.

Claims

1. A method for storing and retrieving unstructured data of large-scale complex objects, characterized in that, The method for storing and retrieving unstructured data of large-scale complex objects includes: The unstructured data of heterogeneous objects is abstracted into three categories of storage: object management data, object native data, and object retrieval data. Object management data includes common management attributes such as object type and version number. Object native data contains complete object information and supports serialized data reading and writing in an object-oriented manner. Based on the meta-information model, features are extracted and encoded from the original data of objects to generate object retrieval data, which includes equivalent terms of object feature values ​​and auxiliary retrieval data for efficient retrieval. The database is partitioned and tables are split according to object category, and the object retrieval data is divided into common attribute retrieval group, multiple hidden attribute retrieval group, keyword retrieval group and auxiliary retrieval column according to retrieval needs; Establish the corresponding index structure, including a composite full-text index of regular search groups and hidden search groups, a separate full-text index of keyword search groups, and a regular index of auxiliary search columns; Based on the standard SQL syntax structure, semantic extensions are performed to map the conditional clauses in the user query to the corresponding search groups, index columns, and index structures according to the search requirements, thereby realizing a hybrid search mode.

2. The method for storing and retrieving unstructured data of large-scale complex objects as described in claim 1, characterized in that, The object management data includes object type, version number, creator, modifier, creation date, modification date, and status.

3. The method for storing and retrieving unstructured data of large-scale complex objects as described in claim 1, characterized in that, The object's native data is stored in binary BLOB, JSON, or other proprietary formats, and supports serialized data reading and writing in an object-oriented manner.

4. The method for storing and retrieving unstructured data of large-scale complex objects as described in claim 1, characterized in that, The feature extraction and encoding process includes replacing the positive and negative signs of feature values, standardizing date formats, and processing special characters. It also uses a combination of prefix codes and feature terms to generate equivalent terms for object feature values.

5. The method for storing and retrieving unstructured data of large-scale complex objects as described in claim 1, characterized in that, The commonly used attribute retrieval group is used for basic information of objects that are frequently used and require synchronous data return. The multiple hidden attribute retrieval groups are feature data that are only used as query conditions and do not require data echoing. The keyword retrieval group is used for long text attribute retrieval; The auxiliary search column is used for special search methods, including interval statistics and function queries.

6. The method for storing and retrieving unstructured data of large-scale complex objects as described in claim 1, characterized in that, The establishment of the index structure includes: A composite full-text index is built using the regular search group and the hidden search group, and the Simple word segmenter is used. Keyword search groups are indexed separately using an Ngram tokenizer. Create regular indexes for auxiliary retrieval columns as needed.

7. The method for storing and retrieving unstructured data of large-scale complex objects as described in claim 1, characterized in that, The hybrid retrieval mode includes: Full-text encoded search mode, used for both regular and hidden attribute searches; Full-text character search mode is used for searching attributes in long text. The auxiliary SQL retrieval mode is used to assist in retrieving data from columns.

8. The method for storing and retrieving unstructured data of large-scale complex objects according to claim 1, characterized in that, The process of generating the object retrieval data includes: Filter out inefficient and low-value information from long text data.

9. A system for storing and retrieving unstructured data of large-scale complex objects, implementing the method for storing and retrieving unstructured data of large-scale complex objects according to any one of claims 1-8, characterized in that, The system includes: The object data abstraction module is used to abstract the unstructured data of heterogeneous objects into three categories for storage: object management data, object native data, and object retrieval data. Object management data includes common management attributes such as object type and version number. Object native data contains complete object information and supports serialized data reading and writing in an object-oriented manner. The feature extraction and encoding module extracts feature attribute values ​​from the object's original data based on the meta-information model, performs data transformation and encoding, and generates object retrieval data, which includes object feature value equivalent terms and auxiliary retrieval data for efficient retrieval. The data grouping and index management module is used to partition databases and tables according to object categories, divide object retrieval data into common attribute retrieval groups, multiple hidden attribute retrieval groups, keyword retrieval groups and auxiliary retrieval columns according to retrieval needs, and establish corresponding index structures, including composite full-text indexes of regular retrieval groups and hidden retrieval groups, separate full-text indexes of keyword retrieval groups and ordinary indexes of auxiliary retrieval columns; The query processing and execution module is used to semantically extend the standard SQL syntax structure, mapping the conditional clauses in the user query to the corresponding search groups, index columns and index structures according to the search requirements, so as to realize the hybrid search mode. The data storage module is for relational databases and enables integrated storage.

10. The unstructured data storage and retrieval system for large-scale complex objects according to claim 9, characterized in that, The object management data includes object type, version number, creator, modifier, creation date, modification date, and status; The object's native data is stored in binary BLOB, JSON, or other proprietary formats, and supports serialized data reading and writing in an object-oriented manner.

Images