Query method and device for multi-version unstructured data semantic information

Abstract

The application discloses a kind of multi-version unstructured data semantic information-oriented query method and device.The method comprises: obtaining each version of unstructured data;For each version of unstructured data, based on the artificial intelligence model used for extracting semantic information, the version of the semantic information is obtained;Based on the modification relationship between each artificial intelligence model, the semantic information version tree of unstructured data is constructed and saved;According to the storage mode of the semantic information version tree, the expression mode between the parent node and the child node in the semantic information version tree is established;When querying the semantic information of unstructured data, based on the query filter condition and the expression mode, the semantic information version tree is searched, and the query result of semantic information is obtained.The application can manage and query multi-version unstructured data semantic information.

Description

Technical Field

[0001] This invention relates to the fields of unstructured data, artificial intelligence, query languages, and databases, and aims to provide a method and apparatus for querying semantic information of multi-version unstructured data. Background Technology

[0002] Unstructured data refers to long texts, images, videos, audio, and other unstructured data. Unstructured data is typically stored in computer systems as a string of binary code. A common characteristic of this type of data is its large size and incomprehensibility by computers. User queries regarding unstructured data primarily focus on retrieving information from within the data—information that is semantic and understandable; this is called semantic information. Traditional data query techniques cannot target the semantic information of unstructured data. However, advancements in artificial intelligence (AI) have opened up new directions for the analysis and querying of unstructured data. Currently, AI technology can achieve high accuracy in tasks such as facial recognition, object recognition, speech recognition, and sentiment analysis. Therefore, AI technology can be used to extract information from unstructured data, thereby driving the development of unstructured data querying techniques.

[0003] Semantic information querying of unstructured data is essentially a query for a certain state of an object under certain rules. Unstructured data is essentially a description of a certain state of an object; when the object's state changes, the content of the unstructured data also changes, which can be considered a version change of the unstructured data object. Simultaneously, changes in semantic information extraction rules also cause version changes in semantic information. Existing technologies can only query one or a few semantic information items of a single unstructured data object, and cannot achieve querying of specific versions of unstructured data objects and their semantic information.

[0004] Not only is the semantic information in unstructured data affected by computational rules, resulting in numerous versions, but version queries of unstructured data also face challenges such as complex relationships between versions and version-related queries. These problems pose significant challenges to version queries of unstructured data.

[0005] Therefore, it is very important to study a method for managing and querying semantic information of multi-version unstructured data. Summary of the Invention

[0006] To address the aforementioned problems, this invention discloses a method and apparatus for querying semantic information of multi-version unstructured data. Based on existing graph databases, through research, design, and implementation of technologies such as query language, methods for obtaining version semantic information of unstructured data objects, caching and indexing of multi-version semantic information, and versioned calculation of semantic information, it enables the management and querying of semantic information of multi-version unstructured data.

[0007] The technical content of this invention includes:

[0008] A method for querying semantic information of multi-version unstructured data, the method comprising:

[0009] Obtain unstructured data from various versions;

[0010] For each version of unstructured data, the version of the semantic information is obtained based on the artificial intelligence model used to extract semantic information;

[0011] Based on the modification relationships between various artificial intelligence models, a semantic information version tree of unstructured data is constructed and stored; wherein each node in the semantic information version tree represents a version of semantic information;

[0012] Based on the storage method of the semantic information version tree, establish the expression method between parent nodes and child nodes in the semantic information version tree;

[0013] When querying the semantic information of unstructured data, the semantic information version tree is retrieved based on the query filtering conditions and the expression method to obtain the query results of the semantic information.

[0014] Furthermore, the query filtering conditions include one or more of the following: the ID of the unstructured data object corresponding to the query level and the query semantic information, the version number of the unstructured data object corresponding to the query semantic information, the name of the query semantic information, and the actual content of the query semantic information.

[0015] Furthermore, the storage method includes: storing the semantic information version tree on a hard disk or storing the semantic information version tree in a Key-Value database.

[0016] Furthermore, when the semantic information version tree is stored on a hard disk, the step of establishing the expression method between parent nodes and child nodes in the semantic information version tree based on the storage method of the semantic information version tree includes:

[0017] A three-way pointer is used to maintain the association between parent and child nodes; wherein the three-way pointer includes: a pointer to the parent version, a pointer to the actual semantic information content, and a pointer to all child nodes.

[0018] Furthermore, when querying the semantic information of unstructured data, the semantic information version tree is retrieved based on query filtering conditions to obtain the query results of the semantic information, including:

[0019] Based on the id of the unstructured data object corresponding to the semantic information to be queried, the version number of the unstructured data object corresponding to the semantic information to be queried, the name of the semantic information to be queried, or the semantic information to be queried, a pointer to a three-way pointer to the semantic information to be queried is obtained.

[0020] Based on the pointer to the actual semantic information content, the actual semantic information content of the target node is obtained;

[0021] Using the pointers to the parent version and / or pointers to all child nodes in the target node, and based on the query level, obtain the corresponding parent node and / or child node;

[0022] Based on the pointers to the actual semantic information in the parent node and / or child node, the actual semantic information of the parent node and / or child node is obtained.

[0023] By combining the semantic information and actual content obtained, the query results of semantic information are obtained.

[0024] Furthermore, when the semantic information version tree is stored in a Key-Value database, the step of establishing the expression method between parent nodes and child nodes in the semantic information version tree based on the storage method of the semantic information version tree includes:

[0025] Nodes with names containing the same semantic information are stored in a key-value database;

[0026] Based on the relationship between parent and child nodes, the key values ​​of each version in any Kv database can be obtained.

[0027] Furthermore, when querying the semantic information of unstructured data, the semantic information version tree is retrieved based on query filtering conditions to obtain the query results of the semantic information, including:

[0028] Find the version of semantic information that matches the filtering criteria, and obtain the actual content of the semantic information for that version;

[0029] Based on the Key value of this version and the query level, the versions of the parent node and / or child node are obtained;

[0030] Obtain the semantic information of the parent node and / or child node versions;

[0031] By combining the semantic information and actual content obtained, the query results of semantic information are obtained.

[0032] Further, deleting a version of semantic information from the semantic information version tree includes:

[0033] Remove the corresponding nodes of this version and all its sub-versions from the semantic information tree;

[0034] or,

[0035] The corresponding node of this version is deleted from the semantic information tree, and the parent node of the direct child node of this version is set as the parent node of the corresponding node of this version.

[0036] A storage medium storing a computer program, wherein the computer program is configured to execute the above-described query method for semantic information of multi-version unstructured data at runtime.

[0037] An electronic device includes a memory and a processor, the memory storing a computer program and the processor being configured to run the above-described query method for semantic information of multi-version unstructured data.

[0038] Compared with the prior art, the method proposed in this invention has the following advantages and effects:

[0039] To address the issue that existing technologies do not support versioned management of semantic information in unstructured data, this invention organizes semantic information into a "semantic information version tree" based on version relationships. This preserves the version relationships between semantic information, allowing users to quickly retrieve data based on parent-child relationships. Furthermore, a storage and indexing method for the semantic information version tree is proposed, and two implementations are provided: one based on a local file system and the other on key-value storage. This improves the storage and retrieval efficiency of the semantic information version tree. Attached Figure Description

[0040] Figure 1 The method flowchart of the present invention.

[0041] Figure 2 A schematic diagram of the version tree of semantic information for unstructured data. Detailed Implementation

[0042] The technical solutions in the embodiments of the present invention will now be clearly and completely described with reference to the accompanying drawings. Obviously, the described embodiments are merely specific embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0043] To better manage the relationship between versions of semantic information, this invention defines the relationship as follows:

[0044] Define a version of unstructured data: For an unstructured data object, each update to its content will generate a new version.

[0045] Semantic information is defined as the information contained in unstructured data that can be understood by humans.

[0046] Define the relationship between versions of unstructured data: For an unstructured data object, if version b is obtained by modifying version a, then a is called the parent version of b, and b is called the child version of a. For example... Figure 2 As shown, v-b2 is a modified version of v-b1, therefore v-b1 is the parent version of v-b2, and v-b2 is the child version of v-b1.

[0047] Defining the version of semantic information in unstructured data: This invention uses an artificial intelligence model to extract semantic information from unstructured data. The version of the artificial intelligence model corresponds one-to-one with the version of the semantic information.

[0048] Define the relationship between versions of semantic information in unstructured data: For a certain semantic information of an unstructured data object, the version number of the semantic information extracted based on the model with version v-m1 is denoted as v-sp1; the model is modified based on v-m1 to obtain version v-m2; the version number of the semantic information extracted based on v-m2 is denoted as v-sp2. At this time, v-sp1 is called the parent version of v-sp2, and v-sp2 is the child version of v-sp1.

[0049] To maintain the relationship between semantic information versions, this invention proposes a query method for semantic information of multi-version unstructured data based on the relationships defined above. By constructing a "semantic information version tree", the query of semantic information of multi-version unstructured data can be completed.

[0050] like Figure 1 As shown, the method includes:

[0051] Step 1: Obtain unstructured data for each version.

[0052] Step 2: For each version of unstructured data, obtain the version of the semantic information based on the artificial intelligence model used to extract semantic information; construct and save the semantic information version tree of the unstructured data based on the modification relationship between each artificial intelligence model; wherein each node in the semantic information version tree represents a version of semantic information.

[0053] By treating all versions of semantic information as a tree structure, a "semantic information version tree" is constructed. The parent-child relationship between semantic information versions is mapped one-to-one to the parent-child relationship in the tree. Thus, any version of the semantic information can correspond to a node in the tree. Specifically:

[0054] 1) For a given version of semantic information, the operation of finding its sub-versions is transformed into finding all child nodes of the node corresponding to that version in the semantic information version tree. The return value is a subtree in the semantic information version tree.

[0055] 2) For a given version of semantic information, the operation of finding its parent version is transformed into finding all nodes along the path from the node corresponding to that version to the root node in the semantic information version tree. The return value is a path in the semantic information version tree.

[0056] 3) For deleting semantic information of a specific version, two types of deletion operations are supported: forced deletion and merge deletion. When forcibly deleting semantic information of a specific version, that version and all its child versions are removed from the semantic information tree. When merging deletion of semantic information of a specific version, that version is removed from the semantic information tree, and the parent node of all its direct child nodes is set to the parent node of that version.

[0057] 4) When updating the semantic information version, the newly added version is regarded as a leaf node on the semantic information version tree.

[0058] In one example, assume that initially the system only has the unstructured data version tree on the left, such as... Figure 2 As shown, there is no semantic information version tree on the right. This example explains what kind of user action will result in... Figure 2 Version tree of semantic information of unstructured data in the data.

[0059] 1) First, the user specifies an AI model with version number v-m1 to extract face information. The binary large object (Blob) with version number v-b4 is processed using v-m1 to obtain face information with version number v-sp1.

[0060] 2) Based on v-m1, upgrade the AI ​​model to v-m2. Use v-m2 to process the blob with version number v-b4 to obtain face information with version number v-sp2. Since the parent version of v-m2 is v-m1, the parent version of v-sp2 is v-sp1.

[0061] 3) Similarly, v-sp3's parent version is also v-sp1.

[0062] 4) Based on v-m3, upgrade the AI ​​model to v-m4. Use v-m4 to process the blob with version number v-b4 to obtain face information with version number v-sp4. Since the parent version of v-m4 is v-m3, the parent version of v-sp4 is v-sp3.

[0063] 5) Similarly, v-sp5's parent version is v-sp1.

[0064] 6) Based on the above steps, we can obtain Figure 2 A version tree of unstructured data semantic information (face) in the data.

[0065] The purpose of this design is to ensure that the relationship between versions of unstructured data semantic information is independent of the creation time of these versions, and is only related to the relationship between the AI ​​model that extracts the semantic information of that version and its parent model.

[0066] Step 3: Based on the storage method of the semantic information version tree, establish the expression method between parent nodes and child nodes in the semantic information version tree.

[0067] This invention proposes a method for storing and indexing semantic information version trees, enabling efficient storage and retrieval of semantic information version trees. This invention provides two methods for storing and indexing semantic information version trees: the native method and the key-value (KV) method, which are described below:

[0068] Method 1: Navigation method.

[0069] 1) Based on the semantic information version tree stored on the hard disk.

[0070] 2) Use a "three-way pointer" to maintain the association between two versions that are parent and child versions. The purpose of the pointer is to quickly find the data that the pointer points to on the disk.

[0071] 3) The data structure of the "three-way pointer" is: prev* / version-content* / next-file*. Among them, prev* points to the "three-way pointer" of the parent version of the semantic information version, version-content* points to the actual content of the semantic information of the version, and next-file* points to a file that stores pointers to all child nodes of the current version.

[0072] 4) The three parts of the three-way pointer, namely prev*, version-content*, and next-file*, are all fixed lengths, and their lengths are 16 bytes.

[0073] 5) The actual content of the semantic information is stored in files on the disk, collectively referred to as contentfiles. Within a content file, each piece of information exists in the format: blobId / blobVersionNum / semName / semContent / pointer*. Here, blobId and blobVersionNum refer to the ID and version number of the unstructured data object corresponding to the semantic information, respectively; semName refers to the name of the semantic information, such as: face, plateNum; semContent refers to the actual content of the semantic information. pointer* is a pointer to a "three-way pointer" to the semantic information.

[0074] In one example, native methods are used for storage and management. Figure 2 Methods for using version trees:

[0075] 1) Construct a "three-way pointer" for v-sp1. In this three-way pointer, the value of prev* is null because v-sp1 is the root node of the version tree and it has no parent node. The value of Version-content* depends on the size of $v-sp1. If $v-sp1 is less than 16 bytes, then the value of Version-content* is $v-sp1; if $v-sp1 is greater than 16 bytes, then the value of Version-content* is a pointer to a file that stores the actual value of $v-sp1.

[0076] 2) For the “three-way pointer” in 1), namely the three-way pointer of v-sp1, the value of its next-file* is a pointer to a file. This file stores all the pointers to the child nodes of v-sp1, namely the “three-way pointers” of v-sp2, v-sp3, and v-sp5.

[0077] 3) For v-sp3 in the diagram, its "three-way pointer" has the following values: prev* is a pointer to the three-way pointer of v-sp1; Version-content* is the value of $v-sp3, or a pointer to that value; and next-file* is a pointer to the three-way pointer of v-sp4.

[0078] 4) The pointer maintenance of the remaining nodes is similar.

[0079] Method 2: KV method.

[0080] 1) Store semantic information version trees based on key-value databases.

[0081] 2) Store semantic information with the same semantic information name in a Kv database, such as storing all face information in one database and all plateNum data in another database.

[0082] 3) In a key-value database, data is organized in key-value format, and the database name corresponds to the name of the semantic information. In this database, the key value is `verCode`, which represents the Huffman code of the version number (a well-known technique). The value is `blobId / blobVersionNum / semContent`, where `blobId` and `blobVersionNum` refer to the ID and version number of the unstructured data object corresponding to the semantic information, respectively; `semContent` refers to the actual content of the semantic information.

[0083] In one example, the key-value method is used for storage and management. Figure 2 Methods for using version trees:

[0084] Assuming the id of a Blob with version number v-b4 is id(v-b4), then the key-value method is used for storage and management. Figure 2 The method for the version tree is shown in Table 1. Each row of data in the figure corresponds to the semantic information of a version (except for the table header). From top to bottom, they are v-sp1, v-sp2, v-sp3, v-sp5, and v-sp4.

[0085] Key Value 0 id(v-b4) / vb4 / $v-sp1 0-00 id(v-b4) / vb4 / $v-sp2 0-01 id(v-b4) / vb4 / $v-sp3 0-10 id(v-b4) / vb4 / $v-sp5 0-01-0 id(v-b4) / vb4 / $v-sp4

[0086] Table 1

[0087] Step 4: When querying the semantic information of unstructured data, the semantic information version tree is retrieved based on the query filtering conditions and the expression method to obtain the query results of the semantic information.

[0088] Method 1: Navigation method.

[0089] In the Navigate algorithm, when a user performs a query related to the version of semantic information, they first need to provide filter conditions, which may include one or more of blobId, blobVersionNum, semName, and semContent. Then, the system queries version information related to that version, such as the content of its parent or child versions. Upon receiving such a request, the system first iterates through the content file to find versions of semantic information that match the filter conditions. Then, based on the value of pointer*, it finds the "three-way pointer" for that semantic information. It then recursively searches for related version information based on the values ​​of the "three-way pointer" until the value of prev* or next-file* in the "three-way pointer" is null.

[0090] One example is a query performed on versioned semantic information managed using native methods.

[0091] Suppose a user needs to find all sub-versions of the version corresponding to the value $v-spx:

[0092] 1) The system first filters the files pointed to by each "three-way pointer" in the system and finds the version with semantic information of value $v-spx. Let's assume that the version is v-sp1.

[0093] 2) Locate the "three-way pointer" of v-sp1, extract the value of next-file* from it, and find the file pointed to by next-file* based on this value, extracting all the "three-way pointers" from it.

[0094] 3) Three "three-way pointers" were retrieved in 2), namely v-sp2, v-sp3, and v-sp5. First, v-sp2, v-sp3, and v-sp5 were added to the result list.

[0095] 4) Recursively process the "three-way pointers" of v-sp2, v-sp3, and v-sp5 respectively, following the method in 2), until the value of next-file* for all newly found "three-way pointers" is null. During this process, find v-sp4 and add v-sp4 to the result list.

[0096] 5) Return the results.

[0097] Method 2: KV method.

[0098] In the key-value (KV) algorithm, when a user performs a query related to the version of semantic information, they first need to provide filter conditions, which may include one or more of the following: blobId, blobVersionNum, semName, and semContent. Then, they query version information related to that version, such as the content of its parent or child versions. Upon receiving such a request, the system first finds the versions of semantic information that match the filter conditions, and then finds the semantic information of other related versions based on the verCode of that version.

[0099] The above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit them. Those skilled in the art can modify or make equivalent substitutions to the technical solutions of the present invention without departing from the spirit and scope of the present invention. The scope of protection of the present invention should be determined by the claims.

Claims

1. A method for querying semantic information of multi-version unstructured data, the method comprising: Obtain unstructured data from various versions; For each version of unstructured data, the version of the semantic information is obtained based on the artificial intelligence model used to extract semantic information; Based on the modification relationships between various artificial intelligence models, a semantic information version tree of unstructured data is constructed and stored; wherein, the nodes in the semantic information version tree include: semantic information version and artificial intelligence model version used to extract semantic information; Based on the storage method of the semantic information version tree, establish the expression method between parent nodes and child nodes in the semantic information version tree; When querying the semantic information of unstructured data, the semantic information version tree is retrieved based on the query filtering conditions and the expression method to obtain the query results of the semantic information; Among these, based on the modification relationships between various artificial intelligence models, a semantic information version tree of unstructured data is constructed and stored, including: For a given unstructured data, obtain the semantic information version v-sp1 extracted by the artificial intelligence model with version number v-m1, and use the artificial intelligence model version number v-m1 as the root node of the semantic information version tree, and use the semantic information version v-sp1 as the child node of the root node. For this unstructured data, obtain the semantic information version v-sp2 extracted by the AI ​​model with version number v-m2, and use the AI ​​model version number v-m2 as a child node of the semantic information version v-sp1, and use the semantic information version v-sp1 as a child node of the AI ​​model version number v-m2; wherein, the parent version of the AI ​​model with version number v-m2 is the AI ​​model with version number v-m1. The storage method includes: saving the semantic information version tree on a hard disk; Where the semantic information version tree is stored on a hard disk, the step of establishing the expression method between parent nodes and child nodes in the semantic information version tree based on the storage method of the semantic information version tree includes: A three-way pointer is used to maintain the association between parent and child nodes; wherein, the three-way pointer includes: a pointer to the parent version, a pointer to the actual semantic information content, and a pointer to all child nodes; The query filtering conditions include one or more of the following: the id of the unstructured data object corresponding to the query level and the query semantic information, the version number of the unstructured data object corresponding to the query semantic information, the name of the query semantic information, and the actual content of the query semantic information; In the case where the semantic information version tree is stored on a hard disk, when querying the semantic information of unstructured data, the semantic information version tree is retrieved based on query filtering conditions to obtain the query results of the semantic information, including: Based on the id of the unstructured data object corresponding to the semantic information to be queried, the version number of the unstructured data object corresponding to the semantic information to be queried, the name of the semantic information to be queried, or the semantic information to be queried, a pointer to a three-way pointer to the semantic information to be queried is obtained. Based on the pointer to the actual semantic information content, the actual semantic information content of the target node is obtained; Using the pointers to the parent version and / or pointers to all child nodes in the target node, and based on the query level, obtain the corresponding parent node and / or child node; Based on the pointers to the actual semantic information in the parent node and / or child node, the actual semantic information of the parent node and / or child node is obtained. By combining the semantic information and actual content obtained, the query results of semantic information are obtained.

2. The method as described in claim 1, characterized in that, The storage method also includes: storing the semantic information version tree in a Key-Value database.

3. The method as described in claim 2, characterized in that, When the semantic information version tree is stored in a key-value database, the step of establishing the expression method between parent nodes and child nodes in the semantic information version tree based on the storage method of the semantic information version tree includes: Nodes with names containing the same semantic information are stored in a key-value database; Based on the relationship between parent and child nodes, the key values ​​of each version in any Kv database can be obtained.

4. The method as described in claim 3, characterized in that, When the semantic information version tree is stored in a key-value database, when querying the semantic information of unstructured data, the semantic information version tree is retrieved based on query filtering conditions to obtain the query results of the semantic information, including: Find the version of semantic information that matches the filtering criteria, and obtain the actual content of the semantic information for that version; Based on the Key value of this version and the query level, the versions of the parent node and / or child node are obtained; Obtain the semantic information of the parent node and / or child node versions; By combining the semantic information and actual content obtained, the query results of semantic information are obtained.

5. The method according to any one of claims 1-4, characterized in that, Deleting a version of semantic information from the semantic information version tree includes: Remove the corresponding nodes of this version and all its sub-versions from the semantic information tree; or, The corresponding node of this version is deleted from the semantic information tree, and the parent node of the direct child node of this version is set as the parent node of the corresponding node of this version.

6. A storage medium storing a computer program, wherein, The computer program is configured to execute any of the methods in claims 1-5 at runtime.

7. An electronic device comprising a memory and a processor, the memory storing a computer program, the processor being configured to run the computer program to perform any one of the methods of claims 1-5.

Images