Data processing method and vector database

Abstract

The invention discloses a data processing method and a vector database, and the method comprises the steps: receiving at least one first data block; the first data block is a data block determined on the basis of executing an updating operation on a target database; the target database characterizes a vector database comprising at least one data item; determining at least one first hash value corresponding to the at least one first data block in parallel; and based on the at least one first hash value, determining update information for the hash value index of the target database.

Description

Technical Field

[0001] The present application relates to but is not limited to the field of computer technology, and in particular to a data processing method and a vector database. Background Art

[0002] The Retrieval-Augmented Generation (RAG) model is a model architecture that combines retrieval and generation, and aims to enhance the generation capability of the model by accessing an external database. The implementation of RAG relies on embedded search engine technology, such as using a vector database to build a search index.

[0003] Vector databases are frequently updated and contain a large amount of high-value information, so the integrity of their content is crucial. Therefore, how to efficiently protect the data security of vector databases during frequent content updates has become an important technical challenge. Summary of the invention

[0004] In view of this, the present application at least provides a data processing method and a vector database.

[0005] The technical solution of the embodiment of the present application is implemented as follows:

[0006] On the one hand, an embodiment of the present application provides a data processing method, the method comprising:

[0007] receiving at least one first data block; the first data block is a data block determined based on performing an update operation on a target database; the target database represents a vector database including at least one data block;

[0008] Determine in parallel at least one first hash value corresponding to at least one first data block;

[0009] Based on the at least one first hash value, update information of the hash value index of the target database is determined.

[0010] In some embodiments, the image processing method further includes:

[0011] A segmentation process is performed on at least one target data item to determine at least one first data block.

[0012] In some embodiments, determining in parallel at least one first hash value corresponding to at least one first data block includes at least one of the following:

[0013] In the case where the update operation is a delete operation, determining a first hash value of the first data block from the hash value index;

[0014] When the update operation is a replace operation or an insert operation, a first hash value is determined based on identification information of the first data block and identification information of the second data block; the second data block represents a data block stored adjacent to the corresponding first data block in the target database.

[0015] In some embodiments, based on at least one first hash value, determining update information for a hash value index of a target database includes:

[0016] Determining at least one third hash value corresponding to at least one third data block; the third data block represents a data block stored adjacent to the corresponding first data block in the target database;

[0017] Based on the at least one first hash value and the at least one third hash value, update information is determined.

[0018] In some embodiments, based on at least one first hash value and at least one third hash value, determining update information includes:

[0019] Determining a root hash value based on at least one first hash value and at least one third hash value;

[0020] Update information is determined based on the at least one first hash value, the at least one third hash value, and the root hash value.

[0021] In some embodiments, the image processing method further includes at least one of the following:

[0022] Determining the integrity of the target database based on identification information and / or stored hash values ​​of at least one data block in the hash value index;

[0023] Determine in parallel at least one hash value to be verified for at least one data block in the target database; determine the integrity of the target database based on the at least one hash value to be verified and the corresponding stored hash value in the hash value index;

[0024] Determine the root hash value to be verified corresponding to the target database; and determine the integrity of the target database based on the root hash value to be verified and the stored root hash value in the hash value index.

[0025] In some embodiments, the at least one data block includes at least one of a fourth data block, a fifth data block, and a sixth data block;

[0026] The image processing method further includes at least one of the following:

[0027] In response to the to-be-checked hash value corresponding to the fourth data block being different from the stored hash value, determining that the fourth data block is illegal data;

[0028] In response to the presence of a duplicate hash value in the hash value index, determining the duplicate hash value and its corresponding data block as illegal data;

[0029] In response to the presence of duplicate hash values ​​in the hash value index, determining whether a data block corresponding to each duplicate hash value exists; determining duplicate hash values ​​for which no corresponding data block exists as illegal data;

[0030] In response to the identification information of the sixth data block appearing only in the hash value index information corresponding to the fifth data block, it is determined that the sixth data block is illegally deleted; wherein the storage hash value of the fifth data block is determined based on the identification information of the sixth data block.

[0031] In some embodiments, the target database further includes a signature index; the signature index includes signature information corresponding to each data block in the target database;

[0032] The image processing method also includes:

[0033] Determine in parallel at least one signature information corresponding to at least one first data block; the signature information is determined based on a first hash value corresponding to the first data block;

[0034] Based on the at least one signature information, a signature index is updated.

[0035] In some embodiments, the method further comprises:

[0036] Based on the stored hash value corresponding to at least one data block in the target database, at least one signature information to be verified corresponding to the at least one data block is calculated in parallel;

[0037] The integrity of the target database is determined based on at least one signature information to be verified and corresponding stored signature information in the signature index.

[0038] On the other hand, the present application provides a vector database, including a processor and a data interface; the processor is connected to the data interface; wherein,

[0039] The processor receives at least one first data block through a data interface; determines in parallel at least one first hash value corresponding to the at least one first data block; and determines update information of a hash value index for a target database based on the at least one first hash value;

[0040] The first data block is a data block determined based on performing an update operation on the vector database.

[0041] It should be understood that the above general description and the following detailed description are merely exemplary and explanatory, and are not intended to limit the technical solutions of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] The drawings herein are incorporated into the specification and constitute a part of the specification. These drawings illustrate embodiments consistent with the present application and are used together with the specification to illustrate the technical solution of the present application.

[0043] Figure 1 A schematic diagram of the implementation flow of a data processing method provided in this application;

[0044] Figure 2 A schematic diagram of the implementation flow of another data processing method provided by the present application;

[0045] Figure 3 A schematic diagram of determining a root hash value in response to a deletion operation according to the data processing method provided by the present application;

[0046] Figure 4 A schematic diagram of determining a root hash value in response to a replacement operation according to the data processing method provided by the present application;

[0047] Figure 5 A schematic diagram of determining a root hash value in response to an insert operation according to the data processing method provided by the present application;

[0048] Figure 6 A schematic diagram of an implementation flow of an embodiment of performing a hash value index update according to the data processing method provided by the present application;

[0049] Figure 7 A schematic diagram of an implementation flow of an embodiment of performing a database integrity check according to the data processing method provided by the present application;

[0050] Figure 8 A hardware entity schematic diagram of a vector database provided in this application;

[0051] Fig. 9 A schematic diagram of the interaction between the modules in the vector database provided in this application. DETAILED DESCRIPTION

[0052] In order to make the purpose, technical solutions and advantages of the present application clearer, the technical solutions of the present application are further elaborated in detail below in conjunction with the drawings and embodiments. The described embodiments should not be regarded as limiting the present application. All other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present application.

[0053] In the following description, reference is made to “some embodiments”, which describe a subset of all possible embodiments, but it will be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0054] The terms "first / second / third" involved are merely used to distinguish similar objects and do not represent a specific ordering of the objects. It is understandable that "first / second / third" can be interchanged with a specific order or sequence where permitted so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein.

[0055] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing this application and are not intended to limit this application.

[0056] The present application provides a data processing method, which can be executed by a processor of a computer device, wherein the computer device may refer to a device with data processing capabilities, such as a server, a laptop, a tablet computer, a desktop computer, a mobile device, etc.

[0057] Figure 1 A schematic diagram of the implementation flow of a data processing method provided in this application, such as Figure 1 As shown, the method includes the following steps S101 to S103:

[0058] Step S101, receiving at least one first data block; the first data block is a data block determined based on performing an update operation on a target database; the target database represents a vector database including at least one data block.

[0059] Here, the target database is a vector database that can store, manage and query high-dimensional vector data.

[0060] In some embodiments, the target database is a RAG database. The RAG database has the characteristics of being not fixed in size and changing in real time, and the changes of the RAG database (ie, the types of update operations) are diverse.

[0061] Data block refers to the storage method of vector data in the target database. In some embodiments, the data block can be composed of multiple floating point numbers or integers.

[0062] In some embodiments, a head pointer is initialized for the target database to point to the first data block, and the last block pointer position is empty.

[0063] The update operation refers to the operations such as deletion, replacement and / or insertion performed on the vector data in the target database. Here, when the update operation is a deletion operation, the first data block refers to the deleted data block; when the update operation is a replacement operation, the first data block refers to the data block used to replace the data block currently stored in the target database; when the update operation is an insertion operation, the first data block is the inserted data block.

[0064] Step S102: Determine in parallel at least one first hash value corresponding to the at least one first data block.

[0065] Here, the hash value is verification information generated for the data block stored in the target database using a specified hash algorithm. That is, the hash value can be used to verify the legitimacy or integrity of the corresponding data block, and accordingly, the hash value index of the target database can be used to verify the integrity of the target database.

[0066] In some embodiments, the specified hash algorithm is an exclusive-OR hash algorithm (XOR-hash).

[0067] After receiving at least one first data block, at least one first hash value corresponding to the at least one first data block is calculated; and when there are multiple first data blocks, multiple first hash values ​​of the multiple first data blocks are respectively determined by parallel processing technology.

[0068] In some embodiments, the parallel processing technology refers to a technology for implementing parallel processing in space, that is, by using multiple processors or computing nodes to simultaneously calculate the first hash values ​​corresponding to multiple first data blocks.

[0069] In some embodiments, the parallel processing technology refers to Single Instruction Multiple Data (SIMD) technology. SIMD technology uses a controller to control multiple processors to process multiple data simultaneously, thereby achieving spatial parallelism.

[0070] Here, through parallel processing, the calculation efficiency of at least one first hash value of at least one data block can be improved to adapt to the characteristic of frequent updates of the target database.

[0071] Step S103: determining update information of the hash value index of the target database based on the at least one first hash value.

[0072] Here, after determining at least one first hash value, update information for the hash value index is determined based on the at least one first hash value. For example, when the update operation is a delete operation, the first hash value of the first data block is deleted from the hash value index; when the update operation is a data replacement operation, the replaced hash value in the hash value index is replaced with the first hash value of the corresponding first data block; when the update operation is an insert operation, the first hash value of the first data block is inserted into the hash value index.

[0073] In the data processing method provided by the present application, after receiving at least one first data block determined based on an update operation performed on a target database, at least one first hash value of the at least one first data block is determined in parallel, and update information of a hash value index for the target database is determined based on the at least one first hash value. In this way, on the one hand, by determining at least one first hash value in a parallel processing manner, the calculation efficiency of the hash value can be significantly improved; on the other hand, after an update operation occurs on the target database, the hash value index of the target database is updated based on the hash value of the first data block related to the update operation, without the need to calculate the hash values ​​of all data blocks in the target database, which can reduce the computing resources and time required for hash value calculation, thereby improving the update efficiency of the hash value index.

[0074] In some embodiments, before receiving at least one first data block, that is, before the above step S101, the data processing method further includes the following step S104:

[0075] Step S104: performing segmentation processing on at least one target data item to determine the at least one first data block.

[0076] Here, the target data item refers to the high-dimensional vector data stored in the target database.

[0077] When determining the first hash value of the first data block, it is necessary to use the register of the corresponding processor (for example, the SIMD register) to store the first data block, and the target data item is usually high-dimensional vector data with a large amount of data. Therefore, the target data item is segmented and at least one first data block is determined so that the data amount of the first data block can match the storage capacity of the register.

[0078] In some embodiments, at least one target data item is determined based on at least one update operation on the target database, for example, based on a delete operation, a replace operation or an insert operation on the target database.

[0079] In the above embodiment, when update operations for multiple target data items occur in the target database, hash value updates for the multiple target data items can be processed in parallel; and, through a splitting operation, the target data items are split into multiple first data blocks, and the first hash values ​​of the multiple first data blocks are determined in parallel using parallel processing technology, that is, parallel processing at the data block level is implemented, further improving the data processing speed.

[0080] In some embodiments, the parallel determination of at least one first hash value corresponding to the at least one first data block, that is, the above step S102, can be implemented as at least one of the following steps S1021 to S1022:

[0081] Step S1021: when the update operation is a delete operation, determine a first hash value of the first data block from the hash value index.

[0082] Here, when the update operation is a delete operation, the corresponding first data block is the deleted data block. Therefore, when determining the first hash value of the first data block, it can be determined from the hash value index of the target database.

[0083] Step S1022, when the update operation is a replacement operation or an insert operation, determine the first hash value based on the identification information of the first data block and the identification information of the second data block; the second data block represents a data block stored adjacent to the corresponding first data block in the target database.

[0084] When the update operation is a replacement operation or an insert operation, the first data block is a replaced data block or an inserted data block, and therefore the hash value of the first data block needs to be recalculated.

[0085] Identification information refers to information used to uniquely identify a data block in a target database.

[0086] In some embodiments, the target database generates corresponding identification information for each first data block using a random number generator.

[0087] In some embodiments, the target database generates corresponding identification information based on the first data block using a specified hash algorithm. For example, the target database calculates the identification information of the first data block using a secure hash algorithm (SHA-256).

[0088] Since each data block in the target database is a high-dimensional vector, it can be considered that the data blocks in the target database are different from each other. Therefore, the identification information generated for each data block using a random number generator or a hash algorithm is also different from each other.

[0089] In some embodiments, the target database creates a corresponding data block index for each first data block, and the data block index includes identification information corresponding to the first data, a timestamp determined based on the update time of the update operation corresponding to the first data block, and identification information of the second data block stored adjacent to the first data block.

[0090] In some embodiments, the data block index corresponding to the first data block is associated with its hash value and stored in the hash value index.

[0091] In some embodiments, the second data block is a data block whose storage location is before the first data block.

[0092] In this way, the first hash value of the first data block is determined based on the data block index of the first data block, that is, the identification information of the second data block stored adjacent to the first data block is used as a calculation parameter for calculating the hash value of the first data block.

[0093] For example, when the second data block is a data block whose storage position is before the first data block, the first hash value of the first data block can be calculated using the following formula (1):

[0094] H i =sha256(sha256(M i ||ID i-1 ||t i ||ID i ),key) (1);

[0095] Among them, H i represents the first hash value of the first data block; sha256 represents the sha-256 hash algorithm; M i Indicates the first data block; ID i-1 Indicates the identification information of the second data block; ID i Indicates identification information of the first data block; i Indicates the update time of the update operation corresponding to the first data block, that is, the timestamp of the first data block; key indicates the hash key.

[0096] In some embodiments, the target database generates a hash key using a random number generator. Here, only legitimate users of the target database can obtain and use the hash key.

[0097] In the above embodiment, the first hash value of the first data block is calculated using the identification information of the second data block stored adjacent to the first data block, so that different data blocks in the target database are associated with each other using the identification information of the data block to form a chain structure. At the same time, since the data association between adjacently stored data blocks is strong, when performing data retrieval on the target database, the chain structure can be used to assist in improving the retrieval efficiency and accuracy. In addition, the hash value of the data block is determined using the hash key as a parameter, and the hash key can only be obtained and used by legitimate users, so the security of the hash calculation can be improved.

[0098] In some embodiments, Figure 2 As shown, the data processing method provided by the present application can be implemented as the following steps S201 to S204:

[0099] Step S201, receiving at least one first data block; the first data block is a data block determined based on performing an update operation on a target database; the target database represents a vector database including at least one data block;

[0100] Step S202, determining in parallel at least one first hash value corresponding to the at least one first data block;

[0101] Step S203, determining at least one third hash value corresponding to at least one third data block; the third data block represents a data block stored adjacent to the corresponding first data block in the target database;

[0102] Step S204: Determine the update information based on the at least one first Hash value and the at least one third Hash value.

[0103] The above steps S201 and S202 correspond to the above steps S101 and S102, respectively. Therefore, the specific implementation of steps S201 and S202 can refer to the above descriptions of steps S101 and S102, respectively.

[0104] The third data block is a data block stored adjacent to the first data block, for example, a data block stored after the first data block.

[0105] Here, when at least one first hash value of at least one first data block is determined in parallel, a third hash value of a third data block corresponding to each first data block is determined accordingly.

[0106] For example, when the update operation corresponding to the first data block is a delete operation, the specific calculation method of the third hash value is as shown in the following formula (2):

[0107] H′ i+1 =sha256(sha256(M i+1 ||ID i-1 ||t′ i+1 ||ID i+1 ),key) (2);

[0108] Among them, H′ i+1 represents the third hash value of the third data block; M i+1 Indicates the third data block; ID i-1 Indicates the identification information of the data block whose storage location is before the first data block; t′ i+1 Indicates the operation time of the deletion operation on the first data block; ID i+1 Indicates identification information of the third data block.

[0109] It can be seen that after deleting the first data block, a data block whose original storage position is before the first data block (for example, the second data block mentioned above) becomes a data block whose current storage position is before the third data block. Therefore, the third hash value of the third data block is calculated using the identification information of the second data block.

[0110] For another example, when the update operation corresponding to the first data block is a replace operation, the specific calculation method of the third hash value is as shown in the following formula (3):

[0111] H′ i+1 =sha256(sha256(M i+1 ||ID i '||t′ i+1 ||ID i+1 ),key) (3);

[0112] Among them, H′ i+1 represents the third hash value of the third data block; M i+1 Indicates the third data block; ID' i Indicates the identification information corresponding to the first data block; t' i+1 Indicates the operation time of the replacement operation for the first data block; ID i+1 Indicates identification information of the third data block.

[0113] It can be seen that after the first data block is used to replace the original data block whose storage position is before the third data block, the third hash value of the third data block is recalculated using the identification information of the first data block.

[0114] For another example, when the update operation corresponding to the first data block is an insert operation, that is, when the first data block is inserted before the third data block, the specific calculation method of the third hash value is as shown in the following formula (4):

[0115] H′ i+1 =sha256(sha256(M i+1 ||ID i ||t′ i+1 ||ID i+1 ),key) (4);

[0116] Among them, H′ i+1 represents the third hash value of the third data block; M i+1 Indicates the third data block; ID i Indicates the identification information corresponding to the first data block; t' i+1 Indicates the operation time of the insert operation for the first data block; ID i+1 Indicates identification information of the third data block.

[0117] It can be seen that when the update operation is to insert the first data block before the third data block, the third hash value of the third data block is recalculated using the identification information of the inserted first data block.

[0118] In this way, after determining the third hash value of the third data block stored adjacent to the first data block, update information for the hash value index is determined based on the first hash value and the third hash value.

[0119] For example, when the update operation is a delete operation, the first hash value in the hash value index is deleted, and the original hash value corresponding to the third data block in the hash value index is updated using the third hash value; when the update operation is a replace operation, the hash value of the data block replaced by the first data block in the hash value index is replaced using the first hash value, and the original hash value corresponding to the third data block in the hash value index is updated using the third hash value; when the update operation is an insert operation, the first hash value is inserted before the hash value corresponding to the third data item in the hash value index, and the original hash value corresponding to the third data block in the hash value index is updated using the third hash value.

[0120] In some embodiments, the determining of the update information based on the at least one first hash value and the at least one third hash value, that is, the above step S204, can be implemented as the following steps S2041 to S2042:

[0121] Step S2041, determining a root hash value based on the at least one first hash value and the at least one third hash value;

[0122] Step S2042: Determine the update information based on the at least one first hash value, the at least one third hash value and the root hash value.

[0123] In some embodiments, the root hash value in the hash value index of the target database is determined based on the hash value of each data block using a specified hash algorithm.

[0124] In some embodiments, the root hash value is calculated using an XOR hash algorithm, as shown in the following formula (5):

[0125]

[0126] Among them, H root Represents the root hash value; H n Represents the hash value of the nth data block; N is an integer greater than 1.

[0127] It can be seen that the root hash value of the target database is directly calculated using the hash value of each data block, that is, the generated hash value index only includes each leaf node and the root node, and does not involve other nodes between the leaf node and the root node. Therefore, after updating the hash value of the leaf node, the hash value of the root node can be directly calculated based on the update operation, without calculating the hash values ​​of other nodes between the leaf node and the root node, thereby simplifying the update steps of the hash value index.

[0128] In this way, after determining at least one first hash value of at least one first data block and at least one third hash value of at least one third data block based on the update operation, the hash value of the root node can be directly calculated based on the at least one first hash value and the at least one third hash value.

[0129] For example, when the update operation is a delete operation, the root hash value can be calculated using the following formula (6):

[0130]

[0131] Among them, H′ root Represents the updated root hash value; H root Indicates the root hash value before update; H i Indicates the hash value of the first data block to be deleted; H′ i+1 represents the third hash value of the third data block; H i+1 Indicates the hash value of the third data block before update.

[0132] Thus, in the operation of deleting the first data block, the process of calculating the first hash value, the third hash value and the root hash value can be as follows: Figure 3 shown.

[0133] exist Figure 3 In the example, before the deletion operation, the target database includes a portion of data blocks 310, including: data block M i 312 (i.e., the first data block), data block M i-1 312 (i.e., the second data block stored before the first data block) and data block M i+1 313 (i.e., the third data block stored after the first data block). i-1 312 includes data block information M of this data block i-1 , identification information ID i-1 , timestamp t i-1 And the identification information ID of the previous data block i-2 ; Data block M i 312 includes data information M of this data block i , identification information ID i , timestamp t i And the identification information ID of the previous data block i-1 ; Data block M i+1 313 includes data information M of this data block i+1 , identification information ID i+1 , timestamp t i+1 And the identification information ID of the previous data block i .

[0134] After the deletion operation is performed on the first data block, the original partial data block 310 is updated to a partial data block 320, which includes the data block M. i-1 321 and data block M i+1 323, where the dotted line represents the data block M i 322 is the first data block to be deleted.

[0135] Thus, when calculating the updated root hash value, the original hash value H stored in the hash value index is root In 360, delete the deleted data block M i The hash value H of 322 i 340 information, add data block M i+1 The updated hash value H of 323 i+1 '350 information, and delete data block M i+1 The hash value H before the update of 323 i+1 330, thereby determining the updated root hash value H root '370.

[0136] For another example, when the update operation is a replacement operation, the root hash value can be calculated using the following formula (7):

[0137]

[0138] Among them, H′ root Represents the updated root hash value; H root Indicates the root hash value before update; H i The hash value of the data block replaced by the first data block; H′ i Represents the hash value of the first data block; H′ i+1 represents the third hash value of the third data block; H i+1 Indicates the hash value of the third data block before update.

[0139] Thus, in the operation of replacing a data block before the third data block with the first data block, the process of calculating the root hash value can be as follows: Figure 4 shown.

[0140] exist Figure 4 In the example, before the replacement operation, the target database includes a portion of data blocks 410, including: data block M i 411 (i.e., the original data block replaced by the first data block) and data block M i+1 412 (i.e., the third data block before updating). i 411 includes data information M of this data block i , identification information ID i, timestamp t i And the identification information ID of the previous data block i-1 ; Data block M i+1 412 includes data information M of this data block i+1 , identification information ID i+1 , timestamp t i+1 And the identification information ID of the previous data block i .

[0141] After the corresponding original data is replaced by the first data block, the partial data block 410 is updated to the partial data block 420, including: data block M i '421 (ie, the first data block) and data block M i+1 '422 (ie, the updated third data block). Wherein, data block M i '421 includes data information M of this data block i ', identification information ID i ', timestamp t i 'And the identification information ID of the previous data block i-1 ; Data block M i+1 '422 includes data information of this data block M i+1 , identification information ID i+1 , timestamp t i+1 'And the identification information ID of the previous data block i '.

[0142] Thus, when calculating the updated root hash value, the original hash value H stored in the hash value index is root In 470, the replaced original data block M is deleted i The hash value H of 411 i 430 information, delete the data block M before the update i+1 The hash value H of 412 i+1 440 information, add data block M i '421 hash value H i '450 information, and add the updated data block M i+1 '422 hash value H i+1 '460 information, thereby determining the updated root hash value H root '480.

[0143] For another example, when the update operation is an insert operation, the root hash value can be calculated using the following formula (8):

[0144]

[0145] Among them, H′ root Represents the updated root hash value; Hroot Indicates the root hash value before update; H p represents the hash value of the data block (i.e., the first data block) inserted before the third data block; H m Indicates the hash value of the third data block before updating; H' m Represents the updated hash value of the third data block (ie, the third hash value).

[0146] Thus, in the operation of inserting the first data block before the third data block, the process of calculating the root hash value can be as follows: Figure 5 shown.

[0147] exist Figure 5 In the example, before the insert operation, the target database includes a portion of data blocks 510, including: data block M m- 1511 and data block M m 512 (i.e., the third data block before updating). m-1 511 includes data information M of this data block m-1 , the identification information ID of the previous data block m-2 , timestamp t m And the ID of this data block m-1 ; Data block M m 512 includes data information M of this data block m , the identification information ID of the previous data block m+1 , timestamp t m And the ID of this data block m .

[0148] In data block M m Insert data block M before 512 p (ie, the first data block) after which the partial data block 510 is updated to a partial data block 520, the partial data block 520 comprising: data block M m-1 521. Data Block M p 522 and data block M m 523 (i.e., the updated third data block). m-1 521 contains information and data block data block M m-1 511 same; data block M p 522 includes data information M of this data p , the identification information ID of the previous data block m-1 , timestamp information t p And the ID of this data block p ; The updated third data block, i.e. data block M m 523 includes data information M of this data block m, the identification information ID of the previous data block p , updated timestamp information t m 'And the identification information ID of this data block m .

[0149] Thus, when calculating the updated root hash value, the original hash value H stored in the hash value index is root In 560, the data block M before the update is deleted m The hash value H m 530 information, and increase the inserted data block M p The hash value H p 540 information and updated data block M m The hash value H m '550 information, thereby obtaining the updated root hash value H root '570.

[0150] It can be seen from the above embodiments that when an update operation occurs in the target database, the data processing method provided by the present application completes the update of the hash value index by updating the hash values ​​of the updated data block and its adjacent data blocks, and using the hash values ​​of the updated data block and its adjacent data blocks to update the root hash value, without involving the update of the hash values ​​of other nodes between the data block and the root hash value in the entire hash value index. Therefore, the amount of hash value calculation can be reduced and the update speed of the hash value index can be improved. At the same time, when an update operation occurs for multiple data blocks, the hash value updates of multiple data blocks can be processed synchronously and in parallel, further improving the update speed of the hash value index.

[0151] As described above, the hash value index includes at least a data block index and a hash value corresponding to each data block, wherein the data block index includes corresponding data block information, a timestamp, identification information of the current data block, and identification information of the previous data block stored adjacently.

[0152] Thus, in some embodiments, the data processing method provided by the present application further includes at least one of the following steps S105 to S106:

[0153] Step S105: determining the integrity of the target database based on identification information and / or stored hash value of at least one data block in the hash value index.

[0154] Here, since the data block index includes the identification information of the current data block and the identification information of the previous data block, except for the first data block and the last data block in the hash value index, the identification information of other data blocks in the hash value index will appear twice in the hash value index.

[0155] In this way, when the identification information of the data blocks stored in the hash value index appears twice except for the first data block and the last data block, the target database data is determined to be complete; otherwise, it is determined that malicious update behavior occurs in the target database.

[0156] In some embodiments, when the hash values ​​stored in the hash value index are different from each other, it is determined that the target database data is complete; otherwise, it is determined that malicious update behavior occurs in the target database.

[0157] Step S106, determining in parallel at least one to-be-verified hash value of at least one data block in the target database; and determining the integrity of the target database based on the at least one to-be-verified hash value and the corresponding stored hash value in the hash value index.

[0158] In some embodiments, when calculating the hash value of a data block, a pre-stored hash key needs to be used, and the hash key is only available to legitimate database users. In this way, since illegal users cannot obtain the hash key, they cannot generate the correct hash value.

[0159] Therefore, a legitimate user can recalculate the hash value of at least one data block based on the index information and hash key of at least one data block stored in the hash value index to obtain at least one hash value to be verified; and compare the hash value to be verified with the corresponding stored hash value in the hash value index to see if they are the same. If they are the same, it proves that the target database data is complete; if they are different, it proves that the target database has been maliciously updated.

[0160] Here, for multiple data blocks in the hash value index, parallel processing technology is used to parallely calculate multiple hash values ​​to be verified, and then compare them with the corresponding stored hash values, which can improve the efficiency of hash value verification, that is, improve the efficiency of target database integrity verification.

[0161] Step S107, determining the root hash value to be verified corresponding to the target database; and determining the integrity of the target database based on the root hash value to be verified and the stored root hash value in the hash value index.

[0162] Here, on the basis of recalculating at least one hash value to be verified of at least one data block, the root hash value to be verified can also be recalculated to compare whether the root hash value to be verified is the same as the root hash value stored in the hash value index; if they are the same, it is determined that the target database data is complete; if they are not the same, it is determined that the target database has been maliciously updated.

[0163] In some embodiments, an integrity check is performed on the target database periodically or in response to a data access operation to the target database.

[0164] Based on the above embodiment of determining the integrity of the target database, a malicious database attack may be determined by at least one of the following steps S108 to S1011; wherein the target database includes at least one of the fourth database, the fifth database and the sixth database.

[0165] Step S108: In response to the fact that the to-be-checked hash value corresponding to the fourth data block is different from the stored hash value, determining that the fourth data block is illegal data.

[0166] Here, when an illegal user uses the fourth data block to replace the replaced data block in the target database, since the illegal user cannot obtain the hash key, the illegal user cannot correctly calculate the hash value of the fourth data block, resulting in the hash value corresponding to the fourth data block in the hash value index still being the hash value of the replaced data block.

[0167] In this way, by recalculating the hash value of the fourth data block, and the recalculated hash value to be checked is different from the hash value corresponding to the fourth data block stored in the hash value index, it can be determined that the fourth data block is an illegal data block.

[0168] Step S109: In response to the existence of duplicate hash values ​​in the hash value index, determining that the duplicate hash values ​​and their corresponding data blocks are illegal data.

[0169] Here, when an illegal user uses the data block index and hash value of a legal data block in the target database to replace the data block index and hash value of other legal data blocks, the hash value to be verified corresponding to each data block is the same as the stored hash value, but the problem of hash value duplication will occur.

[0170] Therefore, when there are duplicate hash values ​​in the hash value index, the duplicate hash values ​​and their corresponding data blocks and data block indexes are all determined to be illegal data.

[0171] Step S1010: In response to the existence of duplicate hash values ​​in the hash value index, determine whether a data block corresponding to each duplicate hash value exists; and determine the duplicate hash value without a corresponding data block as illegal data.

[0172] Here, as described above, the hash value of the first data block is determined based on the identification information of the first data block and the identification information of the second data block stored adjacent to the first data block. Therefore, after the first data block is illegally replaced, the identification information of the first data block stored in the hash index of the second data block will not have a corresponding data block index.

[0173] Therefore, when an illegal user uses the data block index and hash value of a legitimate data block in the target database to replace the data block index and hash value of other legitimate data blocks, duplicate hash values ​​and data block indexes will appear in the hash value index. At this time, it can be determined whether the data block corresponding to each duplicate hash value exists. If not, it is determined to be illegal data.

[0174] For example, if an illegal user replaces the first data block {M1, null, t1, ID1} with the second data block {M2, ID1, t2, ID2}, a duplicate hash value and data block index of the second data block will appear in the hash value index. At this time, it is possible to check whether the data block pointed to by the identification information ID1 associated with the second data block in the data block index exists, and by verifying that the data block pointed to by ID1 does not exist, it is determined that the hash value corresponding to the second data block is illegal data.

[0175] Step S1011, in response to the identification information of the sixth data block only appearing in the hash value index information corresponding to the fifth data block, determining that the sixth data block is illegally deleted; wherein the storage hash value of the fifth data block is determined based on the identification information of the sixth data block.

[0176] As described above, except for the first data block and the last data block, the identification information of each data block in the target database will appear in both the data block index of the data block and the data block index of the adjacent data block. Therefore, when the storage hash value of the fifth data block is determined based on the identification information of the sixth data block, and the identification information of the sixth data block only appears in the data block index corresponding to the fifth data block, it is determined that the sixth data block is illegally deleted.

[0177] In some embodiments, after determining that the target database is maliciously updated or illegal data is determined in the target database, the malicious update event or illegal data is recorded.

[0178] In some embodiments, depending on the type and severity of the malicious update event, or the importance of the illegal data, a corresponding response strategy is executed, such as alarm, restricted access or data recovery, etc., to ensure that the target database can be quickly restored and prevent further damage.

[0179] In some embodiments, when malicious update events for multiple data blocks occur, parallel processing technology can be used to record multiple malicious update events in parallel, and corresponding response strategies can be executed to improve the efficiency of handling abnormal events.

[0180] In some embodiments, the target database further includes a signature index; the signature index includes signature information corresponding to each data block in the target database.

[0181] Here, signature information refers to the electronic signature of data through cryptographic technology, which can be used to verify the authenticity and integrity of the data.

[0182] In some embodiments, based on the hash value of the data block and the signature private key, a specified signature algorithm is used to generate corresponding signature information for each data block. q , the data block M can be determined using the following formula (9): q Signature information:

[0183] σ(M q )=sign(H q ,SK) (9);

[0184] Among them, σ(M q ) indicates data block M q Signature; sign represents the signature algorithm; H q Indicates data block M q SK represents the signature private key.

[0185] At the same time, the combined signature of multiple data blocks in the target database can be determined using the following formula (10):

[0186]

[0187] Among them, σ(M) represents the combined signature; σ(M n ) represents the signature of the nth database; N is an integer greater than 1.

[0188] Thus, the data processing method provided by the present application further includes the following steps S1012 to S1013:

[0189] Step S1012, determining in parallel at least one signature information corresponding to the at least one first data block; the signature information is determined based on a first hash value corresponding to the first data block;

[0190] Step S1013: Update the signature index based on the at least one signature information.

[0191] Here, after determining at least one first data block based on the update operation on the target database, at least one signature information corresponding to the at least one first data block is determined in parallel using parallel processing technology.

[0192] In some embodiments, the parallel processing technique for determining at least one signature information of at least one first data block and the parallel processing technique for determining at least one first hash value may be the same or different.

[0193] After determining at least one signature information of at least one first data block, the signature index is updated based on the at least one signature information.

[0194] For example, when the operation corresponding to the first data block is a deletion operation, the signature information corresponding to the first data block is deleted from the signature index.

[0195] For another example, when the operation corresponding to the first data block is a replacement operation, in the signature index, the signature information of the first data block is used to replace the signature information of the data block replaced by the first data block.

[0196] For another example, when the operation corresponding to the first data block is an insert operation, the signature information of the first data block is inserted into the corresponding position in the signature index.

[0197] In some embodiments, updating the signature index based on at least one signature information further includes: calculating a combined signature based on at least one signature information of at least one first database; and updating the combined signature stored in the signature index using the calculated combined signature.

[0198] In the above embodiment, when data is updated in the target database, it is only necessary to calculate the signature information of the data block corresponding to the update operation, and update the signature index based on the signature information, thereby improving the update speed of the signature index of the target database; in addition, when there are multiple data blocks corresponding to the update operation, the signature information of each data block is calculated in parallel processing, which can further improve the update speed of the signature information.

[0199] In this way, in the case where the target database contains the signature index, the integrity of the target database can be verified based on the signature index.

[0200] Therefore, the data processing method provided by the present application further includes the following steps S1014 to S1015:

[0201] Step S1014, based on the stored hash value corresponding to at least one data block in the target database, at least one signature information to be verified corresponding to the at least one data block is calculated in parallel;

[0202] Step S1015: Determine the integrity of the target database based on the at least one signature information to be verified and the corresponding stored signature information in the signature index.

[0203] Here, when the integrity of the target database is verified, the hash value index of the target database is obtained, and based on the stored hash value corresponding to at least one data block in the hash value index, at least one signature information to be verified corresponding to the at least one data block is calculated in parallel; then, it is determined whether the calculated at least one signature information to be verified is the same as the corresponding stored signature information in the signature index; if they are the same, it is determined that the target database data is complete; if they are not the same, it is determined that the target database has been maliciously updated.

[0204] Next, combine Figure 6 , an embodiment of updating the hash value index of the target database according to the data processing method provided by the present application is described. Figure 6 As shown, this embodiment includes the following steps S601 to S611:

[0205] Step S601, receiving an update operation on a target data item; then, executing step S602;

[0206] Step S602, determining the update operation type; then, when the update operation type is an insert operation, executing step S603; when the update operation type is a replace operation, executing step S606; when the update operation type is a delete operation, executing step S610;

[0207] Step S603, dividing the target data item into a plurality of target data blocks; then, executing step S604;

[0208] Here, according to the capacity of the SIMD register, the target data item is divided into multiple target data blocks, and corresponding identification information and timestamp are generated for each target data block.

[0209] Step S604, calculating multiple hash values ​​of multiple target data blocks in parallel; then, executing step S605;

[0210] Here, SIMD instructions are used to calculate multiple hash values ​​of multiple target data blocks in parallel.

[0211] Step S605, updating the hash value index based on multiple hash values ​​of multiple target data blocks;

[0212] Here, based on multiple hash values ​​of multiple target data blocks, hash values ​​of adjacent data blocks of each target data block are calculated; based on the hash values ​​of each target data block and its adjacent data blocks, a root hash value is calculated; based on the hash values ​​of multiple target data blocks and their adjacent data blocks, and the calculated root hash value, the hash value index is updated.

[0213] Step S606, obtaining the replaced data item; then, executing step S607;

[0214] Here, multiple data blocks corresponding to the replaced data item and their corresponding hash values ​​and data block index information are obtained.

[0215] Step S607, dividing the target data item into a plurality of target data blocks; then, executing step S608;

[0216] Here, according to the capacity of the SIMD register, the target data item is divided into multiple target data blocks, and corresponding identification information and timestamp are generated for each target data block.

[0217] Step S608, calculating multiple hash values ​​of multiple target data blocks in parallel; then, executing step S609;

[0218] Here, SIMD instructions are used to calculate multiple hash values ​​of multiple target data blocks in parallel.

[0219] Step S609, updating the hash value index based on multiple hash values ​​of multiple target data blocks and the hash value of the replaced data item;

[0220] Here, based on multiple hash values ​​of multiple target data blocks, hash values ​​of adjacent data blocks of each target data block are calculated; based on the hash values ​​of each target data block and its adjacent data blocks, and the hash value corresponding to each data block of the replaced data item, a root hash value is calculated; based on the hash values ​​of each target data block and its adjacent data blocks, and the calculated root hash value, the hash value index is updated.

[0221] Step S610, determining a plurality of target data blocks corresponding to the target data item; then, executing step S611;

[0222] Here, in the case where the update operation is a delete operation, the target data item is a deleted data item, or some data blocks in the target data item are deleted objects.

[0223] Step S611: remove the hash value and data block index corresponding to at least one target data block from the hash value index.

[0224] Here, at least one target data block to be deleted is determined from multiple target databases of the target data item, and the hash value corresponding to the at least one target data block is removed from the hash value index; at the same time, based on the hash values ​​of the at least one removed target data block and its adjacent data blocks, the root hash value in the hash value index is calculated and updated.

[0225] Next, combine Figure 7 , an embodiment of performing integrity verification on a target database according to the data processing method provided by the present application is described. Figure 7 As shown, this embodiment includes the following steps S701 to S706:

[0226] Step S701, start integrity check; then, execute step S702;

[0227] Step S702, obtaining the current hash value index; then, executing step S703;

[0228] Step S703, calculating in parallel at least one to-be-verified hash value of at least one data block; then, executing step S704;

[0229] Here, the parallel processing technology is used to utilize the hash values ​​of multiple data blocks of multiple data items.

[0230] Step S704, determining whether the hash value to be verified is the same as the corresponding stored hash value in the hash value index; if so, executing step S705; if not, executing step S706;

[0231] Step S705, confirming that the database data is complete;

[0232] Step S706, record the unmatched hash value and report the abnormal event.

[0233] Based on the aforementioned embodiments, the present application further provides a vector database, and each unit included in the vector database and each module included in each unit can be implemented by a processor.

[0234] Next, combine Figure 8 , the vector database provided by this application is described. Figure 8 As shown, the vector database 800 includes a processor 810 and a data interface 820; the processor 810 is connected to the data interface 820; wherein,

[0235] The processor 810 receives at least one first data block through the data interface 820; determines at least one first hash value corresponding to the at least one first data block in parallel; and determines update information for a hash value index of a vector database based on the at least one first hash value;

[0236] The first data block is a data block determined based on performing an update operation on the vector database.

[0237] In some embodiments, the vector database 800 includes multiple functional modules running on the processor 810 .

[0238] In some embodiments, the vector database 800 includes an incremental hash module; the incremental hash module is used to receive at least one first data block; determine at least one first hash value corresponding to the at least one first data block in parallel; based on the at least one first hash value, determine update information for the hash value index of the vector database; wherein the first data block is a data block determined based on performing an update operation on the vector database.

[0239] In some embodiments, the vector database 800 includes a content management module; the content management module performs segmentation processing on at least one target data item to determine at least one first data block.

[0240] Here, the content management module is responsible for the daily maintenance and content update operations of the vector database. For example, the content management module can be used to generate hash keys, signature private keys, data block identification information and timestamps, etc.

[0241] In some embodiments, when the vector database system is deployed or restarted, the content management module scans all data in the vector database and checks for update information of the vector data.

[0242] In some embodiments, the enhanced hash module is further configured to perform at least one of the following:

[0243] In the case where the update operation is a delete operation, determining a first hash value of the first data block from the hash value index;

[0244] When the update operation is a replace operation or an insert operation, a first hash value is determined based on identification information of the first data block and identification information of the second data block; wherein the second data block represents a data block stored adjacent to the corresponding first data block in the vector database.

[0245] In some embodiments, the incremental hash module is further configured to:

[0246] Determine at least one third hash value corresponding to at least one third data block; wherein the third data block represents a data block stored adjacent to the corresponding first data block in the vector database;

[0247] Based on the at least one first hash value and the at least one third hash value, update information is determined.

[0248] In some embodiments, the incremental hash module is used to:

[0249] Determining a root hash value based on at least one first hash value and at least one third hash value;

[0250] Update information is determined based on the at least one first hash value, the at least one third hash value, and the root hash value.

[0251] In some embodiments, the vector database 800 further includes an integrity check module; the integrity check module is used to perform at least one of the following:

[0252] Determining integrity of the vector database based on identification information of at least one data block in the hash value index and / or a stored hash value;

[0253] Determine in parallel at least one hash value to be checked for at least one data block in the vector database; determine the integrity of the vector database based on the at least one hash value to be checked and the corresponding stored hash value in the hash value index;

[0254] Determine a root hash value to be checked corresponding to the vector database; and determine the integrity of the vector database based on the root hash value to be checked and the stored root hash value in the hash value index.

[0255] In some embodiments, the at least one data block in the vector database includes at least one of a fourth data block, a fifth data block, and a sixth data block;

[0256] The integrity check module is further configured to perform at least one of the following:

[0257] In response to the to-be-checked hash value corresponding to the fourth data block being different from the stored hash value, determining that the fourth data block is illegal data;

[0258] In response to the presence of a duplicate hash value in the hash value index, determining the duplicate hash value and its corresponding data block as illegal data;

[0259] In response to the presence of duplicate hash values ​​in the hash value index, determining whether a data block corresponding to each duplicate hash value exists; determining duplicate hash values ​​for which no corresponding data block exists as illegal data;

[0260] In response to the identification information of the sixth data block appearing only in the storage hash value information corresponding to the fifth data block, it is determined that the sixth data block is illegally deleted; wherein the storage hash value of the fifth data block is determined based on the identification information of the sixth data block.

[0261] In some embodiments, the integrity check module is configured to perform an integrity check on the vector database periodically or in response to a data access operation on the vector database.

[0262] In some embodiments, the vector database further includes an exception response module; the exception response module is used to execute a corresponding response strategy in response to an abnormal event determined by the integrity check module.

[0263] Next, combine Fig. 9 , the interaction process between various functional modules in the vector database provided by this application is explained.

[0264] like Fig. 9 As shown, the vector database includes a content management module 910, an incremental hash module 920, an integrity check module 930 and an abnormal response module 940; wherein, the process of the vector database performing a data update operation includes the following steps S901 to S905:

[0265] Step S901, updating a vector database using a plurality of first data blocks;

[0266] Here, the content management module 910 receives an update operation of the vector database and performs a delete, replace, or insert operation on the target data item.

[0267] The content management module 910 divides the target data item into a plurality of first data blocks, and generates data block index information including identification information of the data block, identification information of the next data block and a timestamp for each first data block.

[0268] Step S902, sending data block update information;

[0269] Here, after determining the data block indexes corresponding to the plurality of first data blocks, the content management module 910 sends the corresponding data block indexes of the plurality of first data blocks to the incremental hash module 920 .

[0270] Step S903, determining update information for the hash value index based on the data block update information;

[0271] Here, the incremental hash module 920 calculates the hash values ​​corresponding to the plurality of first data blocks and a subsequent adjacent data block in parallel based on the data block indexes of the plurality of first data blocks; when the update operation corresponding to the first data block is a delete operation, the incremental hash module 920 can directly determine the hash value of the first data block from the hash value index. At the same time, the incremental hash module 920 recalculates the root hash value.

[0272] The incremental hash module 920 determines the calculated hash values ​​of the first data block and its adjacent data blocks, and the root hash value, as update information for the hash value index.

[0273] Step S904, sending update information for the hash value index;

[0274] Here, the incremental hash module 920 sends update information for the hash value index to the content management module 910 .

[0275] Step S905: update the hash value index.

[0276] Here, the module for implementing the hash value index update function in the content management module 910 updates the hash value index based on the update information for the hash value index, thereby completing the update process of the hash value index of the vector database.

[0277] In addition, the process of performing integrity check of the vector database includes the following steps S906 to S909:

[0278] Step S906, reading the hash value index;

[0279] Here, the vector database performs integrity check periodically or in response to a data read operation. When performing the integrity check, the integrity check module 930 of the vector database reads the hash value index from the content management module 910 and reads the hash key at the same time.

[0280] Step S907, verifying the integrity of the vector database based on the hash value index;

[0281] Here, the specific implementation of the integrity checking module 930 verifying the integrity of the vector database based on the hash value index can refer to the detailed description above.

[0282] Step S908, detecting an abnormal event and sending the abnormal event;

[0283] Here, after the integrity check module 930 determines that the vector database is subjected to a malicious attack according to the integrity check result, the malicious attack is sent to the abnormal response module 940 as an abnormal event.

[0284] Step S909, record the abnormal event and issue an alarm message.

[0285] Here, the abnormal response module records the event type, illegal data items, event information, etc. of the abnormal event, and issues corresponding alarm information.

[0286] In some embodiments, the functions or components included in the vector database provided by the present disclosure can be used to execute the methods described in the above method embodiments. Therefore, for technical details not disclosed in the vector database embodiments of the present application, please refer to the description of the method embodiments of the present application for understanding.

[0287] It should be noted that in the embodiment of the present application, if the above-mentioned data processing method is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the embodiment of the present application is essentially or the part that contributes to the relevant technology can be embodied in the form of a software product, which is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a U disk, a mobile hard disk, a read-only memory (ROM), a disk or an optical disk. In this way, the embodiment of the present application is not limited to any specific hardware, software or firmware, or any combination of hardware, software, and firmware.

[0288] An embodiment of the present application provides a computer device, including a memory and a processor, wherein the memory stores a computer program that can be run on the processor, and when the processor executes the program, some or all of the steps in the above method are implemented.

[0289] The embodiment of the present application provides a computer-readable storage medium on which a computer program is stored, and when the computer program is executed by a processor, some or all of the steps in the above method are implemented. The computer-readable storage medium can be transient or non-transient.

[0290] An embodiment of the present application provides a computer program, including a computer-readable code. When the computer-readable code is run in a computer device, a processor in the computer device executes some or all of the steps for implementing the above method.

[0291] The embodiment of the present application provides a computer program product, which includes a non-transitory computer-readable storage medium storing a computer program, and when the computer program is read and executed by a computer, some or all of the steps in the above method are implemented. The computer program product can be implemented specifically by hardware, software or a combination thereof. In some embodiments, the computer program product is specifically embodied as a computer storage medium, and in other embodiments, the computer program product is specifically embodied as a software product, such as a software development kit (SDK) and the like.

[0292] It should be noted here that the description of the various embodiments above tends to emphasize the differences between the various embodiments, and the same or similar aspects can be referenced to each other. The description of the above device, storage medium, computer program and computer program product embodiments is similar to the description of the above method embodiment, and has similar beneficial effects as the method embodiment. For technical details not disclosed in the embodiments of the device, storage medium, computer program and computer program product of this application, please refer to the description of the method embodiment of this application for understanding.

[0293] It should be understood that "one embodiment" or "an embodiment" mentioned throughout the specification means that specific features, structures or characteristics related to the embodiment are included in at least one embodiment of the present application. Therefore, "in one embodiment" or "in an embodiment" appearing throughout the specification does not necessarily refer to the same embodiment. In addition, these specific features, structures or characteristics can be combined in one or more embodiments in any suitable manner. It should be understood that in various embodiments of the present application, the size of the serial numbers of the above-mentioned steps / processes does not mean the order of execution, and the execution order of each step / process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application. The above-mentioned serial numbers of the embodiments of the present application are for description only and do not represent the advantages and disadvantages of the embodiments.

[0294] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element.

[0295] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as: multiple units or components can be combined, or can be integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the components shown or discussed can be through some interfaces, and the indirect coupling or communication connection of the devices or units can be electrical, mechanical or other forms.

[0296] The units described above as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units; they may be located in one place or distributed on multiple network units; some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0297] In addition, all functional units in the embodiments of the present application may be integrated into one processing unit, or each unit may be a separate unit, or two or more units may be integrated into one unit; the above-mentioned integrated units may be implemented in the form of hardware or in the form of hardware plus software functional units.

[0298] A person skilled in the art can understand that all or part of the steps of implementing the above method embodiment can be completed by hardware related to program instructions, and the aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it executes the steps of the above method embodiment; and the aforementioned storage medium includes: mobile storage devices, read-only memories (ROM), magnetic disks or optical disks, etc., various media that can store program codes.

[0299] Alternatively, if the above-mentioned integrated unit of the present application is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application can essentially or in other words, the part that contributes to the relevant technology can be embodied in the form of a software product, which is stored in a storage medium and includes a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as mobile storage devices, ROMs, magnetic disks, or optical disks.

[0300] The above is only an implementation method of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application.

Claims

1. A data processing method, comprising: receiving at least one first data block; The first data block is a data block determined based on an update operation performed on a target database; The target database represents a vector database including at least one data item; Determine in parallel at least one first hash value corresponding to the at least one first data block; Based on the at least one first hash value, update information of the hash value index of the target database is determined.

2. The method according to claim 1, further comprising: A segmentation process is performed on at least one target data item to determine the at least one first data block.

3. The method according to claim 1, wherein the step of determining in parallel at least one first hash value corresponding to the at least one first data block comprises at least one of the following: In a case where the update operation is a delete operation, determining a first hash value of the first data block from the hash value index; In the case where the update operation is a replace operation or an insert operation, the first hash value is determined based on identification information of the first data block and identification information of the second data block; the second data block represents a data block stored adjacent to the corresponding first data block in the target database.

4. The method according to any one of claims 1 to 3, wherein determining update information of a hash value index for the target database based on the at least one first hash value comprises: Based on the at least one update operation, determining at least one third hash value corresponding to at least one third data block; The third data block represents a data block stored adjacent to the corresponding first data block in the target database; The update information is determined based on the at least one first hash value and the at least one third hash value.

5. The method according to claim 4, wherein determining the update information based on the at least one first hash value and the at least one third hash value comprises: Determining a root hash value based on the at least one first hash value and the at least one third hash value; The update information is determined based on the at least one first hash value, the at least one third hash value, and the root hash value.

6. The method according to claim 1, wherein: The method further comprises at least one of the following: Determining the integrity of the target database based on identification information and / or stored hash values ​​of at least one data block in the hash value index; Determine in parallel at least one to-be-verified hash value of at least one data block in the target database; Determining the integrity of the target database based on the at least one hash value to be verified and a corresponding stored hash value in the hash value index; Determine the root hash value to be verified corresponding to the target database; The integrity of the target database is determined based on the root hash value to be verified and the stored root hash value in the hash value index.

7. The method of claim 6, wherein the at least one data block comprises at least one of a fourth data block, a fifth data block, and a sixth data block; The method further comprises at least one of the following: In response to the to-be-checked hash value corresponding to the fourth data block being different from the stored hash value, determining that the fourth data block is illegal data; In response to the presence of a duplicate hash value in the hash value index, determining that the duplicate hash value and its corresponding data block are illegal data; In response to the presence of duplicate hash values ​​in the hash value index, determining whether a data block corresponding to each of the duplicate hash values ​​exists; and determining duplicate hash values ​​for which no corresponding data block exists as illegal data; In response to the identification information of the sixth data block only appearing in the hash value index information corresponding to the fifth data block, it is determined that the sixth data block is illegally deleted; wherein, The storage hash value of the fifth data block is determined based on the identification information of the sixth data block.

8. The method according to any one of claims 1 to 7, wherein: The target database also includes a signature index; the signature index includes signature information corresponding to each data block in the target database; The method further comprises: Determine in parallel at least one signature information corresponding to the at least one first data block; the signature information is determined based on a first hash value corresponding to the first data block; Based on the at least one signature information, the signature index is updated.

9. The method according to claim 8, further comprising: Based on the stored hash value corresponding to at least one data block in the target database, at least one signature information to be verified corresponding to the at least one data block is calculated in parallel; Based on the at least one signature information to be verified and the corresponding stored signature information in the signature index, the integrity of the target database is determined.

10. A vector database, comprising a processor and a data interface; the processor is connected to the data interface; wherein, The processor receives at least one first data block through the data interface; determines at least one first hash value corresponding to the at least one first data block in parallel; and determines update information of the hash value index for the target database based on the at least one first hash value; The first data block is a data block determined based on performing an update operation on the vector database.

Images