Database processing systems, methods, apparatus, servers, and media

Abstract

The application relates to the technical field of databases, and particularly discloses a database processing system, method, device, server and medium, which are used for realizing data write services of a single-machine vector database server and reducing the deployment cost of the vector database. A business system is used for sending at least one data table update request to a message queue, so as to write an update message body of the data table update request into the message queue, the update message body comprising a data table name, a data table version and vector information of a first vector data table to be updated; a single-machine vector database server is used for listening to the updated message body in the message queue; when the updated message body is listened to, the data table name, the data table version and the vector information are read out from the updated message body; the first vector data table is queried from the single-machine vector database server according to the data table name and the data table version, and the first vector data table is updated according to the vector information.

Description

Technical Field

[0001] This application relates to the field of database technology, and in particular to a database processing system, method, apparatus, server, and medium. Background Technology

[0002] With the rapid development of large model technology, vector database products on the market are also developing rapidly. Current vector database products include distributed vector databases and standalone vector databases. The current types of distributed vector databases include Milvus, and the types of standalone vector databases include Faiss. For distributed vector databases, distributed deployment requires a cluster environment and multiple microservice components to realize data writing functions, which has high deployment and hardware costs. Standalone vector databases only require memory, but each service of a standalone vector database needs to deploy a Faiss service to realize high-concurrency data writing functions.

[0003] Therefore, it can be seen that in traditional solutions, if a project needs to deploy a vector database, the technology selection falls into the two extremes mentioned above. Implementing the write function for single-machine vector data will result in high deployment costs or make it impossible to implement the required functionality. Summary of the Invention

[0004] Therefore, it is necessary to provide a database processing system, method, device, server, and medium to address the aforementioned technical problems and resolve the technical issues that would lead to high deployment costs or make it impossible to implement write functionality for single-machine vector data.

[0005] To solve the above problems, the following technical solution is provided:

[0006] In a first aspect, a database processing system is provided, comprising a business system, a standalone vector database server, and a middleware service system, wherein the middleware service system includes a message queue, wherein:

[0007] The business system is configured to send at least one data table update request to the message queue, and write the update message body of the data table update request into the message queue. The update message body includes the data table name, data table version and vector information to be updated of the first vector data table to be updated.

[0008] The standalone vector database server is configured to listen for updated message bodies in the message queue after the database write service starts; when the updated message body is detected, the data table name, data table version, and vector information are read from the updated message body; the first vector data table is retrieved from the standalone vector database server according to the data table name and data table version, and the first vector data table is updated according to the vector information.

[0009] Furthermore, the middleware service system also includes an intermediate database, which is used to maintain the latest data table metadata information of the stand-alone vector database server in real time. The data table metadata information includes the data table name, data table version, and offline file storage address of the second vector data table to be read. The stand-alone vector database server is used for:

[0010] After the database read service starts, it periodically reads the metadata information of the intermediate database tables through a scheduled task. It then performs a full comparison of the read metadata information with its own available metadata information. If the compared table version information is inconsistent, it automatically loads the latest offline data table file from the offline file storage address and responds to the corresponding vector retrieval request from the business system through the offline data table file.

[0011] Further, the stand-alone vector database server updates the first vector data table according to the vector information, including:

[0012] Based on the data table name and data table version of the first vector data table, retrieve the corresponding third vector data table from the dynamic database;

[0013] The third vector data table is updated first according to the vector information;

[0014] The first vector data table is then updated using the updated third vector data table.

[0015] Furthermore, the single-machine vector database server is deployed with multiple database write services, each of which is used to respond to different database write service start requests.

[0016] Furthermore, the single-machine vector database server is deployed with multiple database read services, and each database write service is used to respond to different database read service start requests.

[0017] Secondly, a database processing method based on a database processing system is provided. The database processing system includes a business system, a single-machine vector database server, and a middleware service system. The middleware service system includes a message queue. The method includes:

[0018] The business system sends at least one data table update request to the message queue to write the update message body of the data table update request into the message queue. The update message body includes the data table name, data table version and vector information of the first vector data table to be updated.

[0019] After the database write service starts, the stand-alone vector database server listens for updated message bodies in the message queue. When the updated message body is detected, the server reads the data table name, data table version, and vector information from the updated message body. The server then queries the corresponding first vector data table from the stand-alone vector database server according to the data table name and data table version, and updates the first vector data table according to the vector information.

[0020] Furthermore, the middleware service system also includes an intermediate database, which maintains the data table metadata information of the standalone vector database server in real time. The data table metadata information includes the data table name, data table version, and offline file storage address of the second vector data table to be read. The method further includes:

[0021] After the database read service is started, the standalone vector database server periodically reads the metadata information of the intermediate database tables through a scheduled task. It then performs a full comparison of the read metadata information with its own available metadata information. If the compared table version information is inconsistent, it automatically loads the latest offline data table file from the offline file storage address and responds to the corresponding vector retrieval request from the business system through the offline data table file.

[0022] Thirdly, a database processing apparatus is provided for a standalone vector database server, the database processing apparatus comprising:

[0023] The listening module is used to listen for update message bodies in the message queue after the database write service starts. The update message body in the message queue includes the data table name, data table version, and vector information of the first vector data table to be updated.

[0024] The reading module is used to read the data table name, data table version, and vector information from the update message body when the update message body is detected.

[0025] The query module is used to query the corresponding first vector data table from the single-machine vector database server according to the data table name and data table version;

[0026] An update module is used to update the first vector data table according to the vector information.

[0027] Fourthly, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the computer program, performs the function of a stand-alone vector database server in a database processing system as described in any of the preceding claims.

[0028] Fifthly, a computer-readable storage medium is provided, the computer-readable storage medium storing a computer program, which, when executed by a processor, cooperates to implement the functions of any of the database processing systems described in the present invention.

[0029] As can be seen, the database processing system provided in this application adopts a single-machine vector database server, avoiding the direct use of distributed vector databases such as Milvus. This directly avoids the need for a cluster environment and the deployment of multiple microservice components, reducing deployment and hardware costs. In addition, the single-machine vector database server is extended by using a message queue to first write update messages and then updating the first vector data table of the single-machine vector database through a consumption subscription method. Since the message queue message can only be consumed once, it can respond to the high concurrency requirements of writing and updating multiple first vector data tables at the same time, realizing the distributed processing requirements. That is, for the vector database, it reduces deployment costs and achieves high concurrency when implementing the write function. Attached Figure Description

[0030] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the description of the embodiments of this application will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0031] Figure 1 This is a schematic diagram of a system framework for a database processing system according to one embodiment of this application;

[0032] Figure 2 This is a schematic diagram of another system framework of a database processing system according to one embodiment of this application;

[0033] Figure 3 This is a flowchart illustrating a database processing method according to an embodiment of this application;

[0034] Figure 4 This is a schematic diagram of a database processing device according to one embodiment of this application;

[0035] Figure 5 This is a schematic diagram of the structure of a stand-alone vector database server according to one embodiment of this application. Detailed Implementation

[0036] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0037] Understandably, deploying distributed vector databases like Milvus requires a cluster environment and multiple microservice components, resulting in high deployment and hardware costs. Deploying single-machine vector databases like Faiss, for example, while only requiring memory, necessitates deploying a Faiss service within each read or write service of the single-machine vector database; otherwise, distributed write functionality within the single-machine vector database cannot be achieved under multi-replica scaling. To address the issues of high deployment costs or the inability to fulfill requirements when deploying distributed vector databases in traditional solutions, this application provides a database processing system that optimizes the overall system architecture and read / write processes, thereby achieving a lightweight distributed deployment scheme for single-machine vector databases. This scheme not only has low deployment costs but also ensures the availability and data consistency of the vector database under certain concurrent traffic requirements, guaranteeing synchronization needs. The following detailed embodiments illustrate this approach.

[0038] First, there is the database write process of a single-machine vector database server.

[0039] Please refer to the following first. Figure 1 As described in one embodiment of this application, a database processing system is provided. The database processing system includes a business system, a single-machine vector database server, and a middleware service system. The middleware service system includes a message queue (MQ). Exemplarily, the message queue includes, but is not limited to, message queue forms such as ActiveMQ, RabbitMQ, ZeroMQ, Kafka, MetaMQ, or RocketMQ. Specific embodiments of this application are not limited to these.

[0040] The business system is configured to send at least one data table update request to the message queue, and to write the update message body of the data table update request into the message queue. The update message body includes the data table name, data table version and vector information of the first vector data table to be updated.

[0041] The standalone vector database server is configured to listen for updated message bodies in the message queue after the database write service starts; when the updated message body is detected, the data table name, data table version, and vector information are read from the updated message body; the corresponding first vector data table is queried from the standalone vector database server according to the data table name and data table version, and the first vector data table is updated according to the vector information.

[0042] In this context, "business system" refers to a system that requires vector retrieval and / or data table updates from a single-machine vector database server. For example, if the project is an internal enterprise resource management system, then the business system could be an internal enterprise resource query system, and so on, without specific limitations. In this embodiment, a message queue is deployed, and the publish-subscribe pattern of the message queue is used to optimize the write process of the single-machine vector database server.

[0043] From the perspective of the business system, when the business system has a data table update requirement, it sends a data table update request to the message queue. The update message body of the data table update request is then written into the message queue. In this way, the update message from the data table update request is placed into the message queue as the update message body. The update message body includes the data table name, data table version, and vector information of the first vector data table to be updated. In other words, every time there is an update to a vector data table, the corresponding update content is written into the message queue as the update message body. The message queue operates on a consumer-subscribe model; the update message bodies within it serve as consumer messages for the standalone vector database server, which in turn acts as a subscriber to the message bodies in the message queue.

[0044] For a standalone vector database server, a database write service is deployed. When the database write service starts, the server, as a subscriber to the message queue, continuously listens for updated message bodies in the message queue. When an updated message body is detected, the server reads the message from the updated message body, which includes the data table name, data table version, and vector information to be updated for the first vector data table. After reading the update message, the server queries the standalone vector database service to find the first vector data table according to the data table name and data table version, updates the first vector data table according to the vector information, and then sends a message queue response message to the message queue to notify the database write service that the update is complete.

[0045] The updated file can be saved to NAS storage, and a message queue response message can be sent to the message queue to notify the system that the write service update is complete.

[0046] As can be seen, this embodiment provides a database processing system. This embodiment uses a single-machine vector database server, avoiding the direct use of distributed vector databases such as Milvus, thereby directly avoiding the need for a cluster environment and the deployment of multiple microservice components, reducing deployment and hardware costs. In addition, this embodiment extends the single-machine vector database server by using a message queue to first write update messages, and then updating the first vector data table of the single-machine vector database through a consumption subscription method. Since the message queue message can only be consumed once, it can respond to high-concurrency write requirements of multiple vector data tables being written and updated simultaneously, realizing distributed processing requirements. That is, for the single-machine vector database server, when implementing the write function, the implementation of this application embodiment can reduce deployment costs and achieve high-concurrency write functions.

[0047] In one embodiment, the single-machine vector database server is deployed with multiple database write services, each of which is used to respond to different database write service start requests.

[0048] It should be understood that, based on the system architecture of this application embodiment, the single-machine vector database server can be deployed with multiple database write services. In other words, the database write service can be horizontally scaled. Since the message body of the message queue can only be consumed once, when there are multiple replicas of database write services listening in one or more business systems, there will be no problem of simultaneous writing when the above-mentioned multiple database write servers are started at the same time, and it can also respond to the high concurrency requirements of simultaneous updates of multiple data tables.

[0049] Secondly, it should be noted that the above process is a database write process based on the database processing system framework provided in the embodiments of this application. In this embodiment, the database read process is based on the database processing system framework.

[0050] Please continue reading. Figure 1 Furthermore, the middleware service system further includes an intermediate database. For example, this intermediate database includes, but is not limited to, in-memory databases such as Redis or Memcached. Specific embodiments of this application are not limited to this. The intermediate database maintains the data table metadata information of the standalone vector database server in real time. This data table metadata information characterizes data describing the data information in the standalone vector database server. The data table metadata information includes the latest available data table name of the second vector data table to be read by the standalone vector database server, the latest data table version, and the offline file storage address of the second vector data table. Wherein:

[0051] The standalone vector database server is used to: after the database read service starts, periodically read the data table metadata information of the intermediate database through a scheduled task, and perform a full comparison between the read data table metadata information and its own available data table metadata information. If the compared data table version information is inconsistent, the latest data table offline file is automatically loaded through the data table offline file storage address, and the corresponding vector retrieval request of the business system is responded to through the data table offline file.

[0052] In this embodiment, the middleware service system also includes an intermediate database to implement the read process of the stand-alone vector database server. Upon receiving a vector retrieval request from the business system and after the database read service is started, the intermediate database continuously refreshes its status data via a scheduled task to obtain the latest data table metadata information. For example, the stand-alone vector database server can implement the aforementioned scheduled task by creating a subprocess; the specific implementation is not limited. After the vector database server periodically reads the data table name, data table version, and offline file storage address of the second vector data table to be read, it performs a full comparison with its own available data table metadata information. If the compared data table version information is inconsistent, the latest offline data table file is automatically loaded into the memory of the stand-alone vector database server via the offline file storage address to complete the replacement of the second vector data table. Finally, the system responds to the corresponding vector retrieval request from the business system using the read offline data table file, thus providing the latest retrieval service after replacement, thereby completing the read process.

[0053] It should be noted that, as an example, when the intermediate database maintains the data table metadata information of the single-machine vector database server, the state of the intermediate database can be maintained in the following way: other business systems parse the previous data read and write process business, realize the business logic transformation, and realize the addition, deletion, modification and query of the second vector data table at the logical level through the transformation result, but still retain the physical file of the second vector data table, and do not modify the physical file of the second vector data table, so as to avoid changing the data of the second vector data table during the data reading process.

[0054] In one embodiment, the single-machine vector database server is deployed with multiple database read services, and each database write service is used to respond to different database read service start requests.

[0055] It should be understood that, based on the system architecture of this application embodiment, the single-machine vector database server can deploy multiple database read services. In other words, the database read service can also be horizontally scaled. It can be seen that after the database read service is horizontally scaled into multiple replicas to achieve distributed deployment, each database read service replica will periodically refresh the status of the intermediate database. Since this operation is read-only, it will not make any changes to the physical files of the second vector data table. Therefore, on the one hand, there will be no performance and data consistency issues when multiple replica database read services read at the same time. On the other hand, during the periodic task refresh process, it will not affect the response to vector retrieval requests in the main process. It can respond to high-concurrency retrieval processing and facilitate rapid expansion, reducing expansion costs. This is very beneficial for projects with continuously increasing query volume.

[0056] As can be seen from the above embodiments, this application provides a database processing system. This system achieves read / write service separation for a single-machine vector database by setting up a message queue and an intermediate database, which is beneficial for maintaining read / write service functions. During the database read service, the latest second vector data table is loaded into memory by comparison to respond to external vector data retrieval. The database write service is mainly responsible for updating the offline first vector data table file to respond to external update requests for vector data. Both database read and write services can horizontally scale service replicas according to the corresponding request volume. The status interaction of the database read and write services is maintained through the middleware message queue and the intermediate database, and the maintenance mechanism is implemented through scheduled tasks within the read and write services.

[0057] like Figure 2 As shown in the specific application scenario embodiment, the message queue can be a RabbitMQ queue, the vector database server can be implemented using Faiss, the intermediate database can be a Redis database, db_write represents the database write service, and db_read represents the database read service. In this case, each first vector data table of the vector database server can be designed as a Faiss instance, and each database read service will load all vector databases. It can be seen that this embodiment can be directly extended based on the Faiss vector database to obtain a lightweight distributed database design, which can realize high-concurrency vector data synchronous loading and distributed requests. It can directly achieve rapid expansion on projects that use the Faiss database, which is of great significance, especially for the maintenance and update of Faiss projects; it reduces the hardware cost and operation and maintenance cost of deploying other distributed databases; and it can meet the vector database requirements during the rapid business expansion period. Of course, other single-machine vector databases can be extended using the method of this application embodiment, which will not be specifically illustrated.

[0058] In one embodiment, two sets of vector data tables can be maintained in the database processing system, denoted as a first vector data table and a third vector data table, respectively. The stand-alone vector database server updates the first vector data table according to the vector information, including: querying the corresponding third vector data table from the dynamic database according to the data table name and data table version; updating the third vector data table according to the vector information; and then using the updated third vector data table to update the first vector data table.

[0059] Taking a single-machine vector database server as an example, it should be further understood that for the first vector data table, which has high complexity and a large amount of data, directly adjusting the table structure, creating or deleting table indexes may cause table locking. The duration of table locking may depend on the actual situation of the data table, which is quite troublesome. Therefore, during the data write service, another Faiss vector data table can be maintained. This Faiss vector data table can be updated, and then the Faiss vector data table can be used to update the other Faiss vector data table in the single-machine vector database server. This reduces the use of the first Faiss vector data during the update process, reduces the occurrence of table read errors during the write process, and improves the stability of the project application.

[0060] In one embodiment, as an optimization, an Embbedding model can also be deployed in the database read service of the vector database server. The Embbedding model is used to transform vector information, directly converting the vector information that needs to be updated synchronously into data information, thereby reducing the time spent on updating data transmission and improving data synchronization efficiency.

[0061] It should be noted that the above embodiments describe the database processing system provided in this application. Based on this database processing system, a database processing method and other related topics are also provided, which will be described below.

[0062] like Figure 3 As shown, in one embodiment, a database processing method based on a database processing system is provided. The database processing system includes a business system, a single-machine vector database server, and a middleware service system. The middleware service system includes a message queue. The method includes:

[0063] S10: The business system sends at least one data table update request to the message queue to write the update message body of the data table update request into the message queue. The update message body includes the data table name, data table version and vector information of the first vector data table to be updated.

[0064] S20: After the database write service starts, the stand-alone vector database server listens for updated message bodies in the message queue; when the updated message body is detected, the data table name, data table version, and vector information are read from the updated message body; the corresponding first vector data table is queried from the stand-alone vector database server according to the data table name and data table version, and the first vector data table is updated according to the vector information.

[0065] In this context, "business system" refers to a system that requires vector retrieval and / or data table updates from a single-machine vector database server. For example, if the project is an internal enterprise resource management system, then the business system could be an internal enterprise resource query system, and so on, without specific limitations. In this embodiment, a message queue is deployed, and the publish-subscribe pattern of the message queue is used to optimize the write process of the single-machine vector database server.

[0066] From the perspective of the business system, when the business system has a data table update requirement, it sends a data table update request to the message queue. The update message body of the data table update request is then written into the message queue. In this way, the update message from the data table update request is placed into the message queue as the update message body. The update message body includes the data table name, data table version, and vector information of the first vector data table to be updated. In other words, every time there is an update to a vector data table, the corresponding update content is written into the message queue as the update message body. The message queue operates on a consumer-subscribe model; the update message bodies within it serve as consumer messages for the standalone vector database server, which in turn acts as a subscriber to the message bodies in the message queue.

[0067] For a standalone vector database server, a database write service is deployed. When the database write service starts, the server, as a subscriber to the message queue, continuously listens for updated message bodies in the message queue. When an updated message body is detected, the server reads the message from the updated message body, which includes the data table name, data table version, and vector information to be updated for the first vector data table. After reading the update message, the server queries the standalone vector database service to find the first vector data table according to the data table name and data table version, updates the first vector data table according to the vector information, and then sends a message queue response message to the message queue to notify the database write service that the update is complete.

[0068] Furthermore, in one embodiment, the single-machine vector database server is deployed with multiple database write services, each of which is used to respond to different database write service start requests.

[0069] It should be understood that, based on the system architecture of this application embodiment, the single-machine vector database server can be deployed with multiple database write services. In other words, the database write service can be horizontally scaled. Since the message body of the message queue can only be consumed once, when there are multiple replicas of database write services listening in one or more business systems, there will be no problem of simultaneous writing when the above-mentioned multiple database write servers are started at the same time, and it can also respond to the high concurrency requirements of simultaneous updates of multiple data tables.

[0070] Furthermore, in one embodiment, the middleware service system further includes an intermediate database, which maintains in real time the data table metadata information of the standalone vector database server. The data table metadata information includes the data table name, data table version, and offline file storage address of the second vector data table to be read. The method further includes:

[0071] S30: After the database read service is started, the standalone vector database server periodically reads the data table metadata information of the intermediate database through a scheduled task, and performs a full comparison between the read data table metadata information and its own available data table metadata information. When the compared data table version information is inconsistent, the latest data table offline file is automatically loaded through the data table offline file storage address, and the corresponding vector retrieval request of the business system is responded to through the data table offline file.

[0072] For example, a standalone vector database server can implement the above-mentioned scheduled task by creating a subprocess, without specific limitations. After periodically reading the data table name, data table version, and offline file storage address of the second vector data table to be read, the vector database server performs a full comparison with its own available data table metadata information. If the compared data table version information is inconsistent, the latest offline data table file is automatically loaded into the memory of the standalone vector database server through the offline file storage address to complete the replacement of the second vector data table. Finally, the read offline data table file is used to respond to the corresponding vector retrieval request from the business system, that is, to provide the latest retrieval service after replacement, thus realizing the read process.

[0073] It should be noted that, as an example, when the intermediate database maintains the data table metadata information of the single-machine vector database server, the state of the intermediate database can be maintained in the following way: other business systems parse the previous data read and write process business, realize the business logic transformation, and realize the addition, deletion, modification and query of the second vector data table at the logical level through the transformation result, but still retain the physical file of the second vector data table, and do not modify the physical file of the second vector data table, so as to avoid changing the data of the second vector data table during the data reading process.

[0074] In one embodiment, two sets of vector data tables can be maintained in the database processing system, denoted as a first vector data table and a third vector data table, respectively. The stand-alone vector database server updates the first vector data table according to the vector information, including: querying the corresponding third vector data table from the dynamic database according to the data table name and data table version; updating the third vector data table according to the vector information; and then using the updated third vector data table to update the first vector data table.

[0075] Taking a single-machine vector database server as an example, it should be further understood that for the first vector data table, which has high complexity and a large amount of data, directly adjusting the table structure, creating or deleting table indexes may cause table locking. The duration of table locking may depend on the actual situation of the data table, which is quite troublesome. Therefore, during the data write service, another Faiss vector data table can be maintained. This Faiss vector data table can be updated, and then the Faiss vector data table can be used to update the other Faiss vector data table in the single-machine vector database server. This reduces the use of the first Faiss vector data during the update process, reduces the occurrence of table read errors during the write process, and improves the stability of the project application.

[0076] In one embodiment, as an optimization, an Embbedding model can also be deployed in the database read service of the vector database server. The Embbedding model is used to transform vector information, directly converting the vector information that needs to be updated synchronously into data information, thereby reducing the time spent on updating data transmission and improving data synchronization efficiency.

[0077] As can be seen, the database processing method provided in this application uses a single-machine vector database server, avoiding the direct use of distributed vector databases such as Milvus. This directly avoids the need for a cluster environment and the deployment of multiple microservice components, reducing deployment and hardware costs. In addition, the single-machine vector database server is extended by using a message queue to first write update messages and then updating the first vector data table of the single-machine vector database through a consumption subscription method. Since the message queue message can only be consumed once, it can respond to the high concurrency requirements of writing and updating multiple first vector data tables at the same time, realizing the distributed processing requirements. That is, for the vector database, the deployment cost is reduced and the high concurrency function is achieved when implementing the write function.

[0078] For further explanation of this method, please refer to the description of the aforementioned system embodiments. The technical effects of the corresponding improvement schemes can also be found in the description of the aforementioned method embodiments. To avoid redundant descriptions, they will not be repeated here.

[0079] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0080] In one embodiment, a database processing apparatus is provided, comprising a listening module 101, a reading module 102, a query module 103, and an update module 104, such as... Figure 4 As shown, the named entity processing apparatus includes:

[0081] Listening module 101 is used to listen for update message bodies in the message queue after the database write service starts. The update message body in the message queue includes the data table name, data table version, and vector information of the first vector data table to be updated.

[0082] The reading module 102 is used to read the data table name, data table version and vector information from the update message body when the update message body is detected.

[0083] The query module 103 is used to query the corresponding first vector data table from the single-machine vector database server according to the data table name and data table version;

[0084] The update module 104 is used to update the first vector data table according to the vector information.

[0085] As can be seen, the database processing device provided in this application embodiment uses a single-machine vector database server, avoiding the direct use of distributed vector databases such as Milvus, thereby directly avoiding the need for a cluster environment and the deployment of multiple microservice components, reducing deployment and hardware costs. In addition, it extends the single-machine vector database server by using a message queue to first write update messages, and then updating the first vector data table of the single-machine vector database through a consumption subscription method. Since the message queue message can only be consumed once, it can respond to the high concurrency requirements of writing and updating multiple first vector data tables at the same time, realizing the distributed processing requirements. That is, for the vector database, it reduces the deployment cost and realizes the high concurrency function when implementing the write function.

[0086] For details regarding the specific implementation of each module or step in the database processing device, please refer to the descriptions in the aforementioned system embodiments. For the technical effects of the corresponding improvement schemes, please refer to the descriptions in the aforementioned method embodiments. To avoid redundant descriptions, these descriptions will not be repeated here.

[0087] In one embodiment, a server is provided to implement the functions of the stand-alone vector database server of this application, and its internal structure diagram can be as follows. Figure 4 As shown, the server includes a processor and memory connected via a system bus. The processor provides computing and control capabilities. The server also includes a computer-readable storage medium and memory. When executed by the processor, the computer-readable instructions implement the functions or steps of a stand-alone vector database server in a database processing system. The computer-readable storage medium provided in this embodiment includes both non-volatile and volatile readable storage media.

[0088] In one embodiment, a server is provided, including a memory, a processor, and computer-readable instructions stored in the memory and executable on the processor. When the processor executes the computer-readable instructions, it implements the functions or steps of a stand-alone vector database server in the aforementioned database processing system.

[0089] In one embodiment, a computer-readable storage medium is provided, the computer-readable storage medium storing a computer program, which, when executed by a processor, cooperates in implementing the functions of the aforementioned database processing system.

[0090] In one embodiment, a computer-readable storage medium is provided, the computer-readable storage medium storing a computer program that, when executed by a processor, performs the following steps:

[0091] A computer program product comprising a computer program that, when executed by a processor, performs the functions of the aforementioned database processing system.

[0092] It should be noted that for more details and technical effects regarding the implementation of this computer device or computer-readable storage medium, please refer to the description in the foregoing embodiments, which will not be repeated here.

[0093] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by instructing related hardware with computer-readable instructions. These computer-readable instructions can be stored in a non-volatile readable storage medium or a volatile readable storage medium. When executed, these computer-readable instructions can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0094] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is used as an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0095] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. A database processing system, characterized in that, The database processing system includes a business system, a single-machine vector database server, and a middleware service system. The middleware service system includes a message queue, wherein: The business system is configured to send at least one data table update request to the message queue, and write the update message body of the data table update request into the message queue. The update message body includes the data table name, data table version and vector information to be updated of the first vector data table to be updated. The standalone vector database server is used to listen for updated message bodies in the message queue after the database write service starts; when the updated message body is detected, the data table name, data table version and vector information are read from the updated message body; the first vector data table is queried from the standalone vector database server according to the data table name and data table version, and the first vector data table is updated according to the vector information. The middleware service system also includes an intermediate database, which is used to maintain the latest data table metadata information of the stand-alone vector database server in real time. The data table metadata information includes the data table name, data table version, and offline file storage address of the second vector data table to be read. The stand-alone vector database server is used for: After the database read service starts, it periodically reads the data table metadata information of the intermediate database through a scheduled task. It then performs a full comparison of the read data table metadata information with its own available data table metadata information. If the compared data table version information is inconsistent, it automatically loads the latest data table offline file through the data table offline file storage address and responds to the corresponding vector retrieval request of the business system through the data table offline file. The standalone vector database server updates the first vector data table according to the vector information, including: Based on the data table name and data table version of the first vector data table, retrieve the corresponding third vector data table from the dynamic database; The third vector data table is updated first according to the vector information; The first vector data table is then updated using the updated third vector data table.

2. The database processing system as described in claim 1, characterized in that, The single-machine vector database server is deployed with multiple database write services, each of which is used to respond to different database write service start requests.

3. The database processing system as described in claim 1, characterized in that, The single-machine vector database server is deployed with multiple database read services, and each database write service is used to respond to different database read service start requests.

4. A database processing method based on a database processing system, characterized in that, The database processing system includes a business system, a single-machine vector database server, and a middleware service system. The middleware service system includes a message queue. The method includes: The business system sends at least one data table update request to the message queue to write the update message body of the data table update request into the message queue. The update message body includes the data table name, data table version and vector information of the first vector data table to be updated. After the database write service starts, the stand-alone vector database server listens for updated message bodies in the message queue. When the updated message body is detected, the data table name, data table version, and vector information are read from the updated message body. The corresponding first vector data table is retrieved from the stand-alone vector database server according to the data table name and data table version, and the first vector data table is updated according to the vector information. The middleware service system further includes an intermediate database, which maintains real-time metadata information of the data tables on the standalone vector database server. This metadata information includes the name of the second vector data table to be read, the table version, and the offline file storage address of the data table. The method further includes: After the database read service is started, the standalone vector database server periodically reads the data table metadata information of the intermediate database through a scheduled task. It then performs a full comparison of the read data table metadata information with its own available data table metadata information. If the compared data table version information is inconsistent, it automatically loads the latest data table offline file through the data table offline file storage address and responds to the corresponding vector retrieval request of the business system through the data table offline file. Updating the first vector data table according to the vector information includes: Based on the data table name and data table version of the first vector data table, retrieve the corresponding third vector data table from the dynamic database; The third vector data table is updated first according to the vector information; The first vector data table is then updated using the updated third vector data table.

5. A database processing device for a stand-alone vector database server, characterized in that, The database processing device includes: The listening module is used to listen for update message bodies in the message queue after the database write service starts. The update message body in the message queue includes the data table name, data table version, and vector information of the first vector data table to be updated. The reading module is used to read the data table name, data table version, and vector information from the update message body when the update message body is detected. The query module is used to query the corresponding first vector data table from the single-machine vector database server according to the data table name and data table version; The update module is used to update the first vector data table according to the vector information; The database processing device is also used for: After the database read service starts, it periodically reads the metadata information of the intermediate database tables through a scheduled task. It then performs a full comparison of the read metadata information with its own available metadata information. If the compared table version information is inconsistent, it automatically loads the latest offline file of the table through the offline file storage address and responds to the corresponding vector retrieval request from the business system through the offline file of the table. Updating the first vector data table according to the vector information includes: Based on the data table name and data table version of the first vector data table, retrieve the corresponding third vector data table from the dynamic database; The third vector data table is updated first according to the vector information; The first vector data table is then updated using the updated third vector data table.

6. A server comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the function of a stand-alone vector database server in the database processing system as described in any one of claims 1-3.

7. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it cooperates to implement the functions of the database processing system as described in any one of claims 1-3.

Images