Knowledge base data processing method and device, electronic equipment and storage medium

Abstract

The application discloses a knowledge base data processing method and device, electronic equipment and a storage medium, and relates to the field of financial technology. The method comprises the following steps: in response to a data access request of a user, determining a target knowledge base ID to be accessed; according to the target knowledge base ID, querying a target index associated with the target knowledge base from a metadata table; wherein the target index is uniquely determined from candidate indexes of a vector database cluster when the target knowledge base is created; using the target knowledge base ID as a routing value, determining a target shard corresponding to the target knowledge base uniquely from a plurality of shards of the target index; and sending the data access request to the target shard, so that the target shard performs request processing.

Description

Technical Field

[0001] This application relates to the field of computer technology, and more particularly to the field of financial technology, specifically to a knowledge base data processing method, apparatus, electronic device, and storage medium. Background Technology

[0002] With the rapid development of artificial intelligence and big data technologies, vector databases, with their efficient vector storage and retrieval capabilities, have become a core infrastructure supporting semantic search, intelligent recommendation, and other applications in multi-knowledge base scenarios. As independent data units, knowledge bases face increasing demands for vectorized storage and retrieval, especially in multi-tenant or cross-project scenarios, where a unified vector database cluster is urgently needed to provide high-performance retrieval services for massive knowledge bases.

[0003] However, existing technical solutions face multiple challenges in managing large-scale knowledge bases and are unable to meet the needs of rapid business expansion. Summary of the Invention

[0004] This application provides a knowledge base data processing method, apparatus, electronic device, and storage medium to improve the resource utilization and data processing efficiency of a vector database cluster.

[0005] In a first aspect, embodiments of this application also provide a knowledge base data processing method, including: In response to a user's data access request, determine the ID of the target knowledge base to be accessed; Based on the target knowledge base ID, query the target index associated with the target knowledge base from the metadata table; wherein, the target index is uniquely determined in advance from the candidate indexes of the vector database cluster when the target knowledge base is created; Using the target knowledge base ID as the routing value, a unique target fragment corresponding to the target knowledge base is determined from multiple fragments of the target index; The data access request is sent to the target shard, causing the target shard to perform request processing.

[0006] Secondly, embodiments of this application also provide a knowledge base data processing apparatus, including: The Target ID module is used to determine the ID of the target knowledge base to be accessed in response to a user's data access request. The target index module is used to query the target index associated with the target knowledge base from the metadata table based on the target knowledge base ID; wherein, the target index is uniquely determined in advance from the candidate indexes of the vector database cluster when the target knowledge base is created; The target sharding module is used to determine the target shard uniquely corresponding to the target knowledge base from multiple shards of the target index, using the target knowledge base ID as the routing value. The request sending module is used to send the data access request to the target shard, so that the target shard performs request processing.

[0007] Thirdly, embodiments of this application also provide an electronic device, which includes: One or more processors; Storage device for storing one or more programs; When one or more programs are executed by one or more processors, the one or more processors implement any of the knowledge base data processing methods provided in the embodiments of this application.

[0008] Fourthly, embodiments of this application also provide a computer-readable storage medium storing a computer program, which, when executed by a processor, implements any of the knowledge base data processing methods provided in embodiments of this application.

[0009] Fifthly, embodiments of this application also provide a computer program product, which includes a computer program that, when executed by a processor, implements any of the knowledge base data processing methods provided in embodiments of this application.

[0010] The technical solution of this application integrates multiple knowledge base data into a few large indexes of a vector database cluster through an index reuse mechanism. This enables efficient management of massive knowledge bases with limited cluster resources, significantly improving resource utilization and data processing efficiency. During the data access phase, data is precisely routed to the uniquely corresponding target shard based on the target knowledge base ID, ensuring centralized data storage and rapid location. Attached Figure Description

[0011] Figure 1 This is a path diagram of a knowledge base data processing method provided according to Embodiment 1 of this application; Figure 2 This is a path diagram of another knowledge base data processing method provided according to Embodiment 2 of this application; Figure 3 This is a schematic diagram of the structure of a knowledge base data processing device according to Embodiment 3 of this application; Figure 4 This is a schematic diagram of the structure of an electronic device that implements the knowledge base data processing method of the embodiments of this application. Detailed Implementation

[0012] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present application.

[0013] It should be noted that the terms "first" and "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0014] Example 1 Figure 1 This is a path diagram of a knowledge base data processing method according to Embodiment 1 of this application. This embodiment is applicable to situations where multiple knowledge bases are processed in a vector database cluster environment. It can be executed by a knowledge base data processing device, which can be implemented in hardware and / or software and can be configured in an electronic device. Figure 1 As shown, the method includes: S101. In response to the user's data access request, determine the ID of the target knowledge base to be accessed; S102. Based on the target knowledge base ID, query the target index associated with the target knowledge base from the metadata table; wherein, the target index is uniquely determined in advance from the candidate indexes of the vector database cluster when the target knowledge base is created; S103. Using the target knowledge base ID as the routing value, determine the target fragment that uniquely corresponds to the target knowledge base from multiple fragments of the target index; S104. Send the data access request to the target shard, so that the target shard performs request processing.

[0015] A vector database cluster consists of multiple nodes used to store and retrieve high-dimensional vector data, such as text and image embedding vectors. M large indexes (M being a natural number, e.g., 3) are pre-created within the vector database cluster, and each index is configured with multiple shards (e.g., 10). By dividing a single index into multiple shards and distributing them concurrently across different nodes, the system's concurrent processing capabilities can be improved.

[0016] The metadata table is used to persist the mapping relationship between the knowledge base and the index, and can be a relational database or other metadata storage system. The metadata table also records the real-time status information of both the index and the knowledge base. During the creation of the target knowledge base, a unique candidate index can be determined from M candidate indexes, ensuring that all data of the target knowledge base is stored in the same candidate index, and a single candidate index can hold data from multiple knowledge bases, thus realizing a data processing mechanism for index reuse.

[0017] While the "single knowledge base - single index" model in related technologies can achieve basic retrieval functions, it has significant drawbacks: Firstly, the number of indexes is limited by physical resources, and the expansion of the knowledge base easily reaches the cluster limit, preventing the system from scaling horizontally. Secondly, the problem of uneven resource allocation is prominent, with a large number of small knowledge bases monopolizing index resources, resulting in idle and wasted computing resources such as memory and CPU, and low overall cluster utilization. Furthermore, the operation and maintenance of thousands of indexes (such as monitoring, backup, and upgrades) is costly and complex, further exacerbating system stability risks. The index reuse technology proposed in this invention integrates data from multiple knowledge bases into a few pre-built large indexes, supporting a number of knowledge bases far exceeding the number of indexes within limited cluster resources, thus improving the utilization rate of vector database cluster resources and data processing efficiency.

[0018] During the data access phase, the client can include the target knowledge base ID when submitting a data access request. In response to the data access request, the target knowledge base ID is extracted from the request. Based on the target knowledge base ID, the target index associated with the target knowledge base is queried from the metadata table. Using the target knowledge base ID as the routing value, a unique target shard corresponding to the target knowledge base is determined from multiple shards of the target index. This means that data for the target knowledge base ID is stored on the same shard of the target index, thereby improving data processing efficiency.

[0019] A data access request is sent to the target shard, causing the target shard to process the request. If the data access request is a data write request, the vector data to be written, carried in the data access request, is written to the target shard. If the data access request is a data query request, the data access request also carries the vector to be queried; filtering conditions for the query statement can be constructed based on the target knowledge base ID, so that the target shard only retrieves data from the target knowledge base, avoiding cross-knowledge base interference and thus improving query efficiency.

[0020] In one optional implementation, the step of using the target knowledge base ID as the routing value to determine the target fragment uniquely corresponding to the target knowledge base from multiple fragments of the target index includes: processing the target knowledge base ID using a preset hash algorithm, and determining the target fragment from multiple fragments of the target index based on the processing result.

[0021] For example, a preset hash algorithm is used to calculate the hash value of the target knowledge base ID. Based on this hash value and the identification information of multiple shards in the target index, the target knowledge base ID is hashed onto a uniquely corresponding target shard. Through the hash routing mechanism, requests from different knowledge bases can be evenly distributed across multiple shards of the target index, achieving load balancing among shards and improving the overall stability of the vector database.

[0022] The technical solution in this embodiment integrates multiple knowledge base data into a few large indexes of a vector database cluster through an index reuse mechanism. This enables efficient management of massive knowledge bases with limited cluster resources, significantly improving resource utilization and data processing efficiency. During the data access phase, data is precisely routed to the uniquely corresponding target shard based on the target knowledge base ID, ensuring centralized data storage and rapid location.

[0023] In one optional implementation, determining the target knowledge base ID to be accessed includes: determining whether the data access request includes the target knowledge base ID to be accessed; if not, determining the knowledge base that the user has access to based on the access control list and the user identifier as the target knowledge base, and obtaining the target knowledge base ID.

[0024] In response to a data access request, the system determines whether the request includes the target knowledge base ID. If it does, the ID is extracted directly from the request. If not, the user identifier is extracted, and the system identifies the knowledge base the user has access to based on the access control list (ACL), designating it as the target knowledge base and obtaining its corresponding ID. Centralized management of user and knowledge base access permissions through ACLs flexibly adapts to the permission isolation requirements in multi-tenant scenarios and simplifies the permission verification process.

[0025] In addition, if the data access request already includes the target knowledge base ID to be accessed, the user's access rights to the target knowledge base can be further verified based on the user identifier according to the access control list; if the user does not have access rights, the data access request is rejected, thereby improving data security.

[0026] Example 2 Figure 2This is a path diagram of another knowledge base data processing method provided in Embodiment 2 of this application. The technical solution of this embodiment is further refined based on the above technical solution. See also Figure 2 The knowledge base data processing method shown includes: S201. In response to the user's request to create a new knowledge base, query the index real-time status table and initially screen out candidate indexes. S202. Determine the health of each candidate index, and select the first index for the uniqueness of the new knowledge base from the candidate indexes based on the health. S203. Persist the mapping relationship between the new knowledge base ID and the first index to the metadata table; S204. In response to the user's data access request, determine the ID of the target knowledge base to be accessed; S205. Based on the target knowledge base ID, query the target index associated with the target knowledge base from the metadata table; wherein, the target index is uniquely determined in advance from the candidate indexes of the vector database cluster when the target knowledge base is created; S206. Using the target knowledge base ID as the routing value, determine the target fragment that uniquely corresponds to the target knowledge base from multiple fragments of the target index; S207. Send the data access request to the target shard, so that the target shard performs request processing.

[0027] Steps S201-S203 are used to assign corresponding indexes to the newly created knowledge base; steps S204-S207 are used to execute data access requests initiated by users for the knowledge base. In actual execution, the execution order between the two sets of steps is not fixed, but is flexibly adjusted according to the actual needs of the scenario.

[0028] The real-time index status table records the real-time status of each candidate index, such as the number of documents or the frequency of queries. During the new knowledge base creation phase, users can initiate a new knowledge base creation request, extract the new knowledge ID from the request, query the real-time index status table, and perform initial screening of the indexes in the vector database cluster to obtain candidate indexes, for example, indexes that meet the load requirements are selected as candidate indexes.

[0029] Health is used to evaluate the overall performance of candidate indexes, encompassing key parameters such as storage capacity and query performance. The health of candidate indexes is calculated by dynamically collecting data such as the remaining storage space of shards in each candidate index, the query frequency of the associated knowledge base, and the data volume. The candidate index with the highest health is then selected as the primary index. The mapping between the new knowledge base ID and the primary index is persisted to the metadata table. Quantifying index health enables dynamic resource scheduling, accurately matching knowledge base needs, avoiding unreasonable resource allocation, and improving the stability of the vector database cluster.

[0030] In one optional implementation, determining the health of each candidate index includes: determining the remaining space based on the current number of documents, the number of shards, and a preset upper limit for shard capacity of the candidate index; determining the remaining query frequency based on the current query frequency of the candidate index and the estimated query frequency of the new knowledge base; determining the data distribution uniformity based on the amount of data in each knowledge base of the candidate index; and fusing the remaining space, the remaining query frequency, and the data distribution uniformity to obtain the health score.

[0031] The remaining space is used to quantify the current capacity of a candidate index to hold new data. For each candidate index, we can obtain the number of storage units into which it is divided (the number of shards), the maximum number of documents that each shard can store (the shard capacity limit), and the total number of documents already stored in the candidate index (the current document count). Multiplying the number of shards by the shard capacity limit and then subtracting the current document count yields the remaining space for that candidate index. The larger the remaining space value, the stronger the storage capacity of the candidate index.

[0032] The remaining query frequency characterizes the query load capacity of a candidate index. For each candidate index, we can obtain the maximum query load it can handle (total query frequency limit), the number of query requests it has already processed per unit time (e.g., per second) (current query frequency), and the estimated query frequency, which is determined based on historical data or business predictions. The remaining query frequency is obtained by subtracting the current query frequency and the estimated query frequency from the total query frequency limit. The higher the remaining query frequency, the stronger the query processing capacity of the candidate index.

[0033] Data distribution uniformity measures the balance of data volume across knowledge bases in a candidate index. For each candidate index, the number of documents (i.e., data volume) associated with each knowledge base is obtained, and the standard deviation of each data volume is calculated to obtain the data distribution uniformity. A smaller standard deviation indicates a more uniform data distribution, which helps avoid performance bottlenecks caused by excessive data in certain shards. Preset first, second, and third weighting coefficients are used to weight and fuse remaining space, remaining query frequency, and data distribution uniformity to obtain the health of the candidate index. The values ​​of each weighting coefficient can be dynamically adjusted according to the business priority of the new knowledge base. For example, for storage-intensive businesses, the first weighting coefficient can be appropriately increased; for query-intensive businesses, the value of the second weighting coefficient can be appropriately adjusted.

[0034] The technical solution of this embodiment calculates the health status by dynamically collecting data such as remaining storage space, query frequency, and data volume when creating a new knowledge base. It then accurately selects the first index with the best health status from the candidate indexes and persists the mapping relationship to the metadata table, thereby realizing dynamic resource scheduling and further improving the stability of the vector database cluster.

[0035] In one optional implementation, the method further includes: real-time monitoring of the current document count and current query frequency of each candidate index; if any candidate index is determined to meet the data migration conditions based on the current document count and the current query frequency, then querying multiple knowledge bases associated with the candidate index from the metadata table, and selecting the knowledge base to be migrated from the multiple knowledge bases; determining the health of each candidate index, and selecting a second index from the candidate indexes for the uniqueness of the knowledge base to be migrated based on the health; using the knowledge base ID to be migrated as the routing value, determining the shard uniquely corresponding to the database to be migrated from the multiple shards included in the second index, and performing data migration; and persisting the mapping relationship between the knowledge base ID to be migrated and the second index to the metadata table.

[0036] Real-time monitoring of the current document count and query frequency of each candidate index; if the current document count of any candidate index reaches or exceeds the preset storage limit threshold, or if the current query frequency is too high, causing excessive system load, then the candidate index meets the data migration conditions. Query the multiple knowledge bases associated with the candidate index from the metadata table; considering factors such as the data volume, business importance, and access frequency of the knowledge bases, select the knowledge base to be migrated from multiple knowledge bases.

[0037] The health of each candidate index is calculated based on factors such as the remaining storage space of the shards in each candidate index, the query frequency of the associated knowledge base, and the data volume. The candidate index with the highest health is then selected as the second index. Using the knowledge base ID to be migrated as the routing value, the unique shard corresponding to the knowledge base to be migrated is determined from multiple shards of the second index, and the knowledge base to be migrated is migrated to that shard. The mapping relationship between the knowledge base ID to be migrated and the second index is persisted to the metadata table for subsequent data access. Through real-time monitoring and intelligent decision-making, efficient data migration and reasonable resource allocation are achieved, effectively improving the system's performance and stability.

[0038] Example 3 Figure 3 This is a schematic diagram of a knowledge base data processing device according to Embodiment 3 of this application. This embodiment is applicable to situations where multiple knowledge bases are processed in a vector database cluster environment. The knowledge base data processing device can be implemented in hardware and / or software, and can be configured in an electronic device. (Reference) Figure 3 The specific structure of the knowledge base data processing device 300 is as follows: The target ID module 310 is used to determine the target knowledge base ID to be accessed in response to a user's data access request. The target index module 320 is used to query the target index associated with the target knowledge base from the metadata table according to the target knowledge base ID; wherein, the target index is uniquely determined in advance from the candidate indexes of the vector database cluster when the target knowledge base is created; The target sharding module 330 is used to determine the target shard uniquely corresponding to the target knowledge base from multiple shards of the target index using the target knowledge base ID as the routing value; The request sending module 340 is used to send the data access request to the target shard, so that the target shard performs request processing.

[0039] In one optional implementation, the target fragmentation module 330 is specifically used for: The target knowledge base ID is processed using a preset hash algorithm, and the target fragment is determined from multiple fragments of the target index based on the processing result.

[0040] In one alternative implementation, the target ID module 310 includes: The ID extraction unit is used to determine whether the data access request includes the target knowledge base ID to be accessed. The target ID unit is used, if not included, to determine, based on the access control list and the user identifier, the knowledge base to which the user has access rights, as the target knowledge base, and to obtain the target knowledge base ID.

[0041] In one optional embodiment, the device 300 further includes a knowledge base creation module, the knowledge base creation module comprising: The candidate index unit is used to respond to a user's request to create a new knowledge base by querying the real-time status table of the index and initially screening out candidate indexes. The health unit is used to determine the health of each candidate index; The first indexing unit is used to select a first index from the candidate indexes for the uniqueness of the new knowledge base based on the health status. The relationship persistence unit is used to persist the mapping relationship between the new knowledge base ID and the first index to the metadata table.

[0042] In one optional implementation, the health unit is specifically used for: The remaining space is determined based on the current number of documents, the number of shards, and the preset maximum shard capacity of the candidate index; The remaining query frequency is determined based on the current query frequency of the candidate index and the estimated query frequency of the new knowledge base; The uniformity of data distribution is determined based on the amount of data in each knowledge base of the candidate indexes; The remaining space, the remaining query frequency, and the data distribution uniformity are combined to obtain the health score.

[0043] In one optional embodiment, the device 300 further includes a knowledge base migration module, which is specifically used for: Real-time monitoring of the current document count and current query frequency of each candidate index; If any candidate index is determined to meet the data migration conditions based on the current number of documents and the current query frequency, then the multiple knowledge bases associated with the candidate index are queried from the metadata table, and the knowledge base to be migrated is selected from the multiple knowledge bases. Determine the health of each candidate index, and select a second index from the candidate indexes based on the health of the candidate indexes to ensure the uniqueness of the knowledge base to be migrated; Using the knowledge base ID to be migrated as the routing value, the unique shard corresponding to the database to be migrated is determined from the multiple shards included in the second index, and data migration is performed; The mapping relationship between the knowledge base ID to be migrated and the second index is persisted to the metadata table.

[0044] The knowledge base data processing apparatus provided in this application embodiment can execute the knowledge base data processing method provided in any embodiment of this application, and has the corresponding functional modules and beneficial effects for executing the knowledge base data processing method.

[0045] According to embodiments of the present invention, the present invention also provides an electronic device, a readable storage medium, and a computer program product.

[0046] Example 4 Figure 4 This is a schematic diagram of the structure of an electronic device 410 implementing the knowledge base data processing method of the embodiments of this application. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices (such as helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present application described and / or claimed herein.

[0047] like Figure 4 As shown, the electronic device 410 includes at least one processor 411 and a memory, such as a read-only memory (ROM) 412 or a random access memory (RAM) 413, communicatively connected to the at least one processor 411. The memory stores computer programs executable by the at least one processor. The processor 411 can perform various appropriate actions and processes based on the computer program stored in the ROM 412 or loaded from storage unit 418 into the RAM 413. The RAM 413 may also store various programs and data required for the operation of the electronic device 410. The processor 411, ROM 412, and RAM 413 are interconnected via a bus 414. An input / output (I / O) interface 415 is also connected to the bus 414.

[0048] Multiple components in electronic device 410 are connected to I / O interface 415, including: input unit 416, such as keyboard, mouse, etc.; output unit 417, such as various types of displays, speakers, etc.; storage unit 418, such as disk, optical disk, etc.; and communication unit 419, such as network card, modem, wireless transceiver, etc. Communication unit 419 allows electronic device 410 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0049] Processor 411 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of processor 411 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. Processor 411 performs the various methods and processes described above, such as knowledge base data processing methods.

[0050] In some embodiments, the knowledge base data processing method may be implemented as a computer program tangibly contained in a computer-readable storage medium, such as storage unit 418. In some embodiments, part or all of the computer program may be loaded and / or installed on electronic device 410 via ROM 412 and / or communication unit 419. When the computer program is loaded into RAM 413 and executed by processor 411, one or more steps of the knowledge base data processing method described above may be performed. Alternatively, in other embodiments, processor 411 may be configured as the knowledge base data processing method by any other suitable means (e.g., by means of firmware).

[0051] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0052] Computer programs used to implement the methods of this application may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable device, such that when executed by the processor, the computer programs cause the functions / operations specified in the path diagrams and / or block diagrams to be implemented. The computer programs may be executed entirely on a machine, partially on a machine, or as a standalone software package, partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0053] In the context of this application, a computer-readable storage medium can be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, apparatus, or device. A computer-readable storage medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium can be a machine-readable signal medium. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0054] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0055] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or middleware components (e.g., application servers), or frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), blockchain networks, and the Internet.

[0056] A computing system can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a hosting product within the cloud computing service system to address the shortcomings of traditional physical hosts and VPS services, such as high management difficulty and weak business scalability.

[0057] It should be understood that the various paths shown above can be used to reorder, add, or delete steps. For example, the steps described in this application can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this application can be achieved, and this is not limited herein.

[0058] The specific embodiments described above do not constitute a limitation on the scope of protection of this application. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. A knowledge base data processing method, characterized in that, include: In response to a user's data access request, determine the ID of the target knowledge base to be accessed; Based on the target knowledge base ID, query the target index associated with the target knowledge base from the metadata table; wherein, the target index is uniquely determined in advance from the candidate indexes of the vector database cluster when the target knowledge base is created; Using the target knowledge base ID as the routing value, the target fragment uniquely corresponding to the target knowledge base is determined from multiple fragments of the target index; The data access request is sent to the target shard, causing the target shard to perform request processing.

2. The method according to claim 1, characterized in that, The step of using the target knowledge base ID as the routing value to determine the unique target fragment corresponding to the target knowledge base from multiple fragments of the target index includes: The target knowledge base ID is processed using a preset hash algorithm, and the target fragment is determined from multiple fragments of the target index based on the processing result.

3. The method according to claim 1, characterized in that, The determination of the target knowledge base ID to be accessed includes: Determine whether the data access request includes the target knowledge base ID to be accessed; If not included, the knowledge base with which the user has access is determined based on the access control list and the user identifier, and this knowledge base is used as the target knowledge base, and the target knowledge base ID is obtained.

4. The method according to any one of claims 1-3, characterized in that, The method further includes: In response to a user's request to create a new knowledge base, the index real-time status table is queried to initially screen and obtain candidate indexes; Determine the health of each candidate index, and select a first index from the candidate indexes for the uniqueness of the new knowledge base based on the health score; The mapping relationship between the new knowledge base ID and the first index is persisted to the metadata table.

5. The method according to claim 4, characterized in that, Determining the health of each candidate index includes: The remaining space is determined based on the current number of documents, the number of shards, and the preset maximum shard capacity of the candidate index; The remaining query frequency is determined based on the current query frequency of the candidate index and the estimated query frequency of the new knowledge base; The uniformity of data distribution is determined based on the amount of data in each knowledge base of the candidate indexes; The remaining space, the remaining query frequency, and the data distribution uniformity are combined to obtain the health score.

6. The method according to claim 1, characterized in that, The method further includes: Real-time monitoring of the current document count and current query frequency of each candidate index; If any candidate index is determined to meet the data migration conditions based on the current number of documents and the current query frequency, then the multiple knowledge bases associated with the candidate index are queried from the metadata table, and the knowledge base to be migrated is selected from the multiple knowledge bases. Determine the health of each candidate index, and select a second index from the candidate indexes based on the health of the candidate indexes to ensure the uniqueness of the knowledge base to be migrated; Using the knowledge base ID to be migrated as the routing value, the unique shard corresponding to the database to be migrated is determined from the multiple shards included in the second index, and data migration is performed; The mapping relationship between the knowledge base ID to be migrated and the second index is persisted to the metadata table.

7. A knowledge base data processing device, characterized in that, include: The Target ID module is used to determine the ID of the target knowledge base to be accessed in response to a user's data access request. The target index module is used to query the target index associated with the target knowledge base from the metadata table based on the target knowledge base ID; wherein, the target index is uniquely determined in advance from the candidate indexes of the vector database cluster when the target knowledge base is created; The target sharding module is used to determine the target shard uniquely corresponding to the target knowledge base from multiple shards of the target index, using the target knowledge base ID as the routing value. The request sending module is used to send the data access request to the target shard, so that the target shard performs request processing.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the knowledge base data processing method as described in any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the knowledge base data processing method as described in any one of claims 1-6.

10. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, implements the knowledge base data processing method according to any one of claims 1-6.

Images