Data storage method, data retrieval method, data storage device, data retrieval device and server

Abstract

The embodiment of the invention discloses a data storage method and device, a data retrieval method and device and a server, in the method, after a data uploading request of a user is received, a corresponding user set identifier can be determined according to a user identifier of the user, and then a storage set corresponding to the user set identifier is obtained; and storing the target data into the storage set. In this way, under the condition that the same user set identifier corresponds to a plurality of user identifiers, users identified by the plurality of user identifiers can share the storage set corresponding to the same user set identifier; therefore, the data uploaded by a plurality of users can be stored in the storage set corresponding to the same user set identifier, so that the condition of more storage fragments caused by independently configuring one storage set for each user is avoided, and the waste of the storage sets is further avoided.

Description

Technical Field

[0001] This application relates to the field of computer technology, and more specifically, to a data storage method, a data retrieval method, an apparatus, and a server. Background Technology

[0002] In the current field of data storage and management, with the rapid growth of users and the explosive increase in data volume, how to efficiently and economically store user data has become a significant challenge for technological development. Among the various storage methods, severe storage fragmentation remains a problem, leading to substantial waste of storage space. Summary of the Invention

[0003] In view of the above problems, this application proposes a data storage method, a data retrieval method, an apparatus, and a server to improve the above problems.

[0004] In a first aspect, this application provides a data storage method, the method comprising: responding to a user's data upload request, determining a corresponding user set identifier based on the user's user identifier; obtaining a storage set corresponding to the user set identifier, wherein there are multiple user identifiers corresponding to the user set identifier; and storing target data in the storage set, wherein the target data is the data requested to be uploaded by the data upload request.

[0005] Secondly, this application provides a data retrieval method, the method comprising: responding to a retrieval request sent by a user, obtaining retrieval requirements corresponding to the retrieval request; obtaining reference content corresponding to the retrieval requirements from a target storage set, wherein the target storage set is determined according to a user set identifier corresponding to the user's user identifier, wherein there are multiple user identifiers corresponding to the user set identifier; and obtaining retrieval results based on the retrieval requirements and the reference content.

[0006] Thirdly, this application provides a data storage device, the device comprising: a set determination unit, configured to determine a corresponding user set identifier based on the user identifier of the user in response to a user's data upload request; a storage set determination unit, configured to obtain a storage set corresponding to the user set identifier, wherein there are multiple user identifiers corresponding to the user set identifier; and a data storage unit, configured to store target data into the storage set, wherein the target data is the data requested to be uploaded by the data upload request.

[0007] Fourthly, this application provides a data retrieval device, the device comprising: a retrieval request acquisition unit, configured to acquire retrieval requests corresponding to the retrieval request in response to a retrieval request sent by a user; a reference content acquisition unit, configured to acquire reference content corresponding to the retrieval request from a target storage set, the target storage set being determined based on a user set identifier corresponding to the user's user identifier, wherein there are multiple user identifiers corresponding to the user set identifier; and a retrieval feedback unit, configured to obtain retrieval results based on the retrieval request and the reference content.

[0008] Fifthly, this application provides a server including one or more processors and a memory; one or more programs are stored in the memory and configured to be executed by the one or more processors to implement the above-described method.

[0009] Sixthly, this application provides a computer-readable storage medium storing program code, wherein the above-described method is executed when the program code is run by a processor.

[0010] This application provides a data storage method, a data retrieval method, an apparatus, and a server. In this method, upon receiving a user's data upload request, a corresponding user set identifier can be determined based on the user's user identifier. Then, the storage set corresponding to the user set identifier is obtained, and the target data is stored in the storage set. This approach allows multiple user identifiers corresponding to the same user set identifier to share the same storage set. This ensures that data uploaded by multiple users can be stored in the same storage set, avoiding storage fragmentation caused by configuring a separate storage set for each user and thus preventing wasted storage space. Attached Figure Description

[0011] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0012] Figure 1 A schematic diagram illustrating an application scenario proposed in an embodiment of this application is shown;

[0013] Figure 2 A flowchart of a data storage method proposed in an embodiment of this application is shown;

[0014] Figure 3A schematic diagram of a virtual user group is shown in an embodiment of this application;

[0015] Figure 4 This application illustrates the functional diagrams of the RAG server and storage server in an embodiment of the present application.

[0016] Figure 5 A schematic diagram illustrating the correspondence between virtual user groups and uploaded files in an embodiment of this application is shown;

[0017] Figure 6 A flowchart of a data storage method proposed in an embodiment of this application is shown;

[0018] Figure 7 A schematic diagram of the system architecture provided in an embodiment of this application is shown;

[0019] Figure 8 A flowchart of a data retrieval method proposed in an embodiment of this application is shown;

[0020] Figure 9 A flowchart illustrating the retrieval results in an embodiment of this application is shown;

[0021] Figure 10 A structural block diagram of a data storage device according to an embodiment of this application is shown;

[0022] Figure 11 This paper shows a structural block diagram of a data retrieval device according to an embodiment of this application;

[0023] Figure 12 This invention illustrates a structural block diagram of an electronic device for performing a data storage method according to an embodiment of the present invention.

[0024] Figure 13 This application illustrates a storage unit for storing or carrying program code that implements the data storage method according to embodiments of this application. Detailed Implementation

[0025] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0026] In today's digital age, with the rapid development of the internet, the Internet of Things, cloud computing, and big data technologies, the user base is expanding at an unprecedented rate, and the amount of data generated is also experiencing explosive growth. This trend is not only reflected in the massive amounts of social media content, online shopping records, and health monitoring data generated by individual users, but also encompasses business data, transaction records, and video surveillance footage in enterprise applications. Therefore, how to efficiently and economically store and manage this massive amount of data has become a key challenge that urgently needs to be addressed in the field of information technology.

[0027] The inventors discovered that some storage methods suffer from severe storage fragmentation, leading to significant waste of storage sets. For example, these methods often create a separate storage set for each user, each with its own complex structure. When the number of users increases dramatically, system memory usage surges. Furthermore, the amount of data each user needs to store varies, resulting in some users having smaller storage sets with less data actually stored. However, the unused data in these user sets cannot be used by other users, further contributing to storage fragmentation.

[0028] Therefore, the inventors have proposed a data storage method, a data retrieval method, an apparatus, and a server in this application. In this method, upon receiving a user's data upload request, the corresponding user set identifier can be determined based on the user's user identifier. Then, the storage set corresponding to the user set identifier is obtained, and the target data is stored in the storage set. This approach allows multiple user identifiers corresponding to the same user set identifier to share the same storage set, ensuring that data uploaded by multiple users is stored in the same storage set. This avoids the fragmentation caused by configuring a separate storage set for each user, thus preventing a waste of storage space.

[0029] The following is an exemplary application scenario provided by the embodiments of this application.

[0030] like Figure 1 As shown, in Figure 1 The scenario shown includes server 100 and server 200.

[0031] Server 100 can function as a RAG (Retrieval-Augmented Generation) server. The RAG server runs the RAG system (i.e., a retrieval-augmented generation system), a model architecture that combines retrieval and generation technologies. This system can be used to answer open-domain questions, generate relevant text, and handle tasks requiring rich background information.

[0032] Server 200 can function as a storage server. The storage server can be used to store data uploaded by users. In this embodiment, the data uploaded by users can be documents or images, etc. Figure 1 As shown, user-uploaded documents can first be processed through the RAG server, including analysis, segmentation, and vectorization, before the resulting vectorized data is stored on the storage server. When a user needs to perform a search, relevant content can be retrieved from the storage server as reference content based on the user's search query. This reference content is then transmitted to the Large Language Model (LLM) to obtain the search feedback content.

[0033] The embodiments of this application will now be described in detail with reference to the accompanying drawings.

[0034] Please see Figure 2 This application provides a data storage method, the method comprising:

[0035] S110: In response to the user's data upload request, determine the corresponding user set identifier based on the user's user identifier.

[0036] When a user needs to upload data, they can do so via electronic device (smartphone, tablet, smartwatch, etc.) to the server executing the data storage method (e.g., the aforementioned). Figure 1 The server 100 sends a data upload request. This data upload request may include a user identifier, allowing the server to identify the specific user requesting the data upload upon receiving the request.

[0037] After obtaining the user identifier, the corresponding user set identifier can be further obtained based on the user identifier. The user set identifier is used to uniquely identify the user set. In this embodiment, the user set can be understood as a virtual user group. In this embodiment, a correspondence between user identifiers and user set identifiers is established. Through this correspondence, the virtual user set to which each user belongs can be clearly identified.

[0038] In this application embodiment, there are multiple ways to determine the corresponding user set identifier based on the user identifier.

[0039] One approach is to calculate the corresponding user set identifier in real time based on the user identifier and a preset algorithm.

[0040] As another approach, the user set identifier corresponding to a user identifier can be obtained based on the user attributes associated with that user identifier. The user attributes can be understood as the user attributes of the user identified by the user identifier. These user attributes can represent a user's preferences or interests. Assigning users with similar or identical attributes to the same virtual user group can facilitate providing the same services to users with similar or identical attributes, while providing differentiated services to users with dissimilar or different attributes.

[0041] As another approach, the corresponding user set identifier can be obtained based on the pre-stored correspondence between user identifiers and user set identifiers. Optionally, after obtaining the user identifier, it can be first checked whether a corresponding user set identifier is stored. If a corresponding user set identifier is already stored, it can be directly read. If a corresponding user set identifier is not stored, the corresponding user set identifier can be calculated in real time based on the user identifier and a preset algorithm, and the calculated user set identifier can be stored in correspondence with the user identifier.

[0042] For example, the correspondence between user identifiers and user set identifiers can be shown in the following table:

[0043]

[0044]

[0045] As shown in the table above, user identifiers U1 and U2 both correspond to user set identifier J1, meaning that user identifiers U1 and U2 belong to the same virtual user group (the virtual user group corresponding to user set identifier J1). Similarly, user identifiers U3 and U4 both correspond to user set identifier J2, meaning that user identifiers U3 and U4 belong to the same virtual user group (the virtual user group corresponding to user set identifier J2). According to the table, if the obtained user identifier is U1, the user set identifier J1 can be directly obtained based on the correspondence in the table. If the obtained user identifier is U5, it will be determined that there is no corresponding user set identifier, and the corresponding user set identifier can be calculated in real time according to a preset algorithm.

[0046] S120: Obtain the storage set corresponding to the user set identifier, wherein there are multiple user identifiers corresponding to the user set identifier.

[0047] In this embodiment, a storage set corresponds to a storage space. This can be understood as the storage set itself representing a type of storage space, or as the storage set being associated with a single storage space. Each user set identifier corresponds to multiple user identifiers. In this case, users identified by multiple user identifiers can share the storage set corresponding to the same user set identifier. Sharing the storage set corresponding to the same user set identifier can also be understood as sharing the storage space corresponding to the same user set identifier.

[0048] For example, the multiple user identifiers corresponding to user set identifier J1 include: user identifier U1, user identifier U2, user identifier U3, and user identifier U4. In this case, the users identified by user identifiers U1, U2, U3, and U4 can be understood as belonging to the same virtual user group. When user set identifier J1 corresponds to storage set S1, the users identified by user identifiers U1, U2, U3, and U4 can share storage set S1.

[0049] In this embodiment, different user set identifiers correspond to different storage sets. This can be understood as the storage sets themselves being different, and their sizes being different. For example, if there are user set identifiers J1, J2, and J3, where user set identifier J1 corresponds to storage set S1, user set identifier J2 corresponds to storage set S2, and user set identifier J3 corresponds to storage set S3, then storage sets S1, S2, and S3 are themselves different, and their sizes are also different.

[0050] One approach is to configure the size of the storage set corresponding to different user set identifiers. Another approach is to determine the size of the storage set corresponding to different user set identifiers based on the number of user identifiers associated with that user set identifier. It should be noted that within the same virtual user group, the more users included, the more data will likely be stored in the storage set corresponding to that virtual user group (user set identifier). In this case, to dynamically match the number of users included in each virtual user group, the more user identifiers an individual user set identifier corresponds to, the larger the storage set corresponding to that user set identifier will be.

[0051] S130: Store the target data in the storage set, where the target data is the data requested to be uploaded by the data upload request.

[0052] like Figure 3 As shown, data can be stored in a vectorized form within the storage set corresponding to a user set identifier (virtual user group). One approach is... Figure 4 and Figure 5 As shown, the target data can be split into multiple data blocks, which are then converted into corresponding vector forms and stored in the storage set. Alternatively, the target data can be directly stored in the storage set.

[0053] This embodiment provides a data storage method that, in the case where multiple user identifiers correspond to the same user set identifier, the storage set corresponding to the same user set identifier can be shared for the users identified by the multiple user identifiers. This allows the data uploaded by multiple users to be stored in the same storage set corresponding to the same user set identifier, avoiding the situation of configuring a separate storage set for each user, which would result in a lot of storage fragmentation and thus avoid the waste of storage sets.

[0054] Please see Figure 6 This application provides a data storage method, the method comprising:

[0055] S210: In response to a user's data upload request, perform a modulo operation based on the user's user identifier and the target capacity value, and use the modulo result as a determined user set identifier. The target capacity value is the maximum capacity value of the storage set corresponding to the user set identifier, and the user identifier consists of numbers.

[0056] The modulo operation is used to find the remainder after dividing two numbers. In this embodiment, the user identifier is moduloed with the target capacity value, and the result can be used as the identifier of the user set. For example, assuming there are 100 users and the target capacity value is 10, the result of the modulo operation on user identifier 11 may be 1, and 1 can be used as the identifier of the user set to which this user belongs.

[0057] As one approach, performing a modulo operation based on the user identifier and the target capacity value to use the modulo result as a determined user set identifier may include: performing a hash operation on the user identifier to obtain a hash operation result, and performing a modulo operation based on the hash operation result and the target capacity value to use the modulo result as a determined user set identifier.

[0058] Hash operations are algorithms that convert inputs of arbitrary length into outputs of fixed length. Therefore, by hashing a user identifier, it can be converted into a fixed-length hash value. Then, the hash result (hash value) is modulo-divided by the target capacity value, and the result of this modulo operation is used as the identifier of the user set. For example, suppose a user identifier "user123" has a hash value of "123456" and a target capacity of 100. After the modulo operation, the result might be 36, and 36 becomes the identifier of the user set to which that user belongs.

[0059] The above method allows users to be evenly distributed into different sets within a certain range, which facilitates management and operation.

[0060] Optionally, the modulo operation based on the hash result and the target capacity value, to use the modulo result as the identifier of the determined user set, may include: performing a modulo operation based on the hash result and the target capacity value to obtain the modulo result, and adding a salt value to the modulo result as the identifier of the determined user set.

[0061] As mentioned earlier, the modulo operation is an arithmetic operation used to find the remainder when two numbers are divided. In this process, performing a modulo operation on the hash result maps the hash value to a smaller range, determined by the "target capacity value." The salt is a random value added during the hashing process to increase complexity and uniqueness. The main purpose of adding a salt to the modulo result in this process is to further improve the complexity and uniqueness of the generated user set identifiers. The salt can be any unpredictable string or number. By adding a unique salt to each user set identifier, even if the hash value and modulo result are the same, the final user set identifier will be different due to the different salt values.

[0062] S220: Obtain the storage set corresponding to the user set identifier, wherein there are multiple user identifiers corresponding to the user set identifier.

[0063] S230: Store the target data in the storage set, wherein the target data is the data requested to be uploaded by the data upload request.

[0064] This embodiment provides a data storage method that avoids the storage fragmentation caused by configuring a separate storage set for each user, thus preventing waste of storage sets. Furthermore, in this embodiment, a modulo operation is performed based on the user identifier and the target capacity value, and the modulo result is used as the determined user set identifier. This allows the corresponding user set identifier to be determined based on the user identifier, improving the flexibility of the data storage process. Moreover, in the further determination of the user set identifier, hash operations and the addition of a salt value can be used to further ensure the uniqueness of the user set identifier.

[0065] In the embodiments of this application, such as Figure 7 As shown, there can be multiple storage servers used for data storage, and these multiple storage servers can form a storage server cluster. In this case, the storage set corresponding to a user set identifier can be located in a certain storage server, and the storage sets corresponding to different user set identifiers are evenly distributed among the multiple storage servers to achieve load balancing. Load balancing is a technique that distributes workloads (such as network traffic, data requests, computing tasks, data storage, etc.) to multiple computing resources (such as servers, virtual machines, containers, etc.). Its main purpose is to optimize resource utilization, improve system response speed, and enhance system fault tolerance by distributing requests across multiple servers.

[0066] In this scenario, multiple storage servers are connected together via a network, and each storage server has certain storage resources and computing power to store and process the data assigned to it, thus jointly undertaking the task of data storage and processing.

[0067] As one approach, if the number of users increases and meets preset conditions, the number of storage servers is increased. These storage servers are used to store the data uploaded by users. It should be noted that the number of users in the system (the system constituted by the servers in this embodiment) changes dynamically. In some cases, the number of users may increase significantly. This significant increase may put excessive pressure on the currently running storage servers. In such cases, additional storage servers can be added to meet the data storage needs of the newly added users.

[0068] Optionally, a maximum number of users that a storage server can support can be pre-configured. In this case, if the number of users currently supported by the server is close to, equal to, or exceeds the maximum number of users, a new storage server can be added. After adding a new storage server, some users can store their data on the new storage server.

[0069] Please see Figure 8 This application provides a data retrieval method, the method comprising:

[0070] S310: In response to the search request sent by the user, obtain the search requirements corresponding to the search request.

[0071] In this context, the search query can be the actual content the user inputs. In this case, the user directly enters the keywords, phrases, or sentences they want to search for, and the system uses these inputs as search queries directly in the search process. This method is simple and direct, suitable for situations where the user can accurately describe their needs. For example, if a user enters "What restaurants are nearby?", the system will directly use this string as the search query.

[0072] Alternatively, the system can perform semantic recognition on the user's input search terms and use the recognized content as the search query. In this case, the system not only accepts the raw text input by the user but also uses Natural Language Processing (NLP) technology to perform semantic analysis on this text, understand the user's true intent, and transform this understanding into a more precise and comprehensive search query. This approach can handle the ambiguity and vagueness of user input, improving the accuracy and relevance of the search. For example, if the user inputs "What will the weather be like tomorrow?", the system, through semantic recognition technology, understands that the user wants to query the weather forecast for "current date + 1 day", and thus transforms the search query into a form like "Weather forecast for XXXX year XX month XX day" for the search. If the system is further intelligent, it may also automatically add a regional prefix based on the user's geographical location information, such as "Chengdu weather forecast for XXXX year XX month XX day".

[0073] S320: Obtain reference content corresponding to the retrieval requirement from the target storage set, wherein the target storage set is determined based on the user set identifier corresponding to the user's user identifier, and there are multiple user identifiers corresponding to the user set identifier.

[0074] In this embodiment, the reference content can be understood as content used for reference when performing retrieval based on search requirements. Optionally, reference content corresponding to the search requirements can be obtained from the target storage set by calculating similarity.

[0075] After obtaining the search requirements, these requirements can be converted into a corresponding vectorized form to obtain the vectorized search requirements. Then, as... Figure 9 As shown, content stored on the storage server that has a high similarity to the vectorized search query can be calculated and used as reference content. Optionally, if there is content stored on the storage server that has a similarity greater than a similarity threshold with the vectorized search query, then the similarity is determined to be high.

[0076] In this application embodiment, there are multiple ways to calculate similarity. One method is to use cosine similarity to evaluate the similarity between two vectorized contents. The formula is:

[0077] sim(A,B)=cos(θ)=(A·B) / (|A||B|), where A and B are two vectors, A·B represents their dot product, and |A| and |B| are their moduli.

[0078] Cosine similarity works by measuring the similarity of two vectors in direction, not their size or length. Specifically, it's calculated by taking the dot product of the two vectors and dividing by the product of their lengths. Cosine similarity values ​​range from -1 to 1. A cosine similarity of 1 means two vectors are in exactly the same direction; -1 means they are in completely opposite directions; and 0 means they are perpendicular. Cosine similarity is particularly useful for textual data or any data that can be represented as vectors, where we are more concerned with the directional differences between data points than their size or absolute position.

[0079] As another approach, the similarity between two vectorized contents can also be calculated using Euclidean distance.

[0080] In the case where the data uploaded by the user is a document, the reference content obtained may be a portion of the content in the uploaded document.

[0081] S330: Based on the search requirements and the reference content, the search results are obtained.

[0082] After obtaining the search requirements and reference content, the search requirements and reference content can be transmitted together to the large language model so that the search results can be obtained through the large language model.

[0083] The reference content is retrieved from the storage server, and the content stored on the storage server is also uploaded by the user. Therefore, reference content can be understood as information that the user may already possess or understand. This information can serve as context or supplementary material during the retrieval process to enhance the search. Reference content can take various forms, such as text, images, and videos, but in most large language model applications, it is primarily in text form. Reference content helps the large language model better understand the user's search needs, especially when dealing with complex or ambiguous questions.

[0084] The process of transmitting search requests and reference content to a large language model can be achieved through an API interface or a specific interactive interface. In this step, the user-input search requests and reference content are encoded into a computer-understandable format and then sent to the large language model for processing. Upon receiving this information, the large language model leverages its powerful natural language processing capabilities and knowledge base to conduct in-depth understanding and analysis of the search requests and reference content.

[0085] When processing received information, the large language model performs the following steps: Step 1: Understanding. First, the model attempts to understand the user's search requirements and reference content, including keywords, topics, and context. Step 2: Inference. Based on the understood content, the model performs inference, attempting to find the most relevant and valuable information related to the search requirements. Step 3: Generation. Finally, the model generates corresponding search results based on the inference results. These results may be answers in text form, recommended article lists, related knowledge fragments, etc. The search results generated by the large language model are returned to the user through the same interface (i.e., the interface or interface where the user inputs their search requirements).

[0086] Please see Figure 10 This application provides a data storage device 400, which includes:

[0087] The set determination unit 410 is used to determine the corresponding user set identifier based on the user's user identifier in response to the user's data upload request.

[0088] Storage set determination unit 420 is used to obtain the storage set corresponding to the user set identifier, wherein there are multiple user identifiers corresponding to the user set identifier.

[0089] Data storage unit 430 is used to store target data into the storage set, wherein the target data is the data requested to be uploaded by the data upload request.

[0090] In one approach, the user identifier is composed of numbers. In this approach, the set determination unit 410 is specifically used to perform a modulo operation based on the user identifier and the target capacity value, so as to use the modulo result as the determined user set identifier, where the target capacity value is the maximum capacity value of the storage set corresponding to the user set identifier.

[0091] Optionally, the set determination unit 410 is specifically used to perform a hash operation on the user identifier to obtain a hash operation result; and to perform a modulo operation based on the hash operation result and the target capacity value to use the modulo result as the determined user set identifier.

[0092] Optionally, the set determination unit 410 is specifically used to perform a modulo operation based on the hash operation result and the target capacity value to obtain the modulo result; and to add a salt value to the modulo result as the identifier of the determined user set.

[0093] In one approach, the data storage unit 430 is specifically used to split the target data to obtain multiple data blocks; after converting the multiple data blocks into corresponding vector forms, they are stored in the storage set.

[0094] Please see Figure 11 This application provides a data retrieval device 500, which includes:

[0095] The retrieval request acquisition unit 510 is used to acquire the retrieval request corresponding to the retrieval request in response to the retrieval request sent by the user.

[0096] The reference content acquisition unit 520 is used to acquire reference content corresponding to the retrieval requirement from the target storage set, wherein the target storage set is determined according to the user set identifier corresponding to the user's user identifier, and there are multiple user identifiers corresponding to the user set identifier.

[0097] The retrieval feedback unit 530 is used to obtain retrieval results based on the retrieval requirements and the reference content.

[0098] This application provides a data storage device and a data retrieval device, which enable the sharing of storage sets corresponding to the same user set identifier when multiple user identifiers correspond to the same user set identifier. This allows data uploaded by multiple users to be stored in the same storage set corresponding to the same user set identifier, avoiding the situation of configuring a separate storage set for each user, which would result in a lot of storage fragmentation and thus avoid the waste of storage sets.

[0099] It should be noted that the device embodiments in this application correspond to the aforementioned method embodiments. The specific principles in the device embodiments can be found in the content of the aforementioned method embodiments, and will not be repeated here.

[0100] The following will combine Figure 12 This application describes one type of server.

[0101] Please see Figure 12 Based on the aforementioned data storage method and apparatus, this application embodiment also provides another server 2000 capable of executing the aforementioned data storage method. The server 2000 includes one or more (only one shown in the figure) processors 202, a memory 204, and a network module 206 coupled together. The memory 204 stores programs capable of executing the contents of the aforementioned embodiments, and the processors 202 can execute the programs stored in the memory 204.

[0102] The processor 202 may include one or more processing cores. The processor 202 connects to various parts of the server 2000 via various interfaces and lines, and performs various functions and processes data of the server 2000 by running or executing instructions, programs, code sets, or instruction sets stored in the memory 204, and by calling data stored in the memory 204. Optionally, the processor 202 may be implemented using at least one hardware form of Digital Signal Processing (DSP), Field-Programmable Gate Array (FPGA), or Programmable Logic Array (PLA). The processor 202 may integrate one or a combination of several of the following: Central Processing Unit (CPU), Graphics Processing Unit (GPU), and modem. The CPU primarily handles the operating system, user interface, and applications; the GPU is responsible for rendering and drawing the displayed content; and the modem handles wireless communication. It is understood that the modem may also not be integrated into the processor 202 and may be implemented separately using a communication chip.

[0103] The memory 204 may include random access memory (RAM) or read-only memory (ROM). The memory 204 can be used to store instructions, programs, code, code sets, or instruction sets. The memory 204 may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for implementing at least one function (such as touch functionality, sound playback functionality, image playback functionality, etc.), and instructions for implementing the various method embodiments described below.

[0104] The network module 206 is used to enable information interaction between the server 2000 and other devices, such as transmitting device control commands, manipulation request commands, and status information acquisition commands. However, the network module 206 may differ depending on the specific device that the server 2000 is used for.

[0105] In one approach, the network module of the server 2000 is a radio frequency (RF) module. This RF module is used to receive and transmit electromagnetic waves, converting electromagnetic waves into electrical signals, thereby enabling communication with communication networks or other devices. The RF module may include various existing circuit elements used to perform these functions, such as antennas, RF transceivers, digital signal processors, encryption / decryption chips, user identity modules (SIM cards), memory, etc. For example, the RF module can interact with external devices by transmitting or receiving electromagnetic waves, thereby receiving audio signals transmitted by external devices.

[0106] Please refer to Figure 13 This diagram illustrates a structural block diagram of a computer-readable storage medium provided in an embodiment of this application. The computer-readable medium 800 stores program code that can be called by a processor to execute the methods described in the above method embodiments.

[0107] The computer-readable storage medium 800 may be an electronic memory such as flash memory, EEPROM (Electrically Erasable Programmable Read-Only Memory), EPROM, hard disk, or ROM. Optionally, the computer-readable storage medium 800 includes a non-transitory computer-readable storage medium. The computer-readable storage medium 800 has a storage set of program code 810 that performs any of the method steps described above. This program code can be read from or written to one or more computer program products. The program code 810 may, for example, be compressed in a suitable form.

[0108] This application provides a data storage method, a data retrieval method, an apparatus, and a server. In this method, upon receiving a user's data upload request, a corresponding user set identifier can be determined based on the user's user identifier. Then, the storage set corresponding to the user set identifier is obtained, and the target data is stored in the storage set. This approach allows multiple user identifiers corresponding to the same user set identifier to share the same storage set. This ensures that data uploaded by multiple users can be stored in the same storage set, avoiding storage fragmentation caused by configuring a separate storage set for each user and thus preventing wasted storage space.

[0109] This application employs a hash algorithm to ensure that data uploaded by the same user (e.g., documents) can be centrally stored (e.g., in a storage set containing user identifiers and their corresponding user combination identifiers), thereby reducing the search scope during subsequent data retrieval. For example, when user U1 uploads document content, the user set identifier corresponding to user U1 is determined to be user set identifier J2. The document content uploaded by user U1, after vectorization, is then stored in the storage set corresponding to user set identifier J2. Subsequently, if user U1 needs to perform data retrieval using a large language model, the corresponding user set identifier J2 is first obtained through user U1, and the data is directly retrieved from the storage set corresponding to user set identifier J2, without needing to search elsewhere, thus reducing the search scope during data retrieval.

[0110] Furthermore, this application employs a multi-node cluster design, achieving load balancing and incremental expansion through service control. Moreover, the system is easily scalable, capable of handling surges in user volume while maintaining system stability.

[0111] Furthermore, this application proposes a highly efficient storage and retrieval system for RAG systems with massive user volumes and batch document uploads through novel optimizations of the vectorized storage structure and network topology. This system not only significantly improves storage efficiency and retrieval speed but also possesses strong scalability, providing RAG systems with efficient and stable vector library services, thus enhancing user experience and system stability. Moreover, the solution provided in this application is not only applicable to semantic retrieval but can also be applied to other scenarios requiring text vectorized storage and efficient retrieval, such as recommendation systems, knowledge graph construction, and intelligent question-answering systems. In the future, this solution can be further optimized and expanded to meet the needs of different scenarios, improving its versatility and adaptability.

[0112] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A data storage method, characterized in that, The method includes: In response to a user's data upload request, the corresponding user set identifier is determined based on the user's user identifier; Obtain the storage set corresponding to the user set identifier, wherein there are multiple user identifiers corresponding to the user set identifier; The target data is stored in the storage set, and the target data is the data requested to be uploaded by the data upload request.

2. The method according to claim 1, characterized in that, The user identifier consists of numbers, and determining the corresponding user set identifier based on the user's user identifier includes: A modulo operation is performed based on the user identifier and the target capacity value, and the modulo result is used as the identifier of the determined user set. The target capacity value is the maximum capacity value of the storage set corresponding to the user set identifier.

3. The method according to claim 2, characterized in that, The step of performing a modulo operation based on the user identifier and the target capacity value, and using the modulo result as the identifier of the determined user set, includes: Perform a hash operation on the user identifier to obtain the hash result; A modulo operation is performed based on the hash result and the target capacity value, and the modulo result is used as the identifier of the determined user set.

4. The method according to claim 3, characterized in that, The step of performing a modulo operation based on the hash result and the target capacity value, and using the modulo result as the identifier of the determined user set, includes: A modulo operation is performed based on the hash result and the target capacity value to obtain the modulo result; A salt value is added to the modulus result to serve as an identifier for the determined user set.

5. The method according to any one of claims 1-4, characterized in that, The step of storing the target data into the storage set includes: The target data is split into multiple data blocks; After the multiple data blocks are converted into corresponding vector forms, they are stored in the storage set.

6. The method according to any one of claims 1-4, characterized in that, The storage set corresponding to the user set identifier is located in one of the multiple storage servers, and the storage sets corresponding to different user set identifiers are evenly distributed among the multiple storage servers.

7. The method according to any one of claims 1-4, characterized in that, The method further includes: If the number of users meets a preset condition after the increase, the number of storage servers will be increased. The storage servers are used to store the data uploaded by users.

8. A data retrieval method, characterized in that, The method includes: In response to a user's search request, obtain the search requirements corresponding to the search request; Retrieve reference content corresponding to the retrieval requirement from the target storage set, wherein the target storage set is determined based on the user set identifier corresponding to the user's user identifier, and there are multiple user identifiers corresponding to the user set identifier; Based on the search requirements and the reference content, the search results are obtained.

9. A data storage device, characterized in that, The device includes: The set determination unit is used to determine the corresponding user set identifier based on the user identifier in response to the user's data upload request; A storage set determination unit is used to obtain the storage set corresponding to the user set identifier, wherein there are multiple user identifiers corresponding to the user set identifier; A data storage unit is used to store target data into the storage set, wherein the target data is the data requested to be uploaded by the data upload request.

10. A data retrieval device, characterized in that, The device includes: The retrieval request acquisition unit is used to acquire the retrieval request corresponding to the retrieval request in response to the retrieval request sent by the user. The reference content acquisition unit is used to acquire reference content corresponding to the retrieval requirement from the target storage set, wherein the target storage set is determined according to the user set identifier corresponding to the user's user identifier, and there are multiple user identifiers corresponding to the user set identifier; The retrieval feedback unit is used to obtain retrieval results based on the retrieval requirements and the reference content.

11. A server, characterized in that, It includes one or more processors and memory; one or more programs are stored in the memory and configured to be executed by the one or more processors according to any one of claims 1-7, or according to claim 8.

12. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores program code, wherein when the program code is run by a processor, it performs the method of any one of claims 1-7, or the method of claim 8.

Images