Computing power task fault tolerance method and device of intelligent computing center

Abstract

The invention provides a computing power task fault tolerance method and device of an intelligent computing center, and the method comprises the steps: S1, after receiving an import data request for a DiskANN object sent by an Index service node, checking whether the DiskANN object allows to import vector data or not; s2, if yes, judging whether a data import request for the DiskANN object is received for the first time or not; s3, if yes, recording first information carried in the data importing request, and executing an operation of importing vector data; the first information comprises first verification information, and the first verification information is obtained from the first service node when the Index service node initiates to import vector data to the DiskANN object and has uniqueness; and S4, if not, comparing whether the first information carried in the currently received import data request is the same as the first information carried in the previously received import data request for the DiskANN object, if so, executing the operation of importing the vector data, and if not, refusing the currently received import data request.

Description

Technical Field

[0001] Embodiments of the present invention relate to the technical field of computing power infrastructure, and in particular to a computing power task fault tolerance method and device for an intelligent computing center. Background Art

[0002] With the development of artificial intelligence technology and computing power technology, the concept of intelligent computing center has emerged. "Intelligent computing center" refers to the use of large-scale heterogeneous computing power resources, including general computing power and intelligent computing power, mainly for artificial intelligence applications (such as artificial intelligence deep learning model development, model training and model reasoning and other scenarios) to provide the required computing power, data and algorithms. Intelligent computing center covers facilities, hardware, software, and can provide full-stack capabilities from bottom-level computing power to top-level application enablement.

[0003] The Intelligent Computing Center supports vector retrieval based on DiskANN. DiskANN is a vector retrieval engine based on distributed storage, which can store and retrieve billions of vector data on a single computer. Compared with traditional vector retrieval algorithms, DiskANN has higher storage efficiency and faster retrieval speed.

[0004] Since the DiskANN service node usually needs to process large amounts of data when processing computing tasks for DiskANN objects, how to achieve fault tolerance for computing tasks is an urgent problem to be solved by the intelligent computing center. Summary of the invention

[0005] The embodiment of the present invention provides a computing task fault tolerance method and device for an intelligent computing center, which is used to solve the problem of how a DiskANN service node can achieve computing task fault tolerance.

[0006] In order to solve the above-mentioned technical problems, the present invention is achieved as follows:

[0007] In a first aspect, an embodiment of the present invention provides a computing task fault tolerance method for an intelligent computing center, including:

[0008] Step S1: after receiving the data import request for the DiskANN object sent by the Index service node, the DiskANN service node checks whether the DiskANN object currently allows the import of vector data;

[0009] Step S2: If the DiskANN object currently allows the import of vector data, the DiskANN service node determines whether it is the first time to receive a data import request for the DiskANN object;

[0010] Step S3: If it is the first time to receive the import data request for the DiskANN object, the DiskANN service node records the first information carried in the import data request, and performs the operation of importing vector data on the DiskANN object; the first information includes: first verification information, which is obtained by the Index service node from the first service node when initiating the import of vector data for the DiskANN object, and is unique;

[0011] Step S4: If it is not the first time that an import data request for the DiskANN object is received, the DiskANN service node compares the first information carried in the currently received import data request for the DiskANN object with the first information carried in the previously received import data request for the DiskANN object to see if they are the same; if they are the same, an operation of importing vector data is performed on the DiskANN object; if they are not the same, the currently received import data request for the DiskANN object is rejected.

[0012] Optionally, the step S1 includes:

[0013] Step S11: the DiskANN service node checks the status of the DiskANN object;

[0014] Step S12: If the state of the DiskANN object is the initialization completed state, the DiskANN service node determines that the DiskANN object currently allows the import of vector data;

[0015] Step S13: If the status of the DiskANN object is the data import completion status, the graph index building status, the graph index building status, the graph index loading into the memory status, or the graph index already loaded into the memory status, the DiskANN service node determines that the DiskANN object is currently not allowed to import vector data.

[0016] Optionally, the first verification information is timestamp information.

[0017] Optionally, the first information further includes: second verification information, where the second verification information is used to indicate version information of the vector data carried in the import data request.

[0018] Optionally, also include:

[0019] Step S4: the DiskANN service node obtains the second information carried in the import data request, where the second information is used to indicate whether there is more vector data to be sent later;

[0020] Step S5: The DiskANN service node determines the message type of the received import data request according to the second information, and the message type includes: the first import data request for the DiskANN object, the intermediate import data request, and the last import data request.

[0021] Optionally, also include:

[0022] Step S6: The DiskANN service node obtains its own computing power resource information, determines third information based on the computing power resource information, where the third information is used to indicate the recommended number of vector data to be sent, and sends the third information to the Index service node.

[0023] In a second aspect, an embodiment of the present invention provides a computing task fault-tolerant device for an intelligent computing center, comprising:

[0024] A detection module, configured to check whether the DiskANN object currently allows the import of vector data after receiving a data import request for the DiskANN object sent by the Index service node;

[0025] A judgment module, used for judging whether it is the first time to receive a request for importing data for the DiskANN object if the DiskANN object currently allows importing vector data;

[0026] A first processing module, configured to record first information carried in the data import request and perform an operation of importing vector data on the DiskANN object if it is the first time to receive the data import request for the DiskANN object; the first information includes: first verification information, which is obtained by the Index service node from the first service node when initiating the import of vector data for the DiskANN object, and is unique;

[0027] The second processing module is used for comparing the first information carried in the currently received import data request for the DiskANN object with the first information carried in the previously received import data request for the DiskANN object to see if they are the same; if they are the same, performing an operation of importing vector data on the DiskANN object; if they are not the same, rejecting the currently received import data request for the DiskANN object.

[0028] In a third aspect, an embodiment of the present invention provides a server comprising: a processor, a memory, and a program stored in the memory and executable on the processor, wherein when the program is executed by the processor, the steps of the computing power task fault tolerance method of the intelligent computing center as described in the first aspect above are implemented.

[0029] In a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the steps of the computing power task fault tolerance method of the intelligent computing center as described in the first aspect above are implemented.

[0030] In a fifth aspect, a computer program product is provided, comprising computer instructions, which, when executed by a processor, implement the steps of the computing power task fault tolerance method of the intelligent computing center as described in the first aspect above.

[0031] In an embodiment of the present invention, after receiving the import data request for the DiskANN object sent by the Index service node, the DiskANN service node checks whether the DiskANN object currently allows the import of vector data, and compares the first information carried in the currently received import data request for the DiskANN object with whether the first information carried in the previously received import data request for the DiskANN object is the same, to check whether the currently received import data request for the DiskANN object is wrong. If an error occurs, the currently received import data request for the DiskANN object can be rejected to ensure that errors occurring during the data import process do not affect the normal import process. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Various other advantages and benefits will become apparent to those of ordinary skill in the art by reading the detailed description of the preferred embodiments below. The accompanying drawings are only for the purpose of illustrating the preferred embodiments and are not to be considered as limiting the present invention. Moreover, the same reference symbols are used throughout the accompanying drawings to represent the same components. In the accompanying drawings:

[0033] Figure 1 This is a schematic diagram of the overall process of vector data query based on DiskANN;

[0034] Figure 2 Schematic diagram of the process of importing data from the Index service node to the DiskANN service node;

[0035] Figure 3 This is one of the flow charts of the computing task fault tolerance method of the intelligent computing center according to an embodiment of the present invention;

[0036] Figure 4This is a second flow chart of the computing task fault tolerance method of the intelligent computing center according to an embodiment of the present invention;

[0037] Figure 5 This is a flowchart of the third method for fault tolerance of computing tasks in an intelligent computing center according to an embodiment of the present invention;

[0038] Figure 6 It is a structural schematic diagram of a computing task fault-tolerant device of an intelligent computing center according to an embodiment of the present invention;

[0039] Figure 7 Schematic diagram of the structure of a server according to an embodiment of the present invention. DETAILED DESCRIPTION

[0040] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0041] First, the technical terms involved in the present invention are briefly explained below.

[0042] The "computing power" mentioned in the present invention is the ability of computer equipment or computing / data center to process information. It is the ability of computer hardware and software to work together to execute certain computing requirements. It is the computing power to achieve target result output by processing information data. It is a new type of productivity that integrates information computing power, network carrying capacity, and data storage capacity. It mainly provides services to the society through computing power infrastructure.

[0043] The "computing power" (Computational Power, CP) described in the present invention is the ability of a data center server to process data and output results. It is a comprehensive indicator to measure the computing power of a data center, including general computing power, super computing power and intelligent computing power. The commonly used unit of measurement is the number of floating point operations performed per second (FLOPS: Floating Point Operations Per Second, 1EFLOPS = 10^18FLOPS). The larger the value, the stronger the comprehensive computing power. According to calculations, 1EFLOPS is approximately the computing power output of 5 Tianhe-2A or 500,000 mainstream server CPUs or 2 million mainstream notebooks. The calculation formula is: CP = CP 通用 +CP 智能 +CP 超级 .

[0044] The "Network Power" (NP) described in the present invention is a manifestation of the data transmission capability of computing facilities, including comprehensive capabilities such as network architecture, network bandwidth, transmission latency, intelligent management and scheduling, etc. The carrying capacity involves network transmission within and between data centers, and is a comprehensive indicator for measuring network transmission scheduling capabilities. In the embodiment of the present invention, the carrying capacity uses the memory bandwidth.

[0045] The "Storage Power" (SP) described in the present invention is the comprehensive ability of a data center in terms of data storage capacity, performance, safety and reliability, and green and low-carbon. It is a comprehensive indicator for measuring the data storage capacity of a data center, including external storage devices such as storage arrays and server built-in storage devices. The commonly used unit of measurement for storage capacity is exabyte (EB, 1EB = 2^60bytes), and the commonly used unit of measurement for performance is the number of reads and writes per second per unit capacity (IOPS / TB, Input / Output Operations Per Second / TB). The disaster recovery ratio is an important manifestation of safety and reliability.

[0046] The "computing power infrastructure" described in the present invention is a new type of information infrastructure that integrates information computing power, network carrying capacity, and data storage capacity. It can realize centralized computing, storage, transmission and application of information, and presents characteristics such as diversity and ubiquity, intelligence and agility, security and reliability, and green and low-carbon.

[0047] The “computing power” mentioned in the present invention includes general computing power, intelligent computing power and super computing power.

[0048] The “general computing power” mentioned in the present invention refers to the computing power provided by servers based on CPU (Central Processing Unit) chips, which is used to support basic general computing such as cloud computing and edge computing.

[0049] The "intelligent computing power" described in the present invention is a computing platform for large-scale deployment of special chips such as GPU (Graphics Processing Unit), FPGA (Field Programmable Gate Array), ASIC (Application Specific Integrated Circuit) for various innovative applications of artificial intelligence, such as natural language processing and machine vision.

[0050] The "super computing power" mentioned in the present invention is mainly the computing power provided by high-performance computing clusters such as supercomputers. It utilizes the centralized computing resources of multiple computer systems working in parallel and uses a dedicated operating system to handle extremely complex or data-intensive problems. It is mainly used for calculations in cutting-edge scientific fields, such as planetary simulation, drug molecule design, and gene analysis.

[0051] The "intelligent computing center" described in the present invention refers to a facility that provides the required computing power, data and algorithms for artificial intelligence applications (such as artificial intelligence deep learning model development, model training and model reasoning scenarios) by using large-scale heterogeneous computing power resources, including general computing power (CPU: Central Processing Unit) and intelligent computing power (GPU: Graphics Processing Unit, FPGA: Field Programmable Gate Array, ASIC: Application Specific Integrated Circuit, etc.). The intelligent computing center covers facilities, hardware, and software, and can provide full-stack capabilities from bottom-level computing power to top-level application enablement.

[0052] The "computing resources" mentioned in the present invention refer to the technologies and facilities with information calculation, transmission, storage and application capabilities required for the development of the digital society, including but not limited to computing resources such as CPU (Central Processing Unit), GPU (Graphics Processing Unit), network resources such as switches and routers, storage resources such as storage arrays and distributed storage, security resources such as firewalls and intrusion detection systems, as well as supporting and guarantee resources such as wind, fire, water and electricity.

[0053] The "DiskANN" described in the present invention is a vector search engine based on distributed storage, which can store and retrieve billions of vector data on a single computer. Compared with traditional vector search algorithms, DiskANN has higher storage efficiency and faster search speed.

[0054] In the embodiment of the present invention, the overall process of vector data query based on DiskANN can be found in the attached Figure 1The overall process mainly includes the following computing tasks: creating a DiskANN object (abbreviated as Create), importing vector data to the DiskANN object (abbreviated as ImportData), importing vector data into the DiskANN service node (abbreviated as PushData), building a graph index for the imported vector data (abbreviated as Build), loading the graph index into the memory (abbreviated as Load), querying vector data based on the graph index (abbreviated as Search), closing the DiskANN object (abbreviated as Close) and destroying the DiskANN object (abbreviated as Destroy), etc.

[0055] Among them, creating a DiskANN object means that the DiskANN service node creates a DiskANN object according to the Create request of the client. When the DiskANN object is created, there is no data in it. It is in a dataless state and cannot provide any services to the outside. The Create request of the client is sent to the DiskANN service node through the Index service node. After the DiskANN service node completes the creation of the DiskANN object, it notifies the client through the Index service node.

[0056] In the task of importing vector data for the DiskANN object, the client sends an ImportData request containing vector data to the Index service node, and the Index service temporarily stores the vector data in the ImportData request in the database of the Index service node. The Index service node notifies the client after the data import is completed.

[0057] After the data import task, the client can send a build request to the Index service node. The Index service node executes the Push Data task according to the build request, that is, the Index service node imports the vector data in the database of the Index service node to the DiskANN service node. The DiskANN service node notifies the Index service node after the vector data is stored on the disk to form a file.

[0058] The build task refers to the process by which the DiskANN service node processes the vector data of the imported DiskANN object based on the build request sent by the Index service node, obtains the graph index of the vector data, and saves the vector data and the graph index. After the DiskANN service node completes the build task, it notifies the client through the Index service node.

[0059] The load task refers to the client sending a load request to the DiskANN service node through the Index service node, and the DiskANN service node loads at least part of the graph index of the vector data of the DiskANN object into the memory based on the load request. After the load is completed, the DiskANN service node notifies the client through the Index service node.

[0060] The search task refers to the client sending a vector data query request to the DiskANN service node through the Index service node. The DiskANN service node retrieves the vector data most similar to the vector data to be queried based on the graph index, and returns the retrieved most similar vector data to the client through the Index service node.

[0061] The Close task refers to the client sending a DiskANN object close request to the DiskANN service node through the Index service node, and the DiskANN service node closes the DiskANN object based on the close request. After the close is completed, the DiskANN service node notifies the client through the Index service node.

[0062] The Destroy task refers to the client sending a destruction request of the DiskANN object to the DiskANN service node through the Index service node, and the DiskANN service node destroys the DiskANN object based on the destruction request. After the destruction is completed, the DiskANN service node notifies the client through the Index service node.

[0063] The above process includes three execution entities: the client, the Index service node and the DiskANN service node. The purpose of using the Index service node is to make the DiskANN service node as lightweight as possible. The DiskANN service node only handles the DiskANN core business, and other businesses are outsourced to the Index service node to ensure that the overall service is stable and reliable. Before executing the above process, the Index service node and the DiskANN service node need to make a handshake connection in advance to provide services for subsequent businesses. The above-mentioned Index service node and DiskANN service node can be deployed on different physical machines or on the same physical machine, and the present invention does not limit this.

[0064] In the relevant scheme, multiple Index service nodes may be included, and multiple Index service nodes form an Index service node cluster. In the computing task of importing vector data into the DiskANN service node (referred to as Push Data), under normal circumstances, for a DiskANN object, during the period of importing vector data, only one Index service node in the Index service node cluster can send a PushData Request (import data request) of the DiskANN object to the DiskANN service node. However, in some cases, the Index service node cluster may be incorrectly scheduled. During the period of importing vector data, multiple Index service nodes may send PushData request requests of the DiskANN object to the DiskANN service node. Figure 2 As shown, when Index service node 1 imports the vector data of the DiskANN object to the DiskANN service node, if Index service node 2 also imports the vector data of the DiskANN object to the DiskANN service node, this will cause confusion in the DiskANN service node and cannot ensure the normal import of the vector data.

[0065] To solve the above problems, please refer to Figure 3 The embodiment of the present invention provides a computing task fault tolerance method for an intelligent computing center, and the computing task fault tolerance method for an intelligent computing center includes:

[0066] Step S1: after receiving the data import request for the DiskANN object sent by the Index service node, the DiskANN service node checks whether the DiskANN object currently allows the import of vector data;

[0067] In the embodiment of the present invention, the import data request of the DiskANN object needs to include the identifier (ID) of the DiskANN object, and each DiskANN object has a unique identifier. The DiskANN service node can process computing tasks of multiple DiskANN objects at the same time.

[0068] Step S2: If the DiskANN object currently allows the import of vector data, the DiskANN service node determines whether it is the first time to receive a data import request for the DiskANN object;

[0069] Step S3: If it is the first time to receive the import data request for the DiskANN object, the DiskANN service node records the first information carried in the import data request, and performs the operation of importing vector data on the DiskANN object; the first information includes: first verification information, which is obtained by the Index service node from the first service node when initiating the import of vector data for the DiskANN object, and is unique;

[0070] Step S4: If it is not the first time that an import data request for the DiskANN object is received, the DiskANN service node compares the first information carried in the currently received import data request for the DiskANN object with the first information carried in the previously received import data request for the DiskANN object to see if they are the same; if they are the same, an operation of importing vector data is performed on the DiskANN object; if they are not the same, the currently received import data request for the DiskANN object is rejected.

[0071] In an embodiment of the present invention, after receiving the import data request for the DiskANN object sent by the Index service node, the DiskANN service node checks whether the DiskANN object currently allows the import of vector data, and compares the first information carried in the currently received import data request for the DiskANN object with whether the first information carried in the previously received import data request for the DiskANN object is the same, to check whether the currently received import data request for the DiskANN object is wrong. If an error occurs, the currently received import data request for the DiskANN object can be rejected to ensure that errors occurring during the data import process do not affect the normal import process.

[0072] In some embodiments, optionally, step S1 includes:

[0073] Step S11: the DiskANN service node checks the status of the DiskANN object;

[0074] Step S12: If the state of the DiskANN object is the initialization completed state, the DiskANN service node determines that the DiskANN object currently allows the import of vector data;

[0075] Step S13: If the status of the DiskANN object is the data import completion status, the graph index building status, the graph index building status, the graph index loading into the memory status, or the graph index already loaded into the memory status, the DiskANN service node determines that the DiskANN object is currently not allowed to import vector data.

[0076] In this embodiment, by checking the status of the DiskANN object, errors such as duplicate data import can be avoided.

[0077] In some embodiments, optionally, the method further includes: if the DiskANN object currently does not allow the import of vector data, rejecting a currently received request to import data for the DiskANN object.

[0078] In an embodiment of the present invention, optionally, rejecting the currently received import data request for the DiskANN object may include at least one of the following: not processing or discarding the currently received import data request for the DiskANN object, and sending an import data response (PushData Response) to the Index service node, wherein the import data response indicates that the currently received import data request for the DiskANN object is rejected.

[0079] In an embodiment of the present invention, optionally, when the import data response indicates rejection of the currently received import data request for the DiskANN object, the rejection reason may also be carried, for example, the rejection reason is that the DiskANN object currently does not allow the import of vector data, or the DiskANN object is currently in the data import completion state and does not allow the import of vector data, or the first information carried in the currently received import data request for the DiskANN object is different from the first information carried in the previously received import data request for the DiskANN object, etc.

[0080] by Figure 4 Take this as an example to illustrate.

[0081] When the DiskANN service node receives the import data request (PushData Request) for the DiskANN object sent by the Index service node 1, it checks whether the DiskANN object currently allows the import of vector data. If the DiskANN object currently allows the import of vector data, the DiskANN service node determines whether it is the first time that the import data request for the DiskANN object is received; if it is the first time that the import data request for the DiskANN object is received, the DiskANN service node records the first information carried in the import data request and performs the operation of importing vector data on the DiskANN object.

[0082] When the DiskANN service node receives the push data request (PushData Request) for the DiskANN object sent by the Index service node 2, it checks whether the DiskANN object currently allows the import of vector data. If the DiskANN object currently allows the import of vector data, the DiskANN service node determines whether it is the first time that the push data request for the DiskANN object is received; if it is not the first time that the push data request for the DiskANN object is received, the DiskANN service node compares the first information carried in the currently received push data request for the DiskANN object with the first information carried in the previously received push data request for the DiskANN object to see if they are the same. If they are not the same, the currently received push data request for the DiskANN object is rejected. The DiskANN service node can send an import data response (PushData Response) to the Index service node 2, which indicates the rejection of the currently received import data request for the DiskANN object, and can also carry the reason for the rejection: the first verification information carried in the currently received import data request for the DiskANN object is different from the first verification information carried in the previously received import data request for the DiskANN object.

[0083] When the DiskANN service node receives the import data request for the DiskANN object sent by the Index service node 1 again, it checks whether the DiskANN object currently allows the import of vector data. If the DiskANN object currently allows the import of vector data, the DiskANN service node determines whether it is the first time to receive the import data request for the DiskANN object; if it is not the first time to receive the import data request for the DiskANN object, the DiskANN service node compares the first information carried in the currently received import data request for the DiskANN object with the first information carried in the previously received import data request for the DiskANN object. If they are the same, the operation of importing vector data is performed on the DiskANN object. It can be seen that the erroneous behavior of the Index service node 2 sending the import data request for the DiskANN object did not affect the normal import of data by the Index service node 1.

[0084] In the embodiment of the present invention, please refer to Figure 5 When the Index service node initiates the import of vector data for the DiskANN object, it needs to send a request to the first service node to obtain the first verification information (tso). The first service node allocates a unique first verification information (tso) to the Index service node and sends it to the Index service node. Each time the Index service node sends an import data request for the DiskANN object to the DiskANN service node, it needs to carry the first verification information (tso).

[0085] In the embodiment of the present invention, the first service node is a service node that allocates first verification information to the Index service node, and can be deployed on different physical machines from the Index service node and / or the DiskANN service node, or can be deployed on the same physical machine, which is not limited by the present invention.

[0086] In the embodiment of the present invention, optionally, the first verification information is timestamp information. Optionally, the timestamp information may be the time when the first service node allocates the first verification information to the Index service node.

[0087] In some embodiments, optionally, the Index service node may store multiple versions of vector data of the DiskANN object. Optionally, the first information further includes: second verification information, where the second verification information is used to indicate version information of the vector data carried in the import data request. In an import data process of a DiskANN object, only one version of vector data is allowed to be imported. If it is the first time that an import data request for the DiskANN object is received, the DiskANN service node can record the first verification information (tso) and the second verification information (version information) carried in the import data request. If it is not the first time that an import data request for the DiskANN object is received, the DiskANN service node compares the first verification information (tso) and the second verification information (version information) carried in the currently received import data request for the DiskANN object with the first verification information (tso) and the second verification information (version information) carried in the previously received import data request for the DiskANN object to see if they are the same. If they are the same, the operation of importing vector data is performed on the DiskANN object. If one of the first verification information (tso) and the second verification information (version information) is different, the currently received import data request for the DiskANN object is rejected.

[0088] In the embodiment of the present invention, double verification of the first verification information (tso) and the second verification information (version information) can effectively avoid errors in importing data.

[0089] In the embodiment of the present invention, optionally, the second verification information may also be timestamp information.

[0090] In some embodiments, optionally, the method further comprises:

[0091] Step S4: the DiskANN service node obtains the second information (has_more) carried in the import data request, where the second information is used to indicate whether there is more vector data to be sent later;

[0092] Step S5: The DiskANN service node determines the message type of the received import data request according to the second information, and the message type includes: the first import data request for the DiskANN object, the intermediate import data request, and the last import data request.

[0093] For example, when the second information (has_more) is true, it indicates that for a DiskANN object, there is still vector data to be sent later. When the second information (has_more) is false, it indicates that there is still no vector data to be sent later.

[0094] In the related scheme, three different types of messages need to be designed for the first import data request, the import data request in the middle stage, and the last import data request, such as First PushData Request, PushDataRequest, and Last PushData Request, to distinguish the import data requests in different stages. However, in the embodiment of the present invention, only one message type needs to be designed, such as PushData Request. The second information (has_more) carried in the import data request can distinguish whether it is the first import data request, the import data request in the middle stage, or the last import data request, thereby simplifying the design.

[0095] In the embodiment of the present invention, optionally, the method further includes:

[0096] Step S6: The DiskANN service node obtains its own computing power resource information, determines third information based on the computing power resource information, where the third information is used to indicate the recommended number of vector data to be sent, and sends the third information to the Index service node.

[0097] In the embodiment of the present invention, optionally, the computing power resource information may include computing power resource information such as CPU and / or memory.

[0098] Generally, the import data request sent by the Index service node carries a vector data set (vectors) and a vector data identifier set (vector_ids), and the vector data set (vectors) includes N vector data, where N is a positive integer greater than 1, for example, N is equal to 100. That is to say, for a DiskANN object, the Index service node can send multiple import data requests, and each import data request carries N vector data. N is usually a fixed value. In an embodiment of the present invention, the DiskANN service node obtains its own computing power resource information, and determines third information based on the computing power resource information, and the third information is used to indicate the recommended number of vector data to be sent. For example, the number of vector data to be sent recommended by the DiskANN service node is 50, which is used for current limiting.

[0099] In an embodiment of the present invention, the Index service node can determine whether to adopt the number of vector data lines recommended by the DiskANN service node to send based on the third information. For example, the number of vector data lines recommended by the DiskANN service node to send is 200, and the number of vector data lines normally sent by the Index service node is 100. After receiving the third information, the number of vector data lines to send can be modified to 200 or a value between 100 and 200.

[0100] That is, the DiskANN service node can give actual recommendations on the number of vector data to be sent based on its own computing resources, and dynamically adjust the conditions for sending vector data to make the use of computing resources more reasonable.

[0101] In some embodiments, optionally, the method further comprises:

[0102] Step S7: the DiskANN service node obtains the fourth information in the import data request, where the fourth information is used to indicate the number of vector data of the DiskANN that the Index service node has previously sent.

[0103] Step S8: The DiskANN service node verifies whether the number of received vector data of the DiskANN is correct according to the fourth information.

[0104] In some embodiments, optionally, the method further comprises:

[0105] Step S8: The DiskANN service node sends an import data response to the Index service node, and the import data response includes fifth information, and the fifth information is used to indicate the number of vector data of the DiskANN that the DiskANN service node has received, so that the Index service node can verify whether the number of vector data of the DiskANN that has been sent is correct.

[0106] In some embodiments, the DiskANN service node and the Index service node may perform the above-mentioned two-way verification to further improve the correctness of data import.

[0107] In an embodiment of the present invention, optionally, the method may further include: when the Index service node verifies that an error occurs in the imported data, the Index service node terminates the request to import the data. Optionally, the Index service node may send a notification of terminating the request to import the data to the DiskANN service node. The DiskANN service node terminates the import of the data according to the notification. Optionally, the DiskANN service node may delete all vector data of the DiskANN object that has been received.

[0108] In some embodiments, optionally, the method further comprises:

[0109] Step S9: When the received import data request for the DiskANN object is not the last import data request, and the interval between the current moment and the last time the import data request for the DiskANN object was received exceeds the preset sending time interval, the DiskANN service node determines that the Index service node is abnormal.

[0110] When the Index service node is abnormal, the DiskANN service node may terminate the data import operation and delete all vector data of the DiskANN object that has been received, thereby preventing the current data import from taking too long.

[0111] Please refer to Figure 6 The embodiment of the present invention further provides a computing task fault-tolerant device 10 of an intelligent computing center, comprising:

[0112] The detection module 11 is used to check whether the DiskANN object currently allows the import of vector data after receiving the data import request for the DiskANN object sent by the Index service node;

[0113] A judgment module 12 is used to judge whether it is the first time to receive a data import request for the DiskANN object if the DiskANN object currently allows the import of vector data;

[0114] The first processing module 13 is used to record the first information carried in the import data request and perform an operation of importing vector data on the DiskANN object if it is the first time to receive the import data request for the DiskANN object; the first information includes: first verification information, which is obtained by the Index service node from the first service node when initiating the import of vector data for the DiskANN object, and is unique;

[0115] The second processing module 14 is used for comparing the first information carried in the currently received import data request for the DiskANN object with the first information carried in the previously received import data request for the DiskANN object to see if they are the same; if they are the same, performing an operation of importing vector data on the DiskANN object; if they are not the same, rejecting the currently received import data request for the DiskANN object.

[0116] In an embodiment of the present invention, after receiving an import data request for a DiskANN object sent by an Index service node, it is checked whether the DiskANN object currently allows the import of vector data, and by comparing the first information carried in the currently received import data request for the DiskANN object with the first information carried in the previously received import data request for the DiskANN object to see whether it is the same, to check whether the currently received import data request for the DiskANN object is erroneous. If an error occurs, the currently received import data request for the DiskANN object can be rejected to ensure that errors occurring during the data import process do not affect the normal import process.

[0117] Optionally, the detection module 11 includes:

[0118] A checking submodule, used to check the status of the DiskANN object;

[0119] A first determination submodule is used to determine whether the DiskANN object currently allows the import of vector data if the state of the DiskANN object is an initialization completion state;

[0120] The second determination submodule is used to determine that the DiskANN object is currently not allowed to import vector data if the status of the DiskANN object is a data import completion status, a graph index building status, a graph index building completion status, a graph index loading into memory status, or a graph index already loaded into memory status.

[0121] Optionally, the first verification information is timestamp information.

[0122] Optionally, the first information further includes: second verification information, where the second verification information is used to indicate version information of the vector data carried in the import data request.

[0123] Optionally, the computing task fault-tolerant device 10 of the intelligent computing center further includes:

[0124] An acquisition module, used to acquire second information carried in the import data request, where the second information is used to indicate whether there is more vector data to be sent later;

[0125] A determination module is used to determine the message type of the received import data request according to the second information, and the message type includes: the first import data request for the DiskANN object, the intermediate import data request, and the last import data request.

[0126] Optionally, the computing task fault-tolerant device 10 of the intelligent computing center further includes:

[0127] A sending module is used to obtain its own computing power resource information, determine third information based on the computing power resource information, and the third information is used to indicate the recommended number of vector data to be sent, and send the third information to the Index service node.

[0128] Please refer to Figure 7 The embodiment of the present invention further provides a server 20, including a processor 21, a memory 22, and a computer program stored in the memory 22 and executable on the processor 21. When the computer program is executed by the processor 21, each process of the computing power task fault-tolerant method embodiment of the above-mentioned intelligent computing center is implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0129] The embodiment of the present invention further provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, each process of the above-mentioned intelligent computing center computing task fault tolerance method embodiment is implemented, and the same technical effect can be achieved. To avoid repetition, it is not repeated here. The computer-readable storage medium is, for example, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0130] The present application also provides a computer program product, including computer instructions, which, when executed by a processor, implement the above Figure 4 The various processes of the method embodiment shown can achieve the same technical effect, and will not be described again here to avoid repetition.

[0131] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element.

[0132] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, a magnetic disk, or an optical disk), and includes a number of instructions for enabling a terminal (which can be a mobile phone, a computer, a server, an air conditioner, or a network device, etc.) to execute the methods described in each embodiment of the present invention.

[0133] The embodiments of the present invention are described above in conjunction with the accompanying drawings, but the present invention is not limited to the above-mentioned specific implementation modes, which are merely illustrative rather than restrictive. Under the guidance of the present invention, ordinary technicians in this field can also make many forms without departing from the scope of protection of the present invention and the claims, all of which are within the protection of the present invention.

Claims

1. A computing task fault tolerance method for an intelligent computing center, characterized in that: include: Step S1: after receiving the data import request for the DiskANN object sent by the Index service node, the DiskANN service node checks whether the DiskANN object currently allows the import of vector data; Step S2: If the DiskANN object currently allows the import of vector data, the DiskANN service node determines whether it is the first time to receive a data import request for the DiskANN object; Step S3: if it is the first time to receive the import data request for the DiskANN object, the DiskANN service node records the first information carried in the import data request, and performs an operation of importing vector data on the DiskANN object; The first information includes: first verification information, which is obtained by the Index service node from the first service node when initiating the import of vector data to the DiskANN object, and is unique; Step S4: If it is not the first time that an import data request for the DiskANN object is received, the DiskANN service node compares the first information carried in the currently received import data request for the DiskANN object with the first information carried in the previously received import data request for the DiskANN object to see if they are the same; if they are the same, an operation of importing vector data is performed on the DiskANN object; if they are not the same, the currently received import data request for the DiskANN object is rejected.

2. The method according to claim 1, characterized in that The step S1 comprises: Step S11: the DiskANN service node checks the status of the DiskANN object; Step S12: If the state of the DiskANN object is the initialization completed state, the DiskANN service node determines that the DiskANN object currently allows the import of vector data; Step S13: If the status of the DiskANN object is the data import completion status, the graph index building status, the graph index building status, the graph index loading into the memory status, or the graph index already loaded into the memory status, the DiskANN service node determines that the DiskANN object is currently not allowed to import vector data.

3. The method according to claim 1, characterized in that The first verification information is timestamp information.

4. The method according to claim 1, characterized in that: The first information also includes: second verification information, where the second verification information is used to indicate version information of the vector data carried in the import data request.

5. The method according to claim 1, characterized in that Also includes: Step S4: the DiskANN service node obtains the second information carried in the import data request, where the second information is used to indicate whether there is more vector data to be sent later; Step S5: The DiskANN service node determines the message type of the received import data request according to the second information, and the message type includes: the first import data request for the DiskANN object, the intermediate import data request, and the last import data request.

6. The method according to claim 1, characterized in that Also includes: Step S6: The DiskANN service node obtains its own computing power resource information, determines third information based on the computing power resource information, where the third information is used to indicate the recommended number of vector data to be sent, and sends the third information to the Index service node.

7. A computing task fault-tolerant device for an intelligent computing center, characterized in that: include: A detection module, configured to check whether the DiskANN object currently allows the import of vector data after receiving a data import request for the DiskANN object sent by the Index service node; A judgment module, used for judging whether it is the first time to receive a request for importing data for the DiskANN object if the DiskANN object currently allows importing vector data; A first processing module is used for recording the first information carried in the import data request and performing an operation of importing vector data on the DiskANN object if it is the first time to receive the import data request for the DiskANN object; The first information includes: first verification information, which is obtained by the Index service node from the first service node when initiating the import of vector data to the DiskANN object, and is unique; The second processing module is used for comparing the first information carried in the currently received import data request for the DiskANN object with the first information carried in the previously received import data request for the DiskANN object to see if they are the same; if they are the same, performing an operation of importing vector data on the DiskANN object; if they are not the same, rejecting the currently received import data request for the DiskANN object.

8. A server, characterized in that: include: A processor, a memory, and a program stored in the memory and executable on the processor, wherein when the program is executed by the processor, the steps of the computing task fault tolerance method of the intelligent computing center as described in any one of claims 1 to 6 are implemented.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of the computing task fault tolerance method of the intelligent computing center according to any one of claims 1 to 6.

10. A computer program product, characterized in that It comprises computer instructions, which, when executed by a processor, implement the steps of the computing task fault tolerance method of the intelligent computing center as described in any one of claims 1 to 6.

Images