Data processing method and device
By using the global index table in the distributed architecture to determine the target processing node and output the input data, the problem of inconsistent generation results under the distributed architecture is solved, and the stable and efficient processing of query requests is achieved.
Patent Information
- Application Number
- CN202510400621.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-03-31
- Publication Date
- 2025-07-18
AI Technical Summary
Under a distributed architecture, the same query request generates inconsistent results when input at different times.
The target processing node is determined from multiple processing nodes through a global index table, and only input data is output to the target processing node, and the response data is generated using its knowledge base and big models to ensure the consistency of the generated results.
It avoids waste of computing power, ensures that the same query request input at different times obtains the same generation results, and achieves the stability and consistency of the generation results.
Smart Images
Figure CN120336448A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of computer technology, and in particular, to a data processing method and apparatus. Background Art
[0002] By performing knowledge base retrieval before the generation of a large model, the generation result of the large model can be strengthened, so that the certainty of the knowledge base can be used to reduce the "hallucination" problem of the large model.
[0003] Currently, there is a way to perform knowledge base retrieval through a distributed architecture including multiple processing nodes. In this way, when processing the same request input at different times, there is a problem of inconsistent generation results. Summary of the Invention
[0004] An embodiment of this application provides a data processing method, which is applied to a current processing node and includes: obtaining target input data; determining at least one target processing node from multiple processing nodes including itself according to the target input data and a global index table, where each processing node corresponds to a knowledge base and a large model, and the global index table represents the index information of the stored content in each knowledge base; outputting the target input data to the target processing node; and determining a data processing result corresponding to the target input data according to the response data output by the target processing node, where the response data is obtained by the target processing node according to the target input data, using its corresponding knowledge base and large model.
[0005] In some embodiments, determining at least one target processing node from multiple processing nodes including itself according to the target input data and a global index table includes: determining relevant knowledge bases related to the target input data from each knowledge base according to the target input data and the global index table; determining relevant processing nodes corresponding to the relevant knowledge bases from multiple processing nodes; and if the relevant processing nodes are in a target state, determining the relevant processing nodes as the target processing nodes.
[0006] In some embodiments, it further includes: in response to establishing a communication connection with each other processing node, sending a first global update request to each other processing node, where the first global update request includes a local index table, and the local index table represents the index information of the stored content in the knowledge base corresponding to itself; and obtaining the global index table according to the second global update request returned by the master processing node in each other processing node.
[0007] In some embodiments, it further includes: in response to obtaining a third global update request sent by the other processing node, determining a local index table corresponding to the other processing node in the third global update request; and updating the global index table according to the local index table corresponding to the other processing node.
[0008] In some embodiments, after updating the global index table according to the local index table corresponding to the other processing node, it further includes: in the case that itself belongs to the main processing node and the processing node that sends the third global update request belongs to a new processing node, returning the updated global index table to the processing node that sends the third global update request; and in response to entering the offline state, deleting the index information corresponding to the knowledge base of the other processing node from the global index table.
[0009] In some embodiments, each processing node corresponds to a rule table, and the rule table includes the device identifier and node type of each processing node. The data processing method further includes: in response to entering the online state, determining the number of processing nodes that have a communication connection with itself; if the number is zero, determining that itself belongs to the main processing node, and setting the node type of itself in the corresponding rule table of itself to the main processing node; if the number is not zero, determining that itself belongs to a new processing node, setting the node type of itself in the corresponding rule table of itself to the new processing node, and sending a new online message including its own device identifier to the other processing nodes.
[0010] In some embodiments, it further includes: if itself belongs to the main processing node and there are other processing nodes, sending heartbeat information to the other processing nodes; if itself belongs to a new processing node and receives a new online message sent by the other processing node, determining that itself belongs to a working processing node, setting the node type of itself in the corresponding rule table of itself to the working processing node, and sending an aging message including its own device identifier to the other processing nodes; if receiving an aging message sent by the other processing node, setting the node type of the processing node corresponding to the aging message in the corresponding rule table of itself to the working processing node; if receiving a new online message sent by the other processing node, setting the node type of the processing node corresponding to the new online message in the corresponding rule table of itself to the new processing node.
[0011] In some embodiments, it further includes: if the number of the other processing nodes is one, and itself belongs to a new processing node and has not received the heartbeat information within the specified duration, determining that itself belongs to the master processing node, and setting the node type of itself in the corresponding rule table to the master processing node; if the number of the other processing nodes is greater than one, and itself belongs to a working processing node and has not received the heartbeat information within the specified duration, sending a master processing node application message including its own device identifier to the new processing node, and determining that itself belongs to the master processing node and setting the node type of itself in the corresponding rule table to the master processing node when receiving the master processing node approval message returned by the new processing node; if itself belongs to a new processing node and receives a master processing node application message sent by at least one working processing node, returning a master processing node approval message to the working processing node corresponding to the first received master processing node application message; if receiving the heartbeat information sent by other processing nodes after sending the master processing node application message, setting the node type of the processing node corresponding to the heartbeat information in the corresponding rule table of itself to the master processing node; if entering the offline state, deleting the device identifier and node type of the other processing nodes in the corresponding rule table of itself, and setting the node type of itself in the corresponding rule table to the empty node.
[0012] In some embodiments, determining the data processing result corresponding to the target input data according to the response data output by the target processing node includes one of the following: when the number of the target processing nodes is multiple, fusing the response data returned by each of the target processing nodes according to the first fusion rule to obtain the fused response data, and determining the fused response data as the data processing result; when the number of the target processing nodes is multiple, obtaining the target response data output by the proxy processing node, and determining the target response data as the data processing result, where the target response data is obtained after the proxy processing node fuses the response data returned by each of the target processing nodes according to the second fusion rule.
[0013] An embodiment of the present application further provides a data processing device, which is applied to the current processing node and includes: an obtaining module, configured to obtain target input data; a first determining module, configured to determine at least one target processing node from multiple processing nodes including itself according to the target input data and the global index table, each processing node corresponding to a knowledge base and a large model, and the global index table representing the index information of the stored content in each knowledge base; an output module, configured to output the target input data to the target processing node; a second determining module, configured to determine the data processing result corresponding to the target input data according to the response data output by the target processing node, where the response data is obtained by the target processing node using the corresponding knowledge base and large model according to the target input data. Description of the Drawings
[0014] To more clearly illustrate the technical solutions of this application, the following will briefly introduce the accompanying drawings required in the description of the embodiments. Obviously, the accompanying drawings in the following description are only some embodiments described in this application. For those of ordinary skill in the art, without creative efforts, other accompanying drawings can be obtained based on these drawings.
[0015] Figure 1 The flowchart of the data processing method according to the embodiment of this application Figure 1 ;
[0016] Figure 2 The flowchart of determining the target processing node according to the embodiment of this application;
[0017] Figure 3 The flowchart of the data processing method according to the embodiment of this application Figure 2 ;
[0018] Figure 4 The flowchart of the data processing method according to the embodiment of this application Figure 3 ;
[0019] Figure 5 The schematic diagram of node type transfer in the rule table according to the embodiment of this application;
[0020] Figure 6 The timing diagram of the main processing node change according to the embodiment of this application;
[0021] Figure 7 The timing of the data processing method according to the embodiment of this application Figure 1 ;
[0022] Figure 8 The schematic diagram of the principle of the data processing method according to the embodiment of this application;
[0023] Figure 9 The timing of the data processing method according to the embodiment of this application Figure 2 ;
[0024] Figure 10 The structural block diagram of the data processing device according to the embodiment of this application. Specific embodiments
[0025] Reference is made herein to the various solutions and features of this application with reference to the accompanying drawings.
[0026] It should be understood that various modifications can be made to the embodiments applied herein. Therefore, the above description should not be regarded as a limitation, but only as an example of the embodiments. Those skilled in the art will think of other modifications within the scope and spirit of this application.
[0027] The accompanying drawings, which are included in and form a part of this specification, illustrate embodiments of the present application and, together with the general description of the present application given above and the detailed description of the embodiments given below, serve to explain the principles of the present application.
[0028] These and other features of the present application will become apparent from the following description of the preferred forms of the embodiments given by way of non-limiting examples with reference to the accompanying drawings.
[0029] It should also be understood that although the present application has been described with reference to some specific examples, those skilled in the art can surely implement many other equivalent forms of the present application.
[0030] When combined with the accompanying drawings, the above and other aspects, features and advantages of the present application will become more apparent in view of the following detailed description.
[0031] Specific embodiments of the present application will be described hereinafter with reference to the accompanying drawings; however, it should be understood that the claimed embodiments are merely examples of the present application and can be implemented in various ways. Well-known and / or repetitive functions and structures are not described in detail to avoid obscuring the present application with unnecessary or redundant details. Therefore, the specific structural and functional details claimed herein are not intended to be limiting, but are merely used as a basis for the claims and a representative basis for teaching those skilled in the art to use the present application in substantially any suitable detailed structure in a variety of ways.
[0032] This specification may use the phrases "in one embodiment", "in another embodiment", "in yet another embodiment" or "in other embodiments", each of which may refer to one or more of the same or different embodiments according to the present application.
[0033] A data processing method according to an embodiment of the present application, since only the target input data is output to the target processing node, avoids the waste of computing power caused by outputting the target input data to all processing nodes, and since the data processing result is determined by the response data output by the target processing node, makes the data processing result only related to the target processing node, realizing that for the same query request input at different times, the same generation result can be obtained, ensuring the consistency of the generation result.
[0034] This data processing method is applied to the current processing node, such as Figure 1 shown, and includes the following steps:
[0035] Step S101, obtain the target input data.
[0036] In this embodiment, the current processing node can be an electronic device, such as a mobile phone, a tablet computer, a personal computer, a server, etc. There is a communication connection between the current processing node and multiple processing nodes. The target input data can be a query request input by the user on the current processing node, specifically, it can be the query text input by the user, or the query text determined according to the voice input by the user, or the query text and the target picture input by the user.
[0037] Step S102, determine at least one target processing node from multiple processing nodes including itself according to the target input data and the global index table. Each processing node corresponds to a knowledge base and a large model. The global index table represents the index information of the stored content in each knowledge base.
[0038] In this embodiment, each processing node corresponds to a knowledge base and a large model. Each processing node can perform RAG (Retrieval-augmented Generation) through its corresponding knowledge base and large model to obtain corresponding response data. Among them, a knowledge base (Knowledge Base) is a structured, easy-to-operate, easy-to-utilize, and comprehensive and organized knowledge cluster in knowledge engineering. For the need to solve problems in a certain or certain fields, it is a collection of interconnected knowledge pieces stored, organized, managed, and used in a computer memory by adopting a certain or several knowledge representation methods. The form of the knowledge base can be a relational database or a vector database. A large model refers to a class of artificial intelligence models with a large number of parameters constructed by artificial neural networks.
[0039] A knowledge base corresponding to each processing node can be a local knowledge base deployed in the processing node or a knowledge base deployed in the cloud that the processing node can access. A large model corresponding to each processing node can be a local large model deployed in the processing node or a large model deployed in the cloud that the processing node can call. A global index table is established in advance, and this global index table represents the index information of the stored content in the knowledge bases corresponding to each processing node. After obtaining the target input data, determine at least one target processing node from multiple processing nodes including itself according to the target input data and the global index table.
[0040] In some embodiments of the present application, the global index table may include device ID, file ID, file update time, and file summary, as shown in Table 1 for example.
[0041] Table 1
[0042] Device ID File ID File Update Time File Abstract 192.168.7.7 / local / doc / a.txt 2024 / 7 / 19 16:00:02 xxxxxxx 192.168.7.8 / doc / b.pdf 2024 / 7 / 20 18:20:02 xxxxxxxxxxx 192.168.7.9 / a / b.pptx 2024 / 5 / 18 09:32:17 xxxxxxx
[0043] Step S103, output the target input data to the target processing node.
[0044] After determining the target processing node, the target input data is output to the target processing node. It can be understood that when the current processing node belongs to the target processing node, the current processing node only outputs the target input data to the processing nodes other than the current processing node in the target processing node. For example, if the current processing node is Node 1 and the target processing nodes include Node 2, Node 3, and Node 4, then Node 1 outputs the target input data to Node 2, Node 3, and Node 4. If the current processing node is Node 1 and the target processing nodes include Node 1, Node 3, and Node 4, then Node 1 outputs the target input data to Node 3 and Node 4.
[0045] Step S104: Determine the data processing result corresponding to the target input data according to the response data output by the target processing node, where the response data is obtained by the target processing node according to the target input data by using its corresponding knowledge base and large model.
[0046] In this embodiment, after receiving the target input data, the target processing node obtains the response data according to the target input data by using its corresponding knowledge base and large model. Specifically, the target processing node retrieves its corresponding knowledge base according to the target input data to obtain the corresponding target knowledge, then generates the target prompt information according to the target knowledge and the target input data, and finally guides its corresponding large model to perform reasoning through the target prompt information, and the corresponding large model outputs the response data. For example, if the target input data is Question 1 and the target knowledge is Knowledge 1 and Knowledge 2, then the target prompt information can be "Please answer Question 1 based on Knowledge 1 and Knowledge 2".
[0047] Obtain the response data output by the target processing node, and determine the data processing result according to the response data.
[0048] In some embodiments of the present application, before retrieving its corresponding knowledge base according to the target input data, the target processing node also performs pre-retrieval processing, and after retrieving its corresponding knowledge base according to the target input data, it also performs post-retrieval processing, so as to ensure the accuracy and availability of the retrieval result. The pre-retrieval processing includes, for example, query intent routing, query rewriting, query expansion, etc., and the post-retrieval processing includes, for example, rough ranking, fine ranking, summarization, fusion, etc.
[0049] The data processing method of the embodiment of the present application obtains target input data; determines at least one target processing node from multiple processing nodes including itself according to the target input data and the global index table, each processing node corresponds to a knowledge base and a large model, and the global index table represents the index information of the stored content in each knowledge base; outputs the target input data to the target processing node; determines the data processing result corresponding to the target input data according to the response data output by the target processing node, and the response data is obtained by the target processing node according to the target input data by using its corresponding knowledge base and large model. Since only the target input data is output to the target processing node, it avoids the waste of computing power caused by outputting the target input data to all processing nodes, and since the data processing result is determined by the response data output by the target processing node, the data processing result is only related to the target processing node, realizing that for the same query request input at different times, the same generation result can be obtained, ensuring the consistency of the generation result.
[0050] In some embodiments of the present application, determining at least one target processing node from multiple processing nodes including itself according to the target input data and the global index table, as Figure 2 shown, includes the following steps:
[0051] Step S1021, determine the relevant knowledge bases related to the target input data from each of the knowledge bases according to the target input data and the global index table.
[0052] In this embodiment, the target input data is compared with each index information corresponding to the global index table. If the target input data matches the index information of the knowledge base, the knowledge base is determined as the relevant knowledge base. Optionally, the keywords in the target input data can be compared with the index information of the knowledge base to determine whether the target input data matches the index information of the knowledge base. It is also possible to compare the vector data of the target input data with the vector data of the index information of the knowledge base to determine whether the target input data matches the index information of the knowledge base.
[0053] Step S1022, determine the relevant processing nodes corresponding to the relevant knowledge bases from the multiple processing nodes.
[0054] Determine the relevant processing nodes from the multiple processing nodes according to each relevant knowledge base. For example, the processing nodes include Node 1, Node 2, Node 3, and Node 4, and the relevant knowledge bases are the knowledge base corresponding to Node 3 and the knowledge base corresponding to Node 4, then Node 3 and Node 4 are the relevant processing nodes.
[0055] Step S1023, if the relevant processing nodes are in the target state, determine the relevant processing nodes as the target processing nodes.
[0056] In this embodiment, the target state may be an online state, or an online state that meets the target permissions. The relevant processing nodes in the target state are determined as target processing nodes.
[0057] By determining the processing nodes corresponding to the relevant knowledge base and in the target state as target processing nodes, the processing nodes related to the knowledge corresponding to the target input data are determined, avoiding waste of computing power and ensuring reliable processing of the target input data.
[0058] In some embodiments of the present application, as Figure 3 shown, the following steps are further included:
[0059] Step S201, in response to establishing a communication connection with each other processing node, send a first global update request to each of the other processing nodes. The first global update request includes a local index table, and the local index table represents the index information of the stored content in the knowledge base corresponding to itself.
[0060] In this embodiment, each processing node includes a local index table, which represents the index information of the stored content in the knowledge base corresponding to itself. If the knowledge base corresponding to the processing node itself is updated, the local index table corresponding to itself is updated. The current processing node can establish a communication connection with each other processing node by logging in to the domain account of the device domain where each processing node is located. After establishing the communication connection, send the first global update request including the local index table to each other processing node, so that each other processing node updates the global index table corresponding to itself according to the local index table in the first global update request. For example, each other processing node adds the device ID of the current processing node to Table 1 according to the received local index table, and updates the file ID, file update time, and file digest under the device ID.
[0061] Step S202, obtain the global index table according to the second global update request returned by the main processing node in each of the other processing nodes.
[0062] In this embodiment, a main processing node is set in each of the other processing nodes. The main processing node completes the update of the global index table according to the received local index table, and returns the second global update request to the current processing node. The current processing node obtains the global index table according to the second global update request returned by the main processing node.
[0063] By sending the first global update request to each other processing node after establishing a communication connection with each other processing node, and obtaining the global index table according to the second global update request returned by the main processing node, it is realized that each processing node updates the global index table efficiently and accurately, and the current processing node can obtain the global index table efficiently and accurately after going online.
[0064] In some embodiments of the present application, it further includes:
[0065] In response to obtaining a third global update request sent by the other processing node, determine the local index table corresponding to the other processing node in the third global update request;
[0066] Update the global index table according to the local index table corresponding to the other processing node.
[0067] In this embodiment, after the other processing node goes online, it will send a third global update request including the local index table. After the current processing node obtains the third global update request, it determines the local index table corresponding to the other processing node in the third global update request, and updates the global index table according to the local index table corresponding to the other processing node. For example, the current processing node adds the corresponding device ID of the other processing node to Table 1 according to the local index table corresponding to the other processing node, and updates the file ID, file update time, and file digest under the device ID.
[0068] Obtain the global index table corresponding to the other processing node through the third global update request returned by the other processing node, and update the global index table according to the local index table corresponding to the other processing node, so as to efficiently and accurately update the global index table.
[0069] In some embodiments of the present application, after updating the global index table according to the local index table corresponding to the other processing node, it further includes:
[0070] In the case where itself belongs to the main processing node and the processing node that sends the third global update request belongs to a new processing node, return the updated global index table to the processing node that sends the third global update request;
[0071] In response to entering the offline state, delete the index information corresponding to the knowledge base of the other processing node from the global index table.
[0072] In this embodiment, in the case where the current processing node is the main processing node and receives the third global update request from the new processing node, in order to enable the new processing node to obtain the global index table, the current processing node completes the update of the global index table according to the local index table of the new processing node in the third global update request, and returns the updated global index table to the new processing node, so that the new processing node can efficiently and accurately obtain the global index table.
[0073] The current processing node can enter the offline state by logging out of the domain account of the device domain where each processing node is located. In response to entering the offline state, the index information corresponding to the knowledge bases of other processing nodes is deleted from the global index table. At this time, the content of the global index table is the same as that of the local index table, so as to accurately adjust its own global index table after the processing node goes offline.
[0074] In some embodiments of the present application, each of the processing nodes corresponds to a rule table, and the rule table includes the device identifiers and node types of the respective processing nodes. As Figure 4 shown, the data processing method further includes the following steps:
[0075] Step S301, in response to entering the online state, determine the number of processing nodes that have a communication connection with itself.
[0076] In this embodiment, each processing node corresponds to a rule table. The rule table includes the device identifiers and node types of the respective processing nodes. The node types may include a main processing node, a working processing node, a new processing node, and an empty node. For example, the form of the rule table may be as shown in Table 2.
[0077] Table 2
[0078] Device Identification Node Type 192.168.7.7 Main Processing Node 192.168.7.8 Working Processing Node 192.168.7.9 New Processing Node 192.168.7.6 Working Processing Node
[0079] It is possible to enter the online state by logging in to the domain account of the device domain where each processing node is located. In response to entering the online state, determine the number of processing nodes that have a communication connection with itself. Execute step S302 or step S303 according to different numbers.
[0080] Step S302, if the number is zero, determine that itself belongs to the main processing node, and set the node type of itself in the corresponding rule table to the main processing node.
[0081] In this embodiment, if the number is zero, it means that there is no processing node connected to the current processing node. Determine that itself belongs to the main processing node, and set the node type of itself in the corresponding rule table to the main processing node.
[0082] Step S303, if the number is not zero, determine that itself belongs to the new processing node, set the node type of itself in the corresponding rule table to the new processing node, and send a new online message including its own device identifier to the other processing nodes.
[0083] In this embodiment, if the number is not zero, then determine that itself belongs to the new processing node, set the node type of itself in the corresponding rule table to the new processing node, and send a new online message including its own device identifier to the other processing nodes, so that each of the other processing nodes updates its corresponding rule table.
[0084] By determining the node type to which it belongs based on the number of processing nodes with a communication connection in the online state, an efficient and accurate update rule table is realized.
[0085] In some embodiments of the present application, it further includes:
[0086] If it belongs to the main processing node and there are other processing nodes, send heartbeat information to the other processing nodes;
[0087] If it belongs to the new processing node and receives the new online message sent by the other processing node, determine that it belongs to the working processing node, set the node type of itself in the corresponding rule table of itself to the working processing node, and send an aging message including its own device identifier to the other processing nodes;
[0088] If it receives the aging message sent by the other processing node, set the node type of the processing node corresponding to the aging message in the corresponding rule table of itself to the working processing node;
[0089] If it receives the new online message sent by the other processing node, set the node type of the processing node corresponding to the new online message in the corresponding rule table of itself to the new processing node.
[0090] In this embodiment, referring to Figure 5 , after an empty node goes online, it may become the main processing node or the new processing node. After a new processing node exists and a new processing node goes online, it becomes a working processing node through aging. After the main processing node, the new processing node, and the working processing node go offline, they become empty nodes. Among them, the node types of each processing node will change with the online of the new processing node. Specifically, there are the following situations:
[0091] Situation 1, it belongs to the main processing node and there are other processing nodes. In this case, send heartbeat information to the other processing nodes so that the other processing nodes can determine that the main processing node is running normally according to the heartbeat message.
[0092] Situation 2, it belongs to the new processing node and receives the new online message sent by the other processing node. In this case, since it receives the new online message sent by the other processing node, it indicates that a new processing node has gone online. Determine that itself becomes a working processing node, set the node type of itself in the corresponding rule table of itself to the working processing node, and send an aging message including its own device identifier to the other processing nodes, so that the other processing nodes can set the node type of the current processing node in each rule table to the working processing node according to the aging message.
[0093] Situation 3, it receives the aging message sent by the other processing node. In this case, it indicates that a new processing node has become a working processing node. Set the node type of the processing node corresponding to the aging message in the corresponding rule table of itself to the working processing node.
[0094] Case 4: Receive a new online message sent by another processing node. In this case, it indicates that there is a newly online processing node, and set the node type corresponding to the processing node in the corresponding rule table of itself to the new processing node.
[0095] By adjusting the node types in the rule tables in different situations, the accurate update of each rule table is realized.
[0096] In some embodiments of the present application, it further includes:
[0097] If the number of the other processing nodes is one, and itself belongs to the new processing node, and the heartbeat information is not received within the specified duration, determine that itself belongs to the main processing node, and set the node type corresponding to itself in the corresponding rule table of itself to the main processing node;
[0098] If the number of the other processing nodes is greater than one, and itself belongs to the working processing node, and the heartbeat information is not received within the specified duration, send a main processing node application message including its own device identifier to the new processing node. In the case of receiving the main processing node consent message returned by the new processing node, determine that itself belongs to the main processing node, and set the node type corresponding to itself in the corresponding rule table of itself to the main processing node;
[0099] If itself belongs to the new processing node and receives the main processing node application message sent by at least one working processing node, return the main processing node consent message to the working processing node corresponding to the first received main processing node application message;
[0100] If the heartbeat information sent by another processing node is received after sending the main processing node application message, set the node type corresponding to the processing node in the corresponding rule table of itself to the main processing node.
[0101] If entering the offline state, delete the device identifier and node type of the other processing node in the corresponding rule table of itself, and set the node type corresponding to itself in the corresponding rule table of itself to the empty node.
[0102] In this embodiment, the following situations are further included.
[0103] Case 5: The number of the other processing nodes is one, and itself belongs to the new processing node, and the heartbeat information is not received within the specified duration. In this case, since the other processing node is one and the heartbeat information is not received, it indicates that the other processing node is not the main processing node. Determine that itself belongs to the main processing node, and set the node type corresponding to itself in the corresponding rule table of itself to the main processing node.
[0104] Case 6: The number of other processing nodes is greater than one, and itself belongs to a working processing node and has not received a heartbeat message within the specified duration. In this case, it indicates that there is an abnormality in the main processing node and it cannot send a heartbeat message. It is necessary to re-determine the main processing node. Specifically, send a main processing node application message including its own device identifier to the new processing node. If a main processing node approval message is received from the new processing node, then determine that itself belongs to the main processing node and set the node type of itself in its corresponding rule table to the main processing node.
[0105] Case 7: Itself belongs to a new processing node and receives a main processing node application message sent by at least one working processing node. In this case, it is necessary to re-determine the main processing node. Return a main processing node approval message to the working processing node corresponding to the first received main processing node application message, so that the corresponding working processing node determines itself as the main processing node according to the main processing node approval message. For example, if the working processing nodes include Node 1 and Node 2, and the new processing node is Node 3, then if both Node 1 and Node 2 send main processing node application messages to Node 3, and Node 3 first receives the main processing node application message from Node 1, Node 3 will return a main processing node approval message to Node 1, so that Node 1 determines itself as the main processing node according to the main processing node approval message.
[0106] Case 8: After sending the main processing node application message, a heartbeat message sent by another processing node is received. In this case, the main processing node has been re-determined, and the node type of the processing node corresponding to the heartbeat message in its corresponding rule table is set to the main processing node. For example, if the working processing nodes include Node 1 and Node 2, and the new processing node is Node 3, then if both Node 1 and Node 2 send main processing node application messages to Node 3, and Node 3 returns a main processing node approval message to Node 1, Node 1 determines itself as the main processing node according to the main processing node approval message, and Node 1 sends heartbeat messages to Node 3 and Node 2, and Node 2 determines Node 1 as the main processing node according to the heartbeat message.
[0107] Case 9: Enter the offline state. In this case, delete the device identifiers and node types of other processing nodes in its corresponding rule table, and set the node type of itself in its corresponding rule table to the empty node. For example, if the processing nodes include Node 1, Node 2, and Node 3, if Node 1 enters the offline state, then Node 1 deletes the device identifiers and node types of Node 2 and Node 3 in its corresponding rule table, and sets the node type of Node 1 to the empty node. For example, if the processing node is in the offline state, the form of the rule table of this processing node can be as shown in Table 3.
[0108] Table 3
[0109] Device Identification Node Type 127.0.0.1 Empty Node
[0110] For example, as Figure 6 shown, each processing node includes a main processing node, a working processing node 1, a working processing node 2, and a new processing node. If heartbeat information is not received within the specified duration, the working processing node 1 and the working processing node 2 respectively send main processing node application messages to the new processing node. The new processing node first receives the main processing node application message sent by the working processing node 1, returns a main processing node approval message to the working processing node 1. The working processing node 1 sets its own node type in the corresponding rule table to the main processing node according to the main processing node approval message, and sends heartbeat information to the working processing node 2 and the new processing node. The working processing node 2 and the new processing node update the main processing node in their corresponding rule tables to the working processing node 1 according to the heartbeat information.
[0111] By adjusting the node types in the rule table according to different situations, accurate updates of each rule table are realized.
[0112] In some embodiments of the present application, determining a data processing result corresponding to the target input data according to the response data output by the target processing node includes one of the following:
[0113] When there are multiple target processing nodes, fusing the response data returned by each of the target processing nodes according to the first fusion rule to obtain fused response data, and determining the fused response data as the data processing result;
[0114] When there are multiple target processing nodes, obtaining target response data output by the proxy processing node, and determining the target response data as the data processing result, where the target response data is obtained after the proxy processing node fuses the response data returned by each of the target processing nodes according to the second fusion rule.
[0115] In this embodiment, if there are multiple target processing nodes, multiple response data can be obtained, and the data processing result can be determined by method 1 or method 2.
[0116] Method 1: The current processing node fuses the response data according to the first fusion rule to obtain fused response data, and determines the fused response data as the data processing result, thereby realizing efficient determination of the processing result. For example, as Figure 7As shown in the figure, each processing node includes Node 1, Node 2, Node 3, and Node 4. Node 1 is the current processing node. The user inputs the target input data into Node 1. Node 1 determines the relevant processing nodes according to the global index table, deletes the offline nodes from the relevant processing nodes, determines the target processing nodes (Node 3 and Node 4), and performs multicast. Node 1 outputs the target input data to Node 3 and Node 4. Then, Node 3 and Node 4 respectively retrieve their corresponding knowledge bases according to the target input data, sort the retrieval results, and generate response data according to the sorted retrieval results and the large model. Node 3 and Node 4 send the response data they generate to Node 1. Node 1 fuses each response data according to the first fusion rule and returns the obtained data processing result to the user.
[0117] For another example, as Figure 8 shown in the figure, each processing node is a PC (Personal Computer), tablet, mobile phone, and server with a communication connection. The user inputs the target input data into the PC. The PC performs pre-processing on the target input data, determines the relevant processing nodes according to the pre-processed target input data and the global index table, deletes the offline nodes from the relevant processing nodes, determines the target processing nodes (tablet and mobile phone), and performs multicast. The PC outputs the pre-processed target input data to the tablet and mobile phone. Then, the tablet retrieves its corresponding knowledge base B according to the pre-processed target input data, performs post-processing on the retrieval results, and generates response data according to the post-processed retrieval results and the large model B. At the same time, the mobile phone retrieves its corresponding knowledge base C according to the pre-processed target input data, performs post-processing on the retrieval results, and generates response data according to the post-processed retrieval results and the large model C. Then, the tablet and the mobile phone send the response data they generate to the PC. The PC fuses each response data and then returns the obtained data processing result to the user.
[0118] Method 2: A proxy processing node is set in advance. The proxy processing node fuses each response data according to the second fusion rule to obtain the target response data, and determines the target response data as the data processing result, thereby reducing the load of the current processing node and improving the consistency of the data processing result. For example, as Figure 9As shown in the figure, each processing node includes Node 1, Node 2, Node 3, Node 4, and Node 5. Node 1 is the current processing node, and Node 5 is the proxy processing node. The user inputs the target input data into Node 1. Node 1 determines the relevant processing nodes according to the global index table, deletes the offline nodes from the relevant processing nodes, determines the target processing nodes (Node 3 and Node 4), and performs multicast. Node 1 outputs the target input data to Node 3 and Node 4. Then, Node 3 and Node 4 respectively retrieve their corresponding knowledge bases according to the target input data, sort the retrieval results, and generate response data according to the sorted retrieval results and the large model. Node 3 and Node 4 send the response data they generate to Node 5. Node 5 fuses each response data according to the second fusion rule and outputs the obtained data processing result to Node 1. Node 1 returns the data processing result to the user.
[0119] The first fusion rule and the second fusion rule may include deduplication, rearrangement, and mixing of each response data. Optionally, the first fusion rule and the second fusion rule may be different fusion rules or the same fusion rule.
[0120] In some embodiments of the present application, each processing node may communicate through any one of protocols including HTTP, TCP / UDP, etc. The specific communication format is, for example, as shown in Table 4. In Table 4, the client sends a message to the sever.
[0121] Table 4
[0122]
[0123]
[0124]
[0125] An embodiment of the present application also proposes a data processing device, which is applied to the current processing node, as Figure 10 shown, including: an obtaining module, configured to obtain target input data; a first determination module, configured to determine at least one target processing node from a plurality of processing nodes including itself according to the target input data and the global index table, each processing node corresponding to a knowledge base and a large model, the global index table representing the index information of the stored content in each knowledge base; an output module, configured to output the target input data to the target processing node; a second determination module, configured to determine a data processing result corresponding to the target input data according to the response data output by the target processing node, the response data being obtained by the target processing node according to the target input data, using its corresponding knowledge base and large model.
[0126] In the data processing device according to the embodiment of the present application, since the output module only outputs the target input data to the target processing node, it avoids the waste of computing power caused by outputting the target input data to all processing nodes. Moreover, when the second determination module determines the data processing result, it is determined based on the response data output by the target processing node, so that the data processing result is only related to the target processing node. This enables the same generation result to be obtained for the same query requests input at different times, ensuring the consistency of the generation result.
[0127] In a specific application scenario, the first determination module is specifically configured to: determine relevant knowledge bases related to the target input data from each of the knowledge bases according to the target input data and the global index table; determine relevant processing nodes corresponding to the relevant knowledge bases from multiple processing nodes; if the relevant processing nodes are in a target state, determine the relevant processing nodes as the target processing nodes.
[0128] In a specific application scenario, it further includes a first update module, which is configured to: in response to establishing a communication connection with each other processing node, send a first global update request to each of the other processing nodes, where the first global update request includes a local index table, and the local index table represents index information of the stored content in the knowledge base corresponding to itself; obtain the global index table according to the second global update requests returned by the main processing nodes in each of the other processing nodes.
[0129] In a specific application scenario, it further includes a second update module, which is configured to: in response to obtaining a third global update request sent by the other processing node, determine the local index table corresponding to the other processing node in the third global update request; update the global index table according to the local index table corresponding to the other processing node.
[0130] In a specific application scenario, it further includes a third update module, which is configured to: in the case that itself belongs to the main processing node and the processing node sending the third global update request belongs to a new processing node, return the updated global index table to the processing node sending the third global update request; in response to entering the offline state, delete the index information corresponding to the knowledge bases of the other processing nodes from the global index table.
[0131] In a specific application scenario, each of the processing nodes corresponds to a rule table. The rule table includes the device identifier and node type of each of the processing nodes, and further includes a fourth update module for: in response to entering the online state, determining the number of processing nodes that have a communication connection with itself; if the number is zero, determining that itself belongs to the main processing node, and setting the node type of itself in the corresponding rule table to the main processing node; if the number is not zero, determining that itself belongs to a new processing node, setting the node type of itself in the corresponding rule table to the new processing node, and sending a new online message including its own device identifier to the other processing nodes.
[0132] In a specific application scenario, it further includes a fifth update module for: if itself belongs to the main processing node and there are other processing nodes, sending heartbeat information to the other processing nodes; if itself belongs to a new processing node and receives the new online message sent by the other processing nodes, determining that itself belongs to the working processing node, setting the node type of itself in the corresponding rule table to the working processing node, and sending an aging message including its own device identifier to the other processing nodes; if receiving the aging message sent by the other processing nodes, setting the node type of the processing node corresponding to the aging message in the corresponding rule table of itself to the working processing node; if receiving the new online message sent by the other processing nodes, setting the node type of the processing node corresponding to the new online message in the corresponding rule table of itself to the new processing node.
[0133] In a specific application scenario, it further includes a sixth update module for: if the number of the other processing nodes is one, and itself belongs to a new processing node, and the heartbeat information has not been received within the specified duration, determining that itself belongs to the main processing node, and setting the node type of itself in the corresponding rule table to the main processing node; if the number of the other processing nodes is greater than one, and itself belongs to the working processing node, and the heartbeat information has not been received within the specified duration, sending a main processing node application message including its own device identifier to the new processing node, and in the case of receiving the main processing node approval message returned by the new processing node, determining that itself belongs to the main processing node, and setting the node type of itself in the corresponding rule table to the main processing node; if itself belongs to a new processing node and receives the main processing node application message sent by at least one working processing node, returning the main processing node approval message to the working processing node corresponding to the first received main processing node application message; if receiving the heartbeat information sent by other processing nodes after sending the main processing node application message, setting the node type of the processing node corresponding to the heartbeat information in the corresponding rule table of itself to the main processing node; if entering the offline state, deleting the device identifier and node type of the other processing nodes in the corresponding rule table of itself, and setting the node type of itself in the corresponding rule table to the empty node.
[0134] In a specific application scenario, the second determination module is specifically configured to perform one of the following: when there are multiple target processing nodes, fuse the response data returned by each of the target processing nodes according to a first fusion rule to obtain fused response data, and determine the fused response data as the data processing result; when there are multiple target processing nodes, obtain the target response data output by a proxy processing node, and determine the target response data as the data processing result, where the target response data is obtained by the proxy processing node fusing the response data returned by each of the target processing nodes according to a second fusion rule.
[0135] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the processes or functions described in the embodiments of the present application are generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center in a wired manner (such as coaxial cable, optical fiber, digital subscriber line) or a wireless manner (such as infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that can be accessed by a computer, or a data storage device such as a server or data center that includes one or more available media integrated. The available medium can be a magnetic medium (such as a floppy disk, hard disk, magnetic tape), an optical medium (such as a DVD), or a semiconductor medium (such as a solid-state drive), etc.
[0136] The above embodiments are only exemplary embodiments of the present application and are not used to limit the present application. The protection scope of the present application is defined by the claims. Those skilled in the art can make various modifications or equivalent replacements within the essence and protection scope of the present application, and such modifications or equivalent replacements should also be regarded as falling within the protection scope of the present application.
Claims
1. A data processing method, applied to a current processing node, includes: Obtaining target input data; Determining at least one target processing node from multiple processing nodes including itself according to the target input data and a global index table, where each processing node corresponds to a knowledge base and a large model, and the global index table represents index information of the stored content in each knowledge base; Outputting the target input data to the target processing node; Determining a data processing result corresponding to the target input data according to the response data output by the target processing node, where the response data is obtained by the target processing node according to the target input data by using its corresponding knowledge base and large model.
2. The data processing method according to claim 1, determining at least one target processing node from multiple processing nodes including itself according to the target input data and the global index table, includes: Determining relevant knowledge bases related to the target input data from each knowledge base according to the target input data and the global index table; Determining relevant processing nodes corresponding to the relevant knowledge bases from multiple processing nodes; If the relevant processing node is in a target state, determining the relevant processing node as the target processing node.
3. The data processing method according to claim 1, further includes: In response to establishing a communication connection with each other processing node, sending a first global update request to each other processing node, where the first global update request includes a local index table, and the local index table represents index information of the stored content in the knowledge base corresponding to itself; Obtaining the global index table according to the second global update request returned by the main processing node in each other processing node.
4. The data processing method according to claim 3, further includes: In response to obtaining a third global update request sent by the other processing node, determining the local index table corresponding to the other processing node in the third global update request; Updating the global index table according to the local index table corresponding to the other processing node.
5. The data processing method according to claim 4, after updating the global index table according to the local index table corresponding to the other processing node, further includes: In the case that itself belongs to the main processing node and the processing node sending the third global update request belongs to a new processing node, returning the updated global index table to the processing node sending the third global update request; In response to entering the offline state, deleting the index information corresponding to the knowledge base of the other processing node from the global index table.
6. The data processing method according to claim 1, each processing node corresponds to a rule table, and the rule table includes the device identifier and node type of each processing node. The data processing method further includes: In response to entering the online state, determining the number of processing nodes having a communication connection with itself; If the number is zero, determining that itself belongs to the main processing node, and setting the node type of itself in its corresponding rule table as the main processing node; If the quantity is not zero, determine that itself belongs to a new processing node, set the node type of itself in its corresponding rule table to the new processing node, and send a new online message including its own device identifier to the other processing nodes.
7. The data processing method according to claim 6, further comprising: If itself belongs to the main processing node and there are other processing nodes, send heartbeat information to the other processing nodes; If itself belongs to the new processing node and receives the new online message sent by the other processing nodes, determine that itself belongs to the working processing node, set the node type of itself in its corresponding rule table to the working processing node, and send an aging message including its own device identifier to the other processing nodes; If receiving the aging message sent by the other processing nodes, set the node type of the processing node corresponding to the aging message in its corresponding rule table to the working processing node; If receiving the new online message sent by the other processing nodes, set the node type of the processing node corresponding to the new online message in its corresponding rule table to the new processing node.
8. The data processing method according to claim 7, further comprising: If the number of the other processing nodes is one, and itself belongs to the new processing node, and the heartbeat information is not received within the specified duration, determine that itself belongs to the main processing node, and set the node type of itself in its corresponding rule table to the main processing node; If the number of the other processing nodes is greater than one, and itself belongs to the working processing node, and the heartbeat information is not received within the specified duration, send a main processing node application message including its own device identifier to the new processing node. In the case of receiving the main processing node consent message returned by the new processing node, determine that itself belongs to the main processing node, and set the node type of itself in its corresponding rule table to the main processing node; If itself belongs to the new processing node and receives the main processing node application message sent by at least one working processing node, return the main processing node consent message to the working processing node corresponding to the first received main processing node application message; If receiving the heartbeat information sent by the other processing nodes after sending the main processing node application message, set the node type of the processing node corresponding to the heartbeat information in its corresponding rule table to the main processing node; If entering the offline state, delete the device identifier and node type of the other processing nodes in its corresponding rule table, and set the node type of itself in its corresponding rule table to the empty node.
9. The data processing method according to claim 1, determining the data processing result corresponding to the target input data according to the response data output by the target processing node, including one of the following: In the case where the target processing nodes are multiple, fuse the response data returned by each of the target processing nodes according to the first fusion rule to obtain the fused response data, and determine the fused response data as the data processing result; In the case where there are multiple target processing nodes, obtain the target response data output by the proxy processing node, and determine the target response data as the data processing result. The target response data is obtained after the proxy processing node fuses each response data returned by each target processing node according to the second fusion rule.
10. A data processing device, applied to a current processing node, comprising: An obtaining module, configured to obtain target input data; A first determining module, configured to determine at least one target processing node from multiple processing nodes including itself according to the target input data and a global index table. Each processing node corresponds to a knowledge base and a large model, and the global index table represents index information of the stored content in each knowledge base; An output module, configured to output the target input data to the target processing node; A second determining module, configured to determine a data processing result corresponding to the target input data according to the response data output by the target processing node, where the response data is obtained by the target processing node according to the target input data by using its corresponding knowledge base and large model.