A data processing system and method

Abstract

The embodiment of the application discloses a data processing system and method, the system comprises: a producer module, a document queue module and a consumer module; wherein the producer module is used for caching path data to the document queue module; the document queue module is used for sending the path data to the consumer module when receiving request information sent by the consumer module; the consumer module is used for acquiring a document corresponding to the path data according to the path data; and the document is classified through a preset keyword.

Description

Technical Field

[0001] This application relates to the field of computer technology, and more particularly to a data processing system and method. Background Technology

[0002] Many companies possess a vast number of technical documents and manuals, and the rules for storing these documents vary across departments. This makes it difficult for staff to quickly find relevant information from a massive amount of data. To prevent this, it is necessary to categorize these massive amounts of corporate technical documents, allowing staff to easily locate the documents they need according to the categories.

[0003] Therefore, how to achieve rapid document classification is a technical problem that urgently needs to be solved. Summary of the Invention

[0004] In view of this, embodiments of this application provide a data processing system and method to solve at least one problem existing in the prior art.

[0005] To achieve the above objectives, the technical solution of this application embodiment is implemented as follows:

[0006] In a first aspect, embodiments of this application provide a data processing system, the system comprising: a producer module, a document queue module, and a consumer module; wherein...

[0007] The producer module is used to cache path data to the document queue module;

[0008] The document queue module is used to send the path data to the consumer module when it receives a request message from the consumer module.

[0009] The consumer module is used to obtain documents corresponding to the path data based on the path data; and to classify the documents using preset keywords.

[0010] In one optional implementation, the system further includes: a storage module; the storage module is used to store documents;

[0011] The producer module is also used to obtain path data corresponding to the document from the storage module.

[0012] In one alternative implementation, the system further includes: a classification database;

[0013] The consumer module is also used to store the categorized documents into a categorization database and send categorization completion information to the document queue module.

[0014] In one optional implementation, the document queue module is further configured to delete the path data corresponding to the document based on the classification completion information.

[0015] In one optional implementation, the producer module includes a producer master node and multiple producer slave nodes;

[0016] The producer slave node is used to send heartbeat information to the producer master node at a first preset time interval, wherein the producer master node is used to return heartbeat feedback including the heartbeat information when it receives the heartbeat information;

[0017] The producer module is used to assign unfinished tasks on the producer master node to any of the multiple producer slave nodes when the heartbeat feedback timeout is detected.

[0018] In one optional implementation, the document queue module includes a document queue master node and multiple document queue slave nodes;

[0019] The document queue slave node is used to send heartbeat information to the document queue master node at a second preset time interval, wherein the document queue master node is used to return heartbeat feedback including the heartbeat information when it receives the heartbeat information;

[0020] The document queue module is used to assign unfinished tasks on the document queue master node to any one of the multiple document queue slave nodes when the heartbeat feedback timeout is detected.

[0021] In one optional implementation, the document queue module includes a document queue master node and multiple document queue slave nodes;

[0022] The document queue master node is used to synchronize the path data to the multiple document queue slave nodes when it receives the path data cached by the producer module.

[0023] In one optional implementation, the document queue module includes a document queue master node and multiple document queue slave nodes;

[0024] The document queue master node is used to synchronize the request information to the multiple document queue slave nodes when it receives the request information for obtaining path data sent by the consumer module.

[0025] In one optional implementation, the consumer module includes multiple consumer nodes; the multiple consumer nodes are used to process data from different paths sent by the document queue module in parallel.

[0026] In one optional implementation, the path data includes a document path and a tag corresponding to the document path; the tag is used to characterize the processing status of the document path.

[0027] In one optional implementation, the tag includes a first state value and a second state value;

[0028] The document queue module is also used to change the tag corresponding to the document path from a first state value to a second state value when sending the path data to the consumer module.

[0029] In an optional implementation, the document queue module is further configured to change the tag from the second state value to the first state value when it is detected that the tag has been maintained in the second state value for more than a third preset duration.

[0030] In one optional implementation, the consumer module is specifically used to: obtain the document corresponding to the path data based on the path data, extract the content of the document, compare the content of the document with preset keywords; if they match, classify the document into the category corresponding to the preset keywords.

[0031] Secondly, embodiments of this application provide a data processing method applied to a producer module, the method comprising:

[0032] Cache path data in the document queue module;

[0033] The document queue module is used to send the path data to the consumer module when it receives a request information from the consumer module; the consumer module is used to obtain the document corresponding to the path data according to the path data; and classify the document by preset keywords.

[0034] Thirdly, embodiments of this application provide a data processing method applied to a consumer module, the method comprising:

[0035] The document corresponding to the path data is obtained based on the path data sent by the document queue module; the document is then classified using preset keywords.

[0036] The document queue module is used to send the path data to the consumer module when it receives a request information sent by the consumer module; wherein the path data in the document queue module is stored by the producer module.

[0037] Fourthly, embodiments of this application provide a data processing method applied to a document queue module, the method comprising:

[0038] Upon receiving a request from the consumer module, path data is sent to the consumer module.

[0039] The path data in the document queue module is stored by the producer module; the consumer module is used to retrieve the document corresponding to the path data; and to classify the document by preset keywords.

[0040] The technical solution provided in this application offers a data processing system comprising a producer module, a document queue module, and a consumer module. The producer module caches path data in the document queue module, and the consumer module retrieves documents based on the path data and then categorizes them. This application implements a distributed document processing architecture using a producer-consumer pattern. By simply scaling up the number of consumer modules in parallel, the speed and efficiency of document categorization can be significantly improved. Attached Figure Description

[0041] In the accompanying drawings, unless otherwise specified, the same reference numerals throughout the various drawings denote the same or similar parts or elements. These drawings are not necessarily drawn to scale. It should be understood that these drawings depict only some embodiments disclosed in this application and should not be construed as limiting the scope of this application.

[0042] Figure 1 This is a schematic diagram of the structure of a data processing system provided in an embodiment of this application;

[0043] Figure 2 A schematic diagram illustrating data storage within the document queue module provided in this application embodiment;

[0044] Figure 3 A schematic diagram of the structure of a consumer module provided in an embodiment of this application;

[0045] Figure 4 This is a schematic diagram of the structure of a document queue module provided in an embodiment of this application;

[0046] Figure 5 This is a schematic diagram of the structure of a producer module provided in an embodiment of this application;

[0047] Figure 6 A schematic diagram of the implementation flow of a data processing method provided in this application embodiment. Figure 1 ;

[0048] Figure 7 A schematic diagram of the implementation flow of a data processing method provided in this application embodiment. Figure 2 ;

[0049] Figure 8A schematic diagram of the implementation flow of a data processing method provided in this application embodiment. Figure 3 . Detailed Implementation

[0050] In the following description, numerous specific details are set forth in order to provide a more thorough understanding of this application. However, it will be apparent to those skilled in the art that this application can be practiced without one or more of these details. In other instances, to avoid confusion with this application, some technical features well-known in the art have not been described; that is, not all features of actual embodiments are described herein, nor are well-known functions and structures described in detail.

[0051] Furthermore, the accompanying drawings are merely illustrative of this application and are not necessarily drawn to scale. The same reference numerals in the drawings denote the same or similar parts, and therefore repeated descriptions of them will be omitted. Some block diagrams shown in the drawings are functional entities and do not necessarily correspond to physically or logically independent entities. These functional entities can be implemented in software, in one or more hardware modules or integrated circuits, or in different network and / or processor devices and / or microcontroller devices.

[0052] The flowchart shown in the attached diagram is merely an illustrative example and does not necessarily include all steps. For example, some steps may be broken down, while others may be combined or partially combined; therefore, the actual execution order may change depending on the specific circumstances.

[0053] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the scope of this application. When used herein, the singular forms “a,” “an,” and “the” are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the terms “comprising” and / or “including,” when used in this specification, identify the presence of the stated features, integers, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups. When used herein, the term “and / or” includes any and all combinations of the associated listed items.

[0054] This application provides a data processing system. Figure 1 This is a schematic diagram of the structure of a data processing system provided in an embodiment of this application, such as... Figure 1 As shown, the system includes: a producer module 100, a document queue module 200, and a consumer module 300; wherein,

[0055] The producer module 100 is used to cache path data to the document queue module 200;

[0056] The document queue module 200 is used to send the path data to the consumer module 300 when it receives the request information sent by the consumer module 300;

[0057] The consumer module 300 is used to obtain documents corresponding to the path data based on the path data; and to classify the documents using preset keywords.

[0058] The technical solution provided in this application implements a distributed document processing architecture through a producer-consumer model. By simply scaling up the number of consumer modules in parallel, the speed and efficiency of document classification can be greatly improved. This allows for quick retrieval of required documents based on the classified documents, solving the problem in existing technologies of being unable to quickly find effective information from massive amounts of documents. Furthermore, the technical solution provided in this application can be widely applied to applications with massive amounts of technical documents and manuals, such as semiconductor companies, communication companies, and computer companies. Taking semiconductor companies as an example, the massive technical documents of semiconductor companies can be effectively classified using preset keywords. Preset keywords can be used to detect whether any document contains a preset keyword, thereby classifying documents based on that preset keyword. This unifies the classification rules and ensures the rationality of the classification.

[0059] In the producer-consumer pattern, the producer module caches path data in a buffer, while the consumer module retrieves path data from the buffer. That is, in the producer-consumer pattern, both the producer and consumer modules access the same buffer. When the producer and consumer modules perform data storage or retrieval operations on the buffer, the results depend on the relative time of these two processes during execution, i.e., on the order of instruction execution between the two modules. During this process, data access anomalies can easily occur due to contention for resources in the buffer between the producer and consumer modules. Therefore, this embodiment uses a document queue module with a First-In, First-Out (FIFO) data structure as the buffer. All insertion operations are performed at one end of the queue, while all retrieval operations (deletion operations) are performed at the other end. Thus, even if the producer and consumer modules process resources in the document queue module (buffer), resource contention anomalies will not occur.

[0060] Figure 2 This is a schematic diagram of data storage within the document queue module provided in an embodiment of this application, as shown below. Figure 2As shown in this embodiment, the path data includes a document path and a tag corresponding to the document path; the tag is used to characterize the processing status of the document path. Here, the document path is the storage path of the document, through which the corresponding document can be found. For example, the document path is D: / Document / file.txt.

[0061] In this embodiment, the tag includes a first state value and a second state value. The first state value indicates that the document path in the path data is in an unprocessed state; the second state value indicates that the document path in the path data is in a processed state. In practical applications, the first state value can be 0, and the second state value can be 1. It should be noted that when the producer module caches the path data to the document queue module, all tags in the path data are the first state value.

[0062] In the embodiments of this application, there is a one-to-one correspondence between document and path data, and each path data includes a document path and a tag.

[0063] In some embodiments, such as Figure 1 As shown, the system further includes: a storage module 400; the storage module 400 is used to store documents; the producer module 100 is also used to obtain path data corresponding to the documents from the storage module 400. In practical applications, the storage module can be a document disk, which includes multiple folders, each folder storing multiple documents. The producer module 100 obtains the path data corresponding to the documents from the storage module 400. Here, each document corresponds to one set of path data.

[0064] In other embodiments, the producer module 100 obtains the document path corresponding to the document from the storage module 400, and caches the document path and the tag corresponding to the document path as path data in the document queue module.

[0065] In the embodiments of this application, such as Figure 1 As shown, the system also includes a classification database 500; the consumer module 300 is further configured to store the classified documents into the classification database 500 and send classification completion information to the document queue module 200. Thus, when staff search for technical information, they can quickly find the required documents from the classification database according to category or keywords.

[0066] In this embodiment, the document queue module 200 is further configured to delete the path data corresponding to the document based on the classification completion information. In this embodiment, after completing the classification of the document corresponding to the path data, the consumer module 300 sends classification completion information to the document queue module 200. Upon receiving the classification completion information, the document queue module 200 deletes the path data corresponding to the classified document.

[0067] Figure 3 This application provides a schematic diagram of the structure of a consumer module, as shown in the embodiment. Figure 3 As shown in this embodiment, the consumer module 300 includes multiple consumer nodes 310; the multiple consumer nodes 310 are used to process data from different paths sent by the document queue module 200 in parallel. In other words, the multiple consumer nodes 310 in the consumer module 300 can all process data from different paths in parallel, thereby greatly improving the speed and efficiency of document classification.

[0068] Here, we take a consumer node 310 in the consumer module 300 as an example. Consumer node 310 sends a request to obtain path data to the document queue module 200. After receiving the request, the document queue module 200 sends the path data to the consumer node 310. The consumer node 310 then retrieves the document corresponding to the path data and categorizes the document using preset keywords. It should be noted that when the document queue module receives multiple request messages from multiple consumer nodes, the document queue module sends the path data to the consumer nodes in a first-in, first-out manner according to the time order in which the request messages are received.

[0069] In this embodiment, the document queue module 200 is further configured to change the tag corresponding to the document path from a first state value to a second state value when sending the path data to the consumer module 300. At this time, the document path in the path data is in a processing state, meaning that a consumer node in the consumer module is processing the document path in the path data.

[0070] In this embodiment, the document queue module 200 is further configured to change the tag from the second state value to the first state value when it detects that the tag's duration in the second state value exceeds a third preset duration. If a consumer node in the consumer module fails during document path processing, the tag corresponding to the document path remains in the second state value in the document queue module. If the tag's duration in the second state value exceeds the third preset duration, the document queue module will change the tag from the second state value to the first state value. At this time, the document path in the path data is in an unprocessed state. In subsequent processes, the document queue module can send the path data to other consumer nodes in the consumer module for reprocessing. The above process can be called a retry redundancy mechanism. Through this retry redundancy mechanism, the failure of a single consumer node does not affect the normal operation of other consumer nodes, and the unprocessed path data on that consumer node can be switched to other consumer nodes for reprocessing, thus having high reliability.

[0071] Here, the third preset duration can be determined based on the processing time of the path data by the consumer nodes in the consumer module. In practical applications, the maximum historical processing time of the path data by all consumer nodes in the consumer module can be set as the third preset duration. In other words, if the tag maintains the second state value for a longer period than the third preset duration, it indicates that the consumer node has not completed the processing of the path data during this period. In this case, it can be determined that the consumer node may be malfunctioning, and the tag is changed from the second state value to the first state value. Upon receiving a request from another consumer node, the path data is sent to that other consumer node for reprocessing.

[0072] Figure 4 This is a schematic diagram of the structure of a document queue module provided in an embodiment of this application, as shown below. Figure 4As shown in this embodiment, the document queue module 200 includes a document queue master node 210 and multiple document queue slave nodes 220. The document queue master node 210, upon receiving path data cached by the producer module 100, synchronizes the path data to the multiple document queue slave nodes 220. Therefore, when the document queue master node fails and cannot function properly, the document queue slave nodes can quickly take over the work of the document queue master node based on the synchronized path data. Furthermore, even if a document queue slave node also fails, the tasks on the failed document queue slave node can be distributed to other document queue slave nodes, ensuring that the failure of a single document queue master node or document queue slave node does not affect the normal operation of other document queue slave nodes, thus exhibiting high reliability.

[0073] In this embodiment, the document queue master node 210, upon receiving a request for path data from the consumer module 300, synchronizes the request information to the multiple document queue slave nodes 220. Thus, when the document queue master node fails and cannot function properly, the document queue slave nodes can quickly take over the work of the master node based on the synchronized request information. Furthermore, even if a document queue slave node also fails, the tasks on the failed slave node can be distributed to other slave nodes, ensuring that the failure of a single master or slave node does not affect the normal operation of other slave nodes, thus providing high reliability.

[0074] Figure 5 This is a schematic diagram of the structure of a producer module provided in an embodiment of this application, such as... Figure 5As shown in this embodiment, the producer module 100 includes a producer master node 110 and multiple producer slave nodes 120. The producer slave nodes 120 are used to send heartbeat information to the producer master node 110 at a first preset time interval. The producer master node 110 is used to return heartbeat feedback including the heartbeat information when it receives the heartbeat information. The producer module 100 is used to allocate unfinished tasks on the producer master node 110 to any of the multiple producer slave nodes 120 when it detects that the heartbeat feedback has timed out. Here, each producer slave node sends a corresponding heartbeat message to the producer master node at a first preset time interval. Correspondingly, the producer master node returns a heartbeat feedback upon receiving the heartbeat message. While maintaining the heartbeat between the producer master and slave nodes at this first preset time interval, if a producer slave node detects a heartbeat feedback timeout, it assigns the unfinished tasks on the producer master node to any of the plurality of producer slave nodes. In some embodiments, in subsequent processes, producer slave nodes that have not been assigned tasks still send corresponding heartbeat messages to producer slave nodes that have been assigned tasks at the first preset time interval. Correspondingly, producer slave nodes that have been assigned tasks return a heartbeat feedback upon receiving the heartbeat message. If a producer slave node that has not been assigned tasks detects a heartbeat feedback timeout, it assigns the unfinished tasks on the producer slave node to which the timed-out heartbeat feedback belongs to other producer slave nodes. In this embodiment, the producer slave node monitors the producer master node by maintaining a heartbeat with it. Therefore, when the heartbeat feedback of the producer master node times out, the unfinished tasks on the producer master node are assigned to any of the multiple producer slave nodes. Based on the above-mentioned heartbeat redundancy mechanism, timely detection and switching of faulty producer master nodes are achieved, ensuring the normal execution of tasks.

[0075] In this embodiment, the document queue slave node 220 is used to send heartbeat information to the document queue master node 210 at a second preset time interval. The document queue master node 210, upon receiving the heartbeat information, returns heartbeat feedback including the heartbeat information. The document queue module 200, upon detecting a timeout in the heartbeat feedback, assigns unfinished tasks on the document queue master node 210 to any one of the multiple document queue slave nodes 220. Here, the document queue slave nodes send corresponding heartbeat information to the document queue master node at a first preset time interval. Correspondingly, the document queue master node returns heartbeat feedback upon receiving the heartbeat information. While maintaining the heartbeat between the document queue master and slave nodes at the first preset time interval, if a document queue slave node detects a timeout in the heartbeat feedback, it assigns the unfinished tasks on the document queue master node to any one of the multiple document queue slave nodes. In some embodiments, during subsequent processes, document queue slave nodes that have not been assigned tasks still send corresponding heartbeat information to document queue slave nodes that have been assigned tasks at first preset time intervals. Correspondingly, document queue slave nodes that have been assigned tasks return heartbeat feedback upon receiving heartbeat information. If a document queue slave node that has not been assigned tasks detects a heartbeat feedback timeout, it assigns the unfinished tasks on the document queue slave node to which the timed-out heartbeat feedback belongs to other document queue slave nodes. In this embodiment, document queue slave nodes monitor the document queue master node by maintaining a heartbeat with it. Therefore, when a heartbeat feedback timeout is detected on the document queue master node, the unfinished tasks on that document queue master node are assigned to any of the multiple document queue slave nodes. Based on the above-mentioned heartbeat redundancy mechanism, timely detection and switching of faulty document queue master nodes are achieved, ensuring the normal execution of tasks.

[0076] In this embodiment, the consumer module 300 is specifically used to: obtain the document corresponding to the path data based on the path data, extract the content of the document, and compare the content of the document with preset keywords; if they match, the document is categorized into the category corresponding to the preset keywords. It should be noted that each category may correspond to multiple preset keywords.

[0077] The technical solution provided in this application offers a data processing system comprising a producer module, a document queue module, and a consumer module. The producer module caches path data in the document queue module, and the consumer module retrieves documents based on the path data and then categorizes them. This application implements a distributed document processing architecture using a producer-consumer pattern. By simply scaling up the number of consumer modules in parallel, the speed and efficiency of document categorization can be significantly improved.

[0078] This application also proposes a data processing method, such as... Figure 6 The diagram shown illustrates the implementation flow of the data processing method in this application embodiment. Figure 1 Applied to the producer module, such as Figure 6 As shown, the method includes the following steps:

[0079] Step 601: Cache the path data in the document queue module;

[0080] Step 602: The document queue module is used to send the path data to the consumer module when it receives the request information sent by the consumer module; the consumer module is used to obtain the document corresponding to the path data according to the path data; and classify the document by preset keywords.

[0081] This application also proposes a data processing method, such as... Figure 7 The diagram shown illustrates the implementation flow of the data processing method in this application embodiment. Figure 2 Applied to consumer modules, such as Figure 7 As shown, the method includes the following steps:

[0082] Step 701: Obtain the document corresponding to the path data sent by the document queue module; classify the document by preset keywords;

[0083] Step 702: The document queue module is used to send the path data to the consumer module when it receives the request information sent by the consumer module; wherein the path data in the document queue module is stored by the producer module.

[0084] This application also proposes a data processing method, such as... Figure 8 The diagram shown illustrates the implementation flow of the data processing method in this application embodiment. Figure 3 Applied to the document queue module, such as Figure 8 As shown, the method includes the following steps:

[0085] Step 801: Upon receiving the request information sent by the consumer module, the path data is sent to the consumer module;

[0086] Step 802: The path data in the document queue module is stored by the producer module; the consumer module is used to obtain the document corresponding to the path data according to the path data; and classify the document by preset keywords.

[0087] It should be noted that the description of the above data processing method is similar to the description of the above data processing system embodiments, and has similar beneficial effects, therefore it will not be repeated. For technical details not disclosed in the data processing methods of this application embodiments, please refer to the description of the data processing system in the embodiments of this application for understanding.

[0088] The methods disclosed in the several method embodiments provided in this application can be arbitrarily combined without conflict to obtain new method embodiments.

[0089] The features disclosed in the several device embodiments provided in this application can be arbitrarily combined without conflict to obtain new device embodiments.

[0090] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A data processing system, characterized in that, The system includes: a producer module, a document queue module, and a consumer module; wherein... The producer module is used to cache path data to the document queue module; The document queue module is used to send the path data to the consumer module when it receives a request message from the consumer module. The consumer module is used to obtain documents corresponding to the path data based on the path data; and to classify the documents using preset keywords. The producer module includes one producer master node and multiple producer slave nodes; The producer slave node is used to send heartbeat information to the producer master node at a first preset time interval, wherein the producer master node is used to return heartbeat feedback including the heartbeat information when it receives the heartbeat information; The producer module is used to assign unfinished tasks on the producer master node to any of the multiple producer slave nodes when the heartbeat feedback timeout is detected. The document queue module includes a document queue master node and multiple document queue slave nodes; The document queue slave node is used to send heartbeat information to the document queue master node at a second preset time interval, wherein the document queue master node is used to return heartbeat feedback including the heartbeat information when it receives the heartbeat information; The document queue module is used to assign unfinished tasks on the document queue master node to any one of the multiple document queue slave nodes when the heartbeat feedback timeout is detected.

2. The data processing system according to claim 1, characterized in that, The system further includes: a storage module; the storage module is used to store documents; The producer module is also used to obtain path data corresponding to the document from the storage module.

3. The data processing system according to claim 1, characterized in that, The system also includes: a classification database; The consumer module is also used to store the categorized documents into the categorization database and send categorization completion information to the document queue module.

4. The data processing system according to claim 3, characterized in that, The document queue module is also used to delete the path data corresponding to the document based on the classification completion information.

5. The data processing system according to claim 1, characterized in that, The document queue module includes a document queue master node and multiple document queue slave nodes; The document queue master node is used to synchronize the path data to the multiple document queue slave nodes when it receives the path data cached by the producer module.

6. The data processing system according to claim 1, characterized in that, The document queue module includes a document queue master node and multiple document queue slave nodes; The document queue master node is used to synchronize the request information to the multiple document queue slave nodes when it receives the request information for obtaining path data sent by the consumer module.

7. The data processing system according to claim 1, characterized in that, The consumer module includes multiple consumer nodes; the multiple consumer nodes are used to process data from different paths sent by the document queue module in parallel.

8. The data processing system according to claim 1, characterized in that, The path data includes a document path and a tag corresponding to the document path; the tag is used to characterize the processing status of the document path.

9. The data processing system according to claim 8, characterized in that, The label includes a first state value and a second state value; The document queue module is also used to change the tag corresponding to the document path from a first state value to a second state value when sending the path data to the consumer module.

10. The data processing system according to claim 9, characterized in that, The document queue module is also used to change the tag from the second state value to the first state value when it is detected that the tag has been maintained in the second state value for more than a third preset duration.

11. The data processing system according to claim 1, characterized in that, The consumer module is specifically used to: obtain the document corresponding to the path data based on the path data, extract the content of the document, compare the content of the document with preset keywords; if they match, classify the document into the category corresponding to the preset keywords.

12. A data processing method, comprising the data processing system according to any one of claims 1-11, characterized in that, Applied to the producer module, the method includes: Cache path data in the document queue module; The document queue module is used to send the path data to the consumer module when it receives a request information from the consumer module; the consumer module is used to obtain the document corresponding to the path data according to the path data; and classify the document by preset keywords.

13. A data processing method, comprising the data processing system according to any one of claims 1-11, characterized in that, Applied to the consumer module, the method includes: The document corresponding to the path data is obtained based on the path data sent by the document queue module; the document is then classified using preset keywords. The document queue module is used to send the path data to the consumer module when it receives a request information sent by the consumer module; wherein the path data in the document queue module is stored by the producer module.

14. A data processing method, comprising the data processing system according to any one of claims 1-11, characterized in that, Applied to the document queue module, the method includes: Upon receiving a request from the consumer module, path data is sent to the consumer module. The path data in the document queue module is stored by the producer module; the consumer module is used to retrieve the document corresponding to the path data; and to classify the document by preset keywords.

Images