Master-slave distributed file processing method and system, master node, and slave node
A technology of distributed files and processing methods, applied in the field of distributed cluster systems, can solve the problems of large space resources, consumption, occupying relatively large disk space and memory space, etc., so as to reduce space resources, reduce cumbersomeness, and improve accuracy Effect
Active Publication Date: 2019-11-29
济南浪潮数据技术有限公司
View PDF14 Cites 0 Cited by
- Summary
- Abstract
- Description
- Claims
- Application Information
AI Technical Summary
Problems solved by technology
However, due to the large number of slave nodes and the need for information interaction between slave nodes during maintenance and deployment to ensure the integrity of the file list and database, the use of slave nodes for maintenance and deployment will not only increase Document processing is cumbersome and will reduce the accuracy of document processing
In addition, since each slave node is deployed with a database, and the file list is stored in the database, it will occupy relatively large disk space and memory space, thus consuming relatively large space resources
Method used
the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View moreImage
Smart Image Click on the blue labels to locate them in the text.
Smart ImageViewing Examples
Examples
Experimental program
Comparison scheme
Effect test
Embodiment Construction
[0045] The following will clearly and completely describe the technical solutions in the embodiments of the application with reference to the drawings in the embodiments of the application. Apparently, the described embodiments are only some of the embodiments of the application, not all of them. Based on the embodiments in this application, all other embodiments obtained by persons of ordinary skill in the art without making creative efforts belong to the scope of protection of this application.
[0046] see figure 1 , which shows a flow chart of a master-slave distributed file processing method applied to a master node provided in an embodiment of the present application, which may include:
[0047] S11: Send a file scanning command to the slave node, so that the slave node scans according to the scan command.
[0048] In the master-slave mode of Master (master)-Worker (slave) or master-slave mode of Master (master)-Slave (slave), the master node sends a file scanning comma...
the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More PUM
Login to View More Abstract
The invention discloses a master-slave distributed file processing method and system, a master node, a slave node, and a computer readable storage medium. The master-slave distributed file processingmethod applied to the master node comprises the steps: transmitting a file scanning command to the slave node, so as to enable the slave node to carry out the scanning according to the scanning command; receiving node information of the slave nodes sent by the slave nodes and a file list obtained by scanning the slave nodes; judging whether a file list which is repeated with the file list exists in a database contained in the server or not, and if so, carrying out duplicate removal operation; and storing the node information and the file list into a database. According to the technical schemedisclosed by the invention, the slave nodes are used for scanning to obtain the file list, and the master node is used for deploying the database and maintaining the file list, so that the complexityof file processing and occupied space resources are reduced, and the accuracy of file processing is improved.
Description
technical field [0001] The present application relates to the technical field of distributed cluster systems, and more specifically, to a master-slave distributed file processing method, a master node, a slave node, a system, and a computer-readable storage medium. Background technique [0002] With the rapid development of cloud technology and big data, data growth is getting faster and faster, and distributed cluster systems are more and more widely used. A distributed cluster system consists of several or hundreds of nodes, among which these nodes are divided into master nodes and slave nodes, and each node needs to cooperate to process files so that the processed files can be processed accordingly operate. [0003] At present, in the process of master-slave distributed file processing, the maintenance of the file list and the deployment of the database are generally performed by the slave nodes. The maintenance mentioned here specifically refers to operations such as ob...
Claims
the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More Application Information
Patent Timeline
Login to View More IPC IPC(8): G06F16/182G06F16/27
CPCG06F16/182G06F16/27
Inventor 刘洋
Owner 济南浪潮数据技术有限公司