Data writing method and device

Abstract

This specification provides a data writing method, which is applied to a front-end machine. The front-end machine is used to write data to a distributed data storage system. The distributed data storage system includes a target partition node and multiple associated proxy nodes, and stream files corresponding to each associated proxy node are created on the target partition node. The method includes: obtaining proxy node information corresponding to the target partition node, where the proxy node information includes information of multiple proxy nodes associated with the target partition node; determining a target proxy node among the multiple proxy nodes according to the proxy node information, where the target proxy node corresponds to a target stream file created in the target partition node; and sending the data to be written to the target proxy node so that the target proxy node writes the data to the target stream file.

Description

Technical Field

[0001] This specification relates to the field of data storage, and particularly to a data writing method and apparatus. Background Art

[0002] In a large-scale distributed data storage system, it is a major challenge to balance data aggregation and traffic. Among them, too high a degree of data aggregation will lead to data access hotspots. Although data traffic balance can solve the above hotspots, it will also reduce the space utilization rate of the storage system.

[0003] In related technologies, the above hotspots are generally solved by splitting data sets. However, this operation will reduce the availability of the split data sets, thereby affecting the overall service level agreement (SLA) of the system. In addition, the operation of splitting data sets is relatively complex and may cause the storage system to be unable to handle sudden data access pressure in a timely manner. Summary of the Invention

[0004] In view of this, this specification provides a data writing method and apparatus to solve the deficiencies in related technologies.

[0005] Specifically, this specification is implemented through the following technical solutions:

[0006] According to a first aspect of an embodiment of this specification, a data writing method is provided, which is applied to a front-end machine. The front-end machine is used to write data to a distributed data storage system. The distributed data storage system includes a target partition node and a plurality of associated proxy nodes, and a flow file corresponding to each associated proxy node is created on the target partition node. The method includes:

[0007] Obtain proxy node information corresponding to the target partition node. The proxy node information includes information of a plurality of proxy nodes associated with the target partition node;

[0008] Determine a target proxy node among the plurality of proxy nodes according to the proxy node information. The target proxy node corresponds to a target flow file created in the target partition node;

[0009] Send the data to be written to the target proxy node, so that the target proxy node writes the data to the target flow file.

[0010] According to a second aspect of the embodiments of the present specification, a data writing device is provided, which is applied to a front-end machine. The front-end machine is used to write data into a distributed data storage system. The distributed data storage system includes a target partition node and a plurality of associated proxy nodes, and stream files corresponding to each associated proxy node are created on the target partition node. The device includes:

[0011] A proxy node information acquisition unit, configured to acquire proxy node information corresponding to the target partition node, where the proxy node information includes information of a plurality of proxy nodes associated with the target partition node;

[0012] A target proxy node determination unit, configured to determine a target proxy node from the plurality of proxy nodes according to the proxy node information, where the target proxy node corresponds to a target stream file created in the target partition node;

[0013] A data writing unit, configured to send data to be written to the target proxy node, so that the target proxy node writes the data into the target stream file.

[0014] According to a third aspect of the embodiments of the present specification, a computer-readable storage medium is provided, on which a computer program is stored. When the program is executed by a processor, the steps of the method described in the first aspect are implemented.

[0015] According to a fourth aspect of the embodiments of the present specification, an electronic device is provided, including a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the program, the steps of the method described in the first aspect are implemented.

[0016] In the technical solution provided in the present specification, by introducing the technology of target partition nodes and proxy nodes in the distributed data storage system, data aggregation and traffic balance are taken into account without splitting the data set. On the one hand, the target partition node can uniformly create files in the corresponding partition file directory, improving the data aggregation degree and the data recovery efficiency at the same time; on the other hand, the front-end machine can select a target proxy node from multiple proxy nodes to write data, ensuring that the traffic reaches a certain traffic balance among different proxy nodes, effectively solving the data access hot spot problem and improving the system availability.

[0017] It should be understood that the above general description and the following detailed description are only exemplary and explanatory, and cannot limit the present specification. Brief Description of the Drawings

[0018] Figure 1 is a schematic diagram of the architecture of a data writing system shown in an exemplary embodiment of the present specification;

[0019] Figure 2 is a schematic flowchart of a data writing method shown in an exemplary embodiment of this specification;

[0020] Figure 3 is a schematic diagram of a three - party interaction among a front - end machine, a target partition node, and a target proxy node shown in an exemplary embodiment of this specification;

[0021] Figure 4 is a schematic diagram of an amplification stream file and a reduction - expansion stream method in a data writing system shown in an exemplary embodiment of this specification;

[0022] Figure 5 is a schematic structural diagram of an electronic device shown in an exemplary embodiment of this specification;

[0023] Figure 6 is a schematic structural diagram of a data writing device shown in an exemplary embodiment of this specification. Detailed implementation manners

[0024] Here, exemplary embodiments will be described in detail, and examples are shown in the drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The implementation manners described in the following exemplary embodiments do not represent all implementation manners consistent with this specification. Instead, they are merely examples of devices and methods consistent with some aspects of this specification as detailed in the appended claims.

[0025] The terms used in this specification are for the purpose of describing specific embodiments only and are not intended to limit this specification. The singular forms "a", "the", and "said" used in this specification and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term "and / or" as used herein refers to and includes any or all possible combinations of one or more of the associated listed items.

[0026] It should be understood that although the terms first, second, third, etc. may be used in this specification to describe various information, such information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of this specification, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the word "if" as used herein may be interpreted as "when" or "while" or "in response to determining".

[0027] Figure 1 is a schematic architecture diagram of a data writing system shown in an exemplary embodiment of this specification. As Figure 1As shown in the figure, it may include a front-end machine 11, a master node 12, partition nodes 13, proxy nodes 14, and a distributed data storage system 15. Among them, the distributed data storage system 15 may include the master node 12, partition nodes 13, and proxy nodes 14.

[0028] The front-end machine 11 refers to an electronic device that supports data writing technology and can be used by users. During the operation of the system, the front-end machine 11 can be used to receive the data to be written input by the user and write it into the distributed data storage system 15. Among them, the user can use electronic devices of the following types as the front-end machine 11: mobile phones, tablet devices, laptop computers, personal digital assistants (PDAs), wearable devices (such as smart glasses, smart watches, etc.). One or more embodiments of this specification do not limit this.

[0029] The master node 12 is a cluster of devices in the distributed data storage system 15 that follows a certain high-availability protocol. Among them, the high-availability protocol can be protocols such as Paxos, Raft, Zab (Zookeeper Atomic Broadcast), etc. At the same time, the master node 12 can maintain relevant information about the partition nodes 13 to be used to query the address information of the existing partition nodes 13 in the distributed data storage system 15 for the front-end machine 11.

[0030] The partition node 13 is a device that stores the corresponding partition data under the distributed data storage system 15. Among them, the above-mentioned corresponding partition data can be stored in the form of a stream file. The partition node 13 can be used to manage the reading and writing of data in the above-mentioned corresponding partition data and can provide information about the proxy node 14 associated with the partition node 13 for the front-end machine 11.

[0031] The proxy node 14 is the corresponding device in the distributed data storage system 15 that receives the data to be written by the front-end machine 11. A write interface of the stream file created by the partition node 13 can be set in the proxy node 14 to be used to write the above-mentioned data to be written into the stream file of the associated partition node 13.

[0032] Figure 2 It is a schematic flowchart of a data writing method shown in an exemplary embodiment of this specification. As Figure 2 shown, this method is applied to the front-end machine. The above front-end machine is used to write data into the distributed data storage system. The above distributed data storage system includes a target partition node and multiple proxy nodes associated with it, and stream files corresponding to each associated proxy node are created on the above target partition node. This method may include the following steps:

[0033] S201. Obtain the proxy node information corresponding to the target partition node, where the proxy node information includes information of multiple proxy nodes associated with the target partition node.

[0034] The above proxy node information may include basic information such as the naming, IP addresses, and creation times of the above multiple proxy nodes, or load information of the multiple proxy nodes associated with the target partition node, such as CPU utilization rate, bandwidth size, and available memory space size. Similarly, the above proxy node information may also include the load information of the flow files corresponding to the above multiple proxy nodes in the above target partition node.

[0035] Among them, the method for the front-end machine to obtain the above proxy node information can be changed according to actual needs. For example, the target partition node can return the corresponding proxy node information according to the acquisition request for proxy node information initiated by the front-end machine, or the front-end machine can directly read the specified node storing the proxy node information in the distributed data storage system, or the target partition node can actively send the proxy node information to the front-end machine at regular intervals according to a preset period. This specification does not limit this.

[0036] In addition, when the front-end machine does not have the address of the target partition node, it can obtain the address of the target partition node by accessing the master node in the distributed data storage system.

[0037] In an embodiment, in the case of not obtaining the address information of the target partition node or the target partition node not responding to the information acquisition request, the front-end machine can initiate a request to obtain the target partition node from the master node; and obtain the address of the specified partition node returned by the master node, and update the target partition node to the specified partition node. In this embodiment, by utilizing the stable and reliable characteristics of the master node, the front-end machine can obtain an effective target partition node at any time, improving the overall availability of the system.

[0038] S202. Determine a target proxy node from the multiple proxy nodes according to the proxy node information, where the target proxy node corresponds to the target flow file created in the target partition node.

[0039] As described above, the above proxy node information may include the load information of the proxy nodes. When there are multiple proxy nodes, the front-end machine can determine the target proxy node according to the load information of the proxy nodes.

[0040] In one embodiment, the front-end machine determines a proxy node with better performance as the target proxy node among the multiple proxy nodes according to the load information of the above-mentioned proxy nodes. For example, there are proxy nodes 1, 2, and 3 with the same performance configuration. Among them, there are more users accessing proxy nodes 1 and 2, and according to the load information of the above-mentioned proxy nodes, it can be known that the available bandwidth of proxy nodes 1 and 2 is less than that of proxy node 3. Then the front-end machine can determine proxy node 3 as the target proxy node and write data through the stream file corresponding to this target proxy node.

[0041] When there are multiple stream files corresponding to the target proxy node, the front-end machine can determine the target stream file in the target proxy node according to the load information of the above-mentioned stream files.

[0042] In one embodiment, when the number of target stream files corresponding to the target proxy node is multiple, the front-end machine can select a stream file with better performance among the stream files corresponding to the above-mentioned target proxy node as the target stream file according to the load information of the above-mentioned stream files.

[0043] When facing multiple proxy nodes and each proxy node corresponds to multiple stream files, the front-end machine can combine the load information of the above-mentioned proxy nodes and the load information of the above-mentioned stream files to determine the target proxy node and the target stream file.

[0044] In one embodiment, the front-end machine first determines a part of the proxy nodes with better performance among the multiple proxy nodes as candidate proxy nodes according to the load information of the above-mentioned proxy nodes, and then determines a stream file with better performance as the target stream file according to a preset stream selection rule among the stream files corresponding to the above-mentioned candidate proxy nodes, and uses the proxy node corresponding to this target stream file as the target proxy node. For example, there are proxy nodes 1, 2, and 3, and stream files A, B, C, and D. Among them, proxy node 1 corresponds to stream file A, proxy node 2 corresponds to stream file B, and proxy node 3 corresponds to stream files C and D. When the performance of proxy node 2 is significantly lower than that of proxy nodes 1 and 3 due to hard disk damage or other reasons, proxy node 2 is excluded from the candidate proxy nodes, that is, the target stream file is selected only from the stream files A, C, and D corresponding to proxy nodes 1 and 3.

[0045] Those skilled in the art can understand that the so-called "better performance" can be used to characterize that the target proxy node has more advantages than other proxy nodes in one or more aspects such as bandwidth and traffic processing speed, or has a more excellent performance in the performance quantification specifications that are authoritative and representative in the industry. This specification does not limit this. A part of the proxy nodes with better performance is a relatively loose range, which can be set to the top 3 or the top 30% in terms of performance among all proxy nodes according to actual needs. And the above preset flow selection rule can also be set to use the flow file with the best performance as the target flow file, or use a selection algorithm similar to the median random method to determine the target flow file, that is, randomly select a flow file from half of the flow files with better performance as the target flow file, so as to balance the flow selection requirements of efficiency and fairness.

[0046] Before determining the target flow file, the front-end machine can obtain the current load information according to the proxy node information. The current load information includes: the load conditions of multiple proxy nodes and / or the load conditions of the flow files corresponding to the above multiple proxy nodes in the target partition node, and determine the flow expansion / contraction strategy for the target partition node based on the current load information, so as to enable the target partition node to dynamically adjust the number of flow files in the face of different data access pressures, and then ensure the availability of the distributed data storage system.

[0047] In one embodiment, when the load parameter value of the current load information in the preset dimension is greater than the preset load upper limit value, a flow expansion request is sent to the target partition node, so that the target partition node creates a new flow file and establishes an association relationship between the new flow file and a proxy node.

[0048] In another embodiment, when the load parameter value of the current load information in the preset dimension is less than the preset load lower limit value, a flow contraction request is sent to the target partition node, so that the target partition node deletes the existing flow file and removes the association relationship between the existing flow file and the corresponding proxy node.

[0049] The above preset dimension can be the memory usage rate, bandwidth size of the target partition node, or a quantifiable evaluation standard obtained by integrating the above multiple pieces of information. This specification does not limit this. The above preset load is a numerical range set based on the above preset dimension for judging the load degree of the target partition node. Among them, when the load parameter value of the current load information in the preset dimension is greater than the preset load upper limit value, the front-end machine determines that the load degree of the target partition node is relatively high at this time and needs to quickly expand the flow file to reduce its load degree. Or when the load parameter value of the current load information in the preset dimension is less than the preset load lower limit value, the front-end machine determines that the load degree of the target partition node is relatively low at this time and can stably reduce the flow file to save system resources.

[0050] S203. Send the data to be written to the target proxy node for the target proxy node to write to the target stream file.

[0051] After the target proxy node writes to the target stream file, the front-end machine can enable the target partition node to obtain and record the position information of the data written by the target proxy node, thus ensuring the correct management of the data in the target partition node.

[0052] In one embodiment, the front-end machine receives a write completion prompt message from the target proxy node for the data to be written. The write completion prompt message includes the storage position information of the data to be written in the target partition node, and sends a write completion request including the storage position information to the target partition node, so that the target partition node records the storage position information of the target data. The storage position information may include: the address offset, length, and status information of the data to be written in the target partition node, etc., which are used to locate the storage position of the data to be written. This specification does not limit this.

[0053] The front-end machine can also establish an information interaction channel with the target partition node to facilitate the front-end machine to timely obtain information about the target partition node or the proxy node associated with the target partition node and make corresponding processing.

[0054] In one embodiment, through the information interaction channel established with the target partition node, the front-end machine receives a change notification message about the proxy node sent by the target partition node. The change notification message is used to indicate the running status of the proxy node associated with the target partition node. The front-end machine can update the pre-obtained proxy node information according to the change notification message to filter out abnormal proxy nodes. For example: after the front-end machine uses proxy node 3 to write data, proxy node 3 has abnormal situations such as crashing, resulting in the front-end machine being unable to continue writing new data through proxy node 3. At this time, the target partition node timely notifies the front-end machine of the above situation through the above information interaction channel, so that the front-end machine will no longer obtain the proxy node information about proxy node 3 when writing data next time, and thus will not be able to write data to the stream file corresponding to proxy node 3.

[0055] As can be seen from the above embodiments, the data writing method in this specification decouples the partition management service and the data writing service in the related art by adding proxy nodes, enabling the front-end machine to balance data aggregation and traffic during the process of writing data to the distributed data storage system. In addition, the use of the scale-out flow technology can flexibly handle sudden data access pressure, thereby further improving the availability of the system.

[0056] Figure 3Schematic diagram of a three-party interaction among a front-end machine, a target partition node, and a target proxy node shown in an exemplary embodiment of this specification. As Figure 3 shown, the interaction process may include the following steps:

[0057] S301. Receive data to be written.

[0058] In one embodiment, a user inputs data "hello, world!" to be written into the front-end machine so that the data to be written is written into a distributed data storage system.

[0059] S302. Send a request to obtain proxy node information.

[0060] In one embodiment, after receiving the data to be written, the front-end machine sends a request to obtain proxy node information to the target partition node in the above-mentioned distributed data storage system.

[0061] S303. Query proxy node information.

[0062] In one embodiment, after receiving the above-mentioned request to obtain proxy node information, the target partition node queries the list of proxy nodes associated with the target partition node provided in the preset nodes of the distributed data storage system, and queries the stream files created on the target partition node respectively corresponding to each associated proxy node in the proxy node list to form a stream file list, and carries the proxy node list and the stream file list in the proxy node information. At the same time, the load information of each proxy node in the proxy node list and the load information of each stream file in the stream file list are carried in the proxy node information as the current load information of each proxy node. Assume that there are proxy nodes A, B, and C in the proxy node list and each proxy node corresponds to two stream files "A1", "A2", "B1", "B2", "C1", "C2" created on the target partition node. Among them, the current load information of proxy node C indicates that the current maximum bandwidth of proxy node C is significantly lower than that of proxy node A, and the performance difference between each stream file is small. At the same time, proxy node B has a disk failure, resulting in almost no response from proxy node B.

[0063] S304. Return proxy node information.

[0064] In one embodiment, the front-end machine receives the proxy node information returned by the target partition node.

[0065] S305. Determine the target proxy node and the associated target stream file according to the proxy node information.

[0066] In one embodiment, the front-end machine screens out the proxy nodes with lower performance in the proxy node list according to the current load information of each proxy node. Obviously, since the disk of proxy node B is damaged, affecting its normal operation, its performance can be regarded as non-existent, resulting in the screening out of proxy node B from the proxy node list. At the same time, the stream files "B1" and "B2" in the stream file list are also screened out. At this time, the front-end machine can select 2 stream files with better performance from the 4 stream files "A1", "A2", "C1", and "C2" in the stream file list according to the median random method. Since the current maximum bandwidth of proxy node C is significantly lower than that of proxy node A, and the performance difference between the stream files is small, so "A1" and "A2" are selected, and then the stream file A1 is selected as the target stream file from these two stream files through a random algorithm.

[0067] S306. Send a write request for the data to be written.

[0068] In one embodiment, the front-end machine sends the data to be written "hello, world!" to the target proxy node associated with the target stream file in the way of backup write.

[0069] S307. Write the data to be written into the target stream file.

[0070] In one embodiment, after receiving the data to be written, the target proxy node writes the data to be written into the target stream file.

[0071] S308. Return the storage location information of the data to be written in the target partition node.

[0072] In one embodiment, after the target proxy node successfully writes the data to be written, it can return the data storage address information of the above data to be written on the target partition node to the front-end machine, where the storage address information can indicate that the offset of the data to be written in the storage space of the target partition node is 100 and the length is 12.

[0073] S309. Send the storage location information.

[0074] In one embodiment, the front-end machine sends the storage location information to the target partition node.

[0075] S310. Record the storage location information

[0076] In one embodiment, the target partition node records the received storage location information of the data to be written in the specified space.

[0077] S311. Feedback the recording situation.

[0078] S312. Initiate a request to establish an information interaction channel.

[0079] In one embodiment, after the front-end machine officially writes the data to be written, it can initiate a request to establish an information interaction channel to the target partition node, and an information interaction channel similar to a "heartbeat" mechanism has been established.

[0080] S313. Continuously detect and generate change notification information of the proxy node.

[0081] In one embodiment, when the target partition node detects that the proxy node A has crashed, it can timely send the change notification information including the above situation to the front-end machine.

[0082] S314. Feedback the change notification information.

[0083] In one embodiment, according to the received change notification information, the front-end machine updates the previously obtained proxy node information to screen out the abnormal proxy node B in the proxy node list and the stream files B1 and B2 associated with the proxy node B in the stream file list.

[0084] Before determining the target proxy node and the associated target stream file according to the proxy node information, the front-end machine can determine whether to expand or reduce the stream files for the target partition node according to the current load information of each proxy node in the proxy node information.

[0085] Figure 4 It is a schematic diagram of a method for expanding and reducing stream files in a data writing system shown in an exemplary embodiment of this specification. As Figure 4 shown, assuming that in the initial state, 4 stream files, namely "Stream File 1-4", are created in the target partition node, and as Figure 4 shown by the two-way connection lines below, "Stream File 1-4" respectively correspond to each proxy node "Proxy Node 1-3" associated with the target partition node.

[0086] When the stream expansion and reduction policy defines that the number of stream files in the target partition node is increased to twice the original when expanding the stream, and when the front-end machine determines that the current partition node faces greater data access pressure according to the current load information included in the proxy node information, the front-end machine can select to expand the number of stream files according to the stream expansion and reduction policy, that is, create new "Stream File 5-8", and can establish the association relationship with the specified proxy node for the newly added stream files respectively according to the rule of "sequential allocation".

[0087] In the case where the target partition node has performed a flow expansion operation above, when the flow reduction and expansion policy defines flow reduction, the number of flow files in the target partition node is reduced to 1 / 2 of the original. And when the front-end machine determines that the current partition node is facing a gentle data access pressure according to the current load information included in the proxy node information, the front-end machine can select to reduce the number of flow files according to the flow reduction and expansion policy, that is, delete the created "flow files 5-8" and remove the association relationship between the "flow files 5-8" and the corresponding proxy nodes.

[0088] When the flow file 1 cannot work properly due to disk damage, the front-end machine can determine that the flow file 1 is abnormal according to the current load information included in the proxy node information, and the proxy node 1 corresponding to the flow file 1 is normal. Then the front-end machine can send a request for replacing the flow file 1 to the target partition node, so that the target partition node removes the flow file 1 and creates a new flow file to ensure the stability of the number of flow files. At the same time, the target partition node can determine the proxy node 2 with the smallest current data access pressure corresponding to the new flow file according to the load information of the proxy nodes 1-3.

[0089] In addition, regarding Figure 4 in the case where the proxy node fails to write the flow file or the flow file response times out, the front-end machine can be notified again to reselect the target flow file. As for the flow selection policy for the specific process of reselecting the target flow file, Figure 3 and the related embodiments have been described and will not be elaborated here.

[0090] Figure 5 is a schematic structural diagram of an electronic device in an exemplary embodiment. Please refer to Figure 5 . At the hardware level, the electronic device includes a processor, an internal bus, a network interface, a memory, and a non-volatile memory. Of course, other required hardware may also be included. The processor reads the corresponding computer program from the non-volatile memory into the memory and then runs it, forming a data writing device logically. Of course, in addition to the software implementation method, this specification does not exclude other implementation methods, such as logical devices or a combination of software and hardware, etc. That is to say, the execution subject of the following processing flow is not limited to each logical unit, and can also be hardware or logical devices.

[0091] Corresponding to the embodiment of the foregoing data writing method, this specification also provides an embodiment of a data writing device.

[0092] Please refer to Figure 6 , Figure 6 is a schematic structural diagram of a data writing device shown in an exemplary embodiment of this specification. As Figure 6As shown, in the software implementation, the data writing device is applied to a front-end machine, which is used to write data to a distributed data storage system. The distributed data storage system includes a target partition node and multiple associated proxy nodes, and stream files corresponding to each associated proxy node are created on the target partition node. The device includes:

[0093] A proxy node information acquisition unit 601, configured to acquire proxy node information corresponding to the target partition node, where the proxy node information includes information of multiple proxy nodes associated with the target partition node;

[0094] A target proxy node determination unit 602, configured to determine a target proxy node from the multiple proxy nodes according to the proxy node information, where the target proxy node corresponds to a target stream file created in the target partition node;

[0095] A data writing unit 603, configured to send data to be written to the target proxy node, so that the target proxy node writes the data to the target stream file.

[0096] Optionally, the proxy node information further includes: load information of the multiple proxy nodes. The target proxy node determination unit 602 is specifically configured to:

[0097] Determine a proxy node with better performance as the target proxy node from the multiple proxy nodes according to the load information of the multiple proxy nodes.

[0098] Optionally, the proxy node information further includes current load information, where the current load information includes: load information of the multiple proxy nodes and / or load information of stream files corresponding to the multiple proxy nodes in the target partition node. The device further includes:

[0099] A target stream file determination unit 604, when the number of target stream files corresponding to the target proxy node is multiple, selects a stream file with better performance from the stream files corresponding to the target proxy node as the target stream file according to the load information of the stream files.

[0100] Optionally, the proxy node information further includes current load information, where the current load information includes: load information of the multiple proxy nodes and / or load information of stream files corresponding to the multiple proxy nodes in the target partition node; the device further includes:

[0101] The dynamic flow scaling unit 605 initiates a flow expansion request to the target partition node when the load parameter value of the current load information in the preset dimension is greater than the preset load upper limit value, so that the target partition node creates a new flow file and establishes an association relationship between the new flow file and a proxy node; or

[0102] When the load parameter value of the current load information in the preset dimension is less than the preset load lower limit value, it initiates a flow contraction request to the target partition node, so that the target partition node deletes the existing flow file and removes the association relationship between the existing flow file and the corresponding proxy node.

[0103] Optionally, the data writing unit 603 is specifically configured to:

[0104] Receive a write completion prompt message for the data to be written from the target proxy node, where the write completion prompt message includes the storage location information of the data to be written in the target partition node;

[0105] Initiate a write completion request including the storage location information to the target partition node, so that the target partition node records the storage location information of the target data.

[0106] Optionally, the device further includes:

[0107] An information interaction unit 606, configured to receive change notification information about the proxy node sent by the target partition node through an information interaction channel established with the target partition node, where the change notification information is used to indicate the running state of the proxy node associated with the target partition node;

[0108] Update the pre-acquired proxy node information according to the change notification information to screen out abnormal proxy nodes.

[0109] Optionally, the distributed data storage system further includes a master node; the proxy node information acquisition unit 601 is specifically configured to initiate an information acquisition request to the target partition node and receive the proxy node information returned by the target partition node.

[0110] The device further includes:

[0111] A target partition node acquisition unit 607, configured to initiate a request to acquire a target partition node to the master node when the address information of the target partition node has not been acquired or the target partition node does not respond to the information acquisition request; and acquire the address of the specified partition node returned by the master node and update the target partition node to the specified partition node.

[0112] For the implementation processes of the functions and roles of each unit in the above device, please refer to the implementation processes of the corresponding steps in the above method for details, which will not be elaborated here.

[0113] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to the partial descriptions of the method embodiments. The device embodiments described above are only illustrative. The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed to multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the solution in this specification. Those of ordinary skill in the art can understand and implement it without creative efforts.

[0114] The embodiments of the subject matter and functional operations described in this specification can be implemented in the following: digital electronic circuits, tangible computer software or firmware, computer hardware including the structures disclosed in this specification and their structural equivalents, or a combination of one or more of them. The embodiments of the subject matter described in this specification can be implemented as one or more computer programs, that is, one or more modules in computer program instructions encoded on a tangible non-transitory program carrier to be executed by a data processing device or to control the operation of a data processing device. Alternatively or additionally, the program instructions can be encoded on a manually generated propagated signal, such as a machine-generated electrical, optical or electromagnetic signal, which is generated to encode information and transmit it to a suitable receiver device for execution by a data processing device. A computer storage medium can be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or a combination of one or more of them.

[0115] The processes and logical flows described in this specification can be executed by one or more programmable computers executing one or more computer programs to perform corresponding functions by operating on input data and generating outputs. The processes and logical flows can also be executed by dedicated logic circuits - such as FPGAs (Field Programmable Gate Arrays) or ASICs (Application Specific Integrated Circuits), and the device can also be implemented as dedicated logic circuits.

[0116] Computers suitable for executing computer programs include, for example, general and / or special-purpose microprocessors, or any other type of central processing unit. Generally, the central processing unit will receive instructions and data from read-only memory and / or random access memory. The basic components of a computer include a central processing unit for implementing or executing instructions and one or more memory devices for storing instructions and data. Generally, a computer will also include one or more mass storage devices for storing data, such as magnetic disks, magneto-optical disks, or optical disks, etc., or the computer will be operatively coupled to such mass storage devices to receive data therefrom or transfer data thereto, or both. However, a computer is not necessarily required to have such devices. In addition, a computer may be embedded in another device, such as a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a global positioning system (GPS) receiver, or a portable storage device such as a universal serial bus (USB) flash drive, to name just a few examples.

[0117] Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and memory devices, such as including semiconductor memory devices (e.g., EPROM, EEPROM, and flash memory devices), magnetic disks (e.g., internal hard disks or removable disks), magneto-optical disks, and CD ROM and DVD-ROM disks. The processor and memory may be supplemented by, or incorporated in, special logic circuitry.

[0118] Although this specification contains many specific implementation details, these should not be construed as limiting the scope of any invention or the scope of what is claimed, but rather as mainly describing the features of specific embodiments of a particular invention. Certain features described in multiple embodiments in this specification may also be implemented in combination in a single embodiment. On the other hand, the various features described in a single embodiment may also be implemented separately in multiple embodiments or in any suitable sub-combination. In addition, although features may operate in certain combinations as described above and even be claimed as such initially, one or more features from a claimed combination may in some cases be removed from that combination, and the claimed combination may be directed to a sub-combination or a variation of a sub-combination.

[0119] Similarly, although operations are depicted in the drawings in a particular order, this should not be construed as requiring that the operations be performed in the particular order shown or sequentially, or that all illustrated operations be performed, to achieve the desired result. In some cases, multitasking and parallel processing may be advantageous. In addition, the separation of various system modules and components in the above embodiments should not be construed as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.

[0120] Accordingly, specific embodiments of the subject matter have been described. Other embodiments are within the scope of the appended claims. In some cases, the acts recited in the claims can be performed in a different order and still achieve the desired result. In addition, the processes depicted in the figures are not necessarily in the particular order or sequential order shown to achieve the desired result. In some implementations, multitasking and parallel processing may be advantageous.

[0121] The above are only the preferred embodiments of this specification and are not intended to limit this specification. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principle of this specification shall be included within the scope of protection of this specification.

Claims

1. A data writing method, characterized in that, Applied to a front-end machine, the front-end machine is used to write data to a distributed data storage system. The distributed data storage system includes a target partition node and multiple associated proxy nodes, and stream files corresponding to each of the associated proxy nodes are created on the target partition node. The target partition node and the multiple proxy nodes are independent devices. The method includes: Obtain the proxy node information corresponding to the target partition node, where the proxy node information includes information of multiple proxy nodes associated with the target partition node; Determine a target proxy node among the multiple proxy nodes according to the proxy node information, and the target proxy node corresponds to a target stream file created in the target partition node; Send the data to be written to the target proxy node so that the target proxy node writes it into the target stream file.

2. The method according to claim 1, wherein The proxy node information further includes: the load information of the multiple proxy nodes; the determining the target proxy node among the multiple proxy nodes according to the proxy node information includes: Determine the target proxy node among the multiple proxy nodes according to the load information of the multiple proxy nodes, and the bandwidth and / or traffic processing speed of the target proxy node are higher than those of other proxy nodes among the multiple proxy nodes.

3. The method according to claim 1 or 2, characterized in that, The proxy node information further includes: the load information of the stream files corresponding to the multiple proxy nodes in the target partition node; the method further includes: When the number of target stream files corresponding to the target proxy node is multiple, select the target stream file according to the load information of the stream files, and the bandwidth and / or traffic processing speed of the target stream file are higher than those of other stream files corresponding to the target proxy node.

4. The method according to claim 1, wherein The proxy node information further includes current load information, and the current load information includes: the load information of the multiple proxy nodes and / or the load information of the stream files corresponding to the multiple proxy nodes in the target partition node; the method further includes: When the load parameter value of the current load information in a preset dimension is greater than a preset load upper limit value, initiate a flow expansion request to the target partition node so that the target partition node creates a new stream file and establishes an association relationship between the new stream file and a proxy node; or When the load parameter value of the current load information in a preset dimension is less than a preset load lower limit value, initiate a flow contraction request to the target partition node so that the target partition node deletes an existing stream file and removes the association relationship between the existing stream file and the corresponding proxy node.

5. The method according to claim 1, wherein The writing the data to be written into the target stream file through the target proxy node includes: Receive a write completion prompt message for the data to be written from the target proxy node, where the write completion prompt message includes the storage location information of the data to be written in the target partition node; Initiate a write completion request including the storage location information to the target partition node so that the target partition node records the storage location information of the data to be written.

6. The method according to claim 1, characterized in that, Further includes: Receive, through the information interaction channel established with the target partition node, the change notification information sent by the target partition node, where the change notification information is used to indicate the running status of the proxy node associated with the target partition node; Update the pre-acquired proxy node information according to the change notification information to filter out abnormal proxy nodes.

7. The method according to claim 1, wherein The distributed data storage system further includes a master node; The obtaining the proxy node information corresponding to the target partition node includes: sending an information obtaining request to the target partition node and receiving the proxy node information returned by the target partition node; The method further includes: in the case where the address information of the target partition node is not obtained or the target partition node does not respond to the information obtaining request, sending a request to obtain the target partition node to the master node; and obtaining the address of the specified partition node returned by the master node and updating the target partition node to the specified partition node.

8. A data writing device, characterized in that, Applied to a front-end machine, the front-end machine is used to write data into a distributed data storage system, the distributed data storage system includes a target partition node and a plurality of proxy nodes associated therewith, and stream files respectively corresponding to the respective associated proxy nodes are created on the target partition node, and the target partition node and the plurality of proxy nodes are independent devices; the apparatus includes: A proxy node information obtaining unit, configured to obtain proxy node information corresponding to the target partition node, where the proxy node information includes information of a plurality of proxy nodes associated with the target partition node; A target proxy node determining unit, configured to determine a target proxy node from the plurality of proxy nodes according to the proxy node information, where the target proxy node corresponds to a target stream file created in the target partition node; A data writing unit, configured to send the data to be written to the target proxy node so that the target proxy node writes the data into the target stream file.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

10. An electronic device, comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, When the processor executes the program, the steps of the method according to any one of claims 1 to 7 are implemented.

Images