Metadata-based writing method and device, computer equipment and storage medium

Abstract

The invention belongs to the field of data processing and finance, and relates to a metadata-based writing method, which comprises the following steps: analyzing a data writing request sent by a client to obtain file metadata, selecting a target data node from a node list according to the file metadata, and determining a current data node by the client according to node identification information, a current data node writes a data packet sent by a client into a preset storage position, generates a corresponding response packet, puts the response packet into a preset to-be-processed queue, calls a callback function to obtain the length of the data packet and slice metadata information, and synchronizes the slice metadata information to other nodes. And taking out the response packet from the to-be-processed queue according to the data packet length, putting into a preset response queue and returning to the client. The invention further provides a metadata-based writing device, computer equipment and a storage medium. In addition, the invention also relates to a block chain technology, and the data packet can be stored in a block chain. The metadata is transmitted at the data node, and the efficiency and timeliness of data writing are improved.

Description

Technical Field

[0001] The present application relates to the fields of data processing and financial technology, and in particular to a metadata-based writing method, apparatus, computer equipment and storage medium. Background Art

[0002] In the financial sector, with the rapid development of big data technology, processing and analyzing massive data has become the key for financial institutions to improve business efficiency, optimize decision-making processes and enhance risk management capabilities. As one of the core frameworks in the field of big data processing, the Hadoop ecosystem has been widely used in the financial industry due to its high scalability, high fault tolerance and efficient data processing capabilities. Among them, the Hadoop Distributed File System (HDFS), as the basic component of the Hadoop ecosystem, bears the heavy responsibility of big data storage and access.

[0003] HDFS is designed to handle large-scale data sets, has high fault tolerance, and is optimized for high-throughput data access. This enables HDFS to run stably on commercial hardware and effectively support the complex and large data processing needs in the financial field. The HDFS system architecture is mainly composed of two types of nodes: NameNode and DataNode. They each perform their own duties and jointly maintain the stable operation of the entire file system.

[0004] As the master node of HDFS, NameNode (NN) plays a vital role. It is responsible for managing the mapping relationship between the file system namespace and file blocks, and stores metadata of all files and directories, such as file names, permissions, block locations and other key information. These metadata are essential to ensure correct access and efficient management of data.

[0005] DataNode (DN) is the working node of HDFS, responsible for the actual data storage task. Each DataNode stores a certain number of data blocks, which are the basic units of HDFS data storage. DataNode receives data from clients or other DataNodes and writes it to the local disk, thus achieving distributed data storage.

[0006] In HDFS, the data writing process adopts a streaming approach. When the client writes data to HDFS, the data will flow through multiple DataNode nodes in sequence. After receiving the data, each DataNode node will return an Ack confirmation message to the client or upstream DataNode, indicating that the node has successfully received and stored the data. In order to ensure the reliability and fault tolerance of the data, HDFS usually writes data to three different DataNode nodes, and only after all three nodes have returned the Ack confirmation message, the client will receive the final Ack confirmation indicating that the data has been successfully written.

[0007] However, in actual applications, this data writing mechanism also brings certain performance loss. Since the actual data and reply messages need to be transmitted across the network between the three DN nodes, this not only increases the delay of data transmission, but may also be affected by network conditions, resulting in challenges to the timeliness of data writing. Especially in the financial field, the requirements for data processing and response speed are extremely high, and any delay may have an adverse impact on business decisions and risk control. Summary of the invention

[0008] The purpose of the embodiments of the present application is to propose a metadata-based writing method, apparatus, computer device and storage medium to solve the technical problems of data transmission delay and low writing efficiency in the data writing mechanism of the related art.

[0009] In order to solve the above technical problems, the embodiment of the present application provides a metadata-based writing method, which adopts the following technical solution:

[0010] Receiving a write data request sent by a client, parsing the write data request, and obtaining file metadata of a file to be written;

[0011] Selecting an available target data node from a pre-established node list according to the file metadata;

[0012] Sending the node identification information of the target data node to the client, and determining the current data node by the client according to the node identification information;

[0013] The data packet sent by the client is received through the current data node, and the data packet is written into a preset storage location. After the writing is completed, a corresponding response packet is generated and put into a preset queue for processing;

[0014] Calling a preset callback function through the current data node to obtain the data packet length and slice metadata information of the data packet, and synchronizing the slice metadata information to other nodes through a preset distributed coordination service, wherein the other nodes are target data nodes other than the current data node;

[0015] Through the current data node, the corresponding response packet is taken out from the queue to be processed according to the length of the data packet, put into the preset response queue, and returned to the client.

[0016] In order to solve the above technical problems, the embodiment of the present application further provides a metadata-based writing device, which adopts the following technical solution:

[0017] The parsing module is used to receive a write data request sent by a client, parse the write data request, and obtain file metadata of a file to be written;

[0018] A selection module, used to select an available target data node from a pre-established node list according to the file metadata;

[0019] A determination module, configured to send the node identification information of the target data node to the client, and determine the current data node according to the node identification information by the client;

[0020] A writing module is used to receive the data packet sent by the client through the current data node, and write the data packet into a preset storage location. After writing is completed, a corresponding response packet is generated and put into a preset queue for processing;

[0021] A synchronization module, used to obtain the data packet length and slice metadata information of the data packet by calling a preset callback function through the current data node, and synchronize the slice metadata information to other nodes through a preset distributed coordination service, wherein the other nodes are target data nodes other than the current data node;

[0022] The response module is used to take out the corresponding response packet from the queue to be processed according to the length of the data packet through the current data node, put it into the preset response queue, and return it to the client.

[0023] In order to solve the above technical problems, the embodiment of the present application further provides a computer device, which adopts the following technical solution:

[0024] The computer device includes a memory and a processor, wherein the memory stores computer-readable instructions, and the processor implements the steps of the metadata-based writing method as described above when executing the computer-readable instructions.

[0025] In order to solve the above technical problems, the embodiment of the present application further provides a computer-readable storage medium, which adopts the following technical solution:

[0026] The computer-readable storage medium stores computer-readable instructions, and when the computer-readable instructions are executed by a processor, the steps of the metadata-based writing method described above are implemented.

[0027] Compared with the prior art, this application has the following beneficial effects:

[0028] The present application provides a metadata-based writing method, which parses the write data request sent by the client to obtain the file metadata of the file to be written, selects the target data node from a pre-established node list according to the file metadata, sends the node identification information of the target data node to the client, and the client determines the current data node according to the node identification information, and sends the data packet to the current data node. The current data node receives the data packet and writes the data packet to a preset storage location. After the writing is completed, a corresponding response packet is generated and put into a preset queue to be processed, and a preset callback function is called to obtain the data packet length and slice metadata information of the data packet, and the slice metadata information is synchronized to other nodes. The current data node takes out the corresponding data packet from the queue to be processed according to the length of the data packet. The response packet is placed in a preset response queue and returned to the client through the preset response queue to notify the client that the data writing is completed; the present application selects available target data nodes for data writing, which can avoid excessive pressure on a single data node when a large number of files are written, improve data reliability and fault tolerance, improve the parallelism and overall efficiency of data access, and optimize resource utilization; the data packet is written to the preset storage location through the current data node, and the preset callback function is called to synchronize the slice metadata information of the data packet to other nodes through the preset distributed coordination service. Only metadata information is transmitted at the data node, which reduces a large amount of network data transmission between data nodes. While maintaining data uniformity, it improves the system's data writing performance and improves the efficiency and timeliness of data writing. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] In order to more clearly illustrate the scheme in the present application, a brief introduction is given below to the drawings required for use in the description of the embodiments of the present application. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0030] Figure 1 is an exemplary system architecture diagram to which the present application may be applied;

[0031] Figure 2 is a flowchart of an embodiment of a metadata-based writing method according to the present application;

[0032] Figure 3 is a structural schematic diagram of an embodiment of a metadata-based writing device according to the present application;

[0033] Figure 4 It is a structural diagram of an embodiment of a computer device according to the present application. DETAILED DESCRIPTION

[0034] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by technicians in the technical field of the present application; the terms used in the specification of the application herein are only for the purpose of describing specific embodiments and are not intended to limit the present application; the terms "including" and "having" and any variations thereof in the specification and claims of the present application and the above-mentioned drawings are intended to cover non-exclusive inclusions. The terms "first", "second", etc. in the specification and claims of the present application or the above-mentioned drawings are used to distinguish different objects, not to describe a specific order.

[0035] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0036] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the accompanying drawings.

[0037] like Figure 1 As shown, the system architecture 100 may include a terminal device 101, a network 102 and a server 103. The terminal device 101 may be a laptop 1011, a tablet computer 1012 or a mobile phone 1013. The network 102 is used to provide a medium for a communication link between the terminal device 101 and the server 103. The network 102 may include various connection types, such as wired, wireless communication links or optical fiber cables.

[0038] The user can use the terminal device 101 to interact with the server 103 through the network 102 to receive or send messages, etc. Various communication client applications can be installed on the terminal device 101, such as web browser applications, shopping applications, search applications, instant messaging tools, email clients, social platform software, etc.

[0039] The terminal device 101 can be any electronic device with a display screen and supporting web browsing. In addition to a laptop computer 1011, a tablet computer 1012 or a mobile phone 1013, the terminal device 101 can also be an e-book reader, an MP3 player (Moving Picture Experts Group Audio Layer III), an MP4 (Moving Picture Experts Group Audio Layer IV), a laptop computer, a desktop computer, etc.

[0040] The server 103 may be a server that provides various services, such as a background server that provides support for a web page displayed on the terminal device 101 .

[0041] It should be noted that the metadata-based writing method provided in the embodiment of the present application is generally executed by a server / terminal device, and accordingly, the metadata-based writing device is generally set in the server / terminal device.

[0042] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to implementation requirements.

[0043] Continue to refer Figure 2 , shows a flowchart of an embodiment of a metadata-based writing method according to the present application, comprising the following steps:

[0044] Step S201: receiving a write data request sent by a client, parsing the write data request, and obtaining file metadata of a file to be written.

[0045] This application adopts an improved HDFS (Hadoop Distributed File System) for data writing and storage. Specifically, the registration and discovery of DataNode (DN) services are implemented through a distributed key-value database. When a new DN is registered to the key-value database or an unavailable DN is removed from the key-value database, the NameNode (NN, master node) can update the DN information in the memory in time, so that the client can obtain the DN node information from the NameNode.

[0046] When the client has file data to be written to HDFS for storage, it sends a write data request to HDFS, which carries the file metadata of the file to be written.

[0047] In this embodiment, the electronic device (eg Figure 1 The server / terminal device shown in the figure can receive the write data request sent by the client through a wired connection or a wireless connection. It should be noted that the above wireless connection method may include but is not limited to 3G / 4G / 5G connection, WiFi connection, Bluetooth connection, WiMAX connection, Zigbee connection, UWB (ultra wideband) connection, and other wireless connection methods currently known or to be developed in the future.

[0048] The file metadata of the write data request includes the file name, file identifier, file size, file type, creation time, modification time, user identity, storage path, etc. After receiving the write data request, HDFS parses the write data request to obtain the file metadata of the file to be written.

[0049] In some optional implementations, after the step of obtaining the file metadata of the file to be written, the method further includes:

[0050] Determine whether the file metadata is complete. If the file metadata is incomplete, return an error message to the client, prompting that the file metadata needs to be supplemented;

[0051] If the file metadata is complete, the system determines whether the file to be written is a file type that the system supports based on the file type field in the file metadata. If the file type is not supported by the system, an error message is returned to the client, indicating that the file type does not support processing.

[0052] If the file type is supported by the system, the system determines whether the file to be written exceeds the preset processing file size threshold based on the file size field in the file metadata. If the file exceeds the threshold, an error message is returned to the client, indicating that the file is too large.

[0053] If the threshold is not exceeded, the file identifier field in the file metadata is used to determine whether the file to be written already exists. If it does exist, an error message is returned to the client, indicating that the file to be written already exists.

[0054] If the file to be written does not exist, determine whether the user has the permission to write the file based on the user identity field in the file metadata;

[0055] When the user has write permission, the step of selecting an available target data node from a pre-established node list according to the file metadata is performed.

[0056] Specifically, the key fields of the file metadata are extracted, including the file type, file size, file identifier, user identity, storage path, etc., and whether the key fields are complete is determined according to the preset judgment rules, wherein the preset judgment rules may be a field matching algorithm or a preset regular expression. When the preset judgment rule adopts the field matching algorithm, the matching degree between each extracted key field and each preset key field is calculated. If each preset key field has an extracted key field with a matching degree greater than or equal to a preset threshold, the file metadata is complete, otherwise it is incomplete; when the preset judgment rule is a preset regular expression, matching is performed through the preset regular expression. If there are unmatched characters, it means that the file metadata is incomplete, otherwise, the file metadata is complete.

[0057] If the key field is missing, the file metadata is considered incomplete. If the file metadata information is incomplete, an error message is generated, which includes the name of the missing key field, and the generated error message is sent to the client, prompting the client to complete the file metadata information. If the file metadata is complete, the file type field is parsed and compared with the file type in the list of file types supported by the HDFS system. If it is not in the file type list, an error message is generated, which includes a prompt message that the file type does not support processing. If it is in the file type list, the file size field is parsed again. If the file size of the file to be written exceeds the maximum threshold of the file size set by the system, An error message is generated, which includes a prompt message that the file is too large to be processed; if the file size of the file to be written does not exceed the maximum threshold of the file size set by the system, a hash calculation is performed on the file identifier, and a query is made in the preset metadata database to see whether it already exists. If so, an error message is generated, which includes a prompt message that the file already exists and does not need to be written again; if the file to be written does not exist, the user identity field is parsed, and a query is made in the permission management system to see whether the user has write permission. If not, an error message is returned to the client, which includes a prompt message that the user does not have write permission; if the user has write permission, subsequent data write operations are performed.

[0058] By judging the legitimacy of file metadata, we can ensure the authenticity and integrity of files, improve the efficiency of file management, and ensure the security of file data.

[0059] Step S202: Select an available target data node from a pre-established node list according to the file metadata.

[0060] The pre-established node list is stored in the key-value database in the form of a key-value pair, with the node identifier as the key and the node information as the value. The node information includes node attribute information, node status information, etc., wherein the node information is updated in real time, which can improve the flexibility and scalability of the system. For example, the node attribute information includes disk capacity, memory size, disk usage, I / O performance, etc., and the node status information is the running status of the node, including online, offline, busy, etc.

[0061] In this embodiment, the client sends a write data request to the HDFS system, and can obtain node information of the data node from the master node of the HDFS system. The master node selects a certain number of available data nodes from the node list and returns them to the client.

[0062] In some optional implementations, the step of selecting an available target data node from a pre-established node list according to the file metadata includes:

[0063] Obtain a pre-established node list, and obtain node attribute information and node status information of each data node from the node list;

[0064] Determine available nodes in the node list according to node attribute information and node status information;

[0065] Calculate the required number of nodes based on the data identifier of the file metadata and the number of available nodes;

[0066] Based on the required number of nodes, the corresponding data nodes are selected from the available nodes as the target data nodes.

[0067] Specifically, node attribute information and node status information of all registered data nodes are obtained from the node list, and available nodes are determined according to the node attribute information and node status information. Specifically, it is determined whether the node attribute information and node status information meet the preset threshold conditions. If so, the data node is marked as an available node to obtain a list of available nodes. For example, if the disk space of a data node is close to or reaches saturation, the data node may no longer be able to receive new data blocks or perform read and write operations on data blocks, and the data node is determined to be unavailable; if the disk space of a data node meets the requirements for writing new data blocks, but the I / O performance of the data node, such as slow read and write speeds and long response time, the data node is determined to be unavailable; if the node status information of a data node meets all the requirements, but the data node is in a busy state, the data node is determined to be unavailable.

[0068] A random algorithm is used to select a certain number of target data nodes from the available node list. Specifically, a random number key is generated according to the data identifier inode number of the file metadata. The calculation formula is key = inode% len (remainDNs), and remaining DNs represents available nodes. The random number is used to perform a modulo operation on the available nodes until the operation result is less than or equal to 0, and the final number of operation cycles is output as the required number of nodes. The required number of data nodes are selected from the available node list as the target data nodes.

[0069] Available target data nodes are selected through a random algorithm. The selection of available nodes is even, so that each available node has a balanced load, avoiding hot spots and improving data writing efficiency and performance.

[0070] Step S203: Send the node identification information of the target data node to the client, and the client determines the current data node according to the node identification information.

[0071] In this embodiment, the node identification information corresponding to the target data node is obtained, and the node identification information is transmitted to the client. During the transmission process, the network status is dynamically monitored. When the network status does not meet the preset transmission requirements, the transmission is suspended and the current transmission progress is saved. When the network status recovers to meet the preset transmission requirements, the remaining node identification information is continuously transmitted according to the saved transmission progress. The preset transmission requirements include load level, network delay, transmission distance, link status, etc.

[0072] After the client obtains the node identification information, it selects a target data node as the first data node to receive the data sent by the client, which is the current data node.

[0073] Step S204, receiving the data packet sent by the client through the current data node, and writing the data packet into a preset storage location. After writing is completed, a corresponding response packet is generated and put into a preset queue for processing.

[0074] In this embodiment, when the client obtains the node identification information of the target data node, the data nodes do not transmit data but metadata. The current data node receives the data packet sent by the client, and the current data node directly calls the write function to write the data packet to the preset storage location and generates an Ack response packet corresponding to the data packet.

[0075] It should be emphasized that in order to further ensure the privacy and security of the data packet, the above data packet can also be stored in a blockchain node.

[0076] The blockchain referred to in this application is a new application model of computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanism, encryption algorithm, etc. Blockchain is essentially a decentralized database, a string of data blocks generated by cryptographic methods. Each data block contains a batch of network transaction information, which is used to verify the validity of its information (anti-counterfeiting) and generate the next block. Blockchain can include the underlying blockchain platform, platform product service layer, and application service layer.

[0077] In some optional implementations of this embodiment, the step of receiving a data packet sent by a client through the current data node and writing the data packet into a preset storage location includes:

[0078] A data packet is received through the current data node, and the data packet is parsed to obtain a data packet identifier of the data packet;

[0079] According to the data packet identifier, determine whether the data packet is received repeatedly;

[0080] When the data packet is not received repeatedly, the integrity of the data packet is verified by a verification algorithm;

[0081] When the data packet verification passes, the data packet is written to the preset storage.

[0082] Specifically, a data packet is obtained, and the data packet metadata is obtained, including a data packet identifier, a check code, etc. The data packet identifier is compared to determine whether the data packet is received repeatedly. If so, the data packet is discarded. Otherwise, the integrity of the data packet is verified through a verification algorithm, and the verification result is compared with the extracted verification information. If they are inconsistent, the data packet is considered damaged and discarded. If they are consistent, the data packet is considered intact and undamaged, and a write function is called to write the data packet to the preset storage location of the current data node.

[0083] As a specific example, the integrity of the data packet is verified by a verification algorithm, and the steps of comparing the verification result with the extracted verification information include: using a hash algorithm to verify the extracted slice metadata information, calculating the verification result, and comparing the verification result with the extracted verification code.

[0084] By checking the data packets, we can ensure the correctness and integrity of the received data packets, detect and handle transmission errors in a timely manner, and ensure the reliability of the entire data transmission process, thereby improving the reliability and availability of data.

[0085] After the data packet is written, a corresponding response packet is generated to notify the client that the data has been successfully written. In this embodiment, the generated response packet is placed in a preset pending queue (PendingQueue) and waits for all data nodes to complete synchronization before returning to the client.

[0086] Step S205, calling a preset callback function through the current data node to obtain the data packet length and slice metadata information of the data packet, and synchronizing the slice metadata information to other nodes through a preset distributed coordination service, wherein the other nodes are target data nodes other than the current data node.

[0087] In this embodiment, after the data packet is written, the current data node calls a preset callback function to obtain the data packet length and slice metadata information of the data packet, where the data packet length is the length of the written data; the slice metadata information includes the slice identifier, file metadata, slice position, slice size, etc. Then, the slice metadata is synchronized to other nodes to achieve the transmission of only metadata information between data nodes to ensure data consistency.

[0088] In some optional implementations, the step of synchronizing the slice metadata information to other nodes through a preset distributed coordination service includes:

[0089] Encapsulate the slice metadata information to obtain the encapsulated synchronized metadata;

[0090] Establish a communication connection between the current data node and other nodes through the distributed coordination service. After the connection is successfully established, send the synchronization metadata to other nodes;

[0091] In response to the update results of other nodes updating according to the synchronization metadata, the current data node determines whether the slice metadata is successfully synchronized;

[0092] If the synchronization fails, re-acquire the slice metadata information and initiate the synchronization request again until the synchronization succeeds or the maximum number of retries is reached.

[0093] In this embodiment, the slice metadata information is encapsulated according to the preset structure format to obtain the encapsulated synchronization metadata, and the communication connection between the current data node and other nodes is established through the preset distributed coordination service. After the connection is successfully established, the synchronization metadata is synchronized to other nodes. After receiving the synchronization metadata, other nodes parse the synchronization metadata to obtain the slice metadata information, and update the node's slice metadata according to the metadata information. After the update is completed, the update result is returned to the current data node. The current data node determines whether the slice metadata is synchronized successfully according to the returned update result. If the synchronization is successful, the subsequent process is executed. If the synchronization fails, the current data node re-acquires the slice metadata information and initiates a synchronization request to other nodes again until the synchronization is successful or the maximum number of retries is reached. Abnormal situations in the synchronization process are captured and processed, error logs are recorded, and an alarm mechanism is triggered to notify the administrator to perform manual intervention and troubleshooting.

[0094] The preset distributed coordination service refers to a service for unified coordination management and scheduling of collaborative work between nodes to ensure the consistency and reliability of the system. As a specific example, the preset distributed coordination service is implemented through a distributed key-value pair storage system.

[0095] By synchronizing slice metadata information to other nodes, only metadata is transmitted between data nodes, ensuring data consistency while improving system data writing performance, reducing large amounts of data transmission between data nodes, and improving the efficiency and timeliness of data writing.

[0096] Step S206, through the current data node, according to the length of the data packet, the corresponding response packet is taken out from the queue to be processed, put into the preset response queue, and returned to the client.

[0097] Specifically, through the current data node, the number of response packets taken out from the queue to be processed is determined according to the length of the data packet; according to the number of response packets, starting from the head of the queue to be processed, a corresponding number of response packets are continuously taken out as target response packets; the target response packets are placed in the preset response queue; according to the write data request, the target response packets are taken out from the preset response queue, and assembled into response data, and returned to the client.

[0098] Specifically, the data packet length corresponding to the response packet in the queue to be processed is obtained in sequence, starting from the queue head of the queue to be processed, and the data packet length corresponding to the response packet is accumulated in sequence. In this process, it is checked whether the accumulated length exceeds the value of the current data packet length. If not, the data packet length of the next response packet is accumulated. If it exceeds, the accumulation is stopped, and all the previously accumulated responses are selected and put into the preset response queue AckQ, and returned to the client. Among them, the current data packet length is the data packet length of the data packet written by the current data node.

[0099] For example, assuming that the current data packet length is 9, and the data packet lengths corresponding to the response packets in the queue to be processed are 3, 5, 2, and 1, respectively, then 3+5≤9 is obtained, and the first two response packets are sent to AckQ and returned to the client.

[0100] By placing the response packet into a preset response queue and returning it to the client, it is possible to optimize data transmission efficiency, reduce network load, quickly respond to client requests, improve the real-time performance of data processing, enhance system flexibility and scalability, and improve resource utilization.

[0101] The present application selects available target data nodes for data writing, thereby avoiding excessive pressure on a single data node when a large number of files are written, improving data reliability and fault tolerance, increasing the parallelism and overall efficiency of data access, and optimizing resource utilization; the data packet is written to the preset storage through the current data node, and the preset callback function is called to synchronize the slice metadata information of the data packet to other nodes through the preset distributed coordination service. Only metadata information is transmitted at the data node, which reduces a large amount of network data transmission between data nodes. While maintaining data uniformity, it improves the system's data writing performance and improves the efficiency and timeliness of data writing.

[0102] In some optional implementations, after the step of taking out the corresponding response packet from the queue to be processed according to the length of the data packet and putting it into the preset response queue and returning it to the client, the method further includes:

[0103] If a fault is detected in the current data node, the fault handling process is triggered, the slice metadata information of the data packet is obtained through other nodes, the slice metadata information is written into the preset metadata storage, and the corresponding response packet is generated.

[0104] In this embodiment, the current data node fails, indicating that the slice data of the data packet has been written, but the slice metadata has not been saved successfully. If the previous data node fails, the fault handling process is triggered. When the client sends a write data request to other nodes, the other nodes write the slice metadata information of the previously written file to the preset metadata storage, and generate a corresponding response packet and return it to the client, so that the response packet received by the client actually writes the data and metadata of the file to be written.

[0105] Through the fault handling process, it can be ensured that when a data node fails, data can continue to be read and written from other data nodes, ensuring data consistency.

[0106] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through computer-readable instructions, and the computer-readable instructions can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, the aforementioned storage medium can be a non-volatile storage medium such as a disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM).

[0107] It should be understood that, although the steps in the flowchart of the accompanying drawings are displayed in sequence as indicated by the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least a part of the steps in the flowchart of the accompanying drawings may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be executed in turn or alternately with other steps or at least a part of the sub-steps or stages of other steps.

[0108] Further references Figure 3 , as a response to the above Figure 2 The present application provides an embodiment of a metadata-based writing device, and the device embodiment is similar to Figure 2 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.

[0109] like Figure 3 As shown, the metadata-based writing device 300 described in this embodiment includes: a parsing module 301, a selection module 302, a determination module 303, a writing module 304, a synchronization module 305 and a response module 306. Among them:

[0110] The parsing module 301 is used to receive a write data request sent by a client, parse the write data request, and obtain file metadata of a file to be written;

[0111] The selection module 302 is used to select an available target data node from a pre-established node list according to the file metadata;

[0112] The determination module 303 is used to send the node identification information of the target data node to the client, and the client determines the current data node according to the node identification information;

[0113] The writing module 304 is used to receive the data packet sent by the client through the current data node, and write the data packet into a preset storage location. After the writing is completed, a corresponding response packet is generated and put into a preset queue for processing;

[0114] The synchronization module 305 is used to call a preset callback function through the current data node to obtain the data packet length and slice metadata information of the data packet, and synchronize the slice metadata information to other nodes through a preset distributed coordination service, and the other nodes are target data nodes other than the current data node;

[0115] The response module 306 is used to take out the corresponding response packet from the queue to be processed according to the length of the data packet through the current data node, put it into the preset response queue, and return it to the client.

[0116] It should be emphasized that in order to further ensure the privacy and security of the data packet, the above data packet can also be stored in a blockchain node.

[0117] The metadata-based writing device 300 of the present application can avoid excessive pressure on a single data node when a large number of files are written by selecting available target data nodes for data writing, improve data reliability and fault tolerance, improve the parallelism and overall efficiency of data access, and optimize resource utilization; write the data packet to the preset storage through the current data node, and call the preset callback function to synchronize the slice metadata information of the data packet to other nodes through the preset distributed coordination service. Only metadata information is transmitted at the data node, which reduces a large amount of network data transmission between data nodes. While maintaining data uniformity, it improves the system's data writing performance and improves the efficiency and timeliness of data writing.

[0118] In some optional implementations, the metadata-based writing device 300 further includes a verification module, which is used to:

[0119] Determine whether the file metadata is complete, and if the file metadata is incomplete, return an error message to the client, prompting that the file metadata needs to be supplemented;

[0120] If the file metadata is complete, judging whether the file to be written is a file type supported by the system according to the file type field in the file metadata; if the file type is a file type not supported by the system, returning an error message to the client, indicating that the file type does not support processing;

[0121] If the file type is supported by the system, then judging whether the file to be written exceeds a preset processing file size threshold according to the file size field in the file metadata, and if it exceeds the threshold, returning an error message to the client, indicating that the file is too large;

[0122] If the threshold is not exceeded, judging whether the file to be written already exists according to the file identification field in the file metadata, and if it does exist, returning an error message to the client to prompt that the file to be written already exists;

[0123] If the file to be written does not exist, judging whether the user has the permission to write the file according to the user identity field in the file metadata;

[0124] When the user has write permission, the step of selecting an available target data node from a pre-established node list according to the file metadata is performed.

[0125] By judging the legitimacy of file metadata, we can ensure the authenticity and integrity of files, improve the efficiency of file management, and ensure the security of file data.

[0126] In some optional implementations, the selection module 302 is further configured to:

[0127] Obtain a pre-established node list, and obtain node attribute information and node status information of each data node from the node list;

[0128] Determine the available nodes in the node list according to the node attribute information and the node status information;

[0129] Calculate the required number of nodes according to the data identifier of the file metadata and the available nodes;

[0130] Based on the required number of nodes, corresponding data nodes are selected from the available nodes as target data nodes.

[0131] Available target data nodes are selected through a random algorithm. The selection of available nodes is even, so that each available node has a balanced load, avoiding hot spots and improving data writing efficiency and performance.

[0132] In some optional implementations, the writing module 304 includes:

[0133] A parsing submodule, configured to receive the data packet through the current data node, and parse the data packet to obtain a data packet identifier of the data packet;

[0134] A judgment submodule, used for judging whether the data packet is received repeatedly according to the data packet identifier;

[0135] A check submodule, used for checking the integrity of the data packet by using a check algorithm when the data packet is not received repeatedly;

[0136] The writing submodule is used to write the data packet into a preset storage location when the data packet passes the verification.

[0137] By checking the data packets, we can ensure the correctness and integrity of the received data packets, detect and handle transmission errors in a timely manner, and ensure the reliability of the entire data transmission process, thereby improving the reliability and availability of data.

[0138] In some optional implementations, the synchronization module 305 is further configured to:

[0139] Encapsulating the slice metadata information to obtain encapsulated synchronization metadata;

[0140] Establishing a communication connection between the current data node and other nodes through a distributed coordination service, and after the connection is successfully established, sending the synchronization metadata to the other nodes;

[0141] In response to an update result of the other nodes being updated according to the synchronization metadata, determining, through the current data node, whether the slice metadata is successfully synchronized;

[0142] If the synchronization fails, re-acquire the slice metadata information and initiate the synchronization request again until the synchronization succeeds or the maximum number of retries is reached.

[0143] By synchronizing slice metadata information to other nodes, only metadata is transmitted between data nodes, ensuring data consistency while improving system data writing performance, reducing large amounts of data transmission between data nodes, and improving the efficiency and timeliness of data writing.

[0144] In some optional implementations of this embodiment, the response module 306 is further configured to:

[0145] Determining the number of response packets taken out from the queue to be processed according to the length of the data packet;

[0146] According to the number of response packets, starting from the head of the queue to be processed, continuously taking out a corresponding number of response packets as target response packets;

[0147] Putting the target response packet into the preset response queue;

[0148] According to the write data request, the target response packet is taken out from the preset response queue, assembled into response data, and returned to the client.

[0149] By placing the response packet into a preset response queue and returning it to the client, it is possible to optimize data transmission efficiency, reduce network load, quickly respond to client requests, improve the real-time performance of data processing, enhance system flexibility and scalability, and improve resource utilization.

[0150] In some optional implementations, the metadata-based writing device 300 also includes a fault handling module, which is used to: if it is detected that the current data node fails, trigger a fault handling process, obtain the slice metadata information of the data packet through the other nodes, write the slice metadata information into the preset metadata storage, and generate a corresponding response packet.

[0151] Through the fault handling process, it can be ensured that when a data node fails, data can continue to be read and written from other data nodes, ensuring data consistency.

[0152] To solve the above technical problems, the present application also provides a computer device. Figure 4 , Figure 4 This is a basic structural block diagram of the computer device in this embodiment.

[0153] The computer device 4 includes a memory 41, a processor 42, and a network interface 43 that are interconnected and communicated through a system bus. It should be noted that the figure only shows a computer device 4 with a memory 41, a processor 42, and a network interface 43, but it should be understood that it is not required to implement all the components shown, and more or fewer components can be implemented instead. Among them, those skilled in the art can understand that the computer device here is a device that can automatically perform numerical calculations and / or information processing according to pre-set or stored instructions, and its hardware includes but is not limited to microprocessors, application specific integrated circuits (Application Specific Integrated Circuit, ASIC), programmable gate arrays (Field-Programmable Gate Array, FPGA), digital processors (Digital Signal Processor, DSP), embedded devices, etc.

[0154] The computer device may be a computing device such as a desktop computer, a notebook, a PDA, a cloud server, etc. The computer device may interact with a user through a keyboard, a mouse, a remote controller, a touch pad, or a voice control device.

[0155] The memory 41 includes at least one type of readable storage medium, and the readable storage medium includes flash memory, hard disk, multimedia card, card-type memory (for example, SD or DX memory, etc.), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, disk, optical disk, etc. In some embodiments, the memory 41 can be an internal storage unit of the computer device 4, such as a hard disk or memory of the computer device 4. In other embodiments, the memory 41 can also be an external storage device of the computer device 4, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (FlashCard), etc. equipped on the computer device 4. Of course, the memory 41 can also include both the internal storage unit of the computer device 4 and its external storage device. In this embodiment, the memory 41 is generally used to store the operating system and various application software installed on the computer device 4, such as computer-readable instructions based on the metadata writing method, etc. In addition, the memory 41 can also be used to temporarily store various types of data that have been output or are to be output.

[0156] The processor 42 may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chips in some embodiments. The processor 42 is generally used to control the overall operation of the computer device 4. In this embodiment, the processor 42 is used to run the computer-readable instructions stored in the memory 41 or process data, such as running the computer-readable instructions of the metadata-based writing method.

[0157] The network interface 43 may include a wireless network interface or a wired network interface. The network interface 43 is generally used to establish a communication connection between the computer device 4 and other electronic devices.

[0158] By selecting available target data nodes for data writing, it is possible to avoid excessive pressure on a single data node when writing a large number of files, improve data reliability and fault tolerance, improve the parallelism and overall efficiency of data access, and optimize resource utilization; write the data packet to the preset storage through the current data node, and call the preset callback function to synchronize the slice metadata information of the data packet to other nodes through the preset distributed coordination service. Only metadata information is transmitted at the data node, reducing a large amount of network data transmission between data nodes. While maintaining data uniformity, it improves the system's data writing performance and improves the efficiency and timeliness of data writing.

[0159] The present application also provides another embodiment, namely, providing a computer-readable storage medium, wherein the computer-readable storage medium stores computer-readable instructions, and the computer-readable instructions can be executed by at least one processor to enable the at least one processor to perform the steps of the metadata-based writing method as described above.

[0160] By selecting available target data nodes for data writing, it is possible to avoid excessive pressure on a single data node when writing a large number of files, improve data reliability and fault tolerance, improve the parallelism and overall efficiency of data access, and optimize resource utilization; write the data packet to the preset storage through the current data node, and call the preset callback function to synchronize the slice metadata information of the data packet to other nodes through the preset distributed coordination service. Only metadata information is transmitted at the data node, reducing a large amount of network data transmission between data nodes. While maintaining data uniformity, it improves the system's data writing performance and improves the efficiency and timeliness of data writing.

[0161] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in each embodiment of the present application.

[0162] Obviously, the embodiments described above are only some embodiments of the present application, rather than all embodiments. The preferred embodiments of the present application are given in the accompanying drawings, but they do not limit the patent scope of the present application. The present application can be implemented in many different forms. On the contrary, the purpose of providing these embodiments is to make the understanding of the disclosure of the present application more thorough and comprehensive. Although the present application is described in detail with reference to the aforementioned embodiments, for those skilled in the art, it is still possible to modify the technical solutions recorded in the aforementioned specific implementation methods, or to perform equivalent replacement of some of the technical features therein. Any equivalent structure made using the contents of the specification and drawings of this application, directly or indirectly used in other related technical fields, is similarly within the scope of patent protection of this application.

Claims

1. A metadata-based writing method, characterized in that: The steps include: Receiving a write data request sent by a client, parsing the write data request, and obtaining file metadata of a file to be written; Selecting an available target data node from a pre-established node list according to the file metadata; Sending the node identification information of the target data node to the client, and determining the current data node by the client according to the node identification information; The data packet sent by the client is received through the current data node, and the data packet is written into a preset storage location. After the writing is completed, a corresponding response packet is generated and put into a preset queue for processing; Calling a preset callback function through the current data node to obtain the data packet length and slice metadata information of the data packet, and synchronizing the slice metadata information to other nodes through a preset distributed coordination service, wherein the other nodes are target data nodes other than the current data node; Through the current data node, the corresponding response packet is taken out from the queue to be processed according to the length of the data packet, put into the preset response queue, and returned to the client.

2. The metadata-based writing method according to claim 1, characterized in that: After the step of obtaining the file metadata of the file to be written, the method further includes: Determine whether the file metadata is complete, and if the file metadata is incomplete, return an error message to the client, prompting that the file metadata needs to be supplemented; If the file metadata is complete, judging whether the file to be written is a file type supported by the system according to the file type field in the file metadata; if the file type is a file type not supported by the system, returning an error message to the client, indicating that the file type does not support processing; If the file type is supported by the system, then judging whether the file to be written exceeds a preset processing file size threshold according to the file size field in the file metadata, and if it exceeds the threshold, returning an error message to the client, indicating that the file is too large; If the threshold is not exceeded, judging whether the file to be written already exists according to the file identification field in the file metadata, and if it does exist, returning an error message to the client to prompt that the file to be written already exists; If the file to be written does not exist, judging whether the user has the permission to write the file according to the user identity field in the file metadata; When the user has write permission, the step of selecting an available target data node from a pre-established node list according to the file metadata is performed.

3. The metadata-based writing method according to claim 1, characterized in that: The step of selecting an available target data node from a pre-established node list according to the file metadata comprises: Obtain a pre-established node list, and obtain node attribute information and node status information of each data node from the node list; Determine the available nodes in the node list according to the node attribute information and the node status information; Calculate the required number of nodes according to the data identifier of the file metadata and the available nodes; Based on the required number of nodes, corresponding data nodes are selected from the available nodes as target data nodes.

4. The metadata-based writing method according to claim 1, characterized in that: The step of receiving the data packet sent by the client through the current data node and writing the data packet into a preset storage location includes: The data packet is received through the current data node, and the data packet is parsed to obtain a data packet identifier of the data packet; Determining, according to the data packet identifier, whether the data packet is received repeatedly; When the data packet is not received repeatedly, verifying the integrity of the data packet by using a verification algorithm; When the data packet passes the verification, the data packet is written into a preset storage location.

5. The metadata-based writing method according to claim 1, characterized in that: The step of synchronizing the slice metadata information to other nodes through a preset distributed coordination service includes: Encapsulating the slice metadata information to obtain encapsulated synchronization metadata; Establishing a communication connection between the current data node and other nodes through a distributed coordination service, and after the connection is successfully established, sending the synchronization metadata to the other nodes; In response to an update result of the other nodes being updated according to the synchronization metadata, determining whether the slice metadata is successfully synchronized through the current data node; If the synchronization fails, re-acquire the slice metadata information and initiate the synchronization request again until the synchronization succeeds or the maximum number of retries is reached.

6. The metadata-based writing method according to claim 1, characterized in that: The step of taking out a corresponding response packet from the queue to be processed according to the length of the data packet and putting it into a preset response queue and returning it to the client comprises: Determining the number of response packets taken out from the queue to be processed according to the length of the data packet; According to the number of response packets, starting from the head of the queue to be processed, continuously taking out a corresponding number of response packets as target response packets; Putting the target response packet into the preset response queue; According to the write data request, the target response packet is taken out from the preset response queue, assembled into response data, and returned to the client.

7. The metadata-based writing method according to claim 1, characterized in that: After the step of taking out the corresponding response packet from the queue to be processed according to the length of the data packet and putting it into the preset response queue and returning it to the client, the step further includes: If it is detected that the current data node fails, the fault handling process is triggered, the slice metadata information of the data packet is obtained through the other nodes, the slice metadata information is written into the preset metadata storage, and a corresponding response packet is generated.

8. A metadata-based writing device, characterized in that: include: The parsing module is used to receive a write data request sent by a client, parse the write data request, and obtain file metadata of a file to be written; A selection module, used to select an available target data node from a pre-established node list according to the file metadata; A determination module, configured to send the node identification information of the target data node to the client, and determine the current data node according to the node identification information by the client; A writing module is used to receive the data packet sent by the client through the current data node, and write the data packet into a preset storage location. After writing is completed, a corresponding response packet is generated and put into a preset queue for processing; A synchronization module, used to call a preset callback function through the current data node to obtain the data packet length and slice metadata information of the data packet, and synchronize the slice metadata information to other nodes through a preset distributed coordination service, wherein the other nodes are target data nodes other than the current data node; The response module is used to take out the corresponding response packet from the queue to be processed according to the length of the data packet through the current data node, put it into the preset response queue, and return it to the client.

9. A computer device, characterized in that: The method comprises a memory and a processor, wherein the memory stores computer-readable instructions, and the processor implements the steps of the metadata-based writing method as claimed in any one of claims 1 to 7 when executing the computer-readable instructions.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-readable instructions, and when the computer-readable instructions are executed by a processor, the steps of the metadata-based writing method according to any one of claims 1 to 7 are implemented.

Images