Metadata-based writing method, device, computer equipment, and storage medium

Abstract

The present application belongs to the field of data processing and finance, and relates to a metadata-based writing method, including parsing a write data request sent by a client to obtain file metadata, selecting a target data node from a node list based on the file metadata, the client determining the current data node based on the node identification information, the current data node writing the data packet sent by the client to a preset storage location, generating a corresponding response packet and placing it into a preset queue to be processed, calling a callback function to obtain the data packet length and slice metadata information, synchronizing the slice metadata information to other nodes, taking out the response packet from the queue to be processed based on the data packet length, placing it into a preset response queue and returning it to the client. The present application also provides a metadata-based writing device, computer equipment, and storage medium. In addition, the present application also relates to blockchain technology, and data packets can be stored in a blockchain. The present application transmits metadata at data nodes to improve the efficiency and timeliness of data writing.

Description

Technical Field

[0001] The present application relates to the fields of data processing and financial technology, and in particular to a metadata-based writing method, apparatus, computer equipment, and storage medium. Background Art

[0002] In the financial sector, with the rapid development of big data technologies, processing and analyzing massive amounts of data has become crucial for financial institutions to improve business efficiency, optimize decision-making processes, and enhance risk management capabilities. As one of the core frameworks for big data processing, the Hadoop ecosystem has gained widespread adoption in the financial industry due to its high scalability, high fault tolerance, and efficient data processing capabilities. The Hadoop Distributed File System (HDFS), a foundational component of the Hadoop ecosystem, carries the heavy responsibility of storing and accessing big data.

[0003] HDFS is designed specifically to handle large-scale data sets, offering high fault tolerance and optimized for high-throughput data access. This enables HDFS to run stably on commodity hardware and effectively support the complex and massive data processing needs of the financial sector. The HDFS system architecture primarily consists of two types of nodes: NameNode and DataNode. Each node performs its own specific functions and works together to maintain the stable operation of the entire file system.

[0004] The NameNode (NN), as the master node of HDFS, plays a crucial role. It manages the file system's namespace and the mapping between files and blocks, storing metadata for all files and directories, such as file names, permissions, and block locations. This metadata is crucial for ensuring correct data access and efficient management.

[0005] DataNodes (DNs) are HDFS worker nodes responsible for actual data storage. Each DataNode stores a certain number of data blocks, which are the basic units of HDFS data storage. DataNodes receive data from clients or other DataNodes and write it to their local disk, thus achieving distributed data storage.

[0006] In HDFS, data is written in a streaming manner. When a client writes data to HDFS, the data flows sequentially through multiple DataNodes. After receiving the data, each DataNode returns an Ack message to the client or upstream DataNode, indicating that the node has successfully received and stored the data. To ensure data reliability and fault tolerance, HDFS typically writes data to three different DataNodes. Only after all three nodes have returned Ack messages does the client receive the final Ack confirmation indicating that the data has been successfully written.

[0007] However, in practice, this data writing mechanism also introduces certain performance losses. Since actual data and reply messages must be transmitted across the network between three DN nodes, this not only increases data transmission latency but can also be affected by network conditions, making it challenging to write data in a timely manner. This is especially true in the financial sector, where data processing and response speeds are extremely demanding. Any delay can negatively impact business decisions and risk control. Summary of the Invention

[0008] The purpose of the embodiments of the present application is to propose a metadata-based writing method, apparatus, computer device, and storage medium to solve the technical problems of data transmission delay and low writing efficiency in the data writing mechanism of the related art.

[0009] In order to solve the above technical problems, the embodiment of the present application provides a metadata-based writing method, which adopts the following technical solutions:

[0010] Receive a write data request sent by a client, parse the write data request, and obtain file metadata of the file to be written;

[0011] Selecting an available target data node from a pre-established node list according to the file metadata;

[0012] Sending the node identification information of the target data node to the client, and determining the current data node by the client according to the node identification information;

[0013] Receive the data packet sent by the client through the current data node, and write the data packet into a preset storage location. After writing is completed, generate a corresponding response packet and put it into a preset queue for processing;

[0014] Calling a preset callback function through the current data node to obtain the data packet length and slice metadata information of the data packet, and synchronizing the slice metadata information to other nodes through a preset distributed coordination service, where the other nodes are target data nodes other than the current data node;

[0015] Through the current data node, the corresponding response packet is taken out from the queue to be processed according to the length of the data packet, put into the preset response queue, and returned to the client.

[0016] In order to solve the above technical problems, the embodiment of the present application further provides a metadata-based writing device, which adopts the following technical solution:

[0017] A parsing module is configured to receive a write data request sent by a client, parse the write data request, and obtain file metadata of a file to be written;

[0018] A selection module, configured to select an available target data node from a pre-established node list according to the file metadata;

[0019] a determination module, configured to send the node identification information of the target data node to the client, and determine the current data node through the client according to the node identification information;

[0020] A writing module is used to receive the data packet sent by the client through the current data node and write the data packet into a preset storage location. After writing is completed, a corresponding response packet is generated and placed into a preset queue for processing;

[0021] a synchronization module, configured to call a preset callback function through the current data node to obtain the data packet length and slice metadata information of the data packet, and synchronize the slice metadata information to other nodes through a preset distributed coordination service, where the other nodes are target data nodes other than the current data node;

[0022] The response module is used to take out the corresponding response packet from the queue to be processed according to the length of the data packet through the current data node, put it into the preset response queue, and return it to the client.

[0023] In order to solve the above technical problems, the embodiment of the present application further provides a computer device, which adopts the following technical solution:

[0024] The computer device includes a memory and a processor, wherein the memory stores computer-readable instructions, and the processor implements the steps of the metadata-based writing method described above when executing the computer-readable instructions.

[0025] In order to solve the above technical problems, the embodiment of the present application further provides a computer-readable storage medium, which adopts the following technical solution:

[0026] The computer-readable storage medium stores computer-readable instructions, which, when executed by a processor, implement the steps of the metadata-based writing method described above.

[0027] Compared with the prior art, this application has the following beneficial effects:

[0028] The present application provides a metadata-based writing method, which obtains the file metadata of the file to be written by parsing the write data request sent by the client, selects the target data node from the pre-established node list according to the file metadata, and sends the node identification information of the target data node to the client. The client determines the current data node according to the node identification information and sends the data packet to the current data node. The current data node receives the data packet and writes the data packet to a preset storage location. After the writing is completed, a corresponding response packet is generated and put into a preset queue for processing. The preset callback function is called to obtain the data packet length and slice metadata information of the data packet, and the slice metadata information is synchronized to other nodes. The current data node takes the corresponding data packet from the queue for processing according to the data packet length. The response packet is placed in the preset response queue and returned to the client through the preset response queue to notify the client that the data writing is completed; this application selects available target data nodes for data writing, which can avoid excessive pressure on a single data node when a large number of files are written, improve data reliability and fault tolerance, improve the parallelism and overall efficiency of data access, and optimize resource utilization; the data packet is written to the preset storage location through the current data node, and the preset callback function is called to synchronize the slice metadata information of the data packet to other nodes through the preset distributed coordination service. Only metadata information is transmitted at the data node, which reduces a large amount of network data transmission between data nodes. While maintaining data uniformity, it improves the system's data writing performance and improves the efficiency and timeliness of data writing. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] In order to more clearly illustrate the solutions in this application, a brief introduction will be given below to the drawings required for use in the description of the embodiments of this application. Obviously, the drawings described below are some embodiments of this application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0030] Figure 1 is an exemplary system architecture diagram to which the present application may be applied;

[0031] Figure 2 is a flowchart of an embodiment of a metadata-based writing method according to the present application;

[0032] Figure 3 is a structural diagram of an embodiment of a metadata-based writing device according to the present application;

[0033] Figure 4 It is a structural diagram of an embodiment of a computer device according to the present application. DETAILED DESCRIPTION

[0034] Unless otherwise defined, all technical and scientific terms used herein have the same meanings as commonly understood by those skilled in the art to which this application belongs. The terms used in the specification of the application are for the purpose of describing specific embodiments only and are not intended to limit this application. The terms "including" and "having" and any variations thereof in the specification and claims of this application and the above-mentioned drawings are intended to cover non-exclusive inclusions. The terms "first", "second", etc. in the specification and claims of this application or the above-mentioned drawings are used to distinguish different objects, not to describe a specific order.

[0035] References herein to "embodiments" mean that a particular feature, structure, or characteristic described in connection with the embodiments may be included in at least one embodiment of the present application. The appearance of this phrase in various places in the specification does not necessarily refer to the same embodiment, nor does it constitute an independent or alternative embodiment that is mutually exclusive of other embodiments. It is understood, both explicitly and implicitly, by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0036] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the accompanying drawings.

[0037] like Figure 1 As shown, system architecture 100 may include a terminal device 101, a network 102, and a server 103. Terminal device 101 may be a laptop computer 1011, a tablet computer 1012, or a mobile phone 1013. Network 102 is a medium for providing a communication link between terminal device 101 and server 103. Network 102 may include various connection types, such as wired or wireless communication links or fiber optic cables.

[0038] The user can use the terminal device 101 to interact with the server 103 via the network 102 to receive or send messages, etc. Various communication client applications can be installed on the terminal device 101, such as web browser applications, shopping applications, search applications, instant messaging tools, email clients, social platform software, etc.

[0039] The terminal device 101 can be various electronic devices with a display screen and supporting web browsing. In addition to the laptop computer 1011, tablet computer 1012 or mobile phone 1013, the terminal device 101 can also be an e-book reader, an MP3 player (Moving Picture Experts Group Audio Layer III), an MP4 (Moving Picture Experts Group Audio Layer IV) player, a laptop computer and a desktop computer, etc.

[0040] The server 103 may be a server that provides various services, such as a background server that provides support for web pages displayed on the terminal device 101 .

[0041] It should be noted that the metadata-based writing method provided in the embodiment of the present application is generally executed by a server / terminal device, and accordingly, the metadata-based writing device is generally set in the server / terminal device.

[0042] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is merely illustrative. Any number of terminal devices, networks and servers may be provided as required.

[0043] Continue to refer Figure 2 , shows a flowchart of an embodiment of a metadata-based writing method according to the present application, comprising the following steps:

[0044] Step S201: receiving a write data request sent by a client, parsing the write data request, and obtaining file metadata of a file to be written.

[0045] This application adopts an improved HDFS (Hadoop Distributed File System) for data writing and storage. Specifically, the registration and discovery of DataNode (DN) services are implemented through a distributed key-value database. When a new DN is registered to the key-value database or an unavailable DN is removed from the key-value database, the NameNode (NN, master node) can promptly update the DN information in the memory, making it easier for the client to obtain DN node information from the NameNode.

[0046] When the client has file data to be written to HDFS for storage, it sends a write data request to HDFS. The write data request carries the file metadata of the file to be written.

[0047] In this embodiment, the electronic device (eg Figure 1 The server / terminal device shown in the figure can receive the write data request sent by the client through a wired connection or a wireless connection. It should be noted that the above-mentioned wireless connection method may include but is not limited to 3G / 4G / 5G connection, WiFi connection, Bluetooth connection, WiMAX connection, Zigbee connection, UWB (ultra wideband) connection, and other wireless connection methods currently known or to be developed in the future.

[0048] The file metadata of a write request includes the file name, file identifier, file size, file type, creation time, modification time, user identity, storage path, etc. After receiving a write request, HDFS parses the request to obtain the file metadata of the file to be written.

[0049] In some optional implementations, after the step of obtaining the file metadata of the file to be written, the method further includes:

[0050] Determine whether the file metadata is complete. If the file metadata is incomplete, return an error message to the client, prompting the client to complete the file metadata.

[0051] If the file metadata is complete, the system determines whether the file to be written is a file type supported by the system based on the file type field in the file metadata. If the file type is not supported by the system, an error message is returned to the client, indicating that the file type cannot be processed.

[0052] If the file type is supported by the system, the system will determine whether the file to be written exceeds the preset processing file size threshold based on the file size field in the file metadata. If it exceeds the threshold, an error message will be returned to the client, indicating that the file is too large.

[0053] If the threshold is not exceeded, the file identifier field in the file metadata is used to determine whether the file to be written already exists. If so, an error message is returned to the client, indicating that the file to be written already exists.

[0054] If the file to be written does not exist, the user's permission to write the file is determined based on the user identity field in the file metadata.

[0055] When the user has write permission, the step of selecting an available target data node from a pre-established node list according to the file metadata is performed.

[0056] Specifically, the key fields of the file metadata are extracted, including the file type, file size, file identifier, user identity, storage path, etc., and the completeness of the key fields is determined according to the preset judgment rules, wherein the preset judgment rules can be a field matching algorithm or a preset regular expression. When the preset judgment rule adopts the field matching algorithm, the matching degree between each extracted key field and each preset key field is calculated. If each preset key field has an extracted key field with a matching degree greater than or equal to a preset threshold, the file metadata is complete; otherwise, it is incomplete. When the preset judgment rule is a preset regular expression, matching is performed using the preset regular expression. If there are unmatched characters, it means that the file metadata is incomplete. Otherwise, the file metadata is complete.

[0057] If the key field is missing, the file metadata is considered incomplete. If the file metadata information is incomplete, an error message is generated, which includes the name of the missing key field, and the generated error message is sent to the client, prompting the client to complete the file metadata information. If the file metadata is complete, the file type field is parsed and compared with the file type in the list of file types supported by the HDFS system. If it is not in the file type list, an error message is generated, which includes a prompt message that the file type does not support processing. If it is in the file type list, the file size field is parsed. If the file size of the file to be written exceeds the maximum threshold of the processing file size set by the system, An error message is generated, which includes a prompt message that the file is too large and cannot be processed; if the file size of the file to be written does not exceed the maximum threshold of the file size processed by the system, the file identifier is hashed and the preset metadata database is checked to see if it already exists. If it does, an error message is generated, which includes a prompt message that the file already exists and does not need to be written again; if the file to be written does not exist, the user identity field is parsed, and the user is queried in the permission management system to see if he has write permission. If he does not have permission, an error message is returned to the client, which includes a prompt message that the user does not have write permission; if the user has write permission, subsequent data write operations are performed.

[0058] By judging the legitimacy of file metadata, we can ensure the authenticity and integrity of files, improve the efficiency of file management, and ensure the security of file data.

[0059] Step S202: Select an available target data node from a pre-established node list according to the file metadata.

[0060] The pre-established node list is stored in a key-value database as a key-value pair, with the node identifier as the key and node information as the value. Node information includes node attribute information and node status information. Node information is updated in real time, improving system flexibility and scalability. For example, node attribute information includes disk capacity, memory size, disk usage, and I / O performance. Node status information indicates the node's operating status, including online, offline, and busy.

[0061] In this embodiment, the client sends a write request to the HDFS system, which can obtain node information of the data nodes from the master node of the HDFS system. The master node selects a certain number of available data nodes from the node list and returns them to the client.

[0062] In some optional implementations, the step of selecting an available target data node from a pre-established node list based on the file metadata includes:

[0063] Obtain a pre-established node list, and obtain node attribute information and node status information of each data node from the node list;

[0064] Determine available nodes in the node list according to node attribute information and node status information;

[0065] Calculate the required number of nodes based on the data identifier of the file metadata and the number of available nodes;

[0066] Based on the required number of nodes, the corresponding data node is selected from the available nodes as the target data node.

[0067] Specifically, the node attribute information and node status information of all registered data nodes are obtained from the node list, and the available nodes are determined based on the node attribute information and node status information. Specifically, it is determined whether the node attribute information and node status information meet the preset threshold conditions. If so, the data node is marked as an available node to obtain a list of available nodes. For example, if the disk space of a data node is close to or reaches saturation, the data node may no longer be able to receive new data blocks or perform read and write operations on data blocks, and the data node is determined to be unavailable; if the disk space of a data node meets the requirements for writing new data blocks, but the I / O performance of the data node, such as slow read and write speeds and long response time, the data node is determined to be unavailable; if the node status information of a data node meets all the requirements, but the data node is in a busy state, the data node is determined to be unavailable.

[0068] A random algorithm is used to select a certain number of target data nodes from the available node list. Specifically, a random number key is generated based on the inode number of the file metadata. The calculation formula is key = inode% len(remainDNs), where remainDNs represents the available nodes. The random number is used to perform a modulo operation on the available nodes until the operation result is less than or equal to 0. The final number of operation cycles is output as the required number of nodes, and the required number of data nodes are selected from the available node list as the target data nodes.

[0069] Available target data nodes are selected through a random algorithm. The selection of available nodes is even, so that the load of each available node is balanced, hot spots are avoided, and data writing efficiency and performance are improved.

[0070] Step S203: Send the node identification information of the target data node to the client, and the client determines the current data node according to the node identification information.

[0071] In this embodiment, node identification information corresponding to the target data node is obtained and transmitted to the client. During the transmission process, the network status is dynamically monitored. When the network status does not meet the preset transmission requirements, the transmission is suspended and the current transmission progress is saved. When the network status recovers and meets the preset transmission requirements, the remaining node identification information is transmitted according to the saved transmission progress. The preset transmission requirements include load level, network latency, transmission distance, link status, etc.

[0072] After the client obtains the node identification information, it selects a target data node as the first data node to receive the data sent by the client, which is the current data node.

[0073] Step S204: receiving the data packet sent by the client through the current data node, and writing the data packet into a preset storage location. After writing is completed, generating a corresponding response packet and putting it into a preset queue for processing.

[0074] In this embodiment, when the client obtains the node identification information of the target data node, the data nodes do not transmit data but metadata. When the current data node receives the data packet sent by the client, it directly calls the write function to write the data packet to the preset storage location and generates an Ack response packet corresponding to the data packet.

[0075] It should be emphasized that in order to further ensure the privacy and security of the data packet, the above data packet can also be stored in a blockchain node.

[0076] The blockchain referred to in this application is a new application model for computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanisms, and encryption algorithms. Blockchain is essentially a decentralized database, a series of data blocks generated using cryptographic methods. Each data block contains information about a batch of network transactions, which is used to verify the validity of this information (to prevent counterfeiting) and generate the next block. Blockchain can include the underlying blockchain platform, the platform product service layer, and the application service layer.

[0077] In some optional implementations of this embodiment, the step of receiving, through the current data node, a data packet sent by the client and writing the data packet to a preset storage location includes:

[0078] A data packet is received through the current data node, and the data packet is parsed to obtain a data packet identifier of the data packet;

[0079] According to the data packet identifier, determine whether the data packet is received repeatedly;

[0080] When the data packet is not received repeatedly, the integrity of the data packet is verified by a verification algorithm;

[0081] When the data packet passes the verification, the data packet is written to the preset storage.

[0082] Specifically, a data packet is obtained, and the data packet metadata, including the data packet identifier, check code, etc., is obtained. The data packet identifier is compared to determine whether the data packet is received repeatedly. If so, the data packet is discarded. Otherwise, the integrity of the data packet is verified through a verification algorithm, and the verification result is compared with the extracted verification information. If they are inconsistent, the data packet is considered damaged and discarded. If they are consistent, the data packet is considered intact and undamaged, and the write function is called to write the data packet to the preset storage location of the current data node.

[0083] As a specific example, the integrity of the data packet is verified through a verification algorithm, and the steps of comparing the verification result with the extracted verification information include: using a hash algorithm to verify the extracted slice metadata information, calculating the verification result, and comparing the verification result with the extracted verification code.

[0084] By checking the data packets, we can ensure the correctness and integrity of the received data packets, detect and handle transmission errors in a timely manner, ensure the reliability of the entire data transmission process, and thus improve the reliability and availability of data.

[0085] After the data packet is written, a corresponding response packet is generated to notify the client that the data has been successfully written. In this embodiment, the generated response packet is placed in a preset pending queue (PendingQueue) and waits for all data nodes to complete synchronization before returning it to the client.

[0086] Step S205, calling the preset callback function through the current data node to obtain the data packet length and slice metadata information of the data packet, and synchronizing the slice metadata information to other nodes through the preset distributed coordination service, and the other nodes are target data nodes other than the current data node.

[0087] In this embodiment, after a data packet is written, the current data node calls a preset callback function to obtain the packet length and slice metadata information of the data packet. The packet length is the length of the written data; the slice metadata information includes the slice identifier, file metadata, slice location, slice size, etc. The slice metadata is then synchronized to other nodes, ensuring that only metadata information is transmitted between data nodes to ensure data consistency.

[0088] In some optional implementations, the step of synchronizing the slice metadata information to other nodes through a preset distributed coordination service includes:

[0089] Encapsulate the slice metadata information and obtain the encapsulated synchronized metadata;

[0090] Establish a communication connection between the current data node and other nodes through the distributed coordination service. After the connection is successfully established, the synchronization metadata is sent to other nodes.

[0091] In response to the update results of other nodes based on the synchronization metadata, the current data node determines whether the slice metadata is successfully synchronized;

[0092] If synchronization fails, re-acquire the slice metadata information and initiate the synchronization request again until synchronization succeeds or the maximum number of retries is reached.

[0093] In this embodiment, slice metadata information is encapsulated according to a preset structural format to obtain encapsulated synchronization metadata. A communication connection is established between the current data node and other nodes through a preset distributed coordination service. After the connection is successfully established, the synchronization metadata is synchronized to the other nodes. After receiving the synchronization metadata, the other nodes parse the synchronization metadata to obtain slice metadata information and update the node's slice metadata based on the metadata information. After the update is complete, the update result is returned to the current data node. The current data node determines whether the slice metadata synchronization is successful based on the returned update result. If the synchronization is successful, the subsequent process is executed. If the synchronization fails, the current data node re-acquires the slice metadata information and re-initiates synchronization requests to other nodes until the synchronization is successful or the maximum number of retries is reached. Abnormal situations during the synchronization process are captured and processed, error logs are recorded, and an alarm mechanism is triggered to notify the administrator for manual intervention and problem troubleshooting.

[0094] The preset distributed coordination service refers to a service for unified coordination management and scheduling of collaborative work between nodes to ensure system consistency and reliability. As a specific example, the preset distributed coordination service is implemented through a distributed key-value storage system.

[0095] By synchronizing slice metadata information to other nodes, only metadata is transmitted between data nodes, ensuring data consistency while improving system data writing performance, reducing large amounts of data transmission between data nodes, and improving the efficiency and timeliness of data writing.

[0096] Step S206: Take out the corresponding response packet from the queue to be processed according to the length of the data packet through the current data node, put it into the preset response queue, and return it to the client.

[0097] Specifically, through the current data node, the number of response packets taken out from the queue to be processed is determined according to the length of the data packet; according to the number of response packets, starting from the head of the queue to be processed, a corresponding number of response packets are continuously taken out as target response packets; the target response packets are placed in the preset response queue; according to the write data request, the target response packets are taken out from the preset response queue, assembled into response data, and returned to the client.

[0098] Specifically, the packet lengths corresponding to the response packets in the pending queue are sequentially obtained, starting from the queue head of the pending queue. The packet lengths corresponding to the response packets are accumulated one by one. During this process, the accumulated length is checked to see if it exceeds the current packet length. If not, the packet length of the next response packet is accumulated. If it exceeds, the accumulation stops and all previously accumulated responses are selected and placed into the preset response queue AckQ, which is then returned to the client. The current packet length is the packet length of the packet written by the current data node.

[0099] For example, assuming that the current data packet length is 9, and the data packet lengths corresponding to the response packets in the queue to be processed are 3, 5, 2, and 1 respectively, then 3+5≤9 is obtained, and the first two response packets are sent to AckQ and returned to the client.

[0100] By placing the response packet into a preset response queue and returning it to the client, it is possible to optimize data transmission efficiency, reduce network load, quickly respond to client requests, improve the real-time performance of data processing, enhance the flexibility and scalability of the system, and improve resource utilization.

[0101] This application selects available target data nodes for data writing, which can avoid excessive pressure on a single data node when a large number of files are written, improve data reliability and fault tolerance, increase the parallelism and overall efficiency of data access, and optimize resource utilization; write the data packet to the preset storage through the current data node, and call the preset callback function to synchronize the slice metadata information of the data packet to other nodes through the preset distributed coordination service. Only metadata information is transmitted at the data node, which reduces a large amount of network data transmission between data nodes. While maintaining data uniformity, it improves the system's data writing performance and improves the efficiency and timeliness of data writing.

[0102] In some optional implementations, after the step of taking out the corresponding response packet from the queue to be processed according to the length of the data packet and placing it into the preset response queue and returning it to the client, the method further includes:

[0103] If a failure is detected in the current data node, the fault handling process is triggered, the slice metadata information of the data packet is obtained through other nodes, the slice metadata information is written into the preset metadata storage, and the corresponding response packet is generated.

[0104] In this embodiment, a failure of the current data node indicates that the slice data of the data packet has been written, but the slice metadata has not been successfully saved. A failure of the previous data node triggers the fault handling process. When the client sends a write request to another node, the other node writes the slice metadata information of the previously written file to the preset metadata storage and generates a corresponding response packet back to the client. This ensures that the response packet received by the client actually contains the data and metadata of the file to be written.

[0105] Through the fault handling process, it can be ensured that when a data node fails, data can continue to be read and written from other data nodes, ensuring data consistency.

[0106] Those skilled in the art will appreciate that all or part of the processes in the above-described method embodiments can be implemented by instructing related hardware via computer-readable instructions. The computer-readable instructions can be stored in a computer-readable storage medium, and when the program is executed, it can include the processes in the above-described method embodiments. The aforementioned storage medium can be a non-volatile storage medium such as a magnetic disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM).

[0107] It should be understood that although the steps in the flowcharts of the accompanying drawings are shown in sequence as indicated by the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some of the steps in the flowcharts of the accompanying drawings may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be executed in turn or alternately with other steps or at least a portion of the sub-steps or stages of other steps.

[0108] Further references Figure 3 , as a response to the above Figure 2 The present application provides an embodiment of a metadata-based writing device, which is similar to the embodiment of the present invention. Figure 2 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.

[0109] like Figure 3 As shown, the metadata-based writing device 300 described in this embodiment includes: a parsing module 301, a selection module 302, a determination module 303, a writing module 304, a synchronization module 305, and a response module 306.

[0110] The parsing module 301 is used to receive a write data request sent by a client, parse the write data request, and obtain file metadata of the file to be written;

[0111] The selection module 302 is used to select an available target data node from a pre-established node list according to the file metadata;

[0112] The determination module 303 is configured to send the node identification information of the target data node to the client, and the client determines the current data node according to the node identification information;

[0113] The writing module 304 is used to receive the data packet sent by the client through the current data node and write the data packet into a preset storage location. After writing is completed, a corresponding response packet is generated and placed into a preset queue for processing;

[0114] The synchronization module 305 is configured to call a preset callback function through the current data node to obtain the data packet length and slice metadata information of the data packet, and synchronize the slice metadata information to other nodes through a preset distributed coordination service, where the other nodes are target data nodes other than the current data node;

[0115] The response module 306 is used to take out the corresponding response packet from the to-be-processed queue according to the length of the data packet through the current data node, put it into the preset response queue, and return it to the client.

[0116] It should be emphasized that in order to further ensure the privacy and security of the data packet, the above data packet can also be stored in a blockchain node.

[0117] The metadata-based writing device 300 of the present application can avoid excessive pressure on a single data node when a large number of files are written by selecting an available target data node for data writing, thereby improving data reliability and fault tolerance, improving the parallelism and overall efficiency of data access, and optimizing resource utilization; the data packet is written to the preset storage through the current data node, and the preset callback function is called to synchronize the slice metadata information of the data packet to other nodes through the preset distributed coordination service. Only metadata information is transmitted at the data node, which reduces a large amount of network data transmission between data nodes. While maintaining data uniformity, it improves the system's data writing performance and improves the efficiency and timeliness of data writing.

[0118] In some optional implementations, the metadata-based writing device 300 further includes a verification module configured to:

[0119] Determine whether the file metadata is complete. If the file metadata is incomplete, return an error message to the client, prompting that the file metadata needs to be supplemented;

[0120] If the file metadata is complete, determine whether the file to be written is a file type supported by the system based on the file type field in the file metadata. If the file type is not supported by the system, return an error message to the client, indicating that the file type cannot be processed;

[0121] If the file type is supported by the system, the system determines whether the file to be written exceeds a preset processing file size threshold based on the file size field in the file metadata. If the file exceeds the threshold, the system returns an error message to the client, indicating that the file is too large.

[0122] If the threshold is not exceeded, determining whether the file to be written already exists based on the file identification field in the file metadata; if so, returning an error message to the client, prompting that the file to be written already exists;

[0123] If the file to be written does not exist, determining whether the user has permission to write the file based on the user identity field in the file metadata;

[0124] When the user has write permission, the step of selecting an available target data node from a pre-established node list according to the file metadata is performed.

[0125] By judging the legitimacy of file metadata, we can ensure the authenticity and integrity of files, improve the efficiency of file management, and ensure the security of file data.

[0126] In some optional implementations, the selection module 302 is further configured to:

[0127] Obtaining a pre-established node list, and obtaining node attribute information and node status information of each data node from the node list;

[0128] Determine the available nodes in the node list according to the node attribute information and the node status information;

[0129] Calculating the required number of nodes based on the data identifier of the file metadata and the available nodes;

[0130] A corresponding data node is selected from the available nodes as a target data node based on the required number of nodes.

[0131] Available target data nodes are selected through a random algorithm. The selection of available nodes is even, so that the load of each available node is balanced, hot spots are avoided, and data writing efficiency and performance are improved.

[0132] In some optional implementations, the writing module 304 includes:

[0133] A parsing submodule, configured to receive the data packet through the current data node, parse the data packet and obtain a data packet identifier of the data packet;

[0134] A judgment submodule, configured to judge whether the data packet is received repeatedly according to the data packet identifier;

[0135] A check submodule, configured to check the integrity of the data packet using a check algorithm when the data packet is not received repeatedly;

[0136] The writing submodule is used to write the data packet into a preset storage location when the data packet passes the verification.

[0137] By checking the data packets, we can ensure the correctness and integrity of the received data packets, detect and handle transmission errors in a timely manner, ensure the reliability of the entire data transmission process, and thus improve the reliability and availability of data.

[0138] In some optional implementations, the synchronization module 305 is further configured to:

[0139] Encapsulating the slice metadata information to obtain encapsulated synchronization metadata;

[0140] Establishing a communication connection between the current data node and other nodes through a distributed coordination service, and after the connection is successfully established, sending the synchronization metadata to the other nodes;

[0141] In response to an update result of the other nodes updating according to the synchronization metadata, determining, through the current data node, whether the slice metadata is successfully synchronized;

[0142] If synchronization fails, re-acquire the slice metadata information and initiate the synchronization request again until synchronization succeeds or the maximum number of retries is reached.

[0143] By synchronizing slice metadata information to other nodes, only metadata is transmitted between data nodes, ensuring data consistency while improving system data writing performance, reducing large amounts of data transmission between data nodes, and improving the efficiency and timeliness of data writing.

[0144] In some optional implementations of this embodiment, the response module 306 is further configured to:

[0145] Determining the number of response packets taken out from the queue to be processed according to the length of the data packet;

[0146] According to the number of response packets, starting from the head of the queue to be processed, a corresponding number of response packets are continuously taken out as target response packets;

[0147] Putting the target response packet into the preset response queue;

[0148] According to the write data request, the target response packet is taken out from the preset response queue, assembled into response data, and returned to the client.

[0149] By placing the response packet into a preset response queue and returning it to the client, it is possible to optimize data transmission efficiency, reduce network load, quickly respond to client requests, improve the real-time performance of data processing, enhance the flexibility and scalability of the system, and improve resource utilization.

[0150] In some optional implementations, the metadata-based writing device 300 also includes a fault handling module, which is used to: if it is detected that the current data node fails, trigger the fault handling process, obtain the slice metadata information of the data packet through the other nodes, write the slice metadata information into the preset metadata storage, and generate a corresponding response packet.

[0151] Through the fault handling process, it can be ensured that when a data node fails, data can continue to be read and written from other data nodes, ensuring data consistency.

[0152] To solve the above technical problems, the present application also provides a computer device. Figure 4 , Figure 4 This is a basic structural block diagram of the computer device in this embodiment.

[0153] The computer device 4 includes a memory 41, a processor 42, and a network interface 43 that are interconnected through a system bus. It should be noted that the figure only shows a computer device 4 with a memory 41, a processor 42, and a network interface 43, but it should be understood that it is not required to implement all the components shown, and more or fewer components can be implemented instead. Among them, those skilled in the art can understand that the computer device here is a device that can automatically perform numerical calculations and / or information processing according to pre-set or stored instructions, and its hardware includes but is not limited to a microprocessor, an application specific integrated circuit (ASIC), a field-programmable gate array (FPGA), a digital signal processor (DSP), an embedded device, etc.

[0154] The computer device may be a desktop computer, notebook computer, PDA, cloud server, etc. The computer device may interact with the user via a keyboard, mouse, remote control, touchpad, or voice control device.

[0155] The memory 41 includes at least one type of readable storage medium, including flash memory, a hard disk, a multimedia card, a card-type memory (e.g., SD or DX memory), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic storage, a magnetic disk, an optical disk, etc. In some embodiments, the memory 41 may be an internal storage unit of the computer device 4, such as the hard disk or memory of the computer device 4. In other embodiments, the memory 41 may also be an external storage device of the computer device 4, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash memory card, etc. equipped on the computer device 4. Of course, the memory 41 may also include both the internal storage unit of the computer device 4 and its external storage device. In this embodiment, the memory 41 is generally used to store the operating system and various application software installed on the computer device 4, such as computer-readable instructions based on the metadata writing method. In addition, the memory 41 can also be used to temporarily store various types of data that have been output or are to be output.

[0156] In some embodiments, the processor 42 may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chip. The processor 42 is generally used to control the overall operation of the computer device 4. In this embodiment, the processor 42 is used to execute computer-readable instructions stored in the memory 41 or process data, such as executing computer-readable instructions for the metadata-based writing method.

[0157] The network interface 43 may include a wireless network interface or a wired network interface. The network interface 43 is generally used to establish a communication connection between the computer device 4 and other electronic devices.

[0158] By selecting available target data nodes for data writing, it is possible to avoid excessive pressure on a single data node when writing a large number of files, improve data reliability and fault tolerance, increase the parallelism and overall efficiency of data access, and optimize resource utilization; write the data packet to the preset storage through the current data node, and call the preset callback function to synchronize the slice metadata information of the data packet to other nodes through the preset distributed coordination service. Only metadata information is transmitted at the data node, reducing a large amount of network data transmission between data nodes. While maintaining data uniformity, it improves the system's data writing performance and improves the efficiency and timeliness of data writing.

[0159] The present application also provides another embodiment, namely, providing a computer-readable storage medium, which stores computer-readable instructions, and the computer-readable instructions can be executed by at least one processor to enable the at least one processor to perform the steps of the metadata-based writing method as described above.

[0160] By selecting available target data nodes for data writing, it is possible to avoid excessive pressure on a single data node when writing a large number of files, improve data reliability and fault tolerance, increase the parallelism and overall efficiency of data access, and optimize resource utilization; write the data packet to the preset storage through the current data node, and call the preset callback function to synchronize the slice metadata information of the data packet to other nodes through the preset distributed coordination service. Only metadata information is transmitted at the data node, reducing a large amount of network data transmission between data nodes. While maintaining data uniformity, it improves the system's data writing performance and improves the efficiency and timeliness of data writing.

[0161] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus the necessary general hardware platform, and of course can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for enabling a terminal device (which can be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in each embodiment of the present application.

[0162] Obviously, the embodiments described above are only some of the embodiments of the present application, rather than all of the embodiments. The preferred embodiments of the present application are given in the accompanying drawings, but they do not limit the patent scope of the present application. The present application can be implemented in many different forms. On the contrary, the purpose of providing these embodiments is to make the understanding of the disclosure of the present application more thorough and comprehensive. Although the present application has been described in detail with reference to the aforementioned embodiments, for those skilled in the art, it is still possible to modify the technical solutions described in the aforementioned specific embodiments, or to make equivalent replacements for some of the technical features therein. Any equivalent structure made using the contents of the present application specification and the accompanying drawings, directly or indirectly used in other related technical fields, is also within the scope of patent protection of the present application.

Claims

1. A metadata-based writing method, characterized in that: The steps include: Receive a write data request sent by a client, parse the write data request, and obtain file metadata of the file to be written; Selecting an available target data node from a pre-established node list according to the file metadata; Sending the node identification information of the target data node to the client, and determining the current data node by the client according to the node identification information; Receive the data packet sent by the client through the current data node, and write the data packet into a preset storage location. After writing is completed, generate a corresponding response packet and put it into a preset queue for processing; Calling a preset callback function through the current data node to obtain the data packet length and slice metadata information of the data packet, and synchronizing the slice metadata information to other nodes through a preset distributed coordination service, where the other nodes are target data nodes other than the current data node; Through the current data node, the corresponding response packet is taken out from the queue to be processed according to the length of the data packet, put into the preset response queue, and returned to the client.

2. The metadata-based writing method according to claim 1, characterized in that: After the step of obtaining the file metadata of the file to be written, the method further includes: Determine whether the file metadata is complete. If the file metadata is incomplete, return an error message to the client, prompting that the file metadata needs to be supplemented; If the file metadata is complete, determine whether the file to be written is a file type supported by the system based on the file type field in the file metadata. If the file type is not supported by the system, return an error message to the client, indicating that the file type cannot be processed; If the file type is supported by the system, the system determines whether the file to be written exceeds a preset processing file size threshold based on the file size field in the file metadata. If the file exceeds the threshold, the system returns an error message to the client, indicating that the file is too large. If the threshold is not exceeded, determining whether the file to be written already exists based on the file identification field in the file metadata; if so, returning an error message to the client, prompting that the file to be written already exists; If the file to be written does not exist, determining whether the user has permission to write the file based on the user identity field in the file metadata; When the user has write permission, the step of selecting an available target data node from a pre-established node list according to the file metadata is performed.

3. The metadata-based writing method according to claim 1, characterized in that: The step of selecting an available target data node from a pre-established node list according to the file metadata includes: Obtaining a pre-established node list, and obtaining node attribute information and node status information of each data node from the node list; Determine the available nodes in the node list according to the node attribute information and the node status information; Calculating the required number of nodes based on the data identifier of the file metadata and the available nodes; A corresponding data node is selected from the available nodes as a target data node based on the required number of nodes.

4. The metadata-based writing method according to claim 1, wherein: The step of receiving, through the current data node, a data packet sent by the client and writing the data packet into a preset storage location includes: receiving the data packet through the current data node, parsing the data packet to obtain a data packet identifier of the data packet; Determining, based on the data packet identifier, whether the data packet is received repeatedly; When the data packet is not received repeatedly, verifying the integrity of the data packet using a verification algorithm; When the data packet passes the verification, the data packet is written into a preset storage location.

5. The metadata-based writing method according to claim 1, characterized in that: The step of synchronizing the slice metadata information to other nodes through a preset distributed coordination service includes: Encapsulating the slice metadata information to obtain encapsulated synchronization metadata; Establishing a communication connection between the current data node and other nodes through a distributed coordination service, and after the connection is successfully established, sending the synchronization metadata to the other nodes; In response to an update result of the other nodes updating according to the synchronization metadata, determining, through the current data node, whether the slice metadata is successfully synchronized; If synchronization fails, re-acquire the slice metadata information and initiate the synchronization request again until synchronization succeeds or the maximum number of retries is reached.

6. The metadata-based writing method according to claim 1, characterized in that: The step of taking out a corresponding response packet from the to-be-processed queue according to the length of the data packet and placing it into a preset response queue and returning the response packet to the client comprises: Determining the number of response packets taken out from the queue to be processed according to the length of the data packet; According to the number of response packets, starting from the head of the queue to be processed, a corresponding number of response packets are continuously taken out as target response packets; Putting the target response packet into the preset response queue; According to the write data request, the target response packet is taken out from the preset response queue, assembled into response data, and returned to the client.

7. The metadata-based writing method according to claim 1, characterized in that: After the step of taking out a corresponding response packet from the to-be-processed queue according to the length of the data packet and placing it into a preset response queue and returning the response packet to the client, the step further includes: If it is detected that the current data node fails, the fault handling process is triggered, the slice metadata information of the data packet is obtained through the other nodes, the slice metadata information is written into the preset metadata storage, and a corresponding response packet is generated.

8. A metadata-based writing device, characterized in that: include: A parsing module is configured to receive a write data request sent by a client, parse the write data request, and obtain file metadata of a file to be written; A selection module, configured to select an available target data node from a pre-established node list according to the file metadata; a determination module, configured to send the node identification information of the target data node to the client, and determine the current data node through the client according to the node identification information; A writing module is used to receive the data packet sent by the client through the current data node and write the data packet into a preset storage location. After writing is completed, a corresponding response packet is generated and placed into a preset queue for processing; a synchronization module, configured to call a preset callback function through the current data node to obtain the data packet length and slice metadata information of the data packet, and synchronize the slice metadata information to other nodes through a preset distributed coordination service, where the other nodes are target data nodes other than the current data node; The response module is used to take out the corresponding response packet from the queue to be processed according to the length of the data packet through the current data node, put it into the preset response queue, and return it to the client.

9. A computer device, characterized in that: The method comprises a memory and a processor, wherein the memory stores computer-readable instructions, and the processor implements the steps of the metadata-based writing method according to any one of claims 1 to 7 when executing the computer-readable instructions.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores computer-readable instructions, which, when executed by a processor, implement the steps of the metadata-based writing method according to any one of claims 1 to 7.

Images