Storage control method and related device
Through the combination of data processing units and elastic devices, the problem of cloud data center storage request processing occupies CPU resources is solved, high-performance storage and data integrity in abnormal situations is achieved, and the storage system of cloud data center is lossless.
Patent Information
- Application Number
- CN202310662142.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2023-06-05
- Publication Date
- 2024-12-06
AI Technical Summary
When existing cloud data centers process storage requests, they occupy the server's CPU resources, affect performance, and may lead to the loss of data storage messages in abnormal situations.
Storage control is performed through the data processing unit (DPU), and the combination of elastic devices and back-end storage warehouses is used to realize the processing and response of storage requests, avoid occupying server CPU resources, and prevent data loss through the resend mechanism in abnormal situations.
It realizes the improvement of cloud data center storage performance without occupying server CPU resources, and ensures losslessness of the storage system in abnormal scenarios and prevents data loss.
Smart Images

Figure CN119094609A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of data processing, and in particular to a storage control method and related devices. Background Art
[0002] A cloud data center can be implemented through multiple servers, which provide computing, storage, network data transmission and other functions, so that a cloud data center can realize cloud storage of data. Currently, the central processing unit (CPU) in the server used to provide computing obtains storage requests and processes the storage requests. However, this service provision method occupies the CPU resources in the server and affects the performance of the server. Summary of the invention
[0003] In order to solve the above technical problems, the present application provides a storage control method and related devices, which process storage requests through a data processing unit (DPU), thereby realizing lossless storage of the storage system without occupying CPU resources in the server and improving the storage performance of the cloud data center.
[0004] The embodiments of the present application disclose the following technical solutions:
[0005] In one aspect, the present application provides a storage control method, which is applied to a data processing unit, wherein the data processing unit includes an elastic device corresponding to a server, and the method includes:
[0006] Obtaining an object storage request from the server through the elastic device;
[0007] Acquire storage information of a data storage message from a first device, and send the data storage message to a second device according to the storage information; wherein, when the first device is the elastic device, the second device is a backend storage warehouse, and the data storage message is the object storage request; when the first device is the backend storage warehouse, the second device is the elastic device, and the data storage message includes an object storage response corresponding to the object storage request; the backend storage warehouse is used to perform storage processing according to the object storage request, and generate an object storage response corresponding to the object storage request according to the storage processing result;
[0008] In response to the abnormal recovery instruction for the data processing unit, if it is determined according to the storage information that the target message in the data storage message has not been sent, the target message is sent to the second device.
[0009] On the other hand, the present application provides a storage control device, characterized in that it is applied to a data processing unit, the data processing unit includes an elastic device corresponding to a server, and the device includes:
[0010] a request acquisition unit, configured to acquire an object storage request from the server through the elastic device;
[0011] A sending unit, configured to obtain storage information of a data storage message from a first device, and send the data storage message to a second device according to the storage information; wherein, when the first device is the elastic device, the second device is a backend storage warehouse, and the data storage message is the object storage request; when the first device is the backend storage warehouse, the second device is the elastic device, and the data storage message includes an object storage response corresponding to the object storage request; the backend storage warehouse is configured to perform storage processing according to the object storage request, and generate an object storage response corresponding to the object storage request according to the storage processing result;
[0012] An exception response unit is used to respond to the exception recovery instruction for the data processing unit, and if it is determined according to the storage information that the target message in the data storage message has not been sent, send the target message to the second device.
[0013] In another aspect, the present application provides a computer device, the device comprising a processor and a memory:
[0014] The memory is used to store a computer program and transmit the computer program to the processor;
[0015] The processor is used to execute the storage control method described in the above aspect according to the instructions in the computer program.
[0016] On the other hand, an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium is used to store a computer program, and the computer program is used to execute the storage control method described in the above aspect.
[0017] On the other hand, an embodiment of the present application provides a computer program product including a computer program, which, when executed on a computer device, enables the computer device to execute the storage control method.
[0018] It can be seen from the above technical solution that storage control can be performed through a data processing unit, and the data processing unit has an elastic device corresponding to the server, so that an object storage request from the server can be obtained through the elastic device. In the storage control process of the data processing unit, storage information of the data storage information can be obtained from the first device, and a data storage message is sent to the second device according to the storage information, wherein when the first device is an elastic device, the second device is a backend storage warehouse, and the data storage message is an object storage request. The backend storage warehouse is used to perform storage processing according to the object storage request, and generate an object storage response corresponding to the object storage request according to the storage processing result. When the first device is a backend storage warehouse, the second device is an elastic device, and the data storage message is an object storage response corresponding to the object storage request. In other words, the data processing unit can send the object storage request in the elastic device to the backend storage warehouse, and send the object storage response in the backend storage warehouse to the elastic device, thereby transferring the object storage request from the server to the backend storage warehouse, and transferring the object storage response from the backend storage warehouse to the server. It realizes the processing of storage requests through the combination of software and hardware of the elastic device and the storage control process, without occupying the CPU resources in the server, and will not affect the performance of the server.
[0019] In addition, after the data processing unit has an exception and recovers, in response to the exception recovery instruction for the data processing unit, if it is determined based on the storage information that the target message in the data storage message has not been sent, it means that at least one of the object storage request and the object storage response has not been sent in time due to the occurrence of the exception. Therefore, the data storage message can be sent to the second device. In this way, the data storage messages that have not been processed in time due to the exception in the data processing unit can be cleared by resending, thereby preventing the loss of data storage messages, and realizing lossless storage of the storage system in abnormal scenarios. That is, in the embodiment of the present application, lossless storage of the storage system can be realized without occupying the central processing unit resources in the server, thereby improving the storage performance of the cloud data center. BRIEF DESCRIPTION OF THE DRAWINGS
[0020] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.
[0021] Figure 1 A schematic diagram of an application scenario of a storage control method provided in an embodiment of the present application;
[0022] Figure 2A flowchart of a storage control method provided in an embodiment of the present application;
[0023] Figure 3 A schematic diagram of the structure of a data processing unit provided in an embodiment of the present application;
[0024] Figure 4 A queue mapping schematic diagram provided for an embodiment of the present application;
[0025] Figure 5 A schematic diagram of a transmission process provided in an embodiment of the present application;
[0026] Figure 6 Another transmission process schematic diagram provided for an embodiment of the present application;
[0027] Figure 7 A schematic diagram of another transmission process provided in an embodiment of the present application;
[0028] Figure 8 A schematic diagram of another transmission process provided in an embodiment of the present application;
[0029] Fig. 9 A schematic diagram of another transmission process provided in an embodiment of the present application;
[0030] Fig.10 A structural block diagram of a storage control device provided in an embodiment of the present application;
[0031] Fig.11 A structural diagram of a terminal device provided in an embodiment of the present application;
[0032] Fig.12 A structural diagram of a server provided in an embodiment of the present application. DETAILED DESCRIPTION
[0033] The embodiments of the present application are described below in conjunction with the accompanying drawings.
[0034] Currently, a storage request can be obtained and processed through a central processing unit in a server. This service provision method occupies CPU resources in the server and affects the performance of the server.
[0035] In order to solve the above technical problems, storage control can be performed through a data processing unit, which has an elastic device corresponding to the server, and the processing of storage requests is realized through the combination of software and hardware of the elastic device and the storage control process. In addition, after the data processing unit has an exception and recovers, the data storage messages in the data processing unit that have not been processed in time due to the exception can be cleared by resending, thereby preventing the loss of data storage messages, realizing lossless storage of the storage system without occupying CPU resources in the server, and improving the storage performance of the cloud data center.
[0036] The storage control method provided in the embodiment of the present application can be implemented by a computer device, which can be a storage system. The storage system includes a data processing unit and a server. The server can be connected to a terminal device, wherein the server can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. The terminal device can be a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart watch, etc., but is not limited to this. The terminal device and the server can be directly or indirectly connected via wired or wireless communication, and this application does not limit this.
[0037] The storage control method provided in the embodiment of the present application involves cloud technology. Cloud technology refers to a hosting technology that unifies a series of resources such as hardware, software, and network in a wide area network or a local area network to achieve data calculation, storage, processing, and sharing.
[0038] Cloud technology is a general term for network technology, information technology, integration technology, management platform technology, application technology, etc. based on the cloud computing business model. It can form a resource pool, which can be used on demand and is flexible and convenient. Cloud computing technology will become an important support. The backend services of the technical network system require a large amount of computing and storage resources, such as video websites, picture websites and more portal websites. With the rapid development and application of the Internet industry, in the future, each item may have its own identification mark, which needs to be transmitted to the backend system for logical processing. Data of different levels will be processed separately. All kinds of industry data need strong system backing support, which can only be achieved through cloud computing.
[0039] Cloud storage is a new concept that extends and develops from the concept of cloud computing. A distributed cloud storage system (hereinafter referred to as storage system) refers to a storage system that uses cluster applications, grid technology, and distributed storage file systems to bring together a large number of different types of storage devices (storage devices are also called storage nodes) in the network through application software or application interfaces to work together and provide external data storage and business access functions.
[0040] At present, the storage method of the storage system is: create a logical volume, and when creating a logical volume, allocate physical storage space for each logical volume. The physical storage space may be composed of disks of a storage device or several storage devices. The client stores data on a logical volume, that is, stores the data on the file system. The file system divides the data into many parts, each of which is an object. The object contains not only data but also additional information such as data identification (ID, ID entity). The file system writes each object into the physical storage space of the logical volume, and the file system records the storage location information of each object, so that when the client requests to access the data, the file system can allow the client to access the data according to the storage location information of each object.
[0041] The process of the storage system allocating physical storage space to a logical volume is as follows: based on the estimated capacity of the objects stored in the logical volume (this estimate often has a large margin relative to the actual capacity of the objects to be stored) and the grouping of independent redundant disk arrays (RAID, Redundant Array of Independent Disks), the physical storage space is pre-divided into stripes. A logical volume can be understood as a stripe, thereby allocating physical storage space to the logical volume.
[0042] In order to facilitate understanding of the technical solution provided by the present application, a storage control method provided by an embodiment of the present application will be introduced below in combination with an actual application scenario.
[0043] Figure 1A schematic diagram of an application scenario of a storage control method provided by an embodiment of the present application is shown, in which a server 10, a data processing unit 30, a terminal device 20 and a back-end storage warehouse 40 are included, the terminal device 20 and the server 10 are connected via a network, and the server 10 and the data processing unit 30 are connected, for example, the server 10 and the data processing unit 30 can be integrated into a server system having a data processing unit 30, and the data processing unit is also called a smart network card. The server 10 can be a virtual machine (VM) server or a bare metal server (Bare Metal Server, BMS), etc. The server 10 is connected to the back-end storage warehouse 40 through the data processing unit 30, so that the terminal device 20 can realize cloud storage through the server 10 and the data processing unit 30. The data processing unit 30 can be used as the aforementioned computer device to perform storage control.
[0044] The data processing unit 30 has an elastic device corresponding to the server 10, so that an object storage request from the server 10 can be obtained through the elastic device. In the storage control process of the data processing unit 30, storage information of the data storage information can be obtained from the first device, and a data storage message is sent to the second device according to the storage information. When the first device is an elastic device, the second device is a back-end storage warehouse 40, and the data storage message is an object storage request. The back-end storage warehouse 40 is used to perform storage processing according to the object storage request, and generate an object storage response corresponding to the object storage request according to the storage processing result. When the first device is the back-end storage warehouse 40, the second device is an elastic device, and the data storage message is an object storage response corresponding to the object storage request.
[0045] That is to say, the data processing unit 30 can send the object storage request in the elastic device to the backend storage warehouse 40, and send the object storage response in the backend storage warehouse 40 to the elastic device, thereby transferring the object storage request from the server 10 to the backend storage warehouse 40, and transferring the object storage response from the backend storage warehouse 40 to the server 10. It implements the processing of storage requests through the combination of hardware and software of the elastic device and the storage control process, without occupying the CPU resources in the server 10 and will not affect the performance of the server 10.
[0046] In addition, after the data processing unit 30 has an exception and recovers, in response to the exception recovery instruction for the data processing unit 30, if it is determined based on the storage information that the target message in the data storage message has not been sent, it means that at least one of the object storage request and the object storage response has not been sent in time due to the occurrence of the exception. Therefore, a data storage message can be sent to the second device, such as sending an object storage response to an elastic device, or sending an object storage request to the back-end storage warehouse 40. In this way, the data storage messages that have not been processed in time due to the exception in the data processing unit 30 can be cleared by resending, thereby preventing the loss of data storage messages, and realizing lossless storage of the storage system in abnormal scenarios. That is, in an embodiment of the present application, lossless storage of the storage system can be realized without occupying the central processing unit resources in the server 10, thereby improving the storage performance of the cloud data center.
[0047] Figure 2 A flowchart of a storage control method provided in an embodiment of the present application. In this embodiment, a server system is used as the aforementioned computer device for explanation. The server system includes a server and a data processing unit. The data processing unit includes an elastic device corresponding to the server. The data processing unit can execute the storage control method. The storage control method may include:
[0048] S101, obtaining an object storage request from a server through an elastic device.
[0049] In a cloud storage scenario, the server can send an object storage request to the backend storage warehouse according to the operation instructions of the terminal device. The backend storage warehouse can perform storage processing according to the object storage request and generate an object storage response based on the storage processing result, thereby using the server to implement cloud storage functions.
[0050] Currently, object storage requests can be processed by the CPU in the server. For example, the CPU can perceive the newly added object storage requests of the server's storage process through polling, and then copy the object storage requests to the storage engine in the backend storage warehouse for storage processing. This requires binding the polling operation to the CPU core, occupying the CPU resources of the server, and during the object storage request processing, the CPU needs to participate in the memory copy, which affects the CPU performance and also puts pressure on the memory.
[0051] In the embodiment of the present application, a data processing unit can be added to the server so that the server and the data processing unit constitute a server system, and the storage service of the terminal device is supported by the server system. The server can be a virtual machine server or a bare metal server, etc. The same data processing unit can be connected to one or more servers, for example, the same data processing unit can be connected to a virtual machine server and a bare metal server at the same time. The server system can be connected to a back-end storage warehouse, which is used to provide a storage area. The server system and the back-end storage warehouse constitute a cloud data center. The server acts as a visitor (guest), which can also be called a host (host) or a cloud host. The server system can send a storage request to the back-end storage warehouse based on the storage request of the terminal device, and the back-end storage warehouse performs storage processing according to the storage request to realize cloud storage of data.
[0052] The data processing unit can allocate elastic devices for the server's operating system (guest OS), see Figure 3 As shown, it is a structural diagram of a data processing unit provided in an embodiment of the present application. The elastic device can provide storage areas, interfaces, etc. for the server. When the server is a virtual machine server, the elastic device can be used as a virtual function (Virtual Function, VF) device. When the server is a bare metal server, the elastic device can be used as a physical function (Physical Function, PF) device. The server can send an object storage request to the elastic device, that is, the data processing unit can obtain the object storage request from the server through the elastic device. The object storage request may include at least one of a storage request and a query request for the target object. In this way, through the additional setting of the data processing unit, zero copy of the CPU in the server can be achieved, reducing the main memory pressure of the server.
[0053] S102: Acquire storage information of a data storage message from a first device, and send the data storage message to a second device according to the storage information.
[0054] In an embodiment of the present application, a data processing unit can be set between the server and the back-end storage warehouse, and the sending of messages between the server and the back-end storage warehouse can be implemented through the data processing unit, that is, the data processing unit can send the data storage message in the first device to the second device. Specifically, the storage information of the data storage message can be obtained from the first device, and the data storage message can be sent to the second device according to the storage information. The data storage message can be at least one of an object storage request and an object storage response. When the data storage message is an object storage request, the storage information of the data storage message is the request information of the object storage request. When the data storage message is an object storage response, the storage information of the data storage message is the response information of the object storage response.
[0055] When the first device is an elastic device, the second device is a backend storage warehouse, the data storage message is an object storage request, and the backend storage warehouse is used to perform storage processing according to the object storage request and generate an object storage response corresponding to the object storage request according to the storage processing result. When the first device is a backend storage warehouse, the second device is an elastic device, and the data storage message is an object storage response corresponding to the object storage request.
[0056] That is to say, the data processing unit can send the object storage request in the elastic device to the backend storage warehouse, and send the object storage response in the backend storage warehouse to the elastic device, thereby transferring the object storage request from the server to the backend storage warehouse, and transferring the object storage response from the backend storage warehouse to the server. It implements the processing of storage requests through the combination of hardware and software of the elastic device and the storage control process, without occupying the CPU resources in the server and will not affect the performance of the server.
[0057] The request information of the object storage request is used to indicate the request operation and request object of the object storage request. The request operation is, for example, a read operation or a write operation. The request object is, for example, a data identifier of the data to be processed. The request information of the object storage request may be header data of the object storage request. The processing status of the object storage request may be monitored based on the request information of the object storage request. For example, a sending list may be established for the object storage request based on the request information, or a processing status identifier may be added to the request information. The sending list and the processing status identifier are used to indicate the processing status of the object storage request corresponding to the request information. The processing status includes whether the request has been sent to the backend storage warehouse or has not been sent to the backend storage warehouse.
[0058] The response information of the object storage response is used to indicate the operation result and operation object of the object storage response. The operation result is, for example, the success or failure of a read operation, or the success or failure of a write operation. The operation object is, for example, a data identifier of processed data. The response information of the object storage response may be the header data of the object storage response. The processing status of the object storage response may be monitored based on the response information of the object storage response. For example, a sending list may be established for the object storage response based on the response information, or a processing status identifier may be added to the response information. The sending list and the processing status identifier are used to indicate the processing status of the object storage response corresponding to the response information. The processing status includes whether it has been sent to the elastic device or not sent to the elastic device.
[0059] The data processing unit may include a lossless transmission device, reference Figure 3As shown, the lossless transmission device is a hardware device, which can also be called a lossless engine, and can realize the internal transfer of data storage messages. The data processing unit can also have a transmission process, which can communicate with the back-end storage warehouse through the hardware interface of the data processing unit. The transmission process can also be called a cloud disk front-end device or spdk software. The transmission process can have a storage area corresponding to the transmission process, and the storage area and the lossless transmission device are connected through an elastic device, and the elastic device can at least include an interface.
[0060] During the storage control process of the data processing unit, an object storage request can be sent. Specifically, if the data storage message is an object storage request, the request information of the object storage request can be obtained from the first request queue corresponding to the elastic device through a lossless transmission device, and the object storage request is moved from the first request queue to the second request queue corresponding to the transmission process of the data processing unit according to the request information, and the object storage request is sent to the backend storage warehouse through the transmission process according to the second request queue; During the storage control process of the data processing unit, an object storage response can be sent. Specifically, if the data storage message is an object storage response, the object storage response can be obtained from the backend storage warehouse through the transmission process, and the response information of the object storage response can be obtained from the first response queue corresponding to the transmission process through a lossless transmission device, and the object storage response is moved from the first response queue to the second response queue corresponding to the elastic device according to the response information. Among them, the first request queue and the second response queue can be used as attributes of the elastic device, and the second request queue and the first response queue can be used as configuration information of the transmission process, refer to Figure 4 As shown, a queue mapping diagram provided in an embodiment of the present application is provided. The second request queue and the first response queue can be acquired and processed by the lossless transmission device, and the second request queue and the first response queue can also be acquired and processed by the transmission process. In this way, through the hardware transportation of the lossless transmission device and the software sending of the transmission process, object storage requests and object storage responses can be sent to achieve stable operation of the storage business.
[0061] Each elastic device may have one or more first request queues and one or more second response queues. The first request queue and the second response queue of the same elastic device may be independent of each other or combined into a comprehensive queue. Figure 4 As shown, the server includes n servers, and the corresponding elastic devices are represented as PF / VF#0 to PF / VF#n. A data processing unit may include one or more transmission processes, each of which may correspond to one or more second request queues, or one or more first response queues. The second request queue and first response queue corresponding to the same transmission process may be independent of each other, or may be combined into a comprehensive queue.
[0062] The object storage request is moved from the first request queue to the second request queue corresponding to the transmission process, which can be specifically, the object storage request is moved from the first storage area corresponding to the corresponding queue position in the first request queue to the second storage area corresponding to the corresponding queue position in the second request queue corresponding to the transmission process, so that the server can send the object storage request to the first storage area of the elastic device, and after the object storage request is moved to the second storage area, the transmission process can obtain the object storage request from the second storage area; the object storage response is moved from the first response queue to the second response queue of the elastic device object, which can be specifically, the object storage response is moved from the third storage area corresponding to the corresponding queue position in the first response queue to the fourth storage area corresponding to the corresponding queue position in the second response queue corresponding to the elastic device, so that the transmission process can place the obtained object storage response in the third storage area, and after the object storage response is moved to the fourth storage area, the server can obtain the object storage response from the fourth storage area. The second storage area and the third storage area belong to the storage space in the data processing unit.
[0063] The lossless transmission device can realize the transfer of object storage requests and object storage responses based on the queue mapping function, so that the transfer of object storage requests and object storage responses has a fixed strategy, and ensures the load balancing of object storage requests and object storage responses. Among them, the object storage request is transferred from the first request queue to the second request queue according to the request information, which can be specifically determined according to the request information and the queue mapping relationship, and then the object storage request is transferred from the first request queue to the second request queue; similarly, the object storage response is transferred from the first response queue to the second response queue according to the response information, which can be specifically determined according to the response information and the queue mapping relationship, and then the object storage response is transferred from the first response queue to the second response queue. The request information can reflect the information of the first request queue, the response information can reflect the information of the first response queue, and the queue mapping relationship can reflect the precise mapping of the queue level. It is implemented based on the internal metadata transmission (Internal Metadata Transmission, IMT) protocol. The queue mapping function is a function possessed by lossless transmission equipment. It can determine the queue to be moved to according to the configured queue mapping relationship, realize lossless transmission of storage traffic, and realize stable operation of storage services.
[0064] In specific implementation, data transfer can be achieved through direct memory access (DMA), which can send data from the elastic device directly to the memory of the data processing unit, thereby improving the efficiency of data copying. Of course, since data transfer is achieved through the data processing unit, it does not need to occupy the CPU resources and memory resources of the server, thereby reducing the pressure on the server main memory. That is, transferring the object storage request from the first request queue to the second request queue can be specifically achieved by transferring the object storage request from the first request queue to the second request queue through direct memory access; transferring the object storage response from the first response queue to the second response queue can be specifically achieved by transferring the object storage response from the first response queue to the second response queue through direct memory access. The data transfer achieved through DMA can be achieved by calling the DMA engine.
[0065] Before obtaining the request information of the object storage request from the first request queue corresponding to the elastic device through the lossless transmission device, the lossless transmission device can also obtain the storage notification from the server for indicating the first request queue, so that the lossless transmission device obtains the request information after obtaining the storage notification, and then starts sending the object storage request, without the need for the CPU or data processing unit to obtain the object storage request through polling, and without the need for the server system to fall into the kernel, thereby improving data transmission efficiency. In specific implementation, the storage notification can be sent to the lossless transmission device through the operating system (OS) of the server after updating the first request queue corresponding to the elastic device.
[0066] Before obtaining the response information of the object storage response from the first response queue corresponding to the transmission process through the lossless transmission device, the response notification indicating the first response queue from the transmission process can be obtained through the lossless transmission device. Similarly, after obtaining the response notification, the lossless transmission device obtains the response information and then starts sending the object storage response. There is no need for the CPU or data processing unit to obtain the object storage response through polling, and there is no need for the server system to fall into the kernel, thereby improving data transmission efficiency. In specific implementation, the transmission process can send the response notification to the lossless transmission device after updating the first response queue.
[0067] S103, in response to the abnormal recovery instruction for the data processing unit, if it is determined according to the storage information that the target message in the data storage message has not been sent, sending the target message to the second device.
[0068] In the embodiment of the present application, the data processing unit may have an exception, and when the exception occurs, at least one of the object storage request and the object storage response may not be sent in time, so that the data processing unit is prone to omissions in processing the object storage request, which affects the storage of the storage system. Therefore, after the data processing unit has an exception and recovers, in response to the exception recovery instruction for the data processing unit, if it is determined according to the storage information that the target message in the data storage message has not been sent, it means that at least one of the object storage request and the object storage response has not been sent in time due to the occurrence of the exception, so the data storage message can be sent to the second device, so that the data storage message that has not been processed in time due to the occurrence of the exception in the data processing unit can be cleared by resending, thereby preventing the loss of the data storage message and realizing lossless storage of the storage system in the abnormal scenario.
[0069] As mentioned above, the processing status of the object storage request can be monitored based on the request information, and the processing status of the object storage response can be monitored based on the response information of the object storage response. Specifically, the lossless transmission device can provide a query interface for querying the acquired storage information, and the data storage message corresponding to the acquired storage information is used as the message that needs to be processed.
[0070] The lossless transmission device can form a transport log after transporting the data storage message. The query interface can also be used to query the transport log. According to the comparison result of the storage information and the transport log, the messages that need to be processed and the messages that have been transported can be determined, thereby determining the messages to be transported among the messages that need to be processed. In the storage information, the response information of the object storage response corresponding to the object storage request has a corresponding relationship with the request information of the object storage request. For example, if they have the same data identifier, the object storage request without response information can be determined based on the request information and the response information. Among them, the object storage request with response information and its corresponding object response information can no longer be stored in the lossless transmission device to save the storage resources of the lossless transmission device.
[0071] After the transmission process sends the data storage message, a sending log can be formed. Here, the sending of the data storage message refers to sending an object storage request to the backend storage warehouse, and sending the obtained object storage response to the storage area corresponding to the first response queue. In this way, based on the comparison result of the storage information from the lossless transmission device and the sending log, the messages that need to be processed and the messages that have been sent can be determined, thereby determining the messages to be sent among the messages that need to be processed.
[0072] Therefore, in an embodiment of the present application, messages that need to be processed in the data storage messages, messages that have been transported by lossless transmission equipment, messages that have not been transported by lossless transmission equipment, messages sent by the transmission process, and messages that have not been sent by the transmission process can be determined based on the storage information, thereby controlling the processing nodes of each data storage message. When an exception occurs, the data storage message can continue to be processed according to the processing node of the data storage message to prevent the loss of the data storage message in the data processing unit.
[0073] In the embodiment of the present application, there is a transmission process in the data processing unit, and the abnormal recovery instruction for the data processing unit may be an abnormal recovery instruction for the transmission process. Since the transmission process is a software process, the transmission process often exits when an abnormality occurs. Figure 5 As shown, it is a schematic diagram of a transmission process provided by an embodiment of the present application, S103 may include S1031-S1033, wherein S1031, in response to an abnormal recovery instruction for the transmission process, starts the transmission process; S1032, calls the traffic pause interface of the lossless transmission device through the transmission process, so that the lossless transmission device suspends the operation of transferring the object storage request from the first request queue to the second request queue according to the request information, and the transmission process works normally, so it can first send the target message that has been transferred by the lossless transmission device and has not been sent by the transmission process, so as to clear the target message that has not been sent by the transmission process and prevent the loss of the target message.
[0074] Specifically, in S1033, if it is determined according to the storage information that there is an unsent target message in the transmission process, the target message can be restored in the queue of the transmission process, so that the transmission process sends the target message to the second device according to the queue of the transmission process. Wherein, if the target message is a target request in an object storage request, the queue of the transmission process is the second request queue, and if the target message is a target response in an object storage response, the queue of the transmission process is the first response queue. In other words, if it is determined according to the storage information that there is an unsent target request in the transmission process, the target request can be restored in the second request queue, and then the target request is sent to the backend storage warehouse according to the second request queue through the transmission process; if it is determined according to the storage information that there is an unsent target response in the transmission process, the target response can be restored in the first response queue, and then the response information of the target response is obtained from the first response queue through the lossless transmission device, and the target response is moved from the first response queue to the second response queue corresponding to the elastic device according to the response information. In this way, before starting the normal working process, the target request that was not sent in time when the exception occurred can be restored first, ensuring that the object storage request and the object storage response will not be lost, thereby ensuring the losslessness of the storage business.
[0075] After the target message is restored in the queue of the transmission process, S1034 can also be executed, in which the transmission process calls the traffic recovery interface of the lossless transmission device, so that the lossless transmission device can restore the operation of transferring the object storage request from the first request queue to the second request queue according to the request information, thereby sending the object storage request according to the normal sending process. In this way, through the hardware support including the traffic suspension interface and the traffic recovery interface, as well as the recovery process of the target message, the coordination of software and hardware is achieved to ensure that the target message will not be lost or erroneous due to the occurrence of abnormal scenarios. Among them, the transmission process calls the traffic recovery interface of the lossless transmission device before sending the target request to the backend storage warehouse, or after sending the target request to the backend storage warehouse; it can be done before the lossless transmission device obtains the response information of the target response, or after the lossless transmission device obtains the response information of the target response.
[0076] The transfer process can include a parent process and a child process called by the parent process. The child process is used to process data storage messages, such as sending object storage requests to the backend storage warehouse or obtaining object storage responses from the backend storage warehouse. Figure 6 As shown, it is another transmission process schematic diagram provided by an embodiment of the present application, wherein S1032 may be specifically, S1032A, calling the child process through the parent process in the transmission process; S1032B, calling the traffic pause interface of the lossless transmission device through the child process. S1032 may also include S1032C-S1032E, wherein S1032C, through the child process, determines whether there is an unsent target message, if so, executes S1032D to save the target message, if not, then executes S1032E to exit the child process; S1032F, after determining that each target message has been saved, it can be considered that the target message has been emptied, and then S1032E can be executed, that is, the child process is exited normally. Of course, if the child process fails to exit, the transmission process can be restarted to enter S1031.
[0077] After the subprocess is exited normally, the transmission process is initialized, and the transmission process can enter the normal working process. Then, S1033 can be executed to restore the target message in the queue of the transmission process, so that the target message in the queue is processed by using the re-called subprocess of the transmission process, and the flow recovery interface of the lossless transmission device can be called by the re-called subprocess to realize the lossless recovery of the storage service.
[0078] In actual operation, the object storage response contains less content, so it takes less time to obtain and move the object storage response, so it can usually be sent in time. The unsent target message of the transmission process is usually the target request in the object storage request. Therefore, after starting the transmission process, it is not necessary to judge whether there is an unsent target response in the transmission process, but only to judge whether there is an unsent target request. If there is a target request, the target request is restored in the first request queue of the transmission process.
[0079] The abnormal recovery instruction for the transmission process can be a restart instruction after the abnormal exit of the transmission process. When the transmission process exits abnormally, the target request is usually not sent, and of course the target response may not be sent. Therefore, the transmission process can suspend obtaining object storage requests, and then resume obtaining object storage requests after restoring the target message in the queue of the transmission process, so as to avoid the acquisition of too many object storage requests affecting the sending of the target message, and realize lossless recovery of storage services.
[0080] The abnormal recovery instruction for the transmission process may also be a hot upgrade completion instruction for the transmission process. The hot upgrade completion instruction is used to indicate that the hot upgrade of the transmission process is completed and can be put into use again. After that, the transmission process can be restarted and it is determined whether there is a target message to avoid the loss of the target message caused by the hot upgrade operation. In addition, if the abnormal recovery instruction for the transmission process is a hot upgrade completion instruction, the transmission process can be hot upgraded in response to the hot upgrade start instruction for the transmission process. After the hot upgrade of the transmission process is completed, the hot upgrade completion instruction for the transmission process can be obtained. The hot upgrade completion instruction and the hot upgrade start instruction can be obtained from the upgrade process used for process upgrade.
[0081] In the embodiment of the present application, before performing the hot upgrade, the target messages that have not been sent in the transmission process can be emptied. For details, refer to Figure 7As shown, it is another transmission process schematic diagram provided by an embodiment of the present application. In response to an abnormal recovery instruction for the transmission process in S1031, before starting the transmission process, S1030A and S1030B can be executed. In S1030A, in response to a hot upgrade start instruction for the transmission process, the traffic pause interface of the lossless transmission device can be called to make the lossless transmission device pause the operation of transferring the object storage request from the first request queue to the second request queue according to the request information. The transmission process works normally, so it can first send the target storage message that has been transferred by the lossless transmission device and has not been sent by the transmission process to clear the data storage message that has not been sent by the transmission process and prevent the loss of the data storage message. Afterwards, in S1030B, when it is determined that there is no unsent data storage message in the transmission process according to the storage information, the transmission process can be exited so as to upgrade the configuration information of the transmission process. Upgrading the configuration information of the transmission process can be achieved through the aforementioned upgrade process.
[0082] In actual operation, there may be a situation where there are still unsent data storage messages in the transmission process for a period of time after the operation of moving the object storage request from the first request queue to the second request queue according to the request information is suspended. Therefore, after a preset time period after the operation of moving the object storage request from the first request queue to the second request queue according to the request information is suspended, the transmission process can be enabled, and the unsent data storage messages that still exist in the transmission process can continue to be sent by starting a new process after hot upgrading.
[0083] In an embodiment of the present application, there is a lossless transmission device in the data processing unit, so the abnormal recovery instruction for the data processing unit can be an abnormal recovery instruction for the lossless transmission device, such as a hot upgrade completion instruction for the lossless transmission device. During the hot upgrade process of the lossless transmission device, the lossless transmission device does not have the carrying capacity, so that the storage service cannot operate normally. Therefore, after the hot upgrade of the lossless transmission device is completed, it is necessary to restore the operation of the storage service.
[0084] Specifically, refer to Figure 8As shown, it is another transmission process schematic diagram provided by an embodiment of the present application, S103 may include S103A-S103B, in S103A, in response to the hot upgrade completion instruction for the lossless transmission device, the target configuration information and the information to be processed in the acquired storage information may be configured for the lossless transmission device, wherein the target configuration information may indicate the workflow of the lossless transmission device, for example, it may include the aforementioned queue mapping relationship, etc., the data storage message corresponding to the information to be processed in the acquired storage information has not been transported by the lossless transmission device, that is, the data storage message corresponding to the information to be processed is the message to be processed. In this way, the lossless transmission device that has completed the hot upgrade can be used to transport the information to be processed, for example, when the information to be processed is an object storage request, the object storage request can be transported from the first request queue to the second request queue through the lossless transmission device, and when the information to be processed is an object storage response, the object storage response can be transported from the first response queue to the second response queue through the lossless transmission device, so as to prevent the loss of storage requests or storage responses caused by the hot upgrade of the lossless transmission device. Afterwards, in S103B, the operation of obtaining the object storage request information from the first request queue corresponding to the elastic device through the lossless transmission device, and the operation of obtaining the response information of the object storage response from the first response queue corresponding to the transmission process through the lossless transmission device can be restored. At this time, the lossless transmission device has normal carrying capacity, so that the storage business traffic can be restored.
[0085] Before performing a hot upgrade, you can drain the untransferred data storage messages in the lossless transmission device. For details, refer to Fig. 9As shown, it is a schematic diagram of another transmission process in an embodiment of the present application. In response to the hot upgrade completion instruction for the lossless transmission device, the target configuration information is configured for the lossless transmission device, and before the pending information in the storage information is processed, S103C-S103E can also be executed. In S103C, in response to the hot upgrade start instruction for the lossless transmission device, the operation of obtaining the request information of the object storage request from the first request queue corresponding to the elastic device through the lossless transmission device and the operation of obtaining the response information of the object storage response from the first response queue corresponding to the transmission process through the lossless transmission device can be suspended, so that the lossless transmission device stops obtaining the object storage request and the object storage response, the object storage request is retained in the elastic device, and the object storage response is temporarily stored in the storage space corresponding to the transmission process. The lossless transmission device works according to the normal handling process, that is, it can carry the object storage request corresponding to the obtained request information and the object storage response corresponding to the obtained response information. In S103D, if it is determined based on the storage information that there is no storage information corresponding to the data storage message that has not been moved in the lossless transmission device, the target configuration information of the lossless transmission device can be saved to perform a hot upgrade on the lossless transmission device; correspondingly, in S103E, if after a preset time has passed after the hot upgrade start instruction is obtained, it is determined based on the storage information that there is storage information corresponding to the data storage information that has not been moved in the lossless transmission device, then no further waiting is required, and the storage information corresponding to the data storage message that has not been moved is taken as information to be processed, and the target configuration information and the information to be processed of the lossless transmission device are saved to perform a hot upgrade on the lossless transmission device.
[0086] In actual operation, if after a preset time has passed after the hot upgrade start instruction is obtained, it is determined based on the storage information that there is request information corresponding to the object storage request that has not been transported in the lossless transmission device, then the waiting may not continue, and the request information corresponding to the object storage request that has not been transported may be used as pending information. If after a preset time has passed after the hot upgrade start instruction is obtained, it is determined based on the storage information that there is response information corresponding to the object storage response that has not been transported in the lossless transmission device, the response information corresponding to the object storage response that has not been transported may be used as pending information, or it may be considered that the hot upgrade conditions are not met, and the operation of obtaining the object storage request information from the first request queue corresponding to the elastic device through the lossless transmission device, and the operation of obtaining the response information of the object storage response from the first response queue corresponding to the transmission process through the lossless transmission device are restored, and the next hot upgrade retry may be waited for. The next hot upgrade retry may be after a certain time has passed after the hot upgrade start instruction is received, or it may be after the next hot upgrade start instruction is received. This is because the transport of the object storage response usually takes less time. If the transport cannot be completed after the preset time, the lossless transmission device may be abnormal, and therefore the hot upgrade conditions are not met.
[0087] In the embodiment of the present application, the sending of object storage requests can be controlled according to the flow rate. Specifically, the request flow rate of the object storage request can be obtained according to the first request queue, and the response flow rate of the object storage response can be obtained according to the first response queue. If the request flow rate and the response flow rate meet the first rate difference condition, for example, the difference between the two is greater than or equal to the first threshold, it means that the data processing unit is not sufficient to send these object storage requests in time, or the back-end storage warehouse cannot timely implement the storage processing corresponding to the object storage request, so the operation of transferring the object storage request from the first request queue to the second request queue corresponding to the transmission process of the data processing unit according to the request information can be suspended first, that is, the obtained request information can be cached in the lossless transmission device to avoid the untimely response caused by excessive sending of object storage requests. On the contrary, if the request flow rate and the response flow rate do not meet the first rate difference condition, for example, the difference between the two is less than the first threshold, it means that the data processing unit is capable of implementing the sending of the object storage request, and the back-end storage warehouse is capable of implementing the storage processing corresponding to the object storage request, so the operation of transferring the object storage request from the first request queue to the second request queue corresponding to the transmission process of the data processing unit according to the request information can be restored, that is, the object storage request is sent normally. Among them, the first threshold can be 0 or a smaller value greater than 0. The larger the first threshold, the more request information that needs to be cached by the data processing unit. Therefore, the actual value of the first threshold can be determined according to the actual storage situation of the data processing unit.
[0088] In the embodiment of the present application, the acquisition of the request information of the object storage request can also be controlled according to the flow rate. Specifically, if the request flow rate and the response flow rate meet the second rate difference condition, for example, the difference between the two is greater than or equal to the second threshold, it means that the data processing unit is not sufficient to send these object processing requests in time, or the back-end storage warehouse cannot timely implement the storage processing corresponding to the object storage request, so the operation of obtaining the request information of the object storage request from the first request queue corresponding to the elastic device through the lossless transmission device can be suspended first, that is, the acquisition of the request information can be limited to avoid the waste of memory caused by too much request information cached in the lossless transmission device, and will not affect the normal operation of the storage business. On the contrary, if the request flow rate and the response flow rate do not meet the second rate difference condition, for example, the difference between the two is less than the second threshold, it means that the data processing unit is capable of implementing the sending of the object storage request, and the back-end warehouse is capable of implementing the storage processing corresponding to the object storage request, so the operation of obtaining the request information of the object storage request from the first request queue corresponding to the elastic device through the lossless transmission device can be performed, that is, the request information is normally obtained. Of course, the second threshold can be greater than the first threshold, so that the number of times the request information is suspended is limited. Among them, the number of first request queues can be one or more. In this way, through the mapping relationship between the first request queue and the second request queue, the first request queue can be accurately back-pressured without affecting the request queues corresponding to other corresponding queues, thereby ensuring data losslessness.
[0089] In this way, in normal business scenarios, the sending of object storage requests and object storage responses can be realized through the elastic device, lossless transmission device and transmission process in the data processing unit, and the sending of object storage requests can be controlled according to the traffic rate. In abnormal scenarios, such as during the hot upgrade of the lossless transmission device and the transmission process, and after the troubleshooting of the transmission process, the loss of object storage requests and object storage responses can be avoided by controlling the work of the lossless transmission device and the transmission process, thereby realizing lossless storage of the storage system in abnormal scenarios. That is, in the embodiment of the present application, lossless storage of the storage system can be realized based on the full scenario without occupying the central processing unit resources in the server, thereby improving the storage performance of the cloud data center.
[0090] Based on a storage control method provided in an embodiment of the present application, an embodiment of the present application also provides a storage control device, referring to Fig.10 , which is a structural block diagram of a storage control device provided in an embodiment of the present application, wherein the storage control device 1300 is applied to a data processing unit, wherein the data processing unit includes an elastic device corresponding to a server, and the device includes:
[0091] A request acquisition unit 1301 is used to acquire an object storage request from the server through an elastic device;
[0092] The sending unit 1302 is used to obtain storage information of the data storage message from the first device, and send the data storage message to the second device according to the storage information; wherein, when the first device is the elastic device, the second device is the backend storage warehouse, and the data storage message is the object storage request; when the first device is the backend storage warehouse, the second device is the elastic device, and the data storage message includes an object storage response corresponding to the object storage request; the backend storage warehouse is used to perform storage processing according to the object storage request, and generate an object storage response corresponding to the object storage request according to the storage processing result;
[0093] The exception response unit 1303 is used to respond to the exception recovery instruction for the data processing unit, and if it is determined according to the storage information that the target message in the data storage message has not been sent, send the target message to the second device.
[0094] Optionally, the data processing unit further includes a lossless transmission device, and the sending unit 1302 includes:
[0095] a request information acquisition unit, configured to acquire, if the data storage message is an object storage request, request information of the object storage request from a first request queue corresponding to the elastic device through the lossless transmission device;
[0096] a request transfer unit, configured to transfer the object storage request from the first request queue to a second request queue corresponding to the transmission process of the data processing unit according to the request information;
[0097] A sending unit, configured to send the object storage request to a backend storage warehouse according to the second request queue through the transmission process;
[0098] a response acquisition unit, configured to acquire an object storage response from the backend storage repository through the transmission process if the data storage message is an object storage response;
[0099] a response information acquisition unit, configured to acquire the response information of the object storage response from the first response queue corresponding to the transmission process through the lossless transmission device;
[0100] The response transfer unit is used to transfer the object storage response from the first response queue to a second response queue corresponding to the elastic device according to the response information.
[0101] Optionally, the abnormality recovery instruction for the data processing unit is an abnormality recovery instruction for the transmission process, and the abnormality response unit 1303 includes:
[0102] A process starting unit, configured to start a transmission process in response to an abnormal recovery instruction for the transmission process;
[0103] an interface calling unit, configured to call the traffic pause interface of the lossless transmission device through the transmission process, so that the lossless transmission device pauses the operation of transferring the object storage request from the first request queue to the second request queue corresponding to the transmission process of the data processing unit according to the request information;
[0104] a message recovery unit, configured to recover the target message in the queue of the transmission process if it is determined according to the storage information that there is an unsent target message in the transmission process, so that the transmission process sends the target message to the second device according to the queue of the transmission process; wherein, if the target message is a target request in the object storage request, the queue of the transmission process is the second request queue, and if the target message is a target response in the object storage response, the queue of the transmission process is the first response queue;
[0105] The device also includes:
[0106] A traffic recovery unit is used to call the traffic recovery interface of the lossless transmission device through the transmission process after restoring the target message in the queue of the transmission process, so that the lossless transmission device can restore the operation of transferring the object storage request from the first request queue to the second request queue corresponding to the transmission process of the data processing unit according to the request information.
[0107] Optionally, the abnormal recovery instruction for the transmission process is a hot upgrade completion instruction for the transmission process, and the device further includes:
[0108] a traffic suspension unit, configured to, before starting the transmission process in response to the abnormal recovery instruction for the transmission process, call the traffic suspension interface of the lossless transmission device in response to the hot upgrade start instruction for the transmission process, so that the lossless transmission device suspends the operation of transferring the object storage request from the first request queue to the second request queue corresponding to the transmission process of the data processing unit according to the request information;
[0109] The process exit unit is used to exit the transmission process when it is determined according to the storage information that there is no unsent data storage message in the transmission process, so as to upgrade the configuration information of the transmission process.
[0110] Optionally, the abnormal recovery instruction for the data processing unit is a hot upgrade completion instruction for the lossless transmission device, and the abnormal response unit 1303 includes:
[0111] a configuration unit, configured to configure target configuration information and the to-be-processed information in the storage information for the lossless transmission device in response to a hot upgrade completion instruction for the lossless transmission device, wherein the data storage message corresponding to the to-be-processed information is not moved by the lossless transmission device;
[0112] The device also includes:
[0113] A traffic recovery unit is used to restore the operation of obtaining the request information of the object storage request from the first request queue corresponding to the elastic device through the lossless transmission device, and the operation of obtaining the response information of the object storage response from the first response queue corresponding to the transmission process through the lossless transmission device, after configuring the target configuration information for the lossless transmission device and obtaining the pending information in the request information.
[0114] Optionally, the device further comprises:
[0115] a pausing unit, configured to, in response to a hot upgrade completion instruction for the lossless transmission device, configure target configuration information for the lossless transmission device, and before the to-be-processed information in the storage information is configured, in response to a hot upgrade start instruction for the lossless transmission device, suspend the operation of obtaining the request information of the object storage request from the first request queue corresponding to the elastic device through the lossless transmission device, and the operation of obtaining the response information of the object storage response from the first response queue corresponding to the transmission process through the lossless transmission device;
[0116] A saving unit is used for saving target configuration information of the lossless transmission device if it is determined based on the storage information that there is no storage information corresponding to the data storage message that has not been transported in the lossless transmission device, so as to perform a hot upgrade on the lossless transmission device; if after a preset time has passed after the hot upgrade start instruction is obtained, it is determined based on the storage information that there is storage information corresponding to the data storage message that has not been transported in the lossless transmission device, then the storage information corresponding to the data storage message that has not been transported is used as information to be processed, and the target configuration information of the lossless transmission device and the information to be processed are saved, so as to perform a hot upgrade on the lossless transmission device.
[0117] Optionally, the device further comprises:
[0118] a rate determination unit, configured to obtain a request flow rate of the object storage request according to the first request queue, and obtain a response flow rate of the object storage response according to the first response queue;
[0119] a request pausing unit, configured to suspend the operation of transferring the object storage request from the first request queue to the second request queue corresponding to the transmission process of the data processing unit according to the request information if the request flow rate and the response flow rate satisfy a first rate difference condition;
[0120] A request recovery unit is used to recover the operation of transferring the object storage request from the first request queue to the second request queue corresponding to the transmission process of the data processing unit according to the request information if the request traffic rate and the response traffic rate do not satisfy the first rate difference condition.
[0121] Optionally, the device further comprises:
[0122] an information pausing unit, configured to suspend the operation of obtaining the request information of the object storage request from the first request queue corresponding to the elastic device through the lossless transmission device if the request traffic rate and the response traffic rate satisfy a second rate difference condition;
[0123] An information recovery unit is used to recover the operation of obtaining the request information of the object storage request from the first request queue corresponding to the elastic device through the lossless transmission device if the request traffic rate and the response traffic rate do not satisfy the second rate difference condition.
[0124] Optionally, the request handling unit includes:
[0125] A first queue determination unit, configured to determine, according to the request information and the queue mapping relationship, a second request queue corresponding to the object storage request in the request queue of the transmission process of the data processing unit;
[0126] A first transfer unit, configured to transfer the object storage request from the first request queue to the second request queue;
[0127] The response handling unit comprises:
[0128] A second queue determining unit, configured to determine, in the response queue of the elastic device, a second response queue corresponding to the object storage response according to the response information and the queue mapping relationship;
[0129] The second transfer unit is used to transfer the object storage response from the first response queue to the second response queue.
[0130] Optionally, the first transport unit is specifically used for:
[0131] Transferring the object storage request from the first request queue to the second request queue through direct memory access;
[0132] The second transport unit is specifically used for:
[0133] The object store response is moved from the first response queue to the second response queue via direct memory access.
[0134] Optionally, the device further comprises:
[0135] a storage notification acquisition unit, configured to acquire, through the lossless transmission device, a storage notification from the server indicating the first request queue before acquiring the request information of the object storage request from the first request queue corresponding to the elastic device through the lossless transmission device;
[0136] The device also includes:
[0137] A response notification acquisition unit is used to obtain a response notification indicating the first response queue from the transmission process through the lossless transmission device before obtaining the response information of the object storage response from the first response queue corresponding to the transmission process through the lossless transmission device.
[0138] It can be seen from the above technical solution that storage control can be performed through a data processing unit, and the data processing unit has an elastic device corresponding to the server, so that an object storage request from the server can be obtained through the elastic device. In the storage control process of the data processing unit, storage information of the data storage information can be obtained from the first device, and a data storage message is sent to the second device according to the storage information, wherein when the first device is an elastic device, the second device is a backend storage warehouse, and the data storage message is an object storage request. The backend storage warehouse is used to perform storage processing according to the object storage request, and generate an object storage response corresponding to the object storage request according to the storage processing result. When the first device is a backend storage warehouse, the second device is an elastic device, and the data storage message is an object storage response corresponding to the object storage request. In other words, the data processing unit can send the object storage request in the elastic device to the backend storage warehouse, and send the object storage response in the backend storage warehouse to the elastic device, thereby transferring the object storage request from the server to the backend storage warehouse, and transferring the object storage response from the backend storage warehouse to the server. It realizes the processing of storage requests through the combination of software and hardware of the elastic device and the storage control process, without occupying the CPU resources in the server, and will not affect the performance of the server.
[0139] In addition, after the data processing unit has an exception and recovers, in response to the exception recovery instruction for the data processing unit, if it is determined based on the storage information that the target message in the data storage message has not been sent, it means that at least one of the object storage request and the object storage response has not been sent in time due to the occurrence of the exception. Therefore, the data storage message can be sent to the second device. In this way, the data storage messages that have not been processed in time due to the exception in the data processing unit can be cleared by resending, thereby preventing the loss of data storage messages, and realizing lossless storage of the storage system in abnormal scenarios. That is, in the embodiment of the present application, lossless storage of the storage system can be realized without occupying the central processing unit resources in the server, thereby improving the storage performance of the cloud data center.
[0140] The embodiment of the present application further provides a computer device, which is the computer device described above and may include a server system, the server system includes a server and a data processing unit, and the storage control device described above may be configured in the computer device. The computer device is described below in conjunction with the accompanying drawings.
[0141] See also Fig.11 As shown, the embodiment of the present application provides a terminal device, taking the terminal device as a mobile phone as an example:
[0142] Fig.11 FIG. 1 is a block diagram showing a partial structure of a mobile phone related to a terminal device provided in an embodiment of the present application. Fig.11 The mobile phone includes: a radio frequency (RF) circuit 1410, a memory 1420, an input unit 1430, a display unit 1440, a sensor 1450, an audio circuit 1460, a wireless fidelity (WiFi) module 1470, a processor 1480, and a power supply 1490. Those skilled in the art can understand that Fig.11 The mobile phone structure shown in the figure does not constitute a limitation on the mobile phone, and may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.
[0143] Combine the following Fig.11 A detailed introduction to the various components of the mobile phone:
[0144] The RF circuit 1410 may be used for receiving and sending signals during information transmission or communication or during a call. In particular, after receiving downlink information from a base station, the information is sent to the processor 1480 for processing. In addition, the uplink data is sent to the base station.
[0145] The memory 1420 can be used to store software programs and modules. The processor 1480 executes various functional applications and data processing of the mobile phone by running the software programs and modules stored in the memory 1420. The memory 1420 can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc.; the data storage area can store data created according to the use of the mobile phone (such as audio data, a phone book, etc.), etc. In addition, the memory 1420 can include a high-speed random access memory, and can also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage devices.
[0146] The input unit 1430 may be used to receive input digital or character information and generate key signal input related to user settings and function control of the mobile phone. Specifically, the input unit 1430 may include a touch panel 1431 and other input devices 1432 .
[0147] The display unit 1440 may be used to display information input by the user or information provided to the user and various menus of the mobile phone. The display unit 1440 may include a display panel 1441.
[0148] The mobile phone may also include at least one sensor 1450, such as a light sensor, a motion sensor, and other sensors.
[0149] The audio circuit 1460 , the speaker 1461 , and the microphone 1462 can provide an audio interface between the user and the mobile phone.
[0150] WiFi is a short-range wireless transmission technology. The mobile phone can help users send and receive emails, browse web pages and access streaming media through the WiFi module 1470. It provides users with wireless broadband Internet access.
[0151] Processor 1480 is the control center of the mobile phone. It uses various interfaces and lines to connect various parts of the entire mobile phone. It executes various functions of the mobile phone and processes data by running or executing software programs and / or modules stored in memory 1420 and calling data stored in memory 1420.
[0152] The mobile phone also includes a power source 1490 (such as a battery) for supplying power to various components.
[0153] In this embodiment, the processor 1480 included in the terminal device also has the following functions:
[0154] The data processing unit obtains the object storage request from the server through the elastic device, and the data processing unit includes the elastic device corresponding to the server;
[0155] Acquire storage information of a data storage message from a first device, and send the data storage message to a second device according to the storage information; wherein, when the first device is the elastic device, the second device is a backend storage warehouse, and the data storage message is the object storage request; when the first device is the backend storage warehouse, the second device is the elastic device, and the data storage message includes an object storage response corresponding to the object storage request; the backend storage warehouse is used to perform storage processing according to the object storage request, and generate an object storage response corresponding to the object storage request according to the storage processing result;
[0156] In response to the abnormal recovery instruction for the data processing unit, if it is determined according to the storage information that the target message in the data storage message has not been sent, the target message is sent to the second device.
[0157] The present application also provides a server. Fig.12 As shown, Fig.12 The structural diagram of the server 1500 provided in the embodiment of the present application, the server 1500 may have relatively large differences due to different configurations or performances, and may include one or more processors 1522, such as central processing units (CPU), memory 1532, and one or more storage media 1530 (e.g., one or more mass storage devices) storing application programs 1542 or data 1544. Among them, the memory 1532 and the storage medium 1530 can be short-term storage or permanent storage. The program stored in the storage medium 1530 may include one or more modules (not shown in the figure), and each module may include a series of instruction operations on the server. Furthermore, the processor 1522 can be configured to communicate with the storage medium 1530 and execute a series of instruction operations in the storage medium 1530 on the server 1500.
[0158] The server 1500 may also include one or more power supplies 1526, one or more wired or wireless network interfaces 1550, one or more input and output interfaces 1558, and / or one or more operating systems 1541, such as Windows Server 2003 or Windows Server 2008. TM , Mac OS X TM , Unix TM ,Linux TM , FreeBSD TM etc.
[0159] The steps performed by the server in the above embodiment can be based on Fig.12 The server structure shown.
[0160] In addition, an embodiment of the present application further provides a storage medium, wherein the storage medium is used to store a computer program, and the computer program is used to execute the method provided in the above embodiment.
[0161] The embodiments of the present application also provide a computer program product including instructions, which, when executed on a computer, enables the computer to execute the method provided in the above embodiments.
[0162] A person of ordinary skill in the art can understand that all or part of the steps of implementing the above method embodiment can be completed by hardware related to program instructions, and the aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it executes the steps of the above method embodiment; and the aforementioned storage medium can be at least one of the following media: read-only memory (English: Read-only Memory, abbreviated: ROM), RAM, magnetic disk or optical disk, etc. Various media that can store program codes.
[0163] It should be noted that each embodiment in this specification is described in a progressive manner, and the same and similar parts between the embodiments can refer to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device and system embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can refer to the partial description of the method embodiments. The device and system embodiments described above are merely schematic, in which the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative work.
[0164] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions that can be easily thought of by a technician familiar with the technical field within the technical scope disclosed in the present application should be included in the protection scope of the present application. Moreover, based on the implementation methods provided in the above aspects, the present application can also be further combined to provide more implementation methods. Therefore, the protection scope of the present application should be based on the protection scope of the claims.
Claims
1. A storage control method, characterized in that: Applied to a data processing unit, the data processing unit includes an elastic device corresponding to a server, the method includes: Obtaining an object storage request from the server through the elastic device; Acquire storage information of a data storage message from a first device, and send the data storage message to a second device according to the storage information; wherein, when the first device is the elastic device, the second device is a backend storage warehouse, and the data storage message is the object storage request; when the first device is the backend storage warehouse, the second device is the elastic device, and the data storage message includes an object storage response corresponding to the object storage request; the backend storage warehouse is used to perform storage processing according to the object storage request, and generate an object storage response corresponding to the object storage request according to the storage processing result; In response to the abnormal recovery instruction for the data processing unit, if it is determined according to the storage information that the target message in the data storage message has not been sent, the target message is sent to the second device.
2. The method according to claim 1, characterized in that: The data processing unit further includes a lossless transmission device, wherein acquiring storage information of the data storage message from the first device and sending the data storage message to the second device according to the storage information includes: If the data storage message is an object storage request, obtaining request information of the object storage request from a first request queue corresponding to the elastic device through the lossless transmission device, transferring the object storage request from the first request queue to a second request queue corresponding to the transmission process of the data processing unit according to the request information, and sending the object storage request to a backend storage warehouse through the transmission process according to the second request queue; If the data storage message is an object storage response, the object storage response is obtained from the backend storage warehouse through the transmission process, and the response information of the object storage response is obtained from the first response queue corresponding to the transmission process through the lossless transmission device. According to the response information, the object storage response is moved from the first response queue to the second response queue corresponding to the elastic device.
3. The method according to claim 2, characterized in that The abnormal recovery instruction for the data processing unit is an abnormal recovery instruction for the transmission process, and the response to the abnormal recovery instruction for the data processing unit, if it is determined according to the storage information that the target message in the data storage message has not been sent, sending the target message to the second device, includes: In response to an abnormal recovery instruction for the transmission process, starting the transmission process; Calling the traffic pause interface of the lossless transmission device through the transmission process, so that the lossless transmission device pauses the operation of transferring the object storage request from the first request queue to the second request queue corresponding to the transmission process of the data processing unit according to the request information; If it is determined according to the storage information that there is an unsent target message in the transmission process, the target message is restored in the queue of the transmission process so that the transmission process sends the target message to the second device according to the queue of the transmission process; wherein, if the target message is a target request in the object storage request, the queue of the transmission process is the second request queue, and if the target message is a target response in the object storage response, the queue of the transmission process is the first response queue; After recovering the target message in the queue of the transmission process, the method further comprises: The traffic recovery interface of the lossless transmission device is called by the transmission process, so that the lossless transmission device recovers the operation of transferring the object storage request from the first request queue to the second request queue corresponding to the transmission process of the data processing unit according to the request information.
4. The method according to claim 3, characterized in that The abnormal recovery instruction for the transmission process is a hot upgrade completion instruction for the transmission process. In response to the abnormal recovery instruction for the transmission process, before starting the transmission process, the method further includes: In response to a hot upgrade start instruction for the transmission process, calling a traffic pause interface of the lossless transmission device to cause the lossless transmission device to pause the operation of transferring the object storage request from the first request queue to the second request queue corresponding to the transmission process of the data processing unit according to the request information; When it is determined according to the storage information that there is no unsent data storage message in the transmission process, the transmission process is exited so as to update the configuration information of the transmission process.
5. The method according to claim 2, characterized in that: The abnormal recovery instruction for the data processing unit is a hot upgrade completion instruction for the lossless transmission device, and the response to the abnormal recovery instruction for the data processing unit, if it is determined according to the storage information that the target message in the data storage message has not been sent, sending the target message to the second device, includes: In response to a hot upgrade completion instruction for the lossless transmission device, configuring target configuration information and the to-be-processed information in the storage information for the lossless transmission device, wherein the data storage message corresponding to the to-be-processed information is not moved by the lossless transmission device; After configuring the target configuration information for the lossless transmission device and the information to be processed in the acquired request information, the method further includes: Restore the operation of obtaining the request information of the object storage request from the first request queue corresponding to the elastic device through the lossless transmission device, and the operation of obtaining the response information of the object storage response from the first response queue corresponding to the transmission process through the lossless transmission device.
6. The method according to claim 5, characterized in that Before configuring the target configuration information and the to-be-processed information in the storage information for the lossless transmission device in response to the hot upgrade completion instruction for the lossless transmission device, the method further includes: In response to a hot upgrade start instruction for the lossless transmission device, suspending the operation of obtaining the request information of the object storage request from the first request queue corresponding to the elastic device through the lossless transmission device, and the operation of obtaining the response information of the object storage response from the first response queue corresponding to the transmission process through the lossless transmission device; If it is determined according to the storage information that the lossless transmission device does not contain storage information corresponding to the data storage message that has not been transported, then saving the target configuration information of the lossless transmission device so as to perform a hot upgrade on the lossless transmission device; If a preset time has passed after the hot upgrade start instruction is obtained, it is determined based on the storage information that there is storage information corresponding to the data storage message that has not been transferred in the lossless transmission device, then the storage information corresponding to the data storage message that has not been transferred is used as information to be processed, and the target configuration information of the lossless transmission device and the information to be processed are saved to perform a hot upgrade on the lossless transmission device.
7. The method according to any one of claims 2 to 6, characterized in that: The method further comprises: Acquire a request flow rate of the object storage request according to the first request queue, and acquire a response flow rate of the object storage response according to the first response queue; If the request flow rate and the response flow rate satisfy a first rate difference condition, suspending the operation of transferring the object storage request from the first request queue to the second request queue corresponding to the transmission process of the data processing unit according to the request information; If the request traffic rate and the response traffic rate do not satisfy the first rate difference condition, the operation of transferring the object storage request from the first request queue to the second request queue corresponding to the transmission process of the data processing unit according to the request information is restored.
8. The method according to claim 7, characterized in that The method further comprises: If the request traffic rate and the response traffic rate satisfy a second rate difference condition, suspending the operation of obtaining the request information of the object storage request from the first request queue corresponding to the elastic device through the lossless transmission device; If the request traffic rate and the response traffic rate do not satisfy the second rate difference condition, the operation of obtaining the request information of the object storage request from the first request queue corresponding to the elastic device through the lossless transmission device is resumed.
9. The method according to any one of claims 2 to 6, characterized in that: The method of transferring the object storage request from the first request queue to a second request queue corresponding to the transmission process of the data processing unit according to the request information includes: Determine, according to the request information and the queue mapping relationship, a second request queue corresponding to the object storage request in the request queue of the transmission process of the data processing unit; Transferring the object storage request from the first request queue to the second request queue; The transferring the object storage response from the first response queue to a second response queue corresponding to the elastic device according to the response information includes: Determining, according to the response information and the queue mapping relationship, a second response queue corresponding to the object storage response in the response queue of the elastic device; The object store response is moved from the first response queue to the second response queue.
10. The method according to claim 9, characterized in that The transferring the object storage request from the first request queue to the second request queue includes: Transferring the object storage request from the first request queue to the second request queue through direct memory access; The moving the object storage response from the first response queue to the second response queue includes: The object store response is moved from the first response queue to the second response queue via direct memory access.
11. The method according to any one of claims 2 to 6, characterized in that: Before obtaining the request information of the object storage request from the first request queue corresponding to the elastic device through the lossless transmission device, the method further includes: Obtaining, through the lossless transmission device, a storage notification from the server indicating the first request queue; Before obtaining the response information of the object storage response from the first response queue corresponding to the transmission process through the lossless transmission device, the method further includes: A response notification indicating the first response queue is obtained from the transmission process through the lossless transmission device.
12. A storage control device, characterized in that: Applied to a data processing unit, the data processing unit includes an elastic device corresponding to a server, and the device includes: a request acquisition unit, configured to acquire an object storage request from the server through the elastic device; A sending unit, configured to obtain storage information of a data storage message from a first device, and send the data storage message to a second device according to the storage information; wherein, when the first device is the elastic device, the second device is a backend storage warehouse, and the data storage message is the object storage request; when the first device is the backend storage warehouse, the second device is the elastic device, and the data storage message includes an object storage response corresponding to the object storage request; the backend storage warehouse is configured to perform storage processing according to the object storage request, and generate an object storage response corresponding to the object storage request according to the storage processing result; An exception response unit is used to respond to the exception recovery instruction for the data processing unit, and if it is determined according to the storage information that the target message in the data storage message has not been sent, send the target message to the second device.
13. A computer device, characterized in that: The computer device comprises a processor and a memory: The memory is used to store a computer program and transmit the computer program to the processor; The processor is used to execute the storage control method described in any one of claims 1 to 11 according to the instructions in the computer program.
14. A computer-readable storage medium, characterized in that: The computer-readable storage medium is used to store a computer program, and the computer program is used to execute the storage control method according to any one of claims 1 to 11.
15. A computer program product comprising a computer program, characterized in that When it is run on a computer device, the computer device is enabled to execute the storage control method described in any one of claims 1 to 11.
Citation Information
Patent Citations
Data processing system, method and controller
CN115576661A
Storage breakpoint resuming method and system on intelligent network card
CN116107805A
Data migration method and device and electronic equipment
CN116185553A
