Data processing method, electronic equipment, medium and program product

Abstract

The embodiment of the invention discloses a data processing method, electronic equipment, a medium and a program product. The method comprises the following steps: creating a first virtual device corresponding to a mechanical hard disk for a system through the mechanical hard disk, and registering the first virtual device as a back-end device in a cache management device; creating a second virtual device corresponding to the solid state disk for the system through the solid state disk, and registering the second virtual device as a cache device in the cache management device; and creating a third virtual device for the system through the mechanical hard disk, and taking the third virtual device as a resource access entrance for data processing. According to the scheme, the third virtual equipment with higher performance can be externally provided as a resource access entry under the condition that a rear end supports rapid cache storage by using a cache technology, so that the input, output and transmission rates of data are improved, the advantages of the cache technology are further exerted, and the resource utilization rate is improved. And under dual acceleration of cache storage and high-speed transmission, the data processing efficiency is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data processing, and in particular to a data processing method, an electronic device, a medium and a program product. BACKGROUND

[0002] There are mainly two types of storage devices on the market at present: solid state drives (SSDs) and hard disk drives (HDDs). There are significant differences between the two in terms of technical principles, performance, durability, power consumption and application scenarios. SSDs are becoming the first choice of more and more users due to their excellent performance, stable durability and good user experience. HDDs are still indispensable in certain scenarios due to their large capacity and low cost.

[0003] At present, in the case of using a mechanical hard disk for data processing, the disk array created by the mechanical hard disk provides a SCSI device as a virtual device to the outside, so as to use the disk array resource by the upper layer. However, the processing speed of this data processing architecture is limited, and it is difficult to meet the real-time requirements of current data processing. SUMMARY

[0004] The embodiments of the present application provide a data processing method, an electronic device, a medium and a program product to double improve the data processing speed from the data access level and the data level.

[0005] According to an aspect of the present application, a data processing method is provided, which comprises:

[0006] creating, by a mechanical hard disk, a first virtual device corresponding to the mechanical hard disk to a system, and registering the first virtual device as a backend device in a cache management device;

[0007] creating, by a solid state disk, a second virtual device corresponding to the solid state disk to the system, registering the second virtual device as a cache device in the cache management device, and binding the backend device and the cache device; wherein the transmission performance and input / output performance of the second virtual device are higher than those of the first virtual device;

[0008] creating, by the mechanical hard disk, a third virtual device to the system, taking the third virtual device as a resource access entry for data processing, and performing data transmission based on the resource access entry and data processing based on the first virtual device and the second virtual device; wherein the transmission performance and input / output performance of the third virtual device are higher than those of the first virtual device.

[0009] According to an aspect of the present application, a data processing apparatus is provided, which comprises:

[0010] a first virtual device creation module, configured to create a first virtual device corresponding to the mechanical hard disk by the mechanical hard disk, and register the first virtual device as a back-end device in a cache management device;

[0011] a second virtual device creation module, configured to create a second virtual device corresponding to the solid state disk by the solid state disk, register the second virtual device as a cache device in the cache management device, and bind the back-end device and the cache device; wherein the transmission performance and input / output performance of the second virtual device are higher than those of the first virtual device;

[0012] a third virtual device creation module, configured to create a third virtual device by the mechanical hard disk, and access the third virtual device as a resource access entry for data processing, so as to perform data transmission based on the resource access entry and data processing based on the first virtual device and the second virtual device; wherein the transmission performance and input / output performance of the third virtual device are higher than those of the first virtual device.

[0013] According to another aspect of the present application, an electronic device is provided, which comprises:

[0014] at least one processor; and

[0015] a memory connected with the at least one processor in communication; wherein

[0016] the memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor to enable the at least one processor to perform the data processing method of any embodiment of the present application.

[0017] According to another aspect of the present application, a computer readable storage medium is provided, which stores computer instructions for enabling a processor to perform the data processing method of any embodiment of the present application when executed by the processor.

[0018] According to another aspect of the present application, a computer program product is provided, which comprises a computer program for enabling a processor to perform the data processing method of any embodiment of the present application when executed by the processor.

[0019] The technical scheme of the embodiment of the application creates a first virtual device corresponding to the mechanical hard disk to a system through the mechanical hard disk, registers the first virtual device as a back-end device in a cache management device; creates a second virtual device corresponding to the solid state disk to the system through the solid state disk, registers the second virtual device as a cache device in the cache management device, and binds the back-end device and the cache device; wherein the transmission performance and the input / output performance of the second virtual device are higher than those of the first virtual device, and the above scheme can realize acceleration at the data access and read / write level, and improve the data processing capability after receiving data. A third virtual device is created to the system through the mechanical hard disk, the third virtual device is used as a resource access entrance for data processing, data transmission is performed based on the resource access entrance, and data processing is performed based on the first virtual device and the second virtual device; wherein the transmission performance and the input / output performance of the third virtual device are higher than those of the first virtual device, so that the high-speed cache technology is further played under the high-speed transmission of the third virtual device, the speed of receiving data is further improved at the data receiving level, double acceleration at the data receiving and data processing level is realized, and the data processing efficiency is effectively improved.

[0020] It should be understood that the content described in this part is not intended to identify the key or important features of the embodiments of the application, nor is it used to limit the scope of the application. Other features of the application will become apparent from the following description. BRIEF DESCRIPTION OF DRAWINGS

[0021] In order to more clearly illustrate the technical solutions in the embodiments of the application, the drawings needed in the embodiment description will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the application, and other drawings can be obtained by those skilled in the art without creative labor.

[0022] Figure 1 A flowchart of a data processing method provided by an embodiment of the application;

[0023] Figure 2 A flowchart of a data processing method provided by another embodiment of the application;

[0024] Figure 3 A flowchart of a specific implementation provided by the application;

[0025] Figure 4 A flowchart of a data access process provided by the application;

[0026] Figure 5 A structural schematic diagram of a data processing device provided by an embodiment of the application;

[0027] Figure 6 A structural schematic diagram of an electronic device is provided for an embodiment of the present application. DETAILED DESCRIPTION

[0028] In order to enable persons skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be described clearly and completely below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by persons skilled in the art without creative work should fall within the scope of protection of the present application.

[0029] It should be noted that the terms "first", "second", "third", "fourth", "actual", "preset" and the like in the specification and claims of the present application and the above drawings are used to distinguish similar objects, and do not necessarily have to describe a specific order or sequence. It should be understood that the data thus used can be interchanged under appropriate circumstances, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, system, product or device that includes a series of steps or units does not have to be limited to those steps or units clearly listed, but can include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0030] Figure 1 A flowchart of a data processing method is provided for an embodiment of the present application. The embodiment of the present application can be applicable to the case of processing data based on a mechanical hard disk and a solid state disk. The method can be executed by a data processing device, which can be realized in the form of hardware and / or software, and can be configured in an electronic device. As shown in the figure, the method comprises: Figure 1

[0031] S110, creating a first virtual device corresponding to the mechanical hard disk by the mechanical hard disk, and registering the first virtual device as a back-end device in a cache management device.

[0032] ​The system type is not limited, for example, it can be a Linux system, a windows system, an Android system, etc. The mechanical hard disk can be one mechanical hard disk or a disk array composed of multiple mechanical hard disks. In the case of multiple disk arrays, it presents a mechanical hard disk storage device externally. The first virtual device is a virtual device presented externally corresponding to the mechanical hard disk, which is a logical device equivalent to the mechanical hard disk at the software level. The first virtual device as a virtualized storage controller can enable the virtual machine to access the local physical components and the storage device, that is, to access the corresponding mechanical hard disk. The cache management device is a core component responsible for coordinating data exchange between the cache and the main memory in the system, and its core function is to optimize system performance through intelligent data management strategies. In the business scenario of the embodiments of the present application, the cache manager can be a manager of an acceleration access technology introduced in data processing, such as a manager of the Bcache technology used in the Linux system, a manager of the PrimoCache technology used in the windows system, etc. Bcache is a cache technology at the block device layer of the Linux system kernel, which significantly improves the performance of the storage system by using a high-speed storage device (such as an SSD) as a cache for a low-speed storage device (such as an HDD). Similar technologies can also be dm-cache technology, LVM cache technology, etc. In the above cache technologies, it is generally allowed to use a high-speed storage device as a cache device to temporarily store hot data with high access frequency, improve access rate, reduce direct access to the backend device to improve overall performance, and use a low-speed storage device as a backend device to realize low-speed and large-capacity storage and store all data persistently.

[0033] In the embodiments of the present application, in the case of installing a mechanical hard disk, a first virtual device corresponding to the mechanical hard disk can be created for the system through the mechanical hard disk, and the first virtual device serves as a logical device of the mechanical hard disk. The first virtual device is registered as a backend device in the cache manager, that is, the corresponding role of the first virtual device is marked in the cache technology, as the role of the backend device, responsible for persistently storing data. Based on the access performance of the mechanical hard disk, the created first virtual device needs to be a virtual device adapted to it, which is generally a SCSI device, so that the rate of data transmission through the SCSI device matches the rate of data processing of the mechanical hard disk, avoiding the problem of data accumulation due to rate mismatch.

[0034] S120, a second virtual device corresponding to the solid state disk is created for the system through the solid state disk, the second virtual device is registered as a cache device in the cache management device, and the backend device and the cache device are bound; wherein the transmission performance and input / output performance of the second virtual device are higher than those of the first virtual device.

[0035] The solid state disk is a storage device based on a flash chip, has core advantages of high-speed reading and writing, low power consumption, anti-shock and anti-falling, silent operation, etc., but has relatively high price and limited capacity. In the embodiment of the application, in the case of loading the solid state disk, the second virtual device corresponding to the solid state disk is created by the solid state disk to the system, and the second virtual device is used as a virtualization logic device of the solid state disk. Based on the high access performance of the solid state disk, the second virtual device needs to adapt to the performance of the solid state disk to exert the high-speed advantage of the solid state disk. Since the solid state disk has higher processing rate than the mechanical disk, the transmission performance and input / output performance of the second virtual device adapted to the solid state disk are higher than those of the first virtual device, and the second virtual device can be set as an NVMe device. The second virtual device is registered as a cache device in the cache management device, that is, the corresponding role of the second virtual device is marked in the cache technology, as the role of the cache device, so as to facilitate the subsequent caching in the solid state disk through the second virtual device.

[0036] In the embodiment of the application, after the first virtual device is registered as a backend device and the second virtual device is registered as a cache device, the backend device and the cache device are bound, that is, the logical association between the backend device and the cache device is established, a virtual cache device is formed, and the data read / write process is uniformly managed by the cache management device in the system. After binding, the cache device stores hot data of high-frequency access, and the backend device retains a complete data copy, and the two devices work cooperatively through a dynamic strategy to balance performance acceleration and data persistence.

[0037] S130, creating a third virtual device through the mechanical disk to the system, using the third virtual device as a resource access entrance of data processing, transmitting data based on the resource access entrance and processing data based on the first virtual device and the second virtual device; wherein the transmission performance and input / output performance of the third virtual device are higher than those of the first virtual device.

[0038] The transmission performance and input / output performance of the third virtual device need to be higher than those of the first virtual device, so as to further improve the speed at the data transmission level based on the cache technology to improve the data processing capability, adapt to the data processing capability, and cooperatively improve the data processing speed.

[0039] Specifically, a third virtual device is created in the system via a mechanical hard drive. Since a first and second virtual device have already been established, serving as the backend device and cache device respectively, data processing capabilities are effectively improved at the data processing level. In this case, the high-performance third virtual device is compatible with the data processing capabilities of current mechanical hard drives and solid-state drives, meeting the conditions for creating a third virtual device. After creating the third virtual device, it is used as a resource access point for data transmission, enabling high-speed data transmission and interaction. Data caching and persistent storage, or data retrieval and reading, are implemented in the first and second virtual devices, thereby accelerating both data transmission and data access capabilities and effectively improving data processing speed.

[0040] The technical solution of this application embodiment creates a first virtual device corresponding to the mechanical hard drive in the system, and registers the first virtual device as a backend device in the cache management device; it also creates a second virtual device corresponding to the solid-state drive in the system, registers the second virtual device as a cache device in the cache management device, and binds the backend device to the cache device. The second virtual device has higher transmission and input / output performance than the first virtual device. This solution accelerates data access and read / write operations, improving data processing capabilities after data is received. A third virtual device is created in the system through the mechanical hard drive, serving as a resource access point for data processing. Data is transmitted through this resource access point, and data processing is performed based on the first and second virtual devices. The third virtual device has higher transmission and input / output performance than the first virtual device, further leveraging the high-speed transmission of the third virtual device to enhance caching technology. This further improves data reception speed, achieving dual acceleration in both data reception and processing, effectively improving data processing efficiency.

[0041] As a non-limiting implementation, the third virtual device is used as a resource access point for data processing, including:

[0042] Create a configuration file in the system directory and configure the association with the third virtual device in the configuration file so that the associated third virtual device is used as the resource access entry for data processing when the system reads the configuration file.

[0043] For example, when data processing is performed locally on the system, it is necessary to establish an association with a third-party virtual device on the system's local machine, thereby enabling the third-party virtual device to serve as a resource access point for data processing. Specifically, a configuration file is created in the system directory, and the association with the third-party virtual device is configured in the configuration file. That is, during the configuration of the resource access point in the configuration file, the relevant parameters of the third-party virtual device are written into the configuration file as the resource access point. When the system reads the configuration file to determine the resource access point, it uses the third-party virtual device as the resource access point. This solution, by configuring the association with the third-party virtual device in the system kernel's configuration file, enables the external provision of the third-party virtual device, allowing for high-speed data transmission through data interaction based on the third-party virtual device.

[0044] As a non-limiting implementation, the third virtual device is used as a resource access point for data processing, including:

[0045] A network object is created through the communication module in the system, and an association is established between the network object and the third virtual device, so that when a communication connection is established between the client and the network object, the associated third virtual device serves as the resource access entry point for data processing.

[0046] In this embodiment, there are situations where data access is achieved through communication with other devices such as clients. In such cases, a network object can be pre-created using the system's communication module. In this embodiment, a network object refers to a virtual entity connected to the internet and capable of interacting with other entities, serving as a virtual entity to communicate and interact with external entities. After establishing the network object, an association is established between the network object and a third virtual device. When the client and the network object establish a communication connection, the associated third virtual device serves as the resource access entry point for data processing.

[0047] As a non-limiting implementation, the third virtual device is used as a resource access point for data processing, including:

[0048] If the client communicates through the first protocol corresponding to the first virtual device, then at least two network objects are created through the communication module in the system, and the association between each network object and the third virtual device is established, so that when the client and the at least two network objects establish a communication connection, the associated third virtual device is used as the resource access entry point for data processing.

[0049] In this embodiment, if the client communicates via the first protocol corresponding to the first virtual device, the performance of the first protocol is lower than that of the second protocol corresponding to the second virtual device. The third virtual device uses the second protocol for data transmission, and the solid-state drive (SSD) also uses the second protocol of the second virtual device for data access and processing. In this case, the data transmission performance between the client and the network object may not match the data processing performance between the hard disk drive (HDD) and the SSD, resulting in a mismatch between data transmission capability and data processing capability, and failing to fully utilize the advantages of data processing capability. Therefore, at least two network objects can be created through the communication module in the system, and an association can be established between each network object and the third virtual device. Thus, when the client establishes a communication connection with at least two network objects, the associated third virtual device serves as the resource access entry point for data processing. When data is transmitted in parallel through at least two network objects, the data transmission speed of the third virtual device and the data processing speed of the hard disk can be adapted, thereby adapting to the backend data processing capability, achieving dual acceleration, and effectively improving the efficiency of data processing.

[0050] As a non-limiting implementation, the process of determining the number of the at least two network objects includes:

[0051] Determine the performance data of the first protocol and the performance data of the second protocol; wherein the second protocol is the protocol corresponding to the second virtual device; the performance data includes transmission rate or throughput;

[0052] The ratio of the performance data of the second protocol to the performance data of the first protocol is used as the number of the at least two network objects.

[0053] In this embodiment, the first protocol is the protocol corresponding to the first virtual device, and the second protocol is the protocol corresponding to the second virtual device. The performance data of the second protocol is higher than that of the first protocol. Specifically, in determining the number of at least two network objects, the performance data of the first protocol and the second protocol can be determined. The performance data can be the transmission rate or throughput; that is, the transmission rate of the first protocol is compared with the transmission rate of the second protocol, or the throughput of the first protocol is compared with the throughput of the second protocol. The performance data of the second protocol is higher than that of the first protocol. The ratio of the performance data of the second protocol to that of the first protocol is used as the number of at least two network objects. For example, if the transmission rate of the first protocol is Abps and the transmission rate of the second protocol is Bbps, assuming B / A is 10, then the number of at least two network objects is determined to be 10. Ten network objects are created through the communication module in the system, and all ten network objects are associated with the third virtual device. The client communicates with the ten network objects and transmits data in parallel. After being received by the third virtual device, the transmission rates are matched, effectively improving the data transmission rate.

[0054] In practical applications based on the above embodiments, the client can communicate based on either the first protocol or the second protocol; no limitation is made here. During client communication based on the first protocol, only one network object can be created, and the application can still function normally.

[0055] Figure 2 This is a flowchart illustrating a data processing method according to another embodiment of this application. This embodiment is an optimization based on the above embodiment; solutions not described in detail in this embodiment are found in the above embodiment. Figure 2 As shown, the method in this embodiment of the application specifically includes the following steps:

[0056] S210. Create a first virtual device corresponding to the mechanical hard drive in the system through the mechanical hard drive, and register the first virtual device as a backend device in the cache management device.

[0057] S220. Create a second virtual device corresponding to the solid-state drive in the system through the solid-state drive, register the second virtual device as a cache device in the cache management device, and bind the backend device to the cache device; wherein, the transmission performance and input / output performance of the second virtual device are higher than those of the first virtual device.

[0058] S230. Create a third virtual device in the system through the mechanical hard disk, and use the third virtual device as a resource access point for data processing, so as to transmit data based on the resource access point and perform data processing based on the first virtual device and the second virtual device; wherein, the transmission performance and input / output performance of the third virtual device are higher than those of the first virtual device.

[0059] S240. Construct a first block device request adapted to the third virtual device based on the resource access request, and send the first block device request to the third virtual device.

[0060] For example, when the system receives a resource access request, since the third virtual device acts as the resource access entry point to receive data, a first device request adapted to the third virtual device is constructed based on the resource access request, so that the third virtual device can recognize and respond to it. The system kernel sends the first device request to the third virtual device, so that the first virtual device receives and processes the first device request, which is equivalent to processing the resource access request.

[0061] In this embodiment of the application, constructing the first block device request for the third virtual device adaptation based on the resource access request includes:

[0062] If the resource access request is a local resource access request of the system, then the system parses and processes the resource access request, and constructs the first device request adapted by the third virtual device based on the parsing result.

[0063] If the resource access request is a resource access request protocol packet sent by the client, the system processes the protocol packet and constructs the first device request for the third virtual device adaptation based on the processing result.

[0064] For example, if the resource access request is a locally generated resource access request, the system can parse and process the resource access request to extract the data access requirements, that is, to extract the statements that the system can recognize. Based on the parsed structure, the first device request adapted by the third virtual device is constructed so that the third virtual device can recognize and process the first device request.

[0065] If the resource access request is a resource access request protocol packet sent by the client, the system needs to parse the protocol packet to extract the request content, and then construct the first device request for the third virtual device adaptation based on the processed result, so that the third virtual device can identify and process the first device request.

[0066] The above solution can specifically parse and process resource access requests from different sources through the system. Based on the parsed request content, it constructs a first device request that the third virtual device can identify and process, so that the third virtual device can identify and process the first device request, give full play to the high efficiency of the third virtual device, and achieve rapid processing of the first device.

[0067] S250. The first device request is converted into a second device request adapted by the cache device through the third virtual device, and the second device request is processed by the cache device and the processing result is returned to the system.

[0068] In this embodiment, based on caching technology, after receiving the first device request, the third virtual device needs to access the cache device first for processing. Therefore, the third virtual device converts the first device request into a second device request adapted for the cache device, enabling the cache device to recognize and respond to it. The cache device processes the second device request, responds to its request content, obtains the processing result, and returns the result to the system. For example, if the second device request retrieves data identified as "pic," the cache device responds by performing a local retrieval and returns the retrieved data identified as "pic" to the system.

[0069] S260. The processing result is written back to the backend device through the cache device in the background.

[0070] For example, after returning the processing result to the system, the caching device needs to write the processing result back to the backend device for persistent storage. Since the caching device has already processed the response from the second device and returned the processing result to the system, and the caching device has fast data transmission, throughput, and data processing capabilities, the response is fast from the requester's perspective. Furthermore, the process of returning the processing result to the system in the background and then writing it back to the backend device does not consume the time required to return the processing result to the system, and does not affect the speed of data processing by the caching device. Therefore, data processing remains fast from the requester's perspective.

[0071] This application provides a data processing method. Based on a resource access request, a first device request adapted to a third virtual device is constructed and sent to the third virtual device. The third virtual device converts the first device request into a second device request adapted to a cache device, processes the second device request through the cache device, and returns the processing result to the system. In the background, the cache device writes the processing result back to the backend device. This solution, based on high-speed caching technology, first passes the resource access request to the cache device for rapid processing, quickly returns the processing result to the system, and then writes the processing result back to the backend device for persistent storage. This ensures that the data processing speed responding to the requester is not affected while guaranteeing data write-back. Furthermore, using the third virtual device as the resource access entry point further improves data access speed, adapts to the backend's data processing capabilities, and synergistically improves overall data processing efficiency.

[0072] This application provides a specific implementation method, such as... Figure 3 As shown, the specific process is as follows:

[0073] (1) The disk array creates SCSI devices to the Linux system kernel.

[0074] (2) The disk array registers with the Bcache manager and registers the SCSI device as a backing device.

[0075] (3) Solid-state drives create NVMe devices to the Linux system kernel.

[0076] (4) The disk array registers with the Bcache manager and registers the NVMe device as a cache device.

[0077] (5) Bind the cache device to the backing device;

[0078] (6) The disk array creates an NVMe device in the Linux system kernel and provides the device to the outside world as a resource access point.

[0079] In this context, the SCSI device is the first virtual device mentioned in the above embodiments, the NVMe device created in (3) is the second virtual device, and the NVMe device created in (6) is the third virtual device. The Bcache manager is the cache manager. The backing device is the backend device, and the cache device is the cache device.

[0080] The NVME device creation process is described in detail below:

[0081] (1) Define queues and NVME block device pointers

[0082] struct request_queue *queue=NULL;

[0083] struct gendisk *disk=NULL;

[0084] (2) Create an IO queue

[0085] queue=blk_alloc_queue(GFP_KERNEL).

[0086] (3) Set the IO receive callback function:

[0087] The `blk_qc_t raid_bio_request_fn(struct request_queue*q, structbio*bio)` callback function is implemented according to the actual business requirements.

[0088] blk_queue_make_request(queue, raid_bio_request_fn).

[0089] (4) Set the maximum data volume for a single I / O operation

[0090] blk_queue_max_hw_sectors(bdev->queue,1024), this size can be adjusted as needed.

[0091] (5) Set the size of physical blocks and logical blocks

[0092] uint64_t page_size=4×1024;

[0093] blk_queue_physical_block_size(bdev->queue,page_size);

[0094] uint64_t logic_blk_size = 4×1024;

[0095] blk_queue_logical_block_size(bdev->queue, logic_blk_size);

[0096] These two parameters can be adjusted as needed to achieve the best performance.

[0097] (6) Set the block device as a non-rotating device

[0098] blk_queue_flag_set(QUEUE_FLAG_NONROT, bdev->queue);

[0099] (7) Set the DISCARD property and its length.

[0100] This attribute can be set as needed, and the setting method is as follows:

[0101] blk_queue_flag_set(QUEUE_FLAG_DISCARD, queue);

[0102] blk_queue_max_discard_sectors(queue, 1024), parameter 2 is the maximum length of a single discard, which can be set to an appropriate value according to business needs.

[0103] (8) Create NVME block device

[0104] #define NUMA_NO_NODE (-1)

[0105] disk = alloc_disk_node(1<<6, NUMA_NO_NODE), parameter 1 indicates the number of minor device numbers supported, and this value can be set appropriately according to your own needs.

[0106] (9) Bind NVME block devices to queues and activate NVME block devices.

[0107] disk->queue = queue;

[0108] add_disk(disk);

[0109] At this point, the NVME block device creation is complete.

[0110] In the above content, * represents a pointer. In specific data processing procedures, such as... Figure 4 As shown: The IO request reaches the Linux kernel through the network. The Linux kernel then parses and reconstructs the first device request and sends it to the NVMe device. The NVMe device receives the first device request, parses and reconstructs the second device request, and sends it to the cache device of bcache. After receiving the IO request, the cache device responds directly and returns the response result without waiting for it to be sent to the backing device of bcache.

[0111] Figure 5This is a schematic diagram of a data processing apparatus provided in an embodiment of this application. The apparatus can execute the data processing method provided in any embodiment of this application, and possesses the corresponding functional modules and beneficial effects for executing the method. For example... Figure 5 As shown, the device includes:

[0112] The first virtual device creation module 310 is used to create a first virtual device corresponding to the mechanical hard drive in the system through the mechanical hard drive, and register the first virtual device as a backend device in the cache management device.

[0113] The second virtual device creation module 320 is used to create a second virtual device corresponding to the solid-state drive in the system through the solid-state drive, register the second virtual device as a cache device in the cache management device, and bind the backend device to the cache device; wherein, the transmission performance and input / output performance of the second virtual device are higher than those of the first virtual device;

[0114] The third virtual device creation module 330 is used to create a third virtual device in the system through a mechanical hard disk, and use the third virtual device as a resource access entry for data processing, so as to transmit data based on the resource access entry and perform data processing based on the first virtual device and the second virtual device; wherein, the transmission performance and input / output performance of the third virtual device are higher than those of the first virtual device.

[0115] In this embodiment of the application, the third virtual device creation module 330 uses the third virtual device as a resource access entry point for data processing, including:

[0116] Create a configuration file in the system directory and configure the association with the third virtual device in the configuration file so that the associated third virtual device is used as the resource access entry for data processing when the system reads the configuration file.

[0117] In this embodiment of the application, the third virtual device creation module 330 uses the third virtual device as a resource access entry point for data processing, including:

[0118] A network object is created through the communication module in the system, and an association is established between the network object and the third virtual device, so that when a communication connection is established between the client and the network object, the associated third virtual device serves as the resource access entry point for data processing.

[0119] In this embodiment of the application, the third virtual device creation module 330 uses the third virtual device as a resource access entry point for data processing, including:

[0120] If the client communicates through the first protocol corresponding to the first virtual device, then at least two network objects are created through the communication module in the system, and the association between each network object and the third virtual device is established, so that when the client and the at least two network objects establish a communication connection, the associated third virtual device is used as the resource access entry point for data processing.

[0121] In this embodiment of the application, the device further includes a quantity determination module, used for:

[0122] Determine the performance data of the first protocol and the performance data of the second protocol; wherein the second protocol is the protocol corresponding to the second virtual device; the performance data includes transmission rate or throughput;

[0123] The ratio of the performance data of the second protocol to the performance data of the first protocol is used as the number of the at least two network objects.

[0124] In this embodiment of the application, after using the third virtual device as the resource access entry point for data processing, the device further includes a resource access module, used for:

[0125] Construct a first device request adapted to the third virtual device based on the resource access request, and send the first device request to the third virtual device;

[0126] The first device request is converted into a second device request adapted by the cache device through the third virtual device, and the second device request is processed by the cache device and the processing result is returned to the system.

[0127] The processing result is written back to the backend device through the cache device in the background.

[0128] In this embodiment of the application, the resource access module constructs the first device request adapted by the third virtual device based on the resource access request, including:

[0129] If the resource access request is a local resource access request of the system, then the system parses and processes the resource access request, and constructs the first device request adapted by the third virtual device based on the parsing result.

[0130] If the resource access request is a resource access request protocol packet sent by the client, the system processes the protocol packet and constructs the first device request for the third virtual device adaptation based on the processing result.

[0131] The data processing apparatus provided in this application embodiment can execute a data processing method provided in any embodiment of this application, and has the corresponding functional modules and beneficial effects of executing the method.

[0132] Figure 6 A schematic diagram of an electronic device 10, which can be used to implement embodiments of this application, is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices (such as helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the application described and / or claimed herein.

[0133] like Figure 6 As shown, the electronic device 10 includes at least one processor 11 and a memory, such as a read-only memory (ROM) 12 or a random access memory (RAM) 13, communicatively connected to the at least one processor 11. The memory stores computer programs executable by the at least one processor. The processor 11 can perform various appropriate actions and processes based on the computer program stored in the ROM 12 or loaded from storage unit 18 into the RAM 13. The RAM 13 can also store various programs and data required for the operation of the electronic device 10. The processor 11, ROM 12, and RAM 13 are interconnected via a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0134] Multiple components in electronic device 10 are connected to I / O interface 15, including: input unit 16, such as keyboard, mouse, etc.; output unit 17, such as various types of monitors, speakers, etc.; storage unit 18, such as disk, optical disk, etc.; and communication unit 19, such as network card, modem, wireless data processing transceiver, etc. Communication unit 19 allows electronic device 10 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0135] Processor 11 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. Processor 11 performs the various methods and processes described above, such as data processing methods.

[0136] In some embodiments, the data processing method may be implemented as a computer program tangibly contained in a computer-readable storage medium, such as storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or mounted on electronic device 10 via ROM 12 and / or communication unit 19. When the computer program is loaded into RAM 13 and executed by processor 11, one or more steps of the data processing method described above may be performed. Alternatively, in other embodiments, processor 11 may be configured to perform the data processing method by any other suitable means (e.g., by means of firmware).

[0137] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0138] Computer programs used to implement the methods of this application may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when executed by the processor, the computer programs cause the functions / operations specified in the flowcharts and / or block diagrams to be performed. The computer programs may be executed entirely on a machine, partially on a machine, or as a standalone software package, partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0139] In the context of this application, a computer-readable storage medium can be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, apparatus, or device. A computer-readable storage medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium can be a machine-readable signal medium. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0140] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0141] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or middleware components (e.g., application servers), or frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), blockchain networks, and the Internet.

[0142] A computing system can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a hosting product within the cloud computing service system to address the shortcomings of traditional physical hosts and VPS services, such as high management difficulty and weak business scalability.

[0143] This invention also provides a computer program product, including a computer program that, when executed by a processor, implements the data processing method provided in any embodiment of this application.

[0144] In implementing the computer program product, computer program code for performing the operations of this invention can be written in one or more programming languages ​​or a combination thereof. Programming languages ​​include object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as C or similar languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or it can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0145] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this application can be executed in parallel, sequentially, or in different orders, as long as the desired information of the technical solution of this application can be achieved, and this is not limited herein.

[0146] The specific embodiments described above do not constitute a limitation on the scope of protection of this application. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. A data processing method, characterized in that, The method includes: A first virtual device corresponding to the hard drive is created in the system through the hard drive, and the first virtual device is registered as a backend device in the cache management device; A second virtual device corresponding to the solid-state drive is created in the system through the solid-state drive. The second virtual device is registered as a cache device in the cache management device, and the backend device is bound to the cache device. The transmission performance and input / output performance of the second virtual device are higher than those of the first virtual device. A third virtual device is created in the system using a mechanical hard drive. This third virtual device serves as a resource access point for data processing, enabling data transmission based on this resource access point and data processing based on the first and second virtual devices. The transmission and input / output performance of the third virtual device is higher than that of the first virtual device.

2. The method according to claim 1, characterized in that, Using the third virtual device as a resource access point for data processing includes: Create a configuration file in the system directory and configure the association with the third virtual device in the configuration file so that the associated third virtual device is used as the resource access entry for data processing when the system reads the configuration file.

3. The method according to claim 1, characterized in that, Using the third virtual device as a resource access point for data processing includes: A network object is created through the communication module in the system, and an association is established between the network object and the third virtual device, so that when a communication connection is established between the client and the network object, the associated third virtual device serves as the resource access entry point for data processing.

4. The method according to claim 3, characterized in that, Using the third virtual device as a resource access point for data processing includes: If the client communicates through the first protocol corresponding to the first virtual device, then at least two network objects are created through the communication module in the system, and the association between each network object and the third virtual device is established, so that when the client and the at least two network objects establish a communication connection, the associated third virtual device is used as the resource access entry point for data processing.

5. The method according to claim 4, characterized in that, The process for determining the number of at least two network objects includes: Determine the performance data of the first protocol and the performance data of the second protocol; wherein the second protocol is the protocol corresponding to the second virtual device; the performance data includes transmission rate or throughput; The ratio of the performance data of the second protocol to the performance data of the first protocol is used as the number of the at least two network objects.

6. The method according to claim 1, characterized in that, After using the third virtual device as the resource access point for data processing, the method further includes: Construct a first device request adapted to the third virtual device based on the resource access request, and send the first device request to the third virtual device; The first device request is converted into a second device request adapted by the cache device through the third virtual device, and the second device request is processed by the cache device and the processing result is returned to the system. The processing result is written back to the backend device through the cache device in the background.

7. The method according to claim 6, characterized in that, Constructing the first device request adapted by the third virtual device based on the resource access request includes: If the resource access request is a local resource access request of the system, then the system parses and processes the resource access request, and constructs the first device request adapted by the third virtual device based on the parsing result. If the resource access request is a resource access request protocol packet sent by the client, the system processes the protocol packet and constructs the first device request for the third virtual device adaptation based on the processing result.

8. An electronic device, characterized in that, The electronic device includes: At least one processor; and A memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the data processing method according to any one of claims 1-7.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that cause a processor to execute the data processing method according to any one of claims 1-7.

10. A computer program product, characterized in that, It includes a computer program that, when executed by a processor, implements the data processing method as described in any one of claims 1-7.

Images