Method and device for writing and reading data

Abstract

The embodiment of the present invention provides a method and device for writing and reading data, including: receiving a write request of a first data packet; determining a target mechanical disk when the first data packet is a data packet of the first type; and writing a target amount of data in the first data packet to the target mechanical disk according to the relationship between the first time consumption of writing the data packet to the target mechanical disk N times in a historical period and a preset threshold range. The present invention solves the problem of low efficiency of writing and reading data on mechanical disks in the related art.

Description

Technical Field

[0001] The embodiments of the present invention relate to the field of communications, and in particular, to a method and device for writing and reading data. Background Art

[0002] The performance of a mechanical disk (Hard Disk Drive, HDD) is different when reading and writing. Under normal circumstances, when writing sequentially and with a large write I / O granularity, the disk has fewer arm shifts and addressing actions, while reading data is generally random reading. The head needs to find the target data through a large arm shift addressing action to read it. In this case, the disk spends more time on arm shift addressing, resulting in a decrease in the overall performance of the disk. If there is still write business coming in, the addressing time is further aggravated, resulting in an increase in write time and low efficiency, and a decrease in write performance. The above is a general read and write business. Reading will have a negative impact on writing. If a small file is read, that is, only a small amount of data is read each time, each read requires the disk to re-address, causing the disk to shift its arm and address more frequently, and the actual proportion of time for reading data is even lower. This random read with a small I / O granularity has a greater impact on the performance of the disk, causing a more serious performance degradation.

[0003] There is currently no effective solution to the above problems. Summary of the invention

[0004] The embodiments of the present invention provide a method and device for writing and reading data, so as to at least solve the problem of low efficiency of writing and reading data by mechanical disks in the related art.

[0005] According to one embodiment of the present invention, a method for writing and reading data is provided, comprising: receiving a write request for a first data packet, wherein the data volume of the first data packet is a first data volume; in the case where the first data packet is a first type of data packet, determining a target mechanical disk, wherein the current state of the target mechanical disk is a non-read-only state, the state of the mechanical disk that is currently being read or within a preset time after being read is a read-only state, the mechanical disk in the read-only state is prohibited from being written with data, and the data volume of the first type of data packet is greater than or equal to a preset data volume threshold; based on the relationship between the first time consumption of writing data packets to the target mechanical disk N times in a historical period and a preset threshold range, writing the target data volume in the first data packet to the target mechanical disk, wherein N is a preset value, N is an integer greater than 1, and the target data volume is less than or equal to the first data volume.

[0006] In an exemplary embodiment, before writing the target data volume in the first data packet to the target mechanical disk, the method also includes: determining a target entry matching the first time consumption in a preset relationship table, and determining the target data volume in the target entry; wherein the preset relationship table records multiple entries, the multiple entries include the target entry, the entry includes the preset threshold range and the data volume, different entries include different preset threshold ranges, different entries include different data volumes, the target entry includes a target threshold range and the target data volume, and the first time consumption is within the target threshold range.

[0007] In an exemplary embodiment, the method further includes: in a case where the first data packet is a data packet of the second type, writing the first data packet to the solid-state drive, wherein the data volume of the second type of data packet is less than the preset data volume threshold; in a case where the first data packet is a data packet of the first type and the target data volume is less than the first data volume, determining the data in the first data packet except the target data volume as the remaining data of the first data packet, and writing the remaining data of the first data packet to the solid-state drive.

[0008] In an exemplary embodiment, when the first data packet is a data packet of the second type, writing the first data packet to the solid-state drive includes: when the remaining amount of the second capacity of the solid-state drive is greater than or equal to the first data amount, writing the first data packet to the solid-state drive, wherein the second capacity is the capacity of the solid-state drive for writing the second type of data packet in the current cycle; when the remaining amount of the second capacity of the solid-state drive is less than the first data amount, migrating the second type of data packet in the solid-state drive to the target mechanical disk in the order of the writing time of the second type of data packet in the solid-state drive from far to near, and writing the first data packet to the solid-state drive.

[0009] In an exemplary embodiment, writing the remaining data of the first data packet to the solid-state drive includes: when the remaining amount of the first capacity of the solid-state drive is greater than or equal to the amount of the remaining data, writing the remaining data to the solid-state drive, wherein the first capacity is the capacity of the solid-state drive used to write the first type of data packet in the current cycle; when the remaining amount of the first capacity of the solid-state drive is less than the amount of the remaining data, migrating the first type of data packet in the solid-state drive to the target mechanical disk in the order of the writing time of the first type of data packet in the solid-state drive from far to near, and writing the remaining data to the solid-state drive.

[0010] In an exemplary embodiment, the method further includes: counting the number of first type data packets written in a cycle before the current cycle to obtain a first value; counting the number of second type data packets written in the previous cycle to obtain a second value; and allocating usage of the solid-state drive based on the relationship between the first value and the second value.

[0011] In an exemplary embodiment, the solid-state drive is allocated for use based on the relationship between the first value and the second value, including: when the first value is greater than or equal to the second value, the solid-state drive is allocated to a first capacity and a second capacity, wherein the first capacity is used to cache the first type of data packets, the second capacity is used to cache the second type of data packets, and the ratio of the first capacity to the second capacity is a first ratio, and the first ratio is less than 1; when the first value is less than the second value, the solid-state drive is allocated to a third capacity and a fourth capacity, wherein the third capacity is used to cache the first type of data packets, the fourth capacity is used to cache the second type of data packets, and the ratio of the third capacity to the fourth capacity is a second ratio, and the first ratio is greater than the second ratio.

[0012] In an exemplary embodiment, the method further includes: in response to a read request for a second data packet, searching for data segments of the second data packet in a local cache, a solid-state drive, and a mechanical hard disk; when data segments of the second data packet are found in the target mechanical hard disk, reading the data segments of the second data packet in the target mechanical hard disk and setting the target mechanical hard disk to a read-only state.

[0013] In an exemplary embodiment, data segments of the second data packet are read in the target mechanical hard disk, and the target mechanical hard disk is set to a read-only state, including: if the end moment of reading the data segments of the second data packet in the target mechanical hard disk is a first moment, the target mechanical hard disk is set to a non-read-only state at a second moment, wherein the interval between the first moment and the second moment is a preset duration.

[0014] According to another embodiment of the present invention, a device for writing and reading data is provided, including: a receiving module for receiving a write request for a first data packet, wherein the data volume of the first data packet is a first data volume; a determining module for determining a target mechanical disk when the first data packet is a first type of data packet, wherein the current state of the target mechanical disk is a non-read-only state, the state of the mechanical disk currently being read or within a preset time after being read is a read-only state, the mechanical disk in the read-only state is prohibited from being written with data, and the data volume of the first type of data packet is greater than or equal to a preset data volume threshold; a writing module for writing a target data volume in the first data packet to the target mechanical disk based on a relationship between a first time consumption of writing data packets to the target mechanical disk N times in a historical period and a preset threshold range, wherein N is a preset value, N is an integer greater than 1, and the target data volume is less than or equal to the first data volume.

[0015] According to yet another embodiment of the present invention, a computer-readable storage medium is provided, wherein a computer program is stored in the computer-readable storage medium, wherein the computer program implements the steps of any of the above methods when executed by a processor.

[0016] According to yet another embodiment of the present invention, there is provided an electronic device, including a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to execute the steps in any one of the above method embodiments.

[0017] According to yet another embodiment of the present invention, a computer program product is provided, comprising a computer program, wherein when the computer program is executed by a processor, the steps of any of the above methods are implemented.

[0018] Through the present invention, since the state of the mechanical disk is divided into a read-only state and a non-read-only state, data is prohibited from being written to the mechanical disk in the read-only state. And the first data packet with a data volume greater than or equal to the preset data volume threshold is written to the target mechanical disk, which can avoid the problem of low efficiency in reading and writing data caused by the mechanical disk performing write operations and read operations at the same time. And considering the pressure resistance of the target mechanical disk, the target data volume in the first data packet is written to the target mechanical disk according to the relationship between the first time consumption of writing the data packet to the target mechanical disk N times in a historical time and the preset threshold range. In this way, the target data volume of the first data packet is written to the target mechanical disk within the pressure resistance range of the target mechanical hard disk, ensuring the efficiency of writing data. Therefore, the problem of low efficiency in writing and reading data on the mechanical disk can be solved. The effect of improving the efficiency of writing and reading data on the mechanical disk is achieved. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 It is a hardware structure block diagram of a mobile terminal of a method for writing and reading data according to an embodiment of the present invention;

[0020] Figure 2 is a flow chart of a method for writing and reading data according to an embodiment of the present invention;

[0021] Figure 3 This is a schematic diagram of a process according to an embodiment of the present invention. Figure 1 ;

[0022] Figure 4 This is a schematic diagram of a process according to an embodiment of the present invention. Figure 2 ;

[0023] Figure 5 This is a schematic diagram of a process according to an embodiment of the present invention. Figure 3 ;

[0024] Figure 6 is a structural block diagram of an apparatus for writing and reading data according to an embodiment of the present invention. DETAILED DESCRIPTION

[0025] Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings and in combination with the embodiments.

[0026] The following is an explanation of the keywords involved in this application:

[0027] Cloud storage system: A cloud storage system is a distributed data storage system consisting of one or more metadata servers and data node servers. The metadata and real data of the stored files are separated, and the real data is stored in shards. Erasure Code is used to achieve redundancy and fault tolerance.

[0028] Metadata node / service: a node / service that stores and manages metadata, MetaDataServer, hereinafter referred to as MDS;

[0029] Data Node: A server node that stores real file data, DataNode, hereinafter referred to as DN;

[0030] Erasure Code: ErasureCode, referred to as EC;

[0031] Strip: Strip is the original data of a certain length, consisting of N+M pieces;

[0032] Piece: Data fragmentation. Original data of a certain length is divided into N original data blocks according to the specified EC rules. Each data block is called a piece.

[0033] Original data block: Each shard of the original data is called an original data block;

[0034] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0035] The method embodiments provided in the embodiments of the present application can be run in a mobile terminal, a computer terminal or a similar computing device in a cloud storage system. Taking a mobile terminal as an example, a client is installed on the mobile terminal, and the client is a client used to read and write data in the cloud storage system.

[0036] Figure 1 FIG. 1 is a hardware structure block diagram of a mobile terminal of a method for writing and reading data according to an embodiment of the present invention. Figure 1 As shown, the mobile terminal may include one or more ( Figure 1 Only one is shown in the figure) a processor 102 (the processor 102 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA) and a memory 104 for storing data, wherein the mobile terminal may also include a transmission device 106 and an input / output device 108 for communication functions. It can be understood by those skilled in the art that Figure 1 The structure shown is only for illustration and does not limit the structure of the mobile terminal. Figure 1 More or fewer components as shown, or with Figure 1 Different configurations are shown.

[0037] The memory 104 can be used to store computer programs, for example, software programs and modules of application software, such as computer programs corresponding to the methods of writing and reading data in the embodiments of the present invention. The processor 102 executes various functional applications and data processing by running the computer programs stored in the memory 104, that is, to implement the above-mentioned methods. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 104 may further include a memory remotely arranged relative to the processor 102, and these remote memories may be connected to the mobile terminal via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0038] The transmission device 106 is used to receive or send data via a network. The specific example of the above network may include a wireless network provided by a communication provider of the mobile terminal. In one example, the transmission device 106 includes a network adapter (Network Interface Controller, referred to as NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 106 can be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0039] In this embodiment, a method for writing and reading data running on a metadata node in the above cloud storage system is provided. Figure 2 is a flow chart of a method for writing and reading data according to an embodiment of the present invention. Figure 2 As shown, the process includes the following steps:

[0040] Step S202, receiving a write request of a first data packet, wherein the data volume of the first data packet is a first data volume;

[0041] The data included in the first data packet may be image data, and the first data packet is divided into a first type and a second type according to the size of the image, the data packet of a large image is the first type, and the data packet of a small image is the second type. The client receives a read request to read image data, or may receive a write request to write image data.

[0042] The data volume of the first type of data packet is greater than or equal to the preset data volume threshold, and the data volume of the second type of data packet is less than the preset quantity threshold. The preset quantity threshold can be determined according to actual conditions, such as 100 megabytes, 500 megabytes, etc.

[0043] Step S204, in the case where the first data packet is a data packet of the first type, determining a target mechanical disk in the target server, wherein the current state of the target mechanical disk is a non-read-only state, the state of the mechanical disk that is currently being read or within a preset time after being read is a read-only state, the mechanical disk in the read-only state is prohibited from being written data, and the data volume of the first type of data packet is greater than or equal to a preset data volume threshold;

[0044] Among them, the HDD disk status includes: read-write status (not read-only status, data can be read and written), read-only status (only data reading is allowed); the HDD disk status can be marked through the data node; in order to reduce the pressure on the metadata node, the HDD disk status can be maintained on the data node. For example, if the target mechanical hard disk is the hard disk of the target server, the disk status of the target mechanical hard disk can be maintained on the target server (the target server is the data node).

[0045] The change of the HDD disk (including the target mechanical hard disk) status from read-write to read-only takes effect immediately. When the HDD disk status is read-only, the data node (including the target server) can control the data to be written to other available disks. When the HDD disk status changes from read-only to read-write, the HDD disk's read-only status needs to be delayed for a preset period of time (the preset period can be determined based on actual conditions, for example: 10s, 30s). Based on the business scenario of reading data, the HDD disk may continue to read in the short term to reduce unnecessary state change consumption.

[0046] For example, if Figure 3 As shown in the flowchart, the detailed process of reading data includes the following steps:

[0047] Step S11, the client receives a read request, the read request is used to request to read a second data packet (the second data packet may be a data packet of image data); query metadata information of each data slice of the second data packet in the metadata node; obtain the starting offset and length of each data slice in the logical file;

[0048] Step S12, in response to a read request of the second data packet, searching for data fragments of the second data packet in a local cache, a solid state drive, and a mechanical hard disk, includes:

[0049] The client first tries to read whether there is a data fragment of the second data packet in the local cache; if there is in the cache, the data fragment of the second data packet is read and returned; if there is not in the cache, the sequential trigger reads the data fragment of the second data packet from the solid-state drive SSD again, and if there is in the solid-state drive SSD, the data is read and put into the cache, and the data fragment is returned;

[0050] If there is no data fragment of the second data packet in the cache and in the solid-state drive SSD, the data fragment of the second data packet is read from the target mechanical hard disk HDD in the cloud storage data node (target server);

[0051] Step S13, when the second data packet is found in the target mechanical hard disk of the target server, the second data packet is read in the target mechanical hard disk, and the target mechanical hard disk is set to a read-only state through the target server;

[0052] Including: if the end time of reading the data fragment of the second data packet in the target mechanical hard disk is the first time, setting the target mechanical hard disk to a non-read-only state at the second time, wherein the interval between the first time and the second time is a preset time (the preset time can be set according to actual conditions, such as the above 30 seconds).

[0053] Step S14, read the data slices of the second data packet in the target mechanical hard disk HDD of the data node (target server), put them into the client cache, and return the data slices of the second data packet.

[0054] Step S206, writes the target data volume in the first data packet to the target mechanical disk according to the relationship between the first time consumption of writing the data packet to the target mechanical disk N times in succession within the historical time and the preset threshold range, wherein N is a preset value, N is an integer greater than 1, and the target data volume is less than or equal to the first data volume.

[0055] The value of N can be set according to actual conditions, for example, it can be 3 or 5. The pressure resistance of the target mechanical disk can be determined according to the time taken to write data to the target mechanical disk N times.

[0056] Optionally, the executor of the above steps may be a background processor, or other devices with similar processing capabilities, or a machine that integrates at least an image acquisition device and a data processing device, wherein the image acquisition device may include a graphics acquisition module such as a camera, and the data processing device may include a computer, a mobile phone or other terminal, but is not limited thereto.

[0057] Through the above steps, since the state of the mechanical disk is divided into a read-only state and a non-read-only state, the mechanical disk in the read-only state is prohibited from being written with data. And the first data packet whose data volume is greater than or equal to the preset data volume threshold is written to the target mechanical disk, so as to avoid the problem of low efficiency of reading and writing data caused by the mechanical disk performing write operations and read operations at the same time. And considering the pressure resistance of the target mechanical disk, the target data volume in the first data packet is written to the target mechanical disk according to the relationship between the first time consumption of writing the data packet to the target mechanical disk N times in a row in the historical time and the preset threshold range. In this way, the target data volume of the first data packet is written to the target mechanical disk within the pressure resistance of the target mechanical hard disk, ensuring the efficiency of writing data. Therefore, the problem of low efficiency of writing and reading data on the mechanical disk can be solved. The effect of improving the efficiency of writing and reading data on the mechanical disk is achieved.

[0058] As an optional implementation, the pressure resistance of the target mechanical hard disk can be determined in the following manner:

[0059] Determine a target entry matching the first time consumption in a preset relationship table, and determine the target data volume in the target entry; for example, the preset relationship table is shown in Table 1 below, wherein the preset relationship table records multiple entries (for example, entries 1, 2, and 3 shown in Table 1), and the multiple entries include the target entry, and the entry includes the preset threshold range (for example, 0-minimum threshold, minimum threshold-threshold 1, threshold 1-threshold 2 shown in Table 1, wherein the specific values ​​of the minimum threshold, threshold 1, and threshold 2 can be determined according to actual conditions) and the data volume (for example, 100%, 75%, and 50% shown in Table 1, which are examples only and can also be set according to actual conditions), different entries include different preset threshold ranges, and different entries include different data volumes, the target entry includes the target threshold range and the target data volume, and the first time consumption is within the target threshold range.

[0060] Table 1

[0061]

[0062] As an optional embodiment, the method also includes: in a case where the first data packet is a data packet of the second type, writing the first data packet to the solid-state drive, wherein the data volume of the second type of data packet is less than the preset data volume threshold; in a case where the first data packet is a data packet of the first type and the target data volume is less than the first data volume, determining the data in the first data packet except the target data volume as the remaining data of the first data packet, and writing the remaining data of the first data packet to the solid-state drive.

[0063] like Figure 4 In the flowchart shown, the detailed process of writing data is as follows:

[0064] Step S21, writing the first data packet into the cache first, determining the type of the first data packet, if the data volume of the first data packet is greater than or equal to a preset data volume threshold, the first data packet is a data packet of the first type (for example, large image data); if the data volume of the first data packet is less than the preset data volume threshold, the first data packet is a data packet of the second type (for example, small image data);

[0065] Step S22, if it is a first type of data packet (such as large image data), determine whether the time taken to write data to the target mechanical disk for N consecutive times is greater than a minimum threshold; if it is not greater than the minimum time threshold, it means that the pressure resistance of the target mechanical disk writing capability is low, and at this time, 100% of the data is written to the target mechanical disk HDD;

[0066] If it is greater than the threshold 1, it means that the target mechanical disk HDD is under certain pressure and data security is at risk, so 25% of the data in the first data packet is extracted and written to the solid-state drive SSD, and the remaining 75% of the data is continued to be written to the target mechanical disk HDD;

[0067] If it is greater than the threshold 2, it means that the target mechanical disk HDD is under great pressure and data security is at high risk, so 50% of the data in the first data packet is extracted and written to the solid-state drive SSD, and the remaining 50% of the data continues to be written to the target mechanical disk HDD;

[0068] Before all SSD and HDD writing operations, apply to the metadata node for data storage information. SSD selects the current node by default and the writing is completed once completed.

[0069] Step S23, when writing the data in the first data packet into the target mechanical disk HDD, the data node may determine whether the state of the target mechanical disk HDD is marked as read-only;

[0070] If the HDD is in read-only state, the data node selects another available HDD based on the available disk list and load balancing strategy, reports to the metadata node after the data is stored on the disk, and updates the metadata information of the data;

[0071] Step S24: If it is a second type of data packet (eg, small image data), the data in the cache is continuously merged to a size granularity of a preset data amount (eg, 1 MB) and written to the SSD.

[0072] As an optional implementation, the method further includes: counting the number of first type data packets written in the previous cycle of the current cycle to obtain a first value; counting the number of second type data packets written in the previous cycle to obtain a second value; and allocating usage of the solid-state drive based on the relationship between the first value and the second value.

[0073] Specifically, allocating usage of the solid state drive according to the relationship between the first value and the second value includes:

[0074] When the first value is greater than or equal to the second value, allocating the solid-state drive into a first capacity and a second capacity, wherein the first capacity is used to cache the first type of data packets, the second capacity is used to cache the second type of data packets, a ratio of the first capacity to the second capacity is a first ratio, and the first ratio is less than 1;

[0075] When the first value is less than the second value, the solid-state drive is allocated into a third capacity and a fourth capacity, wherein the third capacity is used to cache the first type of data packets, the fourth capacity is used to cache the second type of data packets, the ratio of the third capacity to the fourth capacity is a second ratio, and the first ratio is greater than the second ratio.

[0076] Specifically, the memory size and SSD capacity share of the first type of data packets (such as large images) and the second type of data packets (such as small images) can be adjusted dynamically according to the business. Different priorities and response strategies are different. The characteristics of the cache are mainly determined by the business scenario.

[0077] Since the I / O granularity of the first type of data packets and the second type of data packets are different and have different impacts on the system performance, the second type of data packets have a higher priority, and memory is allocated to the second type of data packets first. The cache life cycle of the second type of data packets is also longer, and the read and write performance of the first type of data packets is relatively high, so a small amount of memory can achieve better performance. At the same time, the memory size allocated to the first type of data packets and the second type of data packets also needs to be adjusted dynamically according to actual business to ensure efficient use of memory.

[0078] like Figure 5 The flowchart shown includes the following steps:

[0079] Step S31, the client starts up and initializes the cache capacity size to 1:1 by default (half is used to store the first type of data packets, and the other half is used to store the second type of data packets), and the SSD ratio is 1:4 (the first ratio 1:4 is only for example, and can be determined according to actual conditions, 1 / 5 of the capacity is used to store the first type of data packets, and 4 / 5 of the capacity is used to store the second type of data packets).

[0080] Step S32, counting the number of first type data packets (first value) and the number of second type data packets (second value) in a timing thread period (the length of each period can be set according to actual conditions, such as 1 hour);

[0081] Step S33: The number of the first type of data packets is greater than or equal to the number of the second type of data packets, indicating that the write operation of the first type of data packets is greater than the write operation of the second type of data packets, and there is no need to adjust the cache and SSD capacity ratio;

[0082] Otherwise, it means that the second type of data packets have more write operations, so the cache ratio of the second type of data packets is adjusted to 80% of the total capacity, more second type of data packets are cached, the cache elimination cycle is longer, and the cache hit rate is further improved; the second type of data packets can be adjusted to occupy a maximum of 90% of the SSD capacity, and the SSD ratio is 1:9 (the second ratio 1:9 is only used as an example, and can be determined according to actual conditions, 1 / 10 of the capacity is used to store the first type of data packets, and 9 / 10 of the capacity is used to store the second type of data packets). More second type of data packets are stored in the SSD to speed up read and write operations.

[0083] As an optional implementation, when the first data packet is a data packet of the second type, writing the first data packet to the solid-state drive includes: when the remaining amount of a second capacity of the solid-state drive is greater than or equal to the first data amount, writing the first data packet to the solid-state drive, wherein the second capacity is a capacity of the solid-state drive used to write the second type of data packet in a current cycle;

[0084] When the remaining amount of the second capacity of the solid-state drive is less than the first data amount, the second type of data packets in the solid-state drive are migrated to the target mechanical disk in the order of the writing time of the second type of data packets in the solid-state drive from far to near, and the first data packets are written to the solid-state drive.

[0085] Specifically, when the type of the first data packet is the second type (e.g., a thumbnail), it is determined whether the available capacity of the SSD (the second capacity allocated to the SSD in the previous cycle is used to store the second type of data packet) can store the first data packet. If the available capacity of the SSD is insufficient, the second type of data in the SSD needs to be moved to the target mechanical disk HDD in chronological order to free up space; if the available capacity of the SSD is sufficient, the current process ends.

[0086] As an optional implementation, writing the remaining data of the first data packet to the solid-state drive includes: when the remaining amount of the first capacity of the solid-state drive is greater than or equal to the amount of the remaining data, writing the remaining data to the solid-state drive, wherein the first capacity is the capacity of the solid-state drive used to write the first type of data packet in the current cycle; when the remaining amount of the first capacity of the solid-state drive is less than the amount of the remaining data, migrating the first type of data packet in the solid-state drive to the target mechanical disk in the order of the writing time of the first type of data packet in the solid-state drive from far to near, and writing the remaining data to the solid-state drive.

[0087] Specifically, when the type of the first data packet is the first type (e.g., a large image), and the remaining data is written to the solid state drive, it is determined whether the available capacity of the SSD (the first capacity allocated to the SSD in the previous cycle is used to store the first type of data packet) can store the remaining data. If the available capacity of the SSD is insufficient, the first type of data in the SSD needs to be moved to the target mechanical disk HDD in chronological order to free up space; if the available capacity of the SSD is sufficient, the current process ends.

[0088] In a write-first business scenario, the first thing to do is to ensure that writing is normal to avoid data loss. If there are both read and write businesses, sequential writing consumes less disk performance, while random reads of mechanical disks consume much more disk performance than sequential writing, especially random I / O reads of small-granularity files, which seriously degrades disk performance. Therefore, under limited hardware cost conditions, it is necessary to reduce the impact of small read I / Os on overall performance, while ensuring normal read functions, improving the overall throughput of image stream business, and thus realizing storage acceleration of image streams. Reducing the impact of small read I / Os on performance means that the system as a whole has a stronger ability to resist the performance impact caused by small I / O reads.

[0089] During the read business, if the cache and SSD have no data to read, the data in the HDD is read. When reading, a certain read merge is performed to amplify the read operation and pre-read more data at one time. At this time, the HDD is marked as being read. Within a certain period of time, the probability of continuous reading of the disk is higher. If the HDD is written at the same time, the disk performance will drop significantly and the performance will be poor. At this time, the new write HDD business will no longer allocate the HDD for writing. The new business uses other HDDs that are not being read for writing. For normal write business, the load balancing at the HDD disk level is mainly based on the disk capacity balancing. When reading small I / O, we set the load priority higher than the capacity load balancing to reduce the length of time that read and write business exist on a HDD disk at the same time. This design can resist the degradation of system performance caused by reading small I / O.

[0090] This application actively intervenes from the underlying physical medium level to reduce the negative cumulative impact of reading and writing the same HDD. Aiming at the multi-node and multi-disk characteristics of cloud storage systems, the HDD read and write mutual exclusion design is used to improve HDD performance, accelerate data storage, and increase the upper limit of system load capacity. Aiming at the business scenarios of image cloud storage, a tiered storage that is strongly related to the business is designed. Logically, there are only two layers, high-performance SSD and HDD, which has more efficient storage performance and reduces performance loss in the middle. The tiering is determined by the priority of large and small images, and the tiering logic is dynamically adjusted according to the business ratio of large and small images to improve the storage efficiency of each layer.

[0091] Through the description of the above implementation methods, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus a necessary general hardware platform, and of course can also be implemented by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for enabling a terminal device (which can be a mobile phone, computer, server, or network device, etc.) to execute the methods described in each embodiment of the present invention.

[0092] In the present embodiment, a device for writing and reading data is also provided, which is used to implement the above-mentioned embodiments and preferred implementation modes, and the descriptions that have been made will not be repeated. As used below, the term "module" can implement a combination of software and / or hardware of a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceived.

[0093] Figure 6 is a structural block diagram of a device for writing and reading data according to an embodiment of the present invention, such as Figure 6 As shown, the device includes: a receiving module 62, which is used to receive a write request for a first data packet, wherein the data volume of the first data packet is a first data volume; a determining module 64, which is used to determine a target mechanical disk when the first data packet is a first type of data packet, wherein the current state of the target mechanical disk is a non-read-only state, the state of the mechanical disk that is currently being read or within a preset time after being read is a read-only state, and the mechanical disk in the read-only state is prohibited from being written to data, and the data volume of the first type of data packet is greater than or equal to a preset data volume threshold; a writing module 66, which is used to write the target data volume in the first data packet to the target mechanical disk according to the relationship between the first time consumption of writing the data packet to the target mechanical disk N times in a historical time and the preset threshold range, wherein N is a preset value, N is an integer greater than 1, and the target data volume is less than or equal to the first data volume.

[0094] In an exemplary embodiment, the above-mentioned device is also used to determine a target entry matching the first time consumption in a preset relationship table before writing the target data volume in the first data packet to the target mechanical disk, and determine the target data volume in the target entry; wherein the preset relationship table records multiple entries, multiple entries include the target entry, the entry includes the preset threshold range and the data volume, different entries include different preset threshold ranges, different entries include different data volumes, the target entry includes the target threshold range and the target data volume, and the first time consumption is within the target threshold range.

[0095] In an exemplary embodiment, the above-mentioned device is also used to write the first data packet to the solid-state drive when the first data packet is a data packet of the second type, wherein the data volume of the second type of data packet is less than the preset data volume threshold; when the first data packet is a data packet of the first type and the target data volume is less than the first data volume, the data in the first data packet except the target data volume is determined as the remaining data of the first data packet, and the remaining data of the first data packet is written to the solid-state drive.

[0096] In an exemplary embodiment, the above-mentioned device is also used to write the first data packet to the solid-state hard disk when the remaining amount of the second capacity of the solid-state hard disk is greater than or equal to the first data amount, wherein the second capacity is the capacity of the solid-state hard disk used to write the second type of data packet in the current cycle; when the remaining amount of the second capacity of the solid-state hard disk is less than the first data amount, the second type of data packet in the solid-state hard disk is migrated to the target mechanical disk in the order of the writing time of the second type of data packet in the solid-state hard disk from far to near, and the first data packet is written to the solid-state hard disk.

[0097] In an exemplary embodiment, the above-mentioned device is also used to write the remaining data to the solid-state hard disk when the remaining amount of the first capacity of the solid-state hard disk is greater than or equal to the data amount of the remaining data, wherein the first capacity is the capacity of the solid-state hard disk used to write the first type of data packets in the current cycle; when the remaining amount of the first capacity of the solid-state hard disk is less than the data amount of the remaining data, the first type of data packets in the solid-state hard disk are migrated to the target mechanical disk in the order of the writing time of the first type of data packets in the solid-state hard disk from far to near, and the remaining data are written to the solid-state hard disk.

[0098] In an exemplary embodiment, the above-mentioned device is also used to count the number of first type data packets written in the previous cycle of the current cycle to obtain a first value; count the number of second type data packets written in the previous cycle to obtain a second value; and allocate the use of the solid-state hard disk according to the relationship between the first value and the second value.

[0099] In an exemplary embodiment, the above-mentioned device is also used to allocate the solid-state drive into a first capacity and a second capacity when the first value is greater than or equal to the second value, wherein the first capacity is used to cache the first type of data packets, the second capacity is used to cache the second type of data packets, and the ratio of the first capacity to the second capacity is a first ratio, and the first ratio is less than 1; when the first value is less than the second value, the solid-state drive is allocated into a third capacity and a fourth capacity, wherein the third capacity is used to cache the first type of data packets, the fourth capacity is used to cache the second type of data packets, and the ratio of the third capacity to the fourth capacity is a second ratio, and the first ratio is greater than the second ratio.

[0100] In an exemplary embodiment, the above-mentioned device is also used to respond to a read request for a second data packet and search for data segments of the second data packet in a local cache, a solid-state drive, and a mechanical hard disk; when data segments of the second data packet are found in the target mechanical hard disk, the data segments of the second data packet are read in the target mechanical hard disk and the target mechanical hard disk is set to a read-only state.

[0101] In an exemplary embodiment, the above-mentioned device is also used to set the target mechanical hard disk to a non-read-only state at a second moment if the end moment of reading the data segment of the second data packet in the target mechanical hard disk is a first moment, wherein the interval between the first moment and the second moment is a preset duration.

[0102] It should be noted that the above modules can be implemented by software or hardware. For the latter, it can be implemented in the following ways, but not limited to: the above modules are all located in the same processor; or the above modules are located in different processors in any combination.

[0103] An embodiment of the present invention further provides a computer-readable storage medium, in which a computer program is stored. When the computer program is executed by a processor, the steps of any of the above methods are implemented.

[0104] In an exemplary embodiment, the computer-readable storage medium may include, but is not limited to, various media that can store computer programs, such as a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk or an optical disk.

[0105] An embodiment of the present invention further provides an electronic device, including a memory and a processor, wherein a computer program is stored in the memory, and the processor is configured to run the computer program to execute the steps in any one of the above method embodiments.

[0106] In an exemplary embodiment, the electronic device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor, and the input / output device is connected to the processor.

[0107] For specific examples in this embodiment, reference may be made to the examples described in the above embodiments and exemplary implementation modes, and this embodiment will not be described in detail herein.

[0108] An embodiment of the present invention further provides a computer program product, including a computer program, which implements the steps of the method described in each embodiment of the present application when executed by a processor.

[0109] Obviously, those skilled in the art should understand that the above modules or steps of the present invention can be implemented by a general computing device, they can be concentrated on a single computing device, or distributed on a network composed of multiple computing devices, they can be implemented by a program code executable by a computing device, so that they can be stored in a storage device and executed by the computing device, and in some cases, the steps shown or described can be executed in a different order than here, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. Thus, the present invention is not limited to any specific combination of hardware and software.

[0110] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A method for writing and reading data, characterized in that: include: receiving a write request for a first data packet, wherein the data volume of the first data packet is a first data volume; In the case where the first data packet is a data packet of the first type, a target mechanical disk is determined, wherein the current state of the target mechanical disk is a non-read-only state, the state of the mechanical disk that is currently being read or within a preset time after the data is read is a read-only state, the mechanical disk in the read-only state is prohibited from being written data, and the data volume of the first type of data packet is greater than or equal to a preset data volume threshold; According to the relationship between the first time consumption of writing the data packet to the target mechanical disk N times in succession within the historical time and the preset threshold range, the target data volume in the first data packet is written to the target mechanical disk, wherein N is a preset value, N is an integer greater than 1, and the target data volume is less than or equal to the first data volume.

2. The method according to claim 1, characterized in that Before writing the target data amount of data in the first data packet into the target mechanical disk, the method further includes: Determine a target entry matching the first time consumption in a preset relationship table, and determine the target data volume in the target entry; Among them, the preset relationship table records multiple entries, the multiple entries include the target entry, the entry includes the preset threshold range and data volume, different entries include different preset threshold ranges, different entries include different data volumes, the target entry includes the target threshold range and the target data volume, and the first time consumption is within the target threshold range.

3. The method according to claim 1 or 2, characterized in that: The method further comprises: When the first data packet is a data packet of the second type, writing the first data packet into the solid-state hard disk, wherein the data volume of the data packet of the second type is less than the preset data volume threshold; When the first data packet is a data packet of the first type and the target data amount is less than the first data amount, the data in the first data packet except the target data amount is determined as the remaining data of the first data packet, and the remaining data of the first data packet is written to the solid state drive.

4. The method according to claim 3, characterized in that: When the first data packet is a data packet of the second type, writing the first data packet to the solid state drive includes: When the remaining amount of the second capacity of the solid state drive is greater than or equal to the first amount of data, writing the first data packet into the solid state drive, wherein the second capacity is the capacity of the solid state drive used for writing the second type of data packets in the current cycle; When the remaining amount of the second capacity of the solid-state drive is less than the first data amount, the second type of data packets in the solid-state drive are migrated to the target mechanical disk in the order of the writing time of the second type of data packets in the solid-state drive from far to near, and the first data packets are written to the solid-state drive.

5. The method according to claim 3, characterized in that: Writing the remaining data of the first data packet into the solid state drive includes: When the remaining amount of the first capacity of the solid state drive is greater than or equal to the amount of the remaining data, writing the remaining data into the solid state drive, wherein the first capacity is the capacity of the solid state drive used for writing the first type of data packets in the current cycle; When the remaining amount of the first capacity of the solid-state hard disk is less than the amount of the remaining data, the first type of data packets in the solid-state hard disk are migrated to the target mechanical disk in the order of the writing time of the first type of data packets in the solid-state hard disk from far to near, and the remaining data are written to the solid-state hard disk.

6. The method according to claim 4 or 5, characterized in that: The method further comprises: Counting the number of data packets of the first type written in a cycle before the current cycle to obtain a first value; Counting the number of the second type of data packets written in the previous cycle to obtain a second value; The solid state drive is allocated usage according to the relationship between the first value and the second value.

7. The method according to claim 6, characterized in that Allocating usage of the solid state drive according to the relationship between the first value and the second value includes: When the first value is greater than or equal to the second value, allocating the solid-state drive into a first capacity and a second capacity, wherein the first capacity is used to cache the first type of data packets, the second capacity is used to cache the second type of data packets, a ratio of the first capacity to the second capacity is a first ratio, and the first ratio is less than 1; When the first value is less than the second value, the solid-state drive is allocated into a third capacity and a fourth capacity, wherein the third capacity is used to cache the first type of data packets, the fourth capacity is used to cache the second type of data packets, the ratio of the third capacity to the fourth capacity is a second ratio, and the first ratio is greater than the second ratio.

8. The method according to claim 1, characterized in that The method further comprises: In response to a read request for a second data packet, searching for data fragments of the second data packet in a local cache, a solid-state drive, and a mechanical hard disk; When the data fragments of the second data packet are found in the target mechanical hard disk, the data fragments of the second data packet are read in the target mechanical hard disk, and the target mechanical hard disk is set to a read-only state.

9. The method according to claim 8, characterized in that Reading the data slices of the second data packet in the target mechanical hard disk and setting the target mechanical hard disk to a read-only state includes: If the end time of reading the data fragment of the second data packet in the target mechanical hard disk is the first time, the target mechanical hard disk is set to a non-read-only state at the second time, wherein the interval between the first time and the second time is a preset time.

10. A device for writing and reading data, characterized in that: include: A receiving module, configured to receive a write request of a first data packet, wherein the data volume of the first data packet is a first data volume; a determination module, configured to determine a target mechanical disk when the first data packet is a data packet of the first type, wherein the current state of the target mechanical disk is a non-read-only state, the state of the mechanical disk that is currently being read or within a preset time after being read is a read-only state, the mechanical disk in the read-only state is prohibited from being written with data, and the data volume of the first type of data packet is greater than or equal to a preset data volume threshold; A writing module is used to write a target data volume in a first data packet to the target mechanical disk according to the relationship between the first time consumption of writing the data packet to the target mechanical disk N times in a historical period and a preset threshold range, wherein N is a preset value, N is an integer greater than 1, and the target data volume is less than or equal to the first data volume.

11. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, wherein the computer program implements the steps of the method described in any one of claims 1 to 9 when executed by a processor.

12. An electronic device comprising a memory and a processor, characterized in that: A computer program is stored in the memory, and the processor is configured to run the computer program to perform the method according to any one of claims 1 to 9.

Images