Information generation method and apparatus

Abstract

The application discloses an information generation method and device, and relates to the technical field of data storage. A specific implementation of the method comprises the following steps: in response to detecting a file application request of an IO processing unit, allocating a file to be written to the IO processing unit; in response to detecting that the current file of the IO processing unit has been written full, obtaining an effective rate of the full file; and based on the effective rate of the full file, generating decision information about whether to start a recycling unit. The implementation achieves flexible scheduling of the storage unit and improves the timeliness and effectiveness of the generated decision information.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computers, in particular to the technical field of data storage, and especially to an information generation method and device. BACKGROUND

[0002] In a storage system, in order to obtain higher write performance or a more easily managed data management mode, an append write mode is used to write data. After the data is appended, the original data is not immediately deleted, so a large amount of garbage data is generated in the system due to the continuous business. Unlike the natural garbage collection unit in solid state storage, an independently writable area is a collection unit. In a distributed system, the collection unit needs to be managed, and then the collection unit is scheduled and controlled to migrate and merge data to achieve the purpose of garbage collection.

[0003] The current data management is based on the fixed allocation of storage space. One of the specific storage spaces may have a large amount of data to be recycled, and the recycling pressure is very large, and even the recycling is not timely enough. Some spaces have a lot of idle space, and the recycling pressure is small, which cannot achieve global balance and has poor scheduling flexibility. SUMMARY

[0004] Embodiments of the present application provide an information generation method, device, equipment and storage medium.

[0005] According to a first aspect, embodiments of the present application provide an information generation method, which comprises: in response to detecting a file application request of an IO processing unit, allocating a file to be written to the IO processing unit; in response to detecting that the current file of the IO processing unit has been written full, obtaining an effective rate of the full file; and generating decision information of whether to start a collection unit based on the effective rate of the full file.

[0006] According to a second aspect, embodiments of the present application provide an information generation device, which comprises: an allocation module configured to allocate a file to be written to an IO processing unit in response to detecting a file application request of the IO processing unit; an obtaining module configured to obtain an effective rate of a full file in response to detecting that the current file of the IO processing unit has been written full; and a generation module configured to generate decision information of whether to start a collection unit based on the effective rate of the full file.

[0007] According to a third aspect, embodiments of the present application provide an electronic device, which comprises one or more processors; a storage device having one or more programs stored thereon, when the one or more programs are executed by the one or more processors, the one or more processors implement the information generation method of any of the embodiments of the first aspect.

[0008] According to a fourth aspect, the embodiments of the present application provide a computer readable medium, which stores a computer program, and the computer program is executed by a processor to implement the information generation method according to any one of the embodiments of the first aspect.

[0009] The present application allocates the file to be written to the IO processing unit in response to detecting the file application request of the IO processing unit; obtains the effective rate of the full file in response to detecting that the current file of the IO processing unit is full; and generates the decision information of whether to start the recycling unit based on the effective rate of the full file, that is, the flexible scheduling of the storage unit is realized by controlling the file size, and the timeliness and effectiveness of the generated decision information are improved.

[0010] It should be understood that the content described in this part is not intended to identify key or important features of the embodiments of the present disclosure, nor to limit the scope of the present disclosure. Other features of the present disclosure will become apparent from the following description. BRIEF DESCRIPTION OF DRAWINGS

[0011] Figure 1 is an exemplary system architecture diagram in which the present application can be applied;

[0012] Figure 2 is a flowchart of one embodiment of the information generation method according to the present application;

[0013] Figure 3 is a timing diagram of one embodiment of the information generation method according to the present application;

[0014] Figure 4 is a flowchart of one embodiment of the information generation method according to the present application;

[0015] Figure 5 is a flowchart of one embodiment of the information generation device according to the present application;

[0016] Figure 6 is a structural schematic diagram of a computer system of a server suitable for implementing the embodiments of the present application. DETAILED DESCRIPTION

[0017] The exemplary embodiments of the present application are described below with reference to the accompanying drawings, which include various details of the embodiments of the present application to assist in understanding, and should be considered as merely exemplary. Therefore, those of ordinary skill in the art should recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the present application. Also, in order to be clear and concise, the description below omits the description of well-known functions and structures.

[0018] It should be noted that the embodiments and features of the embodiments in the present application can be combined with each other in the case of no conflict. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0019] Figure 1 An exemplary system architecture 100 to which embodiments of the information generation method of the present application can be applied is shown.

[0020] As shown in Figure 1 The system architecture 100 can include terminal devices 101, 102, 103, a network 104, and a server 105. The network 104 is a medium to provide a communication link between the terminal devices 101, 102, 103 and the server 105. The network 104 can include various connection types, such as wired, wireless communication links, or fiber optic cables, etc.

[0021] The terminal devices 101, 102, 103 interact with the server 105 through the network 104 to receive or send messages, etc. Various communication client applications can be installed on the terminal devices 101, 102, 103, such as storage class applications, communication class applications, etc.

[0022] The terminal devices 101, 102, 103 can be hardware or software. When the terminal devices 101, 102, 103 are hardware, they can be various electronic devices with display screens, including but not limited to mobile phones and notebook computers. When the terminal devices 101, 102, 103 are software, they can be installed in the above-mentioned electronic devices. They can be implemented as multiple software or software modules (for example, to provide information generation services), or as a single software or software module. No specific limitation is made herein.

[0023] The server 105 can be a server that provides various services, such as allocating a file to be written to an IO processing unit in response to detecting a file application request of the IO processing unit, obtaining an efficiency of a full file in response to detecting that the current file of the IO processing unit has been written full, and generating decision information of whether to start a recycling unit based on the efficiency of the full file.

[0024] It should be noted that the server 105 can be hardware or software. When the server 105 is hardware, it can be implemented as a distributed server cluster composed of multiple servers, or as a single server. When the server is software, it can be implemented as multiple software or software modules (for example, to provide information generation services), or as a single software or software module. No specific limitation is made herein.

[0025] It should be noted that the information generation method provided by the embodiments of the present disclosure can be executed by the server 105, or can be executed by the terminal device 101, 102, or 103, or can be executed by the server 105 and the terminal device 101, 102, or 103 in cooperation with each other. Accordingly, each part (for example, each unit, subunit, module, and sub-module) included in the information generation apparatus can be all arranged in the server 105, or can be all arranged in the terminal device 101, 102, or 103, or can be arranged in the server 105 and the terminal device 101, 102, or 103, respectively.

[0026] It should be understood that Figure 1 The number of terminal devices, networks, and servers in the above-mentioned embodiments is only illustrative. According to the needs of implementation, there can be any number of terminal devices, networks, and servers.

[0027] Figure 2 Flow 200 illustrating an embodiment of the information generation method applicable to the present application is shown. In the present embodiment, the information generation method is applied to a distributed storage system, and the information generation method comprises the following steps:

[0028] In step 201, a file to be written is allocated to an IO processing unit in response to detecting a file application request of the IO processing unit.

[0029] In the present embodiment, the execution subject (such as the server 105 or the terminal device 101, 102, or 103 shown in the above-mentioned embodiments) can monitor the file application request sent by the IO processing unit in real time or periodically, and in response to detecting the file application request, a file to be written can be allocated to the IO processing unit. Figure 1

[0030] The IO processing unit is configured to append business IO to the file and manage metadata in the memory.

[0031] Here, after obtaining the file to be written, the IO processing unit can write business IO and metadata into the file, and can continue to apply for a new file after the file is full.

[0032] The file is composed of storage units, and the size of the file determines the frequency and efficiency of garbage collection. Each file organizes data in blocks, and each data block tail carries metadata information of the current block.

[0033] Here, the size of the file to be written can be determined by the execution subject based on the business pressure and system capacity (i.e., available capacity) of the current distributed storage system.

[0034] ​After obtaining the file application request of the IO processing unit, the execution subject can obtain the service pressure and system capacity of the current distributed storage system, and determine the size of the file allocated to the IO processing unit based on the current service pressure and system capacity. For example, if the service pressure is large and the system capacity is large, a larger file capacity is set; if the service pressure is small and the system capacity is tight, a smaller file capacity is set.

[0035] Specifically, the execution subject can compare the service pressure with a preset first pressure threshold and compare the system capacity with a preset first capacity threshold. If the service pressure is greater than or equal to the preset first pressure threshold and the system capacity is greater than or equal to the preset first capacity threshold, the file size is set to a first preset capacity; if the service pressure is less than a preset second pressure threshold and the system capacity is less than a preset second capacity threshold, the file size is set to a second preset capacity.

[0036] The first preset capacity, the second preset capacity, the first pressure threshold, the second pressure threshold, the first capacity threshold, and the second capacity threshold can be set according to experience and actual demand. Generally, the first preset capacity is greater than the second preset capacity, the first pressure threshold is greater than or equal to the second pressure threshold, and the first capacity threshold is greater than or equal to the second capacity threshold.

[0037] In step 202, in response to detecting that the current file of the IO processing unit has been written full, the effective rate of the full file is obtained.

[0038] In this embodiment, after obtaining the file to be written, the IO processing unit can write service IO and metadata into the file in an append write manner, and can continue to apply for a new file after the file is full. After detecting that the current file of the IO processing unit has been written full, the execution subject can obtain the effective rate of the full file in the IO processing unit.

[0039] Here, the effective rate is used to indicate the ratio of the size of the valid block in the file to the size of the file.

[0040] The number of full files can be one or more, which is not limited in the present application.

[0041] In step 203, based on the effective rate of the full file, decision information of whether to start the recycling unit is generated.

[0042] In this embodiment, after obtaining the effective rate of the full file, the execution subject can directly generate the decision information of whether to start the recycling unit based on the effective rate of the full file; or generate the decision information of whether to start the recycling unit based on the effective rate of the full file and the service pressure of the current distributed storage system, which is not limited in the present application.

[0043] The recovery unit is used to merge the metadata of the valid data of the file to be recovered, and migrate the valid data to a new file, thereby preparing for garbage recovery.

[0044] Specifically, the execution subject can compare the valid rate of the full file with a preset valid rate threshold, and in response to the full file containing a target file with a valid rate less than or equal to the preset valid rate threshold, the decision information for starting the recovery unit is generated.

[0045] Here, the valid rate threshold can be set according to experience and actual needs, for example, 30%, 25%, etc.

[0046] In addition, it should be noted that the number of full files can be one or multiple, which is not limited in the present application.

[0047] Specifically, if the number of full files obtained is one, the execution subject can directly compare the valid rate of the full file with a preset valid rate threshold, and if the valid rate of the full file is less than or equal to the preset valid rate threshold, the decision information for starting the recovery unit is generated to perform garbage recovery on the full file; if the number of full files obtained is multiple, the execution subject can first sort the full files in descending order of valid rate to obtain sorted files, and in response to determining that the sorted files contain a specified number of target files with a valid rate less than or equal to the preset valid rate threshold, the decision information for starting the recovery unit is generated to perform garbage recovery on the target files.

[0048] In some optional manners, the decision information for starting the recovery unit is generated based on the valid rate of the full file, including: in response to the full file containing a specified number of target files with a valid rate less than or equal to a preset valid rate threshold, the decision information for starting the recovery unit is generated.

[0049] In the present implementation, the number of full files is multiple, and the execution subject can first sort the full files in descending order of valid rate to obtain sorted files, and in response to determining that the sorted files contain a specified number of target files with a valid rate less than or equal to the preset valid rate threshold, the decision information for starting the recovery unit is generated to perform garbage recovery on the target files.

[0050] Here, the specified number is greater than or equal to two, which can be set according to experience, actual needs and specific application scenarios, for example, 2, 5, etc., which is not limited in the present application.

[0051] The implementation generates the decision information of starting the recovery unit in response to the fact that the specified number of target files in the full file have an effective rate less than or equal to the preset effective rate threshold, thereby improving the flexibility of generating the decision information of starting the recovery unit.

[0052] In some optional manners, the method further includes: in response to the generation of the decision information of starting the recovery unit, controlling the recovery unit to perform a recovery operation on the target file; and in response to the IO processing unit receiving an instruction indicating that the recovery is completed and sent by the recovery unit, controlling the IO processing unit to load the metadata of the valid data and delete the target file after the loading is completed.

[0053] In the implementation, after the execution subject generates the decision information of starting the recovery unit, the execution subject can control the recovery unit to perform a recovery operation on the target file, and send an instruction indicating that the recovery is completed to the IO processing unit after the recovery is completed. In response to the IO processing unit receiving the instruction indicating that the recovery is completed and sent by the recovery unit, the execution subject can control the IO processing unit to load the metadata of the valid data of the target file and delete the target file after the loading is completed.

[0054] The recovery operation includes: loading the metadata of the valid data of the target file into the cache of the recovery unit, performing a merging process, and migrating the valid data of the target file to a new file.

[0055] The implementation controls the recovery unit to perform a recovery operation on the target file in response to the generation of the decision information of starting the recovery unit, and controls the IO processing unit to load the metadata of the valid data and delete the target file after the loading is completed in response to the IO processing unit receiving an instruction indicating that the recovery is completed and sent by the recovery unit, thereby avoiding a large amount of data transmission between the IO processing unit and the recovery unit, and improving the effectiveness and flexibility of garbage collection on the target file.

[0056] Continuing to refer to Figure 3 , Figure 3 is a timing diagram of an application scenario of the information generation method according to the embodiment.

[0057] In Figure 3In the platform monitoring scenario, the execution subject allocates a file to be written to the IO processing unit 302 in response to detecting a file application request of the IO processing unit 302 through the management unit 301, the size of the file to be written can be determined by the execution subject based on the business pressure and the system capacity, i.e., the available capacity of the system, of the current distributed storage system, the IO processing unit 302 appends the business IO to the file after obtaining the file to be written, and manages the metadata in the memory; the execution subject obtains the effective rate of the full file in response to detecting that the current file of the IO processing unit has been written full; based on the effective rate of the full file, decision information of whether to start the recycling unit is generated, for example, in response to the existence of a target file with an effective rate less than or equal to a preset effective rate threshold in the full file, the decision information of starting the recycling unit 303 is generated and the recycling unit is controlled to perform the recycling operation; in response to the IO processing unit 302 receiving an instruction indicating that the recycling is completed sent by the recycling unit 303, the IO processing unit 302 is controlled to load the metadata of the valid data of the target file cached in the recycling unit 303, and the target file, i.e., the garbage file, is deleted after the loading is completed.

[0058] The IO processing unit, the management unit, and the recycling unit can be in the same storage device or in different storage devices, which is not limited in the present application.

[0059] Figure 4 Flow 400 of another embodiment of the information generation method applicable to the present application is shown. In the present embodiment, the information generation method comprises the following steps:

[0060] Step 401, in response to detecting a file application request of the IO processing unit, allocating a file to be written to the IO processing unit.

[0061] In the present embodiment, the implementation details and technical effects of step 401 can be referred to the description of step 201, which will not be repeated here.

[0062] Step 402, in response to detecting that the current file of the IO processing unit has been written full, obtaining the effective rate of the full file.

[0063] In the present embodiment, the implementation details and technical effects of step 402 can be referred to the description of step 202, which will not be repeated here.

[0064] Step 403, based on the effective rate of the full file and the business pressure of the current distributed storage system, generating decision information of whether to start the recycling unit.

[0065] In the embodiment, the execution subject can further acquire the service pressure of the current distributed storage system after acquiring the efficiency of the full file, and generate the decision information of whether to start the recovery unit based on the efficiency of the full file and the service pressure of the current distributed storage system.

[0066] Here, the execution subject can generate the decision information of whether to start the recovery unit based on the comparison result of the efficiency of the full file and the preset efficiency threshold and the comparison result of the service pressure of the current distributed storage system and the preset pressure threshold, or generate a decision value based on the efficiency of the full file and the corresponding first weight and the service pressure of the current distributed storage system and the corresponding second weight, and generate the decision information of whether to start the recovery unit according to the size of the decision value and the preset decision threshold (for example, generate the decision information of starting the recovery unit in response to the decision value being greater than or equal to the decision threshold), which is not limited in the application.

[0067] The first weight and the second weight can be set according to experience and actual demand, and generally the first weight is greater than the second weight, that is, the efficiency of the full file is given priority, that is, the system capacity is given priority.

[0068] In some optional manners, the decision information of whether to start the recovery unit is generated based on the efficiency of the full file and the service pressure of the current distributed storage system, including: in response to the existence of a target file with an efficiency less than or equal to a preset efficiency threshold in the full file and the service pressure of the current distributed storage system being less than or equal to a preset pressure threshold, generating the decision information of starting the recovery unit.

[0069] In the embodiment, the execution subject can compare the efficiency of the full file with the preset efficiency threshold after acquiring the efficiency of the full file, and generate the decision information of starting the recovery unit in response to the existence of one or a specified number of target files with an efficiency less than or equal to a preset efficiency threshold in the full file and the service pressure of the current distributed storage system being less than or equal to a preset pressure threshold.

[0070] Specifically, if there is a target file with an efficiency less than or equal to a preset efficiency threshold in the full file and the service pressure of the current distributed storage system is less than or equal to a preset pressure threshold, the decision information of starting the recovery unit can be generated to perform garbage collection on the target file; if there is a target file with an efficiency less than or equal to a preset efficiency threshold in the full file and the service pressure of the current distributed storage system is greater than a preset pressure threshold, the decision information of not starting the recovery unit can be generated.

[0071] The implementation generates the decision information of starting the recovery unit in response to the existence of the target file with the valid rate less than or equal to the preset valid rate threshold in the full file, and the service pressure of the current distributed storage system being less than or equal to the preset pressure threshold, that is, under the condition of the existence of the target file with the valid rate less than or equal to the preset valid rate threshold, further judging the service pressure of the current distributed system, if the current service pressure is large, the recovery unit is temporarily suspended for garbage collection, if the current service pressure is small, the recovery unit can be directly started for garbage collection, which further improves the flexibility and effectiveness of garbage collection.

[0072] From Figure 4 It can be seen that, compared with the embodiment corresponding to Figure 2 , the flow 400 of the information generation method in the embodiment embodies the generation of the decision information of whether to start the recovery unit based on the valid rate of the full file, and the service pressure of the current distributed storage system, that is, under the condition that the valid rate of the full file meets the recovery condition, further considering the service pressure of the current distributed storage system, comprehensively determining whether to start the recovery unit, and further improving the effectiveness of the generated decision information.

[0073] Further referring to Figure 5 , as an implementation of the method shown in the above figures, the present application provides an embodiment of an information generation device, which corresponds to the method embodiment shown in Figure 3 , and the device can be specifically applied to various electronic devices.

[0074] As shown in Figure 5 , the information generation device 500 of the embodiment includes an allocation module 501, an acquisition module 502, and a generation module 503.

[0075] The allocation module 501 can be configured to allocate a file to be written to the IO processing unit in response to detecting a file application request of the IO processing unit.

[0076] The acquisition module 502 can be configured to acquire the valid rate of the full file in response to detecting that the current file of the IO processing unit has been full.

[0077] The generation module 503 can be configured to generate the decision information of whether to start the recovery unit based on the valid rate of the full file.

[0078] In some optional manners of the embodiment, the generation module is further configured to generate the decision information of whether to start the recovery unit based on the valid rate of the full file, and the service pressure of the current distributed storage system.

[0079] In some optional manners of the embodiment, the generating module is further configured to generate the decision information of starting the reclaiming unit in response to the fact that there are target files with an effective rate less than or equal to the preset effective rate threshold in the full file, and the fact that the service pressure of the current distributed storage system is less than or equal to a preset pressure threshold.

[0080] In some optional manners of the embodiment, the generating module is further configured to generate the decision information of starting the reclaiming unit in response to the fact that there are a specified number of target files with an effective rate less than or equal to the preset effective rate threshold in the full file.

[0081] In some optional manners of the embodiment, the apparatus further includes a reclaiming module configured to control the reclaiming unit to perform a reclaiming operation on the target file in response to the decision information of starting the reclaiming unit being generated, and control the IO processing unit to load the metadata of the valid data and delete the target file after the loading is completed in response to the IO processing unit receiving an instruction indicating that the reclaiming is completed sent by the reclaiming unit.

[0082] According to the embodiments of the present application, the present application further provides an electronic device and a readable storage medium.

[0083] As shown in Figure 6 , it is a block diagram of an electronic device according to the information generation method of the embodiments of the present application.

[0084] 600 is a block diagram of an electronic device according to the information generation method of the embodiments of the present application. The electronic device is intended to represent a variety of forms of digital computers, such as laptops, desktops, tablets, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular telephones, smart phones, wearable devices, and other similar computing devices. The components shown in the figures, their connections and relationships, and their functions, are meant to be examples only, and are not intended to limit the implementations of the present application described and / or claimed in this document.

[0085] As shown in Figure 6As shown, the electronic device includes one or more processors 601, a memory 602, and interfaces for connecting the various components, including high-speed interfaces and low-speed interfaces. The various components are interconnected by different buses, and can be mounted on a common main board or otherwise installed as desired. The processor can process instructions executed within the electronic device, including instructions stored in the memory or graphical information of a GUI stored in the memory for display on an external input / output device, such as a display device coupled to the interface. In other embodiments, multiple processors and / or buses can be used with multiple memories and multiple memory, if desired. Likewise, multiple electronic devices can be connected, each device providing part of the necessary operations (e.g., as a server array, a group of blade servers, or a multi-processor system). Figure 6 The processor 601 is taken as an example in the embodiment.

[0086] The memory 602 is a non-transitory computer readable storage medium provided by the present application. The memory stores instructions executable by at least one processor, so that the at least one processor executes the information generation method provided by the present application. The non-transitory computer readable storage medium of the present application stores computer instructions for causing a computer to execute the information generation method provided by the present application.

[0087] The memory 602 is a non-transitory computer readable storage medium, which can be used to store non-transitory software programs, non-transitory computer executable programs and modules, such as program instructions / modules of the information generation method in the embodiment of the present application (for example, the allocation module 501, the acquisition module 502 and the generation module 503 shown in the figure). The processor 601 executes various functional applications and data processing of the server by running the non-transitory software programs, instructions and modules stored in the memory 602, that is, implements the information generation method in the above method embodiment. Figure 5 The allocation module 501, the acquisition module 502 and the generation module 503 shown in the figure). The processor 601 executes various functional applications and data processing of the server by running the non-transitory software programs, instructions and modules stored in the memory 602, that is, implements the information generation method in the above method embodiment.

[0088] The memory 602 can include a program storage area and a data storage area, wherein the program storage area can store an operating system and application programs required by at least one function; the data storage area can store data created by the use of the electronic device for information generation and the like. In addition, the memory 602 can include a high-speed random access memory, and can also include a non-transitory memory, such as at least one magnetic disk storage device, a flash memory device, or other non-transitory solid-state storage device. In some embodiments, the memory 602 can optionally include a memory remotely arranged with respect to the processor 601, which can be connected to the electronic device for information generation through a network. Examples of the above network include but are not limited to the Internet, an intranet, a local area network, a mobile communication network, and a combination thereof.

[0089] The electronic device of the information generation method can further include an input device 603 and an output device 604. The processor 601, the memory 602, the input device 603, and the output device 604 can be connected through a bus or other means, Figure 6 The connection through the bus is taken as an example.

[0090] The input device 603 can receive input digital or character information, such as a touch screen, a keypad, a mouse, a trackpad, a touchpad, a pointing stick, one or more mouse buttons, a trackball, a joystick, and the like. The output device 604 can include a display device, an auxiliary lighting device (for example, an LED), a tactile feedback device (for example, a vibration motor), and the like. The display device can include, but is not limited to, a liquid crystal display (LCD), a light-emitting diode (LED) display, and a plasma display. In some embodiments, the display device can be a touch screen.

[0091] Various embodiments of the systems and techniques described here can be realized in digital electronic circuitry, integrated circuitry, specially designed ASICs (application specific integrated circuits), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include implementation in one or more computer programs that are executable and / or interpretable on a programmable system including at least one programmable processor, which can be special or general purpose, coupled to receive data and instructions from, and to transmit data and instructions to, a storage system, at least one input device, and at least one output device.

[0092] These computer programs (also known as programs, software, software applications or code) include machine instructions for a programmable processor, and can be implemented in a high-level procedural and / or object-oriented programming language, and / or in assembly / machine language. As used herein, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, apparatus and / or device (e.g., magnetic discs, optical disks, memory, Programmable Logic Devices (PLDs)) used to provide machine instructions and / or data to a programmable processor, including a machine-readable medium that receives machine instructions as a machine-readable signal. The term "machine-readable signal" refers to any signal used to provide machine instructions and / or data to a programmable processor.

[0093] To provide for interaction with a user, the systems and techniques described here can be implemented on a computer having a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user and a keyboard and a pointing device (e.g., a mouse or a trackball) by which the user can provide input to the computer. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form, including acoustic, speech, or tactile input.

[0094] The systems and techniques described here can be implemented in a computing system that includes a back end component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front end component (e.g., a user computer having a graphical user interface or a Web browser through which a user can interact with an implementation of the systems and techniques described here), or any combination of such back end, middleware, or front end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), and the Internet.

[0095] The computer system can include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other.

[0096] According to the technical scheme of the embodiment of the present application, flexible scheduling of the storage units is achieved, and the timeliness and effectiveness of the generated decision information are improved.

[0097] It should be understood that the various forms of flow shown above can be re-ordered, added to, or deleted from without departing from the scope of the present application. For example, the steps recited in the present application can be executed in parallel, executed in series, or executed in different orders, and the present application is not limited herein as long as the desired results of the technical scheme disclosed in the present application can be achieved.

[0098] The specific embodiments described above are not intended to limit the scope of the present application. Those skilled in the art will understand that various modifications, combinations, sub-combinations, and alternatives of the present application can be made based on the detailed disclosure herein. Any modification, equivalent replacement, and improvement made within the spirit and principle of the present application shall fall within the scope of the present application.

Claims

1. An information generation method applied to a distributed storage system, the method comprising: in response to detecting a file application request of an IO processing unit, allocating a to-be-written file to the IO processing unit, a size of the to-be-written file being determined based on a service pressure and a system capacity of a current distributed storage system, the IO processing unit being configured to append write service IOs to files and manage metadata in memory, wherein each file organizes data in blocks, and a tail of each block carries metadata information of a current block; in response to detecting that a current file of the IO processing unit has been written full, obtaining an efficiency of the written full file, the efficiency being used to indicate a ratio of a size of valid blocks in the file to a size of the file; based on the efficiency of the written full file, generating decision information about whether to start a recycling unit; in response to generating the decision information about starting the recycling unit, controlling the recycling unit to perform a recycling operation on a target file in the written full file, the efficiency of which is less than or equal to a preset efficiency threshold, the recycling operation comprising: loading metadata of valid data of the target file into a cache of the recycling unit, performing merging processing, and migrating the valid data of the target file to a new file; in response to the IO processing unit receiving an instruction indicating that recycling is completed, which is sent by the recycling unit, controlling the IO processing unit to load the metadata of the valid data, and deleting the target file after loading is completed. The generating, based on the efficiency of the written full file, of the decision information about whether to start the recycling unit comprises: based on the efficiency of the written full file and the service pressure of the current distributed storage system, generating the decision information about whether to start the recycling unit. The generating, based on the efficiency of the written full file and the service pressure of the current distributed storage system, of the decision information about whether to start the recycling unit comprises: in response to the written full file including a target file whose efficiency is less than or equal to a preset efficiency threshold, and the service pressure of the current distributed storage system being less than or equal to a preset pressure threshold, generating the decision information about starting the recycling unit. The number of the written full files is a plurality, and the generating, based on the efficiency of the written full file, of the decision information about whether to start the recycling unit comprises: in response to the written full file including a specified number of target files whose efficiency is less than or equal to a preset efficiency threshold, the specified number being greater than or equal to two, generating the decision information about starting the recycling unit. 5.An information generation apparatus applied to a distributed storage system, the apparatus comprising: an allocation module configured to, in response to detecting a file application request of an IO processing unit, allocate a to-be-written file to the IO processing unit, a size of the to-be-written file being determined based on a service pressure and a system capacity of a current distributed storage system, the IO processing unit being configured to append write service IOs to files and manage metadata in memory, wherein each file organizes data in blocks, and a tail of each block carries metadata information of a current block; ​ 2. The method of claim 1, wherein, ​ ​ 3. The method of claim 2, wherein, ​ ​ 4. The method of claim 1, wherein, ​ ​ ​ ​ The obtaining module is configured to, in response to detecting that the current file of the IO processing unit has been written full, obtain an efficiency of the written full file, the efficiency being used to indicate a ratio of a size of valid blocks in the file to a size of the file; The generating module is configured to generate decision information of whether to start the recycling unit based on the efficiency of the written full file; The recycling module is configured to, in response to the decision information of starting the recycling unit being generated, control the recycling unit to perform a recycling operation on a target file in the written full file, the efficiency of which is less than or equal to a preset efficiency threshold, the recycling operation including: loading metadata of valid data of the target file into a cache of the recycling unit, performing a merging process, and migrating the valid data of the target file to a new file; and in response to the IO processing unit receiving an instruction indicating that the recycling is completed, which is sent by the recycling unit, controlling the IO processing unit to load the metadata of the valid data, and deleting the target file after the loading is completed.

6. The apparatus of claim 5, wherein, The generating module is further configured to: generate the decision information of whether to start the recycling unit based on the efficiency of the written full file and a service pressure of a current distributed storage system.

7. The apparatus of claim 6, wherein, The generating module is further configured to: in response to the target file in the written full file having an efficiency less than or equal to a preset efficiency threshold and the service pressure of the current distributed storage system being less than or equal to a preset pressure threshold, generate the decision information of starting the recycling unit.

8. An electronic device, comprising: comprise: at least one processor; and a memory connected with the at least one processor in communication; wherein the memory stores computer instructions that can be executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-4.

9. A non-transitory computer-readable storage medium having stored thereon computer instructions, wherein, The computer instructions are used to enable the computer to perform the method of any one of claims 1-4.

Images