Log information transfer method and device, equipment and medium

Abstract

The invention relates to a log information transfer method and device, equipment and a medium, and the method comprises the steps: respectively obtaining quota associated log information corresponding to each polling cycle in each polling cycle; based on the quota associated log information corresponding to the first polling cycle, generating configuration information corresponding to the quota associated log information; according to the configuration information corresponding to the quota associated log information of each polling cycle, storage operation of the quota associated log information corresponding to each polling cycle is asynchronously completed. According to the method, manual intervention is reduced through an automatic process, the efficiency and reliability of quota associated log management are improved, the method is suitable for various computer systems needing efficient log management, storage operation of quota associated log information corresponding to each polling cycle is asynchronously completed, system blockage caused by synchronous operation can be avoided, and the efficiency and reliability of the quota associated log management are improved. And the system stability is enhanced.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a log information transfer method, device, equipment and medium. Background Art

[0002] In distributed file quota technology, in order to ensure the accuracy of quota data statistics, when the metadata changes, the quota-related data is logged together with the file metadata. When the metadata log is subsequently eliminated, we need to synchronously transfer the quota companion log information.

[0003] If the transfer mapping efficiency and fault reliability are poor during this process, it will make it difficult to evaluate the fault recovery capability, inaccurate quota statistics, and poor quota transfer performance will directly affect business performance.

[0004] Therefore, efficient dump mapping, fault design, and ensuring dump performance are key technologies for quota-associated log data management. Summary of the invention

[0005] In view of this, the present invention provides a log information transfer method, device, equipment and medium to solve the problems of poor mapping efficiency and poor fault reliability of existing quota-associated log information transfer.

[0006] In a first aspect, the present invention provides a log information transfer method, comprising:

[0007] In each polling cycle, obtain the quota log information corresponding to each polling cycle;

[0008] Based on the quota associated log information corresponding to the first polling cycle, generate configuration information corresponding to the quota associated log information, wherein the configuration information includes a storage location of the quota associated log information, a size of the quota associated log information, and identification information corresponding to the first polling cycle, where the first polling cycle is any polling cycle;

[0009] According to the configuration information corresponding to the quota associated log information of each polling cycle, the storage operation of the quota associated log information corresponding to each polling cycle is asynchronously completed.

[0010] The log information transfer method provided by the present invention has the following advantages:

[0011] By obtaining the quota associated log information corresponding to each polling cycle and generating the corresponding configuration information in each polling cycle, the automated process reduces manual intervention and improves the efficiency of log processing. The storage location, size, and identification information corresponding to the polling cycle of the quota associated log information are clearly recorded in the configuration information, which helps to quickly locate and retrieve the quota associated log information, thereby improving the performance of data processing. At the same time, clear configuration information also helps to quickly locate and recover when problems occur in the system, thereby improving the reliability of the system. Asynchronously completing the storage operation of the quota associated log information corresponding to each polling cycle can avoid system blockage caused by synchronous operations and enhance the stability of the system.

[0012] This method can adapt to different polling cycles and log information, and has good scalability. At the same time, through the flexible definition of configuration information, the storage strategy can be adjusted according to different needs to improve the flexibility of the system. By separating the storage and management of log information from business logic, the coupling degree of the system can be reduced, making the system more modular and easy to maintain and upgrade.

[0013] In an optional implementation, the configuration information is embodied in the form of an operation handle.

[0014] Specifically, the handle provides an abstract access method, abstracting different quota associated log information into the same configuration information module for management, reducing programming complexity and improving code readability. Each quota associated log information configuration information memory block has a unique handle reference, making it easier for developers to track and debug resource usage, and quickly locate problems when failures occur. If the location of a resource in memory changes, you only need to change the configuration information in the corresponding handle. The external value and identification of the handle remain unchanged, which does not affect the normal operation of the program and greatly enhances the robustness of the program. In addition, because the handle greatly reduces the complexity of memory addressing operations, it is more efficient than directly accessing the memory address, improving the overall performance of the system.

[0015] In an optional implementation, the storage location of the quota companion log includes a cache write location;

[0016] According to the configuration information corresponding to the quota associated log information of each polling cycle, the storage operation of the quota associated log information corresponding to each polling cycle is asynchronously completed, including:

[0017] Extract the cache write location and quota associated log information size from the configuration information;

[0018] Determine the disk write location based on the cache write location and the size of the quota associated log information;

[0019] The quota companion log is written to the preset file in the pre-built storage pool according to the cache write location and the disk write location.

[0020] Specifically, the sum of the cache write position and the quota associated log information size is the disk write position. The storage position can be pre-calculated through the cache write position and the quota associated log information size. According to the pre-calculated storage position, the quota associated log information is immediately written to the preset file in the pre-built storage pool, realizing automated storage operations, reducing the time waiting for the response to the completion of the previous polling cycle write, and improving the system throughput and the system's parallel processing capabilities. In addition, after extracting the cache write position and the quota associated log information size from the configuration information and determining the disk write position, the quota associated log information of multiple polling cycles can be written simultaneously, improving the efficiency of log information transfer.

[0021] The transfer to disk operation is performed in the background and will not affect the foreground operation, ensuring the smoothness and response speed of the foreground business.

[0022] In an optional implementation, the configuration information may further include: a quota companion log format version, an identifier of a written data object, and one or more parameters of a disk write location of the companion log corresponding to the current polling cycle.

[0023] Specifically, the quota companion log format version can ensure that systems or applications of different versions can correctly parse and process quota companion log information, and write data object identifiers to ensure that quota companion log information is associated with the original data object, which helps to maintain data consistency and integrity when restoring quota companion log information of quotas or analyzing original data.

[0024] The storage location of the quota associated log may include a cache write location or a disk write location. The cache write location is the starting location of the quota associated log information storage, and the disk write location is the ending location of the quota associated log information storage.

[0025] In an optional implementation, when obtaining the quota associated log information corresponding to the second polling cycle fails within the second polling cycle, the method further includes:

[0026] According to the second polling cycle, the original data written in the second polling cycle is obtained from the original data pool, wherein the second polling cycle is any polling cycle except the first polling cycle;

[0027] Restore the size of the quota associated log information in the second polling cycle based on the original data;

[0028] When the second polling cycle is not the first polling cycle, determining the cache write position of the second polling cycle based on the disk write position of the previous polling cycle;

[0029] Based on the quota companion log information size in the second polling cycle and the cache write position in the second polling cycle, restore the disk write position of the quota companion log in the second polling cycle;

[0030] Configuration information of the second polling cycle is generated based on the identification information corresponding to the second polling cycle, the size of the quota associated log information of the second polling cycle, and the storage location of the quota log of the second polling cycle.

[0031] In an optional implementation, when the second polling cycle is the first polling cycle, the method further includes:

[0032] Determine that the first byte of the preset file is the cache write position of the second polling cycle;

[0033] Configuration information of the second polling cycle is generated based on the size of the quota associated log information restored in the second polling cycle, the cache write position of the second polling cycle, and the identification information corresponding to the second polling cycle.

[0034] Specifically, failure to obtain the quota associated log information of the second polling cycle will result in failure to generate the configuration information of the second polling cycle. Based on the polling cycle, the original data stored in the original data pool in each polling cycle can be determined, and the quota associated log information content can be restored based on the original data, including the quota associated log information size.

[0035] Since the next byte of the disk write position of the previous polling cycle is the cache write position in the second polling cycle configuration information, or when the second polling cycle is the first polling cycle, the first byte of the preset file is the cache write position in the second polling cycle configuration information. The sum of the cache write position in the second polling cycle configuration information and the size of the quota associated log information in the second polling cycle configuration information is the disk write position in the second polling cycle configuration information, and thus the second polling cycle configuration information is restored.

[0036] This solution provides a method for handling the abnormal situation of failure to obtain the quota associated log information corresponding to the second polling cycle. By restoring the quota associated log information size and disk write location, the configuration information can be quickly restored, and then the data processing flow can be quickly restored, reducing the processing delay caused by the loss of log information. At the same time, the complete quota associated log configuration information can also reduce the risk of data loss and enhance data security.

[0037] In an optional embodiment, the method further includes:

[0038] Get the remaining storage space in the preset file in the pre-built storage pool;

[0039] Based on the quota associated log information size and the remaining storage space in the current polling cycle, determine whether the remaining storage space can store the quota associated log information;

[0040] When it is determined that the remaining storage space is insufficient to store the quota associated log information of the previous polling cycle, the storage space of the preset file in the pre-built storage pool is expanded until the storage space in the preset file is sufficient to store the quota associated log information of the current polling cycle.

[0041] Specifically, based on the size of the quota associated log information and the remaining storage space, determine whether the remaining storage space can store the quota associated log information, and automatically expand the storage space when the storage space is insufficient to ensure that all quota associated log information can be stored, avoid the loss of quota associated log information, and ensure data integrity.

[0042] The automatic expansion mechanism of this solution ensures that the system can run stably even under high load conditions, improving the reliability of the system. The automatic storage space management of the automatic expansion mechanism reduces the need for manual inspection and intervention, reducing operation and maintenance costs.

[0043] In an optional implementation, the method further includes backing up the handle of each polling cycle to a pre-built storage pool.

[0044] Specifically, the backup handle can be restored from the storage pool when the handle of the quota associated log information is lost or damaged, reducing the risk of data loss. The backup handle can also be used for diagnosis after a failure occurs, helping to quickly locate the problem and perform recovery.

[0045] In a second aspect, the present invention provides a log information transfer device, the device comprising:

[0046] The acquisition module is used to obtain the quota associated log information corresponding to each polling cycle in each polling cycle;

[0047] A processing module is used to generate configuration information corresponding to the quota associated log information based on the quota associated log information corresponding to the first polling cycle, wherein the configuration information includes the storage location of the quota associated log, the size of the quota associated log information, and identification information corresponding to the first polling cycle, and the first polling cycle is any polling cycle; according to the configuration information corresponding to the quota associated log information of each polling cycle, the storage operation of the quota associated log information corresponding to each polling cycle is asynchronously completed.

[0048] The log information transfer device provided by the present invention has the following advantages:

[0049] By obtaining the quota associated log information corresponding to each polling cycle and generating the corresponding configuration information in each polling cycle, the automated process reduces manual intervention and improves the efficiency of log processing. The storage location, size, and identification information corresponding to the polling cycle of the quota associated log information are clearly recorded in the configuration information, which helps to quickly locate and retrieve the quota associated log information, thereby improving the performance of data processing. At the same time, clear configuration information also helps to quickly locate and recover when problems occur in the system, thereby improving the reliability of the system. Asynchronously completing the storage operation of the quota associated log information corresponding to each polling cycle can avoid system blockage caused by synchronous operations and enhance the stability of the system.

[0050] In a third aspect, the present invention provides a computer device, comprising: a memory and a processor, the memory and the processor are communicatively connected to each other, computer instructions are stored in the memory, and the processor executes the log information transfer method of the first aspect or any corresponding embodiment thereof by executing the computer instructions.

[0051] In a fourth aspect, the present invention provides a computer-readable storage medium having computer instructions stored thereon, the computer instructions being used to enable a computer to execute the log information transfer method of the first aspect or any corresponding embodiment thereof.

[0052] In a fifth aspect, the present invention provides a computer program product, including computer instructions, which are used to enable a computer to execute the log information transfer method of the first aspect or any corresponding embodiment thereof. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0054] Figure 1 It is a flow chart of a log information transfer method provided by an embodiment of the present invention;

[0055] Figure 2 It is a flowchart of another log information transfer method provided by an embodiment of the present invention;

[0056] Figure 3 It is a flowchart of another log information transfer method provided by an embodiment of the present invention;

[0057] Figure 4 It is a flowchart of another log information transfer method provided by an embodiment of the present invention;

[0058] Figure 5 It is a flowchart of another log information transfer method provided by an embodiment of the present invention;

[0059] Figure 6 It is a structural schematic diagram of a log information transfer device provided by an embodiment of the present invention;

[0060] Figure 7 It is a schematic diagram of the hardware structure of a computer device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0061] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0062] In distributed file quota technology, in order to ensure the accuracy of quota data statistics, when metadata changes, quota-related data is logged together with file metadata. When metadata logs are subsequently eliminated, we need to transfer the quota companion log information synchronously. Among them, quota companion logs refer to quota-related information when metadata changes in distributed file systems, such as which users, groups, directories generated this data change, and other detailed data are logged together with file metadata.

[0063] If the transfer mapping efficiency and fault reliability are poor during this process, it will make it difficult to evaluate the fault recovery capability, inaccurate quota statistics, and poor quota transfer performance will directly affect business performance.

[0064] To solve the above problems, an embodiment of the present invention provides an embodiment of log information transfer. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system (computer device) including a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0065] In this embodiment, a log information transfer method is provided, which can be used for the above-mentioned terminal devices, such as mobile phones, tablet computers, etc. Figure 1 is a flow chart of a log information transfer method provided by an embodiment of the present invention, such as Figure 1 As shown, the process includes the following steps:

[0066] Step S101: in each polling cycle, respectively obtain the quota associated log information corresponding to each polling cycle.

[0067] In a specific example, the polling cycle can be a logical clock with a specific time interval. Each logical clock will automatically generate a fixed-format sequence identifier, which can be a timestamp, serial number, or other unique representation symbol. Each logical clock will have data written to it. As the data is written, a corresponding quota companion log will be generated in the processor memory. The quota companion log is stored in the form of key-value, where the key is the logical clock and the value is the quota allocation log information.

[0068] The quota log information includes but is not limited to recording which users, groups, and directories generated the data change, the creation, modification, and deletion of quotas caused by data writing, modification, and deletion operations, the amount of resources used by users, quota release events, etc.

[0069] Step S102: Based on the quota associated log information corresponding to the first polling cycle, generate configuration information corresponding to the quota associated log information.

[0070] The configuration information includes the storage location of the quota associated log, the size of the quota associated log information, and identification information corresponding to the first polling cycle, where the first polling cycle is any polling cycle.

[0071] Specifically, first obtain the storage location of the preset file in the quota associated log information pre-constructed storage pool, and pre-construct the storage pool. The pre-constructed storage pool can be a local storage, a cloud storage service, etc. Next, according to the quota associated log information of the first polling cycle, determine the identification information corresponding to the first polling cycle, the size of the quota associated log information of the first polling cycle, and other parameters. Finally, generate specific configuration information corresponding to the polling cycle.

[0072] Step S103 , according to the configuration information corresponding to the quota associated log information of each polling cycle, asynchronously complete the storage operation of the quota associated log information corresponding to each polling cycle.

[0073] Specifically, in an optional example, it can be seen from the above that the configuration information may include configuration information such as the storage location of the quota companion log and the size of the quota companion log information. The storage location of the quota companion log refers to the storage location in the preset file in the pre-built storage pool. The storage location may, for example, include the storage starting position of each quota companion log file stored in the pre-built storage pool.

[0074] Based on the quota associated log information size and the storage starting position of the quota associated log file, the storage ending position of the quota associated log file is further determined. The storage ending position of the quota associated log file can be calculated by adding the storage starting position of the quota associated log file and the size of the quota associated log information. According to the storage starting position of the quota associated log file and the calculated storage ending position of the quota associated log file, the storage operation of the quota associated log information corresponding to each polling cycle is asynchronously completed.

[0075] In a specific example, it is known that the storage end position of the quota companion log file of each polling cycle is obtained by calculation, so the storage start position of the quota companion log file of polling cycle two can be determined by the next byte of the storage end position of the quota companion log file of polling cycle one, and the storage end position of the quota companion log file of polling cycle two is the sum of the storage start position of the quota companion log file of polling cycle two and the size of the quota companion log information of polling cycle two.

[0076] The quota log information of polling cycle 2 does not need to wait for the write completion response notification of polling cycle 1, and can be directly written according to the cache write position in the configuration information of polling cycle 2. The storage operation of the quota log information of each polling cycle is asynchronously completed.

[0077] A log information transfer method provided by an embodiment of the present application obtains the quota associated log information corresponding to each polling cycle and generates corresponding configuration information in each polling cycle. The automated process reduces manual intervention and improves the efficiency of log processing. The storage location, size and identification information corresponding to the polling cycle of the quota associated log information are clearly recorded in the configuration information, which helps to quickly locate and retrieve the quota associated log information, thereby improving the performance of data processing. At the same time, clear configuration information also helps to quickly locate and recover when problems occur in the system, thereby improving the reliability of the system. Asynchronously completing the storage operation of the quota associated log information corresponding to each polling cycle can avoid system blockage caused by synchronous operations and enhance the stability of the system.

[0078] In an optional embodiment, based on the above embodiment, the configuration information is embodied in the form of an operation handle.

[0079] After generating configuration information corresponding to the quota associated log information based on the quota associated log information corresponding to the first polling cycle, the processor allocates a first polling operation handle to the first polling configuration information, and the first polling operation handle points to the configuration information memory block of the first polling.

[0080] The processor obtains or changes the storage location of the quota companion log in the configuration information, the size of the quota companion log information, and the identification information corresponding to the first polling cycle through the operation handle. The implementation of asynchronous storage also needs to obtain the cache write position from the first polling operation handle to start storage. The first polling quota companion log is any polling quota companion log, the first polling configuration information is any polling configuration information, and the first polling operation handle is any polling operation handle.

[0081] Specifically, the handle abstracts different quota associated log information into the same configuration information module for management, which reduces programming complexity and improves code readability. Each quota associated log information configuration information memory block has a unique handle reference, allowing developers to more easily track and debug resource usage, and quickly locate problems when failures occur. If the location of a resource in memory changes, you only need to change the configuration information in the corresponding handle. The external value and identification of the handle remain unchanged, which does not affect the normal operation of the program, greatly enhancing the robustness of the program. The handle greatly reduces the complexity of memory addressing operations, so it is more efficient than directly accessing the memory address, improving the overall performance of the system.

[0082] In this embodiment, a log information transfer method is provided, which can be used for the above-mentioned terminal devices, such as mobile phones, tablet computers, etc. Figure 2 This is a flow chart of another log information transfer method provided by an embodiment of the present invention. Based on any of the above embodiments, the storage location may include, for example, the storage starting location of each quota associated log file stored in the pre-built storage pool. The storage starting location of the quota associated log file may be referred to as the cache write location. According to the configuration information corresponding to the quota associated log information of each polling cycle, the storage operation of the quota associated log information corresponding to each polling cycle is asynchronously completed. For details, see Figure 2 As shown, the method comprises the following steps:

[0083] Step S201, extracting the cache write position and the size of the quota associated log information from the configuration information.

[0084] Step S202: determine the disk write position according to the cache write position and the size of the quota associated log information.

[0085] Specifically, the cache write position in the configuration information is the starting position where the quota companion log information of the current polling cycle is transferred to the preset file of the pre-built storage pool. The sum of the cache write position and the quota companion log information size is the end position where the quota companion log information of the current polling cycle is transferred to the preset file of the pre-built storage pool, that is, the disk write position.

[0086] Step S203: write the quota associated log into a preset file in a pre-built storage pool according to the cache write position and the disk write position.

[0087] The storage operation of the quota associated log information corresponding to each polling cycle is completed asynchronously, which mainly depends on the cache write position and disk write position of the configuration information of each polling cycle.

[0088] In a specific example, the next byte of the disk write position of polling cycle one is the cache write position of polling cycle two. At this time, polling cycle two does not need to wait for the storage operation of polling cycle one to be completed before starting the storage operation, nor does it need to pay attention to the storage status of the quota companion log of polling cycle one. The storage operation can be started directly according to the cache write position in the configuration information.

[0089] In an optional embodiment, the configuration information may further include: a quota companion log format version, an identifier of a written data object, and one or more parameters of a disk write location of the companion log corresponding to the current polling cycle.

[0090] Specifically, the quota companion log format version can ensure that systems or applications of different versions can correctly parse and process quota companion log information, and write data object identifiers to ensure that quota companion log information is associated with the original data object, which helps to maintain data consistency and integrity when restoring quota companion log information of quotas or analyzing original data.

[0091] In this embodiment, a log information transfer method is provided, which can be used for the above-mentioned terminal devices, such as mobile phones, tablet computers, etc. Figure 3 1 is a flow chart of another log information transfer method provided by an embodiment of the present invention. On the basis of the above embodiment, when obtaining the quota associated log information corresponding to the second polling cycle fails within the second polling cycle, the method further includes the following method steps:

[0092] Step S301: According to the second polling cycle, original data written in the second polling cycle is obtained from the original data pool.

[0093] The second polling cycle is any polling cycle.

[0094] Specifically, in each polling cycle, the original data will change, and the quota information generated by the data change will be automatically recorded in the quota companion log. For example, when the client writes data, the corresponding client operation request, the size of the written data, the file name, size, permissions, creation time and other information of the written file will be automatically recorded in the quota companion log. The two are connected by a common polling cycle identifier.

[0095] Step S302: restore the size of the quota associated log information in the second polling cycle according to the original data.

[0096] Specifically, when the automatic recording operation of the second polling cycle quota accompanying log fails, the second polling cycle quota accompanying log information content such as the client's operation request within the second polling cycle, the written data size, the file name of the written file, the file size, the file permissions, the file creation time, etc. can be restored according to the original data, and then the second polling cycle quota accompanying log information size can be restored.

[0097] Step S303: when the second polling cycle is not the first polling cycle, determining the cache write position of the second polling cycle based on the disk write position of the previous polling cycle.

[0098] Specifically, the next byte of the disk write position in the quota companion log configuration information of the previous polling cycle is the cache write position in the second polling cycle configuration information.

[0099] Step S304: based on the quota companion log information size in the second polling cycle and the cache write position of the second polling cycle, restore the disk write position of the quota companion log of the second polling cycle.

[0100] The sum of the cache write position in the second polling cycle configuration information and the size of the quota associated log information restored in the second polling cycle configuration information is the disk write position in the second polling cycle configuration information.

[0101] Step S305: Generate configuration information of the second polling cycle based on the identification information corresponding to the second polling cycle, the size of the quota associated log information of the second polling cycle, and the storage location of the quota log of the second polling cycle.

[0102] Specifically, based on the aforementioned embodiments, it can be known that the configuration information of the second polling cycle may include identification information corresponding to the second polling cycle, the size of the quota associated log information of the second polling cycle, and the storage location of the quota log of the second polling cycle, wherein the corresponding identification of the second polling cycle is a sequential identification automatically generated in a fixed format, which may be a timestamp, a serial number, or other unique representation symbol, and the storage location of the quota associated log of the second polling cycle may include a cache write location, and the cache write location of the second polling cycle and the size of the quota associated log information of the second polling cycle may determine the disk write location of the second polling cycle. Thus, the configuration information of the second polling cycle is restored.

[0103] In another optional embodiment, when the second polling cycle is the first polling cycle, the method further includes the following steps:

[0104] Step a1, determining the first byte of the preset file as the cache write position of the second polling cycle.

[0105] Step a2: Generate configuration information of the second polling cycle based on the size of the quota associated log information restored in the second polling cycle, the cache write position of the second polling cycle, and the identification information corresponding to the second polling cycle.

[0106] Specifically, when the second polling cycle is the first polling cycle, the first byte of the preset file in the pre-built storage pool is the cache write position of the second polling cycle.

[0107] The subsequent method for restoring the configuration information of the quota associated log information based on the original data is the same as above and will not be repeated here.

[0108] Specifically, failure to obtain the quota associated log information of the second polling cycle will result in failure to generate the configuration information of the second polling cycle. Based on the polling cycle, the original data stored in the original data pool in each polling cycle can be determined, and the quota associated log information content can be restored based on the original data, including the quota associated log information size.

[0109] Since the next byte of the disk write position of the previous polling cycle is the cache write position in the second polling cycle configuration information, or when the second polling cycle is the first polling cycle, the first byte of the preset file is the cache write position in the second polling cycle configuration information. The sum of the cache write position in the second polling cycle configuration information and the size of the quota associated log information in the second polling cycle configuration information is the disk write position in the second polling cycle configuration information, and thus the second polling cycle configuration information is restored.

[0110] This solution provides a method for handling the abnormal situation of failure to obtain the quota associated log information corresponding to the second polling cycle. By restoring the quota associated log information size and disk write location, the configuration information can be quickly restored, and then the data processing flow can be quickly restored, reducing the processing delay caused by the loss of log information. At the same time, the complete quota associated log configuration information can also reduce the risk of data loss and enhance data security.

[0111] In another optional example, the method further includes:

[0112] Step b1, obtaining the remaining storage space in the preset file in the pre-built storage pool.

[0113] Step b2: Based on the quota associated log information size and the remaining storage space in the current polling cycle, determine whether the remaining storage space can store the quota associated log information.

[0114] Step b3: when it is determined that the remaining storage space is insufficient to store the quota associated log information of the previous polling cycle, the storage space of the preset file in the pre-built storage pool is expanded until the storage space in the preset file is sufficient to store the quota associated log information of the current polling cycle.

[0115] In this embodiment, based on the size of the quota associated log information and the remaining storage space, it is determined whether the remaining storage space can store the quota associated log information. When the storage space is insufficient, the capacity is automatically expanded to ensure that all quota associated log information can be stored, avoid the loss of quota associated log information, and ensure data integrity.

[0116] In another optional example, the method further includes:

[0117] Back up handles every polling cycle to a pre-built storage pool.

[0118] Specifically, the backup handle can be restored from the storage pool when the handle of the quota associated log information is lost or damaged, reducing the risk of data loss. The backup handle can also be used for diagnosis after a failure occurs, helping to quickly locate the problem and perform recovery.

[0119] In another optional embodiment, based on the above embodiment, the corresponding quota associated log information is generated based on the polling cycle and then asynchronously stored in a preset file in a preset storage pool, and the preset file has an automatic expansion mechanism. As time goes by, the amount of data in the preset file increases, in order to improve the read and write performance. The method also includes the following steps:

[0120] Step c1, presetting the upper limit of the polling cycle times of the storage quota associated log information.

[0121] Step c2, triggering the cold and hot data separation process according to the preset upper limit of the polling cycle of the storage quota associated log information.

[0122] Among them, hot data is usually the quota log information that is recently generated or frequently accessed, while cold data is the quota log information that has a long history (for example, the storage time of historical data is greater than or equal to the preset time threshold) and a low access frequency (for example, the access frequency is lower than the preset access frequency threshold). Store hot data on high-performance, low-latency storage media to ensure fast access. Migrate cold data to storage media with lower costs and slower access speeds, such as hard disk drives (HDDs) or cloud storage services to reduce storage costs.

[0123] Below, a part of the operation process of the above method of the present application will be described in the form of a specific example. Figure 4 As shown:

[0124] Take the two time periods of logical clock key and logical clock key+1 as examples, where the logical clock key is any logical clock, and the logical clock key+1 is the next logical clock of the logical clock key.

[0125] During the time period of the logical clock key, the original data of one or more clients is written into the original data pool, which will generate the quota log information of the logical clock key. The quota log is stored in the form of key-value, where the key is the logical clock and the value is the quota log information. The size of the quota log information and the identification information of the logical clock key are obtained according to the quota log information.

[0126] Before the processor automatically generates configuration information, it should determine whether the logical clock key is the first logical clock. When the logical clock key is the first logical clock, the first byte of the preset file in the pre-built storage pool is obtained as the cache write position of the logical clock key. When the logical clock key is not the first logical clock, the cache write position is the next byte of the disk write position in the previous logical clock configuration information. The disk write position of the logical clock key is calculated by adding the cache write position of the logical clock key and the size of the quota companion log information. Before the storage operation is completed, the disk write position is known.

[0127] After automatically generating the configuration information corresponding to the quota associated log information according to the above method, the key operation handle is used to indicate the configuration information. The purpose of adding the operation handle is to directly obtain the cache write position and disk write position and other parameters in any logical clock configuration information for storage, without waiting for other logical clock storage to complete the response, to achieve asynchronous storage. Even if the configuration information with the logical clock as the key is changed, the identification of the operation handle corresponding to the configuration information will not change, and will not affect the normal operation of the program.

[0128] When performing storage operations, first obtain the operation handle with the logical clock as the key, extract the quota associated log information size in the operation handle, and compare it with the remaining storage space of the preset file in the pre-built storage pool. When it is determined that the remaining storage space is insufficient to store the quota associated log information of the logical clock key, expand the storage space of the preset file in the pre-built storage pool until the storage space in the preset file is sufficient to store the quota associated log information of the logical clock key.

[0129] The quota associated log information of the logical clock key is stored according to the cache write position in the logical clock key operation handle. In the process of storing the quota associated log information of the logical clock as key, the operation handle of the logical clock as key+1 is generated. The generation method and storage operation of the operation handle of the logical clock as key+1 are consistent with the generation method and storage operation of the logical clock key operation handle, and will not be repeated. At this time, there is no need to wait for the response to the completion of the storage operation of the quota associated log information of the logical clock as key. The quota associated log information of the logical clock as key+1 is directly stored according to the cache write position in the logical clock key+1 operation handle, realizing the asynchronous storage of the key and key+1 quota associated log information.

[0130] After the quota companion log information of the logical clock key is written, the cache write position and disk write position in the configuration information indicated by the operation handle key can be updated.

[0131] The above example fully demonstrates the execution process of the quota associated log information handle.

[0132] In particular, when the operation handle key generation fails, the operation handle key is automatically restored. The specific operation process is as follows: Figure 5 As shown:

[0133] When transferring the log associated with the logical clock key, you need to obtain the operation handle key. First, determine whether the operation handle key exists. If the operation handle key of the logical clock key does not exist, determine whether the quota log associated with the operation handle key exists.

[0134] When the quota associated log information corresponding to the logical clock key does not exist, after restoring the quota associated log information of the logical clock key and the size of the quota associated log information based on the original data of the logical clock key, determine whether the logical clock key is the first logical clock.

[0135] When the logical clock key is not in the first polling cycle, the cache write position in the key logical clock configuration information is determined according to the disk write position in the key-1 logical clock operation handle. Based on the cache write position in the key logical clock configuration information and the size of the restored quota companion log information, the disk write position of the configuration information of the logical clock key is determined, and then the configuration information indicated by the operation handle key is restored, and the operation handle key is returned.

[0136] When the logical clock key is in the first polling cycle, the first byte of the preset file in the pre-built storage pool is the cache write position in the key logical clock configuration information. The subsequent method of returning the operation handle key based on the cache write position in the key logical clock configuration information and the size of the restored quota companion log information is the same as above and will not be repeated here.

[0137] When the quota associated log information of the logical clock key exists, the logical clock key configuration information and operation handle key are generated according to the quota associated log information of the logical clock key, and the operation handle key is returned.

[0138] In addition, in order to reduce the data detection interval when the handle cache is restored, a periodic execution task is started to batch update the disk write location in the configuration information to the storage pool based on the original data to back up the configuration information.

[0139] In this embodiment, a log information transfer operation device is also provided, which is used to implement the above-mentioned embodiments and preferred implementation modes, and the descriptions that have been made will not be repeated. As used below, the term "module" can implement a combination of software and / or hardware of a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceivable.

[0140] The operating device for transferring log information in this embodiment is as follows: Figure 6 As shown, it includes: an acquisition module 601 and a processing module 602.

[0141] The acquisition module 601 is used to respectively acquire the quota associated log information corresponding to each polling cycle in each polling cycle;

[0142] Processing module 602 is used to generate configuration information corresponding to the quota associated log information based on the quota associated log information corresponding to the first polling cycle, wherein the configuration information includes the storage location of the quota associated log, the size of the quota associated log information, and identification information corresponding to the first polling cycle, and the first polling cycle is any polling cycle; according to the configuration information corresponding to the quota associated log information of each polling cycle, the storage operation of the quota associated log information corresponding to each polling cycle is asynchronously completed.

[0143] In an optional embodiment, the processing module 602 is specifically configured to reflect the configuration information in the form of an operation handle.

[0144] In an optional embodiment, the processing module 602 is further used to extract the cache write position and the size of the quota associated log information from the configuration information;

[0145] Determine the disk write location based on the cache write location and the size of the quota associated log information;

[0146] The quota companion log is written to the preset file in the pre-built storage pool according to the cache write location and the disk write location.

[0147] In an optional embodiment, the configuration information may further include: a quota companion log format version, an identifier of a written data object, and one or more parameters of a disk write location of the companion log corresponding to the current polling cycle.

[0148] In an optional embodiment, the processing module 602 is further configured to, when the quota associated log information corresponding to the second polling cycle fails to be obtained in the second polling cycle, obtain the original data written in the second polling cycle from the original data pool according to the second polling cycle, wherein the second polling cycle is any polling cycle other than the first polling cycle;

[0149] Restore the size of the quota associated log information in the second polling cycle based on the original data;

[0150] When the second polling cycle is not the first polling cycle, determining the cache write position of the second polling cycle based on the disk write position of the previous polling cycle;

[0151] Based on the quota companion log information size in the second polling cycle and the cache write position in the second polling cycle, restore the disk write position of the quota companion log in the second polling cycle;

[0152] Configuration information of the second polling cycle is generated based on the identification information corresponding to the second polling cycle, the size of the quota associated log information of the second polling cycle, and the storage location of the quota log of the second polling cycle.

[0153] In an optional embodiment, the processing module 602 is further configured to, when the second polling cycle is the first polling cycle, determine that the first byte of the preset file is the cache write position of the second polling cycle;

[0154] Configuration information of the second polling cycle is generated based on the size of the quota associated log information restored in the second polling cycle, the cache write position of the second polling cycle, and the identification information corresponding to the second polling cycle.

[0155] In an optional embodiment, the processing module 602 is further used to obtain the remaining storage space of the preset file in the pre-built storage pool;

[0156] Based on the quota associated log information size and the remaining storage space in the current polling cycle, determine whether the remaining storage space can store the quota associated log information;

[0157] When it is determined that the remaining storage space is insufficient to store the quota associated log information of the previous polling cycle, the storage space of the preset file in the pre-built storage pool is expanded until the storage space in the preset file is sufficient to store the quota associated log information of the current polling cycle.

[0158] In an optional embodiment, the processing module 602 is further configured to back up the handle of each polling cycle to a pre-built storage pool.

[0159] The log information transfer device in this embodiment is presented in the form of a functional module, where the module refers to an application specific integrated circuit (ASIC), a processor and memory that executes one or more software or fixed programs, and / or other devices that can provide the above functions.

[0160] The further functional description of each of the above modules and units is the same as that of the above corresponding embodiments and will not be repeated here.

[0161] The embodiment of the present invention provides a log information transfer device with a modular design, so that each module can work in parallel, thereby improving the overall processing efficiency.

[0162] Obtain the quota log information corresponding to each polling cycle. Automatic acquisition reduces manual intervention and greatly improves processing efficiency. Based on the quota log information corresponding to the first polling cycle, generate configuration information corresponding to the quota log information. Clear configuration information helps to quickly locate and restore when problems occur in the system, improving system reliability.

[0163] The processing module processes the configuration information of each polling cycle, making it easier to locate and retrieve log data. The quota associated log information of each polling cycle generates corresponding configuration information and is stored separately. Even if the quota associated log storage operation of a polling cycle fails, it will not affect the execution of the quota associated log storage operation of the remaining polling cycles, thereby improving the stability of the system.

[0164] An embodiment of the present invention further provides a computer device, Figure 7 is a schematic diagram of the structure of a computer device provided by an optional embodiment of the present invention, such as Figure 6As shown, the computer device includes: one or more processors 10, a memory 20, and interfaces for connecting various components, including high-speed interfaces and low-speed interfaces. Various components are connected to each other using different buses for communication, and can be installed on a common mainboard or installed in other ways as needed. The processor can process the instructions executed in the computer device, including instructions stored in or on the memory to display the graphical information of the GUI on an external input / output device (such as, a display device coupled to the interface). In some optional embodiments, if necessary, multiple processors and / or multiple buses can be used together with multiple memories and multiple memories. Similarly, multiple computer devices can be connected, and each device provides some necessary operations (for example, as a server array, a group of blade servers, or a multi-processor system). Figure 5 A processor 10 is taken as an example.

[0165] The processor 10 may be a central processing unit, a network processor or a combination thereof. The processor 10 may further include an integrated circuit. The integrated circuit may be a dedicated integrated circuit, a programmable logic device or a combination thereof. The programmable logic device may be a complex programmable logic device, a field programmable gate array, a general purpose array logic or any combination thereof.

[0166] The memory 20 stores instructions executable by at least one processor 10, so that at least one processor 10 executes the method shown in the above embodiment.

[0167] The memory 20 may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created by the use of a computer device based on the presentation of a small program landing page, etc. In addition, the memory 20 may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some optional embodiments, the memory 20 may optionally include a memory remotely arranged relative to the processor 10, and these remote memories may be connected to the computer device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0168] The memory 20 may include a volatile memory, such as a random access memory; the memory may also include a non-volatile memory, such as a flash memory, a hard disk or a solid state drive; the memory 20 may also include a combination of the above types of memory.

[0169] The computer device also includes an input device 30 and an output device 40. The processor 10, the memory 20, the input device 30 and the output device 40 may be connected via a bus or other means. Figure 5 The example of connecting through bus is taken in the following.

[0170] The input device 30 can receive input digital or character information, and generate key signal input related to the user settings and function control of the computer device, such as a touch screen, a keypad, a mouse, a track pad, a touch pad, an indicator bar, one or more mouse buttons, a trackball, a joystick, etc. The output device 40 may include a display device, an auxiliary lighting device (e.g., an LED) and a tactile feedback device (e.g., a vibration motor), etc. The above-mentioned display device includes but is not limited to a liquid crystal display, a light emitting diode, a display and a plasma display. In some optional embodiments, the display device can be a touch screen.

[0171] The embodiment of the present invention also provides a computer-readable storage medium. The method provided in the above embodiment of the present invention can be implemented in hardware, firmware, or implemented as a computer code that can be recorded in a storage medium, or is implemented as a computer code that is originally stored in a remote storage medium or a non-temporary machine-readable storage medium and will be stored in a local storage medium and downloaded through a network, so that the method described herein can be stored in such software processing on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. Among them, the storage medium can be a magnetic disk, an optical disk, a read-only storage memory, a random access memory, a flash memory, a hard disk or a solid-state hard disk, etc.; further, the storage medium can also include a combination of the above types of memories. It can be understood that a computer, a processor, a microprocessor controller, or programmable hardware includes a storage component that can store or receive software or computer code. When the software or computer code is accessed and executed by a computer, a processor, or hardware, the method shown in the above embodiment is implemented.

[0172] A part of the present invention may be applied as a computer program product, such as a computer program instruction, which, when executed by a computer, can call or provide the method and / or technical solution according to the present invention through the operation of the computer. Those skilled in the art should understand that the existence of the computer program instruction in a computer-readable medium includes, but is not limited to, a source file, an executable file, an installation package file, etc., and accordingly, the way in which the computer program instruction is executed by the computer includes, but is not limited to: the computer directly executes the instruction, or the computer compiles the instruction and then executes the corresponding compiled program, or the computer reads and executes the instruction, or the computer reads and installs the instruction and then executes the corresponding installed program. Here, the computer-readable medium may be any available computer-readable storage medium or communication medium accessible to the computer.

[0173] Although the embodiments of the present invention have been described in conjunction with the accompanying drawings, those skilled in the art may make various modifications and variations without departing from the spirit and scope of the present invention, and such modifications and variations are all within the scope defined by the appended claims.

Claims

1. A log information transfer method, characterized in that: The method comprises: In each polling cycle, respectively obtain the quota associated log information corresponding to each of the polling cycles; Based on the quota associated log information corresponding to the first polling cycle, generate configuration information corresponding to the quota associated log information, wherein the configuration information includes a storage location of the quota associated log, a size of the quota associated log information, and identification information corresponding to the first polling cycle, and the first polling cycle is any polling cycle; According to the configuration information respectively corresponding to the quota associated log information of each polling cycle, a storage operation of the quota associated log information respectively corresponding to each polling cycle is asynchronously completed.

2. The method according to claim 1, characterized in that The configuration information is embodied in the form of an operation handle.

3. The method according to claim 1 or 2, characterized in that: The storage location of the quota companion log includes a cache write location; The asynchronously completing the storage operation of the quota associated log information corresponding to each of the polling cycles according to the configuration information corresponding to the quota associated log information of each of the polling cycles, includes: Extracting the cache write position and the quota associated log information size from the configuration information; Determine the disk write location according to the cache write location and the size of the quota associated log information; The quota companion log is written into a preset file in a pre-built storage pool according to the cache write position and the disk write position.

4. The method according to claim 1 or 2, characterized in that: The configuration information also includes: The quota companion log format version, the identifier of the data object to be written, and one or more parameters of the disk write location of the companion log corresponding to the current polling cycle.

5. The method according to claim 3, characterized in that: When obtaining the quota associated log information corresponding to the second polling period fails within the second polling period, the method further includes: According to a second polling cycle, obtaining the original data written in the second polling cycle from the original data pool, wherein the second polling cycle is any polling cycle except the first polling cycle; Restoring the size of the quota associated log information in the second polling period according to the original data; When the second polling cycle is not the first polling cycle, determining the cache write position of the second polling cycle based on the disk write position of the previous polling cycle; Based on the size of the quota companion log information in the second polling cycle and the cache write position of the second polling cycle, restore the disk write position of the quota companion log of the second polling cycle; Configuration information of the second polling cycle is generated based on identification information corresponding to the second polling cycle, a size of the quota associated log information of the second polling cycle, and a storage location of the quota log of the second polling cycle.

6. The method according to claim 5, characterized in that When the second polling cycle is the first polling cycle, the method further includes: Determine that the first byte of the preset file is the cache write position of the second polling cycle; Configuration information of the second polling cycle is generated based on the size of the quota companion log information restored in the second polling cycle, the cache write position of the second polling cycle, and the identification information corresponding to the second polling cycle.

7. The method according to claim 3, characterized in that The method further comprises: Obtaining remaining storage space of preset files in the pre-built storage pool; Based on the size of the quota associated log information in the current polling cycle and the remaining storage space, determining whether the remaining storage space can store the quota associated log information; When it is determined that the remaining storage space is insufficient to store the quota associated log information of the previous polling cycle, the storage space of the preset file in the pre-built storage pool is expanded until the storage space in the preset file is sufficient to store the quota associated log information of the current polling cycle.

8. A log information transfer device, characterized in that: The device comprises: The acquisition module is used to respectively acquire the quota associated log information corresponding to each polling cycle in each polling cycle; A processing module is used to generate configuration information corresponding to the quota associated log information based on the quota associated log information corresponding to the first polling cycle, wherein the configuration information includes the storage location of the quota associated log, the size of the quota associated log information, and identification information corresponding to the first polling cycle, and the first polling cycle is any polling cycle; according to the configuration information corresponding to the quota associated log information of each polling cycle, asynchronously complete the storage operation of the quota associated log information corresponding to each polling cycle.

9. A computer device, characterized in that: include: A memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the log information transfer method according to any one of claims 1 to 7 by executing the computer instructions.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a computer to execute the log information transfer method according to any one of claims 1 to 7.

Images