Data processing method for memory limitation and related device

Abstract

The invention provides a data processing method for memory limitation and a related device. The method comprises the following steps: acquiring a first object corresponding to a target application program; determining a first reference address corresponding to the first object definition, and compressing the first reference address according to a preset coding function to obtain a second reference address; filling the first object data according to a preset storage structure and a preset storage rule to obtain reference first object data; creating a self-recovery log file in the target disk, and determining a self-recovery log file write operation mechanism according to the self-recovery log file; acquiring a first data processing instruction associated with the reference first object data; and executing the first data processing instruction on the reference first object data according to the self-recovery log file write operation mechanism to obtain target first object data so as to improve the data processing efficiency.

Description

Technical Field

[0001] The present application relates to the technical field of data storage, and in particular to a data processing method and related device for memory-limited data. Background Art

[0002] When computing large amounts of data, if the machine resources are sufficient, the computing can be completed through distributed storage and distributed computing clusters built by multiple machines. However, when there is only a single machine, insufficient memory will occur during the computing process. At present, in order to solve this problem, the memory is generally compressed and exchanged through page replacement algorithms, memory compression algorithms, and swap space mechanisms, thereby saving and expanding memory space to a certain extent.

[0003] However, the page replacement algorithm in the current solution does not directly optimize the calculation of memory data, but makes decisions based on the access frequency or recent usage of the page. Frequent page replacement may occur, resulting in a large number of disk read and write operations, thereby reducing the overall performance of the system. Among them, the memory compression algorithm can only enjoy the memory space savings brought by compression for the data selected for compression, and cannot satisfy all memory data. In addition, the disk read and write operations will consume a lot of computing resources, resulting in the concentration of computing resources on read and write operations, reducing the processing power of the machine.

[0004] Therefore, how to optimize the storage of memory data and improve memory computing and file interaction mechanisms to optimize memory utilization and improve the processing performance of large-scale data computing has become a problem to be solved. Summary of the invention

[0005] The embodiments of the present application provide a data processing method and related devices for memory-constrained data, which achieve storage optimization of memory data and improve memory computing and file interaction mechanisms to optimize memory utilization and improve processing performance of large-scale data calculations.

[0006] In a first aspect, an embodiment of the present application provides a data processing method for limited memory, which is applied to a data parsing and conversion machine, wherein a target device includes the data parsing and conversion machine, a target application, a target memory, and a target disk, and the method includes:

[0007] Acquire a first object corresponding to the target application; the first object is stored in the target memory; the first object includes first object data and a first object definition;

[0008] Determine a first reference address corresponding to the first object definition, and compress the first reference address according to a preset encoding function to obtain a second reference address; the second reference address is used to point to the first object definition;

[0009] Filling the first object data according to a preset storage structure and a preset storage rule to obtain reference first object data, so as to save data space of the target memory;

[0010] Creating a self-recovery log file in the target disk, and determining a self-recovery log file write operation mechanism based on the self-recovery log file; the self-recovery log file write operation mechanism is used to store data processing operations to ensure that data can be recovered after a system crash or failure;

[0011] Obtaining a first data processing instruction associated with the reference first object data;

[0012] The first data processing instruction is executed on the reference first object data according to the self-recovery log file write operation mechanism to obtain target first object data, so as to improve data processing efficiency.

[0013] In a second aspect, an embodiment of the present application provides a data processing device for memory-constrained data, which is applied to a data parsing and conversion machine, wherein a target device includes the data parsing and conversion machine, a target application, a target memory, and a target disk, and the device includes an acquisition module, a determination module, a filling module, a creation module, and an execution module, wherein:

[0014] The acquisition module is used to acquire a first object corresponding to the target application; the first object is stored in the target memory; the first object includes first object data and a first object definition;

[0015] The determining module is used to determine a first reference address corresponding to the first object definition, and compress the first reference address according to a preset encoding function to obtain a second reference address; the second reference address is used to point to the first object definition;

[0016] The filling module is used to fill the first object data according to a preset storage rule according to a preset storage structure to obtain reference first object data, so as to save data space of the target memory;

[0017] The creation module is used to create a self-recovery log file in the target disk and determine a self-recovery log file write operation mechanism based on the self-recovery log file; the self-recovery log file write operation mechanism is used to store data processing operations to ensure that data can be recovered after a system crash or failure;

[0018] The acquisition module is further used to acquire the first data processing instruction associated with the reference first object data;

[0019] The execution module is used to execute the first data processing instruction on the reference first object data according to the self-recovery log file write operation mechanism to obtain the target first object data, so as to improve data processing efficiency.

[0020] In a third aspect, an embodiment of the present application provides an electronic device, comprising a processor, a memory, a communication interface, and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the processor, and the program includes instructions for executing the steps of any method in the first aspect of the embodiment of the present application.

[0021] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program for electronic data exchange, wherein the computer program enables a computer to execute part or all of the steps described in any method of the first aspect of the embodiment of the present application.

[0022] In a fifth aspect, an embodiment of the present application provides a computer program product, wherein the computer program product includes a non-transitory computer-readable storage medium storing a computer program, and the computer program is operable to cause a computer to execute some or all of the steps described in any method of the first aspect of the embodiment of the present application. The computer program product may be a software installation package.

[0023] By implementing the embodiments of the present application, storage optimization of memory data is achieved, and memory computing and file interaction mechanisms are improved to optimize memory utilization and improve processing performance for large amounts of data calculations. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0025] Figure 1 It is a flowchart of a method for processing data with limited memory provided by an embodiment of the present application;

[0026] Figure 2 is a schematic diagram of a first optimization mechanism provided in an embodiment of the present application;

[0027] Figure 3 is a schematic diagram of a first memory structure of object data provided in an embodiment of the present application;

[0028] Figure 4is a schematic diagram of a second memory structure of object data provided in an embodiment of the present application;

[0029] Figure 5 is a schematic diagram of a memory structure of OBJ1 provided in an embodiment of the present application;

[0030] Figure 6 It is a schematic diagram of the memory structure of OBJ2 provided in an embodiment of the present application;

[0031] Figure 7 It is a structural diagram of a self-recovery log file provided in an embodiment of the present application;

[0032] Figure 8 is a schematic diagram of a second optimization mechanism provided in an embodiment of the present application;

[0033] Fig. 9 It is a structural schematic diagram of an electronic device provided in an embodiment of the present application;

[0034] Fig.10 It is a block diagram of the functional modules of a data processing device with limited memory provided by an embodiment of the present application. DETAILED DESCRIPTION

[0035] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0036] The terms "first", "second", etc. in the specification and claims of this application and the above-mentioned drawings are used to distinguish different objects, rather than to describe a specific order. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not limited to the listed steps or units, but optionally includes steps or units that are not listed, or optionally includes other steps or units inherent to these processes, methods, products or devices.

[0037] It should be understood that the term "and / or" in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article indicates that the associated objects before and after are in an "or" relationship. The "plurality" appearing in the embodiments of the present application refers to two or more.

[0038] In the embodiments of the present application, "at least one item" or similar expressions refer to any combination of these items, including any combination of single items or plural items, and refer to one or more, and multiple refers to two or more. For example, at least one item of a, b, or c can represent the following seven situations: a, b, c, a and b, a and c, b and c, a, b, and c. Among them, each of a, b, and c can be an element or a set containing one or more elements.

[0039] The "connection" that appears in the embodiments of the present application refers to various connection methods such as direct connection or indirect connection to achieve communication between devices, and the embodiments of the present application do not impose any limitations on this.

[0040] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0041] The following is an explanation of the relevant terms involved in this application, as follows:

[0042] Object data: int, char, etc. are basic data types, while object data is a complex data structure that contains multiple basic data types.

[0043] Disk IO: refers to the data reading and writing operations on disk devices in a computer system. I stands for data input, i.e., read operation, and O stands for data output, i.e., write operation.

[0044] Disk random read and write IO: When reading and writing data, the disk needs to seek to the data point first, and positioning is required every time data is read or written. This IO operation is called disk random read and write IO.

[0045] Disk sequential read and write IO: When the head has been positioned at a specified data point and needs to perform subsequent read and write operations, it only needs to continue reading from the specified data point without re-seeking the head. This IO operation is disk sequential read and write IO.

[0046] Page table: records the status of memory data pages.

[0047] Reference address: A number pointing to the starting address of a memory unit.

[0048] Central Processing Unit (CPU): The main processor in a computer system, responsible for executing instructions, processing data, and controlling various operations of the computer. It is one of the core components of a computer and is often referred to as the "brain of the computer."

[0049] Merge operation: specifically refers to the action of merging and updating the data in the "data processing cache page" to the "original data page".

[0050] Purge operation: specifically refers to the action of forming a "data change page" after the data on the "original data page" changes, and writing the data on the "data change page" to the target disk at the appropriate time.

[0051] When computing a large amount of data, if the machine resources are sufficient, the computing can be completed through the distributed storage and distributed computing cluster built by multiple machines. However, when there is only a single machine, the problem of insufficient memory will occur during the computing process. At present, in order to solve this problem, the memory is generally compressed and exchanged through page replacement algorithms, memory compression algorithms, and swap space mechanisms, thereby saving and expanding the memory space to a certain extent. However, the page replacement algorithm in the current scheme does not directly optimize the calculation of memory data, but makes decisions based on the access frequency or recent usage of the page. Frequent page replacement may occur, resulting in a large number of disk read and write operations, thereby reducing the overall performance of the system. Among them, the memory compression algorithm can only enjoy the memory space savings brought by compression for the data selected for compression, and cannot meet the needs of all memory data. In addition, the disk read and write operations will consume a lot of computing resources, resulting in the concentration of computing resources on read and write operations, reducing the processing power of the machine.

[0052] Therefore, how to optimize the storage of memory data and improve memory computing and file interaction mechanisms to optimize memory utilization and improve the processing performance of large-scale data computing has become a problem to be solved.

[0053] To solve the above problems, an embodiment of the present application provides a data processing method and related device for memory-limited data, which are applied to a data parsing and conversion machine, and a target device includes the data parsing and conversion machine, a target application, a target memory and a target disk. First, a first object corresponding to the target application is obtained; the first object is stored in the target memory; the first object includes first object data and a first object definition; then, a first reference address corresponding to the first object definition is determined, and the first reference address is compressed according to a preset encoding function to obtain a second reference address; the second reference address is used to point to the first object definition; then, the first object data is filled according to a preset storage rule according to a preset storage structure to obtain reference first object data to save data space in the target memory; a self-recovery log file is created in the target disk, and a self-recovery log file write operation mechanism is determined according to the self-recovery log file; the self-recovery log file write operation mechanism is used to store data processing operations to ensure that data can be recovered after a system crash or failure; finally, a first data processing instruction associated with the reference first object data is obtained; the first data processing instruction is executed on the reference first object data according to the self-recovery log file write operation mechanism to obtain target first object data to improve data processing efficiency. The storage of memory data is optimized, and the memory computing and file interaction mechanisms are improved to optimize memory utilization and improve the processing performance of large-scale data computing.

[0054] Combine the following Figure 1 A method for processing data with limited memory in an embodiment of the present application is described. Figure 1 A flowchart of a method for processing data with limited memory provided in an embodiment of the present application is applied to a data parsing and conversion machine, and a target device includes the data parsing and conversion machine, a target application, a target memory, and a target disk, and specifically includes the following steps:

[0055] Step S101, obtaining a first object corresponding to the target application.

[0056] The first object is stored in the target memory; the first object includes first object data and a first object definition.

[0057] Specifically, the target device includes but is not limited to a computer, a smart phone, a tablet computer, a game console, and a server, which are not specifically limited here. The memory cache page in the target memory is read by the data parsing and conversion machine, and the first object corresponding to the target application is accessed according to the memory start address corresponding to the memory cache page. The data parsing and conversion machine is used to process and convert data, and can perform data parsing, data conversion, data integration, data cleaning and preprocessing on the data to ensure the consistency and accuracy of the data between different systems and applications.

[0058] Step S102: determine a first reference address corresponding to the first object definition, and compress the first reference address according to a preset encoding function to obtain a second reference address.

[0059] The second reference address is used to point to the first object definition.

[0060] The step of compressing the first reference address according to a preset encoding function to obtain the second reference address specifically includes:

[0061] Obtain the starting offset of the memory area corresponding to the first reference address; subtract the first reference address from the starting offset to obtain a reference reference address; unsignedly right-shift the reference reference address by a bits to perform address compression to obtain the second reference address; a is a positive integer.

[0062] Specifically, when the target application performs a data write operation, the data parsing and conversion machine first determines whether the type corresponding to the data is a reference address. If the data is a reference address, the preset encoding function is called to compress the data. The encoding function can be the encode_oop(ref_address) function, which is not specifically limited here. Then, the compressed data is stored in the memory area. If the data is a non-reference address, it does not need to be compressed and is directly stored in the memory area. Among them, in the process of calling the first encoding function to compress the data, the starting offset of the memory area corresponding to the first reference address can be obtained, and then the first reference address is subtracted from the starting offset to obtain a reference reference address, and then the reference reference address is unsignedly right-shifted by a bit to perform address compression to obtain a second reference address. It should be noted that the algorithm logic of the reference address compression operation is relatively simple, and unlike the existing memory compression algorithm, the computing power resources consumed by the compression operation are negligible.

[0063] In one possible embodiment, when the target application performs a data read operation, the data parsing and conversion machine first determines whether the type corresponding to the data is a reference address. If the data is a reference address, the preset decoding function is called for decompression. The encoding function can be a decode_oop (narrow_address) function, which is not specifically limited here. Then, the decompressed reference address is memory-addressed, and the actual data location is accessed before returning to the target application. If the data is a non-reference address, there is no need to decompress it. The memory addressing is directly performed according to the memory start address pointed to by the data, and the actual data is accessed before returning to the target application. Among them, in the process of calling the preset decoding function for decompression, the compressed second reference address can be unsignedly shifted left by a bit, and then the starting offset is added to obtain the initial first reference address.

[0064] It can be seen that by compressing the reference address, the memory usage can be significantly reduced, thereby reducing the demand for memory bandwidth and having a positive impact on the overall performance of the target device.

[0065] For easier understanding, see Figure 2 , Figure 2 A schematic diagram of a first optimization mechanism provided for an application embodiment shows that the original reference address mapping table is removed from the target memory, a data analysis and conversion machine is added, and the object data area is pointed to the object definition area. The target application points to the page table in the target memory. In the page table, "original" means that the corresponding memory data page is the original data in the target memory; "compressed" means that the corresponding memory data page is the data obtained after the original data in the target memory is compressed; "disk" means that the corresponding memory data page is the data obtained after the original data in the target memory is replaced with the target disk. Through the data analysis and conversion machine, point to the object data area corresponding to the memory data page, and then point to the corresponding definition area from the object data area. Pg is the memory data page identifier, for example, Pg1 corresponds to memory data page 1. OBJ is the object identifier, then OBJ1 and OBJ2 are the first object and the second object respectively, then OBJ1 data is the first object data, OBJ1 is defined as the first object definition, OBJ2 data is the second object data, and OBJ2 is defined as the second object definition.

[0066] It should be noted that the original reference address mapping table stores two reference addresses corresponding to each object, namely the object definition reference address and the object data reference address. Under a 64-bit operating system, the reference addresses will each occupy 8B of storage space, resulting in each object occupying 16B of storage space. In order to save this part of memory storage, the target application can be pointed to the object data area, that is, the memory area pointed to by the object data reference address in the original reference address mapping table, through a data parsing and conversion machine, to save 8B of memory storage. Then, the memory structure of the object data is transformed, that is, an object definition reference address is added to the object data header to replace the object definition reference address in the original reference address mapping table.

[0067] For easier understanding, see Figure 3 , Figure 3A schematic diagram of the first memory structure of an object data provided for the application embodiment shows that the variable name of the object definition reference address corresponding to OBJ1 is ref_md1(OBJ1), the variable name of the object definition reference address corresponding to OBJ2 is ref_md2(OBJ2), and the variable name of the reference address pointing to OBJ1 in OBJ2 is ref_obj1, that is, the reference address ref_obj1 pointing to the object data of OBJ1 is stored in the object data of OBJ2. It should be noted that the gray area is used to represent memory filling, which is actually blank memory data without actual data meaning, and each row area represents 8B of data space. Among them, the memory consumption of all reference addresses can be optimized through the data parsing and conversion machine to save memory storage.

[0068] It should be noted that in the original memory access method, the target application basically reads the reference address of the memory object, and then performs related memory operations. Under a 64-bit operating system, the CPU data bus reads 8B of data at a time, whether writing data to the memory or reading data from the memory. If this processing mechanism is adopted, the reference address occupies 8B of memory space, while the non-reference address does not have this limitation and can be 1B or 2B. In order to avoid the CPU data bus accessing data across cache lines as much as possible, memory alignment and memory padding are generally used to ensure that the memory space of the data is 8B aligned to improve data processing efficiency. For example, if the complete data to be read is scattered in two cache lines, it needs to be read twice to get the data together. Therefore, when compressing and optimizing the reference address of 8B, it is necessary to consider not only the alignment needs of the data, but also whether the addressing capacity of the compressed memory is insufficient. For example, if 8B is compressed into 5B, 6B, and 7B, the addressing capacity of the memory is sufficient, and the corresponding addressing capacities are 2 respectively. 8 ×4GB, 2 16 ×4GB, 2 24 ×4GB, but this compression method will make memory alignment and filling more difficult.

[0069] It can be seen that the eight basic data types of object data include long, double, float, int, short, byte, boolean, and char, of which two occupy 8B (e.g., long and double), two occupy 4B (e.g., float and int), one occupies 2B (e.g., short), and the last three occupy 1B (e.g., byte, boolean, and char). When 8B is compressed to 5B, the remaining 3B of space may need to be filled with blank memory, or it can be filled with short, byte, boolean, and char; when 8B is compressed to 6B, the remaining 2B may need to be filled with blank memory, or it can be filled with short, byte, boolean, and char; when 8B is compressed to 7B, the remaining 1B may need to be filled with blank memory, or it can be filled with byte, boolean, and char. If 8B is compressed into 4B, the memory addressing capacity can still be maintained at 4GB, and there are 6 basic data types that can be used for alignment (for example, float, int, short, byte, boolean, char). Therefore, compression into 4B can save as much memory as possible due to the extra memory consumption caused by memory alignment and padding.

[0070] In addition, in the modified memory structure, the header of the object data area includes an 8B reference address. Together with the data in the object data area, the memory space occupied by the object is always greater than 8B, which means that the memory starting address of the object is a multiple of 8 (including 0), namely 0x0000, 0x0008, 0x0010, 0x0018, and 0x0020. It can be seen that in binary expression, the lowest 3 bits of the binary code of the memory starting address are always "000". Therefore, these 3 bits can be used to optimize memory addressing, that is, the actual memory addressing capacity is 2 3 ×4GB=32GB. It can be seen that compressing the reference address 8B into 4B not only solves the need for memory addressing, but also solves the problem of memory consumption of compressed reference addresses.

[0071] For easier understanding, see Figure 4 , Figure 4 A second memory structure diagram of object data provided for an embodiment of the application shows that not only the reference address defined by the object data can be compressed, but also the reference addresses pointing to other objects can be compressed. For example, the reference address variable ref_obj1 pointing to OBJ1 in OBJ2 is compressed from 8B to 4B in data size.

[0072] It should be noted that after the reference address is compressed, the reading and writing of its data cannot rely on the original memory mechanism, because the operating system stores and reads data in 8B by default, but after compression, the data is stored in 4B. Therefore, the data can be processed by a data parsing and conversion machine, that is, to determine whether the data is a reference address, and to compress or decompress the data according to different data operations. Among them, when 8B is compressed to 4B, an unsigned right shift of 32 bits is required. In order to increase the memory addressing capability, considering that the memory space occupied by the object data is aligned to 8B, the right shift can be reduced by 3 bits, that is, only an unsigned right shift of 29 bits is required. It can be seen from this that a can be 29, which is used to represent an unsigned right shift of 29 bits to perform the compression operation.

[0073] Step S103: filling the first object data according to a preset storage structure and a preset storage rule to obtain reference first object data, so as to save data space of the target memory.

[0074] The first object data includes a plurality of sub-data; the preset storage structure includes multiple layers of data space, and the memory size of each layer of data space is the same; the first object data is filled according to the preset storage rule according to the preset storage structure to obtain the reference first object data, specifically including:

[0075] If the first object data has no inheritance hierarchy, the plurality of sub-data are sorted according to a preset first sorting rule to obtain a first sub-data sequence; the first sub-data sequence is filled into the multi-layer data space in a sequential order to obtain the reference first object data;

[0076] If the first object data includes multiple inheritance levels, the sub-data of each inheritance level in the multiple inheritance levels are sorted according to the first sorting rule to obtain multiple reference second sub-data sequences; each inheritance level corresponds to a reference second sub-data sequence; the multiple reference second sub-data sequences are sorted according to the order of the inheritance levels from top to bottom to obtain a target second sub-data sequence; the target second sub-data sequence is filled into the multi-layer data space in order to obtain the reference first object data.

[0077] It should be noted that the first object data may be OBJ1 object data, and the OBJ1 object data may include multiple sub-data, such as char c1, int i1, int i12, etc. The preset storage structure includes multiple layers of data space, and the memory size of each layer of data space is 8B.

[0078] In a possible embodiment, if the first object data includes multiple inheritance levels, at least one sub-data of inheritance level i is obtained; the inheritance level i is any inheritance level among the multiple inheritance levels; if a preset first type exists in the data type corresponding to the at least one sub-data, and the data space corresponding to the inheritance level i-1 is not full, the at least one sub-data is sorted according to a preset second sorting rule to obtain a reference third sub-data sequence; the reference third sub-data sequence is filled into the data space corresponding to the inheritance level i-1 in order of precedence; when the data space corresponding to the inheritance level i-1 cannot store data, all sub-data not stored in the data space are sorted according to the first sorting rule to obtain a reference fourth sub-data sequence; the reference fourth sub-data sequence is filled into the multi-layer data space in order of precedence to obtain the reference first object data.

[0079] It should be noted that although the reference address is reduced from 8B to 4B through compression, due to memory alignment and padding, the 4B of data space actually saved still needs to be padded with blank data. Therefore, the object data can be padded according to the preset storage structure and the preset storage rules to save the data space of the target memory.

[0080] The preset storage rules are as follows:

[0081] Rule 1: All objects are aligned with 8B.

[0082] Rule 2: Object data in the object data area are aligned according to a preset first sorting rule, and the first sorting rule is to sort the object data in the following order: double=long>float=int>short>byte>char>boolean>ref, where ref represents a reference address.

[0083] Rule 3: If the object data includes multiple inheritance levels, the data areas corresponding to each inheritance level should not be mixed and sorted. You can sort from top to bottom according to the inheritance level. After the upper inheritance level is sorted, the lower inheritance level can be sorted. Among them, the data in the same inheritance level is sorted according to the first sorting rule.

[0084] Rule 4: If the object data includes multiple inheritance levels, the data type of the first sub-data in the current inheritance level is the preset first type, and the previous inheritance level has not used up the 8B data space, then the data parsing and conversion machine breaks Rule 2, sorts the sub-data according to the preset second sorting rule, and aligns and fills them into the remaining data space. After the data space is aligned according to 8B, the remaining data is sorted again according to Rule 2. The preset first type is double or long, and the second sorting rule is to sort the object data in the following order: int>short>byte>ref.

[0085] Rule 5: If the data type of the first sub-data in the object data is the preset first type, and the object definition reference address has not used up 8B of data space, the same processing as Rule 4 is performed.

[0086] It should be noted that the object data storage is stored according to the new rules and is also read according to the new rules. For example, the operations of reference address compression and decompression are the same and can be implemented through a data parsing and conversion machine.

[0087] In a possible embodiment, in the data structure of OBJ1, there is no double or long in its data area. Therefore, although the object definition reference address in the object header only uses 4B and not all 8B, it cannot meet all the conditions of Rule 5 and is therefore unavailable. OBJ1 has no inheritance hierarchy, so Rule 3 and Rule 4 cannot be used for optimization. In this case, optimization and adjustment can be performed using Rule 1 and Rule 2.

[0088] Among them, the original OBJ1 data structure is as follows:

[0089]

[0090] Adjust OBJ1 according to the preset storage rules, and the adjusted OBJ1 is as follows:

[0091]

[0092] It should be noted that OBJ1 (i1=1, s1=1) represents an object data of OBJ1, wherein the variable i1 of the int data type, i.e., i1=1, indicates that the identification of the object data is 1, and the variable s1 of the short data type, i.e., s1=1, indicates that the value of the object data is 1.

[0093] For easier understanding, see Figure 5 , Figure 5A memory structure diagram of OBJ1 is provided for an application embodiment. It can be seen that ref_md1 occupies 4B of the first-level data space, and the remaining 4B is filled with blank data. f1 and i1 respectively occupy 4B of the second-level data space, i12 occupies 2B of the third-level data space, t1 and c1 respectively occupy 1B of the third-level data space, and the total occupied memory size is 24B.

[0094] It can be seen that the original data area of ​​OBJ1 occupies 24B of memory, plus the 16B of the reference address mapping table, a total of 40B of memory space is occupied. Now the reference address mapping table has been removed, and all data is located in the data area, and the total memory size is 24B. Therefore, compared with the original data storage method, its memory size is reduced from 40B to 24B, which saves 40% of memory space.

[0095] In a possible embodiment, the data of OBJ2 includes two layers of inheritance hierarchy, plus the data area of ​​OBJ2 itself, which can constitute a complex object with three layers of inheritance hierarchy. In the first inheritance hierarchy, because there is a double type, and the object definition reference address of the object head does not use up 8B, that is, it meets rule 5, and the original rule 2 sorting rule can be broken by referring to rule 4, and "int ib1" is sorted first to the back of the object definition reference address variable "ref_md2", and then "doubledb1" is sorted. At this time, the first inheritance hierarchy has been sorted and 8B has been used up.

[0096] In the second inheritance level, there is only the variable "float fb2", which can be sorted normally according to rules 2 and 3. At this time, the second inheritance level has not used up 8B. In the third inheritance level, the data type of the first child data is long, and the second inheritance level has not used up 8B, which means that both rule 3 and rule 4 are met. Then rule 3 is prioritized for sorting, that is, "int i2" is moved to the back of "float fb2" in the second inheritance level. In this way, the second inheritance level has also used up 8B, so the sorting rule of rule 2 is restored at this time. Therefore, the next data to be sorted is "long l2", followed by "short s2", followed by "byte b2", and finally "ref_OBJ1 ref_obj1".

[0097] Among them, the original OBJ1 data structure is as follows:

[0098]

[0099] The dotted lines are used to distinguish different inheritance levels and have no practical meaning. OBJ2 includes the data of OBJ_BASE2.

[0100] Adjust OBJ2 according to the preset storage rules, and the adjusted OBJ2 is as follows:

[0101]

[0102] For easier understanding, see Figure 6 , Figure 6 A memory structure diagram of OBJ2 provided in the embodiment of the application shows that the original data area of ​​OBJ2 occupies 56B of memory, plus the 16B of the reference address mapping table, a total of 72B of memory space is occupied. Now there is no reference address mapping table, and all data is located in the data area. The total occupied memory size is based on Figure 6 It can be calculated as 40B. Therefore, compared with the original data storage method, the memory size is reduced from 72B to 40B, which saves 44.45% of the memory space.

[0103] It can be seen that all object data are compressed in advance using the data parsing and conversion machine, that is, the target application is pointed to the memory data area of ​​the object data to save the memory occupied by the object data reference address, and then the object definition reference address is added to the object data header and compressed. Then, according to the preset storage rules, the object data storage structure is optimized to solve the problem of excessive memory consumption caused by blank data filling and save memory space.

[0104] Step S104: creating a self-recovery log file in the target disk, and determining a self-recovery log file write operation mechanism according to the self-recovery log file.

[0105] The self-recovery log file write operation mechanism is used to store data processing operations to ensure that data can be recovered after a system crash or failure.

[0106] The self-recovery log file includes a write pointer and a check pointer, the write pointer is used to indicate the starting position of counterclockwise writing data, and the check pointer is used to indicate the starting position of data in the self-recovery log file. The self-recovery log file is created in the target disk, and the self-recovery log file writing operation mechanism is determined according to the self-recovery log file, specifically including:

[0107] k log files are obtained in the target disk; each log file in the k log files has the same size; k is a positive integer; the k log files are integrated according to a preset ring structure to obtain the self-recovery log file; when the self-recovery log file performs a reference disk sequential write operation, the positions of the write pointer and the check pointer are checked; when the check pointer is after the write pointer, it is determined that the data in the self-recovery log file is not full, and the reference disk sequential write operation is continued; when the check pointer coincides with the write pointer, it is determined that the data in the self-recovery log file is full, and the first position of the check pointer is obtained; the check pointer is controlled to move forward counterclockwise according to a preset moving distance to obtain the second position of the check pointer; the data between the first position and the second position is determined to be the data to be processed, and the data to be processed is processed to obtain the free reference log file space; the reference disk sequential write operation is continued according to the reference log file space.

[0108] For easier understanding, see Figure 7 , Figure 7 A structural diagram of a self-recovery log file provided for the application embodiment, it can be seen that when k is 10, 10 log files form a ring, wherein the log file can be 1GB, and the size of each log file is the same, which is not specifically limited here. Among them, the 10 log files are numbered as revylog_file0, revylog_file1, revylog_file2, revylog_file3, ..., revylog_file9 in sequence. Write_point represents a write pointer, and the file can be written in a counterclockwise order, while check_point represents a check pointer, and data can also be checked in a counterclockwise order. The log file is logically connected end to end to form a ring. When the previous file is written, continue to write the next one, and then the cycle starts from the beginning. The operation of cyclic appending can be supported in the logical implementation to obtain a self-recovery log file write operation mechanism.

[0109] Among them, revylog_file0 is connected to revylog_file9. When the write pointer lags behind the check pointer, it means that the current 10 revylog log files have not been fully written, and the content between the write pointer and the check pointer is the temporarily stored data content; when the write pointer catches up with the check pointer, it means that the 10 revylog log files have been fully written, that is, there is no content to be added at present, and it is necessary to stop and push the check pointer forward first. After processing part of the temporarily stored data and leaving enough space, continue to push the write pointer to write other data. Among them, the position of the check pointer forward can be determined according to the busyness of the system corresponding to the target application. For example, when the system is relatively busy, 10% of the data can be processed first, that is, the check pointer can move forward to the position of the next revylog log file. When the system is relatively idle, all the data can be processed at one time, that is, the check pointer can go around 10 revylog log files and return to the original position.

[0110] Step S105: obtaining a first data processing instruction associated with the reference first object data.

[0111] Specifically, the first data processing instruction includes a write operation instruction for the first object data. For example, the first object data is i=1, and its associated first data processing instructions are OBJ1 (i1=1, s1+2) and OBJ1 (i1=1, s1-1).

[0112] Step S106: Execute the first data processing instruction on the reference first object data according to the self-recovery log file write operation mechanism to obtain target first object data, so as to improve data processing efficiency.

[0113] The self-recovery log file includes a first self-recovery log file, and executing the first data processing instruction on the reference first object data according to the self-recovery log file write operation mechanism to obtain the target first object data specifically includes:

[0114] Determine a first original data page according to the reference first object data; modify the reference first object data in the first original data page according to the first data processing instruction to obtain a first data change page; the first data change page includes changed first object data; store the first data processing instruction in the first self-recovery log file by performing a first disk sequential write operation on the first self-recovery log file; when the read and write operations on the first self-recovery log file meet a preset first condition, write the changed first object data to the target disk by performing a first disk random write operation on the first data change page to obtain the target first object data.

[0115] It should be noted that the preset first condition includes but is not limited to insufficient capacity of the first self-recovery log file, which is not specifically limited here.

[0116] In a possible embodiment, the target memory includes object data corresponding to the three OBJ1s i1=1, i1=2, and i1=3, and the target disk includes object data corresponding to the four OBJ1s i1=1, i1=2, i1=3, and i1=4. When accessing the three object data i1=1, i1=2, and i1=3, the target memory can be accessed directly without loading from the target disk; while when accessing the three object data i1=4, i1=5, and i1=6, it needs to be loaded from the target disk, that is, there are three more random disk read IOs, and the first access without data also needs to be known after loading from the target disk. Then, the target instruction sequence is executed. Because the instructions corresponding to the object data of different OBJ1s are scattered, the file buffer cannot be used for optimization when there is a write operation after the instruction is executed. In fact, each instruction corresponds to a random disk write IO, so a total of 21 instructions require 21 random disk write IOs, plus the random disk read IOs of the first access loading of the three OBJ1s i1=4, i1=5, and i1=6, a total of 24 random disk read and write IOs are consumed.

[0117] Among them, the target instruction sequence is as follows:

[0118] OBJ1(i1=1,s1+2)

[0119] OBJ1(i1=3,s1+1)

[0120] OBJ1(i1=6,s1+2)

[0121] OBJ1(i1=2,s1+2)

[0122] OBJ1(i1=1,s1-1)

[0123] OBJ1(i1=2,s1+3)

[0124] OBJ1(i1=1,s1+3)

[0125] OBJ1(i1=2,s1-5)

[0126] OBJ1(i1=1,s1-1)

[0127] OBJ1(i1=3,s1+1)

[0128] OBJ1(i1=5,s1+5)

[0129] OBJ1(i1=3,s1+1)

[0130] OBJ1(i1=4,s1+10)

[0131] OBJ1(i1=3,s1*)

[0132] OBJ1(i1=5,s1+5)

[0133] OBJ1(i1=4,s1-1)

[0134] OBJ1(i1=4,s1-2)

[0135] OBJ1(i1=6,s1+3)

[0136] OBJ1(i1=4,s1-3)

[0137] OBJ1(i1=5,s1-10)

[0138] OBJ1(i1=6,s1*)

[0139] It should be noted that the target instruction sequence refers to a series of operations or instructions that need to be executed, and these instructions may usually involve the modification or operation of different object data. After the target instruction sequence is executed, the write operations generated are scattered, that is, discontinuous write data, and the write operations cannot be effectively merged into the buffer for unified processing, resulting in frequent file access operations. Therefore, the execution process of the target instruction sequence can be optimized through the subsequent second optimization mechanism to improve data processing efficiency.

[0140] The "int i1" variable can be used as a distinguishing factor for OBJ1 data. For example, the data corresponding to i1=1 and the data corresponding to i1=2 are distinguished, and the "short s1" variable is used as a data change item. S1 is initialized to 1, and OBJ1(i1=1, s1=1) can be obtained. When the OBJ1(i1=1, s1+2) instruction in the target instruction sequence is executed, the object data s1 of i1=1 is increased by 2. If the object data represented by i1=1 does not exist originally, that is, OBJ1(i1=1, s1=1) does not exist, then the operation is to initialize the object data and initialize s1 to 2. When the OBJ1(i1=1, s1-1) instruction in the target instruction sequence is executed, the object data s1 of i1=1 is reduced by 1. If the object data represented by i1=1 does not exist originally, that is, OBJ1(i1=1, s1=1) does not exist, then the operation is to initialize the object data and initialize s1 to -1. When the OBJ1(i1=3, s1*) instruction in the target instruction sequence is executed, the object data of i1=3 is deleted. If the object data represented by i1=3 does not exist originally, this operation can be ignored.

[0141] In a possible embodiment, the object data in OBJ1 includes i1=1, i1=2, and i1=3. The data areas of these three object data in the target memory are "original data pages", which represent the original data that has not been modified and matches the data in the target disk one by one. Among them, the "original data page" records OBJ1 (i1=1, s1=1), and is played back according to the instruction sequence of OBJ1 (i1=1, s1+2), OBJ1 (i1=1, s1-1), OBJ1 (i1=1, s1+3), and OBJ1 (i1=1, s1-1), that is, s1=1+2-1+3-1, and the final result is s1=4, that is, the final state of the "data change page" is recorded as (i1=1, s1=4). Therefore, it is only necessary to write the result of changing s1 from 1 to 4 to the target disk through a random write IO.

[0142] It should be noted that, because the playback results of the write operation instructions are temporarily stored in the "data change page" of the target memory, if the target application crashes or loses power before the "data change page" is written to the target disk, there may be a problem of data loss. Therefore, the data processing process can be optimized through the self-recovery log file write operation mechanism, that is, when the "original data page" of the target memory replays the 4 write operation instructions of the object data of OBJ1 i1=1, not only will the "data change page" be obtained, but the 4 write operation instructions will also be sequentially appended to the log file. Because it is a sequential append write, that is, the disk sequential write IO, and does not involve the head seek problem, then these 4 instructions can be written to the file buffer first, and then flushed to the target disk at one time, that is, only one disk sequential write IO is needed to complete it. Then, when the data parsing and conversion machine is idle or the target application is closed, the data in the "data change page" is written to the target disk through a disk random write IO. Among them, after writing, the check pointer in the first self-recovery log file will move forward a corresponding distance to free up the occupied space. Alternatively, when the read and write operations of the first self-recovery log file meet the preset first condition, that is, when the capacity of the first self-recovery log file is insufficient, a disk sequential write IO will be triggered first, and this operation is called a purge operation.

[0143] It can be seen that the new self-recovery log file writing mechanism does not write the write operation instructions to the target disk in real time through random disk write IO, but by continuously replaying the write operation instructions in the "original data page" of the target memory, thereby forming a "data change page" in the target memory, and at the same time, through the file buffer mechanism, it is optimized to 1 disk sequential IO to append this write operation instruction sequence to the first self-recovery log file, and finally under the preset conditions, that is, the target application is closed, the system is idle, and the first self-recovery log file space is insufficient, the content of the "data change page" is written to the data file of the target disk through a random disk write IO. Among them, compared with the original 4 random disk write IOs for the object data of i1=1, the new mechanism only needs 1 disk sequential write IO and 1 disk random write IO.

[0144] For easier understanding, see Figure 8 , Figure 8 A schematic diagram of a second optimization mechanism provided for an application embodiment. It can be seen that the second optimization mechanism includes a self-recovery log file write operation mechanism, wherein, in the target disk file, revylog is used to represent the self-recovery log file, which is used to sequentially write the change status of memory data and restore the memory data based on the self-recovery log file at startup to avoid data loss, and datafile is used to represent the data persistence disk file in the target disk.

[0145] In one possible embodiment, the OBJ1 object data with i1=2, its "original data page" records OBJ1(i1=2, s1=1), and is replayed according to the instruction sequence of OBJ1(i1=2, s1+2), OBJ1(i1=2, s1+3), and OBJ1(i1=2, s1-5), that is, s1=1+2+3–5, and the final result is s1=1. Among them, the final result of the "original data page" of the target memory has not changed, and there is no need to do random disk write IO. However, in the process of replaying the instruction sequence, it is not known that the final result has not changed, so the instruction sequence will still be appended to the first self-recovery log file to avoid accidental data loss. Therefore, this process only requires one disk sequential write IO. Among them, compared with the original three disk random write IOs, the new mechanism only requires one disk sequential write IO for the object data with i1=2.

[0146] In one possible embodiment, the OBJ1 object data with i1=3, its "original data page" records OBJ1(i1=3, s1=1), and is played back according to the instruction sequence of OBJ1(i1=3, s1+1), OBJ1(i1=3, s1+1), OBJ1(i1=3, s1+1), OBJ1(i1=3, s1*), that is, s1=1+1+1+1*, because the last write operation instruction is *, which means deleting this data. Therefore, only one disk sequential write IO is required to record it in the first self-recovery log file, and one disk random write IO is required to write the deletion operation to the disk data. Among them, compared with the original 4 disk random write IOs for the object data of i1=3, the new mechanism only requires 1 disk sequential write IO and 1 disk random write IO.

[0147] In a possible embodiment, the self-recovery log file further includes a second self-recovery log file, and the method further specifically includes:

[0148] Acquire a second object corresponding to the target application; store the second object on the target disk; determine a second data processing instruction associated with the second object data corresponding to the second object; the second data processing instruction includes at least one data operation; create a data processing cache page in the target memory; store the second data processing instruction in the data processing cache page, and integrate at least one data operation in the second data processing instruction to obtain a target data processing instruction; store the target data processing instruction in the second self-recovery log file by performing a second disk sequential write operation on the second self-recovery log file.

[0149] Wherein, after storing the target data processing instruction in the second self-recovery log file, the method further specifically includes:

[0150] By performing a second disk random write operation on the target disk, the second object data is loaded from the target disk to the target memory to obtain a second original data page; the second object data in the second original data page is processed according to the target data processing instruction to obtain a second data change page; the second data change page includes changed second object data; by performing a third disk sequential write operation on the second self-recovery log file, the target data processing instruction is stored in the second self-recovery log file; when the read and write operations on the second self-recovery log file meet the preset second condition, by performing a third disk random write operation on the second data change page, the changed second object data is written to the target disk to obtain the target second object data.

[0151] It should be noted that the preset second condition includes but is not limited to insufficient capacity of the second self-recovery log file, which is not specifically limited here.

[0152] In one possible embodiment, OBJ1 includes three object data: i1=4, i1=5, and i1=6, and none of the three object data are in the memory. According to the original mechanism, before the write operation instruction is executed, it is necessary to first load it into the memory through 1 disk random read IO, and then replay the write operation instruction to the "original data page" of the memory. In order to optimize this problem, a "data processing cache page" can be created in the target memory, that is, when the data analysis and conversion machine determines that the object data does not exist in the target memory, it can create a "data processing cache page", and then record the change of the write operation instruction in the "data processing cache page", and then continue to execute other write operation instructions. Among them, an independent second self-recovery log file can be created to ensure that data is not lost in abnormal situations.

[0153] Among them, the "data processing cache page" will temporarily store the write operation instructions corresponding to the object data, and will not be updated to the target disk data immediately, nor will it be updated to the "original data page" of the target memory. When other read operations need to access the object data later, the data analysis and conversion machine will load the data from the target disk to the "original data page" of the target memory, and actively merge the "data processing cache page" into the "original data page". This operation is called a merge operation, thereby forming a "data change page". With the help of the purge operation of the original "data change page", the data change content can be written to the target disk data. If the write operation instruction is written to the "data processing cache page" and the OBJ1 data is not accessed later, or the target application is closed, the data analysis and conversion machine actively loads the data from the target disk to the target memory and performs a merge operation. Therefore, the mechanism corresponding to the "data processing cache page" also supports the start of self-recovery.

[0154] In a possible embodiment, the OBJ1 object data with i1=4 is played back in the "data processing cache page" according to the instruction sequence of OBJ1(i1=4,s1+10), OBJ1(i1=4,s1-1), OBJ1(i1=4,s1-2), OBJ1(i1=4,s1-3), that is, s1=10-1-2-3, and the final result s1=4, indicating that it can be written to the target disk through 1 random write IO. Among them, compared with the original 4 disk random write IOs, the new mechanism only needs 1 disk sequential write IO and 1 disk random write IO.

[0155] In one possible embodiment, the OBJ1 object data with i1=5 is played back in the "data processing cache page" according to the instruction sequence of OBJ1(i1=5,s1+5), OBJ1(i1=5,s1+5), OBJ1(i1=5,s1-10), that is, s1=5+5-10, and the final result s1=0, indicating that there is no data change. Since the target disk does not have the object data, there is no need to perform random disk write IO. Among them, the object data of i1=5 only needs one disk sequential write IO compared to the original three random disk write IOs.

[0156] In a possible embodiment, the OBJ1 object data with i1=6 is played back in the "data processing cache page" according to the instruction sequence of OBJ1(i1=6,s1+2), OBJ1(i1=6,s1+3), OBJ1(i1=6,s1*), that is, s1=2+3*. Because the last step is to delete the data, and the target disk does not have the object data, there is no need to do random disk write IO. Among them, the object data with i1=6 only needs one disk sequential write IO compared to the original three random disk write IOs.

[0157] It can be seen that for OBJ1, the new mechanism only needs 6 disk sequential read and write IOs and 3 disk random read and write IOs, while the original mechanism needs 24 disk random read and write IOs. If a disk sequential read and write IO takes 3ms and a disk random read and write IO takes 10ms, the time comparison between the new and old mechanisms is as follows:

[0158] The total time taken by the new mechanism is: 6×3ms+3×10ms=48ms.

[0159] Total time consumed by the old mechanism: 24×10ms=240ms.

[0160] Therefore, the performance difference between the two is quite significant. The new mechanism can convert random disk read and write IO into sequential disk read and write IO, and greatly reduce the number of disk IO, thereby greatly improving the processing performance of large-scale data calculations.

[0161] Combine the following Fig. 9 The electronic device in the embodiment of the present application is described. Fig. 9 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application is shown in FIG. Fig. 9 As shown, the electronic device includes one or more processors, a memory, a communication interface and one or more programs, and the processor is communicatively connected with the memory and the communication interface via an internal communication bus.

[0162] Among them, the processor is mainly used for:

[0163] Acquire a first object corresponding to a target application; the first object is stored in a target memory; the first object includes first object data and a first object definition;

[0164] Determine a first reference address corresponding to the first object definition, and compress the first reference address according to a preset encoding function to obtain a second reference address; the second reference address is used to point to the first object definition;

[0165] Filling the first object data according to a preset storage structure and a preset storage rule to obtain reference first object data, so as to save data space of a target memory;

[0166] Creating a self-recovery log file in the target disk and determining a self-recovery log file write operation mechanism based on the self-recovery log file; the self-recovery log file write operation mechanism is used to store data processing operations to ensure that data can be recovered after a system crash or failure;

[0167] Obtaining a first data processing instruction associated with reference to the first object data;

[0168] The first data processing instruction is executed on the reference first object data according to the self-recovery log file write operation mechanism to obtain the target first object data, so as to improve the data processing efficiency.

[0169] The one or more programs are stored in the above-mentioned memory and are configured to be executed by the above-mentioned processor, and the one or more programs include instructions for executing any step in the above-mentioned method embodiment.

[0170] Among them, the processor can be, for example, a central processing unit (CPU), a general processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, transistor logic devices, hardware components or any combination thereof. It can implement or execute various exemplary logic blocks, units and circuits described in conjunction with the disclosure of this application. The processor can also be a combination that implements a computing function, such as a combination of one or more microprocessors, a combination of a DSP and a microprocessor, and the like. The communication unit can be a communication interface, a transceiver, a transceiver circuit, etc., and the storage unit can be a memory.

[0171] The memory may be a volatile memory or a nonvolatile memory, or may include both volatile and nonvolatile memories. Among them, the nonvolatile memory may be a read-only memory (ROM), a programmable ROM (PROM), an erasable programmable ROM (EPROM), an electrically erasable programmable ROM (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of random access memory (RAM) are available, such as static RAM (SRAM), dynamic random access memory (DRAM), synchronous DRAM (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link DRAM (SLDRAM), and direct RAM bus random access memory (DR RAM).

[0172] It is understandable that the electronic device may include more or fewer structural elements than those in the above structural block diagram, for example, including a power module, physical buttons, a Wi-Fi module, a speaker, a Bluetooth module, a sensor, a display module, etc., which are not limited here. It is understandable that the electronic device may be a target device.

[0173] The above mainly introduces the scheme of the embodiment of the present application from the perspective of the execution process on the method side. It is understandable that, in order to realize the above functions, the electronic device includes a hardware structure and / or software module corresponding to the execution of each function. It should be easily appreciated by those skilled in the art that, in combination with the units and algorithm steps of each example described in the embodiments provided herein, the present application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present application.

[0174] The embodiment of the present application can divide the electronic device into functional units according to the above method example. For example, each functional unit can be divided according to each function, or two or more functions can be integrated into one processing unit. The above integrated unit can be implemented in the form of hardware or in the form of software functional units. It should be noted that the division of units in the embodiment of the present application is schematic and is only a logical function division. There may be other division methods in actual implementation.

[0175] In the case of dividing each functional module into corresponding functional modules, Fig.10 A functional module composition block diagram of a data processing device for limited memory provided in an embodiment of the present application is applied to a data parsing and conversion machine, a target device includes the data parsing and conversion machine, a target application, a target memory and a target disk, and the data processing device for limited memory 10 includes an acquisition module 101, a determination module 102, a filling module 103, a creation module 104 and an execution module 105, wherein:

[0176] The acquisition module 101 is used to acquire a first object corresponding to the target application; the first object is stored in the target memory; the first object includes first object data and a first object definition;

[0177] The determining module 102 is used to determine a first reference address corresponding to the first object definition, and compress the first reference address according to a preset encoding function to obtain a second reference address; the second reference address is used to point to the first object definition;

[0178] The filling module 103 is used to fill the first object data according to a preset storage rule according to a preset storage structure to obtain reference first object data to save data space of the target memory;

[0179] The creation module 104 is used to create a self-recovery log file in the target disk and determine a self-recovery log file write operation mechanism according to the self-recovery log file; the self-recovery log file write operation mechanism is used to store data processing operations to ensure that data can be recovered after a system crash or failure;

[0180] The acquisition module 101 is further used to acquire the first data processing instruction associated with the reference first object data;

[0181] The execution module 105 is used to execute the first data processing instruction on the reference first object data according to the self-recovery log file write operation mechanism to obtain the target first object data, so as to improve data processing efficiency.

[0182] Optionally, in the aspect of compressing the first reference address according to a preset encoding function to obtain the second reference address, the determining module 102 is further specifically configured to:

[0183] Obtaining a starting offset of a memory area corresponding to the first reference address;

[0184] Subtract the first reference address from the starting offset to obtain a reference reference address;

[0185] The reference address is unsignedly shifted right by a bits to perform address compression to obtain the second reference address; a is a positive integer.

[0186] Optionally, the first object data includes a plurality of sub-data; the preset storage structure includes multiple layers of data space, and the memory size of each layer of data space is the same; in terms of filling the first object data according to a preset storage rule according to the preset storage structure to obtain the reference first object data, the filling module 103 is further specifically used to:

[0187] If the first object data has no inheritance hierarchy, sorting the plurality of sub-data according to a preset first sorting rule to obtain a first sub-data sequence;

[0188] Filling the first sub-data sequence into the multi-layer data space in order to obtain the reference first object data;

[0189] If the first object data includes multiple inheritance levels, the sub-data of each inheritance level in the multiple inheritance levels are sorted according to the first sorting rule to obtain multiple reference second sub-data sequences; each inheritance level corresponds to one reference second sub-data sequence;

[0190] Sorting the plurality of reference second sub-data sequences according to the order of inheritance hierarchy from top to bottom to obtain a target second sub-data sequence;

[0191] The target second sub-data sequence is filled into the multi-layer data space in sequence to obtain the reference first object data.

[0192] Optionally, the filling module 103 is further specifically used for:

[0193] If the first object data includes multiple inheritance levels, obtaining at least one sub-data of inheritance level i; the inheritance level i is any inheritance level among the multiple inheritance levels;

[0194] If the data type corresponding to the at least one sub-data contains a preset first type, and the data space corresponding to the inheritance level i-1 is not full, sorting the at least one sub-data according to a preset second sorting rule to obtain a reference third sub-data sequence;

[0195] Fill the reference third sub-data sequence into the data space corresponding to the inheritance level i-1 in order;

[0196] When the data space corresponding to the inheritance level i-1 cannot store data, all sub-data not stored in the data space are sorted according to the first sorting rule to obtain a reference fourth sub-data sequence;

[0197] The reference fourth sub-data sequence is filled into the multi-layer data space in sequence to obtain the reference first object data.

[0198] Optionally, the self-recovery log file includes a write pointer and a check pointer, the write pointer is used to indicate the starting position of data written counterclockwise, and the check pointer is used to indicate the starting position of data in the self-recovery log file. In terms of creating a self-recovery log file in the target disk and determining a self-recovery log file write operation mechanism according to the self-recovery log file, the creation module 104 is further specifically used to:

[0199] Obtain k log files in the target disk; each of the k log files has the same size; k is a positive integer;

[0200] Integrating the k log files according to a preset ring structure to obtain the self-recovery log file;

[0201] When the self-recovery log file performs a reference disk sequential write operation, checking the positions of the write pointer and the check pointer;

[0202] When the check pointer is behind the write pointer, it is determined that the self-recovery log file is not full of data, and the reference disk sequential write operation continues to be performed;

[0203] When the check pointer coincides with the write pointer, it is determined that the data in the self-recovery log file is full, and the first position of the check pointer is obtained;

[0204] Controlling the inspection pointer to move forward counterclockwise according to a preset moving distance to obtain a second position of the inspection pointer;

[0205] Determine that data between the first position and the second position is data to be processed, and process the data to be processed to obtain a free reference log file space;

[0206] The reference disk sequential write operation continues to be performed according to the reference log file space.

[0207] Optionally, the self-recovery log file includes a first self-recovery log file, and in terms of executing the first data processing instruction on the reference first object data according to the self-recovery log file write operation mechanism to obtain the target first object data, the execution module 105 is further specifically used to:

[0208] determining a first original data page according to the reference first object data;

[0209] Modify the reference first object data in the first original data page according to the first data processing instruction to obtain a first data change page; the first data change page includes the changed first object data;

[0210] The first data processing instruction is stored in the first self-recovery log file by performing a first disk sequential write operation on the first self-recovery log file;

[0211] When the read and write operations on the first self-recovery log file meet a preset first condition, the first random disk write operation is performed on the first data change page to write the changed first object data to the target disk to obtain the target first object data.

[0212] Optionally, the self-recovery log file further includes a second self-recovery log file, and the execution module 105 is further specifically configured to:

[0213] Acquire a second object corresponding to the target application; the second object is stored in the target disk;

[0214] Determine a second data processing instruction associated with second object data corresponding to the second object; the second data processing instruction includes at least one data operation;

[0215] Creating a data processing cache page in the target memory;

[0216] storing the second data processing instruction in the data processing cache page, and integrating at least one data operation in the second data processing instruction to obtain a target data processing instruction;

[0217] The target data processing instruction is stored in the second self-recovery log file by performing a second disk sequential write operation on the second self-recovery log file.

[0218] Optionally, after storing the target data processing instruction in the second self-recovery log file, the execution module 105 is further specifically configured to:

[0219] By performing a second disk random write operation on the target disk, the second object data is loaded from the target disk to the target memory to obtain a second original data page;

[0220] Processing the second object data in the second original data page according to the target data processing instruction to obtain a second data change page; the second data change page includes the changed second object data;

[0221] By performing a third disk sequential write operation on the second self-recovery log file, the target data processing instruction is stored in the second self-recovery log file;

[0222] When the read and write operations on the second self-recovery log file meet the preset second condition, the changed second object data is written to the target disk by performing a third disk random write operation on the second data change page to obtain the target second object data.

[0223] It can be seen that by optimizing the storage of memory data and improving the memory computing and file interaction mechanisms, the memory utilization rate can be increased and the processing performance of large-scale data computing can be improved.

[0224] It should be noted that the specific implementation of each operation can adopt the corresponding description of the method embodiment shown above. The data processing device 10 with limited memory can be used to execute the above method embodiment of the present application, which will not be repeated here.

[0225] An embodiment of the present application also provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program for electronic data exchange, wherein the computer program enables a computer to execute part or all of the steps of any method recorded in the above method embodiments, and the above computer includes an electronic device.

[0226] The embodiment of the present application also provides a computer program product, the computer program product includes a non-transitory computer-readable storage medium storing a computer program, the computer program is operable to cause a computer to execute some or all of the steps of any method described in the method embodiment. The computer program product may be a software installation package, and the computer includes an electronic device.

[0227] It should be noted that, for the above-mentioned various embodiments, for the sake of simple description, they are all expressed as a series of action combinations. Those skilled in the art should be aware that the present application is not limited by the described order of actions, because some steps in the embodiments of the present application can be performed in other orders or simultaneously. In addition, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions, steps, modules or units involved are not necessarily required by the embodiments of the present application.

[0228] In the above embodiments, the embodiments of the present application have different focuses on the description of each embodiment. For parts that are not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0229] Those skilled in the art can understand that to implement all or part of the processes in the above-mentioned embodiments, the processes can be completed by computer programs to instruct related hardware, and the programs can be stored in computer-readable storage media. When the programs are executed, they can include the processes of the above-mentioned method embodiments. The aforementioned storage media include: ROM or random access memory RAM, magnetic disk or optical disk and other media that can store program codes.

[0230] The steps of the method or algorithm described in the embodiments of the present application can be implemented in hardware or by executing software instructions by a processor. The software instructions can be composed of corresponding software modules, and the software modules can be stored in RAM, flash memory, ROM, EPROM, electrically erasable programmable read-only memory (electrically EPROM, EEPROM), registers, hard disks, mobile hard disks, read-only compact disks (CD-ROMs) or any other form of storage medium known in the art. An exemplary storage medium is coupled to the processor so that the processor can read information from the storage medium and write information to the storage medium. Of course, the storage medium can also be a component of the processor. The processor and the storage medium can be located in an ASIC. In addition, the ASIC can be located in a terminal device or a management device. Of course, the processor and the storage medium can also be present in a terminal device or a management device as discrete components.

[0231] Those skilled in the art should be aware that in one or more of the above examples, the functions described in the embodiments of the present application can be implemented in whole or in part by software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiments of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website site, computer, server, or data center to another website site, computer, server, or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (digital subscriber line, DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrations. The available medium may be a magnetic medium (eg, a floppy disk, a hard disk, a magnetic tape), an optical medium (eg, a digital video disc (DVD)), or a semiconductor medium (eg, a solid state disk (SSD)).

[0232] The modules / units included in the devices and products described in the above embodiments may be software modules / units or hardware modules / units, or may be partially software modules / units and partially hardware modules / units. For example, for the devices and products applied to or integrated in the chip, the modules / units included therein may all be implemented in the form of hardware such as circuits, or at least some of the modules / units may be implemented in the form of software programs, which run on the processor integrated inside the chip, and the remaining (if any) modules / units may be implemented in the form of hardware such as circuits; for the devices and products applied to or integrated in the chip module, the modules / units included therein may all be implemented in the form of hardware such as circuits, and different modules / units may be located in the same component (such as a chip, circuit module, etc.) or in different components of the chip module, or at least some of the modules / units may be implemented in the form of software programs. The software programs run on the processor integrated inside the chip, and the remaining (if any) modules / units may be implemented in the form of hardware such as circuits. It is implemented in the form of a software program, which runs on a processor integrated inside the chip module, and the remaining (if any) modules / units can be implemented in hardware such as circuits; for various devices and products applied to or integrated in the terminal equipment, the various modules / units contained therein can be implemented in hardware such as circuits, and different modules / units can be located in the same component (for example, chip, circuit module, etc.) or in different components in the terminal equipment, or, at least some modules / units can be implemented in the form of a software program, which runs on a processor integrated inside the terminal equipment, and the remaining (if any) modules / units can be implemented in hardware such as circuits.

[0233] The specific implementation methods described above further illustrate the purpose, technical solutions and beneficial effects of the embodiments of the present application. It should be understood that the above description is only the specific implementation method of the embodiments of the present application and is not intended to limit the protection scope of the embodiments of the present application. Any modifications, equivalent substitutions, improvements, etc. made on the basis of the technical solutions of the embodiments of the present application should be included in the protection scope of the embodiments of the present application.

Claims

1. A method for processing data with limited memory, characterized in that: Applied to a data analysis and conversion machine, the target device includes the data analysis and conversion machine, a target application, a target memory and a target disk, including: Acquire a first object corresponding to the target application; the first object is stored in the target memory; the first object includes first object data and a first object definition; Determine a first reference address corresponding to the first object definition, and compress the first reference address according to a preset encoding function to obtain a second reference address; the second reference address is used to point to the first object definition; Filling the first object data according to a preset storage structure and a preset storage rule to obtain reference first object data, so as to save data space of the target memory; Creating a self-recovery log file in the target disk, and determining a self-recovery log file write operation mechanism based on the self-recovery log file; the self-recovery log file write operation mechanism is used to store data processing operations to ensure that data can be recovered after a system crash or failure; Obtaining a first data processing instruction associated with the reference first object data; The first data processing instruction is executed on the reference first object data according to the self-recovery log file write operation mechanism to obtain target first object data, so as to improve data processing efficiency.

2. The method according to claim 1, characterized in that The compressing the first reference address according to a preset encoding function to obtain a second reference address includes: Obtaining a starting offset of a memory area corresponding to the first reference address; Subtract the first reference address from the starting offset to obtain a reference reference address; The reference address is unsignedly shifted right by a bits to perform address compression to obtain the second reference address; a is a positive integer.

3. The method according to claim 1, characterized in that The first object data includes a plurality of sub-data; the preset storage structure includes multiple layers of data space, and the memory size of each layer of data space is the same; the first object data is filled according to the preset storage rule according to the preset storage structure to obtain the reference first object data, including: If the first object data has no inheritance hierarchy, sorting the plurality of sub-data according to a preset first sorting rule to obtain a first sub-data sequence; Filling the first sub-data sequence into the multi-layer data space in order to obtain the reference first object data; If the first object data includes multiple inheritance levels, the sub-data of each inheritance level in the multiple inheritance levels are sorted according to the first sorting rule to obtain multiple reference second sub-data sequences; each inheritance level corresponds to one reference second sub-data sequence; Sorting the plurality of reference second sub-data sequences according to the order of inheritance hierarchy from top to bottom to obtain a target second sub-data sequence; The target second sub-data sequence is filled into the multi-layer data space in sequence to obtain the reference first object data.

4. The method according to claim 3, characterized in that The method further comprises: If the first object data includes multiple inheritance levels, obtaining at least one sub-data of inheritance level i; the inheritance level i is any inheritance level among the multiple inheritance levels; If the data type corresponding to the at least one sub-data contains a preset first type, and the data space corresponding to the inheritance level i-1 is not full, sorting the at least one sub-data according to a preset second sorting rule to obtain a reference third sub-data sequence; Fill the reference third sub-data sequence into the data space corresponding to the inheritance level i-1 in order; When the data space corresponding to the inheritance level i-1 cannot store data, all sub-data not stored in the data space are sorted according to the first sorting rule to obtain a reference fourth sub-data sequence; The reference fourth sub-data sequence is filled into the multi-layer data space in sequence to obtain the reference first object data.

5. The method according to claim 1, characterized in that The self-recovery log file includes a write pointer and a check pointer, the write pointer is used to indicate the starting position of counterclockwise writing data, and the check pointer is used to indicate the starting position of data in the self-recovery log file. The self-recovery log file is created in the target disk, and the self-recovery log file writing operation mechanism is determined according to the self-recovery log file, including: Obtain k log files in the target disk; each of the k log files has the same size; k is a positive integer; Integrating the k log files according to a preset ring structure to obtain the self-recovery log file; When the self-recovery log file performs a reference disk sequential write operation, checking the positions of the write pointer and the check pointer; When the check pointer is behind the write pointer, it is determined that the self-recovery log file is not full of data, and the reference disk sequential write operation continues to be performed; When the check pointer coincides with the write pointer, it is determined that the data in the self-recovery log file is full, and the first position of the check pointer is obtained; Controlling the inspection pointer to move forward counterclockwise according to a preset moving distance to obtain a second position of the inspection pointer; Determine that data between the first position and the second position is data to be processed, and process the data to be processed to obtain a free reference log file space; The reference disk sequential write operation continues to be performed according to the reference log file space.

6. The method according to claim 5, characterized in that The self-recovery log file includes a first self-recovery log file, and the step of executing the first data processing instruction on the reference first object data according to the self-recovery log file write operation mechanism to obtain the target first object data includes: determining a first original data page according to the reference first object data; Modify the reference first object data in the first original data page according to the first data processing instruction to obtain a first data change page; the first data change page includes the changed first object data; The first data processing instruction is stored in the first self-recovery log file by performing a first disk sequential write operation on the first self-recovery log file; When the read and write operations on the first self-recovery log file meet a preset first condition, the first disk random write operation is performed on the first data change page to write the changed first object data into the target disk to obtain the target first object data.

7. The method according to any one of claims 1 to 6, characterized in that: The self-recovery log file also includes a second self-recovery log file, and the method further includes: Acquire a second object corresponding to the target application; the second object is stored in the target disk; Determine a second data processing instruction associated with second object data corresponding to the second object; the second data processing instruction includes at least one data operation; Creating a data processing cache page in the target memory; storing the second data processing instruction in the data processing cache page, and integrating at least one data operation in the second data processing instruction to obtain a target data processing instruction; The target data processing instruction is stored in the second self-recovery log file by performing a second disk sequential write operation on the second self-recovery log file.

8. The method according to claim 7, characterized in that After storing the target data processing instruction in the second self-recovery log file, the method further includes: By performing a second disk random write operation on the target disk, the second object data is loaded from the target disk to the target memory to obtain a second original data page; Processing the second object data in the second original data page according to the target data processing instruction to obtain a second data change page; the second data change page includes the changed second object data; By performing a third disk sequential write operation on the second self-recovery log file, the target data processing instruction is stored in the second self-recovery log file; When the read and write operations on the second self-recovery log file meet the preset second condition, the changed second object data is written to the target disk by performing a third disk random write operation on the second data change page to obtain the target second object data.

9. A data processing device for memory-constrained applications, characterized in that: Applied to a data analysis and conversion machine, the target device includes the data analysis and conversion machine, a target application, a target memory and a target disk, and the device includes an acquisition module, a determination module, a filling module, a creation module and an execution module, wherein: The acquisition module is used to acquire a first object corresponding to the target application; the first object is stored in the target memory; the first object includes first object data and a first object definition; The determining module is used to determine a first reference address corresponding to the first object definition, and compress the first reference address according to a preset encoding function to obtain a second reference address; the second reference address is used to point to the first object definition; The filling module is used to fill the first object data according to a preset storage rule according to a preset storage structure to obtain reference first object data, so as to save data space of the target memory; The creation module is used to create a self-recovery log file in the target disk and determine a self-recovery log file write operation mechanism based on the self-recovery log file; the self-recovery log file write operation mechanism is used to store data processing operations to ensure that data can be recovered after a system crash or failure; The acquisition module is further used to acquire the first data processing instruction associated with the reference first object data; The execution module is used to execute the first data processing instruction on the reference first object data according to the self-recovery log file write operation mechanism to obtain the target first object data, so as to improve data processing efficiency.

10. An electronic device, characterized in that: include: A processor, a memory, a communication interface, and one or more programs; The one or more programs are stored in the memory and configured to be executed by the processor, the programs comprising instructions for executing the steps in the method according to any one of claims 1 to 8.

Images