Data processing method and device, electronic equipment and storage medium

Abstract

The present disclosure provides a data processing method and device, electronic equipment and storage medium, relates to the technical field of data processing, in particular to the field of memory compression, and can be applied to the intelligent recommendation scene. The specific implementation scheme is: for the initial data loaded into the target container, a target type is determined from a plurality of candidate data types according to the data value of the initial data; wherein the target container includes a basic storage area for storing a mark and an extended storage area for storing the initial data, and the mark represents the address of the initial data in the extended storage area; the converted data is obtained by converting the initial data from an initial type to the target type; the data length of the converted data is less than the data length of the initial data; in response to detecting that the capacity difference between the first demand storage area determined according to the target type and the basic storage area satisfies a first predetermined condition, the converted data is stored in the available storage area other than the basic storage area.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of data processing, in particular to the field of memory compression, and can be applied to intelligent recommendation scenarios. More specifically, the present disclosure provides a data processing method and device, an electronic device, a storage medium, and a computer program product. BACKGROUND

[0002] In the process of software development, some general container libraries can be used to manage memory and data. For example, the general container library can be STL (Standard Template Library, a C++ standard template library). The container library provides data structures and algorithms, which can simplify programming work. However, in some scenarios, the use of general container libraries has high memory overhead. SUMMARY

[0003] The present disclosure provides a data processing method and device, an electronic device, a storage medium, and a computer program product.

[0004] According to an aspect of the present disclosure, a data processing method is provided, including: determining, for initial data loaded into a target container, a target type from a plurality of candidate data types according to a data value of the initial data; wherein the target container includes a basic storage area for storing a mark and an extended storage area for storing the initial data, the mark representing an address of the initial data in the extended storage area; obtaining converted data by converting the initial data from an initial type to the target type; the data length of the converted data being less than the data length of the initial data; and storing the converted data to an available storage area other than the basic storage area in response to detecting that a capacity difference between a first required storage area determined according to the target type and the basic storage area satisfies a first predetermined condition.

[0005] According to another aspect of the present disclosure, a data processing device is provided, including: a type determining module, a conversion module, and a first storage module. The type determining module is configured to determine, for initial data loaded into a target container, a target type from a plurality of candidate data types according to a data value of the initial data; wherein the target container includes a basic storage area for storing a mark and an extended storage area for storing the initial data, the mark representing an address of the initial data in the extended storage area. The conversion module is configured to obtain converted data by converting the initial data from an initial type to the target type; the data length of the converted data being less than the data length of the initial data. The first storage module is configured to store the converted data to an available storage area other than the basic storage area in response to detecting that a capacity difference between a first required storage area determined according to the target type and the basic storage area satisfies a first predetermined condition.

[0006] According to another aspect of the present disclosure, an electronic device is provided, comprising: at least one processor; and a memory connected with the at least one processor in communication; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method provided by the present disclosure.

[0007] According to another aspect of the present disclosure, a non-transitory computer readable storage medium storing computer instructions is provided, wherein the computer instructions are used to make a computer perform the method provided by the present disclosure.

[0008] According to another aspect of the present disclosure, a computer program product is provided, comprising a computer program which, when executed by a processor, implements the method provided by the present disclosure.

[0009] It should be understood that the content described in this part is not intended to identify key or important features of the embodiments of the present disclosure, nor to limit the scope of the present disclosure. Other features of the present disclosure will become apparent from the following description. BRIEF DESCRIPTION OF DRAWINGS

[0010] The accompanying drawings are used to better understand the present scheme, and do not limit the present disclosure. Among them:

[0011] Figure 1 is a schematic flow chart of a data processing method according to an embodiment of the present disclosure;

[0012] Figure 2A is a schematic diagram of a basic storage area according to an embodiment of the present disclosure;

[0013] Figure 2B is a schematic diagram of a basic storage area satisfying a first predetermined condition according to an embodiment of the present disclosure;

[0014] Figure 2C is a schematic diagram of a basic storage area not satisfying a first predetermined condition according to an embodiment of the present disclosure;

[0015] Figure 3 is a schematic principle diagram of a data processing method when a first predetermined condition is satisfied according to an embodiment of the present disclosure;

[0016] Figure 4 is a schematic principle diagram of a data processing method when a first predetermined condition is not satisfied according to an embodiment of the present disclosure;

[0017] Figure 5 is a schematic flow chart of a method of constructing a target container according to an embodiment of the present disclosure;

[0018] Figure 6A is a schematic principle diagram of a method of constructing a target container when a second predetermined condition is satisfied according to an embodiment of the present disclosure;

[0019] Figure 6B yes Figure 6A Schematic diagram of the basic storage area in;

[0020] Figure 7 is a schematic diagram of constructing a target container when the second predetermined condition is not met according to an embodiment of the present disclosure;

[0021] Figure 8 is a schematic structural block diagram of a data processing device according to an embodiment of the present disclosure; and

[0022] Figure 9 It is a structural block diagram of an electronic device used to implement the data processing method of an embodiment of the present disclosure. DETAILED DESCRIPTION

[0023] The following description of exemplary embodiments of the present disclosure is made in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding. These details should be considered as merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0024] In the technical solutions disclosed herein, the collection, storage, use, processing, transmission, provision and disclosure of user personal information involved comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0025] In the technical solution disclosed herein, the user's authorization or consent is obtained before obtaining or collecting the user's personal information.

[0026] In some application scenarios, some data is updated infrequently yet frequently accessed. This data requires a certain amount of memory to store, which in practice results in high memory overhead. This not only increases system resource consumption but can also impact overall system performance, such as response time and throughput. Therefore, it's necessary to reduce the memory required to store data, especially for data that is updated infrequently but frequently accessed.

[0027] Embodiments of the present disclosure aim to provide a data processing method which can achieve the effect of saving memory. The method can be applied to online service scenarios that need to hold a large amount of data for a long time, such as positive index cache objects, dictionary objects, and medium and long-term resource dimension cache objects. For positive index cache objects in online services, applying the technical solutions provided by the present disclosure can optimize memory usage, improve cache hit rate, and reduce memory waste. For dictionary objects, applying the technical solutions provided by the present disclosure can reduce memory occupation, improve query efficiency, and reduce system load. For resource dimension cache objects, applying the technical solutions provided by the present disclosure can optimize memory management, improve data access speed, and improve user experience.

[0028] The technical solutions provided by the present disclosure will be described in detail below with reference to the accompanying drawings and specific embodiments.

[0029] Figure 1 is a schematic flowchart of a data processing method according to an embodiment of the present disclosure.

[0030] As shown in Figure 1 , the data processing method 100 can include operations S110-S130.

[0031] At operation S110, for initial data loaded into a target container, a target type is determined from a plurality of candidate data types according to a data value of the initial data; wherein the target container includes a basic storage area for storing a mark and an extended storage area for storing the initial data, and the mark represents an address of the initial data in the extended storage area.

[0032] For example, the target container includes a basic storage area and an extended storage area, wherein the basic storage area is determined based on a rule when the container is created and generally will not be modified after the container is created, and the basic storage area is used to provide space occupied by basic members of the container. The extended storage area is dynamically adjusted according to capacity requirements after the container is created, for example, if the capacity of the container is insufficient, the capacity size of the extended storage area can be increased, and if the capacity of the container is sufficient, the capacity size of the extended storage area can be reduced. The mark is stored in the basic storage area, and the mark can be a pointer. At the same time, the data loaded into the target container is stored in the extended storage area, and the mark points to the address of the data, so the data can be indexed through the mark.

[0033] For example, the data type includes integer type, floating point type, etc., and the integer type can include int8, int16, int32, etc.

[0034] For example, the initial type of the initial data is int32, but the data value of the initial data is small, causing the high bits of the int32 to be all 0, and int16 can be used to normally represent the initial data without affecting the data precision of the initial data, and thus int16 can be used as the target type. For example, the initial type of the initial data is a floating point type, but the decimal part of the initial data is all 0, and thus an integer type can be used to normally represent the initial data without affecting the data precision of the initial data, and thus the integer type can be used as the target type.

[0035] In operation S120, the converted data is obtained by converting the initial data from the initial type to the target type; the data length of the converted data is less than the data length of the initial data.

[0036] For example, the initial data of the int32 type is converted into converted data of the int16 type, and the data length is changed from 4 bytes to 2 bytes.

[0037] In operation S130, in response to detecting that the capacity difference between the first required storage area determined according to the target type and the basic storage area satisfies a first predetermined condition, the converted data is stored in the available storage area outside the basic storage area.

[0038] For example, the data length required by each data type is known, and thus the storage area capacity required for storing the processed data can be calculated according to the target type, and the storage area capacity is the capacity of the first required storage area. For another example, the initial data can be converted into converted data based on the target type, and then the converted data is read to determine the first required storage area.

[0039] For example, the available storage area outside the basic storage area can include an extension storage area already possessed by the target container, and the converted data can be stored in the extension storage area. Another extension storage area can also be allocated, and the available storage area outside the basic storage area can include the newly allocated extension storage area, and the converted data can be stored in the newly allocated extension storage area.

[0040] The technical scheme provided in the embodiment first converts the initial data from an initial type to a target type, so that the data length of the converted data is less than the data length of the initial data. In this way, the initial data is updated by using the converted data, the required storage space can be reduced, the memory saving effect is achieved by reconstructing the data type, the memory usage of the target container is reduced, the online service memory pressure is relieved, and the system performance and stability are improved. In addition, in the embodiment, the target container includes a basic storage area and an extended storage area, the basic storage area stores a mark, the mark points to an address of the initial data in the extended storage area, when a capacity difference between the first demand storage area and the basic storage area meets a first predetermined condition, the converted data is stored in an available storage area outside the basic storage area, and the mark can also be updated by using the storage address of the converted data in the available storage area, so that the processed data can be indexed by the mark, and after the initial data is updated, the processed data can be normally accessed.

[0041] It should be noted that in some application scenarios, some data has a very low update frequency and needs to be frequently used. For such data, the technical scheme provided in the embodiment can be used to update the initial data to the converted data, and the data compression effect can be achieved to save the storage area. It can be understood that if the target container dynamically adjusts the size according to the actual data, memory fragmentation and performance degradation will be caused. Therefore, after the data type conversion is performed, the size of the target container can be fixed, that is, the processed data loaded into the target container remains unchanged, and the frequent delete and insert operations of the target container in the running process are limited, so that the memory fragmentation is reduced by pre-building the target container and fixing the size.

[0042] Figure 2A FIG. 1 is a schematic diagram of a basic storage area according to an embodiment of the present disclosure.

[0043] As shown in Figure 2A , in the embodiment, the basic storage area includes a high-bit basic storage area and a low-bit basic storage area. Taking a 64-bit machine as an example, a mark will apply for 8 bytes of memory, for example, the high 16 bits are the high-bit basic storage area, and the low 48 bits are the low-bit basic storage area, and generally only the low-bit basic storage area is used to store the mark, and the high-bit basic storage area is not used. For example Figure 2A , the basic storage area includes high-bit basic storage areas Byte1 and Byte2 and low-bit basic storage areas Byte3-Byte8.

[0044] Figure 2B FIG. 2 is a schematic diagram of a basic storage area meeting a first predetermined condition according to an embodiment of the present disclosure.

[0045] As shown in Figure 2BAs shown, in this embodiment, if the capacity difference between the first demand storage area and the basic storage area meets the first predetermined condition, the flag can be stored in the lower basic storage area, and the data attributes of the converted data can be stored in the higher basic storage area. This allows the higher basic storage area to be utilized, avoiding idleness, and storing more information without incurring additional storage overhead. Furthermore, once the attribute information is stored in the higher basic storage area, no additional memory is required to store the data attributes, thereby saving storage space.

[0046] Figure 2C 2 is a schematic diagram of a basic storage area that does not meet the first predetermined condition according to an embodiment of the present disclosure.

[0047] like Figure 2C As shown, in this embodiment, if the capacity difference between the first demand storage area and the basic storage area does not meet the first predetermined condition, the processed data can be stored in the basic storage area, for example, in the low-order basic storage area. The data attributes of the converted data can be stored in the high-order basic storage area.

[0048] In one example, the data attributes include at least one of the following: target type, compression status, and data length of the converted data, and the compression status may indicate whether compression is performed. Figure 2B and Figure 2C As shown, a basic storage area occupies 8 bytes. The first byte, Byte1, can be used to store the data type. The data type of the converted data is the target type, which may include, for example, whether it is a signed integer. The first byte, Byte1, can also be used to store compression, such as whether floating-point compression is used. The second high-order byte, Byte2, stores the length of the converted data. Furthermore, the lower six bytes are used to store flags or processed data. The data type, length of the converted data, and flags allow accurate reading of processed data from the extended storage area. Figure 2B and Figure 2C This is illustrated by taking the target type as an integer and the data length as 1 byte as an example.

[0049] Figure 3 is a schematic diagram of a data processing method when a first predetermined condition is met according to an embodiment of the present disclosure, Figure 4 It is a schematic diagram of a data processing method when the first predetermined condition is not met according to an embodiment of the present disclosure.

[0050] In this embodiment, for the initial data loaded into the target container, the target type is determined from multiple candidate data types based on the data value of the initial data. For example, the initial type of the initial data is int32, and the target type is int16.

[0051] Next, whether the target container satisfies the first predetermined condition can be determined according to the first demand storage area and the base storage area. For example, when the data attribute of the processed data does not need to use the base storage area for storage, the first predetermined condition can include that the capacity of the first demand storage area is greater than the capacity of the base storage area. In this way, it can be determined whether the base storage area can accommodate the processed data. For another example, when the data attribute of the processed data uses the base storage area for storage, the first predetermined condition can include that the capacity of the first demand storage area is greater than the capacity of the lower base storage area. In this way, on the basis of reserving the storage area of the data attribute, it can be ensured that the lower base storage area can accommodate the processed data.

[0052] As shown in Figure 3 , if the first predetermined condition is satisfied, it means that the base storage area cannot accommodate the processed data or that the lower base storage area cannot accommodate the processed data. At this time, the initial data can be updated with the processed data, and the flag can be updated. The specific updating manner can be referred to the above, and the embodiment will not be described again. For example, the converted data of int16 is stored in the extended storage area, and the initial data of int32 is deleted. At this time, the storage area used by the target container includes the base storage area and the storage area occupied by the converted data of int16. The difference between the storage area occupied by the initial data of int32 and the storage area occupied by the converted data of int16 is the saved storage area.

[0053] As shown in Figure 4 , if the first predetermined condition is not satisfied, it means that the base storage area can accommodate the processed data or that the lower base storage area can accommodate the processed data. At this time, the flag can be deleted, and the converted data can be stored in the base storage area. The initial data in the extended storage area can also be deleted to release the extended storage area. In the embodiment, when the data amount of the converted data is small, the base storage area can accommodate the part of the converted data. Therefore, the flag can not be used any more, and the base storage area of the target container can be directly used to store the processed data. In this way, the processed data can reuse the storage area occupied by the flag, and does not need to use the extended storage area other than the base storage area. Therefore, the extended storage area can be released, and the memory overhead can be further reduced, and the use efficiency of the memory can be improved.

[0054] For example, as shown in Figure 4 , the converted data of int16 is stored in the base storage area, and the initial data of int32 is deleted. At this time, the storage area used by the target container includes the base storage area, and does not need to use the extended storage area. The saved storage area is the extended storage area.

[0055] It can be seen that the present disclosure adopts different strategies according to whether the target container meets the first predetermined condition. When the first predetermined condition is not met, the mark is deleted, and the converted data is stored by occupying the storage area of the mark. When the first predetermined condition is met, the target type can be determined according to the data value of the initial data, and the initial data is converted to the data type, and then the converted data is used to update the initial data to achieve the effect of memory compression. In addition, since different strategies are adopted according to whether the target container meets the first predetermined condition, targeted processing can be performed according to the needs of different scenarios, thereby improving the flexibility and efficiency of memory usage.

[0056] According to another embodiment of the present disclosure, the number of initial data is multiple, and the process of determining the target type from the multiple candidate data types according to the data value of the initial data can include: determining the target type according to the maximum data value and the minimum data value in the multiple initial data.

[0057] In this embodiment, the memory can be compressed after the target container loads all the initial data. In the compression process, the maximum data value and the minimum data value are found by traversing the initial data, and then the data type with the minimum data length that can be used is determined according to the range of the maximum data value and the minimum data value, which is the target type, and then the initial data is converted into converted data of the target type. In this way, the storage space of each initial data can be reduced, thereby reducing the total memory occupancy.

[0058] According to another embodiment of the present disclosure, the process of storing the converted data to the available storage area outside the base storage area can include: determining a target storage area in the available storage area according to the capacity size of the first demand storage area. Then the converted data is stored in the target storage area, and the mark is updated according to the storage address of the converted data in the target storage area, and the initial data is also deleted from the extended storage area.

[0059] For example, the initial data in the target container can be traversed, and the capacity of the storage area required for storing the processed data is counted according to the target type and the number of initial data. The capacity of the storage area is the capacity of the first demand storage area. Then, the target storage area can be reallocated according to the size of the first demand storage area. The size of the target storage area matches the size of the first demand storage area, which means that the size of the target storage area is equal to or slightly larger than the size of the first demand storage area. After obtaining the target storage area, the converted data can be stored in the target storage area, and then the initial data can be deleted from the extended storage area, thereby releasing the extended storage area and recycling the useless storage resources. In this embodiment, the size of the target storage area is accurately allocated according to the demand, so that the processed data is completely stored, and unnecessary memory occupation is effectively reduced. In addition, in the process of allocating the target storage area, the size of the first demand storage area is determined in advance, so that the target area can be allocated at one time, without gradually expanding the area in multiple times to obtain the target area. In this way, the allocation efficiency of the storage area can be improved, the application and release of the storage area are avoided, the performance overhead is reduced, and the memory fragmentation is reduced.

[0060] According to another embodiment of the present disclosure, the process of determining the first demand storage area according to the target type can include determining the first demand storage area according to the target type and the number of initial data. For example, for the same target container, the number of initial data loaded into the target container is multiple, and the target type is int16. After the data type conversion of the initial data, each processed data requires a storage area of 2 bytes. The total bytes required by all processed data are counted to determine the first demand information. In this embodiment, the target type and the number of initial data are used to accurately calculate the first demand storage area, so that the size of the target storage area can be accurately allocated according to the demand in the process of allocating the target storage area in the available storage area.

[0061] Figure 5 is a schematic flowchart of a method for constructing a target container according to an embodiment of the present disclosure.

[0062] The above describes the processing process of the target container. It should be noted that the target container is obtained based on the initial container. Next, the process of determining the target container using the initial container is described.

[0063] In this embodiment, the method for determining the target container further includes operations S501 to S504.

[0064] In operation S501, the data to be stored is obtained.

[0065] At operation S502, it is determined whether the initial container meets a second predetermined condition according to the second required storage area of the to-be-stored data and the free space in the base storage area of the initial container.

[0066] For example, a plurality of data needs to be loaded in the initial container, and the to-be-stored data is stored in the base storage area or the extended storage area of the initial container, that is, the loading process is completed. The number of to-be-stored data to be loaded is at least one, and the to-be-stored data can be obtained one by one or in batches.

[0067] After obtaining the to-be-stored data, it is determined whether the initial container meets the second predetermined condition according to the second required storage area of the to-be-stored data and the free space in the initial container. For example, the second predetermined condition indicates whether the free space in the base storage area of the initial container can store the obtained at least one to-be-stored data, if not, it can be determined that the second predetermined condition is met, if yes, it can be determined that the second predetermined condition is met.

[0068] At operation S503, if the initial container does not meet the second predetermined condition, the to-be-stored data is stored in the free space in the base storage area.

[0069] It should be noted that in actual application, a single target container can be used to load data for a certain specific purpose. For example, there are three kinds of data for online use, and the number of data for some purposes is small, while the number of data for some purposes is large, and the number of data loaded by the target container cannot be known in advance. In this embodiment, in the process of constructing the target container based on the initial container, the construction process of the target container is divided into two modes based on whether the second predetermined condition is met, and the free space of the base storage area in the initial container is preferentially used to store the obtained to-be-stored data. When the free space in the initial container cannot accommodate the to-be-stored data, the extended storage area is used to store the to-be-stored data. In this way, in online application, when the number of data for a certain purpose is small, the above operation S503 is used to directly store the data in the base storage area, without using the extended storage area. Therefore, subsequent data type conversion of the to-be-stored data can be avoided, and the data processing efficiency can be further improved.

[0070] At operation S504, if the initial container meets the second predetermined condition, a predetermined operation is performed on the initial container to obtain a target container. The predetermined operation includes: allocating a new storage area to the initial container as an extended storage area according to the second required storage area. The to-be-stored data and the data already stored in the base storage area are stored in the extended storage area as initial data. A mark is also constructed in the base storage area.

[0071] For example, according to the initial type and number of the acquired data to be stored, the size of the storage area required for storing the data to be stored is calculated, and the size of the second required storage area is calculated. The extended storage area can be allocated according to the size of the second required storage area.

[0072] For example, the reserve method can be called or the extended storage area of a predetermined size can be applied by other means, and then the data to be stored is stored as initial data in the extended storage area, and the data stored in the basic storage area can also be moved to the extended storage area as initial data. Next, it can be determined whether the allocated extended storage area is sufficient. If the allocated extended storage area is smaller than the second required storage area, it cannot be stored in the extended storage area. The operation of applying for an extended storage area of a predetermined size can be returned, and the operation is repeated until all initial data is stored in the extended storage area. In addition, a mark is constructed in the basic storage area of the initial container, and the mark points to the address of the initial data in the extended storage area.

[0073] It can be seen that the above operation S504 uses the extended storage area to store the data to be stored when the idle basic storage area of the initial container cannot accommodate the data to be stored. Thus, in the case of insufficient basic storage area space, the data to be stored is normally stored, and the data to be stored is successfully loaded in the initial container.

[0074] Figure 6A is a schematic diagram of a method for constructing a target container according to an embodiment of the present disclosure when the second predetermined condition is met, Figure 6B is a schematic diagram of a method for constructing a target container according to an embodiment of the present disclosure when the second predetermined condition is not met. Figure 6A is a schematic diagram of a method for constructing a target container according to an embodiment of the present disclosure when the second predetermined condition is met, Figure 7 is a schematic diagram of a method for constructing a target container according to an embodiment of the present disclosure when the second predetermined condition is not met.

[0075] For example, the number of data to be stored is multiple, and the multiple data to be stored is loaded into the initial container in the form of a data stream. During the loading of the first five data to be stored, the idle basic storage area of the initial container is relatively large, and thus the first five data to be stored is directly stored in the idle basic storage area.

[0076] As shown in Figure 6, the sixth data to be stored, Data_6, is obtained. At this time, the basic storage area of ​​the initial container can still accommodate the sixth data to be stored, Data_6, and the sixth data to be stored, Data_6, is directly stored in the basic storage area. In addition, the basic storage area can include a high-order basic storage area and a low-order basic storage area. The attributes of the data to be stored, Data_1 to Data_6, can be stored in the high-order basic storage area, and then the data, Data_1 to Data_6, can be stored in the remaining low-order storage area. In addition, it should be noted that the attributes of each data in the basic storage area can be the same, so the attributes can be stored only once, without having to store the attributes of all the data separately, thereby further saving memory.

[0077] like Figure 7 As shown, the seventh data to be stored, Data_7, is obtained. At this time, the idle basic storage area cannot accommodate the seventh data to be stored, Data_7. Therefore, the first six data, Data_1 to Data_6, are taken out from the basic storage area and moved to the extended storage area. The seventh data to be stored, Data_7, is also stored in the extended storage area, and a mark is constructed in the basic storage area to represent the position of the seven data to be stored in the extended storage area. In addition, the basic storage area can include a high-order basic storage area and a low-order basic storage area. The attributes of the data Data_1 to Data_7 can be stored in the high-order storage area. The attributes of Data_1 to Data_7 can be the same, so in order to save storage space, they can be stored only once. The mark can be stored in the low-order storage area. The mark can be a pointer, and the pointer can point to the address of the first data Data_1 in the extended storage area.

[0078] In some embodiments, when loading data to be stored into the initial container, for example, storing the data to be stored in free space in the basic storage area or in the extended storage area, an appropriate data type can be selected based on the data value of the data to be stored, and the data to be stored can be stored according to the selected data type. For example, if the data value to be processed is an integer, the integer type can be selected for storage, thereby reducing memory usage and ensuring data accuracy.

[0079] In one example, the basic storage area includes a high-order basic storage area and a low-order basic storage area. Taking a 64-bit machine as an example, a tag will request 8 bytes of memory, of which the high 16 bits can be the high-order basic storage area and the low 48 bits can be the low-order basic storage area.

[0080] For a case that the initial container satisfies the second predetermined condition, a tag can be constructed in the low-level base storage area, and the high-level base storage area is configured to be in an idle state. In this way, when the data compression process is subsequently performed, if the storage area required by the processed data is not greater than the size of the low-level base storage area, the data attribute of the processed data can be directly written in the high-level base storage area, so that the high-level base storage area can be utilized, avoiding being idle, and further avoiding using other memories to store the data attribute, thereby achieving the effect of saving the storage area. In addition, if the high-level base storage area has stored data in advance, the high-level base storage area needs to be cleared before being used. When the high-level base storage area is in an idle state, the data attribute can be directly written in the high-level base storage area when the high-level base storage area is needed, thereby improving the data storage efficiency.

[0081] For a case that the initial container does not satisfy the second predetermined condition, the to-be-stored data can be stored in the low-level base storage area, and the attribute of the to-be-stored data can also be stored in the high-level base storage area. In this way, the high-level base storage area can be utilized, avoiding being idle, and more information can be stored without increasing additional storage overhead. After the attribute information is stored in the high-level base storage area, other memories are no longer needed to store the data attribute, thereby achieving the effect of saving the storage area.

[0082] For example, the second predetermined condition can include that the capacity of the second required storage area is greater than the capacity of the idle space. It should be noted that the base storage area includes the high-level base storage area and the low-level base storage area, and the idle space can include two parts, one part is located in the high-level base storage area, and the other part is located in the low-level base storage area. When the base storage area is not needed to store the data attribute of the processed data, the second predetermined condition can include that the capacity of the second required storage area is greater than the sum of the capacities of the two parts of the idle space. For another example, when the base storage area is used to store the data attribute of the processed data, the second predetermined condition can include that the capacity of the second required storage area is greater than the capacity of the idle space in the low-level base storage area. In this way, on the basis of reserving the storage area of the data attribute, it can be guaranteed whether the low-level base storage area can accommodate the processed data.

[0083] Figure 8 is a schematic structural block diagram of a data processing apparatus according to an embodiment of the present disclosure.

[0084] As shown in Figure 8 , the data processing apparatus 800 can include a type determination module 810, a conversion module 820, and a first storage module 830.

[0085] The type determining module 810 is configured to determine a target type from a plurality of candidate data types according to data values of initial data loaded into a target container, wherein the target container comprises a basic storage area for storing a mark and an extended storage area for storing the initial data, and the mark represents an address of the initial data in the extended storage area.

[0086] The conversion module 820 is configured to obtain converted data by converting the initial data from an initial type to the target type, wherein a data length of the converted data is less than a data length of the initial data.

[0087] The first storage module 830 is configured to store the converted data to an available storage area other than the basic storage area in response to detecting that a capacity difference between the first required storage area and the basic storage area satisfies a first predetermined condition.

[0088] According to another embodiment of the present disclosure, the apparatus further comprises a first processing module and a data storage module. The first processing module is configured to delete the mark in response to detecting that the capacity difference between the first required storage area and the basic storage area does not satisfy the first predetermined condition. The data storage module is configured to store the converted data to the basic storage area.

[0089] According to another embodiment of the present disclosure, the basic storage area comprises a high-bit basic storage area and a low-bit basic storage area, and the mark is stored in the low-bit basic storage area. The apparatus further comprises a second storage module configured to store a data attribute of the converted data to the high-bit basic storage area.

[0090] According to another embodiment of the present disclosure, the first predetermined condition comprises one of the following: a capacity of the first required storage area is greater than a capacity of the basic storage area; and a capacity of the first required storage area is greater than a capacity of the low-bit basic storage area.

[0091] According to another embodiment of the present disclosure, the data attribute comprises at least one of the following: the target type, a compression state, and a data length of the converted data.

[0092] According to another embodiment of the present disclosure, the number of the initial data is a plurality. The type determining module comprises a type determining submodule configured to determine the target type according to a maximum data value and a minimum data value in the plurality of initial data.

[0093] According to another embodiment of the present disclosure, the first storage module comprises an allocation submodule, a first storage submodule, an update submodule, and a deletion submodule. The allocation submodule is configured to determine a target storage area in the available storage area according to a capacity of the first required storage area. The first storage submodule is configured to store the converted data to the target storage area. The update submodule is configured to update the mark according to a storage address of the converted data in the target storage area. The deletion submodule is configured to delete the initial data from the extended storage area.

[0094] According to another embodiment of the present disclosure, the apparatus further includes a determination sub-module configured to determine the first demand storage area according to the target type and the number of initial data.

[0095] According to another embodiment of the present disclosure, the apparatus further includes a judgment module and a second processing module. The judgment module is configured to determine whether the initial container satisfies a second predetermined condition in response to obtaining the data to be stored, according to the second demand storage area of the data to be stored and the idle basic storage area of the initial container. The second processing module is configured to perform a predetermined operation on the initial container to obtain a target container in response to detecting that the initial container satisfies the second predetermined condition, wherein the predetermined operation includes: allocating a new storage area to the initial container as an extended storage area according to the second demand storage area; storing the data to be stored and the data already stored in the basic storage area as initial data into the extended storage area; and constructing a mark in the basic storage area.

[0096] According to another embodiment of the present disclosure, the basic storage area includes a high basic storage area and a low basic storage area; and the second processing module includes a construction sub-module configured to construct the mark in the low basic storage area and configure the high basic storage area to be in an idle state.

[0097] According to another embodiment of the present disclosure, the apparatus further includes a judgment module and a third storage module. The judgment module is configured to determine whether the initial container satisfies a second predetermined condition in response to obtaining the data to be stored, according to the second demand storage area of the data to be stored and the idle basic storage area of the initial container. The third storage module is configured to store the data to be stored into the idle basic storage area in response to detecting that the initial container does not satisfy the second predetermined condition.

[0098] According to another embodiment of the present disclosure, the basic storage area includes a high basic storage area and a low basic storage area; and the third storage module includes a second storage sub-module and a third storage sub-module. The second storage sub-module is configured to store the data to be stored into the low basic storage area. The third storage sub-module is configured to store the attribute of the data to be stored into the high basic storage area.

[0099] According to another embodiment of the present disclosure, the second predetermined condition includes one of: a capacity of the second demand storage area is greater than a sum of a capacity of idle space in the high basic storage area and a capacity of idle space in the low basic storage area; and a capacity of the second demand storage area is greater than a capacity of idle space in the idle low basic storage area.

[0100] According to an embodiment of the present disclosure, the present disclosure further provides an electronic device, comprising at least one processor; and a memory connected with the at least one processor in communication; the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the above data processing method.

[0101] According to an embodiment of the present disclosure, the present disclosure further provides a non-transitory computer readable storage medium storing computer instructions, wherein the computer instructions are used to enable a computer to perform the above data processing method.

[0102] According to an embodiment of the present disclosure, the present disclosure further provides a computer program product comprising a computer program, wherein the computer program is executed by a processor to implement the above data processing method.

[0103] Figure 9 is a structural block diagram of an electronic device for implementing the data processing method of the embodiments of the present disclosure. The electronic device is intended to represent various forms of digital computers, such as laptops, desktops, tablets, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular telephones, smart phones, wearable devices, and other similar computing devices. The components shown here, their connections and relationships, and their functions, are meant to be examples only, and are not intended to limit the implementations of the present disclosure described and / or claimed in this document.

[0104] As shown in Figure 9 , the device 900 includes a computing unit 901 that can perform various appropriate actions and processes in accordance with a computer program stored in a read-only memory (ROM) 902 or a computer program loaded from a storage unit 908 into a random access memory (RAM) 903. In the RAM 903, various programs and data required for the operation of the device 900 can also be stored. The computing unit 901, the ROM 902, and the RAM 903 are connected to each other through a bus 904. An input / output (I / O) interface 905 is also connected to the bus 904.

[0105] Various components in the device 900 are connected to the I / O interface 905, including: an input unit 906, such as a keyboard, a mouse, etc.; an output unit 907, such as various types of displays, speakers, etc.; a storage unit 908, such as a magnetic disk, an optical disk, etc.; and a communication unit 909, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 909 allows the device 900 to exchange information / data with other devices through a computer network, such as the Internet, and / or various telecommunications networks.

[0106] The computing unit 901 can be various general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 901 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various specialized artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, and the like. The computing unit 901 performs various methods and processes described above, such as the data processing method. For example, in some embodiments, the data processing method can be implemented as a computer software program tangibly embodied in a machine-readable medium, such as the storage unit 908. In some embodiments, part or all of the computer program can be loaded and / or installed onto the device 900 via the ROM 902 and / or the communication unit 909. When the computer program is loaded onto the RAM 903 and executed by the computing unit 901, one or more steps of the data processing method described above can be performed. Alternatively, in other embodiments, the computing unit 901 can be configured to perform the data processing method by any other suitable means, such as by means of firmware.

[0107] Various implementations of the systems and techniques described above can be realized in digital electronic circuitry, integrated circuitry, a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), a system on a chip (SOC), a complex programmable logic device (CPLD), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include implementation in one or more computer programs that are executable and / or interpretable on a programmable system including at least one programmable processor, which can be special or general purpose, coupled to receive data and instructions from, and to transmit data and instructions to, a storage system, at least one input device, and at least one output device.

[0108] Program code for carrying out methods of the present disclosure can be written in any combination of one or more programming languages. The program code can be provided to a processor or controller of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the program code, when executed by the processor or controller, produces a means for implementing the functions / acts specified in the flowcharts and / or block diagrams. The program code can be executed entirely on a machine, partially on a machine, partially on a machine as a stand-alone software package, partially on a machine and partially on a remote machine or entirely on a remote machine or server.

[0109] In the context of this disclosure, a machine-readable medium can be a tangible medium that contains or stores a program for use by or in connection with an instruction execution system, apparatus, or device. The machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include but is not limited to an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of the machine-readable storage medium would include an electrical connection based on one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0110] To provide for interaction with a user, the systems and techniques described here can be implemented on a computer having a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user and a keyboard and a pointing device (e.g., a mouse or a trackball) by which the user can provide input to the computer. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form, including acoustic, speech, or tactile input.

[0111] The systems and techniques described here can be implemented in a computing system that includes a back end component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front end component (e.g., a user computer having a graphical user interface or a Web browser through which a user can interact with an implementation of the systems and techniques described here), or any combination of such back end, middleware, or front end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), and the Internet.

[0112] The computing system can include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other.

[0113] It should be understood that the various forms of flow shown above can be used to reorder, add, or remove steps. For example, the steps recited in the present disclosure can be performed in parallel, in series, or in a different order, as long as the desired results of the technology disclosed in the present disclosure are achieved, which is not limited herein.

[0114] The foregoing detailed description has not been presented to limit the scope of the present disclosure. Various modifications and equivalents can be employed, as would be understood by one skilled in the art, in conjunction with the spirit and principles of the present disclosure. Any modification, substitution, and improvement without departing from the spirit and principles of the present disclosure should be included within the scope of the present disclosure.

Claims

1. A data processing method, comprising: For initial data loaded into a target container, determining a target type from a plurality of candidate data types according to a data value of the initial data; wherein the target container includes a basic storage area for storing a tag and an extended storage area for storing the initial data, the tag representing an address of the initial data in the extended storage area; Converting the initial data from an initial type to the target type to obtain converted data; wherein the data length of the converted data is less than the data length of the initial data; In response to detecting that a capacity difference between a first required storage area determined according to the target type and the basic storage area satisfies a first predetermined condition, storing the converted data in an available storage area other than the basic storage area, and deleting the initial data from the extended storage area; In response to detecting that the capacity difference between the first demand storage area and the basic storage area does not meet the first predetermined condition, the mark is deleted, the converted data is stored in the basic storage area, and the initial data is deleted from the extended storage area.

2. The method according to claim 1, wherein The basic storage area includes a high-order basic storage area and a low-order basic storage area, and the mark is stored in the low-order basic storage area; the method further includes: The data attributes of the converted data are stored in the high-order basic storage area.

3. The method according to claim 2, wherein: The first predetermined condition includes one of the following: The capacity of the first demand storage area is greater than the capacity of the basic storage area; and The capacity of the first demand storage area is greater than the capacity of the low-order basic storage area.

4. The method according to claim 2, wherein: The data attributes include at least one of the following: the target type, a compression state, and a data length of the converted data.

5. The method according to claim 1, wherein The number of the initial data is multiple; and determining the target type from multiple candidate data types according to the data value of the initial data includes: The target type is determined according to a maximum data value and a minimum data value in the plurality of initial data.

6. The method according to claim 1, wherein Storing the converted data in an available storage area other than the basic storage area includes: Determining a target storage area in the available storage area according to the capacity of the first required storage area; storing the converted data in the target storage area; The tag is updated according to the storage address of the converted data in the target storage area.

7. The method according to claim 1, wherein The method further comprises: The first required storage area is determined according to the target type and the amount of the initial data.

8. The method according to claim 1, further comprising: In response to acquiring the data to be stored, determining whether the initial container meets a second predetermined condition based on a second required storage area of ​​the data to be stored and free space of the basic storage area of ​​the initial container; as well as In response to detecting that the initial container meets the second predetermined condition, performing a predetermined operation on the initial container to obtain the target container; The predetermined operation includes: Allocate a new storage area for the initial container according to the second required storage area, and use it as the extended storage area; Taking the data to be stored and the data stored in the basic storage area as the initial data, and storing them in the extended storage area; and The tag is constructed in the base storage area.

9. The method according to claim 8, wherein The basic storage area includes a high-order basic storage area and a low-order basic storage area; and constructing the mark in the basic storage area includes: The tag is constructed in the low-order basic storage area, and the high-order basic storage area is configured to be in an idle state.

10. The method according to claim 1, further comprising: In response to acquiring the data to be stored, determining whether the initial container meets a second predetermined condition according to a second required storage area of ​​the data to be stored and free space in the basic storage area of ​​the initial container; as well as In response to detecting that the initial container does not meet the second predetermined condition, the data to be stored is stored in the free space.

11. The method according to claim 10, wherein: The basic storage area includes a high-order basic storage area and a low-order basic storage area; and storing the data to be stored in the free space includes: storing the data to be stored in the free space in the low-order basic storage area; and The attributes of the data to be stored are stored in the high-order basic storage area.

12. The method according to claim 9 or 11, wherein: The second predetermined condition includes one of the following: The capacity of the second demand storage area is greater than the sum of the capacity of the free space in the high-order basic storage area and the capacity of the free space in the low-order basic storage area; as well as The capacity of the second demand storage area is greater than the capacity of the free space in the low-order basic storage area.

13. A data processing device comprising: a type determination module, configured to determine, for initial data loaded into a target container, a target type from a plurality of candidate data types based on a data value of the initial data; wherein the target container includes a basic storage area for storing a tag and an extended storage area for storing the initial data, the tag representing an address of the initial data in the extended storage area; a conversion module, configured to convert the initial data from an initial type to the target type to obtain converted data; wherein the data length of the converted data is smaller than the data length of the initial data; a first storage module configured to, in response to detecting that a capacity difference between a first required storage area determined according to the target type and the basic storage area satisfies a first predetermined condition, store the converted data in an available storage area other than the basic storage area, and delete the initial data from the extended storage area; A first processing module and a data storage module, wherein the first processing module is used to delete the mark in response to detecting that the capacity difference between the first demand storage area and the basic storage area does not meet the first predetermined condition; the data storage module is used to store the converted data in the basic storage area and delete the initial data from the extended storage area.

14. The device according to claim 13, wherein The basic storage area includes a high-order basic storage area and a low-order basic storage area, and the mark is stored in the low-order basic storage area; the device further includes: The second storage module is used to store the data attributes of the converted data in the high-order basic storage area.

15. The device according to claim 14, wherein The first predetermined condition includes one of the following: The capacity of the first demand storage area is greater than the capacity of the basic storage area; and The capacity of the first demand storage area is greater than the capacity of the low-order basic storage area.

16. The device according to claim 14, wherein The data attributes include at least one of the following: the target type, a compression state, and a data length of the converted data.

17. The device according to claim 13, wherein The number of the initial data is multiple; the type determination module includes: The type determination submodule is used to determine the target type according to the maximum data value and the minimum data value in the multiple initial data.

18. The device according to claim 13, wherein The first storage module includes: an allocation submodule, configured to determine a target storage area in the available storage area according to the capacity of the first required storage area; A first storage submodule, configured to store the converted data into the target storage area; An updating submodule is configured to update the tag according to a storage address of the converted data in the target storage area.

19. The apparatus according to claim 13, further comprising: A determination submodule is configured to determine the first required storage area according to the target type and the amount of the initial data.

20. The apparatus according to claim 13, further comprising: a judgment module, configured to, in response to obtaining the data to be stored, determine whether the initial container satisfies a second predetermined condition based on a second required storage area of ​​the data to be stored and free space in the basic storage area of ​​the initial container; as well as a second processing module, configured to, in response to detecting that the initial container satisfies the second predetermined condition, perform a predetermined operation on the initial container to obtain the target container; The predetermined operation includes: Allocate a new storage area for the initial container according to the second required storage area, and use it as the extended storage area; Taking the data to be stored and the data stored in the basic storage area as the initial data, and storing them in the extended storage area; and The tag is constructed in the base storage area.

21. The device according to claim 20, wherein The basic storage area includes a high-order basic storage area and a low-order basic storage area; the second processing module includes: A construction submodule is used to construct the mark in the low-order basic storage area and configure the high-order basic storage area to be in an idle state.

22. The apparatus according to claim 13, further comprising: a judgment module, configured to, in response to obtaining the data to be stored, determine whether the initial container satisfies a second predetermined condition based on a second required storage area of ​​the data to be stored and free space in the basic storage area of ​​the initial container; as well as A third storage module is configured to store the data to be stored in the free space in response to detecting that the initial container does not satisfy the second predetermined condition.

23. The device according to claim 22, wherein The basic storage area includes a high-order basic storage area and a low-order basic storage area; The third storage module includes: A second storage submodule is configured to store the data to be stored into the free space in the low-order basic storage area; as well as The third storage submodule is configured to store the attributes of the data to be stored in the high-order basic storage area.

24. The device according to claim 21 or 23, wherein The second predetermined condition includes one of the following: The capacity of the second demand storage area is greater than the sum of the capacity of the free space in the high-order basic storage area and the capacity of the free space in the low-order basic storage area; as well as The capacity of the second demand storage area is greater than the capacity of the free space in the free lower basic storage area.

25. An electronic device comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 12.

26. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1 to 12.

27. A computer program product comprising a computer program which, when executed by a processor, implements the method according to any one of claims 1 to 12.

Images