Data processing method and device, electronic equipment and storage medium

Abstract

The invention provides a data processing method and device, electronic equipment and a storage medium, relates to the technical field of data processing, in particular to the field of memory compression and the like, and can be applied to an intelligent recommendation scene. According to the specific implementation scheme, for initial data loaded into a target container, a target type is determined from a plurality of candidate data types according to a data value of the initial data; wherein the target container comprises a basic storage area used for storing the mark and an extended storage area used for storing the initial data, and the mark represents the address of the initial data in the extended storage area; the method comprises the following steps: converting initial data from an initial type to a target type to obtain converted data; the data length of the converted data is smaller than that of the initial data; and in response to detection that the capacity difference between the first demand storage area and the basic storage area determined according to the target type meets a first preset condition, storing the converted data to an available storage area except the basic storage area.

Description

Technical Field

[0001] The present disclosure relates to the field of data processing technology, and in particular to fields such as memory compression, and can be applied to intelligent recommendation scenarios. More specifically, the present disclosure provides a data processing method, device, electronic device, storage medium, and computer program product. Background Art

[0002] In the process of software development, some general container libraries can be used to manage memory and data. For example, the general container library can be STL (Standard Template Library, a C++ standard template library). The container library provides data structures and algorithms to simplify programming, but in some scenarios, the use of general container libraries has a high memory overhead. Summary of the invention

[0003] The present disclosure provides a data processing method, apparatus, electronic device, storage medium, and computer program product.

[0004] According to one aspect of the present disclosure, a data processing method is provided, comprising: for initial data loaded into a target container, determining a target type from a plurality of candidate data types according to a data value of the initial data; wherein the target container comprises a basic storage area for storing a tag and an extended storage area for storing the initial data, the tag representing an address of the initial data in the extended storage area; obtaining converted data by converting the initial data from the initial type to the target type; the data length of the converted data is less than the data length of the initial data; in response to detecting that a capacity difference between a first required storage area determined according to the target type and the basic storage area satisfies a first predetermined condition, storing the converted data in an available storage area other than the basic storage area.

[0005] According to another aspect of the present disclosure, a data processing device is provided, comprising: a type determination module, a conversion module and a first storage module. The type determination module is used to determine a target type from a plurality of candidate data types according to the data value of the initial data loaded into a target container; wherein the target container comprises a basic storage area for storing a mark and an extended storage area for storing the initial data, and the mark represents the address of the initial data in the extended storage area. The conversion module is used to obtain converted data by converting the initial data from the initial type to the target type; the data length of the converted data is less than the data length of the initial data. The first storage module is used to store the converted data in an available storage area other than the basic storage area in response to detecting that the capacity difference between the first demand storage area determined according to the target type and the basic storage area satisfies a first predetermined condition.

[0006] According to another aspect of the present disclosure, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the method provided by the present disclosure.

[0007] According to another aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable a computer to execute the method provided by the present disclosure.

[0008] According to another aspect of the present disclosure, a computer program product is provided, including a computer program, and when the computer program is executed by a processor, the method provided by the present disclosure is implemented.

[0009] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure.

[0011] Figure 1 is a schematic flow chart of a data processing method according to an embodiment of the present disclosure;

[0012] Figure 2A is a schematic diagram of a basic storage area according to an embodiment of the present disclosure;

[0013] Figure 2B is a schematic diagram of a basic storage area that meets a first predetermined condition according to an embodiment of the present disclosure;

[0014] Figure 2C is a schematic diagram of a basic storage area that does not meet a first predetermined condition according to an embodiment of the present disclosure;

[0015] Figure 3 is a schematic diagram of a data processing method when a first predetermined condition is met according to an embodiment of the present disclosure;

[0016] Figure 4 is a schematic diagram of a data processing method when a first predetermined condition is not met according to an embodiment of the present disclosure;

[0017] Figure 5 is a schematic flow chart of a method for constructing a target container according to an embodiment of the present disclosure;

[0018] Fig. 6A is a schematic diagram of a method for constructing a target container when a second predetermined condition is met according to an embodiment of the present disclosure;

[0019] Figure 6B yes Fig. 6A A schematic diagram of the basic storage area in ;

[0020] Figure 7 is a schematic principle diagram of constructing a target container when the second predetermined condition is not met according to an embodiment of the present disclosure;

[0021] Figure 8 is a schematic structural block diagram of a data processing device according to an embodiment of the present disclosure; and

[0022] Fig. 9 It is a structural block diagram of an electronic device used to implement the data processing method of the embodiment of the present disclosure. DETAILED DESCRIPTION

[0023] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0024] In the technical solution of the present disclosure, the collection, storage, use, processing, transmission, provision and disclosure of user personal information involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0025] In the technical solution of the present disclosure, the user's authorization or consent is obtained before obtaining or collecting the user's personal information.

[0026] In some application scenarios, some data is updated very infrequently but needs to be used frequently. This part of data requires a certain amount of memory to store, which will lead to high memory overhead in actual applications. This not only increases the resource consumption of the system, but may also affect the overall performance of the system, such as key indicators such as response time and throughput. Therefore, it is necessary to reduce the memory space required to store data, especially for data with a very low update frequency and a high frequent update frequency, to reduce the memory space required for such data.

[0027] The embodiment of the present disclosure aims to provide a data processing method, which can achieve the effect of saving memory. The method can be applied to online service scenarios that need to hold a large amount of data for a long time, such as positive cache objects, dictionary objects, and medium- and long-term resource dimension cache objects. For positive cache objects in online services, the application of the technical solution provided by the present disclosure can optimize memory usage, improve cache hit rate, and reduce memory waste. For dictionary objects, the application of the technical solution provided by the present disclosure can reduce memory usage, improve query efficiency, and reduce system load. For resource dimension cache objects, the application of the technical solution provided by the present disclosure can optimize memory management, increase data access speed, and enhance user experience.

[0028] The technical solution provided by the present disclosure will be described in detail below with reference to the accompanying drawings and specific embodiments.

[0029] Figure 1 is a schematic flow chart of a data processing method according to an embodiment of the present disclosure.

[0030] like Figure 1 As shown, the data processing method 100 may include operations S110 to S130.

[0031] In operation S110, for the initial data loaded into the target container, a target type is determined from a plurality of candidate data types according to the data value of the initial data; wherein the target container includes a basic storage area for storing a tag and an extended storage area for storing the initial data, and the tag represents an address of the initial data in the extended storage area.

[0032] For example, the target container includes a basic storage area and an extended storage area, wherein the basic storage area is determined based on rules when creating the container and is generally not modified after the container is created. The basic storage area is used to provide space occupied by the basic members of the container. The extended storage area is dynamically adjusted according to capacity requirements after the container is created. For example, if the capacity of the container is insufficient, the capacity of the extended storage area can be increased. If the capacity of the container is surplus, the capacity of the extended storage area can be reduced. The basic storage area stores a mark, which can be a pointer. At the same time, the extended storage area stores data loaded into the target container. The mark points to the address of the data, so the data can be indexed through the mark.

[0033] For example, data types include integer, floating point, etc., and integer types may include int8, int16, int32, etc.

[0034] For example, the initial type of the initial data is int32, but the data value of the initial data is small, resulting in the high bits of int32 being all 0. Int16 can be used to represent the initial data normally, and the data precision of the initial data will not be affected. Therefore, int16 can be used as the target type. For example, the initial type of the initial data is floating point, but the decimal part of the initial data is all 0. Therefore, integer can be used to represent the initial data normally, and the data precision of the initial data will not be affected. Therefore, integer can be used as the target type.

[0035] In operation S120, converted data is obtained by converting the initial data from an initial type to a target type; a data length of the converted data is smaller than a data length of the initial data.

[0036] For example, when initial data of the int32 type is converted to converted data of the int16 type, the data length changes from 4 bytes to 2 bytes.

[0037] In operation S130, in response to detecting that a capacity difference between a first required storage area determined according to a target type and a basic storage area satisfies a first predetermined condition, the converted data is stored in an available storage area outside the basic storage area.

[0038] For example, the data length required for each data type is known, so the storage area capacity required to store the processed data can be calculated according to the target type, and the storage area capacity is the capacity of the first required storage area. For another example, the data type of the initial data can be converted based on the target type to obtain the converted data, and then the converted data can be read to determine the first required storage area.

[0039] For example, the available storage area outside the basic storage area may include an existing extended storage area of ​​the target container, and the converted data may be stored in the extended storage area. Another extended storage area may also be allocated, and the available storage area outside the basic storage area may include the newly allocated extended storage area, and the converted data may be stored in the newly allocated extended storage area.

[0040] The technical solution provided in this embodiment first converts the initial data from the initial type to the target type, so that the data length of the converted data is less than the data length of the initial data. In this way, by updating the initial data with the converted data, the required storage space can be reduced, and the memory saving effect can be achieved by reconstructing the data type, reducing the memory usage of the target container, thereby alleviating the memory pressure of the online service and improving the system performance and stability. In addition, in this embodiment, the target container includes a basic storage area and an extended storage area, and the basic storage area stores a mark, and the mark points to the address of the initial data in the extended storage area. When the capacity difference between the first demand storage area and the basic storage area meets the first predetermined condition, the converted data is stored in the available storage area outside the basic storage area, and the storage address of the converted data in the available storage area can also be updated. The mark is indexed to the processed data through the mark, so that after the initial data is updated, the processed data can be guaranteed to be normally accessed.

[0041] It should be noted that in some application scenarios, some data are updated very infrequently, but need to be used frequently. For these data, the technical solution provided by this embodiment can be used to update the initial data to the converted data, which can achieve the effect of data compression, thereby saving storage area. It is understandable that if the target container is dynamically resized according to the actual data, memory fragmentation and performance degradation will result. Therefore, after the data type conversion is performed, the size of the target container can be fixed, that is, the processed data loaded into the target container remains unchanged, and the target container is restricted from frequent deletion and insertion operations during operation, so that memory fragmentation can be reduced by pre-building the target container and fixing the size.

[0042] Figure 2A is a schematic diagram of a basic storage area according to an embodiment of the present disclosure.

[0043] like Figure 2A As shown, in this embodiment, the basic storage area includes a high-order basic storage area and a low-order basic storage area. Taking a 64-bit machine as an example, a tag will apply for 8 bytes of memory, for example, the high 16 bits are the high-order basic storage area, the low 48 bits are the low-order basic storage area, and generally only the low-order basic storage area is used to store tags, while the high-order basic storage area is not used. Figure 2A In the example, the basic storage area includes the high-order basic storage area Byte1, Byte2 and the low-order basic storage area Byte3~Byte8.

[0044] Figure 2B is a schematic diagram of a basic storage area that meets a first predetermined condition according to an embodiment of the present disclosure.

[0045] like Figure 2BAs shown, in this embodiment, when the capacity difference between the first demand storage area and the basic storage area meets the first predetermined condition, the mark can be stored in the low-order basic storage area, and the data attribute of the converted data can be stored in the high-order basic storage area. In this way, the high-order basic storage area can be utilized to avoid it being idle, and more information can be stored without increasing additional storage overhead. And after the attribute information is stored in the high-order basic storage area, there is no need to use other memory to store data attributes, thereby achieving the effect of saving storage area.

[0046] Figure 2C is a schematic diagram of a basic storage area that does not meet the first predetermined condition according to an embodiment of the present disclosure.

[0047] like Figure 2C As shown, in this embodiment, if the capacity difference between the first demand storage area and the basic storage area does not meet the first predetermined condition, the processed data can be stored in the basic storage area, for example, in the low-order basic storage area. The data attributes of the converted data can be stored in the high-order basic storage area.

[0048] In one example, the data attribute includes at least one of the following: target type, compression status, and data length of the converted data, and the compression status may indicate whether compression is performed. Figure 2B and Figure 2C As shown in the figure, a basic storage area occupies 8 bytes, of which the first byte Byte1 can be used to store the data type. The data type of the converted data is the target type, and the data type includes, for example, whether it is a signed integer. The first byte Byte1 can also be used to store whether it is compressed, such as whether it is floating point compressed. The second high-order byte Byte2 is used to store the data length of the converted data. In addition, the six low-order bytes are used to store tags or processed data. Through the data type, the data length of the converted data and the tag, the processed data can be accurately read from the extended storage area. Figure 2B and Figure 2C The target type is an integer and the data length is 1 byte.

[0049] Figure 3 is a schematic diagram of a data processing method when a first predetermined condition is met according to an embodiment of the present disclosure, Figure 4 It is a schematic diagram of a data processing method when the first predetermined condition is not met according to an embodiment of the present disclosure.

[0050] In this embodiment, for the initial data loaded into the target container, the target type is determined from a plurality of candidate data types according to the data value of the initial data, for example, the initial type of the initial data is int32, and the target type is int16.

[0051] Next, it can be determined whether the target container meets the first predetermined condition based on the first demand storage area and the basic storage area. For example, when there is no need to use the basic storage area to store the data attributes of the processed data, the first predetermined condition may include: the capacity of the first demand storage area is greater than the capacity of the basic storage area. In this way, it can be determined whether the basic storage area can accommodate the processed data. For another example, when the basic storage area is used to store the data attributes of the processed data, the first predetermined condition may include: the capacity of the first demand storage area is greater than the capacity of the low-level basic storage area. In this way, it can be guaranteed that the low-level basic storage area can accommodate the processed data on the basis of the storage area reserved for the data attributes.

[0052] like Figure 3 As shown, if the first predetermined condition is met, it means that the basic storage area cannot accommodate the processed data or that the low-order basic storage area cannot accommodate the processed data. At this time, the processed data can be used to update the initial data and update the mark. The specific update method can refer to the above, and this embodiment will not be repeated. For example, the converted data of int16 is stored in the extended storage area, and the initial data of int32 is deleted. At this time, the storage area used by the target container includes the basic storage area and the storage area occupied by the converted data of int16. The difference between the storage area occupied by the initial data of int32 and the storage area occupied by the converted data of int16 is the saved storage area.

[0053] like Figure 4 As shown, if the first predetermined condition is not met, it means that the basic storage area can accommodate the processed data or that the low-order basic storage area can accommodate the processed data. At this time, the mark can be deleted, and the converted data is stored in the basic storage area. The initial data in the extended storage area can also be deleted to release the extended storage area. In this embodiment, when the amount of converted data is small, the basic storage area can accommodate the part of the converted data, so the mark can no longer be used, but the basic storage area of ​​the target container can be directly used to store the processed data. In this way, the processed data can reuse the storage area occupied by the mark without using the extended storage area outside the basic storage area. Therefore, the extended storage area can be released, and the memory overhead can be further reduced and the memory usage efficiency can be improved.

[0054] For example, Figure 4 As shown, the converted data of int16 is stored in the basic storage area, and the initial data of int32 is deleted. At this time, the storage area used by the target container includes the basic storage area, and there is no need to use the extended storage area. The saved storage area is the extended storage area.

[0055] It can be seen that the present disclosure adopts different strategies according to whether the target container meets the first predetermined condition. When the first predetermined condition is not met, the mark is deleted, and the converted data is stored in the storage area occupied by the mark. When the first predetermined condition is met, the target type can be determined according to the data value of the initial data, and the data type of the initial data can be converted, and then the initial data can be updated with the converted data to achieve a memory compression effect. In addition, since different strategies are adopted depending on whether the target container meets the first predetermined condition, targeted processing can be performed according to the needs of different scenarios, thereby improving the flexibility and efficiency of memory use.

[0056] According to another embodiment of the present disclosure, the number of initial data is multiple, and accordingly, the above process of determining the target type from multiple candidate data types based on the data value of the initial data may include: determining the target type based on the maximum data value and the minimum data value among the multiple initial data.

[0057] In this embodiment, after the target container has loaded all the initial data, the memory can be compressed. During the compression process, the initial data is traversed to find the maximum data value and the minimum data value, and then the data type with the minimum data length that can be used is determined according to the range of the maximum data value and the minimum data value. The data type is the target type, and then the initial data is converted into converted data of the target type. In this way, the storage space of each initial data can be reduced, thereby reducing the total memory usage.

[0058] According to another embodiment of the present disclosure, the process of storing the converted data in the available storage area outside the basic storage area may include: determining a target storage area in the available storage area according to the capacity of the first required storage area, then storing the converted data in the target storage area, and updating a mark according to the storage address of the converted data in the target storage area, and deleting the initial data from the extended storage area.

[0059] For example, the initial data in the target container can be traversed, and the storage area capacity required for storing the processed data can be counted according to the target type and the number of initial data. The capacity of the storage area is the capacity of the first demand storage area. Then, the target storage area can be reallocated according to the size of the first demand storage area counted, and the size of the target storage area matches the size of the first demand storage area. Matching can indicate that the size of the target storage area is equal to or slightly larger than the size of the first demand storage area. After obtaining the target storage area, the converted data can be stored in the target storage area, and then the initial data can be deleted from the extended storage area, thereby releasing the extended storage area and reclaiming useless storage resources. This embodiment accurately allocates the size of the target storage area according to demand, thereby storing the processed data completely and effectively reducing unnecessary memory usage. In addition, in the process of allocating the target storage area, since the size of the first demand storage area has been predetermined, the target area can be allocated at one time, and there is no need to gradually expand the area in multiple times to obtain the target area, which can provide storage area allocation efficiency, avoid frequent storage area application and release, reduce performance overhead, and reduce memory fragmentation.

[0060] According to another embodiment of the present disclosure, the process of determining the first required storage area according to the target type may include: determining the first required storage area according to the target type and the number of initial data. For example, for the same target container, the number of initial data loaded into the target container is multiple, and the target type is int16. After the data type of the initial data is converted, each processed data needs to use a 2-byte storage area. The first demand information can be determined by counting the total bytes required for all processed data. This embodiment can accurately calculate the first required storage area through the target type and the number of initial data, and then in the process of allocating the target storage area in the available storage area, the size of the target storage area can be accurately allocated according to demand.

[0061] Figure 5 is a schematic flowchart of a method for constructing a target container according to an embodiment of the present disclosure.

[0062] The above describes the processing process of the target container. It should be noted that the target container is obtained based on the initial container. Next, the process of determining the target container using the initial container is described.

[0063] In this embodiment, the method for determining the target container may further include operations S501 to S504.

[0064] In operation S501, data to be stored is obtained.

[0065] In operation S502, it is determined whether the initial container satisfies a second predetermined condition according to a second required storage area for data to be stored and free space in a basic storage area of ​​the initial container.

[0066] For example, if multiple data need to be loaded into the initial container, the loading process is completed by storing the data to be stored in the basic storage area or the extended storage area of ​​the initial container. The number of data to be stored that needs to be loaded is at least one, and the data to be stored can be obtained one by one or in batches.

[0067] After acquiring the data to be stored, it is determined whether the initial container meets the second predetermined condition according to the second required storage area of ​​the data to be stored and the free space of the initial container. For example, the second predetermined condition indicates whether the free space in the basic storage area of ​​the initial container can store the at least one acquired data to be stored. If not, it can be determined that the second predetermined condition is met, and if so, it can be determined that the second predetermined condition is met.

[0068] In operation S503, if the initial container does not satisfy the second predetermined condition, the data to be stored is stored in the free space in the basic storage area.

[0069] It should be noted that in actual applications, a single target container can be used to load data for a specific purpose. Taking data with three purposes online as an example, the number of data for some purposes is small, while the number of data for some purposes is large, and the number of data loaded by the target container cannot be known in advance. In this embodiment, in the process of building a target container based on the initial container, the construction process of the target container is divided into two modes based on whether the second predetermined condition is met, and the free space of the basic storage area in the initial container is preferentially used to store the obtained data to be stored. When the free space in the initial container cannot accommodate the data to be stored, the extended storage area is used to store the data to be stored. In this way, in online applications, when the number of data for a certain purpose is small, the data is directly stored in the basic storage area through the above operation S503 without using the extended storage area, so that the data type conversion of the data to be stored can be omitted later, thereby further improving the data processing efficiency.

[0070] In operation S504, if the initial container meets the second predetermined condition, a predetermined operation is performed on the initial container to obtain a target container. The predetermined operation includes: allocating a new storage area for the initial container according to the second required storage area and using it as an extended storage area. The data to be stored and the data stored in the basic storage area are also stored as initial data in the extended storage area. A tag is also constructed in the basic storage area.

[0071] For example, according to the acquired initial type and number of data to be stored, the storage area size required to store the data to be stored is counted, and the storage area size is the second required storage area. The extended storage area can be allocated according to the size of the second required storage area.

[0072] For another example, you can call the reserve method or apply for an extended storage area of ​​a predetermined size through other means, and then store the data to be stored in the extended storage area as initial data, and you can also move the data stored in the basic storage area to the extended storage area as initial data. Next, you can determine whether the extended storage area after allocation is sufficient. If the allocated extended storage area is smaller than the second required storage area, resulting in the inability to store all the initial data in the extended storage area, you can return to the operation of applying for an extended storage area of ​​a predetermined size, and repeat the operation until all the initial data is stored in the extended storage area. In addition, a mark is constructed in the basic storage area of ​​the initial container, and the mark points to the address of the initial data in the extended storage area.

[0073] It can be seen that in the above operation S504, when the free basic storage area in the initial container cannot accommodate the data to be stored, the extended storage area is used to store the data to be stored, so that when the basic storage area has insufficient space, the data to be stored can be stored normally, ensuring that the data to be stored is smoothly loaded into the initial container.

[0074] Fig. 6A is a schematic diagram of a method for constructing a target container when a second predetermined condition is met according to an embodiment of the present disclosure, Figure 6B yes Fig. 6A A schematic diagram of the basic storage area in , Figure 7 It is a schematic principle diagram of constructing a target container when the second predetermined condition is not met according to an embodiment of the present disclosure.

[0075] For example, there are multiple data to be stored, and the multiple data to be stored are loaded into the initial container one by one in the form of data stream. In the process of loading the first 5 data to be stored, since the idle basic storage area of ​​the initial container is relatively large, the first 5 data to be stored are directly stored in the idle basic storage area.

[0076] As shown in Figure 6, the sixth data to be stored Data_6 is obtained. At this time, the basic storage area of ​​the initial container can still accommodate the sixth data to be stored Data_6, and the sixth data to be stored Data_6 is directly stored in the basic storage area. In addition, the basic storage area may include a high-order basic storage area and a low-order basic storage area. The attributes of the data to be stored Data_1~Data_6 can be stored in the high-order basic storage area, and then the data Data_1~Data_6 are stored in the remaining low-order storage area. In addition, it should be noted that the attributes of each data in the basic storage area can be the same, so the attributes can be stored only once, and there is no need to store the attributes of all the data separately, thereby further saving memory.

[0077] like Figure 7 As shown, the seventh data to be stored Data_7 is continued to be obtained. At this time, the free basic storage area cannot continue to accommodate the seventh data to be stored Data_7. Therefore, the first six data Data_1~Data_6 are taken out from the basic storage area and moved to the extended storage area. The seventh data to be stored Data_7 is also stored in the extended storage area, and a mark is constructed in the basic storage area to characterize the position of the seven data to be stored in the extended storage area. In addition, the basic storage area may include a high-order basic storage area and a low-order basic storage area. The properties of the data Data_1~Data_7 may be stored in the high-order storage area, wherein the properties of Data_1~Data_7 may be the same, so in order to save storage space, they may be stored only once. The mark may be stored in the low-order storage area, and the mark may be a pointer, which may point to the address of the first data Data_1 in the extended storage area.

[0078] In some embodiments, in the process of loading the data to be stored in the initial container, for example, storing the data to be stored in the free space in the basic storage area, or storing it in the extended storage area, a suitable data type can be selected according to the data value of the data to be stored, and the data to be stored can be stored according to the selected data type. For example, if the data value to be processed is an integer, an integer is selected for storage, thereby reducing memory usage and ensuring data accuracy.

[0079] In one example, the basic storage area includes a high-order basic storage area and a low-order basic storage area. Taking a 64-bit machine as an example, a tag will apply for 8 bytes of memory, of which the high 16 bits can be the high-order basic storage area and the low 48 bits can be the low-order basic storage area.

[0080] For the case where the initial container satisfies the second predetermined condition, a mark can be constructed in the low-order basic storage area, and the high-order basic storage area can be configured to be in an idle state. In this way, when the data is subsequently compressed, if the storage area required to be occupied by the processed data is not larger than the size of the low-order basic storage area, the data attributes of the processed data can be directly written into the high-order basic storage area, so that the high-order basic storage area can be utilized to avoid being idle, thereby avoiding the use of other memories to store data attributes, thereby achieving the effect of saving storage area. In addition, if the high-order basic storage area has data stored in advance, the existing data needs to be cleared first when the high-order basic storage area needs to be used. If the high-order basic storage area is in an idle state, the data attributes can be directly written when the high-order basic storage area needs to be used, thereby improving data storage efficiency.

[0081] If the initial container does not meet the second predetermined condition, the data to be stored can be stored in the low-order basic storage area, and the attributes of the data to be stored can also be stored in the high-order basic storage area. In this way, the high-order basic storage area can be utilized to avoid it being idle, and more information can be stored without increasing additional storage overhead. After the attribute information is stored in the high-order basic storage area, there is no need to use other memory to store data attributes, thereby achieving the effect of saving storage area.

[0082] For example, the second predetermined condition may include: the capacity of the second demand storage area is greater than the capacity of the free space. It should be noted that the basic storage area includes a high-order basic storage area and a low-order basic storage area, and the free space may include two parts, one part is located in the high-order basic storage area, and the other part is located in the low-order basic storage area. When there is no need to use the basic storage area to store the data attributes of the processed data, the second predetermined condition may include: the capacity of the second demand storage area is greater than the sum of the capacities of the two parts of free space. For another example, when the basic storage area is used to store the data attributes of the processed data, the second predetermined condition may include: the capacity of the second demand storage area is greater than the capacity of the free space in the low-order basic storage area. In this way, on the basis of the storage area reserved for the data attributes, it can be guaranteed whether the low-order basic storage area can accommodate the processed data.

[0083] Figure 8 It is a schematic structural block diagram of a data processing device according to an embodiment of the present disclosure.

[0084] like Figure 8 As shown, the data processing device 800 may include a type determination module 810 , a conversion module 820 and a first storage module 830 .

[0085] The type determination module 810 is used to determine the target type from multiple candidate data types according to the data value of the initial data loaded into the target container; wherein the target container includes a basic storage area for storing a tag and an extended storage area for storing the initial data, and the tag represents the address of the initial data in the extended storage area.

[0086] The conversion module 820 is used to obtain converted data by converting the initial data from an initial type to a target type; the data length of the converted data is smaller than the data length of the initial data.

[0087] The first storage module 830 is used to store the converted data in an available storage area other than the basic storage area in response to detecting that the capacity difference between the first required storage area determined according to the target type and the basic storage area meets a first predetermined condition.

[0088] According to another embodiment of the present disclosure, the device further includes: a first processing module and a data storage module. The first processing module is used to delete the mark in response to detecting that the capacity difference between the first demand storage area and the basic storage area does not meet the first predetermined condition. The data storage module is used to store the converted data in the basic storage area.

[0089] According to another embodiment of the present disclosure, the basic storage area includes a high-order basic storage area and a low-order basic storage area, and the mark is stored in the low-order basic storage area; the device also includes: a second storage module, used to store data attributes of the converted data in the high-order basic storage area.

[0090] According to another embodiment of the present disclosure, the first predetermined condition includes one of the following: the capacity of the first demand storage area is greater than the capacity of the basic storage area; and the capacity of the first demand storage area is greater than the capacity of the low-order basic storage area.

[0091] According to another embodiment of the present disclosure, the data attribute includes at least one of the following: a target type, a compression state, and a data length of the converted data.

[0092] According to another embodiment of the present disclosure, the number of initial data is multiple; the type determination module includes: a type determination submodule, used to determine the target type according to the maximum data value and the minimum data value in the multiple initial data.

[0093] According to another embodiment of the present disclosure, the first storage module includes: an allocation submodule, a first storage submodule, an update submodule and a deletion submodule. The allocation submodule is used to determine a target storage area in the available storage area according to the capacity of the first required storage area. The first storage submodule is used to store the converted data in the target storage area. The update submodule is used to update the mark according to the storage address of the converted data in the target storage area. The deletion submodule is used to delete the initial data from the extended storage area.

[0094] According to another embodiment of the present disclosure, the above-mentioned device further includes: a determination submodule, configured to determine a first required storage area according to the target type and the amount of initial data.

[0095] According to another embodiment of the present disclosure, the above-mentioned device further includes: a judgment module and a second processing module. The judgment module is used to determine whether the initial container meets the second predetermined condition in response to obtaining the data to be stored, according to the second required storage area of ​​the data to be stored and the free basic storage area of ​​the initial container. The second processing module is used to perform a predetermined operation on the initial container in response to detecting that the initial container meets the second predetermined condition to obtain a target container; wherein the predetermined operation includes: allocating a new storage area for the initial container according to the second required storage area and using it as an extended storage area; using the data to be stored and the data stored in the basic storage area as initial data and storing them in the extended storage area; and constructing a mark in the basic storage area.

[0096] According to another embodiment of the present disclosure, the basic storage area includes a high-order basic storage area and a low-order basic storage area; the second processing module includes: a construction submodule, used to construct a mark in the low-order basic storage area and configure the high-order basic storage area to be in an idle state.

[0097] According to another embodiment of the present disclosure, the above-mentioned device further includes: a judgment module and a third storage module. The judgment module is used to determine whether the initial container meets the second predetermined condition according to the second required storage area of ​​the data to be stored and the idle basic storage area of ​​the initial container in response to obtaining the data to be stored. The third storage module is used to store the data to be stored in the idle basic storage area in response to detecting that the initial container does not meet the second predetermined condition.

[0098] According to another embodiment of the present disclosure, the basic storage area includes a high-order basic storage area and a low-order basic storage area; the third storage module includes: a second storage submodule and a third storage submodule. The second storage submodule is used to store the data to be stored in the low-order basic storage area. The third storage submodule is used to store the attributes of the data to be stored in the high-order basic storage area.

[0099] According to another embodiment of the present disclosure, the second predetermined condition includes one of the following: the capacity of the second demand storage area is greater than the sum of the capacity of the free space in the high-order base storage area and the capacity of the free space in the low-order base storage area; and the capacity of the second demand storage area is greater than the capacity of the free space in the idle lower-order base storage area.

[0100] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, comprising at least one processor; and a memory communicatively connected to the at least one processor; the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the above-mentioned data processing method.

[0101] According to an embodiment of the present disclosure, the present disclosure further provides a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to enable a computer to execute the above data processing method.

[0102] According to an embodiment of the present disclosure, the present disclosure further provides a computer program product, including a computer program, and the computer program implements the above data processing method when executed by a processor.

[0103] Fig. 9 1 is a block diagram of an electronic device for implementing the data processing method of an embodiment of the present disclosure. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0104] like Fig. 9 As shown, the device 900 includes a computing unit 901, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 902 or a computer program loaded from a storage unit 908 into a random access memory (RAM) 903. In the RAM 903, various programs and data required for the operation of the device 900 can also be stored. The computing unit 901, the ROM 902, and the RAM 903 are connected to each other via a bus 904. An input / output (I / O) interface 905 is also connected to the bus 904.

[0105] A number of components in the device 900 are connected to the I / O interface 905, including: an input unit 906, such as a keyboard, a mouse, etc.; an output unit 907, such as various types of displays, speakers, etc.; a storage unit 908, such as a disk, an optical disk, etc.; and a communication unit 909, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 909 allows the device 900 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0106] The computing unit 901 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 901 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 901 performs the various methods and processes described above, such as data processing methods. For example, in some embodiments, the data processing method may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 908. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 900 via the ROM 902 and / or the communication unit 909. When the computer program is loaded into the RAM 903 and executed by the computing unit 901, one or more steps of the data processing method described above may be performed. Alternatively, in other embodiments, the computing unit 901 may be configured to perform the data processing method in any other appropriate manner (e.g., by means of firmware).

[0107] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0108] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0109] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or device, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0110] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0111] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0112] A computer system may include clients and servers. Clients and servers are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship to each other.

[0113] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0114] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A data processing method, comprising: For the initial data loaded into the target container, determine the target type from a plurality of candidate data types according to the data value of the initial data; wherein the target container includes a basic storage area for storing a tag and an extended storage area for storing the initial data, and the tag represents an address of the initial data in the extended storage area; Converted data is obtained by converting the initial data from an initial type to the target type; the data length of the converted data is less than the data length of the initial data; In response to detecting that a capacity difference between a first required storage area determined according to the target type and the basic storage area satisfies a first predetermined condition, the converted data is stored in an available storage area other than the basic storage area.

2. The method according to claim 1, further comprising: In response to detecting that the capacity difference between the first demand storage area and the basic storage area does not satisfy the first predetermined condition, deleting the mark; The converted data is stored in the basic storage area.

3. The method according to any one of claims 1 to 2, wherein: The basic storage area includes a high-order basic storage area and a low-order basic storage area, and the mark is stored in the low-order basic storage area; the method further includes: The data attributes of the converted data are stored in the high-order basic storage area.

4. The method according to claim 3, wherein: The first predetermined condition includes one of the following: The capacity of the first demand storage area is greater than the capacity of the basic storage area; and The capacity of the first demand storage area is greater than the capacity of the low-order basic storage area.

5. The method according to claim 3, wherein: The data attributes include at least one of the following: the target type, the compression status, and the data length of the converted data.

6. The method according to claim 1, wherein: The number of the initial data is multiple; and determining the target type from multiple candidate data types according to the data value of the initial data includes: The target type is determined according to a maximum data value and a minimum data value in the plurality of initial data.

7. The method according to claim 1, wherein: The storing the converted data in an available storage area other than the basic storage area comprises: Determining a target storage area in the available storage area according to the capacity of the first required storage area; storing the converted data in the target storage area; updating the tag according to the storage address of the converted data in the target storage area; and The initial data is deleted from the extended storage area.

8. The method according to claim 1, wherein: The method further comprises: The first required storage area is determined according to the target type and the amount of the initial data.

9. The method according to claim 1, further comprising: In response to acquiring the data to be stored, determining whether the initial container satisfies a second predetermined condition according to a second required storage area of ​​the data to be stored and free space of the basic storage area of ​​the initial container; as well as In response to detecting that the initial container satisfies the second predetermined condition, performing a predetermined operation on the initial container to obtain the target container; The predetermined operation includes: Allocate a new storage area for the initial container according to the second required storage area and use it as the extended storage area; taking the data to be stored and the data stored in the basic storage area as the initial data, and storing them in the extended storage area; and The tag is constructed in the base storage area.

10. The method according to claim 9, wherein: The basic storage area includes a high-order basic storage area and a low-order basic storage area; and constructing the mark in the basic storage area includes: The tag is constructed in the lower base storage area, and the upper base storage area is configured to be in an idle state.

11. The method according to claim 1, further comprising: In response to acquiring the data to be stored, determining whether the initial container satisfies a second predetermined condition according to a second required storage area of ​​the data to be stored and free space in the basic storage area of ​​the initial container; as well as In response to detecting that the initial container does not satisfy the second predetermined condition, the data to be stored is stored in the free space.

12. The method according to claim 11, wherein: The basic storage area includes a high-order basic storage area and a low-order basic storage area; and storing the data to be stored in the free space includes: storing the data to be stored in the free space in the low-order basic storage area; and The attributes of the data to be stored are stored in the high-order basic storage area.

13. The method according to claim 10 or 12, wherein: The second predetermined condition includes one of the following: The capacity of the second demand storage area is greater than the sum of the capacity of the free space in the high-order basic storage area and the capacity of the free space in the low-order basic storage area; as well as The capacity of the second demand storage area is greater than the capacity of the free space in the low-order basic storage area.

14. A data processing device, comprising: a type determination module, configured to determine a target type from a plurality of candidate data types according to a data value of the initial data loaded into a target container; wherein the target container comprises a basic storage area for storing a tag and an extended storage area for storing the initial data, the tag representing an address of the initial data in the extended storage area; A conversion module, configured to obtain converted data by converting the initial data from an initial type to the target type; the data length of the converted data is less than the data length of the initial data; The first storage module is configured to store the converted data in an available storage area other than the basic storage area in response to detecting that a capacity difference between a first required storage area determined according to the target type and the basic storage area satisfies a first predetermined condition.

15. The device according to claim 14, further comprising: A first processing module, configured to delete the mark in response to detecting that the capacity difference between the first demand storage area and the basic storage area does not satisfy the first predetermined condition; A data storage module is used to store the converted data in the basic storage area.

16. The device according to any one of claims 14 to 15, wherein: The basic storage area includes a high-order basic storage area and a low-order basic storage area, and the mark is stored in the low-order basic storage area; the device also includes: The second storage module is used to store the data attributes of the converted data in the high-order basic storage area.

17. The device according to claim 16, wherein: The first predetermined condition includes one of the following: The capacity of the first demand storage area is greater than the capacity of the basic storage area; and The capacity of the first demand storage area is greater than the capacity of the low-order basic storage area.

18. The device according to claim 16, wherein: The data attributes include at least one of the following: the target type, the compression status, and the data length of the converted data.

19. The device according to claim 14, wherein: The number of the initial data is multiple; the type determination module includes: The type determination submodule is used to determine the target type according to the maximum data value and the minimum data value in the multiple initial data.

20. The device according to claim 14, wherein: The first storage module comprises: an allocation submodule, configured to determine a target storage area in the available storage area according to the capacity of the first required storage area; A first storage submodule, used for storing the converted data into the target storage area; an updating submodule, configured to update the tag according to a storage address of the converted data in the target storage area; and The deleting submodule is used to delete the initial data from the extended storage area.

21. The apparatus according to claim 14, further comprising: A determination submodule is used to determine the first required storage area according to the target type and the amount of the initial data.

22. The apparatus according to claim 14, further comprising: a judgment module, configured to, in response to obtaining the data to be stored, determine whether the initial container satisfies a second predetermined condition according to a second required storage area of ​​the data to be stored and free space in the basic storage area of ​​the initial container; as well as A second processing module, configured to, in response to detecting that the initial container satisfies the second predetermined condition, perform a predetermined operation on the initial container to obtain the target container; The predetermined operation includes: Allocate a new storage area for the initial container according to the second required storage area and use it as the extended storage area; taking the data to be stored and the data stored in the basic storage area as the initial data, and storing them in the extended storage area; and The tag is constructed in the base storage area.

23. The device according to claim 22, wherein: The basic storage area includes a high-order basic storage area and a low-order basic storage area; the second processing module includes: A construction submodule is used to construct the mark in the low-order basic storage area and configure the high-order basic storage area to be in an idle state.

24. The apparatus according to claim 14, further comprising: a judgment module, configured to, in response to obtaining the data to be stored, determine whether the initial container satisfies a second predetermined condition according to a second required storage area of ​​the data to be stored and free space in the basic storage area of ​​the initial container; as well as A third storage module is configured to store the data to be stored in the free space in response to detecting that the initial container does not satisfy the second predetermined condition.

25. The device according to claim 24, wherein: The basic storage area includes a high-level basic storage area and a low-level basic storage area; The third storage module includes: A second storage submodule, used for storing the data to be stored into the free space in the low-order basic storage area; as well as The third storage submodule is used to store the attributes of the data to be stored in the high-order basic storage area.

26. The device according to claim 23 or 25, wherein: The second predetermined condition includes one of the following: The capacity of the second demand storage area is greater than the sum of the capacity of the free space in the high-order basic storage area and the capacity of the free space in the low-order basic storage area; as well as The capacity of the second demand storage area is greater than the capacity of the free space in the free lower basic storage area.

27. An electronic device comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 13.

28. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1 to 13.

29. A computer program product comprising a computer program which, when executed by a processor, implements the method according to any one of claims 1 to 13.

Images