Method and apparatus for data compression management

Abstract

The application discloses a data compression management method and device, and relates to the technical field of computers. A specific implementation of the method comprises the following steps: obtaining a merging identifier of to-be-compressed data; judging whether the merging identifier exists in a compressed data list; the compressed data list comprises a merging identifier and a scalar array; if yes, determining a data scalar of the to-be-compressed data according to a data value of the to-be-compressed data and the scalar array, and writing the data scalar of the to-be-compressed data into the scalar array; otherwise, generating a new compressed data list based on the to-be-compressed data. The implementation can aggregate to-be-compressed data according to a merging identifier, overcomes the technical problem that a conventional compression carryover method cannot be used for querying, and achieves the technical effect of querying data according to a merging identifier of the data.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a method and device for data compression management. Background Art

[0002] Conventional compression transfer packages files into a compressed package. This is a new file format that takes up less space, but the contents of the package are altered and cannot be used for search. For example, if 20 articles are compressed into one package, you cannot search for the package by article content, nor can you search for the articles within the package by article content. Summary of the Invention

[0003] In view of this, an embodiment of the present invention provides a method and device for data compression management, which aggregates compressed data according to a merge identifier, can overcome the technical problem that conventional compression carry-over methods cannot be used for queries, and achieve the technical effect of querying data according to the merge identifier of the data.

[0004] To achieve the above object, according to one aspect of an embodiment of the present invention, a method for data compression management is provided, comprising:

[0005] Get the merge identifier of the data to be compressed;

[0006] Determine whether the merge identifier exists in a compressed data list; the compressed data list includes: a merge identifier and a scalar array;

[0007] If so, determine the data scalar of the data to be compressed according to the data value of the data to be compressed and the scalar array, and write the data scalar of the data to be compressed into the scalar array; otherwise, generate a new compressed data list based on the data to be compressed.

[0008] Optionally, the compressed data list further includes: a time offset array; and the method of the embodiment of the present invention further includes:

[0009] If the merge identifier exists in the compressed data list, the time scalar of the data to be compressed is determined according to the occurrence time of the data to be compressed and the time offset array, and the time scalar of the data to be compressed is written into the time offset array.

[0010] Optionally, the compressed data list further includes: a span record; and the method of the embodiment of the present invention further includes:

[0011] If the merge identifier exists in the compressed data list, determining the span index of the data to be compressed according to the unique identifier of the data to be compressed and the span record;

[0012] Writing the unique identifier of the data to be compressed into the span record according to the span index;

[0013] The writing position of the data scalar of the data to be compressed in the scalar array and the writing position of the time scalar of the data to be compressed in the time offset array are determined according to the span index.

[0014] Optionally, the method of the embodiment of the present invention further includes:

[0015] In response to a data query request, obtaining a merge identifier of the data to be queried;

[0016] Obtain a compressed data list corresponding to the merged label of the data to be queried;

[0017] Determine a span index of the data to be queried in the compressed data list according to the unique identifier of the data to be queried and the span record of the compressed data list;

[0018] Restoring the occurrence time of the data to be queried according to the span index and the time offset array of the compressed data list, and restoring the data value of the data to be queried according to the span index and the scalar array of the data to be queried, to obtain the data to be queried;

[0019] Return the data to be queried.

[0020] Optionally, the combined identifier is a user PIN, and the unique identifier is a data ID.

[0021] Optionally, the method of the embodiment of the present invention further includes: before obtaining the merge identifier of the data to be compressed, determining an extraction method of the data to be compressed according to the data volume of the data to be compressed.

[0022] According to another aspect of an embodiment of the present invention, a device for data compression management is provided, comprising:

[0023] An acquisition module obtains a merge identifier of the data to be compressed;

[0024] A judging module, for judging whether the merge identifier exists in a compressed data list; the compressed data list includes: a merge identifier and a scalar array;

[0025] The compression module determines the data scalar of the data to be compressed according to the data value of the data to be compressed and the scalar array when the merge identifier exists in the compressed data list, and writes the data scalar of the data to be compressed into the scalar array; when the merge identifier does not exist in the compressed data list, generates a new compressed data list based on the data to be compressed.

[0026] Optionally, the compressed data list further includes: a time offset array; and the compression module is further configured to:

[0027] If the merge identifier exists in the compressed data list, the time scalar of the data to be compressed is determined according to the occurrence time of the data to be compressed and the time offset array, and the time scalar of the data to be compressed is written into the time offset array.

[0028] Optionally, the compressed data list further includes: a span record; and the compression module is further configured to:

[0029] If the merge identifier exists in the compressed data list, determining the span index of the data to be compressed according to the unique identifier of the data to be compressed and the span record;

[0030] Writing the unique identifier of the data to be compressed into the span record according to the span index;

[0031] The writing position of the data scalar of the data to be compressed in the scalar array and the writing position of the time scalar of the data to be compressed in the time offset array are determined according to the span index.

[0032] Optionally, the apparatus according to the embodiment of the present invention further includes a query module, configured to:

[0033] In response to a data query request, obtaining a merge identifier of the data to be queried;

[0034] Obtain a compressed data list corresponding to the merged label of the data to be queried;

[0035] Determine a span index of the data to be queried in the compressed data list according to the unique identifier of the data to be queried and the span record of the compressed data list;

[0036] Restoring the occurrence time of the data to be queried according to the span index and the time offset array of the compressed data list, and restoring the data value of the data to be queried according to the span index and the scalar array of the data to be queried, to obtain the data to be queried;

[0037] Return the data to be queried.

[0038] Optionally, the combined identifier is a user PIN, and the unique identifier is a data ID.

[0039] Optionally, the acquisition module is further configured to: before acquiring the merge identifier of the data to be compressed, determine an extraction method for the data to be compressed according to the data volume of the data to be compressed.

[0040] According to another aspect of an embodiment of the present invention, there is provided an electronic device for data compression management, comprising:

[0041] one or more processors;

[0042] a storage device for storing one or more programs,

[0043] When the one or more programs are executed by the one or more processors, the one or more processors implement the data compression management method provided by the present invention.

[0044] According to another aspect of an embodiment of the present invention, a computer-readable medium is provided, on which a computer program is stored. When the program is executed by a processor, the data compression management method provided by the present invention is implemented.

[0045] One embodiment of the above invention has the following advantages or beneficial effects: by adopting the technical means of writing the data to be compressed into the compressed data list when the merge identifier of the data to be compressed exists in the compressed data list, and generating a new compressed data list when the merge identifier of the data to be compressed does not exist in the compressed data list, the data to be compressed can be aggregated and compressed according to the merge identifier, overcoming the technical problem that the conventional compression carry-over method cannot be used for query, and achieving the technical effect of data query based on the merge identifier of the data.

[0046] The further effects of the above-mentioned non-conventional optional manner will be described below in conjunction with specific embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] The accompanying drawings are provided for a better understanding of the present invention and are not intended to limit the present invention.

[0048] Figure 1 is an exemplary system architecture diagram suitable for use in a method or apparatus for data compression management according to an embodiment of the present invention;

[0049] Figure 2 Schematic diagram of the main process of the data compression management method according to an embodiment of the present invention;

[0050] Figure 3 is a schematic diagram of a data compression process in an optional embodiment of the present invention;

[0051] Figure 4 1 is a flow chart of data transfer in an optional embodiment of the present invention;

[0052] Figure 5 1 is a flow chart of data query in an optional embodiment of the present invention;

[0053] Figure 6is a schematic diagram of main modules of a data compression management apparatus according to an embodiment of the present invention;

[0054] Figure 7 It is a schematic diagram of the structure of a computer system of a terminal device or a server suitable for implementing an embodiment of the present invention. DETAILED DESCRIPTION

[0055] The following description of exemplary embodiments of the present invention is made in conjunction with the accompanying drawings, in which various details of the embodiments of the present invention are included to facilitate understanding. These details should be considered as merely exemplary. Therefore, it should be appreciated by those skilled in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present invention. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0056] Figure 1 FIG. 1 shows an exemplary system architecture diagram of a method or apparatus for data compression management suitable for use in an embodiment of the present invention, such as Figure 1 As shown, an exemplary system architecture of the data compression management method or data compression management apparatus according to an embodiment of the present invention includes:

[0057] like Figure 1 As shown, system architecture 100 may include terminal devices 101, 102, 103, a network 104, and a server 105. Network 104 is a medium for providing communication links between terminal devices 101, 102, 103 and server 105. Network 104 may include various connection types, such as wired or wireless communication links or fiber optic cables.

[0058] Users can use terminal devices 101, 102, and 103 to interact with server 105 via network 104 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 101, 102, and 103, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social platform software, etc.

[0059] The terminal devices 101 , 102 , and 103 may be various electronic devices having a display screen and supporting web browsing, including but not limited to smart phones, tablet computers, laptop computers, and desktop computers.

[0060] Server 105 may be a server that provides various services, such as a backend management server that supports shopping websites browsed by users using terminal devices 101, 102, and 103. The backend management server may analyze and process received data such as product information query requests, and feed back processing results (such as target push information and product information—for example only) to terminal devices 101, 102, and 103.

[0061] It should be noted that the data compression management method provided in the embodiment of the present invention is generally executed by the server 105 , and accordingly, the data compression management device is generally set in the server 105 .

[0062] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is merely illustrative. Any number of terminal devices, networks and servers may be provided as required.

[0063] Figure 2 FIG. 1 is a schematic diagram of the main process of the method for data compression management according to an embodiment of the present invention. Figure 2 As shown, the data compression management method includes step S201, step S202 and step S203.

[0064] Step S201: Obtain a merge identifier of the data to be compressed.

[0065] The merge identifier is information common to multiple pieces of data, and information can be aggregated based on the merge identifier. The content of the merge identifier can be selectively set according to the actual situation. For example, when aggregating based on the user dimension, the user's PIN (Personal Identification Number) is used as the merge identifier; when aggregating based on the terminal device dimension, the terminal device's IMEI (International Mobile Equipment Identity) is used as the merge identifier; when aggregating based on the service dimension, the service type is used as the merge identifier.

[0066] Step S202: Determine whether the merge identifier exists in the compressed data list. The compressed data list includes the merge identifier and a scalar array. If so, the process proceeds to step S203, where the data scalar of the data to be compressed is determined based on the data value of the data to be compressed and the scalar array, and the data scalar of the data to be compressed is written into the scalar array. Otherwise, the process proceeds to step S204, where a new compressed data list is generated based on the data to be compressed.

[0067] In this step, the merge identifier is used to determine whether a new compressed data needs to be generated. If the merge identifier of the data to be compressed exists in the compressed data list, the data to be compressed is aggregated into the compressed data list. If the merge identifier of the data to be compressed does not exist in the compressed data list, a new compressed data list corresponding to the merge identifier of the data to be compressed is generated.

[0068] The conventional compression process involves recoding the file contents and storing them in a new file type (e.g., zip or rar). This storage method offers a high compression ratio, but because the file type changes, it cannot be used for queries. The compression principle is to search for duplicates character by character and record the locations of duplicate characters. For example, the sentence "The weather is good today" would be broken down into "today (0), day (1, 2), weather (3), not (4, 6), wrong (5, 7). By breaking down the coherent content into individual characters, it loses its business meaning and becomes simply computer characters, making it impossible to query.

[0069] In actual application, the data value of the first data to be compressed in the compressed data list can be used as the standard value, and the difference between the data value of the subsequent data to be compressed and the standard value can be written as the data scalar of the compressed data into the scalar array. For example, data 1 is: PIN = abc, data value value = 10; data 2 is: PIN = abc, data value value = 12; data 3 is: PIN = abc, data value value = 14. After compression, the compressed data list is: PIN = abc, standard value standardValue = 10, and the scalar array compressScalar = [2,4]

[0070] Of course, you can also use other values ​​as the standard value, and write the difference between the data value of each data to be compressed and the standard value as the data scalar of the compressed data into the scalar array. For example, data 1 is: PIN = abc, data value = 10; data 2 is: PIN = abc, data value = 12; data 3 is: PIN = abc, data value = 14. After compression, the compressed data list is: PIN = abc, standard value standardValue = 8, and the scalar array compressScalar = [2, 4, 6].

[0071] The compression process of the embodiment of the present invention does not change the file type. It is based on the business content, sets the content with the highest business differentiation as the merge identifier, and merges the data with the same merge identifier into one record, thereby retaining the meaning of the data itself. Since the file type and encoding remain unchanged, data query can be performed. By adopting the technical means of writing the data to be compressed into the compressed data list when the merge identifier of the data to be compressed exists in the compressed data list, and generating a new compressed data list when the merge identifier of the data to be compressed does not exist in the compressed data list, the data to be compressed can be aggregated and compressed according to the merge identifier, overcoming the technical problem that the conventional compression carry-over method cannot be used for query, and achieving the technical effect of data query based on the merge identifier of the data.

[0072] The compressed data list may further include a time offset array. If the merge identifier exists in the compressed data list, a time scalar of the data to be compressed is determined based on the occurrence time of the data to be compressed and the time offset array, and the time scalar of the data to be compressed is written into the time offset array.

[0073] In practice, the occurrence time of the first data item to be compressed in the compressed data list can be used as the start time, and the difference between the occurrence time of subsequent data items to be compressed and the start time can be written as the time scalar of the compressed data into the time offset array. For example, data 1 is: PIN = abc, start time time = 123456, data value value = 10; data 2 is: PIN = abc, start time time = 123458, data value value = 12; and data 3 is: PIN = abc, start time time = 123459, data value value = 14. After compression, the compressed data list is: PIN = abc, start time startTime = 123456, time offset array compressTime = [2, 3], standard value standardValue = 10, and scalar array compressScalar = [2, 4].

[0074] Of course, you can also use other values ​​as the start time, and write the difference between the occurrence time of each data point to be compressed and the start time as the time scalar of the compressed data into the time offset array. For example, data 1 is: PIN = abc, start time time = 123456, data value value = 10; data 2 is: PIN = abc, start time time = 123458, data value value = 12; data 3 is: PIN = abc, start time time = 123459, data value value = 14. After compression, the compressed data list is: PIN = abc, start time startTime = 123455, time offset array compressTime = [1, 3, 4], standard value standardValue = 8, and scalar array compressScalar = [2, 4, 6].

[0075] By setting the time offset array, the occurrence time of the data to be compressed can be recorded for easy query.

[0076] The compressed data list may also include: a span record. If the merge identifier exists in the compressed data list, the span index of the data to be compressed is determined based on the unique identifier of the data to be compressed and the span record; the unique identifier of the data to be compressed is written into the span record based on the span index; the write position of the data scalar of the data to be compressed in the scalar array and the write position of the time scalar of the data to be compressed in the time offset array are determined based on the span index. The unique identifier is used to uniquely represent a piece of data, such as a data ID (Identity document, also known as an identity identification number). The span index is used to represent the write position of the content to be written in the array. For example, if the unique identifier is in the first position, index = 0; if the unique identifier is in the second position, index = 1, and so on. According to the position of the unique identifier in the span record, the span index index is calculated, so that the subsequent records to be merged are written to the same offset position index as the unique identifier, so as to facilitate data restoration during query.

[0077] Optionally, the method of an embodiment of the present invention further includes: before obtaining the merge identifier of the data to be compressed, determining a method for extracting the data to be compressed, such as parallel or serial extraction, based on the amount of the data to be compressed. If the amount of compressed data is small, a serial operation can be used to write the compressed data to a storage medium. Depending on the target storage medium, the data can be saved to a database or a file.

[0078] Figure 3 FIG. 1 is a flow chart of data compression in an optional embodiment of the present invention. Figure 3 As shown, the compression process includes:

[0079] Loop through each piece of data to be compressed in the set of data to be compressed, and determine whether each piece of data to be compressed exists in the compressed data list by merging the identifiers;

[0080] If it does not exist, a new compressed data is generated, with the time of the data to be compressed as the starting time and the data value of the data to be compressed as the standard value. The record content in the new compressed data list is the same as the content of the data to be compressed. The difference is that the field names of the various contents in the data to be compressed are changed to the corresponding field names in the compressed data list. Taking the above example, the field name time in the compressed data is changed to startTime, and value is changed to standardtValue. The rest follow this principle, adding the start prefix to the time type and the standard prefix to the value type. After the new record is merged into the compressed data list, the merge identifier remains unchanged, and other non-merge identifier fields are integrated. Fields with the compress prefix are added, such as the compressTime and compressValue fields, to identify the offset.

[0081] If it exists, the compression span is marked with the data unique identifier, the time offset is recorded according to the span index, and the time scalar (the difference between the occurrence time and the start time of this piece of data to be compressed) is written to the index position of the time offset array; the scalar is replaced according to the span index, and the data scalar (the difference between the data value of this piece of data to be compressed and the standard value) is written to the index position of the scalar array.

[0082] After processing is complete, the compressed data is output.

[0083] Optionally, the method of an embodiment of the present invention further includes: obtaining a merge identifier of the data to be queried in response to a data query request; obtaining a compressed data list corresponding to the merge identifier of the data to be queried; determining the span index of the data to be queried in the compressed data list based on the unique identifier of the data to be queried and the span record of the compressed data list; restoring the occurrence time of the data to be queried based on the span index and the time offset array of the compressed data list, and restoring the data value of the data to be queried based on the span index and the scalar array of the data to be queried, to obtain the data to be queried; and returning the data to be queried.

[0084] Figure 5 : is a flow chart of data query in an optional embodiment of the present invention. Figure 5As shown, the first step is to determine whether the user's unique identifier (e.g., PIN) matches the compressed data. If not, the data does not exist and is returned directly to the front-end. If it does, the data is restored and then returned to the front-end. The data restoration process includes: extracting the compressed data that matches the original data's unique identifier by span; using the index to match the original data's unique identifier in the compressed data; finding the time offset according to the index and restoring the data's occurrence time; finding the corresponding scalar according to the index, performing scalar replacement, and restoring the data value; and returning the restored data to the front-end.

[0085] Carryover is the process of extracting less frequently used portions of business data and saving them to other storage media, thereby reducing the size of hot business data and improving its response speed. Existing technologies for full carryover often extract the entire data in serial or parallel order and write it to historical tables, big data clusters, or other storage media. Due to the rapid growth of log flow and the expansion of existing business operations, saving the entire historical data consumes a significant amount of storage. Figure 4 : is a flow chart of data transfer in an optional embodiment of the present invention, such as Figure 4 As shown, the data rollover process includes extracting data in serial or parallel mode, depending on the data volume, inputting it into a compression service for compression, and then outputting it to the compression service through steps such as merging identical information, calculating time offsets, replacing scalars, and marking compressed spans. The compressed data is then stored and output to the compression service. Using the data compression management method of the present invention for data rollover can solve the problem of excessive storage space occupied by full rollover. The compression service compresses logs, merging multiple logs into a single log, thereby reducing storage space usage. The compression ratio is the ratio of the compressed file size to the uncompressed file size. For example, if a 100MB file is compressed to 5MB, the compression ratio is 5 / 100*100% = 5%. Generally, the lower the compression ratio, the better. Online experimental results using the method of the present invention for data compression show that for structurally similar consumer flow data, the compression ratio is below 5%, while for structurally dissimilar activity data, the average compression ratio is 25%. Assuming a PB (a digital unit of capacity) resource cost of 1 million yuan, each PB can save 900,000 yuan in storage costs.

[0086] Using the data compression management method of the present invention to compress data solves the problem of conventional compression methods being unable to query. Based on identical information merging, during the initial query, a compressed data item (representing multiple records) is hit based on the same information column (e.g., user PIN), and the results are then filtered a second time. During the secondary filtering, the original position index in the compressed data is located through span matching, the data occurrence time is restored through time offset, and the data value is restored through scalar replacement, thereby restoring the data.

[0087] According to another aspect of the embodiments of the present invention, a device for implementing the above method is provided.

[0088] Figure 6 FIG. 1 is a schematic diagram showing a data compression management device according to an embodiment of the present invention. Figure 6 As shown, the data compression management device 600 includes:

[0089] An acquisition module 601 acquires a merge identifier of the data to be compressed;

[0090] The judging module 602 judges whether the merge identifier exists in the compressed data list; the compressed data list includes: the merge identifier and the scalar array;

[0091] Compression module 603, when the merge identifier exists in the compressed data list, determines the data scalar of the data to be compressed according to the data value of the data to be compressed and the scalar array, and writes the data scalar of the data to be compressed into the scalar array; when the merge identifier does not exist in the compressed data list, generates a new compressed data list based on the data to be compressed.

[0092] Optionally, the compressed data list further includes: a time offset array; and the compression module is further configured to:

[0093] If the merge identifier exists in the compressed data list, the time scalar of the data to be compressed is determined according to the occurrence time of the data to be compressed and the time offset array, and the time scalar of the data to be compressed is written into the time offset array.

[0094] Optionally, the compressed data list further includes: a span record; and the compression module is further configured to:

[0095] If the merge identifier exists in the compressed data list, determining the span index of the data to be compressed according to the unique identifier of the data to be compressed and the span record;

[0096] Writing the unique identifier of the data to be compressed into the span record according to the span index;

[0097] The writing position of the data scalar of the data to be compressed in the scalar array and the writing position of the time scalar of the data to be compressed in the time offset array are determined according to the span index.

[0098] Optionally, the apparatus according to the embodiment of the present invention further includes a query module, configured to:

[0099] In response to a data query request, obtaining a merge identifier of the data to be queried;

[0100] Obtain a compressed data list corresponding to the merged label of the data to be queried;

[0101] Determine a span index of the data to be queried in the compressed data list according to the unique identifier of the data to be queried and the span record of the compressed data list;

[0102] Restoring the occurrence time of the data to be queried according to the span index and the time offset array of the compressed data list, and restoring the data value of the data to be queried according to the span index and the scalar array of the data to be queried, to obtain the data to be queried;

[0103] Return the data to be queried.

[0104] Optionally, the combined identifier is a user PIN, and the unique identifier is a data ID.

[0105] Optionally, the acquisition module is further configured to: before acquiring the merge identifier of the data to be compressed, determine an extraction method for the data to be compressed according to the data volume of the data to be compressed.

[0106] Figure 7 FIG. 1 is a schematic diagram of a computer system suitable for implementing a terminal device according to an embodiment of the present invention. Figure 7 As shown, the computer system 700 of the terminal device according to the embodiment of the present invention includes:

[0107] The system 700 includes a central processing unit (CPU) 701, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 702 or a program loaded from a storage unit 708 into a random access memory (RAM) 703. Various programs and data required for the operation of the system 700 are also stored in the RAM 703. The CPU 701, ROM 702, and RAM 703 are connected to each other via a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704.

[0108] The following components are connected to the I / O interface 705: an input section 706 including a keyboard, a mouse, and the like; an output section 707 including devices such as a cathode ray tube (CRT), a liquid crystal display (LCD), and a speaker; a storage section 708 including a hard disk; and a communication section 709 including a network interface card such as a LAN card or a modem. The communication section 709 performs communication processing via a network such as the Internet. A drive 710 is also connected to the I / O interface 705 as needed. A removable medium 711, such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory, is installed in the drive 710 as needed, so that computer programs read therefrom can be installed into the storage section 708 as needed.

[0109] In particular, according to the embodiments disclosed in the present invention, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, the embodiments disclosed in the present invention include a computer program product comprising a computer program carried on a computer-readable medium, the computer program comprising program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network via the communication section 709, and / or installed from a removable medium 711. When the computer program is executed by the central processing unit (CPU) 701, the above-mentioned functions defined in the system of the present invention are executed.

[0110] It should be noted that the computer-readable medium described in the present invention can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of computer-readable storage media can include, but are not limited to, an electrical connection having one or more conductors, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present invention, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In the present invention, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. This propagated data signal can take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium that can transmit, propagate, or transport a program for use by or in conjunction with an instruction execution system, apparatus, or device. Program code embodied on a computer-readable medium may be transmitted using any suitable medium, including but not limited to wireless, wireline, optical fiber cable, RF, or any suitable combination thereof.

[0111] The flowcharts and block diagrams in the accompanying drawings illustrate the possible implementation architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present invention. In this regard, each box in the flowchart or block diagram can represent a module, program segment, or a part of code, and the above-mentioned module, program segment, or a part of code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in an order different from that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flowchart, and the combination of boxes in the block diagram or flowchart, can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0112] The modules described in the embodiments of the present invention may be implemented in software or hardware. The modules described may also be provided in a processor. For example, they may be described as comprising an acquisition module, a determination module, and a compression module. The names of these modules do not, in some cases, limit the modules themselves. For example, the determination module may also be described as a module that writes the data to be compressed into a list of compressed data.

[0113] As another aspect, the present invention further provides a computer-readable medium, which may be included in the device described in the above embodiment; or may exist independently and not be assembled into the device. The above computer-readable medium carries one or more programs, and when the above one or more programs are executed by a device, the device includes: obtaining a merge identifier of the data to be compressed; determining whether the merge identifier exists in a compressed data list; the compressed data list includes: a merge identifier and a scalar array; if so, determining the data scalar of the data to be compressed based on the data value of the data to be compressed and the scalar array, and writing the data scalar of the data to be compressed into the scalar array; otherwise, generating a new compressed data list based on the data to be compressed.

[0114] According to the technical solution of an embodiment of the present invention, by adopting a technical means of writing the data to be compressed into the compressed data list when the merge identifier of the data to be compressed exists in the compressed data list, and generating a new compressed data list when the merge identifier of the data to be compressed does not exist in the compressed data list, the data to be compressed can be aggregated and compressed according to the merge identifier, overcoming the technical problem that the conventional compression carry-forward method cannot be used for query, and achieving the technical effect of data query according to the merge identifier of the data.

[0115] The above specific embodiments do not limit the scope of protection of the present invention. Those skilled in the art will appreciate that various modifications, combinations, sub-combinations, and substitutions may occur depending on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention are intended to be included within the scope of protection of the present invention.

Claims

1. A method for data compression management, characterized in that: include: Obtaining a merge identifier of the data to be compressed, wherein the merge identifier is common information of multiple pieces of data; Determine whether the merge identifier exists in the compressed data list; The compressed data list includes: a merge identifier and a scalar array; If so, determine the data scalar of the data to be compressed based on the data value of the data to be compressed and the scalar array, and write the data scalar of the data to be compressed into the scalar array to aggregate the data to be compressed into the compressed data list; otherwise, generate a new compressed data list based on the data to be compressed.

2. The method according to claim 1, wherein The compressed data list further includes: a time offset array; the method further includes: If the merge identifier exists in the compressed data list, the time scalar of the data to be compressed is determined according to the occurrence time of the data to be compressed and the time offset array, and the time scalar of the data to be compressed is written into the time offset array.

3. The method according to claim 2, wherein The compressed data list further includes: a span record; the method further includes: If the merge identifier exists in the compressed data list, determining the span index of the data to be compressed according to the unique identifier of the data to be compressed and the span record; Writing the unique identifier of the data to be compressed into the span record according to the span index; The writing position of the data scalar of the data to be compressed in the scalar array and the writing position of the time scalar of the data to be compressed in the time offset array are determined according to the span index.

4. The method according to claim 3, wherein Also includes: In response to a data query request, obtaining a merge identifier of the data to be queried; Obtain a compressed data list corresponding to the merge identifier of the data to be queried; Determine a span index of the data to be queried in the compressed data list according to the unique identifier of the data to be queried and the span record of the compressed data list; Restoring the occurrence time of the data to be queried according to the span index and the time offset array of the compressed data list, and restoring the data value of the data to be queried according to the span index and the scalar array of the data to be queried, to obtain the data to be queried; Return the data to be queried.

5. The method according to any one of claims 1 to 4, characterized in that: Also includes: Before obtaining the merge identifier of the data to be compressed, a method for extracting the data to be compressed is determined according to the data volume of the data to be compressed.

6. A data compression management device, characterized in that: include: An acquisition module, which acquires a merge identifier of the data to be compressed, wherein the merge identifier is common information of multiple data; A judging module, for judging whether the merge identifier exists in the compressed data list; The compressed data list includes: a merge identifier and a scalar array; A compression module, when the merge identifier exists in the compressed data list, determines the data scalar of the data to be compressed based on the data value of the data to be compressed and the scalar array, writes the data scalar of the data to be compressed into the scalar array to aggregate the data to be compressed into the compressed data list; when the merge identifier does not exist in the compressed data list, generates a new compressed data list based on the data to be compressed.

7. The device according to claim 6, characterized in that The compressed data list further includes a time offset array; the compression module is further configured to: If the merge identifier exists in the compressed data list, the time scalar of the data to be compressed is determined according to the occurrence time of the data to be compressed and the time offset array, and the time scalar of the data to be compressed is written into the time offset array.

8. The device according to claim 7, wherein The compressed data list further includes: a span record; and the compression module is further configured to: If the merge identifier exists in the compressed data list, determining the span index of the data to be compressed according to the unique identifier of the data to be compressed and the span record; Writing the unique identifier of the data to be compressed into the span record according to the span index; The writing position of the data scalar of the data to be compressed in the scalar array and the writing position of the time scalar of the data to be compressed in the time offset array are determined according to the span index.

9. An electronic device for data compression management, characterized in that: include: one or more processors; a storage device for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 5.

10. A computer-readable medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 5 is implemented.

Images