Data compression method, apparatus, and electronic device

Abstract

The application discloses a data compression method and device and electronic equipment. The method comprises the following steps: acquiring to-be-compressed data, and acquiring a first proportion of different types of data in the to-be-compressed data; selecting a target compression scheme from a plurality of preset compression schemes according to the first proportion; and compressing the to-be-compressed data by using the target compression scheme to achieve a target compression ratio. The method can reduce workload.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data compression, and in particular to a data compression method, device and electronic equipment. BACKGROUND

[0002] Electricity information collection is to collect the electricity information of power users, and to realize functions such as metering anomaly monitoring, power quality monitoring, electricity analysis and management by storing, analyzing and calculating the collected data. The data increment is huge, in order to reasonably use resources and reduce cost and increase efficiency, data compression technology is usually used when storing. However, the data compression technology in the related art does not have a special method for electricity information collection data, and the best compression ratio scheme needs to be adjusted repeatedly, which is time-consuming and costly, and the intelligence level is low. SUMMARY

[0003] The present application aims to at least solve one of the technical problems in the related art. To this end, the first object of the present application is to provide a data compression method to reduce the workload.

[0004] The second object of the present application is to provide an electronic equipment.

[0005] The third object of the present application is to provide a data compression device.

[0006] To achieve the above-mentioned objects, the first aspect of the present application provides a data compression method, which comprises: obtaining to-be-compressed data, and obtaining a first proportion of different types of data in the to-be-compressed data; selecting a target compression scheme from a plurality of preset compression schemes according to the first proportion; and compressing the to-be-compressed data by using the target compression scheme to achieve a target compression ratio.

[0007] To achieve the above-mentioned objects, the second aspect of the present application provides an electronic equipment, which comprises a memory, a processor and a computer program stored in the memory and executable on the processor, and when the computer program is executed by the processor, the data compression method described above is realized.

[0008] To achieve the above-mentioned objects, the third aspect of the present application provides a data compression device, which comprises: an acquisition module for acquiring to-be-compressed data and a first proportion of different types of data in the to-be-compressed data; a selection module for selecting a target compression scheme from a plurality of preset compression schemes according to the first proportion; and a compression module for compressing the to-be-compressed data by using the target compression scheme to achieve a target compression ratio.

[0009] The data compression method, device and electronic equipment of the embodiment of the present application, by acquiring the to-be-compressed data, acquiring the first proportion of different types of data in the to-be-compressed data, selecting the target compression scheme from the plurality of preset compression schemes according to the first proportion, and compressing the to-be-compressed data by using the target compression scheme to achieve the target compression ratio, thereby, the best compression scheme can be automatically adapted when data compression is needed, and the workload of repeated debugging is reduced.

[0010] Additional aspects and advantages of the present application will be described in the following description, will become apparent from the following description, or will be learned by practice of the present application. BRIEF DESCRIPTION OF DRAWINGS

[0011] Figure 1 is a flowchart of a data compression method of an embodiment of the present application;

[0012] Figure 2 is a flowchart of a data compression method of an embodiment of the present application;

[0013] Figure 3 is a structural block diagram of a data compression device of an embodiment of the present application. DETAILED DESCRIPTION

[0014] The data compression method, device and electronic equipment of the embodiment of the present application are described below with reference to the accompanying drawings, wherein the same or similar reference numerals represent the same or similar elements or elements having the same or similar functions throughout. The embodiments described with reference to the accompanying drawings are exemplary and should not be understood as limiting the present application.

[0015] Figure 1 is a flowchart of a data compression method of an embodiment of the present application.

[0016] As shown in Figure 1 , the data compression method comprises:

[0017] S11, acquiring the to-be-compressed data, and acquiring the first proportion of different types of data in the to-be-compressed data.

[0018] Specifically, first, it is judged whether the target compression scheme needs to be determined, for example, in the case of needing to create a new data table in the HBase database, needing to modify the compression attribute, needing to compress data, etc., it is judged that the target compression scheme needs to be determined.

[0019] When the target compression scheme needs to be determined, the to-be-compressed data is acquired, and it is judged whether the to-be-compressed data is specified by a data type, if the to-be-compressed data has been specified by a data type, the first proportion is determined according to the specified data type.

[0020] Specifically, it can be judged whether the table structure description information Schema of the to-be-compressed data can be acquired. If the table structure description information Schema of the to-be-compressed data can be acquired, it is determined that the to-be-compressed data has been specified with a data type, and thus, if the table structure description information Schema can be acquired, the first proportion is obtained according to the table structure description information Schema.

[0021] However, if the table structure description information Schema cannot be acquired, it is indicated that the to-be-compressed data is not specified with a data type, and at this time, fuzzy query needs to be performed according to the to-be-compressed data and the preset N information collection modeling models, where N is a positive integer.

[0022] Among the above information collection modeling models, first, N modeling data tables need to be acquired. The modeling data table is a modeling table, which is a table obtained according to an industry-unified model design, and includes a table name and a description file. The description file includes modeling data and also includes annotation instructions for the modeling data, so that the modeling data table stores the values and meanings of the fields in the modeling data, and thus, the corresponding second proportion can be obtained according to the annotation instructions. For example, if the to-be-compressed data is power consumption information data, the modeling data is pre-acquired power consumption information data, and then the second proportion corresponding to the modeling data table can be determined according to the types of the fields in the modeling data.

[0023] For each modeling data table, a tokenizer is used for splitting to obtain the corresponding topic keyword and second proportion. The tokenizer is an IK tokenizer. Through the IK tokenizer, the table name and the description file of each modeling data table are split. The table name is split to obtain the topic keyword, the description file is split, and the split result is processed to obtain the corresponding second proportion. The topic keyword and the proportion of different types of data corresponding thereto are recorded, and the proportion of the different types of data is taken as the second proportion.

[0024] After obtaining the topic keywords and the second proportions corresponding to the N modeling data tables, the N information collection modeling models can be obtained according to the N modeling data tables and the corresponding topic keywords and second proportions.

[0025] It should be noted that after obtaining the N information collection modeling models, the N information collection modeling models can also be persisted, and saved to a preset position by a tokenizer, such as being saved to a search server ElasticSearch.

[0026] Thus, after judging that the table structure description information Schema cannot be acquired, a fuzzy query is performed through the theme keyword. Specifically, it is judged whether the theme keyword of the to-be-compressed data can be acquired; if the theme keyword can be acquired, the to-be-compressed data is matched with the pre-set information collection and setting model according to the theme keyword; if the matching succeeds, the second proportion of different types of data in the information collection and setting model is acquired, and the first proportion is assigned as the second proportion.

[0027] Further, if the theme keyword of the to-be-compressed data cannot be acquired, or the matching fails, the to-be-compressed data is randomly sampled to obtain sample data; a third proportion of different types of data in the sample data is acquired; and the first proportion is assigned as the third proportion.

[0028] S12, a target compression scheme is selected from the plurality of pre-set compression schemes according to the first proportion.

[0029] S13, the to-be-compressed data is compressed by using the target compression scheme to achieve a target compression ratio.

[0030] The embodiments of the present application will be described in detail below with reference to the specific examples shown in the accompanying drawings. Figure 2

[0031] In the specific example, the first proportion of different types of data includes the proportion of numerical data to the to-be-compressed data, the proportion of character data to the to-be-compressed data, and the proportion of complex type data to the to-be-compressed data. The plurality of pre-set compression schemes include ZSTD compression algorithm, LZ4 compression algorithm, Snappy compression algorithm, and GZ compression algorithm.

[0032] Specifically, after the power consumption information related data is collected, when a table needs to be built in the HBase or data compression needs to be performed, it is first judged whether the table structure description information Schema of the collected power consumption information related data can be acquired. If the table structure description information Schema can be acquired, the acquired table structure description information Schema is analyzed to obtain the first proportion, so as to judge the relationship between the numerical data, the character data, the complex data, and other types of data according to the first proportion.

[0033] If the table structure description information Schema cannot be acquired, a fuzzy query is performed according to the collected power consumption information related data and the power consumption information collection and setting model, that is, it is judged whether the theme keyword in the collected power consumption information related data can be acquired. If the theme keyword can be acquired, a fuzzy query is performed through the IK word segmenter according to the acquired theme keyword and the power consumption information collection and setting model. If the modeling data table corresponding to the power consumption information related data is queried, the second proportion corresponding to the modeling data table is taken as the first proportion, so as to select the target compression scheme according to the first proportion. ​

[0034] If the subject keyword in the power consumption information related data cannot be acquired, or although the subject keyword can be acquired, the fuzzy query according to the acquired subject keyword and the power consumption information collection and setting model fails, the power consumption information related data is sampled to obtain a sample, after the sample is obtained, the sample data is analyzed to obtain a third proportion of different types of data in the sample data, and the third proportion is taken as the first proportion.

[0035] Further, according to the first proportion, the relationship among the numerical type data, the character type data, the complex type data and the other type data is obtained. If the proportion of the numerical type data is the largest, the ZSTD compression algorithm is selected as the target compression scheme; if the proportion of the character type data is the largest, the LZ4 compression algorithm is selected as the target compression scheme; if the proportion of the complex type data is the largest, the Snappy compression algorithm is selected as the target compression scheme; and if the proportion of the other type data is the largest, the GZ compression algorithm is selected as the target compression scheme.

[0036] In summary, the data compression method in the embodiment of the present application acquires the to-be-compressed data, acquires the first proportion of different types of data in the to-be-compressed data, selects the target compression scheme from the plurality of preset compression schemes according to the first proportion, and compresses the to-be-compressed data by using the target compression scheme to achieve the target compression ratio. Therefore, when data compression is needed, the best compression scheme can be automatically adapted, and the workload of repeated debugging can be reduced. Moreover, when the data compression method in the present application is applied to the HBase database, since the compression scheme is directly modified in the present application, after the compression scheme is modified, compression is performed again, data migration is not needed, and since data migration is not needed, the table for storing data does not need to be replaced, and the HBase database service will not be interrupted. At the same time, since the specific steps of data compression, such as the major_compact merging operation in the HBase database, will not interrupt the HBase database service, the data compression method in the present application will not interrupt the HBase database service, and can perform hot operation, thereby having good practicability.

[0037] Further, the present application provides an electronic device.

[0038] In the embodiment of the present application, the electronic device includes a memory, a processor and a computer program stored in the memory and executable on the processor, and when the computer program is executed by the processor, the data compression method described above is implemented.

[0039] The electronic device of the embodiment of the present application can acquire the to-be-compressed data, acquire the first proportion of different types of data in the to-be-compressed data, select a target compression scheme from a plurality of preset compression schemes according to the first proportion, compress the to-be-compressed data by using the target compression scheme, and achieve a target compression ratio, thereby automatically adapting the best compression scheme when data compression is needed and reducing the workload of repeated debugging.

[0040] Further, the present application provides a data compression device.

[0041] Figure 3 FIG. 1 is a structural block diagram of a data compression device according to an embodiment of the present application.

[0042] As shown in FIG. 1, the data compression device 100 comprises an acquisition module 101, a selection module 102, and a compression module 103. Figure 3

[0043] Specifically, the acquisition module 101 is configured to acquire to-be-compressed data and acquire a first proportion of different types of data in the to-be-compressed data; the selection module 102 is configured to select a target compression scheme from a plurality of preset compression schemes according to the first proportion; and the compression module 103 is configured to compress the to-be-compressed data by using the target compression scheme to achieve a target compression ratio.

[0044] It should be noted that other specific embodiments of the data compression device of the embodiment of the present application can refer to the data compression method described above.

[0045] The data compression device of the embodiment of the present application can acquire the to-be-compressed data, acquire the first proportion of different types of data in the to-be-compressed data, select a target compression scheme from a plurality of preset compression schemes according to the first proportion, compress the to-be-compressed data by using the target compression scheme, and achieve a target compression ratio, thereby automatically adapting the best compression scheme when data compression is needed and reducing the workload of repeated debugging.

[0046] ​It is to be appreciated that the logical and / or steps represented in the flowcharts, or described herein in other manners, can be considered as a list of ordered steps for implementing logical functions, and can be embodied in any computer-readable medium for use by or in connection with an instruction execution system, apparatus, or device, such as a computer-based system, processor- imbedded systems, or other systems that can fetch the instructions from the instruction execution system, apparatus, or device and execute the instructions, or in conjunction with such an instruction execution system, apparatus, or device. For purposes of this specification, a "computer-readable medium" can be any apparatus that can contain, store, communicate, propagate, or transport the program for use by or in connection with the instruction execution system, apparatus, or device. The computer-readable medium can be a more specific example (non-exhaustive list) including the following: an electrical connection having one or more wires (electrical apparatus), a portable computer diskette (magnetic apparatus), a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber (optical apparatus), and a portable compact disc read-only memory (CDROM). In addition, the computer-readable medium can even be paper or another suitable medium upon which the program can be printed, as the program can be electronically captured, for example, by the optical scanning of the paper or other medium, followed by the electronic conversion of the optically scanned program into a computer-readable medium, and then the storage of the computer-readable medium in a computer memory.

[0047] It should be understood that portions of the application can be implemented in hardware, software, firmware, or combinations thereof. In the above embodiments, the various steps or methods can be implemented in software or firmware stored in a memory and executed by a suitable instruction execution system. If implemented in hardware, and in another embodiment, any of the following technologies, or combinations thereof, can be used: discrete logic circuitry having logic gates for implementing logic functions upon data signals, application specific integrated circuits having appropriate combinational logic gates, programmable gate arrays (PGA), field programmable gate arrays (FPGA), and the like.

[0048] In the description of the present specification, the description of the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" and the like means that the specific features, structures, materials or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present application. In the present specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Also, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner.

[0049] In the description of the present application, the terms "center", "longitudinal", "lateral", "length", "width", "thickness", "upper", "lower", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inner", "outer", "clockwise", "counterclockwise", "axial", "radial", "circumferential" and the like indicate the orientation or positional relationship shown in the drawings, and do not indicate or imply that the device or element referred to must have a particular orientation, be constructed and operated in a particular orientation, and cannot be understood as a limitation of the present application.

[0050] In addition, the terms "first", "second" are only for descriptive purposes and cannot be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Therefore, the features defined with "first", "second" can explicitly or implicitly include at least one of the features. In the description of the present application, the meaning of "a plurality of" is at least two, for example, two, three, etc., unless otherwise specifically limited.

[0051] In the description of the present application, unless otherwise specified, the terms "mounting", "connecting", "connecting", "fixing" and the like should be broadly understood, for example, it can be fixedly connected, or it can be detachably connected, or it can be integrated; it can be mechanically connected, or it can be electrically connected; it can be directly connected, or it can be indirectly connected through an intermediate medium; it can be the internal communication of two elements or the interaction relationship between two elements, unless otherwise explicitly limited. For those skilled in the art, the specific meaning of the above terms in the present application can be understood according to the specific circumstances.

[0052] In the present application, unless otherwise specifically provided and limited, the first feature is "on" or "under" the second feature, which can be direct contact between the first and second features, or indirect contact between the first and second features through an intermediate medium. Moreover, the first feature "above", "above" and "above" the second feature can be directly above or obliquely above the first feature, or only indicate that the horizontal height of the first feature is higher than that of the second feature. The first feature "below", "below" and "below" the second feature can be directly below or obliquely below the first feature, or only indicate that the horizontal height of the first feature is less than that of the second feature.

[0053] Although the embodiments of the present application have been shown and described above, it can be understood that the above embodiments are exemplary and cannot be understood as a limitation of the present application, and those skilled in the art can make changes, modifications, replacements and variations to the above embodiments within the scope of the present application.

Claims

1. A data compression method, characterized in that, The method includes: Obtain the data to be compressed, and obtain the first proportion of different types of data in the data to be compressed; Select a target compression scheme from a plurality of preset compression schemes according to the first ratio; The data to be compressed is compressed using the target compression scheme to achieve the target compression ratio; The step of obtaining the first proportion of different types of data in the data to be compressed includes: Determine whether the schema, a table structure description of the data to be compressed, can be obtained; If the table structure description information Schema can be obtained, then the first ratio is obtained based on the table structure description information Schema; If the table structure description information (Schema) cannot be obtained, then obtaining the first proportion of different types of data in the data to be compressed includes: Determine whether the topic keywords of the data to be compressed can be obtained; If the topic keywords can be obtained, the data to be compressed is matched with the pre-set information collection benchmark model based on the topic keywords; If a match is successful, the second proportion of different types of data in the information collection and mapping model is obtained; The first ratio is assigned the value of the second ratio.

2. The data compression method according to claim 1, characterized in that, If the topic keyword of the data to be compressed cannot be obtained, or the matching fails, then obtaining the first proportion of different types of data in the data to be compressed includes: The data to be compressed is randomly sampled to obtain sample data; Obtain the third proportion of different types of data in the sample data; The first ratio is assigned the value of the third ratio.

3. The data compression method according to claim 1, characterized in that, The method for setting up the information collection and calibration model includes: Obtain N modeling data tables, where N is a positive integer. Each modeling data table includes a table name and a description file. The description file includes modeling data and comments for the modeling data. For each modeling data table, a word segmenter is used to split it to obtain the corresponding topic keywords and the second ratio; Based on the N modeling data tables and their corresponding topic keywords and second ratios, N information collection and calibration models are obtained.

4. The data compression method according to claim 1, characterized in that, The first ratio of the different types of data includes: the ratio of numerical data to the data to be compressed, the ratio of character data to the data to be compressed, and the ratio of complex data to the data to be compressed.

5. The data compression method according to claim 4, characterized in that, The multiple preset compression schemes include: ZSTD compression algorithm, LZ4 compression algorithm, Snappy compression algorithm, and GZ compression algorithm.

6. The data compression method according to claim 5, characterized in that, The step of selecting a target compression scheme from multiple preset compression schemes according to the first ratio includes: If the proportion of numerical data is the largest, then the ZSTD compression algorithm is selected as the target compression scheme; If the proportion of character data is the largest, then the LZ4 compression algorithm is selected as the target compression scheme; If the proportion of the complex data type is the largest, then the Snappy compression algorithm is selected as the target compression scheme; If other data types constitute the largest proportion, then the GZ compression algorithm is selected as the target compression scheme.

7. An electronic device, characterized in that, It includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the computer program is executed by the processor, it implements the data compression method as described in any one of claims 1-6.

8. A data compression device, characterized in that, include: An acquisition module is used to acquire the data to be compressed, and to acquire the first proportion of different types of data in the data to be compressed; The selection module is used to select a target compression scheme from a plurality of preset compression schemes according to the first ratio; A compression module is used to compress the data to be compressed using the target compression scheme to achieve the target compression ratio; The step of obtaining the first proportion of different types of data in the data to be compressed includes: Determine whether the schema, a table structure description of the data to be compressed, can be obtained; If the table structure description information Schema can be obtained, then the first ratio is obtained based on the table structure description information Schema; If the table structure description information (Schema) cannot be obtained, then obtaining the first proportion of different types of data in the data to be compressed includes: Determine whether the topic keywords of the data to be compressed can be obtained; If the topic keywords can be obtained, the data to be compressed is matched with the pre-set information collection benchmark model based on the topic keywords; If a match is successful, the second proportion of different types of data in the information collection and mapping model is obtained; The first ratio is assigned the value of the second ratio.

Images