Data compression method, device and electronic equipment

Abstract

The embodiment of the present application provides a data compression method, device and electronic device. The method includes: obtaining a first event record; calling a first event definition including a parameter identifier and a preset value of the first event; comparing the first event definition and the first event record, and generating first event record compressed data according to the comparison result, wherein the first event record compressed data includes: first-category parameter data including parameter identifiers and record values ​​of first-category parameters, the first-category parameters are parameters whose record values ​​are different from preset values; second-category parameter data including parameter identifiers or third-category parameter data including parameter identifiers, wherein: second-category parameters are parameters whose record values ​​are the same as preset values; third-category parameters are parameters whose record values ​​are not recorded in the first event record. According to the method of an embodiment of the present application, the amount of log data is effectively reduced by reducing the parameter record values ​​and parameter identifiers recorded in the event record.

Description

Technical Field

[0001] The present application relates to the technical field of intelligent terminals, and in particular to a data compression method, device and electronic device. Background Art

[0002] In the application scenarios of personal terminals (eg, mobile phones, personal computers), as the application functions that can be realized by the personal terminals continue to increase, the amount of log data of the personal terminals is also increasing.

[0003] In some application scenarios, the log data of personal terminals needs to be uploaded to the server for comprehensive analysis (for example, personal terminal failure analysis). As the amount of personal terminal log data increases, the storage space required to store log data on the personal terminal also increases, and the data transmission traffic consumed by uploading log data also increases, which seriously affects the normal operation of the personal terminal and reduces the user experience. Summary of the invention

[0004] In order to solve the problem of excessive amount of log data, the present application provides a data compression method, device and electronic device, as well as a computer-readable storage medium.

[0005] The present application embodiment adopts the following technical solutions:

[0006] In a first aspect, the present application provides a data compression method, comprising:

[0007] Obtaining a first event record corresponding to the first event, the first event record including parameter identifiers and record values ​​of one or more parameters defined in the first event;

[0008] Calling a first event definition corresponding to the first event, the first event definition including parameter identifiers and preset values ​​of all parameters defined in the first event;

[0009] The first event definition and the first event record are compared, and first event record compressed data is generated according to the comparison result, wherein the first event record compressed data includes:

[0010] Event identification data, wherein the event identification data includes an event identification of the first event;

[0011] First-category parameter data, wherein the first-category parameter data includes parameter identifiers and record values ​​of first-category parameters, and the first-category parameters are parameters included in the first event record, the record values ​​of which are different from preset values;

[0012] The second type of parameter data or the third type of parameter data, where:

[0013] The second-category parameter data includes parameter identifiers of the second-category parameters, where the second-category parameters are parameters included in the first event record, and the recorded values ​​of the parameters are the same as the preset values;

[0014] The third type of parameter data includes parameter identifiers of the third type of parameters. The third type of parameters are parameters included in the first event definition, whose recorded values ​​are not recorded in the first event record.

[0015] In a possible implementation manner based on the first aspect, the first event record compressed data includes:

[0016] Among the second and third types of parameter data, the type with less data volume is the second type of parameter data.

[0017] In a possible implementation manner based on the first aspect, the first-category parameter data further includes a first parameter class identifier, the second-category parameter data further includes a second parameter class identifier, and the third-category parameter data further includes a third parameter class identifier.

[0018] In a possible implementation manner based on the first aspect, the event identification data further includes a first compression mode identification or a second compression mode identification, wherein:

[0019] When the first event record compressed data includes the second type of parameter data, the event identification data includes the first compression mode identification;

[0020] When the first event record compressed data includes the third type of parameter data, the event identification data includes the second compression mode identification.

[0021] In a possible implementation method based on the above-mentioned first aspect, the binary encoding of the event identification data is a code generated by replacing the last bit of the event identification offset code with the binary code of the first compression mode identifier or the second compression mode identifier, wherein the event identification offset code is a code generated by shifting the binary code of the event identifier left by 1 bit.

[0022] In a possible implementation manner based on the first aspect, the first-category parameter data further includes a first parameter class identifier, the second-category parameter data further includes a second parameter class identifier, and the third-category parameter data further includes a second parameter class identifier.

[0023] In a possible implementation manner based on the first aspect above:

[0024] The binary code of the first type of parameter data includes a first type of parameter identification code, the first type of parameter identification code is a code generated by replacing the last bit of the first type of parameter offset code with the binary code of the first parameter class identification, wherein the first type of parameter offset code is a code generated by shifting the binary code of the parameter identification of the first type of parameter left by 1 bit;

[0025] The binary code of the second type parameter data includes the second type parameter identification code, the second type parameter identification code is a code generated by replacing the last bit of the second type parameter offset code with the binary code of the second parameter class identification, wherein the second type parameter offset code is a code generated by shifting the binary code of the parameter identification of the second type parameter left by 1 bit;

[0026] The binary encoding of the third category parameter data includes the third category parameter identification code, which is a code generated by replacing the last bit of the third category parameter offset code with the binary code of the second parameter category identifier, wherein the third category parameter offset code is a code generated by shifting the binary code of the parameter identifier of the third category parameter left by 1 bit.

[0027] In a second aspect, the present application provides a data parsing method, comprising:

[0028] Obtaining first event record compressed data, where the first event record compressed data is data generated by compressing the first event record according to the method of the first aspect, and the first event record corresponds to the first event;

[0029] Calling a first event definition corresponding to the first event, the first event definition including parameter identifiers and preset values ​​of all parameters defined in the first event;

[0030] Parsing the compressed data of the first event record according to the first event definition includes:

[0031] When the first event record compressed data includes the second type of parameter data, for the parameter whose parameter identifier is recorded but whose parameter value is not recorded in the first event record compressed data, the preset value in the first event definition is used as the parameter value;

[0032] or,

[0033] When the first event record compressed data includes the third type of parameter data, for the parameters without record parameter identifiers in the first event record compressed data, the preset values ​​in the first event definition are used as parameter values.

[0034] In a third aspect, the present application provides a data compression device, comprising:

[0035] An event record acquisition module, which is used to acquire a first event record corresponding to a first event, wherein the first event record includes parameter identifiers and record values ​​of one or more parameters defined in the first event;

[0036] An event definition calling module, which is used to call a first event definition corresponding to a first event, wherein the first event definition includes parameter identifiers and preset values ​​of all parameters defined in the first event;

[0037] A compression module is used to compare the first event definition and the first event record, and generate first event record compressed data according to the comparison result, wherein the first event record compressed data includes:

[0038] Event identification data, wherein the event identification data includes an event identification of the first event;

[0039] First-category parameter data, wherein the first-category parameter data includes parameter identifiers and record values ​​of first-category parameters, and the first-category parameters are parameters included in the first event record, the record values ​​of which are different from preset values;

[0040] The second type of parameter data or the third type of parameter data, where:

[0041] The second-category parameter data includes parameter identifiers of the second-category parameters, where the second-category parameters are parameters included in the first event record, and the recorded values ​​of the parameters are the same as the preset values;

[0042] The third type of parameter data includes parameter identifiers of the third type of parameters. The third type of parameters are parameters included in the first event definition, whose recorded values ​​are not recorded in the first event record.

[0043] In a fourth aspect, the present application provides a data analysis device, comprising:

[0044] A data acquisition module, which is used to acquire first event record compressed data, where the first event record compressed data is data generated by compressing the first event record according to the first aspect method, and the first event record corresponds to the first event;

[0045] An event definition calling module, which is used to call a first event definition corresponding to a first event, wherein the first event definition includes parameter identifiers and preset values ​​of all parameters defined in the first event;

[0046] A data parsing module, which is used to parse the first event record compressed data according to the first event definition, including:

[0047] When the first event record compressed data includes the second type of parameter data, for the parameter whose parameter identifier is recorded but whose parameter value is not recorded in the first event record compressed data, the preset value in the first event definition is used as the parameter value;

[0048] or,

[0049] When the first event record compressed data includes the third type of parameter data, for the parameters without record parameter identifiers in the first event record compressed data, the preset values ​​in the first event definition are used as parameter values.

[0050] In a fifth aspect, the present application provides an electronic device, the electronic device comprising a memory for storing computer program instructions and a processor for executing the program instructions, wherein when the computer program instructions are executed by the processor, the electronic device is triggered to execute the method steps of the embodiment of the present application.

[0051] In a sixth aspect, the present application provides a computer-readable storage medium, in which a computer program is stored. When the computer-readable storage medium is run on a computer, the computer executes the method of the embodiment of the present application.

[0052] According to the above technical solution proposed in the embodiment of the present application, at least the following technical effects can be achieved:

[0053] According to the method of an embodiment of the present application, by reducing the parameter record values ​​and parameter identifiers recorded in the event record, the amount of log data is effectively reduced, thereby reducing the storage space occupied by the log data and the traffic consumption during the transmission process;

[0054] Furthermore, according to the method of an embodiment of the present application, for files of various data structures, corresponding data compression operations can be performed without modifying the code for generating the data structure, thereby avoiding the risks brought by modifying the program;

[0055] Furthermore, according to the method of an embodiment of the present application, the compression operation depends on the event definition. When there is no corresponding event definition, the compression result cannot be parsed, which greatly improves data security. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] Figure 1 Shown is a flow chart of an embodiment of a data compression method according to the present application;

[0057] Figure 2 Shown is a flow chart of an embodiment of a data parsing method according to the present application;

[0058] Figure 3 It is a structural schematic diagram of an embodiment of a data compression device according to the present application;

[0059] Figure 4 Shown is a structural schematic diagram of an embodiment of a data analysis device according to the present application. DETAILED DESCRIPTION

[0060] In order to make the purpose, technical solution and advantages of the present application clearer, the technical solution of the present application will be clearly and completely described below in combination with the specific embodiments of the present application and the corresponding drawings. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present application.

[0061] The terms used in the implementation section of this application are only used to explain the specific embodiments of this application and are not intended to limit this application.

[0062] In order to solve the problem of excessive amount of log data, the present application provides a data compression method to cut unnecessary data codes in the log data, thereby reducing the amount of log data.

[0063] For log data, a feasible compression method is to use the JS Object Notation (JSON) key-value pair format for data storage and transmission. However, when storing in the form of key-value pairs, there are a large number of key strings, which requires a large storage space; and when transmitting data, it is transmitted in plain text, which poses a potential information security risk.

[0064] Another possible compression method is to use a serial structured data format (Protocol Buffer, Protobuf). It uses the following process:

[0065] Define structured data as message, which is equivalent to class in Java. Use data structure definition file to store message definition.

[0066] Generate corresponding code files according to the data structure definition file, such as generating a .class file for each message;

[0067] Each member in the message is numbered and stored in the form of Tag-Value. The Tag stores the member number and the member encoding type value, and the Value stores the corresponding data value.

[0068] In the above solution, a .class file needs to be generated for each message, and a large amount of auxiliary code is generated. If there are many messages, a large amount of auxiliary code will be generated, and the data compression effect is not ideal. In addition, the implementation of the above solution requires modifying the program source code, which greatly increases the difficulty of implementing the solution.

[0069] Another feasible compression method is to use the data compression system Avro. The overall implementation concept of Avro is similar to Protobuf. In the Avro-based compression scheme, the member data is numbered and encoded in the order of numbers, but Avro does not store tags, and relies on event definition files to encode and decode in order. In the Avro-based compression scheme, since the corresponding code files, such as .calss files, need to be generated, a large amount of auxiliary code will still be generated, and the program source code needs to be modified to implement the Avro-based compression scheme.

[0070] Based on the above analysis of different compression schemes, an embodiment of the present application proposes a new data compression method to achieve better data compression of log data. In order to propose the method of the embodiment of the present application, the inventor first analyzes the data structure of log data. In general, log data is used to record events of personal terminals. For each event, the event may include one or more parameters for describing the event, and the parameters describe different event states through different parameter values. For the event record of an event, the event record is used to record the parameter values ​​of one or more parameters of all the parameters contained in the event (the parameter values ​​recorded in the event record hereinafter are referred to as record values).

[0071] Taking mobile phone log data as an example, the mobile phone will generate log data when it is running, which is used to record the running status of the mobile phone. The log data will be recorded in the form of events. For different event generation times and / or event generation sources, an event can have one or more different event records. Each event record contains one or more parameter (Param) records. Each parameter record contains the parameter identifier corresponding to the parameter and the record value corresponding to the parameter. The event record describes the status information of the event through the parameter record.

[0072] Since the log data includes a collection of multiple event records, if the data volume of the event records can be reduced, it is equivalent to reducing the data volume of the log data, thus achieving data compression of the log data.

[0073] Generally, for a certain event, the parameters included in the event have preset parameter values ​​(hereinafter referred to as preset values). The preset value of the parameter is the parameter value of the parameter when the event is in a preset state. Usually, the value of the parameter when the event is in a normal state is set as the preset value. In this way, if the event is abnormal, then among the parameters included in the event, there are one or more parameters whose parameter values ​​are not the preset values.

[0074] Since the preset value has already determined the specific value in advance, based on the existence of the preset value, if it can be determined that a parameter meets the preset value, then even if the specific parameter value of the parameter is not recorded, the parameter value of the parameter can be confirmed by retrieving the preset value.

[0075] Furthermore, in actual application scenarios, in order to facilitate the interpretation of log data, event definitions are usually predefined, and the data format of log data is standardized through event definitions. The event definition includes the definition of all parameters included in the event, for example, the parameter identifier of the parameter and the data type of the parameter value of the parameter. The event definition is saved by the generation end of the log data (for example, a personal terminal) and the parsing end (for example, a server that needs to analyze log data). When generating log data, if an event record of an event is to be generated, the pre-saved event definition of the event is called, and an event record that meets the event definition data format is generated through the definition of the parameters in the event definition. When parsing log data, if an event record of an event is to be parsed, the pre-saved event definition of the event is called, and the event record is parsed through the definition of the parameters in the event definition.

[0076] Therefore, based on the existence of the event definition, if the preset value of the parameter is written in the event definition, then when compressing the event record, in the compression result, for the parameter whose parameter value is the preset value, there is no need to save the parameter value; when parsing the compression result of the event record, if a parameter without a recorded parameter value is retrieved, the preset value in the event definition is directly called.

[0077] Furthermore, in actual application scenarios, there are parameters in the event record that do not have parameter values ​​recorded, or parameters are defined in the event definition but are not recorded at all in the event record. In view of the above situation, if it can be confirmed that there are parameters with parameter values ​​recorded in the event record, then by comparing with all the parameters defined in the event definition, it is possible to confirm the parameters without parameter values ​​recorded in the event record.

[0078] In summary, in an embodiment of the present application, when compressing an event record, in the compression result:

[0079] For parameters whose parameter values ​​are not preset values, the parameter identifier and parameter value of the parameter are retained;

[0080] For parameters whose parameter values ​​are preset values, the parameter values ​​are not retained, only the parameter identifiers are retained;

[0081] For parameters whose parameter values ​​are not recorded in the event record or parameters that are defined in the event definition but not recorded at all in the event record, neither the parameter value nor the parameter identifier is retained.

[0082] Furthermore, if it is possible to confirm that there are no parameters with parameter value records in the event record, then by comparing with all the parameters defined in the event definition, it is also possible to confirm that there are parameters with parameter value records in the event record.

[0083] Therefore, in an embodiment of the present application, when compressing an event record, in the compression result:

[0084] For parameters whose parameter values ​​are not preset values, the parameter identifier and parameter value of the parameter are retained;

[0085] For parameters whose parameter values ​​are not recorded in the event record or parameters that are defined in the event definition but not recorded in the event record at all, the parameter values ​​are not retained, only the parameter identifiers are retained;

[0086] For parameters whose parameter values ​​are preset values, neither the parameter value nor the parameter identifier is retained.

[0087] The technical solutions provided by various embodiments of the present application are described in detail below in conjunction with the accompanying drawings.

[0088] Figure 1 FIG. 1 is a flow chart of an embodiment of a data compression method according to the present application. In an embodiment of the present application, Figure 1 As shown, the process of compressing the first event record includes:

[0089] Step 110, obtaining a first event record corresponding to the first event, the first event record including parameter identifiers and record values ​​of one or more parameters defined in the first event;

[0090] Step 120, calling a first event definition corresponding to the first event, the first event definition including parameter identifiers and preset values ​​of all parameters defined in the first event;

[0091] Step 130 : Compare the first event definition and the first event record, and generate first event record compressed data according to the comparison result.

[0092] Specifically, in step 130, the first event record compressed data is generated including event identification data and first type parameter data, wherein:

[0093] The event identification data includes an event identification of the first event;

[0094] The first category parameter data includes parameter identifiers and record values ​​of first category parameters. The first category parameters are parameters included in the first event record, whose record values ​​are different from the preset values.

[0095] Further, in step 130, generating the first event record compressed data also includes the second type of parameter data or the third type of parameter data (the first event record compressed data also includes one of the second type of parameter data and the third type of parameter data, and does not include both the second type of parameter data and the third type of parameter data), wherein:

[0096] The second-category parameter data includes parameter identifiers of the second-category parameters, where the second-category parameters are parameters included in the first event record, and the recorded values ​​of the parameters are the same as the preset values;

[0097] The third type of parameter data includes parameter identifiers of the third type of parameters. The third type of parameters are parameters included in the first event definition, whose recorded values ​​are not recorded in the first event record.

[0098] That is, step 130 has at least a first implementation and a second implementation. In actual application, one of the first implementation and the second implementation is selected to implement step 130.

[0099] Specifically, in a first implementation of step 130, the first event record compressed data includes: event identification data, first-category parameter data, and second-category parameter data.

[0100] Specifically, in the second implementation of step 130, the first event record compressed data includes: event identification data, first-category parameter data, and third-category parameter data.

[0101] According to the method of one embodiment of the present application, by reducing the parameter record values ​​and parameter identifiers recorded in the event record, the amount of log data is effectively reduced, thereby reducing the storage space occupied by the log data and the traffic consumption during the transmission process.

[0102] Furthermore, according to the method of an embodiment of the present application, for files of various data structures, corresponding data compression operations can be performed without modifying the code for generating the data structure, thereby avoiding the risks brought by modifying the program.

[0103] Furthermore, according to the method of an embodiment of the present application, the compression operation depends on the event definition. When there is no corresponding event definition, the compression result cannot be parsed, which greatly improves data security.

[0104] Specifically, in the embodiment of the present application, the parameter identifier can be a parameter number (ID) or a parameter name. Specifically, in an actual application scenario, the specific type of the parameter identifier can be determined according to specific application requirements. For example, in one application scenario, considering that the code length of the parameter number is smaller than the parameter name, the parameter number is used as the parameter identifier. In another application scenario, considering that the parameter number is repeated, the parameter name is used as the parameter identifier.

[0105] It should be noted that in the multiple embodiments and application scenario examples of this specification, the object of data compression is the event record of log data. However, the data compression method proposed in the embodiment of the present application is not limited to the event record of log data, and any data with a similar data record structure to the event record of log data can adopt the data compression method proposed in the embodiment of the present application.

[0106] The following is a description of the specific application scenario Figure 1 The specific analysis process and execution process of the embodiment shown.

[0107] In an application scenario, each event has a corresponding event definition, which specifies the event ID and parameters. The function of the event definition is to specify the number of event parameters and the data type of parameter values, which is equivalent to a configuration file. For example, suppose there is the following event definition for event a:

[0108] <Event id=913000007>

[0109] <param id＝0name＝"Age"type＝"int32"default＝10>

[0110] <param id＝1name＝"Money"type＝"double"default＝1.0>

[0111] <param id＝2name＝"Name"type＝"string"default＝”ABC”>

[0112] <param id＝3name＝"Data"type＝"event"classID＝913009001>

[0113]

[0114] In the event definition:

[0115] Event ID: The unique identifier of each event, which is equivalent to each person's ID number. For example, in the event definition of event a above, Event id = 913000007, where 913000007 is the event ID of event a.

[0116] Parameter: An event has one or more parameters, which are used to record event information. In the parameter definition, each parameter has its own parameter number (ID), parameter name, parameter type, and default value. In the description of the application scenario of the following embodiment, when a parameter needs to be specified, the parameter name is used to refer to the specific parameter for the convenience of description.

[0117] Specifically, the parameter definitions include:

[0118] Parameter ID: parameter ID. For example, id=0 above, where 0 is the parameter ID. The parameter ID is unique in an event. That is, in an event, there is only one parameter with parameter ID 0. In different events, the values ​​of parameter IDs do not affect each other. That is, in one event, there is a parameter with parameter ID 0. In another event, there may also be a parameter with parameter ID 0. The two are not related and do not affect each other.

[0119] Parameter name: the name of the parameter, such as name = "Age", Age is the parameter name;

[0120] Parameter type: the data type of the parameter value, for example, if type = "double", double is the parameter type;

[0121] Default value: The value of the parameter that is preset when the event parameter is defined, such as <param id＝0name＝"Age"type＝"int32"default＝10> In the example, default=10, which is the default value of Age. In the above event definition, when the parameter type is event, the parameter has no default value.

[0122] Furthermore, in some application scenarios, due to the presence of multiple events, the event definition file is composed of multiple event definitions. For example, the event definition of event a mentioned above is only one event definition in the event definition file. For example, in a specific application scenario, the event definition file is as follows:

[0123]

[0124]

[0125] Based on the event definition of event a, in a specific application scenario, an event record A for event a may be in the following format:

[0126] {"EventID":913000007;"Age":23;"Money":1.0;"Name":"ABC"}.

[0127] In event record A, there are three types of recorded data for parameters:

[0128] Category A data ( Figure 1 The first type of parameter data in the illustrated embodiment): the recorded value of the parameter in the event record is different from the preset value. In the log data of the mobile phone, this type of data generally indicates abnormal data, that is, where an error occurs in the program. For example, in the example, the preset value of Age in the event definition is default=10, that is, the preset value of Age is 10, and "Age" in event record A is 23, that is, the recorded value of Age is 23. The preset value is different from the recorded value. In the scenario of the mobile phone log, the preset value is the correct value of the parameter when the program is running, so 10 is normal, and 23 is abnormal. Class A data indicates this type of data that is different from the preset value;

[0129] Category B data ( Figure 1 The second type of parameter data in the illustrated embodiment): the recorded value of the parameter in the event record is the same as the preset value. In the log data of the mobile phone, this type of data is generally normal data when the program is running, or a preset value used to mark the data as normal, such as Money and Name;

[0130] Category C data ( Figure 1 The third type of parameter data in the illustrated embodiment): parameters with no recorded values ​​in the event record, such as the parameter Data in the above event. Data is defined in the event definition, but there is no corresponding recorded value in the event record. In the mobile phone log data, this type of data corresponds to such a situation. As in the above event definition, there are 4 parameters Age, Money, Name, and Data. The program assigns values ​​to these 4 parameters during operation. Assuming that the order of assignment is Money, Name, Age, and Data, when Money and Name are assigned values, they are all normal, Money = 1.0, Name = "ABC". When Age is assigned Age = 23, after the assignment is completed, the code runs between Age and Data. The program crashes due to the incorrect assignment of Age. At this time, Data has no value. Therefore, when reporting the log, only the values ​​of the three parameters Money, Name, and Age can be reported, and the data of the Data parameter cannot be reported.

[0131] Taking mobile phone logs as an example, among the above three types of data, Class A data represents the fault information in the log, which is the key information that needs to be stored and must be stored intact; Class B data, because the record value is the same as the preset value of the parameter in the event definition, Class B data can be not stored during data compression, and restored according to the preset value in the event definition during data analysis; Class C data has no assigned value and cannot be stored.

[0132] For Class A data, it must be stored completely; the parameter values ​​of Class B data can be restored according to the preset values ​​in the event definition during data analysis, so they do not need to be stored; Class C data has no value assigned, so there is no way to store parameter values. Therefore, in the data compression result, the parameter values ​​of Class B and Class C data are not stored, and in order to distinguish whether the data without stored parameter values ​​is Class B data or Class C data during data analysis, the parameter flag is used to distinguish whether to store the parameter in the compression result.

[0133] Specifically, the first implementation of step 130 may be described as follows:

[0134] For Class A data, store complete information, such as parameter identifier (parameter name or parameter ID) + parameter value;

[0135] For Class B data, only the parameter identifier (parameter name or parameter ID) is stored to mark the parameter as Class B data;

[0136] For type C data, nothing is stored directly.

[0137] Take the event definition of event a and event record A for event a as an example. In event record A, it can be seen that Age is A-type data, Money and Name are B-type data, and Data is C-type data. After data compression based on the first implementation method of step 130, the following event record compressed data A1 can be obtained:

[0138] {913000007;Age:23;Money;Name}.

[0139] In the above event record compressed data A1:

[0140] The event ID, 913000007, is to find the event definition with event ID 913000007 in the event definition file;

[0141] Age: 23 is a type A data that must be stored completely, so both the parameter identifier of Age and the parameter value 23 are stored;

[0142] Money and Name are Class B data. To compress data, only the parameter names Money and Name are stored during data compression to mark that Money and Name are the same as the preset values. During data parsing, data recovery needs to be performed based on the preset values ​​in the event definition file.

[0143] Data has no value assigned and is not stored directly. When parsing the data, there is no data in the stored information {913000007; "Age": 23; "Money"; "Name"}, so it can be known that Data is Class C data, has no value assigned, and is not assigned using the preset value.

[0144] Compared with event record A, in event record compressed data A1, category C data is not stored, which is equivalent to compressing the space originally used to store category C data. At the same time, category B data only stores parameter names for marking, but does not store parameter values, which also saves space.

[0145] Specifically, the second implementation of step 130 may be described as follows:

[0146] For Class A data, complete information is stored, such as parameter identifier (parameter name or parameter ID) + actual parameter value;

[0147] For Class B data, simply store nothing;

[0148] For C-type data, only the parameter identifier (parameter name or parameter ID) is stored to mark the parameter as C-type data.

[0149] Take the event definition of event a and event record A for event a as an example. After data compression based on the second implementation of step 130, the following event record compressed data A2 can be obtained:

[0150] {913000007;Age:23;Data}.

[0151] In the above event record compressed data A2:

[0152] Same as the event record compressed data A1, the event ID such as 913000007 and the A-type data Age: 23 must be stored completely and remain unchanged;

[0153] The event record compressed data A2 does not store any information about the B-type data. During data analysis, since {913000007; Age: 23; Data} does not contain any information about Money and Name, the preset values ​​in the event definition are used to restore the data.

[0154] For C-type data in the event record compressed data A2, only the parameter name is stored to mark that the Data is C-type data without a value. No value is assigned during data parsing, nor is the preset value in the event definition used for assignment.

[0155] Compared with event record A, in event record compressed data A2, class B data is not stored, which is equivalent to compressing the space originally used to store class B data. At the same time, class C data only stores parameter names for marking, and does not store parameter values, which also saves space.

[0156] Further, based on Figure 1 In addition to the data compression method of the illustrated embodiment, an embodiment of the present application further proposes a data parsing method for compressed data.

[0157] Figure 2 FIG. 1 is a flow chart of an embodiment of a data analysis method according to the present application. In an embodiment of the present application, Figure 2 As shown, the process of parsing the compressed data of the first event record includes:

[0158] Step 210, obtaining first event record compressed data, the first event record compressed data being data generated by compressing the first event record according to any one of the methods of claims 1 to 7, the first event record corresponding to the first event;

[0159] Step 220, calling a first event definition corresponding to the first event, the first event definition including parameter identifiers and preset values ​​of all parameters defined in the first event;

[0160] Step 230, parsing the first event record compressed data according to the first event definition, includes:

[0161] When the first event record compressed data includes the second type of parameter data, for the parameter whose parameter identifier is recorded but whose parameter value is not recorded in the first event record compressed data, the preset value in the first event definition is used as the parameter value;

[0162] or,

[0163] When the first event record compressed data includes the third type of parameter data, for the parameters without record parameter identifiers in the first event record compressed data, the preset values ​​in the first event definition are used as parameter values.

[0164] Specifically, the specific implementation of step 230 depends on the specific implementation of step 130 .

[0165] In an implementation of step 230, when step 130 adopts the first implementation, for the parameter with a parameter identifier recorded but no parameter value recorded in the compressed data of the first event record, the preset value in the first event definition is used as the parameter value;

[0166] In another implementation of step 230, when step 130 adopts the second implementation, for the parameters without parameter identifiers recorded in the compressed data of the first event record, the preset values ​​in the first event definition are used as parameter values.

[0167] It should be noted that in the embodiments and application scenario examples of this specification, the object of data analysis is the compression result of the event record of the log data. However, the data analysis method proposed in the embodiment of the present application is not limited to the compressed data of the event record of the log data, and any data compressed by the method of the embodiment of the present application can use the data analysis method proposed in the embodiment of the present application.

[0168] Furthermore, since the specific implementation method of step 230 is determined by the specific implementation method of step 130, in one embodiment, the default data format of the event record compressed data needs to be unified at the data compression end (for example, a mobile phone that uploads log data) and the data parsing end (for example, a server that parses log data), that is, both ends need to confirm that, by default, the event record compressed data contains the second category parameter data, or contains the third category parameter data.

[0169] Further, in an actual application scenario, in one embodiment, the implementation of step 130 is determined according to the specific actual situation. Specifically, in one implementation of step 130, the data with less data volume in the second type of parameter data and the third type of parameter data is selected as the data stored in the first event record compressed data. That is, the first event record compressed data includes: the data with less data volume in the second type of parameter data and the third type of parameter data.

[0170] For example, for the above-mentioned event record A, there are 1 class A data, 2 class B data, and 1 class C data in event record A, class B data accounts for 50%, and class A and class C data each account for 25%, so step 130 adopts the second implementation method to obtain a better compression effect (the data length of event record compressed data A2 is significantly shorter than that of event record compressed data A1).

[0171] Furthermore, in the above event record compressed data A1 and event record compressed data A2, parameter identifiers are parameter names, such as Age, Name, Money, and Data. Since parameter names are strings, they occupy a large space. Therefore, in a more optimal solution, parameter identifiers use parameter IDs.

[0172] Furthermore, in one embodiment, in an implementation of step 130, description information is written in the event record compressed data, and the description information is used to describe whether step 130 adopts the first implementation method or the second implementation method. In this way, the data parsing end can determine the implementation method to be adopted in step 230 according to the description information in the event record compressed data.

[0173] Specifically, in an implementation of step 130, the event identification data further includes a first compression mode identification or a second compression mode identification, wherein:

[0174] When the first event record compressed data includes the second type of parameter data, the event identification data includes the first compression mode identification;

[0175] When the first event record compressed data includes the third type of parameter data, the event identification data includes a second compression mode identification.

[0176] Furthermore, in actual application scenarios, the event record compressed data is stored and transmitted in binary code form. In one implementation of step 130, the binary code of the event identification data is a code generated by replacing the last bit of the event identification offset code with the binary code of the first compression mode identifier or the second compression mode identifier, wherein the event identification offset code is a code generated by shifting the binary code of the event identification left by 1 bit.

[0177] For example, the binary code of the event identification data is event Tag1, and the definition of event Tag1 is as follows:

[0178] Event Tag1 = Event ID << 1 | Compression method identifier. (1)

[0179] The meaning of formula (1) is to shift the binary code of the event ID left by 1 bit and then replace the last bit after the left shift with the compression mode identifier (compress_type).

[0180] compress_type: the value is 0 (first compression mode identifier) ​​or 1 (second compression mode identifier), corresponding to the first implementation mode and the second implementation mode of step 130 respectively.

[0181] Take event ID 913000007 as an example. The binary code of 913000007 is:

[0182] 0011 0110 0110 1011 0100 0110 0100 0111.

[0183] After shifting left one position:

[0184] 0110 1100 1101 0110 1000 1100 1000 1110.

[0185] If step 130 adopts the second implementation mode, compress_type takes the value of 1, and the last 0 is replaced by 1, so 0110 1100 1101 0110 1000 1100 1000 1111 is obtained.

[0186] By identifying the compression method, the parser can confirm whether the parameter without a parameter value in the data contained in the compressed data of the event record is the second type of parameter data or the third type of parameter data. However, before this, the parser needs to confirm the first type of parameter data by checking whether the parameter contained in the compressed data of the event record has a parameter value.

[0187] In order to distinguish the first type of parameter data, in an implementation of step 130, the compression method identifier is not used, and an identifier is defined for each type of parameter data. Three parameter class identifiers corresponding to the first type of parameter data, the second type of parameter data, and the third type of parameter data are defined respectively, and the parsing end confirms which type the parameter data belongs to through the parameter class identifier contained in the parameter data. Specifically, in an implementation of step 130, the first type of parameter data also includes the first parameter class identifier, the second type of parameter data also includes the second parameter class identifier, and the third type of parameter data also includes the second parameter class identifier.

[0188] For example, in one application scenario, the parameter class identifier is written into the binary code of the parameter identifier to generate parameter Tag1.

[0189] Define the first parameter class identifier as 00, the first parameter class identifier as 01, and the first parameter class identifier as 10.

[0190] The binary encoding of the first type of parameter data includes the binary encoding of the parameter Tag1 of the first type of parameter data and the parameter value;

[0191] The binary encoding of the second type of parameter data includes a parameter Tag1 of the second type of parameter data;

[0192] The binary encoding of the third type of parameter data includes a parameter Tag1 of the third type of parameter data.

[0193] The parameter Tag1 of each type of parameter data is generated according to the parameter identifier corresponding to the type of parameter data.

[0194] The definition of parameter Tag1 is as follows:

[0195] Parameter Tag1 = parameter ID << 2 | parameter class ID. (2)

[0196] The meaning of formula (1) is to shift the binary code of the parameter ID left by 2 bits and then replace the last 2 bits after the left shift with the parameter class identifier.

[0197] Furthermore, in one implementation of step 130, a parameter class identifier is introduced on the basis of using a compression mode identifier. Specifically, a first parameter class identifier and a second parameter class identifier are defined. In the event record compression data, the first type of parameter data includes the first parameter class identifier, the second type of parameter data includes the second parameter class identifier, and the third type of parameter data includes the second parameter class identifier. In this way, the data parsing end can confirm the first type of parameter data based on the parameter class identifier. When the parameter data includes the first parameter class identifier, it is the first type of parameter data. And further, the second type of parameter data or the third type of parameter data is confirmed according to the compression mode identifier. When the event identification data includes the first compression mode identifier, the parameter data including the second parameter class identifier is the second type of parameter data; when the event identification data includes the second compression mode identifier, the parameter data including the second parameter class identifier is the third type of parameter data.

[0198] Furthermore, in actual application scenarios, the event record compressed data is stored and transmitted in binary code. In one implementation of step 130, the parameter class identifier is written into the binary code of the parameter identifier to generate the parameter identifier code. Specifically:

[0199] The binary code of the first-category parameter data includes a first-category parameter identification code, the first-category parameter identification code is a code generated by replacing the last bit of the first-category parameter offset code with the binary code of the first-category parameter class identifier, wherein the first-category parameter offset code is a code generated by shifting the binary code of the parameter name of the first-category parameter left by 1 bit;

[0200] The binary code of the second type parameter data includes the second type parameter identification code, the second type parameter identification code is a code generated by replacing the last bit of the second type parameter offset code with the binary code of the second parameter class identifier, wherein the second type parameter offset code is a code generated by shifting the binary code of the parameter name of the second type parameter left by 1 bit;

[0201] The binary encoding of the third category parameter data includes the third category parameter identification code, which is a code generated by replacing the last bit of the third category parameter offset code with the binary code of the second parameter category identifier, wherein the third category parameter offset code is a code generated by shifting the binary code of the parameter name of the third category parameter left by 1 bit.

[0202] For example, in an application scenario, the parameter identifier is defined as parameter Tag2, and the parameter class identifier is written into the binary code of the parameter identifier to generate parameter Tag2. The idea of ​​parameter Tag2 is similar to that of event Tag1, which is to use the last bit of the parameter ID shifted left by one bit to distinguish data. The definition of parameter Tag2 is as follows:

[0203] Parameter Tag2 = parameter ID << 1 | data_type. (2)

[0204] Formula (2) means that after the binary code of the parameter ID is shifted left by 2 bits, the last 2 bits after the left shift are replaced by the parameter class identifier data_type. data_type is used to distinguish the parameter data type. data_type is a binary number, 0 and 1, and the value meanings are as follows:

[0205] When the parameter data is the first type of parameter data, the corresponding data_type is 0;

[0206] When the parameter data is the second type parameter data or the third type parameter data, the corresponding data_type is 1.

[0207] Combined with the compression type identifier (compress_type), the following relationship can be obtained:

[0208]

[0209]

[0210] Table 1

[0211] Taking event description A as an example, the second implementation method of step 130 is used to compress event description A:

[0212] For the A-type data Age:23, the parameter ID of Age is 0, and its binary code is: 0000 0000;

[0213] After shifting one bit left, it becomes: 0000 0000;

[0214] Because Age is type A data, the value of data_type is 0. Replace 0 with 0 and get: 0000 0000 (Age parameter Tag2).

[0215] Class B data is not stored, so there is no need to store anything;

[0216] For the C-type data Data, the parameter ID of Data is 3, and its binary code is: 0000 0011;

[0217] After shifting one bit left: 0000 0110;

[0218] Because Data is type C data, the value of data_type is 1. Use 1 instead of 0 to get 0000 0111 (parameter Tag2 of Data).

[0219] Ultimately all data is stored in binary form.

[0220] For Class A data, complete data storage is performed in the form of Tag-Value. Since the parameter Tag of Age is 00000000, the binary code of the parameter value 23 of Age is 0001 0111. Therefore, the complete storage of Age is:

[0221] 0000 0000 0001 0111.

[0222] Therefore, the final compression result of event description A is:

[0223] 0110 1100 1101 0110 1000 1100 1000 1111 0000 0000 0001 0111 00000111.

[0224] in:

[0225] 0110 1100 1101 0110 1000 1100 1000 1111 is event Tag1;

[0226] 0000 0000 is the parameter Tag2 of Age;

[0227] 0001 0111 is the actual value of Age, 23;

[0228] 0000 0111 is the parameter Tag2 of Data.

[0229] It is to be understood that some or all of the steps or operations in the above embodiments are merely examples, and the present application embodiments may also perform other operations or variations of various operations. In addition, the various steps may be performed in different orders presented in the above embodiments, and it is possible that not all of the operations in the above embodiments need to be performed.

[0230] Furthermore, based on the data compression method proposed in one embodiment of the present application, one embodiment of the present application also proposes a data compression device. Figure 3 FIG. 1 is a schematic diagram showing the structure of an embodiment of a data compression device according to the present application. Figure 3 As shown, the data compression device 300 includes:

[0231] An event record acquisition module 310, which is used to acquire a first event record corresponding to a first event, wherein the first event record includes parameter names and record values ​​of one or more parameters defined in the first event;

[0232] An event definition calling module 320, which is used to call a first event definition corresponding to a first event, wherein the first event definition includes parameter names and preset values ​​of all parameters defined in the first event;

[0233] The compression module 330 is used to compare the first event definition and the first event record, and generate first event record compressed data according to the comparison result, wherein the first event record compressed data includes:

[0234] Event identification data, wherein the event identification data includes an event identification of the first event;

[0235] First-category parameter data, wherein the first-category parameter data includes parameter names and record values ​​of first-category parameters, and the first-category parameters are parameters included in the first event record, whose record values ​​are different from preset values;

[0236] One of the second type of parameter data or the third type of parameter data, where:

[0237] The second-category parameter data includes parameter names of the second-category parameters, where the second-category parameters are parameters included in the first event record, whose recorded values ​​are the same as the preset values;

[0238] The third type parameter data includes parameter names of the third type parameters. The third type parameters are parameters included in the first event definition, whose recorded values ​​are not recorded in the first event record.

[0239] Further, based on the data parsing method proposed in one embodiment of the present application, one embodiment of the present application also proposes a data parsing device. Figure 4 FIG. 1 is a schematic diagram of the structure of an embodiment of a data analysis device according to the present application. Figure 4 As shown, the data analysis device 400 includes:

[0240] A data acquisition module 410 is used to acquire first event record compressed data, where the first event record compressed data is data generated by the data compression device according to the embodiment of the present application, and the first event record corresponds to the first event;

[0241] An event definition calling module 420, which is used to call a first event definition corresponding to a first event, wherein the first event definition includes parameter names and preset values ​​of all parameters defined in the first event;

[0242] The data parsing module 430 is used to parse the first event record compressed data according to the first event definition, including:

[0243] When the first event record compressed data includes the second type of parameter data, for the parameter whose parameter identifier is recorded but whose parameter value is not recorded in the first event record compressed data, the preset value in the first event definition is used as the parameter value;

[0244] or,

[0245] When the first event record compressed data includes the third type of parameter data, for the parameters without record parameter identifiers in the first event record compressed data, the preset values ​​in the first event definition are used as parameter values.

[0246] Furthermore, in the 1990s, improvements to a technology could be clearly distinguished as hardware improvements (for example, improvements to the circuit structure of diodes, transistors, switches, etc.) or software improvements (improvements to the method flow). However, with the development of technology, many improvements to the method flow today can be regarded as direct improvements to the hardware circuit structure. Designers almost always obtain the corresponding hardware circuit structure by programming the improved method flow into the hardware circuit. Therefore, it cannot be said that an improvement in a method flow cannot be implemented using a hardware entity module. For example, a programmable logic device (PLD) (such as a field programmable gate array (FPGA)) is such an integrated circuit whose logical function is determined by the access party programming the device. Designers can "integrate" a digital device on a PLD by programming it themselves, without having to ask a chip manufacturer to design and produce a dedicated integrated circuit chip. Moreover, nowadays, instead of manually making integrated circuit chips, this kind of programming is mostly implemented by "logic compiler" software, which is similar to the software compiler used when developing and writing programs, and the original code before compilation must also be written in a specific programming language, which is called hardware description language (HDL). There is not only one HDL, but many kinds, such as ABEL (Advanced Boolean Expression Language), AHDL (Altera Hardware Description Language), Confluence, CUPL (Cornell University Programming Language), HDCal, JHDL (Java Hardware Description Language), Lava, Lola, MyHDL, PALASM, RHDL (Ruby Hardware Description Language), etc. The most commonly used ones are VHDL (Very-High-Speed ​​Integrated Circuit Hardware Description Language) and Verilog. Those skilled in the art should also know that it is only necessary to program the method flow slightly in the above-mentioned hardware description languages ​​and program it into the integrated circuit, and then it is easy to obtain the hardware circuit that implements the logic method flow.

[0247] The controller can be implemented in any appropriate manner, for example, the controller can take the form of a microprocessor or processor and a computer-readable medium storing a computer-readable program code (such as software or firmware) that can be executed by the (micro)processor, a logic gate, a switch, an application-specific integrated circuit (ASIC), a programmable logic controller, and an embedded microcontroller. Examples of controllers include, but are not limited to, the following microcontrollers: ARC 625D, Atmel AT91SAM, Microchip PIC18F26K20, and Silicone Labs C8051F320. The memory controller can also be implemented as part of the control logic of the memory. Those skilled in the art also know that in addition to implementing the controller in a purely computer-readable program code manner, the controller can be implemented in the form of a logic gate, a switch, an application-specific integrated circuit, a programmable logic controller, and an embedded microcontroller by logically programming the method steps. Therefore, this controller can be considered as a hardware component, and the devices included therein for implementing various functions can also be regarded as structures within the hardware component. Or even, the devices for implementing various functions can be regarded as both software modules for implementing the method and structures within the hardware component.

[0248] In the description of the embodiments of the present application, for the convenience of description, the device is described by being divided into various modules / units according to their functions. The division of each module / unit is merely a division of logical functions. When implementing the embodiments of the present application, the functions of each module / unit can be implemented in the same or one or more software and / or hardware.

[0249] Specifically, the device proposed in the embodiment of the present application can be fully or partially integrated into a physical entity during actual implementation, or it can be physically separated. And these modules can all be implemented in the form of software calling through processing elements; they can also be all implemented in the form of hardware; some modules can also be implemented in the form of software calling through processing elements, and some modules can be implemented in the form of hardware. For example, the detection module can be a separately established processing element, or it can be integrated in a chip of an electronic device. The implementation of other modules is similar. In addition, all or part of these modules can be integrated together, or they can be implemented independently. In the implementation process, each step of the above method or each of the above modules can be completed by an integrated logic circuit of hardware in a processor element or instructions in the form of software.

[0250] For example, the above modules may be one or more integrated circuits configured to implement the above methods, such as one or more application specific integrated circuits (ASIC), or one or more digital signal processors (DSP), or one or more field programmable gate arrays (FPGA), etc. For another example, these modules may be integrated together and implemented in the form of a system-on-a-chip (SOC).

[0251] An embodiment of the present application further proposes an electronic device, which includes a memory for storing computer program instructions and a processor for executing the program instructions, wherein when the computer program instructions are executed by the processor, the electronic device is triggered to execute the method steps described in the embodiment of the present application.

[0252] Specifically, in one embodiment of the present application, the above-mentioned one or more computer programs are stored in the above-mentioned memory, and the above-mentioned one or more computer programs include instructions. When the above-mentioned instructions are executed by the above-mentioned device, the above-mentioned device executes the method steps described in the embodiment of the present application.

[0253] Specifically, in one embodiment of the present application, the processor of the electronic device may be a SOC device on a chip, which may include a central processing unit (CPU), and may further include other types of processors. Specifically, in one embodiment of the present application, the processor of the electronic device may be a PWM control chip.

[0254] Specifically, in an embodiment of the present application, the processor involved may include, for example, a CPU, a DSP, a microcontroller or a digital signal processor, and may also include a GPU, an embedded neural network processor (Neural-network Process Units, NPU) and an image signal processor (Image Signal Processing, ISP). The processor may also include necessary hardware accelerators or logic processing hardware circuits, such as ASICs, or one or more integrated circuits for controlling the execution of the program of the technical solution of the present application. In addition, the processor may have the function of operating one or more software programs, and the software programs may be stored in a storage medium.

[0255] Specifically, in one embodiment of the present application, the memory of the electronic device may be a read-only memory (ROM), other types of static storage devices that can store static information and instructions, a random access memory (RAM) or other types of dynamic storage devices that can store information and instructions, or an electrically erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compressed optical discs, laser discs, optical discs, digital versatile discs, Blu-ray discs, etc.), a magnetic disk storage medium or other magnetic storage devices, or may be any computer-readable medium that can be used to carry or store desired program codes in the form of instructions or data structures and can be accessed by a computer.

[0256] Specifically, in an embodiment of the present application, the processor and the memory can be combined into a processing device, more commonly, they are independent components, and the processor is used to execute the program code stored in the memory to implement the method described in the embodiment of the present application. In specific implementation, the memory can also be integrated into the processor, or independent of the processor.

[0257] Furthermore, the devices, apparatuses, modules or units described in the embodiments of the present application may be implemented by computer chips or entities, or by products having certain functions.

[0258] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, devices, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media containing computer-usable program code.

[0259] In the several embodiments provided in the present application, if any function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application.

[0260] Specifically, in one embodiment of the present application, a computer-readable storage medium is further provided, in which a computer program is stored. When the computer program is run on a computer, the computer executes the method provided in the embodiment of the present application.

[0261] An embodiment of the present application further provides a computer program product, which includes a computer program. When the computer program is run on a computer, the computer executes the method provided by the embodiment of the present application.

[0262] The embodiments described in this application are described with reference to the flowcharts and / or block diagrams of the methods, devices (apparatus), and computer program products according to the embodiments of the application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0263] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0264] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0265] It should also be noted that in the embodiments of the present application, "at least one" refers to one or more, and "more than one" refers to two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent the existence of A alone, the existence of A and B at the same time, and the existence of B alone. Among them, A and B can be singular or plural. The character " / " generally indicates that the previous and subsequent associated objects are in an "or" relationship. "At least one of the following" and similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one of a, b and c can be represented by: a, b, c, a and b, a and c, b and c, or a and b and c, where a, b, c can be single or multiple.

[0266] In the embodiments of the present application, the terms "include", "comprises" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of further restrictions, an element defined by the sentence "includes a ..." does not exclude the presence of other identical elements in the process, method, commodity or device including the element.

[0267] The present application may be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application may also be practiced in distributed computing environments where tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules may be located in local and remote computer storage media, including storage devices.

[0268] Each embodiment in this application is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0269] Those skilled in the art will appreciate that the various units and algorithm steps described in the embodiments of the present application can be implemented by a combination of electronic hardware, computer software, and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0270] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described devices, apparatuses and units can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0271] The above is only a specific implementation of the present application. Any person skilled in the art can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. The protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A data compression method, characterized in that: include: Acquire a first event record corresponding to a first event, wherein the first event record includes parameter identifiers and record values ​​of one or more parameters defined in the first event; calling a first event definition corresponding to the first event, wherein the first event definition includes parameter identifiers and preset values ​​of all parameters defined in the first event; The first event definition and the first event record are compared, and first event record compressed data is generated according to the comparison result, wherein the first event record compressed data includes: Event identification data, wherein the event identification data includes an event identification of the first event; First-category parameter data, wherein the first-category parameter data includes parameter identifiers and record values ​​of first-category parameters, and the first-category parameters are parameters included in the first event record, the record values ​​of which are different from the preset values; The second type of parameter data or the third type of parameter data, where: The second-category parameter data includes parameter identifiers of second-category parameters, where the second-category parameters are parameters included in the first event record, whose recorded values ​​are the same as the preset values; The third-category parameter data includes parameter identifiers of third-category parameters, and the third-category parameters are parameters included in the first event definition, whose recorded values ​​are not recorded in the first event record.

2. The method according to claim 1, characterized in that The first event record compressed data includes: The second type of parameter data and the third type of parameter data have a smaller amount of data.

3. The method according to claim 1 or 2, characterized in that: The first-category parameter data also includes a first parameter class identifier, the second-category parameter data also includes a second parameter class identifier, and the third-category parameter data also includes a third parameter class identifier.

4. The method according to claim 1 or 2, characterized in that: The event identification data also includes a first compression mode identification or a second compression mode identification, wherein: When the first event record compressed data includes the second type of parameter data, the event identification data includes the first compression mode identification; When the first event record compressed data includes the third type parameter data, the event identification data includes the second compression mode identification.

5. The method according to claim 4, characterized in that The binary code of the event identification data is a code generated by replacing the last bit of the event identification offset code with the binary code of the first compression mode identifier or the second compression mode identifier, wherein the event identification offset code is a code generated by shifting the binary code of the event identification left by 1 bit.

6. The method according to claim 4 or 5, characterized in that: The first type parameter data also includes a first parameter type identifier, the second type parameter data also includes a second parameter type identifier, and the third type parameter data also includes a second parameter type identifier.

7. The method according to claim 6, characterized in that: The binary code of the first-category parameter data includes a first-category parameter identification code, wherein the first-category parameter identification code is a code generated by replacing the last bit of the first-category parameter offset code with the binary code of the first-category parameter class identification, wherein the first-category parameter offset code is a code generated by shifting the binary code of the parameter identification of the first-category parameter left by 1 bit; The binary code of the second-category parameter data includes a second-category parameter identification code, wherein the second-category parameter identification code is a code generated by replacing the last bit of the second-category parameter offset code with the binary code of the second-category parameter class identification, wherein the second-category parameter offset code is a code generated by shifting the binary code of the parameter identification of the second-category parameter left by 1 bit; The binary encoding of the third category parameter data includes a third category parameter identification code, wherein the third category parameter identification code is a code generated by replacing the last bit of the third category parameter offset code with the binary code of the second parameter category identification, wherein the third category parameter offset code is a code generated by shifting the binary code of the parameter identification of the third category parameter left by 1 bit.

8. A data analysis method, characterized in that: include: Acquire first event record compressed data, where the first event record compressed data is data generated by compressing a first event record according to the method of any one of claims 1 to 7, and the first event record corresponds to a first event; calling a first event definition corresponding to the first event, wherein the first event definition includes parameter identifiers and preset values ​​of all parameters defined in the first event; Parsing the first event record compressed data according to the first event definition includes: When the first event record compressed data includes the second type of parameter data, for the parameters in the first event record compressed data that record parameter identifiers but do not record parameter values, the preset values ​​in the first event definition are used as parameter values; or, When the first event record compressed data includes third-category parameter data, for parameters without record parameter identifiers in the first event record compressed data, preset values ​​in the first event definition are used as parameter values.

9. A data compression device, characterized in that: include: An event record acquisition module, configured to acquire a first event record corresponding to a first event, wherein the first event record includes parameter identifiers and record values ​​of one or more parameters defined in the first event; An event definition calling module, which is used to call a first event definition corresponding to the first event, wherein the first event definition includes parameter identifiers and preset values ​​of all parameters defined in the first event; A compression module is used to compare the first event definition and the first event record, and generate first event record compressed data according to the comparison result, wherein the first event record compressed data includes: Event identification data, wherein the event identification data includes an event identification of the first event; First-category parameter data, wherein the first-category parameter data includes parameter identifiers and record values ​​of first-category parameters, and the first-category parameters are parameters included in the first event record, the record values ​​of which are different from the preset values; The second type of parameter data or the third type of parameter data, where: The second-category parameter data includes parameter identifiers of second-category parameters, where the second-category parameters are parameters included in the first event record, whose recorded values ​​are the same as the preset values; The third-category parameter data includes parameter identifiers of third-category parameters, and the third-category parameters are parameters included in the first event definition, whose recorded values ​​are not recorded in the first event record.

10. A data analysis device, characterized in that: include: A data acquisition module, used for acquiring first event record compressed data, wherein the first event record compressed data is data generated by compressing a first event record according to the method of any one of claims 1 to 7, and the first event record corresponds to a first event; An event definition calling module, which is used to call a first event definition corresponding to the first event, wherein the first event definition includes parameter identifiers and preset values ​​of all parameters defined in the first event; A data parsing module, used for parsing the first event record compressed data according to the first event definition, comprising: When the first event record compressed data includes the second type of parameter data, for the parameters in the first event record compressed data that record parameter identifiers but do not record parameter values, the preset values ​​in the first event definition are used as parameter values; or, When the first event record compressed data includes third-category parameter data, for parameters without record parameter identifiers in the first event record compressed data, preset values ​​in the first event definition are used as parameter values.

11. An electronic device, characterized in that: The electronic device comprises a memory for storing computer program instructions and a processor for executing the program instructions, wherein when the computer program instructions are executed by the processor, the electronic device is triggered to execute the method steps as claimed in any one of claims 1 to 8.

12. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, which, when executed on a computer, enables the computer to execute the method according to any one of claims 1 to 8.

Images