Compression methods, devices, vehicles, and media for JSON data in data warehouses

Abstract

The application relates to a compression method and device based on JSON data in a data warehouse, a vehicle and a medium, wherein the method comprises the following steps: obtaining original JSON data of a target vehicle, and removing null values in the original JSON data to generate null value-removed data; target field information of the null value-removed data is extracted, and the target field information is converted into corresponding target data types according to data processing requirements of a target user; the target field information corresponding to the target data types is converted into HIVE table form data, and a preset data compression algorithm is used to compress the HIVE table form data, so that the compressed data is stored in a preset HIVE data warehouse. The application can effectively improve the reading performance of JSON data in the data warehouse, meet the data analysis and query requirements, significantly reduce the data storage requirements, reduce the storage space cost, fully utilize the HIVE distributed data warehouse capacity, and optimize the data storage and query process.

Description

Technical Field

[0001] This application relates to the field of big data analytics technology, and in particular to a method, apparatus, vehicle, and medium for compressing JSON data in a data warehouse. Background Technology

[0002] In the field of vehicle network data processing, facing the challenge of large-scale data, the industry often uses raw hexadecimal messages or parsed JSON format messages. However, both methods have the following technical problems:

[0003] 1. Raw hexadecimal messages: high parsing cost, not feasible, limiting the practical application value of the data.

[0004] 2. Parsed JSON format message: The dataset is huge, with thousands of fields defined, but not every field contains a value. The storage cost is high, the data volume is large, and the storage overhead is significant.

[0005] 3. Frequently changing data structures: While the flexibility of the JSON data format is very suitable for dynamically expanding fields, it is not the optimal choice for a relatively fixed environment like a data warehouse. It is complex, has high maintenance costs, and is difficult to adapt flexibly to dynamic data.

[0006] Currently, existing technologies can utilize common data storage and compression techniques, and the implementation methods are as follows:

[0007] 1. Relational databases: used for storing and retrieving data;

[0008] 2. Data compression algorithms (e.g., Gzip, Snappy): used to compress data to save storage space;

[0009] 3. Custom data processing workflow: In order to meet the needs of a data warehouse, it is usually necessary to build a custom data processing workflow, including data transformation and cleaning.

[0010] However, existing technologies cannot effectively handle null fields in massive datasets, wasting storage space. Furthermore, the flexibility of the JSON format increases the complexity of data processing and reduces query performance. In addition, the efficiency and performance of existing technologies are not ideal in a data warehouse environment, leading to resource waste and performance degradation, which urgently need to be addressed. Summary of the Invention

[0011] This application provides a method, apparatus, vehicle, and medium for compressing JSON data in a data warehouse, in order to solve the problems that existing technologies cannot effectively handle null fields in huge datasets, which wastes storage space. Furthermore, the flexibility of the JSON format increases the complexity of data processing and reduces query performance. In addition, the efficiency and performance of existing technologies are not ideal in a data warehouse environment, leading to problems such as resource waste and performance degradation.

[0012] The first aspect of this application provides a method for compressing JSON data in a data warehouse, comprising the following steps: obtaining the original JSON data of a target vehicle and removing null values ​​from the original JSON data to generate null-free data; extracting target field information from the null-free data and converting the target field information into a corresponding target data type according to the data processing needs of the target user; converting the target field information corresponding to the target data type into HIVE table-format data, and compressing the HIVE table-format data using a preset data compression algorithm to store the compressed data in a preset HIVE data warehouse.

[0013] Optionally, in one embodiment of this application, removing null values ​​from the original JSON data to generate null-free data includes: traversing all levels of the original JSON data, detecting whether each key-value pair in each level has a null value, and obtaining and deleting all null values ​​in the original JSON data to generate the null-free data.

[0014] Optionally, in one embodiment of this application, the step of extracting the target field information of the null-free data and converting the target field information into a corresponding target data type according to the data processing needs of the target user includes: extracting each key-value pair in the first layer of the null-free data, converting the value in each key-value pair in the first layer into a string type to obtain the first-layer value conversion data corresponding to the null-free data, and storing the first-layer value conversion data as first-layer MAP type data; based on the first-layer MAP type data, determining each json_str field in the second layer of the null-free data, and performing a parsing operation on each json_str field to generate target type parsing data, and storing the target type parsing data as second-layer MAP type data.

[0015] Optionally, in one embodiment of this application, converting the target field information corresponding to the target data type into HIVE table format data includes: determining the column names and data types of the first-level MAP type data and the second-level MAP type data respectively, so as to represent the first-level MAP type data and the second-level MAP type data as HIVE table format data according to the column names and data types; adding an extra field ext to the HIVE table format data corresponding to the first-level MAP type data to obtain supplementary HIVE table format data corresponding to the first-level MAP type data; merging the supplementary HIVE table format data and the HIVE table format data corresponding to the second-level MAP type data to obtain the target field information corresponding to the data with null values ​​removed.

[0016] Optionally, in one embodiment of this application, the step of compressing the HIVE table data using a preset data compression algorithm to store the compressed data in a preset HIVE data warehouse includes: converting the HIVE table data into ORC format data, compressing the ORC format data using the Snappy compression algorithm to obtain ORC format compressed data, and storing the ORC format compressed data in the preset HIVE data warehouse.

[0017] A second aspect of this application provides a compression device based on JSON data in a data warehouse, comprising: a removal module for acquiring the original JSON data of a target vehicle and removing null values ​​from the original JSON data to generate null-free data; an extraction module for extracting target field information from the null-free data and converting the target field information into a corresponding target data type according to the data processing needs of the target user; and a compression module for converting the target field information corresponding to the target data type into HIVE table-format data and compressing the HIVE table-format data using a preset data compression algorithm to store the compressed data in a preset HIVE data warehouse.

[0018] Optionally, in one embodiment of this application, the removal module includes: a traversal unit, used to traverse all levels of the original JSON data, detect whether each key-value pair in each level has a null value, and obtain and delete all null values ​​in the original JSON data to generate the null-free data.

[0019] Optionally, in one embodiment of this application, the extraction module includes: a first conversion unit, configured to extract each key-value pair of the first layer in the null-removed data, and convert the value in each key-value pair of the first layer into a string type to obtain the first-layer value conversion data corresponding to the null-removed data, and store the first-layer value conversion data as first-layer MAP type data; and a parsing unit, configured to determine each json_str field of the second layer in the null-removed data based on the first-layer MAP type data, and perform a parsing operation on each json_str field to generate target type parsing data, and store the target type parsing data as second-layer MAP type data.

[0020] Optionally, in one embodiment of this application, the compression module includes: a determining unit, configured to determine the column names and data types of the first-layer MAP type data and the second-layer MAP type data respectively, so as to represent the first-layer MAP type data and the second-layer MAP type data as HIVE table data according to the column names and data types; an adding unit, configured to add an extra field ext to the HIVE table data corresponding to the first-layer MAP type data to obtain supplementary HIVE table data corresponding to the first-layer MAP type data; and a merging unit, configured to merge the supplementary HIVE table data and the HIVE table data corresponding to the second-layer MAP type data to obtain the target field information corresponding to the data with null values ​​removed.

[0021] Optionally, in one embodiment of this application, the compression module further includes: a second conversion unit, configured to convert the HIVE tabular data into ORC format data, compress the ORC format data using the Snappy compression algorithm to obtain ORC format compressed data, and store the ORC format compressed data in the preset HIVE data warehouse.

[0022] A third aspect of this application provides a vehicle, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the compression method based on JSON data in a data warehouse as described in the above embodiments.

[0023] A fourth aspect of this application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the compression method described above based on JSON data in a data warehouse.

[0024] Therefore, the embodiments of this application have the following beneficial effects:

[0025] The embodiments of this application can obtain the original JSON data of the target vehicle, remove null values ​​from the original JSON data to generate null-free data, extract the target field information of the null-free data, and convert the target field information into the corresponding target data type according to the data processing needs of the target user; convert the target field information corresponding to the target data type into HIVE table format data, and compress the HIVE table format data using a preset data compression algorithm, so as to store the compressed data in a preset HIVE data warehouse. This application can effectively improve the reading performance of JSON data in the data warehouse to meet the needs of data analysis and query, significantly reduce data storage requirements, reduce storage space costs, and fully utilize the capabilities of the HIVE distributed data warehouse to optimize the data storage and query process. Therefore, it solves the problems of existing technologies being unable to effectively handle null fields in huge datasets, wasting storage space, and the increased complexity of data processing due to the flexibility of the JSON format, which reduces query performance. Furthermore, the efficiency and performance of existing technologies are not ideal in a data warehouse environment, leading to resource waste and performance degradation.

[0026] Additional aspects and advantages of this application will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of this application. Attached Figure Description

[0027] The above and / or additional aspects and advantages of this application will become apparent and readily understood from the following description of the embodiments taken in conjunction with the accompanying drawings, wherein:

[0028] Figure 1 This is a flowchart illustrating a method for compressing JSON data in a data warehouse according to an embodiment of this application;

[0029] Figure 2 This is an example diagram of raw JSON data provided according to an embodiment of this application;

[0030] Figure 3 This is a schematic diagram illustrating the removal of null values ​​from data according to an embodiment of this application.

[0031] Figure 4 This is a schematic diagram illustrating the conversion of first-layer data in null-free data to MAP type data according to an embodiment of this application.

[0032] Figure 5 This is a schematic diagram illustrating the conversion of json_str data of the second layer of data in data with null values ​​removed, to MAP type data, according to an embodiment of this application.

[0033] Figure 6 This is a schematic diagram illustrating a first-layer data display in the form of a HIVE table, according to an embodiment of this application.

[0034] Figure 7 This is a schematic diagram illustrating how a second-layer json_str displays data in the form of a HIVE table, according to an embodiment of this application.

[0035] Figure 8 This is a complete data illustration in the form of a HIVE table according to an embodiment of this application;

[0036] Figure 9 This is an example diagram of a compression device based on JSON data in a data warehouse according to an embodiment of this application;

[0037] Figure 10 This is a structural schematic diagram of the vehicle provided in an embodiment of this application.

[0038] Among them, 10-compression device based on JSON data in data warehouse; 100-removal module, 200-extraction module, 300-compression module; 1001-memory, 1002-processor, 1003-communication interface. Detailed Implementation

[0039] The embodiments of this application are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain this application, and should not be construed as limiting this application.

[0040] The following describes a method, apparatus, vehicle, and medium for compressing JSON data in a data warehouse, based on embodiments of this application, with reference to the accompanying drawings. Addressing the problems mentioned in the background section, this application provides a method for compressing JSON data in a data warehouse. In this method, the original JSON data of the target vehicle is obtained, and null values ​​are removed to generate null-free data. Target field information is extracted from the null-free data and converted into a corresponding target data type according to the data processing needs of the target user. The target field information corresponding to the target data type is converted into HIVE table-format data, and the HIVE table-format data is compressed using a preset data compression algorithm to store the compressed data in a preset HIVE data warehouse. This application can effectively improve the read performance of JSON data in a data warehouse to meet data analysis and query needs, significantly reduce data storage requirements, lower storage space costs, and fully utilize the capabilities of the HIVE distributed data warehouse to optimize the data storage and query process.

[0041] Specifically, Figure 1 This is a flowchart illustrating a method for compressing JSON data in a data warehouse, as provided in an embodiment of this application.

[0042] like Figure 1 As shown, this compression method based on JSON data in a data warehouse includes the following steps:

[0043] In step S101, the original JSON data of the target vehicle is obtained, and null values ​​in the original JSON data are removed to generate null-free data.

[0044] Those skilled in the art should understand that, since the actual parameters may vary depending on the size and characteristics of the dataset, they generally need to be adjusted according to the Hive configuration and data warehouse requirements. Therefore, the embodiments of this application can implement the corresponding compression operation in the standard environment of Hive and Hadoop clusters, and require corresponding Hive servers and Hadoop distributed storage devices to integrate with Hive and related tools, thereby facilitating data processing, transformation and storage in the Hive environment.

[0045] Before performing efficient compression of JSON data, this embodiment of the application first needs to obtain a set of original JSON data containing key-value pairs, such as... Figure 2 As shown, this data may contain some keys that are null.

[0046] Therefore, the embodiments of this application can analyze the initial form of the original JSON data to obtain the key-value distribution and possible null values ​​in the original data, and perform corresponding null removal operations when there are control conditions to generate null-removed data corresponding to the original JSON data, thereby effectively reducing the data size and providing reliable data guidance and basis for subsequent data processing and compression.

[0047] Optionally, in one embodiment of this application, removing null values ​​from the original JSON data to generate null-free data includes: traversing all levels of the original JSON data, detecting whether each key-value pair in each level has a null value, and obtaining and deleting all null values ​​in the original JSON data to generate null-free data.

[0048] It should be noted that the embodiments of this application firstly traverse all levels of the original JSON data. Regardless of the nesting level of null values, the embodiments of this application ensure that null values ​​at all levels are processed by traversing the entire JSON dataset. Secondly, the embodiments of this application can detect whether the value of each key-value pair is null. If it is null, it is removed from the data. Figure 3 As shown, this is used to generate null-free data.

[0049] Therefore, the embodiments of this application perform comprehensive null value processing on all levels of JSON data, so that even if the original JSON data has a complex structure such as nesting, all null values ​​can be traversed and removed, thereby effectively reducing the data volume and improving storage efficiency.

[0050] In step S102, the target field information of the null-free data is extracted, and the target field information is converted into the corresponding target data type according to the data processing requirements of the target user.

[0051] After generating null-free data, embodiments of this application can further extract the keys (fields) of the first and second layers from the JSON data to obtain the core information of the data, and convert it into ordinary data types or maps as needed. <string:string>type or array <map <string:string>>Type, for keys that are not extracted or are unknown, use map <string:string>The type is stored in a large field.

[0052] Therefore, embodiments of this application accurately extract the keys of the first and second layers by removing null fields from JSON data, and convert them into different data types as needed to obtain core information, reduce data size, and improve storage efficiency. Furthermore, embodiments of this application can also support general data types and maps. <string:string>Type and array <map <string:string>It supports multiple data types, including type, to flexibly adapt to the complexity of JSON data, thereby reducing the impact of unrecognized keys.

[0053] Optionally, in one embodiment of this application, extracting target field information from the null-removed data and converting the target field information into the corresponding target data type according to the data processing needs of the target user includes: extracting each key-value pair in the first layer of the null-removed data and converting the value in each key-value pair in the first layer into a string type to obtain the first-layer value conversion data corresponding to the null-removed data, and storing the first-layer value conversion data as first-layer MAP type data; based on the first-layer MAP type data, determining each json_str field in the second layer of the null-removed data, and performing a parsing operation on each json_str field to generate target type parsing data, and storing the target type parsing data as second-layer MAP type data.

[0054] In the specific implementation process, the embodiments of this application can perform data type conversion operations on the target field information of the extracted null-free data, that is, the corresponding key-value pair information of the first and second layers. The specific process is as follows:

[0055] 1. Convert first-level key-value pairs to MAP type data

[0056] (1) Extracting the first-level key-value pairs: By parsing the JSON structure after removing null values, the embodiments of this application only focus on and extract the first-level key-value pairs, thereby ensuring that the dataset processed by the embodiments of this application is limited to the outermost structure and simplifying the subsequent processing steps;

[0057] (2) Value type conversion: For each extracted key-value pair, a value type conversion operation is performed to convert all values ​​to string type to ensure that the generated map is of string type. <string:string>Data type consistency in values;

[0058] (3) Store as a MAP type: Store the extracted and value-type-converted key-value pairs as a map. <string:string>A data structure of type is used to obtain the first-level MAP type data, such as Figure 4 As shown, this not only facilitates efficient storage in the data warehouse, but also provides reliable technical support for subsequent queries.

[0059] It is important to note that in the actual execution process, in addition to storing the extracted first-level relevant data as a map... <string:string>In addition to data types, those skilled in the art can also store it as a map of arrays or ordinary fields, depending on the actual situation, to be compatible with diverse data structures and adaptable to different types of data scenarios.

[0060] 2. Convert the second-level json_str data to MAP type.

[0061] (1) Locate the json_str field of the second layer: Extract the json_str field of the second layer from the map type defined in the first layer above;

[0062] (2) Parsing the json_str value: For each extracted json_str field, a parsing operation is performed. It is parsed into a map. <string:string>Types of data to ensure that key-value pairs can be processed and stored efficiently;

[0063] (3) Store as a MAP type: Store the parsed key-value pairs as a map. <string:string>A data structure of type is used to obtain second-level MAP type data, such as Figure 5 As shown, this allows for more efficient storage and querying within the data warehouse.

[0064] Understandably, since the json_str field in the second layer may contain complex structures, those skilled in the art need to flexibly choose appropriate data types to ensure that the data can be stored clearly and completely.

[0065] Therefore, this application can formulate rules for handling null values ​​and extracting keys from JSON data, which can be used to remove null value fields from JSON data and extract the keys of the first and second levels. The rules can also be customized according to the characteristics and requirements of the data, such as determining which keys must be retained, which keys can be ignored, and how to handle unrecognized keys, thereby significantly reducing data storage requirements, saving storage space, and improving the efficiency of data conversion.

[0066] In step S103, the target field information corresponding to the target data type is converted into HIVE table data, and the HIVE table data is compressed using a preset data compression algorithm so that the compressed data is stored in a preset HIVE data warehouse.

[0067] Furthermore, embodiments of this application can store the target field information corresponding to the target data type, i.e., the null-free data after data type conversion, in the form of a Hive table to fully utilize Hive's distributed data warehouse capabilities. In addition, embodiments of this application can also significantly reduce the data size by using data compression algorithms on the data stored in the form of a Hive table (i.e., Hive table data), and can optimize the performance of the data model in the data warehouse. Moreover, by performing queries and analysis through Hive, the efficiency of data query can be effectively improved.

[0068] Optionally, in one embodiment of this application, converting the target field information corresponding to the target data type into HIVE table format data includes: determining the column names and data types of the first-level MAP type data and the second-level MAP type data respectively, so as to represent the first-level MAP type data and the second-level MAP type data as HIVE table format data according to the column names and data types; adding an extra field ext to the HIVE table format data corresponding to the first-level MAP type data to obtain supplementary HIVE table format data corresponding to the first-level MAP type data; merging the supplementary HIVE table format data and the HIVE table format data corresponding to the second-level MAP type data to obtain the target field information corresponding to the null-free data.

[0069] Furthermore, embodiments of this application can respectively display the first-level MAP type data and the second-level MAP type data in the form of a HIVE table, and then merge them to display the complete HIVE table data. The specific process is as follows:

[0070] 1. The first level displays the data in the form of HIVE tables.

[0071] (1) Define column names and data types: Explicitly define the name and corresponding data type of each column in the Hive table;

[0072] (2) Save as the corresponding data type: For the data after the first layer of conversion, select the appropriate Hive data type according to the specific data structure. The data type may include arrays, maps and ordinary data types, etc.

[0073] Since raw JSON data may contain complex structures, including arrays and nested map types, Hive table design needs to handle these complex structures flexibly in order to select appropriate data types.

[0074] (3) Data Structure Conversion: Before writing to a Hive table, data structure conversion may be necessary to meet the data type requirements specified by Hive, such as... Figure 6 As shown, for example, the values ​​corresponding to the map are converted to data types supported by Hive to ensure data standardization.

[0075] 2. Display the data from the second-level json_str as a Hive table:

[0076] (1) Define column names and data types: Explicitly define the name and corresponding data type of each column in the Hive table;

[0077] (2) Save as the corresponding data type: For the data after the second layer of conversion, select the appropriate Hive data type according to the specific data structure. The data type may include arrays, maps and ordinary data types, etc.

[0078] (3) Data Structure Conversion: Before writing to a Hive table, data structure conversion may be necessary to meet the data type requirements specified by Hive, such as... Figure 7 As shown, for example, the values ​​corresponding to the map can be converted to data types supported by Hive;

[0079] (4) Handling unparsed key-value pairs: For unparsed key-value pairs in the json_str field of the second layer, they are cleverly saved into a map type field named ext__json_str, which greatly improves the flexibility of the data and can effectively cope with future changes in the data structure.

[0080] 3. Display data completely in the form of a HIVE table.

[0081] (1) Handling unresolved key-value pairs: For unresolved key-value pairs that may exist in the first layer, they are cleverly saved into a map type field named ext, thereby adapting to data changes and flexibility and ensuring data integrity;

[0082] (2) Merge all processed fields: Merge the processed data from the first and second layers to form a complete dataset, such as... Figure 8 As shown; in addition, embodiments of this application may also add an extra field ext to the first layer to store other information about the data.

[0083] Therefore, the embodiments of this application can completely display all processed data in the form of Hive tables, and the tables include all processed fields, which can better support more comprehensive data warehouse queries and analyses. This not only improves data query performance and speeds up data analysis and report generation, but also improves the efficiency and resource utilization of the Hive environment and reduces maintenance costs.

[0084] Optionally, in one embodiment of this application, HIVE table data is compressed using a preset data compression algorithm to store the compressed data in a preset HIVE data warehouse. This includes: converting the HIVE table data into ORC format data, compressing the ORC format data using the Snappy compression algorithm to obtain ORC format compressed data, and storing the ORC format compressed data in the preset HIVE data warehouse.

[0085] It should be noted that the embodiments of this application can perform final data storage and compression on the processed Hive tables (i.e., Hive table-formatted data), and the specific process is as follows:

[0086] 1. Choose ORC format: Choose to use ORC format for data storage. ORC format stores data in columns to significantly reduce data size, provides higher compression ratio and query performance, and can also support complex data types, making it suitable for large-scale data warehouses;

[0087] (2) Default compression method Snappy: The default Snappy compression method is used to provide faster data reading speed while ensuring compression ratio.

[0088] In summary, the embodiments of this application remove null values ​​from the original JSON format string data and extract its first and second level key-value pairs, transforming them into general field types and maps as needed. <string:string>type, array <map <string:string>>The key type, taking into account both the flexibility of JSON and the characteristics of Hive, includes unrecognized keys in a map, in addition to some known and fixed keys. <string:string>The data is stored in a large field, which enables key operations such as removing null values, extracting keys, and converting data types from the original JSON data. Finally, the processed data is stored in the form of a Hive table and compressed using ORC format to improve data storage and query performance.

[0089] Therefore, the embodiments of this application, through reasonable data structure design, flexible data type selection, and ingenious handling of unparsed fields, effectively overcome the challenges in vehicle network data processing, achieving high efficiency and performance optimization in data storage and querying. They can flexibly handle the dynamic nature of JSON data while fully leveraging the powerful capabilities of Hive and related tools. Furthermore, the embodiments of this application not only provide higher efficiency and performance for big data analysis of vehicle network data, supporting faster decision-making and data mining, but also offer feasible and efficient solutions for data warehouse processing in the big data field, providing more efficient support for data analysis of vehicle network data.

[0090] The compression method for JSON data in a data warehouse proposed in this application involves obtaining the original JSON data of the target vehicle, removing null values ​​to generate null-free data, extracting target field information from the null-free data, and converting the target field information into the corresponding target data type according to the data processing needs of the target user. The target field information corresponding to the target data type is then converted into HIVE table-format data, and compressed using a preset data compression algorithm. The compressed data is then stored in a preset HIVE data warehouse. This application effectively improves the read performance of JSON data in the data warehouse to meet data analysis and query needs, significantly reduces data storage requirements, lowers storage space costs, and fully utilizes the capabilities of the HIVE distributed data warehouse to optimize data storage and query processes.

[0091] Secondly, a compression apparatus based on JSON data in a data warehouse according to an embodiment of this application is described with reference to the accompanying drawings.

[0092] Figure 9 This is a block diagram of a compression device for JSON data in a data warehouse according to an embodiment of this application.

[0093] like Figure 9 As shown, the compression device 10 based on JSON data in the data warehouse includes: a removal module 100, an extraction module 200, and a compression module 300.

[0094] The removal module 100 is used to obtain the original JSON data of the target vehicle and remove null values ​​from the original JSON data to generate null-free data.

[0095] The extraction module 200 is used to extract the target field information of the null-free data and convert the target field information into the corresponding target data type according to the data processing needs of the target user.

[0096] Compression module 300 is used to convert the target field information corresponding to the target data type into HIVE table format data, and compress the HIVE table format data through a preset data compression algorithm so as to store the compressed data in a preset HIVE data warehouse.

[0097] Optionally, in one embodiment of this application, the removal module 100 includes: a traversal unit, used to traverse all levels of the original JSON data, detect whether each key-value pair in each level has a null value, and obtain and delete all null values ​​in the original JSON data to generate null-free data.

[0098] Optionally, in one embodiment of this application, the extraction module 200 includes a first conversion unit and a parsing unit.

[0099] The first conversion unit is used to extract each key-value pair in the first layer of the null-removed data, convert the value in each key-value pair in the first layer into a string type, so as to obtain the first layer value conversion data corresponding to the null-removed data, and store the first layer value conversion data as first layer MAP type data.

[0100] The parsing unit is used to determine each json_str field in the second layer of null-free data based on the first layer MAP type data, and to perform parsing operations on each json_str field to generate target type parsed data, and store the target type parsed data as second layer MAP type data.

[0101] Optionally, in one embodiment of this application, the compression module 300 includes: a determining unit, an adding unit, and a merging unit.

[0102] The determining unit is used to determine the column names and data types of the first-level MAP type data and the second-level MAP type data respectively, so as to represent the first-level MAP type data and the second-level MAP type data as HIVE table data according to the column names and data types.

[0103] Add a unit to add an extra field (ext) to the HIVE table format data corresponding to the first-level MAP type data, thereby obtaining supplementary HIVE table format data corresponding to the first-level MAP type data.

[0104] The merging unit is used to merge supplementary HIVE table data and the corresponding HIVE table data of the second-level MAP type data to obtain the target field information corresponding to the data with null values ​​removed.

[0105] Optionally, in one embodiment of this application, the compression module 300 further includes: a second conversion unit, used to convert HIVE tabular data into ORC format data, compress the ORC format data using the Snappy compression algorithm to obtain ORC format compressed data, and store the ORC format compressed data in a preset HIVE data warehouse.

[0106] It should be noted that the foregoing explanation of the method embodiment for compressing JSON data in a data warehouse also applies to the compression device for JSON data in a data warehouse in this embodiment, and will not be repeated here.

[0107] The compression device based on JSON data in a data warehouse, as proposed in this application, includes a removal module for acquiring the original JSON data of a target vehicle and removing null values ​​from the original JSON data to generate null-free data; an extraction module for extracting target field information from the null-free data and converting the target field information into a corresponding target data type according to the data processing needs of the target user; and a compression module for converting the target field information corresponding to the target data type into HIVE table-format data and compressing the HIVE table-format data using a preset data compression algorithm to store the compressed data in a preset HIVE data warehouse. This application can effectively improve the read performance of JSON data in a data warehouse to meet data analysis and query needs, significantly reduce data storage requirements, lower storage space costs, and fully utilize the capabilities of the HIVE distributed data warehouse to optimize the data storage and query process.

[0108] Figure 10 A schematic diagram of the structure of a vehicle provided in an embodiment of this application. The vehicle may include:

[0109] The memory 1001, the processor 1002, and the computer program stored on the memory 1001 and capable of running on the processor 1002.

[0110] When the processor 1002 executes the program, it implements the compression method based on JSON data in the data warehouse provided in the above embodiments.

[0111] Furthermore, the vehicle also includes:

[0112] Communication interface 1003 is used for communication between memory 1001 and processor 1002.

[0113] The memory 1001 is used to store computer programs that can run on the processor 1002.

[0114] The memory 1001 may include high-speed RAM memory, and may also include non-volatile memory, such as at least one disk storage device.

[0115] If the memory 1001, processor 1002, and communication interface 1003 are implemented independently, then the communication interface 1003, memory 1001, and processor 1002 can be interconnected via a bus to complete communication between them. The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized into address buses, data buses, control buses, etc. For ease of representation, Figure 10 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0116] Optionally, in a specific implementation, if the memory 1001, processor 1002, and communication interface 1003 are integrated on a single chip, then the memory 1001, processor 1002, and communication interface 1003 can communicate with each other through an internal interface.

[0117] The processor 1002 may be a central processing unit (CPU), an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of this application.

[0118] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the compression method based on JSON data in a data warehouse as described above.

[0119] In the description of this specification, the references to "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0120] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this application, "N" means at least two, such as two, three, etc., unless otherwise explicitly specified.

[0121] Any process or method described in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or N executable instructions for implementing custom logic functions or processes, and the scope of the preferred embodiments of this application includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the functions involved, as should be understood by those skilled in the art to which embodiments of this application pertain.

[0122] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-included system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of computer-readable media include: an electrical connection having one or more wires (electronic device), a portable computer disk drive (magnetic device), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Alternatively, the computer-readable medium may be paper or other suitable media on which the program can be printed, since the program can be obtained electronically by optically scanning the paper or other medium, followed by editing, interpreting, or otherwise processing as necessary, and then stored in a computer memory.

[0123] It should be understood that the various parts of this application can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, the N steps or methods can be implemented using software or firmware stored in memory and executed by a suitable instruction execution system. If implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0124] Those skilled in the art will understand that all or part of the steps of the methods described in the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, it includes one or a combination of the steps of the method embodiments.

[0125] Furthermore, the functional units in the various embodiments of this application can be integrated into a processing module, or each unit can exist physically separately, or two or more units can be integrated into a module. The integrated module can be implemented in hardware or as a software functional module. If the integrated module is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium.

[0126] The storage medium mentioned above can be a read-only memory, a disk, or an optical disk, etc. Although embodiments of this application have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting this application. Those skilled in the art can make changes, modifications, substitutions, and variations to the above embodiments within the scope of this application.< / string:string> < / string:string> < / string:string> < / string:string> < / string:string> < / string:string> < / string:string> < / string:string> < / string:string> < / string:string> < / string:string> < / string:string> < / string:string>

Claims

1. A compression method based on JSON data in a data warehouse, characterized in that, The method comprises the following steps: obtaining original JSON data of a target vehicle, and removing null values in the original JSON data to generate non-null value data; extracting target field information of the non-null value data, and converting the target field information into a corresponding target data type according to a data processing requirement of a target user; converting the target field information corresponding to the target data type into HIVE table form data, and compressing the HIVE table form data through a preset data compression algorithm to store the compressed data into a preset HIVE data warehouse; wherein the extracting of the target field information of the non-null value data and the converting of the target field information into the corresponding target data type according to the data processing requirement of the target user comprises: extracting each key-value pair of a first layer in the non-null value data, and converting values in each key-value pair of the first layer into a string type to obtain first layer value conversion data corresponding to the non-null value data, and storing the first layer value conversion data as first layer MAP type data; based on the first layer MAP type data, determining each json_str field of a second layer in the non-null value data, and performing a parsing operation on each json_str field to generate target type parsing data, and storing the target type parsing data as second layer MAP type data; the converting of the target field information corresponding to the target data type into HIVE table form data comprises: determining column names and data types of the first layer MAP type data and the second layer MAP type data respectively, to represent the first layer MAP type data and the second layer MAP type data as HIVE table form data according to the column names and the data types; adding an ext additional field in the HIVE table form data corresponding to the first layer MAP type data to obtain supplemented HIVE table form data corresponding to the first layer MAP type data; merging the supplemented HIVE table form data and HIVE table form data corresponding to the second layer MAP type data to obtain target field information corresponding to the non-null value data.

2. The method of claim 1, wherein, the removing of the null values in the original JSON data to generate the non-null value data comprises: traversing all levels of the original JSON data, detecting whether each key-value pair of each level has a null value, and obtaining and deleting all null values of the original JSON data to generate the non-null value data.

3. The method of claim 1, wherein, the compressing of the HIVE table form data through the preset data compression algorithm to store the compressed data into the preset HIVE data warehouse comprises: converting the HIVE table form data into ORC format data, and compressing the ORC format data through a Snappy compression algorithm to obtain ORC format compressed data, and storing the ORC format compressed data into the preset HIVE data warehouse.

4. An apparatus for compressing JSON data in a data warehouse based on, characterized by, comprise: a removing module configured to obtain original JSON data of a target vehicle, and remove null values in the original JSON data to generate non-null value data; The extraction module is configured to extract target field information of the null value removed data, and convert the target field information into a corresponding target data type according to a data processing requirement of a target user. The compression module is configured to convert the target field information corresponding to the target data type into HIVE table form data, compress the HIVE table form data by using a preset data compression algorithm, and store the compressed data into a preset HIVE data warehouse. The extraction of the target field information of the null value removed data and the conversion of the target field information into the corresponding target data type according to the data processing requirement of the target user include: The extraction module is configured to extract target field information of the null value removed data, and convert the target field information into a corresponding target data type according to a data processing requirement of a target user. The extraction of the target field information of the null value removed data and the conversion of the target field information into the corresponding target data type according to the data processing requirement of the target user include: The conversion of the target field information corresponding to the target data type into the HIVE table form data includes: The determination of the column name and the data type of the first layer MAP type data and the second layer MAP type data, respectively, and the representation of the first layer MAP type data and the second layer MAP type data as the HIVE table form data according to the column name and the data type; The addition of the ext additional field in the HIVE table form data corresponding to the first layer MAP type data to obtain the supplemented HIVE table form data corresponding to the first layer MAP type data; The merging of the supplemented HIVE table form data and the HIVE table form data corresponding to the second layer MAP type data to obtain the target field information corresponding to the null value removed data.

5. The apparatus of claim 4, wherein, The elimination module includes: The traversal unit is configured to traverse all levels of the original JSON data, detect whether each key-value pair of each level has a null value, and acquire and delete all null values of the original JSON data to generate the null value removed data.

6. A vehicle characterized by comprising: The computer program is stored in the memory and executable on the processor, and the processor executes the program to implement the compression method based on the JSON data in the data warehouse. The program is executed by the processor to implement the compression method based on the JSON data in the data warehouse.

7. A computer readable storage medium having stored thereon a computer program, characterized in that, ​

Images