Cooperative processing method and device for off-line data, vehicle and storage medium

Abstract

The invention relates to the technical field of vehicles, in particular to an offline data co-processing method and device, a vehicle and a storage medium, and the method comprises the steps: obtaining a business demand; determining at least one target off-line data warehouse management tool according to the business demand, and controlling each target off-line data warehouse management tool to call off-line data in a preset distributed file system by using a preset off-line data warehouse cooperation tool; and using a preset offline data warehouse cooperation tool to control each target offline data warehouse management tool to process the called offline data. Therefore, the problems that the data processing requirements of different business scenes cannot be met by using a single counting tool and the data processing efficiency is low are solved, the flexibility of data processing can be improved, and efficient utilization of data resources is realized.

Description

Technical Field

[0001] The present application relates to the field of vehicle technology, and in particular to a method, device, vehicle and storage medium for collaborative processing of offline data. Background Art

[0002] In the field of big data, there are a variety of data warehouse tools, such as Hadoop, Hive, Spark, Presto, ClickHouse, etc. Each tool has its own unique advantages and applicable scenarios, and these tools have different requirements when processing data.

[0003] Related technologies usually use a data warehouse tool to process offline data. However, in actual business scenarios, the same data often needs to be processed and analyzed in a variety of different ways. A single data warehouse tool cannot meet all needs. In addition, multiple data warehouse tools are difficult to coordinate effectively and it is difficult to ensure data consistency and integrity. Summary of the invention

[0004] The present application provides a collaborative processing method, device, vehicle and storage medium for offline data to solve the problem that a single data warehouse tool cannot meet the data processing requirements of different business scenarios, resulting in low data processing efficiency. It can improve the flexibility of data processing and achieve efficient use of data resources.

[0005] The first embodiment of the present application provides a method for collaborative processing of offline data, comprising the following steps:

[0006] Obtain business requirements;

[0007] Determine at least one target offline data warehouse management tool according to the business requirements, and use a preset offline data warehouse collaboration tool to control each target offline data warehouse management tool to call offline data in a preset distributed file system;

[0008] The preset offline data warehouse collaboration tool is used to control each target offline data warehouse management tool to process the called offline data.

[0009] Optionally, in some embodiments, before obtaining the business requirements, the following steps are included:

[0010] Get offline data;

[0011] The offline data is stored in the preset distributed file system, and the preset distributed file system is associated with at least one offline data warehouse management tool.

[0012] Optionally, in some embodiments, the using a preset offline data warehouse collaboration tool to control each target offline data warehouse management tool to call the offline data in the target distributed file system includes:

[0013] Use each target offline data warehouse management tool to create a corresponding temporary work table;

[0014] The preset offline data warehouse collaboration tool is used to control each target offline data warehouse management tool to call the offline data in the target distributed file system, and store the offline data in the corresponding temporary table.

[0015] Optionally, in some embodiments, after using the preset offline data warehouse collaboration tool to control each target offline data warehouse management tool to process the called offline data, the method further includes:

[0016] Delete the offline data in each temporary worksheet.

[0017] Optionally, in some embodiments, after using the preset offline data warehouse collaboration tool to control each target offline data warehouse management tool to process the called offline data, the method further includes:

[0018] Generate offline data processing results, and visualize the offline data processing results.

[0019] A second aspect of the present application provides a collaborative processing device for offline data, including:

[0020] Acquisition module, used to obtain business requirements;

[0021] A determination module, configured to determine at least one target offline data warehouse management tool according to the business requirements, and use a preset offline data warehouse collaboration tool to control each target offline data warehouse management tool to call offline data in a preset distributed file system;

[0022] The control module is used to control each target offline data warehouse management tool to process the called offline data by using the preset offline data warehouse collaboration tool.

[0023] Optionally, in some embodiments, before obtaining the business requirements, the obtaining module includes:

[0024] An acquisition unit, used for acquiring offline data;

[0025] An association unit is used to store the offline data in the preset distributed file system, and the preset distributed file system is associated with at least one offline data warehouse management tool.

[0026] Optionally, in some embodiments, the control module includes:

[0027] A creation unit is used to create a corresponding temporary work table using each target offline data warehouse management tool;

[0028] The storage unit is used to use the preset offline data warehouse collaboration tool to control each target offline data warehouse management tool to call the offline data in the target distributed file system, and store the offline data in the corresponding temporary table.

[0029] Optionally, in some embodiments, after using the preset offline data warehouse collaboration tool to control each target offline data warehouse management tool to process the called offline data, the determination module further includes:

[0030] Delete unit, used to delete offline data in each temporary worksheet.

[0031] Optionally, in some embodiments, after using the preset offline data warehouse collaboration tool to control each target offline data warehouse management tool to process the called offline data, the control module further includes:

[0032] The visualization unit is used to generate an offline data processing result and visualize the offline data processing result.

[0033] A third aspect of the present application provides a vehicle, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the method for collaborative processing of offline data as described in the above embodiment.

[0034] The fourth aspect of the present application provides a computer-readable storage medium on which a computer program is stored. The program is executed by a processor to implement the collaborative processing method of offline data as described in the above embodiment.

[0035] Therefore, by obtaining business needs and determining at least one target offline data warehouse management tool according to the business needs, the preset offline data warehouse collaboration tool is used to control each target offline data warehouse management tool to call the offline data in the preset distributed file system, and the preset offline data warehouse collaboration tool is used to control each target offline data warehouse management tool to process the called offline data. In this way, the problem that a single data warehouse tool cannot meet the data processing needs of different business scenarios and the data processing efficiency is low is solved, the flexibility of data processing can be improved, and the efficient use of data resources can be achieved.

[0036] Additional aspects and advantages of the present application will be given in part in the description below, and in part will become apparent from the description below, or will be learned through the practice of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:

[0038] Figure 1 A flowchart of a method for collaborative processing of offline data provided according to an embodiment of the present application;

[0039] Figure 2 A schematic diagram of an offline data warehouse construction process provided according to an embodiment of the present application;

[0040] Figure 3 A schematic diagram of the principle of a collaborative processing method for offline data provided according to an embodiment of the present application;

[0041] Figure 4 A block diagram of a collaborative processing device for offline data provided according to an embodiment of the present application;

[0042] Figure 5 A schematic diagram of the structure of a vehicle is provided according to an embodiment of the present application. DETAILED DESCRIPTION

[0043] Embodiments of the present application are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present application, and should not be construed as limiting the present application.

[0044] The following describes the collaborative processing method, device, vehicle and storage medium of offline data of the embodiments of the present application with reference to the accompanying drawings. In view of the problem mentioned in the above background technology that the use of a single data warehouse tool cannot meet the data processing requirements of different business scenarios and the data processing efficiency is low, the present application provides a collaborative processing method for offline data. In this method, by obtaining business needs and determining at least one target offline data warehouse management tool according to the business needs, a preset offline data warehouse collaborative tool is used to control each target offline data warehouse management tool to call the offline data in the preset distributed file system, and a preset offline data warehouse collaborative tool is used to control each target offline data warehouse management tool to process the called offline data. In this way, the problem that the use of a single data warehouse tool cannot meet the data processing requirements of different business scenarios and the data processing efficiency is low is solved, the flexibility of data processing can be improved, and the efficient use of data resources can be achieved.

[0045] Specifically, Figure 1 A flowchart of a collaborative processing method for offline data provided in an embodiment of the present application.

[0046] like Figure 1 As shown, the offline data collaborative processing method includes the following steps:

[0047] In step S101, business requirements are obtained.

[0048] Among them, business requirements include input parameters, and business requirements are specific goals and functions that users or enterprises hope to achieve through offline data warehouse management tools.

[0049] Specifically, the embodiments of the present application need to determine the offline data warehouse management tools and data processing methods based on business needs to avoid developing a system that does not meet actual needs.

[0050] Optionally, in some embodiments, before obtaining business requirements, it includes: obtaining offline data; storing the offline data in a preset distributed file system, and the preset distributed file system is associated with at least one offline data warehouse management tool.

[0051] The preset distributed file system is Hadoop, and the offline data warehouse management tool includes at least one of Hive and Iceberg.

[0052] It should be noted that, combined with Figure 2 As shown, the embodiment of the present application can pre-build an offline data warehouse. First, build a Hadoop cluster, deploy offline data warehouse management tools Hive and Iceberg, deploy the distributed SQL engine Kyuubi, and create an HDFS data storage directory for storing offline data.

[0053] Specifically, the embodiment of the present application uses ETL tools or other data synchronization and acquisition methods to save external offline data to a specified hdfs directory in the hadoop cluster.

[0054] In step S102, at least one target offline data warehouse management tool is determined according to business requirements, and a preset offline data warehouse collaboration tool is used to control each target offline data warehouse management tool to call offline data in a preset distributed file system.

[0055] Among them, the preset offline data warehouse collaboration tool is the Kyuubi engine.

[0056] It is understandable that both hive and iceberg support directly loading data from hadoop files into corresponding hive and iceberg tables, and the Kyuubi engine can directly operate hive and iceberg tables. Hive and iceberg tables can share a copy of hadoop file data.

[0057] Specifically, the embodiment of the present application determines at least one target offline data warehouse management tool by inputting parameters, for example, Figure 3As shown, when the input parameter is H, hive is determined as the target offline data warehouse management tool, when the input parameter is I, iceberg is determined as the target offline data warehouse management tool, and when the input parameter is A, hive and iceberg are determined as the target offline data warehouse management tools at the same time, wherein the input parameters are pre-set by relevant personnel.

[0058] Furthermore, in some embodiments, a preset offline data warehouse collaboration tool is used to control each target offline data warehouse management tool to call offline data in the target distributed file system, including: using each target offline data warehouse management tool to create a corresponding temporary work table; using the preset offline data warehouse collaboration tool to control each target offline data warehouse management tool to call offline data in the target distributed file system, and storing the offline data in the corresponding temporary table work.

[0059] Specifically, after determining at least one target offline data warehouse management tool, a preset offline data warehouse collaboration tool is used to control each target offline data warehouse management tool to call offline data in a preset distributed file system. For example, when hive is determined as the target offline data warehouse management tool, a hive temporary work table is created, and the offline data is called into the hive temporary work table; when iceberg is determined as the target offline data warehouse management tool, an iceberg temporary work table is created, and the offline data is called into the iceberg temporary work table; when hive and iceberg are determined as both the target offline data warehouse management tools, a hive temporary work table and an iceberg temporary work table are created at the same time, and the offline data is called into the hive temporary work table and the iceberg temporary work table.

[0060] In step S103, a preset offline data warehouse collaboration tool is used to control each target offline data warehouse management tool to process the called offline data.

[0061] Specifically, the Kyuubi engine can directly operate hive and iceberg tables to dynamically load data into the tables and execute query SQL statements. For example, in a project, the preset offline data warehouse collaboration tool interface is called to specify the offline data warehouse tool to execute SQL by passing parameters.

[0062] Optionally, in some embodiments, after using the preset offline data warehouse collaboration tool to control each target offline data warehouse management tool to process the called offline data, the method further includes: deleting the offline data in each temporary worksheet.

[0063] Specifically, after data processing is completed, the data in the table is cleared or deleted so that new data can be reloaded for processing later. This can avoid data duplication or save storage space. By releasing data that is no longer needed, the use of storage resources can be optimized and the overall efficiency of the system can be improved. In addition, if data is temporarily loaded into the table for specific queries or analysis, then after the query is completed, this temporary data can be released so that system resources can be used by other tasks.

[0064] Optionally, in some embodiments, after using a preset offline data warehouse collaboration tool to control each target offline data warehouse management tool to process the called offline data, it also includes: generating offline data processing results and visualizing the offline data processing results.

[0065] Specifically, after using the preset offline data warehouse collaboration tool to control each target offline data warehouse management tool to process the called offline data, the embodiment of the present application summarizes, counts or analyzes the processed offline data to generate offline data processing results, and displays the generated offline data processing results in a graphical manner, such as charts, etc., which can help users understand the data more intuitively.

[0066] According to the offline data collaborative processing method proposed in the embodiment of the present application, by obtaining business needs and determining at least one target offline data warehouse management tool according to the business needs, the preset offline data warehouse collaborative tool is used to control each target offline data warehouse management tool to call the offline data in the preset distributed file system, and the preset offline data warehouse collaborative tool is used to control each target offline data warehouse management tool to process the called offline data. In this way, the problem that a single data warehouse tool cannot meet the data processing needs of different business scenarios and the data processing efficiency is low is solved, the flexibility of data processing can be improved, and the efficient use of data resources can be achieved.

[0067] Next, the offline data collaborative processing device proposed in accordance with the embodiment of the present application is described with reference to the accompanying drawings.

[0068] Figure 4 It is a block diagram of a collaborative processing device for offline data according to an embodiment of the present application.

[0069] like Figure 4 As shown, the offline data collaborative processing device 10 includes: an acquisition module 100 , a determination module 200 and a control module 300 .

[0070] The acquisition module 100 is used to acquire business requirements.

[0071] The determination module 200 is used to determine at least one target offline data warehouse management tool according to business needs, and use a preset offline data warehouse collaboration tool to control each target offline data warehouse management tool to call offline data in a preset distributed file system.

[0072] The control module 300 is used to control each target offline data warehouse management tool to process the called offline data by using the preset offline data warehouse collaboration tool.

[0073] Optionally, in some embodiments, before obtaining the business requirements, the obtaining module 100 includes: an obtaining unit and an associating unit.

[0074] The acquisition unit is used to acquire offline data.

[0075] The association unit is used to store offline data in a preset distributed file system, and the preset distributed file system is associated with at least one offline data warehouse management tool.

[0076] Optionally, in some embodiments, the control module 300 includes: a creation unit and a storage unit.

[0077] The creation unit is used to create a corresponding temporary work table using each target offline data warehouse management tool.

[0078] The storage unit is used to use the preset offline data warehouse collaboration tool to control each target offline data warehouse management tool to call the offline data in the target distributed file system, and store the offline data in the corresponding temporary table.

[0079] Optionally, in some embodiments, after using the preset offline data warehouse collaboration tool to control each target offline data warehouse management tool to process the called offline data, the determination module 200 further includes: a deletion unit.

[0080] The deletion unit is used to delete the offline data in each temporary worksheet.

[0081] Optionally, in some embodiments, after using a preset offline data warehouse collaboration tool to control each target offline data warehouse management tool to process the called offline data, the control module further includes: a visualization unit.

[0082] The visualization unit is used to generate and visualize offline data processing results.

[0083] It should be noted that the above explanation of the offline data collaborative processing method embodiment is also applicable to the offline data collaborative processing device of this embodiment, and will not be repeated here.

[0084] According to the offline data collaborative processing device proposed in the embodiment of the present application, by obtaining business needs and determining at least one target offline data warehouse management tool according to the business needs, the preset offline data warehouse collaborative tool is used to control each target offline data warehouse management tool to call the offline data in the preset distributed file system, and the preset offline data warehouse collaborative tool is used to control each target offline data warehouse management tool to process the called offline data. In this way, the problem that a single data warehouse tool cannot meet the data processing needs of different business scenarios and the data processing efficiency is low is solved, the flexibility of data processing can be improved, and the efficient use of data resources can be achieved.

[0085] Figure 5 A schematic diagram of the structure of a vehicle provided in an embodiment of the present application. The vehicle may include:

[0086] A memory 501 , a processor 502 , and a computer program stored in the memory 501 and executable on the processor 502 .

[0087] When the processor 502 executes the program, the offline data collaborative processing method provided in the above embodiment is implemented.

[0088] Furthermore, the vehicle also includes:

[0089] The communication interface 503 is used for communication between the memory 501 and the processor 502 .

[0090] The memory 501 is used to store computer programs that can be executed on the processor 502 .

[0091] The memory 501 may include a high-speed RAM (Random Access Memory) memory, and may also include a non-volatile memory, such as at least one disk memory.

[0092] If the memory 501, the processor 502 and the communication interface 503 are implemented independently, the communication interface 503, the memory 501 and the processor 502 can be connected to each other through a bus and communicate with each other. The bus can be an ISA (Industry Standard Architecture) bus, a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 5 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0093] Optionally, in a specific implementation, if the memory 501, the processor 502 and the communication interface 503 are integrated on a chip, the memory 501, the processor 502 and the communication interface 503 can communicate with each other through an internal interface.

[0094] The processor 502 may be a CPU (Central Processing Unit), or an ASIC (Application Specific Integrated Circuit), or one or more integrated circuits configured to implement the embodiments of the present application.

[0095] An embodiment of the present application also provides a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, the above-mentioned method for collaborative processing of offline data is implemented.

[0096] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representations of the above terms are not necessarily directed to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or N embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, without contradiction.

[0097] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include at least one of the features. In the description of this application, "N" means at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

[0098] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, fragment or portion of code comprising one or more executable instructions for implementing the steps of a custom logical function or process, and the scope of the preferred embodiments of the present application includes alternative implementations in which functions may not be performed in the order shown or discussed, including performing functions in a substantially simultaneous manner or in reverse order depending on the functions involved, which should be understood by technicians in the technical field to which the embodiments of the present application belong.

[0099] It should be understood that the various parts of the present application can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiment, the N steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array, a field programmable gate array, etc.

[0100] A person skilled in the art may understand that all or part of the steps in the method for implementing the above-mentioned embodiment may be completed by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, which, when executed, includes one or a combination of the steps of the method embodiment.

[0101] Although the embodiments of the present application have been shown and described above, it can be understood that the above embodiments are exemplary and cannot be understood as limitations on the present application. Ordinary technicians in this field can change, modify, replace and modify the above embodiments within the scope of the present application.

Claims

1. A collaborative processing method for offline data, characterized in that: The following steps are involved: Obtain business requirements; Determine at least one target offline data warehouse management tool according to the business requirements, and use a preset offline data warehouse collaboration tool to control each target offline data warehouse management tool to call offline data in a preset distributed file system; The preset offline data warehouse collaboration tool is used to control each target offline data warehouse management tool to process the called offline data.

2. The method according to claim 1, characterized in that Before obtaining business requirements, include: Get offline data; The offline data is stored in the preset distributed file system, and the preset distributed file system is associated with at least one offline data warehouse management tool.

3. The method according to claim 1, characterized in that The method of using a preset offline data warehouse collaboration tool to control each target offline data warehouse management tool to call the offline data in the target distributed file system includes: Use each target offline data warehouse management tool to create a corresponding temporary work table; The preset offline data warehouse collaboration tool is used to control each target offline data warehouse management tool to call the offline data in the target distributed file system, and store the offline data in the corresponding temporary table.

4. The method according to claim 3, characterized in that After using the preset offline data warehouse collaboration tool to control each target offline data warehouse management tool to process the called offline data, the method further includes: Delete the offline data in each temporary worksheet.

5. The method according to claim 1, characterized in that After using the preset offline data warehouse collaboration tool to control each target offline data warehouse management tool to process the called offline data, the method further includes: Generate offline data processing results, and visualize the offline data processing results.

6. A collaborative processing device for offline data, characterized in that: include: Acquisition module, used to obtain business requirements; A determination module, configured to determine at least one target offline data warehouse management tool according to the business requirements, and use a preset offline data warehouse collaboration tool to control each target offline data warehouse management tool to call offline data in a preset distributed file system; The control module is used to control each target offline data warehouse management tool to process the called offline data by using the preset offline data warehouse collaboration tool.

7. The device according to claim 6, characterized in that Before obtaining the business requirements, the obtaining module includes: An acquisition unit, used for acquiring offline data; An association unit is used to store the offline data in the preset distributed file system, and the preset distributed file system is associated with at least one offline data warehouse management tool.

8. The device according to claim 6, characterized in that The control module comprises: A creation unit is used to create a corresponding temporary work table using each target offline data warehouse management tool; The storage unit is used to use the preset offline data warehouse collaboration tool to control each target offline data warehouse management tool to call the offline data in the target distributed file system, and store the offline data in the corresponding temporary table.

9. A vehicle, characterized in that: include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the offline data collaborative processing method according to any one of claims 1 to 5.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: The program is executed by a processor to implement the offline data collaborative processing method as described in any one of claims 1 to 5.

Images