Data analysis method based on cloud computing

Abstract

The invention discloses a data analysis method based on cloud computing, and relates to the technical field of data processing, and the method comprises the following steps: S01, generating a to-be-scheduled data set from collected data based on a predetermined integration parameter plan; s02, a multi-channel data storage block used for storing a data set to be scheduled is generated in the cloud media library; s03, identifying the data set, arranging the data set to generate a corresponding Hive SQL (Structured Query Language) statement, synchronously calling a corresponding plan mode in the integrated parameter plan, and generating a Hive SQL statement request; s04, calling a Java database to connect with a JDBC connection technology to establish connection with a data warehouse server Hive Server based on distributed computing Hadoop, and responding to the Hive SQL statement request; and S05, the media library stores and receives the data returned by the Hive Server, and the multi-channel data storage block is returned to zero and formatted. The method solves the problem that errors occur in operation of patch codes and source codes.

Description

Technical Field

[0001] The present invention relates to the technical field of data processing, and in particular to a data analysis method based on cloud computing. Background Art

[0002] Data analysis means that after receiving the data to be analyzed from the log, the data is first checked and formatted using a data inspection script, and then preliminary processing is performed. On this basis, other heterogeneous scripts are used to perform other intermediate analysis. This involves the partitioning, splitting, and merging of relational databases. At the same time, the processing speed of relational databases under tens of millions of data volumes must also be considered, so that a balanced solution for database clusters is adopted, and data is pulled from databases on different servers and then aggregated using other scripts that complete specified requirements. This process needs to be "aggregated" in a very short time. For example, for a search engine term search, data needs to be aggregated within a few seconds.

[0003] Reference Publication (Announcement) Number: CN111460333A, Publication (Announcement) Date: 2020-07-28, discloses a real-time search data analysis system, which uses streaming computing big data technology to analyze and count the search behavior data generated by vertical search engines, and provide a visual display. In the data collection phase, the developed embedding technology is used to collect clicks and browsing behaviors left by users on the search interface, and at the same time collect the data used for the search. In the data analysis phase, offline and real-time are combined to store offline data, and the real-time data is analyzed and stored in the database system. In terms of search data display, in addition to visually displaying the analysis results, search engine technology is used to help users search and analyze results more conveniently and tap the potential value of the data. Users only need to enter the required data analysis dimension in the search box, and then click on the analysis result they want to view, and they can intuitively see the visualized chart results, which is more convenient and quick.

[0004] In the prior art including the above patents, the data is obtained by extracting the keywords of the entries, and the data to be extracted is pre-processed and stored in the form of keywords, so that the data is extracted after the keywords are matched. However, in the process of pre-processing the data, it is necessary to perform complex operations such as splitting, splitting, and merging of the relational database, and in the process of running the search engine, it is also necessary to perform adaptive patch repairs according to the changes in the data, so as to improve it. However, this patch-type expansion has a great disadvantage, because there will be a connection problem between the patch code and the source code of the search engine, so the patch code and the source code need to be repaired in the process, and the entire operation and maintenance method is extremely cumbersome and time-consuming. Summary of the invention

[0005] The purpose of the present invention is to provide a data analysis method based on cloud computing to solve the above problems.

[0006] In order to achieve the above object, the present invention provides the following technical solutions:

[0007] A data analysis method based on cloud computing includes the following steps:

[0008] S01. Generate a data set to be scheduled from the collected data based on a predetermined integration parameter plan;

[0009] S02, a cloud media library generates a multi-channel data storage block for storing a data set to be scheduled;

[0010] S03, identifying the data set and merging and collating it to generate a corresponding Hive SQL statement similar to the structured query language, synchronously calling the corresponding plan mode in the integrated parameter plan, and generating a Hive SQL statement request;

[0011] S04, calling the Java database connection JDBC connection technology to establish a connection with the Hive Server, a data warehouse server based on the distributed computing Hadoop, and responding to the Hive SQL statement request;

[0012] S05. The media library stores and receives data returned by the Hive Server, and resets and formats the multi-channel data storage blocks.

[0013] Preferably, the data set to be scheduled is generated in step S01 based on loading the plan to be scheduled from the integrated parameter plan, and calling the corresponding plan mode according to the plan type of the loaded plan;

[0014] Based on whether the plan mode is responsive, determine whether the corresponding plan mode called by the data parsing is correct, otherwise return to reallocate.

[0015] Preferably, in step S02, the multi-channel data storage block is temporarily stored, and the stored data set to be scheduled is classified and summarized, including:

[0016] S21, after loading the planning mode, dividing the stored data set to be scheduled into a number of sentences S, and analyzing each sentence S including a plurality of characters M;

[0017] S22, extracting key characters M according to the integrated parameter plan, and based on several types of key characters M contained in a plurality of sentences S under a complete data, and defining the complete data under a definition F;

[0018] S23. Determine the priority object with the highest priority based on the plan mode, and call the corresponding plan processing module to perform data analysis according to the plan type of the plan in the acquired priority object.

[0019] Preferably, the priority object in step S23 includes a main priority queue, which is based on the priority sorting of the processing under the multiple Fs determined in the planning mode, and generates multiple channel data storage blocks of the same level for multiple Fs at the same level to execute the data according to plan and perform data analysis.

[0020] Preferably, when determining multiple Fs of the same level under the main priority queue, the priorities of multiple priority objects of the same level are compared, the level of the key character M is determined for comparison, and the priority objects are sorted according to the comparison results.

[0021] Preferably, the step of calling the corresponding plan mode in the integrated parameter plan and generating a Hive SQL statement request in step S3 includes:

[0022] S31, after the planned mode exits operation or fails to operate normally, determine to load the objects to be scheduled into the planned mode, and encapsulate the objects whose execution time in the planned mode has arrived into priority objects;

[0023] S32, sending the priority object to the slave priority queue unit; and obtaining the priority object with the highest priority from the slave priority queue unit;

[0024] S33. Call the corresponding plan processing module to perform data analysis according to the object type of the object in the acquired priority object.

[0025] Preferably, in step 31, the objects arriving at the object execution time are encapsulated into priority objects, and a plan mode is initialized to carry them, and each plan mode calls the corresponding java database respectively.

[0026] As a preferred method, a plan mode is initialized for carrying, and for each parallel plan in other plans, the plan processing threads carrying each parallel plan are initialized in parallel, and the corresponding java database is called by each parallel initialized plan processing thread respectively.

[0027] In the above technical scheme, the present invention provides a data analysis method based on cloud computing, which has the following beneficial effects: it solves the problem of errors occurring in the operation of patch codes and source codes, and enables the patch codes and source codes to repair themselves and maintain smooth operation by autonomously calling the Java database through the code. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present invention. For ordinary technicians in this field, other drawings can also be obtained based on these drawings.

[0029] Figure 1 A schematic diagram of the structure of a process provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0030] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0031] like Figure 1 As shown, a data analysis method based on cloud computing includes the following steps:

[0032] S01. Generate a data set to be scheduled from the collected data based on a predetermined integration parameter plan;

[0033] S02, a cloud media library generates a multi-channel data storage block for storing a data set to be scheduled;

[0034] S03, identifying the data set, merging and collating it to generate a corresponding Hive SQL statement similar to the structured query language, synchronously calling the corresponding plan mode in the integrated parameter plan, and generating a Hive SQL statement request;

[0035] S04, calling the Java database connection JDBC connection technology to establish a connection with the Hive Server, a data warehouse server based on the distributed computing Hadoop, and responding to the Hive SQL statement request;

[0036] S05. The media library storage receives the data returned by the Hive Server and resets the multi-channel data storage block to zero and formats it.

[0037] As an embodiment further provided by the present invention, the data set to be scheduled is generated in step S01 based on loading the plan to be scheduled from the integrated parameter plan, and calling the corresponding plan mode according to the plan type of the loaded plan;

[0038] Based on whether the plan mode is responsive, determine whether the corresponding plan mode called by the data parsing is correct, otherwise return to reallocate.

[0039] As another embodiment further provided by the present invention, in step S02, the multi-channel data storage block is temporarily stored, and the data set to be scheduled is classified and summarized, including:

[0040] S21, after loading the planning mode, dividing the data set to be scheduled into a number of sentences S, and analyzing each sentence S including a plurality of characters M;

[0041] S22, extracting key characters M according to the integrated parameter plan, and based on several types of key characters M contained in multiple sentences S under a complete data, and defining the complete data under a definition F;

[0042] S23. Determine the priority object with the highest priority based on the plan mode, and call the corresponding plan processing module to perform data analysis according to the plan type of the plan in the acquired priority object.

[0043] Furthermore, the priority object in step S23 includes a main priority queue, which prioritizes the processing of multiple Fs determined in the planning mode, and generates multiple channel data storage blocks of the same level for multiple Fs at the same level to execute the data according to the plan and perform data analysis.

[0044] Secondly, after determining multiple Fs of the same order of magnitude under the main priority queue, the priorities of multiple priority objects of the same order are compared, the levels of the key characters M are determined for comparison, and the priority objects are sorted according to the comparison results.

[0045] As another embodiment further provided by the present invention, the step of calling the corresponding plan mode in the integrated parameter plan and generating a Hive SQL statement request in step S3 includes:

[0046] S31, after the planned mode exits operation or fails to operate normally, determine to load the objects to be scheduled into the planned mode, and encapsulate the objects whose execution time in the planned mode has arrived into priority objects;

[0047] S32, sending the priority object to the slave priority queue unit; and obtaining the priority object with the highest priority from the slave priority queue unit;

[0048] S33. Call the corresponding plan processing module to perform data analysis according to the object type of the object in the acquired priority object.

[0049] Furthermore, in step 31, the object arriving at the object execution time is encapsulated into a priority object, and a plan mode is initialized to carry it, and each plan mode calls the corresponding java database respectively.

[0050] Furthermore, a plan mode is initialized for carrying, and for each parallel plan in other plans, the plan processing threads carrying each parallel plan are initialized in parallel, and the corresponding java database is called by each parallel initialized plan processing thread respectively.

[0051] The above technology solves the problem of errors occurring during the operation of the patch code and the source code. By autonomously calling the java database through the code, the patch code and the source code can repair themselves and maintain smooth operation.

[0052] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0053] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0054] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0055] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1The steps for the functions specified in one or more boxes.

[0056] The present invention uses specific embodiments to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea. At the same time, for those skilled in the art, according to the idea of ​​the present invention, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on the present invention.

[0057] The embodiments of the present application also provide a specific implementation of an electronic device capable of implementing all steps of the method in the above embodiments, and the electronic device specifically includes the following contents:

[0058] Processor, memory, communications interface and bus;

[0059] Wherein, the processor, memory, and communication interface communicate with each other via the bus;

[0060] The processor is used to call the computer program in the memory, and when the processor executes the computer program, all the steps in the method in the above embodiment are implemented.

[0061] The embodiments of the present application also provide a computer-readable storage medium capable of implementing all the steps of the method in the above embodiments, wherein a computer program is stored on the computer-readable storage medium, and when the computer program is executed by a processor, all the steps of the method in the above embodiments are implemented.

[0062] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the hardware + program embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment. Although the embodiment of this specification provides the method operation steps described in the embodiment or flow chart, more or less operation steps can be included based on conventional or non-creative means. The order of steps listed in the embodiment is only one way of executing the order of many steps, and does not represent the only execution order. When the device or terminal product in practice is executed, it can be executed in sequence or in parallel according to the method shown in the embodiment or the accompanying drawings (for example, a parallel processor or a multi-threaded processing environment, or even a distributed data processing environment). The term "include", "include" or any other variant thereof is intended to cover non-exclusive inclusion, so that the process, method, product or device including a series of elements includes not only those elements, but also includes other elements that are not explicitly listed, or also includes elements inherent to such a process, method, product or device. In the absence of more restrictions, it is not excluded that there are other identical or equivalent elements in the process, method, product or device including the elements. For the convenience of description, the above device is described by dividing it into various modules according to its functions. Of course, when implementing the embodiments of this specification, the functions of each module can be implemented in the same one or more software and / or hardware, or the module implementing the same function can be implemented by a combination of multiple sub-modules or sub-units. The device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the coupling or direct coupling or communication connection between each other shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms. The present invention is described with reference to the flowchart and / or block diagram of the method, device (system) and computer program product according to the embodiments of the present invention. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the process and / or box in the flowchart and / or block diagram can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the process Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0063] Those skilled in the art will appreciate that the embodiments of this specification may be provided as methods, systems or computer program products. Therefore, the embodiments of this specification may be in the form of a complete hardware embodiment, a complete software embodiment or an embodiment combining software and hardware. Moreover, the embodiments of this specification may be in the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes. Each embodiment in this specification is described in a progressive manner, and the same and similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts refer to the partial description of the method embodiment. In the description of this specification, the description of the reference term "one embodiment", "some embodiments", "example", "specific example", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the embodiment of this specification.

[0064] In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, without contradicting each other. The above is only an embodiment of the embodiment of this specification and is not intended to limit the embodiment of this specification. For those skilled in the art, the embodiment of this specification may have various changes and variations. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the embodiment of this specification shall be included in the scope of the claims of the embodiment of this specification.

Claims

1. A data analysis method based on cloud computing, characterized in that: The following steps are involved: S01. Generate a data set to be scheduled from the collected data based on a predetermined integration parameter plan; S02, a cloud media library generates a multi-channel data storage block for storing a data set to be scheduled; S03, identifying the data set and merging and collating it to generate a corresponding Hive SQL statement similar to the structured query language, synchronously calling the corresponding plan mode in the integrated parameter plan, and generating a Hive SQL statement request; S04, calling the Java database connection JDBC connection technology to establish a connection with the Hive Server, a data warehouse server based on the distributed computing Hadoop, and responding to the Hive SQL statement request; S05. The media library stores and receives data returned by the Hive Server, and resets and formats the multi-channel data storage blocks.

2. The data analysis method based on cloud computing according to claim 1, characterized in that: The data set to be scheduled is generated in step S01 based on loading the plan to be scheduled from the integrated parameter plan, and calling the corresponding plan mode according to the plan type of the loaded plan; Based on whether the plan mode is responsive, determine whether the corresponding plan mode called by the data parsing is correct, otherwise return to reallocate.

3. The data analysis method based on cloud computing according to claim 1, characterized in that: In step S02, the multi-channel data storage block is temporarily stored, and the data set to be scheduled is classified and summarized, including: S21, after loading the planning mode, dividing the stored data set to be scheduled into a number of sentences S, and analyzing each sentence S including a plurality of characters M; S22, extracting key characters M according to the integrated parameter plan, and based on several types of key characters M contained in a plurality of sentences S under a complete data, and defining the complete data under a definition F; S23. Determine the priority object with the highest priority based on the plan mode, and call the corresponding plan processing module to perform data analysis according to the plan type of the plan in the acquired priority object.

4. The data analysis method based on cloud computing according to claim 3, characterized in that: The priority objects in step S23 include a main priority queue, which is based on the priority sorting of the processing of the multiple Fs determined in the planning mode, and generates multiple channel data storage blocks of the same level for the multiple Fs at the same level to execute the data according to the plan and perform data analysis.

5. The data analysis method based on cloud computing according to claim 4, characterized in that: After determining multiple Fs of the same order of magnitude under the main priority queue, the priorities of multiple priority objects of the same order are compared, the levels of the key characters M are determined for comparison, and the priority objects are sorted according to the comparison results.

6. The data analysis method based on cloud computing according to claim 1, characterized in that: The step of calling the corresponding plan mode in the integrated parameter plan and generating a Hive SQL statement request in step S3 includes: S31, after the planned mode exits operation or fails to operate normally, determine to load the objects to be scheduled into the planned mode, and encapsulate the objects whose execution time in the planned mode has arrived into priority objects; S32, sending the priority object to the slave priority queue unit; and obtaining the priority object with the highest priority from the slave priority queue unit; S33. Call the corresponding plan processing module to perform data analysis according to the object type of the object in the acquired priority object.

7. The data analysis method based on cloud computing according to claim 6, characterized in that: In step 31, the object arriving at the object execution time is encapsulated into a priority object, and a plan mode is initialized to carry it, and each plan mode calls the corresponding java database respectively.

8. The cloud computing-based data analysis method according to claim 7, characterized in that: Initialize a plan mode for carrying, and for each parallel plan in other plans, initialize the plan processing threads carrying each parallel plan in parallel, and each parallel initialized plan processing thread calls the corresponding java database respectively.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the steps of the cloud computing-based data analysis method according to any one of claims 1 to 8 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the cloud computing-based data analysis method described in any one of claims 1 to 8 are implemented.

Images