Real-time storage method and device for big data and medium

Abstract

The invention discloses a real-time storage method and device for big data and a medium, and the method comprises the steps: determining a plurality of preset virtual machines, carrying out environment building according to the plurality of virtual machines to determine an integrated environment, and carrying out the deployment of the integrated environment according to a plurality of pieces of preset software; determining a plurality of nodes in the deployed integrated environment, and performing distributed storage on the data through the plurality of nodes; and performing conversion processing on the distributed and stored data to determine streaming data, accessing the streaming data into a preset script, and executing the script to determine a streaming task, thereby determining a change condition of the data through the streaming task. Through optimization measures such as virtual machine integration, stream processing and storage software integration, processing of multiple data types and execution of flinksql task scripts, real-time storage and processing of big data are achieved, the storage efficiency, real-time performance and expandability of the system are improved, and powerful support is provided for big data application.

Description

Technical Field

[0001] The present application relates to the field of big data technology, and in particular to a real-time storage method, device and medium for big data. Background Art

[0002] At a time when the field of big data and artificial intelligence is developing rapidly, the storage process of massive data suffers from high latency and slow speed, which to a certain extent affects data processing efficiency, limits the work efficiency of managers, and increases the manpower and material costs of producers. At the same time, it is difficult to meet the data storage and processing needs in diverse scenarios. Summary of the invention

[0003] In order to solve the above problems, the present application proposes a real-time storage method for big data, including: determining a plurality of pre-set virtual machines, building an environment according to the plurality of virtual machines to determine an integrated environment, and deploying the integrated environment according to a plurality of pre-set software; determining a plurality of nodes in the integrated environment after deployment, and distributing and storing data through the plurality of nodes; converting and processing the distributed stored data to determine streaming data, connecting the streaming data to a pre-set script, and executing the script to determine a streaming task, thereby determining changes in the data through the streaming task.

[0004] In one example, the integrated environment is deployed according to multiple pre-set software, specifically including: determining the pre-set stream processing software and storage software, and integrating the stream processing software and the storage software; placing the compiled jar package of the storage software in the installation and deployment directory of the stream processing software, and determining all the jar packages in the installation and deployment directory of the stream processing software, and copying all the jar packages to the integration directory corresponding to the stream processing software.

[0005] In one example, distributing and storing data through the multiple nodes specifically includes: converting the data into a form to determine a data file; determining pre-set distributed software and the multiple nodes of the distributed software, and distributing and storing the data file through the multiple nodes.

[0006] In one example, the data after distributed storage is converted to determine streaming data, specifically including: determining a script and a client of a stream processing software, and transforming the script through the client to determine a stream processing task; determining a data type of the data, and determining a corresponding stream processing program according to the data type, so as to convert the corresponding data according to the stream processing program, wherein the data type includes offline data and streaming data.

[0007] In one example, the method further includes: determining the data type; if the data type is offline data, capturing the change data of the relational database through CDC technology, including changes in table structure, addition and deletion of data, reflecting such changes in a mapping table, and then synchronizing such changes to the storage software to realize the conversion of static data streams into streaming data.

[0008] In one example, the method also includes: judging the data type. If the data type is streaming data, specifying the connector as kafka when writing tasks in the stream processing software, mapping the data of the topic in kafka to flinkcdc, and directly pulling the data of the specified kafka topic to the flinkcdc client in real time, and then writing a derivative task to insert the data in flinkcdc into the storage software to access the streaming data.

[0009] In an example, the method further includes: executing a written flinksql task script, starting a flink cluster and a flink-sql client, and executing the flinksql task script in the client to suspend a streaming processing task to wait for upstream data.

[0010] In one example, the method also includes: determining the source of the upstream data, inserting the data produced by the source into MySQL, or pushing the data produced by the source to a Kafka topic, mapping and identifying the streaming processing task to determine the changes in the data, and then performing synchronization processing based on the changed data.

[0011] On the other hand, the present application also proposes a real-time storage device for big data, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the real-time storage device for big data can execute: determining a plurality of pre-set virtual machines, building an environment based on the plurality of virtual machines to determine an integrated environment, and deploying the integrated environment based on a plurality of pre-set software; determining a plurality of nodes in the integrated environment after deployment, and distributing and storing data through the plurality of nodes; converting the data after distributed storage to determine streaming data, connecting the streaming data to a pre-set script, and executing the script to determine a streaming task, thereby determining changes in the data through the streaming task.

[0012] On the other hand, the present application also proposes a non-volatile computer storage medium storing computer executable instructions, wherein the computer executable instructions are configured to: determine a plurality of pre-set virtual machines, build an environment based on the plurality of virtual machines to determine an integrated environment, and deploy the integrated environment based on a plurality of pre-set software; determine a plurality of nodes in the integrated environment after deployment, and distribute and store data through the plurality of nodes; convert and process the distributed and stored data to determine streaming data, connect the streaming data to a pre-set script, and execute the script to determine a streaming task, thereby determining changes in the data through the streaming task.

[0013] This application realizes the distributed storage of data and improves the storage efficiency and security of data through the deployment of virtual machine integrated environment and multiple software. At the same time, the setting of multiple nodes also enhances the scalability and fault tolerance of the system, so that the system can cope with the storage needs of large-scale data. Through the integration of stream processing software and storage software, the streaming processing of data is realized, so that the changes in data can be captured and processed in real time, which improves the real-time nature of data and enables the system to better cope with the challenges brought by data changes. For offline data, the change data of the relational database is captured by CDC technology, and the conversion of static data flow to streaming data is realized; for streaming data, real-time pulling and processing of data is realized by writing flink tasks. This flexibility enables the system to process various types of data and meet the needs of different scenarios. Through the execution of flinksql task scripts, the flink cluster and flink-sql client are started, and the functions of suspending and waiting for upstream data of streaming processing tasks are realized, which improves the automation of the system and enables the system to better cope with the real-time and accuracy requirements of data. This application realizes real-time storage and processing of big data through optimization measures such as virtual machine integration, integration of stream processing and storage software, processing of multiple data types, and execution of flinksql task scripts, improves the storage efficiency, real-time and scalability of the system, and provides strong support for big data applications. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0015] Figure 1 A flowchart of a real-time storage method for big data in an embodiment of the present application;

[0016] Figure 2 This is a schematic diagram of a real-time storage device for big data in an embodiment of the present application. DETAILED DESCRIPTION

[0017] In order to make the purpose, technical solution and advantages of the present application clearer, the technical solution of the present application will be clearly and completely described below in combination with the specific embodiments of the present application and the corresponding drawings. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present application.

[0018] The technical solutions provided by various embodiments of the present application are described in detail below in conjunction with the accompanying drawings.

[0019] like Figure 1 As shown, in order to solve the above problems, an embodiment of the present application provides a real-time storage method for big data, the method comprising:

[0020] S101, determining a plurality of pre-set virtual machines, building an environment according to the plurality of virtual machines to determine an integrated environment, and deploying the integrated environment according to a plurality of pre-set software.

[0021] Prepare 3 virtual machines with specifications of 16U64G and installed with CentOS system. The 3 virtual machines are on the same network platform and can communicate with each other.

[0022] Using Datasophon 1.2.1, follow the steps below to deploy the specified versions of the following software components: ZooKeeper, Kafka, Kerberos, HDFS, YARN, Hadoop, Flink, Hudi, and Dinky. ZooKeeper is an open source distributed coordination service that maintains configuration information, naming, provides distributed synchronization, and provides group services. Kafka is a distributed stream processing platform that can handle high throughput data. Kerberos is a network authentication protocol that is used to authenticate users through keys. HDFS is a distributed file system. Yet Another ResourceNegotiator is a resource manager for Hadoop. Hadoop is a distributed system infrastructure developed by the Apache Foundation. Flink is an open source stream processing framework for processing unbounded and bounded data streams. Hudi is an open source data lake storage platform that is used to build an efficient data lake on Hadoop or cloud storage. Dinky refers to a component or tool integrated with Flink, Hudi, etc. The deployment process strictly follows the Datasophon deployment guide. After the deployment of the above components is completed, integrate Flink with Hudi and Dinky. Place the compiled JAR package of Hudi in the lib folder under the Flink installation directory, and copy all JAR packages in the Flink lib directory to the subfolder corresponding to the Flink version in the extend directory of Dinky.

[0023] S102: Determine a plurality of nodes in the integrated environment after deployment, and distribute and store data through the plurality of nodes.

[0024] The distributed storage architecture allows data to be evenly distributed and stored on multiple nodes, thereby significantly improving data storage efficiency and scalability. Data is stored in the HDFS (Hadoop Distributed File System) file system in the form of files. HDFS is a specially designed distributed file system that manages multiple nodes to store file data in a distributed manner. When building the HDFS file system, we should use disks with high IO read and write performance to improve the system throughput. In the HDFS configuration file hdfs-site.xml, nameservice needs to be configured, and in order to achieve high availability (HA) features, corresponding HA configuration is also required.

[0025] By using the open data storage format Hudi, we can store and write massive amounts of data into Hudi tables. The metadata and business data of Hudi tables are actually stored in the HDFS file system. When creating a Hudi table, you need to specify the connector type as Hudi in the table creation statement and set the file storage path in HDFS.

[0026] S103, converting the data after distributed storage to determine streaming data, connecting the streaming data to a pre-set script, and executing the script to determine a streaming task, thereby determining changes in the data through the streaming task.

[0027] Using the capabilities of Flink, a real-time data stream processing technology, we use the Kappa architecture to treat all data to be stored as streaming data. By writing a Flink SQL task script and converting it into a Flink task in the Flink SQL client for execution, real-time access to streaming data is achieved. For static offline data, we use CDC (Change Data Capture) technology to capture change data in relational databases, including changes in table structure and data addition, deletion, and modification operations. These change data are reflected in the binlog file of the database engine. CDC technology reads and parses these binlog log files, reflects them in the mapping table of Flink SQL CDC, and then synchronizes these changes to the Hudi table, thereby achieving the purpose of converting static data streams into streaming data.

[0028] For streaming data from the message queue, we take Kafka as an example. When writing the Flink SQL task, we specify the connector as Kafka. By mapping the topic data in Kafka to Flink SQL CDC, we can pull the data of the specified Kafka topic to the Flink SQL CDC client in real time. Then, we write a data import task to insert the data in Flink SQL CDC into the Hudi table to achieve the purpose of accessing streaming data.

[0029] After writing the Flink SQL task script, follow these steps to start the task: First, go to the bin path of the Flink installation directory in your environment. Start the Flink cluster by executing the . / yarn-session.sh script. Then, in the same directory, start the Flink SQL client. This will take you to the Flink SQL task execution interface. In this interface, paste or enter the pre-written Flink SQL task script content and execute it. After execution, a streaming processing task will be suspended, which is ready to receive and process data from upstream at any time.

[0030] After the task is suspended, the upstream data source, such as MySQL or Kafka, starts producing data. When inserting data into MySQL or pushing data to a specified Kafka topic, the suspended task will detect data changes in real time through CDC (for MySQL) or Kafka mapping (for Kafka topic). Once the data changes are identified, the task will immediately synchronize the newly changed data for processing.

[0031] like Figure 2 As shown, the embodiment of the present application also provides a real-time storage device for big data, including:

[0032] at least one processor; and,

[0033] a memory communicatively connected to at least one processor; wherein,

[0034] The memory stores instructions that can be executed by at least one processor, and the instructions are executed by at least one processor to enable a real-time storage device for big data to perform:

[0035] Determine a plurality of pre-set virtual machines, build an environment according to the plurality of virtual machines to determine an integrated environment, and deploy the integrated environment according to the plurality of pre-set software;

[0036] Determine a plurality of nodes in the integrated environment after deployment, and distribute and store data through the plurality of nodes;

[0037] The data after distributed storage is converted to determine streaming data, the streaming data is connected to a pre-set script, and the script is executed to determine a streaming task, so as to determine the change of the data through the streaming task.

[0038] The embodiment of the present application further provides a non-volatile computer storage medium storing computer executable instructions, wherein the computer executable instructions are configured as follows:

[0039] Determine a plurality of pre-set virtual machines, build an environment according to the plurality of virtual machines to determine an integrated environment, and deploy the integrated environment according to the plurality of pre-set software;

[0040] Determine a plurality of nodes in the integrated environment after deployment, and distribute and store data through the plurality of nodes;

[0041] The data after distributed storage is converted to determine streaming data, the streaming data is connected to a pre-set script, and the script is executed to determine a streaming task, so as to determine the change of the data through the streaming task.

[0042] In the 1990s, improvements to a technology could be clearly distinguished as hardware improvements (for example, improvements to the circuit structure of diodes, transistors, switches, etc.) or software improvements (improvements to the method flow). However, with the development of technology, many improvements to the method flow today can be regarded as direct improvements to the hardware circuit structure. Designers almost always obtain the corresponding hardware circuit structure by programming the improved method flow into the hardware circuit. Therefore, it cannot be said that an improvement in a method flow cannot be implemented using a hardware entity module. For example, a programmable logic device (PLD) (such as a field programmable gate array (FPGA)) is such an integrated circuit whose logical function is determined by the user's programming of the device. Designers can "integrate" a digital system on a PLD by programming it themselves, without having to ask a chip manufacturer to design and produce a dedicated integrated circuit chip. Moreover, nowadays, instead of manually making integrated circuit chips, this kind of programming is mostly implemented by "logic compiler" software, which is similar to the software compiler used when developing and writing programs, and the original code before compilation must also be written in a specific programming language, which is called hardware description language (HDL). There is not only one HDL, but many kinds, such as ABEL (Advanced Boolean Expression Language), AHDL (Altera Hardware Description Language), Confluence, CUPL (Cornell University Programming Language), HDCal, JHDL (Java Hardware Description Language), Lava, Lola, MyHDL, PALASM, RHDL (Ruby Hardware Description Language), etc. The most commonly used ones are VHDL (Very-High-Speed ​​Integrated Circuit Hardware Description Language) and Verilog. Those skilled in the art should also know that it is only necessary to program the method flow slightly in the above-mentioned hardware description languages ​​and program it into the integrated circuit, and then it is easy to obtain the hardware circuit that implements the logic method flow.

[0043] The controller can be implemented in any appropriate manner, for example, the controller can take the form of a microprocessor or processor and a computer-readable medium storing a computer-readable program code (such as software or firmware) that can be executed by the (micro)processor, a logic gate, a switch, an application-specific integrated circuit (ASIC), a programmable logic controller, and an embedded microcontroller. Examples of controllers include, but are not limited to, the following microcontrollers: ARC 625D, Atmel AT91SAM, Microchip PIC18F26K20, and Silicone Labs C8051F320. The memory controller can also be implemented as part of the control logic of the memory. Those skilled in the art also know that in addition to implementing the controller in a purely computer-readable program code manner, the controller can be implemented in the form of a logic gate, a switch, an application-specific integrated circuit, a programmable logic controller, and an embedded microcontroller by logically programming the method steps. Therefore, this controller can be considered as a hardware component, and the devices included therein for implementing various functions can also be regarded as structures within the hardware component. Or even, the devices for implementing various functions can be regarded as both software modules for implementing the method and structures within the hardware component.

[0044] The systems, devices, modules or units described in the above embodiments may be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, the computer may be, for example, a personal computer, a laptop computer, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.

[0045] For the convenience of description, the above device is described in various units according to their functions. Of course, when implementing this specification, the functions of each unit can be implemented in the same or multiple software and / or hardware.

[0046] Each embodiment in this application is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device and medium embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiments.

[0047] The devices and media provided in the embodiments of the present application correspond one-to-one to the methods. Therefore, the devices and media also have similar beneficial technical effects as the corresponding methods. Since the beneficial technical effects of the methods have been described in detail above, the beneficial technical effects of the devices and media will not be repeated here.

[0048] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0049] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0050] These computer program instructions may also be stored in a computer readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0051] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0052] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0053] The memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0054] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0055] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0056] The above are only embodiments of the present application and are not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included within the scope of the claims of the present application.

Claims

1. A real-time storage method for big data, characterized in that: include: Determine a plurality of pre-set virtual machines, build an environment according to the plurality of virtual machines to determine an integrated environment, and deploy the integrated environment according to the plurality of pre-set software; Determine a plurality of nodes in the integrated environment after deployment, and distribute and store data through the plurality of nodes; The data after distributed storage is converted to determine streaming data, the streaming data is connected to a pre-set script, and the script is executed to determine a streaming task, so as to determine the change of the data through the streaming task.

2. The method according to claim 1, characterized in that The integrated environment is deployed according to the pre-set multiple software, specifically including: Determining pre-set stream processing software and storage software, and integrating the stream processing software and the storage software; The compiled jar package of the storage software is placed in the installation and deployment directory of the stream processing software, and all jar packages in the installation and deployment directory of the stream processing software are determined, and all the jar packages are copied to the integration directory corresponding to the stream processing software.

3. The method according to claim 1, characterized in that Distributing and storing data through the multiple nodes specifically includes: Converting the data into a certain format to determine a data file; Determine the pre-set distributed software, and determine the multiple nodes of the distributed software, and perform distributed storage on the data file through the multiple nodes.

4. The method according to claim 1, characterized in that: The distributed and stored data is converted to determine streaming data, specifically including: Determine a script and a client of the stream processing software, and transform the script through the client to determine the stream processing task; Determine the data type of the data, determine the corresponding stream processing program according to the data type, and convert and process the corresponding data according to the stream processing program, wherein the data type includes offline data and streaming data.

5. The method according to claim 1, characterized in that: The method further comprises: Determine the data type. If the data type is offline data, capture the change data of the relational database through CDC technology, including table structure changes, data addition and deletion changes, reflect the changes in the mapping table, and then synchronize the changes to the storage software to realize the conversion of static data streams into streaming data.

6. The method according to claim 1, characterized in that The method further comprises: Determine the data type. If the data type is streaming data, specify the connector as Kafka when writing tasks in the stream processing software. Map the data of the topic in Kafka to FlinkCDC, and pull the data of the specified KafkaTopic directly to the FlinkCDC client in real time. Then write a derivative task to insert the data in FlinkCDC into the storage software to access the streaming data.

7. The method according to claim 6, characterized in that The method further comprises: Execute the written flinksql task script, start the flink cluster and flink-sql client, and execute the flinksql task script in the client to suspend a streaming processing task to wait for upstream data.

8. The method according to claim 7, characterized in that The method further comprises: Determine the source of the upstream data, insert the data produced by the source into MySQL, or push the data produced by the source to Kafka topic, map and identify the streaming processing task to determine the data changes, and then perform synchronization processing according to the changed data.

9. A real-time storage device for big data, characterized in that: include: at least one processor; as well as, a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the real-time storage device for big data to perform: Determine a plurality of pre-set virtual machines, build an environment according to the plurality of virtual machines to determine an integrated environment, and deploy the integrated environment according to the plurality of pre-set software; Determine a plurality of nodes in the integrated environment after deployment, and distribute and store data through the plurality of nodes; The data after distributed storage is converted to determine streaming data, the streaming data is connected to a pre-set script, and the script is executed to determine a streaming task, so as to determine the change of the data through the streaming task.

10. A non-volatile computer storage medium storing computer executable instructions, characterized in that: The computer executable instructions are configured to: Determine a plurality of pre-set virtual machines, build an environment according to the plurality of virtual machines to determine an integrated environment, and deploy the integrated environment according to the plurality of pre-set software; Determine a plurality of nodes in the integrated environment after deployment, and distribute and store data through the plurality of nodes; The data after distributed storage is converted to determine streaming data, the streaming data is connected to a pre-set script, and the script is executed to determine a streaming task, so as to determine the change of the data through the streaming task.

Images