Large data implementation method and device, electronic equipment and computer readable storage medium

Abstract

The invention relates to the technical field of data implementation, and discloses a large data implementation method. Comprising the steps of obtaining multiple pieces of data source information, and integrating the multiple pieces of data source information to generate to-be-processed data; performing parallel processing on the to-be-processed data through CSV Load and Flink real-time writing, and generating pre-processed data; matching the preprocessed data with the order number details, generating matching result information, and sending the matching result information to a data warehouse; the scheduling tasks are processed in batches through Flink, and data are pushed to a cache Tenis platform; acquiring data query information, calling data in the cached Tenis platform according to the data query information to generate query result information, and sending the query result information to the client; the method has the advantage of more efficient data processing.

Description

Technical Field

[0001] The present invention relates to the field of data implementation technology, and in particular to a large-scale data implementation method, device, electronic device and computer-readable storage medium. Background Art

[0002] Logistics and transportation provide great convenience for people in today's society. It is particularly important to process, retrieve and analyze a large amount of logistics data. Currently, complex logistics data is not standardized and query paths are not unified; When analyzing data, most traditional database models are used, which require regular batch calculations to generate statistical analysis reports. This results in high latency and low concurrency in report generation, making it impossible to achieve real-time data statistical analysis; Large amounts of offline computing batch processing also put pressure on system performance.

[0003] Therefore, a large data implementation method, device, electronic device and computer-readable storage medium are provided to solve the above problems. Summary of the invention

[0004] The main purpose of the present invention is to solve the problems of non-standardization of complex logistics data and inconsistent query paths; when analyzing data, most of them adopt the traditional database model, and need to perform batch calculations regularly to generate statistical analysis reports; this leads to a certain delay in report generation, low concurrency, and inability to achieve real-time data statistical analysis; a large number of offline batch calculations also put pressure on system performance and other problems.

[0005] A first aspect of the present invention provides a large data implementation method, the large data implementation method comprising: Acquire information from multiple data sources, and integrate the information from multiple data sources to generate data to be processed; Use CSV Load and Flink real-time writing to process the data in parallel and generate pre-processed data. Match the pre-processed data with the order number details, generate matching result information and send the matching result information to the data warehouse; Use Flink to process batch scheduling tasks and push data to the cache Tendis platform; Obtain data query information, retrieve data cached in the Tendis platform based on the data query information to generate query result information, and send the query result information to the client.

[0006] Optionally, the obtaining of multiple data source information and integrating the multiple data source information to generate data to be processed includes; Acquire multiple data source information, wherein the multiple data source information includes delivery weight data information, volume image status information and customer weight information; Receive data source information through a unified data interface or protocol; Specific data is identified and processed through data scanning.

[0007] Optionally, the step of performing parallel processing of the data to be processed by CSV Load and Flink real-time writing to generate pre-processed data includes: Regularly batch load the data to be processed via CSV Load; Write the data to be processed in real time through Flink; The loaded data to be processed and the written data to be processed are pre-processed data.

[0008] Optionally, the batch processing scheduling task using Flink includes: Receive job submission requests and perform initialization work according to the requirements and configuration of the job submission requests; Build a job graph based on the logical structure and dependencies of the job; According to the dependencies between operators in the job graph, the task execution order is determined, and the tasks are assigned to the corresponding TaskExecutor for execution; Monitor the execution status of tasks and issue alarms for abnormally executed tasks; Complete the task and summarize the execution results.

[0009] Optionally, the acquiring of data query information, retrieving data cached in the Tendis platform according to the data query information to generate query result information, and sending the query result information to the client includes: Obtaining data query information, and extracting multiple data query requests in the data query information; For multiple data query requests, the data in the cached Tendis platform is retrieved through SQL to generate query result information; The query results are distributed cached and sent to the client.

[0010] Optionally, the identifying process of specific data by data scanning includes: Identify and process missing data through data scanning; Identify and process erroneous and missing data through data scanning.

[0011] A second aspect of the present invention provides a large data implementation device, comprising: A data source acquisition module is used to acquire information of multiple data sources and integrate the information of multiple data sources to generate data to be processed; The data preprocessing module is used to process the data to be processed in parallel through CSV Load and Flink real-time writing to generate preprocessed data; The data matching module is used to match the pre-processed data with the order number details, generate matching result information and send the matching result information to the data warehouse; The scheduling cache module is used to use Flink to batch process scheduling tasks and push data to the cache Tendis platform; The data query module is used to obtain data query information, retrieve the data cached in the Tendis platform according to the data query information, generate query result information, and send the query result information to the client.

[0012] A third aspect of the present invention provides an electronic device, the electronic device comprising a memory and at least one processor, the memory storing instructions; The at least one processor calls the instructions in the memory to enable the electronic device to execute the various steps of the large data implementation method described above.

[0013] A fourth aspect of the present invention provides a computer-readable storage medium having instructions stored thereon, and the instructions, when executed by a processor, implement the various steps of the large data implementation method described above.

[0014] The technical solution of the present invention provides a large-scale data implementation method, which integrates data from different data sources, realizes the fusion and correlation analysis of multi-source data, and provides a more comprehensive business perspective; utilizes real-time computing frameworks such as Flink to perform online analysis and prediction of real-time data streams to provide real-time business insights; develops an adaptive data scheduling algorithm to dynamically adjust the allocation and priority of data processing tasks according to system load and data volume; implements a load balancing mechanism to ensure that the system can still maintain efficient and stable operation in high-concurrency scenarios; introduces distributed cache Tendis and combines it with load balancing technology to distribute query requests to multiple cache nodes, realizes parallel processing and load balancing of query requests, and improves the overall throughput and concurrent processing capabilities of the system. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 A flowchart of a large data implementation method provided for the first embodiment of the present invention; Figure 2 A flowchart of a large data implementation method provided for a second embodiment of the present invention; Figure 3 A flowchart of a large data implementation method provided by a third embodiment of the present invention; Figure 4 A schematic diagram of the structure of a large data implementation device provided by an embodiment of the present invention; Figure 5 A schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0016] The embodiment of the present invention provides a large-scale data implementation method, including obtaining information of multiple data sources, integrating the information of multiple data sources to generate data to be processed; processing the data to be processed in parallel through CSV Load and Flink real-time writing to generate pre-processed data; matching the pre-processed data with the order number details, generating matching result information and sending the matching result information to the data warehouse; using Flink batch processing scheduling tasks to push data to the cache Tendis platform; obtaining data query information, calling the data in the cache Tendis platform according to the data query information to generate query result information, and sending the query result information to the client. The present invention solves the problems of the current complex logistics data being non-standard and the query path being non-uniform; when analyzing data, most of them adopt the traditional database mode, and need to perform batch calculations regularly to generate statistical analysis reports; this leads to a certain delay in report generation, low concurrency, and inability to achieve real-time data statistical analysis; a large number of offline calculation batches also put pressure on system performance.

[0017] The terms "first", "second", "third", "fourth", etc. (if any) in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments described herein can be implemented in an order other than that illustrated or described herein. In addition, the terms "including" or "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0018] For ease of understanding, the specific process of the embodiment of the present invention is described below. Figure 1 The first embodiment of the large data implementation method in the embodiment of the present invention includes: Acquire information from multiple data sources, and integrate the information from multiple data sources to generate data to be processed; Use CSV Load and Flink real-time writing to process the data in parallel and generate pre-processed data. Match the pre-processed data with the order number details, generate matching result information and send the matching result information to the data warehouse; Use Flink to process batch scheduling tasks and push data to the cache Tendis platform; Obtain data query information, retrieve data cached in the Tendis platform based on the data query information to generate query result information, and send the query result information to the client.

[0019] See also Figure 2 The second embodiment of the large data implementation method in the embodiment of the present invention includes: Acquire multiple data source information, and integrate the multiple data source information to generate data to be processed; specifically including: acquire multiple data source information, the multiple data source information includes delivery weight data information, volume image status information and customer weight information; receive the data source information through a unified data interface or protocol; identify and process specific data through data scanning.

[0020] The identification and processing of specific data specifically includes: Identify and process missing data through data scanning; Identify and process erroneous and missing data through data scanning.

[0021] The data to be processed is processed in parallel through CSV Load and Flink real-time writing to generate preprocessed data. Specifically, the data to be processed is loaded in batches periodically through CSV Load; the data to be processed is written in real time through Flink; the loaded data to be processed and the written data to be processed are preprocessed data.

[0022] Match the pre-processed data with the order number details, generate matching result information and send the matching result information to the data warehouse; Use Flink to process batch scheduling tasks and push data to the cache Tendis platform; Obtain data query information, retrieve data in the cache Tendis platform according to the data query information to generate query result information, and send the query result information to the client; specifically, it includes: obtaining data query information, extracting multiple data query requests in the data query information; retrieving data in the cache Tendis platform through SQL for multiple data query requests to generate query result information; performing distributed caching on the query results, and sending the query results to the client at the same time.

[0023] See also Figure 3 , the third embodiment of the large data implementation method in the embodiment of the present invention includes: Acquire multiple data source information, and integrate the multiple data source information to generate data to be processed; specifically including: acquire multiple data source information, the multiple data source information includes delivery weight data information, volume image status information and customer weight information; receive the data source information through a unified data interface or protocol; identify and process specific data through data scanning.

[0024] The identification and processing of specific data specifically includes: Identify and process missing data through data scanning; Identify and process erroneous and missing data through data scanning.

[0025] The data to be processed is processed in parallel through CSV Load and Flink real-time writing to generate preprocessed data. Specifically, the data to be processed is loaded in batches periodically through CSV Load; the data to be processed is written in real time through Flink; the loaded data to be processed and the written data to be processed are preprocessed data.

[0026] Match the pre-processed data with the order number details, generate matching result information and send the matching result information to the data warehouse; Use Flink to process batch scheduling tasks and push data to the cache Tendis platform; The specific process of Flink batch processing scheduling tasks includes: Receive job submission requests and perform initialization work according to the requirements and configuration of the job submission requests; Build a job graph based on the logical structure and dependencies of the job; According to the dependencies between operators in the job graph, the task execution order is determined, and the tasks are assigned to the corresponding TaskExecutor for execution; Monitor the execution status of tasks and issue alarms for abnormally executed tasks; Complete the task and summarize the execution results.

[0027] Obtain data query information, retrieve data in the cache Tendis platform according to the data query information to generate query result information, and send the query result information to the client; specifically, it includes: obtaining data query information, extracting multiple data query requests in the data query information; retrieving data in the cache Tendis platform through SQL for multiple data query requests to generate query result information; performing distributed caching on the query results, and sending the query results to the client at the same time.

[0028] The above describes the large data implementation method in the embodiment of the present invention. The following describes the large data implementation device in the embodiment of the present invention. Figure 4In the embodiment of the present invention, the large data implementation device includes: The data source acquisition module 401 is used to acquire information of multiple data sources and integrate the information of multiple data sources to generate data to be processed; The data preprocessing module 402 is used to process the data to be processed in parallel through CSV Load and Flink real-time writing to generate preprocessed data; The data matching module 403 is used to match the pre-processed data with the order number details, generate matching result information and send the matching result information to the data warehouse; The scheduling cache module 404 is used to use Flink to batch process scheduling tasks and push data to the cache Tendis platform; The data query module 405 is used to obtain data query information, retrieve data cached in the Tendis platform according to the data query information, generate query result information, and send the query result information to the client.

[0029] above Figure 4 The large data implementation device in the embodiment of the present invention is described in detail from the perspective of modular functional entities, and the electronic device in the embodiment of the present invention is described in detail from the perspective of hardware processing.

[0030] Figure 5 7 is a schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. The electronic device 700 may have relatively large differences due to different configurations or performances, and may include one or more processors (central processing units, CPU) 710 (for example, one or more processors) and a memory 720, and one or more storage media 730 (for example, one or more mass storage devices) storing application programs 733 or data 732. Among them, the memory 720 and the storage medium 730 can be temporary storage or permanent storage. The program stored in the storage medium 730 may include one or more modules (not shown in the figure), and each module may include a series of instruction operations in the electronic device 700. Furthermore, the processor 710 may be configured to communicate with the storage medium 730 to execute a series of instruction operations in the storage medium 730 on the electronic device 700.

[0031] The electronic device 700 may also include one or more power supplies 740, one or more wired or wireless network interfaces 750, one or more input and output interfaces 750, and / or one or more operating systems 731, such as Windows Serve, Mac OS X, Unix, Linux, FreeBSD, etc. It will be appreciated by those skilled in the art that Figure 5The structure of the electronic device shown does not constitute a limitation on the electronic device, and may include more or less components than shown in the figure, or combine some components, or arrange the components differently.

[0032] The present invention also provides a computer-readable storage medium, which may be a non-volatile computer-readable storage medium or a volatile computer-readable storage medium. Instructions are stored in the computer-readable storage medium, and when the instructions are executed on a computer, the computer executes the steps of the large data implementation method.

[0033] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the above-described system, device, or unit can refer to the corresponding process in the aforementioned method embodiment and will not be repeated here.

[0034] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art or the whole or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk and other media that can store program code.

[0035] As described above, the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features thereof may be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A large data implementation method, characterized in that: The large data implementation method comprises: Acquire information from multiple data sources, and integrate the information from multiple data sources to generate data to be processed; Use CSV Load and Flink real-time writing to process the data in parallel and generate pre-processed data. Match the pre-processed data with the order number details, generate matching result information and send the matching result information to the data warehouse; Use Flink to process batch scheduling tasks and push data to the cache Tendis platform; Obtain data query information, retrieve data cached in the Tendis platform based on the data query information to generate query result information, and send the query result information to the client.

2. The large data implementation method according to claim 1, characterized in that: The obtaining of multiple data source information and integrating the multiple data source information to generate data to be processed includes; Acquire multiple data source information, wherein the multiple data source information includes delivery weight data information, volume image status information and customer weight information; Receive data source information through a unified data interface or protocol; Specific data is identified and processed through data scanning.

3. The large data implementation method according to claim 2, characterized in that: The method of parallel processing of the data to be processed by CSV Load and Flink real-time writing to generate pre-processed data includes: Regularly batch load the data to be processed via CSV Load; Write the data to be processed in real time through Flink; The loaded data to be processed and the written data to be processed are pre-processed data.

4. The large data implementation method according to claim 3, characterized in that: The use of Flink to batch process scheduling tasks includes: Receive job submission requests and perform initialization work according to the requirements and configuration of the job submission requests; Build a job graph based on the logical structure and dependencies of the job; According to the dependencies between operators in the job graph, the task execution order is determined, and the tasks are assigned to the corresponding TaskExecutor for execution; Monitor the execution status of tasks and issue alarms for abnormally executed tasks; Complete the task and summarize the execution results.

5. The large data implementation method according to claim 4, characterized in that: The obtaining of data query information, retrieving data cached in the Tendis platform according to the data query information to generate query result information, and sending the query result information to the client comprises: Obtaining data query information, and extracting multiple data query requests in the data query information; For multiple data query requests, the data in the cached Tendis platform is retrieved through SQL to generate query result information; The query results are distributed cached and sent to the client.

6. The large data implementation method according to claim 5, characterized in that: The identification process of specific data by data scanning includes: Identify and process missing data through data scanning; Identify and process erroneous and missing data through data scanning.

7. A large data implementation device, characterized in that: include: A data source acquisition module is used to acquire information of multiple data sources and integrate the information of multiple data sources to generate data to be processed; The data preprocessing module is used to process the data to be processed in parallel through CSV Load and Flink real-time writing to generate preprocessed data; The data matching module is used to match the pre-processed data with the order number details, generate matching result information and send the matching result information to the data warehouse; The scheduling cache module is used to use Flink to batch process scheduling tasks and push data to the cache Tendis platform; The data query module is used to obtain data query information, retrieve the data cached in the Tendis platform according to the data query information, generate query result information, and send the query result information to the client.

8. An electronic device, comprising a memory and at least one processor, wherein instructions are stored in the memory; The at least one processor calls the instructions in the memory to enable the electronic device to execute each step of the large data implementation method as described in any one of claims 1-6.

9. A computer-readable storage medium having instructions stored thereon, characterized in that: When the instructions are executed by a processor, the steps of the large data implementation method according to any one of claims 1 to 6 are implemented.

Images