Data automatic transfer method, device and equipment and storage medium thereof

Abstract

The embodiment of the application belongs to the technical field of data transfer and digital medical treatment, is applied to the field of automatic transfer of medical data, and relates to a data automatic transfer method, device, equipment and storage medium thereof, and comprises the following steps: acquiring file data in a new business scenario according to a preset HDFS component; and calculating updated data in a Hive data warehouse according to a MapReduce component, replacing the updated data into a corresponding first batch of business data table, and completing automatic updating and transfer of the updated data from the Hive data warehouse to the first batch of business data table. The automatic transfer of file data to the Hive data warehouse and the automatic transfer of Hive data warehouse data to a relational data table are realized in an automatic manner, manual transfer is automatically replaced, time is saved, and the problem that data updating is not timely due to manual transfer and the poor service experience of patients are avoided.

Description

Technical Field

[0001] This application relates to the fields of data transfer and digital medical technology, and is applied to the field of automated medical data transfer, particularly to a method, apparatus, device and storage medium for automated data transfer. Background Technology

[0002] The Hive Big Data Automation System is an HTTP-based big data system that automates the creation of tables and data import. This automation tool creates new tables, synchronizes data from an HVIE database to another database, or synchronizes data from MySQL, Oracle, PostgreSQL, or other databases to a Hive database. It also imports manually prepared data into a Hive database, improving testing quality and efficiency. This helps improve software product quality and increase customer satisfaction.

[0003] Traditional manual data synchronization is cumbersome and error-prone. Users face complex, lengthy, and tedious commands and operations, needing to step through steps to correct errors, download, process data, upload, and execute operations to obtain the required data. This often results in a poor user experience. With large datasets, data creation can take hours or even days, highlighting the increasing challenges of manual operations and providing a poor data transfer experience. As digital healthcare evolves, relying on traditional data transfer and update methods frequently leads to patients spending considerable time waiting for medical data, resulting in a poor patient experience. Therefore, existing technologies for data transfer based on Hive libraries suffer from the problem of untimely data updates due to manual transfer, leading to a poor service experience for patients. Summary of the Invention

[0004] The purpose of this application is to provide an automated data transfer method, apparatus, device and storage medium to solve the problem that the existing technology causes untimely data updates due to manual transfer, resulting in a poor service experience for patients.

[0005] To address the aforementioned technical problems, this application provides an automated data transfer method, employing the following technical solution:

[0006] An automated data transfer method includes the following steps:

[0007] Based on the original business application scenario, source data is obtained from the pre-set Hive data warehouse to build the first batch of business data tables;

[0008] Based on the first monitoring component pre-set in the Hadoop infrastructure, identify whether there are any new business scenarios in the original business application scenario;

[0009] If a new business scenario exists in the original business application scenario, the file data in the new business scenario is accessed according to the preset HDFS component;

[0010] The file data in the newly added business scenario is calculated according to the preset MapReduce component, and the calculation results are transferred to the Hive data warehouse to realize the automatic transfer of file data to the Hive data warehouse;

[0011] Based on the second monitoring component pre-set in the Hadoop infrastructure, it is used to identify whether the source data in the Hive data warehouse has been updated under the original business application scenario;

[0012] If the source data in the Hive data warehouse is updated in the original business application scenario, the updated data is calculated according to the MapReduce component, and the updated data is replaced in the corresponding first batch of business data tables, thus completing the automatic update and transfer of the updated data from the Hive data warehouse to the first batch of business data tables.

[0013] Furthermore, the step of obtaining source data from a pre-set Hive data warehouse based on the original business application scenario and constructing the first batch of business data tables specifically includes:

[0014] Step 301: Combine the pre-set Hive template and the source data to generate a Hive execution statement, wherein the Hive template is composed of non-replaced fields in the Hive execution statement and replaceable example fields in the Hive execution statement;

[0015] Step 302: Execute all combined Hive execution statements, obtain the execution results, and cache the execution results in a preset middleware, wherein the middleware can be a distributed caching component in the Hadoop infrastructure;

[0016] Step 303: Combine the pre-set SQL template and the execution result cached in the middleware to generate an SQL execution statement, wherein the SQL template consists of non-replaced fields in the SQL execution statement and replaceable example fields in the SQL execution statement;

[0017] Step 304: Execute all combined SQL statements, obtain the execution results and output them to the preset database table to complete the construction of the first batch of business data tables.

[0018] Furthermore, the step of combining the pre-set Hive template and the source data to generate the Hive execution statement specifically includes:

[0019] Step 3011: Sequentially obtain each target field in the source data to replace the replaceable example field in the Hive execution statement, and combine them to form the Hive execution statement corresponding to the target field;

[0020] Step 3012: Repeatedly change the target field and repeat step 3011 according to the original business application scenario until all the combined Hive execution statements meet all data operation requirements in the original business application scenario. Then the combination of Hive execution statements corresponding to the original business application scenario is completed.

[0021] Furthermore, the step of combining the pre-set SQL template and the execution results cached in the middleware to generate the SQL execution statement specifically includes:

[0022] Step 3031: Sequentially obtain each result field in the execution result and replace the replaceable example field in the SQL execution statement to combine the result fields into the corresponding SQL execution statement;

[0023] Step 3032: Repeat step 3031 for each result field in the execution result to complete the combination of SQL execution statements corresponding to all result fields in the execution result.

[0024] Furthermore, after executing all the combined SQL statements, obtaining the execution results, and outputting them to a preset database table to complete the construction of the first batch of business data tables, the method further includes:

[0025] Obtain the table names of each table in the first batch of business data tables;

[0026] Construct a target array using the table names of each table as array elements, and set an array name for the target array.

[0027] Furthermore, after performing the step of transferring the calculation results to the Hive data warehouse to automate the transfer of file data to the Hive data warehouse, the method further includes:

[0028] Retrieve newly stored data from the Hive data warehouse;

[0029] Replace the source data in step 301 with the newly stored data, combine them to form Hive execution statements, and execute steps 302 to 304 to complete the construction of the business data table corresponding to the new business scenario;

[0030] Add the table name of the business data table corresponding to the newly added business scenario to the target array to complete the update operation of the first batch of business data tables.

[0031] Furthermore, the step of replacing the updated data into the corresponding first batch of business data tables, completing the automated update and transfer of the updated data from the Hive data warehouse to the first batch of business data tables, specifically includes:

[0032] Retrieve the updated data from the Hive data warehouse;

[0033] Based on a preset log file, determine the data before the update that corresponds to the updated data;

[0034] Identify the Hive execution statement corresponding to the data before it was updated from the preset Hive execution statement cache;

[0035] Based on the Hive execution statement, identify the SQL execution statement corresponding to the data before it was updated;

[0036] Replace the unupdated data in the SQL execution statement with the updated data to generate a new SQL execution statement;

[0037] Execute the new SQL statement to complete the automated update and transfer of the updated data from the Hive data warehouse to the first batch of business data tables.

[0038] To address the aforementioned technical problems, this application also provides an automated data transfer device, which employs the following technical solution:

[0039] An automated data transfer device includes:

[0040] The source table construction module is used to obtain source data from a preset Hive data warehouse based on the original business application scenario and construct the first batch of business data tables.

[0041] The first monitoring module is used to identify whether there are any new business scenarios in the original business application scenario based on the first monitoring component preset in the Hadoop infrastructure.

[0042] The file data access module is used to access the file data in the new business scenario according to the preset HDFS component if there is a new business scenario in the original business application scenario.

[0043] The first transfer module is used to calculate the file data in the new business scenario according to the preset MapReduce component, and transfer the calculation results to the Hive data warehouse to realize the automatic transfer of file data to the Hive data warehouse.

[0044] The second monitoring module is used to identify whether the source data in the Hive data warehouse has been updated under the original business application scenario, based on the second monitoring component preset in the Hadoop infrastructure.

[0045] The second transfer module is used to calculate the updated data according to the MapReduce component if the source data in the Hive data warehouse is updated in the original business application scenario, and replace the updated data into the corresponding first batch of business data tables, thereby completing the automatic update and transfer of the updated data from the Hive data warehouse to the first batch of business data tables.

[0046] To address the aforementioned technical problems, this application also provides a computer device that employs the following technical solution:

[0047] A computer device includes a memory and a processor, wherein the memory stores computer-readable instructions, and the processor executes the computer-readable instructions to implement the steps of the above-described automatic data transfer method.

[0048] To address the aforementioned technical problems, this application also provides a computer-readable storage medium, employing the technical solution described below:

[0049] A computer-readable storage medium storing computer-readable instructions, which, when executed by a processor, implement the steps of the automated data transfer method described above.

[0050] Compared with the prior art, the embodiments of this application have the following main advantages:

[0051] The automated data transfer method described in this application involves: obtaining source data from a preset Hive data warehouse based on the original business application scenario to construct the first batch of business data tables; identifying whether a new business scenario exists in the original business application scenario based on a preset first monitoring component in the Hadoop infrastructure; if a new business scenario exists in the original business application scenario, accessing the file data in the new business scenario based on a preset HDFS component; calculating the file data in the new business scenario based on a preset MapReduce component, and transferring the calculation result to the Hive data warehouse to achieve automated transfer of file data to the Hive data warehouse; identifying whether the source data in the Hive data warehouse is updated under the original business application scenario based on a preset second monitoring component in the Hadoop infrastructure; if the source data in the Hive data warehouse is updated under the original business application scenario, calculating the updated data based on the MapReduce component, and replacing the updated data into the corresponding first batch of business data tables to complete the automated update and transfer of the updated data from the Hive data warehouse to the first batch of business data tables. By automating the transfer of file data to the Hive data warehouse and the transfer of data from the Hive data warehouse to relational data tables, manual transfer steps are reduced. Automation replaces manual transfer, saving time and avoiding the poor business experience that manual transfer can bring to customers. Attached Figure Description

[0052] To more clearly illustrate the solutions in this application, the accompanying drawings used in the description of the embodiments of this application will be briefly introduced below. Obviously, the accompanying drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0053] Figure 1 This is an exemplary system architecture diagram to which this application can be applied;

[0054] Figure 2 A flowchart of an embodiment of the data automated transfer method according to this application;

[0055] Figure 3 yes Figure 2 A flowchart of a specific embodiment of step 201 shown;

[0056] Figure 4 yes Figure 3 A flowchart of a specific embodiment of step 301 shown;

[0057] Figure 5 yes Figure 3A flowchart of a specific embodiment of step 303 shown;

[0058] Figure 6 yes Figure 2 A flowchart of a specific embodiment of step 206 shown;

[0059] Figure 7 A schematic diagram of a structural diagram of an embodiment of the data automatic transfer device according to this application;

[0060] Figure 8 yes Figure 7 A schematic diagram of a specific embodiment of module 701 shown;

[0061] Figure 9 A schematic diagram of the structure of an embodiment of the computer device according to this application. Detailed Implementation

[0062] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains; the terminology used herein in the specification of the application is for the purpose of describing particular embodiments only and is not intended to be limiting of the application; the terms "comprising" and "having," and any variations thereof, in the specification, claims, and foregoing drawings of this application, are intended to cover non-exclusive inclusion. The terms "first," "second," etc., in the specification, claims, or foregoing drawings of this application are used to distinguish different objects, not to describe a particular order.

[0063] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0064] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings.

[0065] like Figure 1 As shown, system architecture 100 may include terminal devices 101, 102, and 103, a network 104, and a server 105. Network 104 serves as the medium for providing communication links between terminal devices 101, 102, and 103 and server 105. Network 104 may include various connection types, such as wired or wireless communication links, or fiber optic cables, etc.

[0066] Users can use terminal devices 101, 102, and 103 to interact with server 105 via network 104 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 101, 102, and 103, such as web browser applications, shopping applications, search applications, instant messaging tools, email clients, social media platform software, etc.

[0067] Terminal devices 101, 102, and 103 can be various electronic devices with displays and support web browsing, including but not limited to smartphones, tablets, e-book readers, MP3 players (Moving Picture Experts Group Audio Layer III), MP4 players (Moving Picture Experts Group Audio Layer IV), laptops, and desktop computers, etc.

[0068] Server 105 can be a server that provides various services, such as a backend server that supports the pages displayed on terminal devices 101, 102, and 103.

[0069] It should be noted that the automated data transfer method provided in this application is generally executed by a server / terminal device, and correspondingly, the automated data transfer device is generally installed in the server / terminal device.

[0070] It should be understood that Figure 1 The number of terminal devices, networks, and servers shown is merely illustrative. Depending on implementation needs, any number of terminal devices, networks, and servers can be included.

[0071] To facilitate understanding, let's first introduce the Hive data warehouse. The Hive data warehouse is a data warehouse based on the Hadoop distributed cluster infrastructure. Hadoop implements a distributed file system, one component of which is HDFS (Hadoop Distributed File System). HDFS is highly fault-tolerant and designed to be deployed on inexpensive hardware; moreover, it provides high throughput access to application data, making it suitable for applications with extremely large datasets. HDFS relaxes POSIX requirements, allowing data in the file system to be accessed in a streaming manner. The core design of the Hadoop framework consists of HDFS and MapReduce. HDFS provides storage for massive amounts of data, while MapReduce provides computation for massive amounts of data. Hive data warehouses are often used for data storage in big data business scenarios, while the Hadoop distributed cluster infrastructure is used for distributed processing in these scenarios.

[0072] Hive data warehouses do not support the direct execution of SQL statements. Therefore, when transferring data between Hive data warehouses and relational data tables, it is not possible to use SQL statements for direct transfer.

[0073] Continue to refer to Figure 2 A flowchart illustrating an embodiment of the automated data transfer method according to this application is shown. The automated data transfer method includes the following steps:

[0074] Step 201: Obtain source data from the preset Hive data warehouse according to the original business application scenario and build the first batch of business data tables.

[0075] In this embodiment, the preset Hive data warehouse includes medical source data stored by hospitals or medical cloud platforms.

[0076] Continue to refer to Figure 3 , Figure 3 yes Figure 2 A flowchart of a specific embodiment of step 201 shown includes:

[0077] Step 301: Combine the pre-set Hive template and the source data to generate a Hive execution statement, wherein the Hive template is composed of non-replaced fields in the Hive execution statement and replaceable example fields in the Hive execution statement;

[0078] Continue to refer to Figure 4 , Figure 4 yes Figure 3 A flowchart of a specific embodiment of step 301 shown includes:

[0079] Step 3011: Sequentially obtain each target field in the source data to replace the replaceable example field in the Hive execution statement, and combine them to form the Hive execution statement corresponding to the target field;

[0080] Step 3012: Repeatedly change the target field and repeat step 3011 according to the original business application scenario until all the combined Hive execution statements meet all data operation requirements in the original business application scenario. Then the combination of Hive execution statements corresponding to the original business application scenario is completed.

[0081] Step 302: Execute all combined Hive execution statements, obtain the execution results, and cache the execution results in a preset middleware, wherein the middleware can be a distributed caching component in the Hadoop infrastructure;

[0082] By executing all the combined Hive execution statements, the execution results are obtained and cached in a preset middleware. In other words, the data to be transferred is obtained and cached in the middleware, so that when transferring relational data tables, the data to be cached can be directly pulled from the middleware.

[0083] Step 303: Combine the pre-set SQL template and the execution result cached in the middleware to generate an SQL execution statement, wherein the SQL template consists of non-replaced fields in the SQL execution statement and replaceable example fields in the SQL execution statement;

[0084] Continue to refer to Figure 5 , Figure 5 yes Figure 3 A flowchart of a specific embodiment of step 303 shown includes:

[0085] Step 3031: Sequentially obtain each result field in the execution result and replace the replaceable example field in the SQL execution statement to combine the result fields into the corresponding SQL execution statement;

[0086] Step 3032: Based on each result field in the execution result, repeat step 3031 to complete the combination of SQL execution statements corresponding to all result fields in the execution result;

[0087] Step 304: Execute all combined SQL statements, obtain the execution results and output them to the preset database table to complete the construction of the first batch of business data tables.

[0088] By executing all the combined SQL statements, the execution results are obtained and output to the preset database tables, thus completing the construction of the first batch of business data tables and realizing the construction and data transfer from the Hive data warehouse to the first batch of business data tables.

[0089] In this embodiment, after executing all the combined SQL statements, obtaining the execution results and outputting them to a preset database table to complete the construction of the first batch of business data tables, the method further includes: obtaining the table names of each table in the first batch of business data tables; constructing a target array using the table names of each table as array elements, and setting an array name for the target array.

[0090] Step 202: Based on the first monitoring component preset in the Hadoop infrastructure, identify whether there are any new business scenarios in the original business application scenario.

[0091] In this embodiment, the newly added business scenario can refer to a new medical data input interface.

[0092] In this embodiment, the first monitoring component monitors the logic program of the original business application scenario.

[0093] Step 203: If there is a new business scenario in the original business application scenario, then access the file data in the new business scenario according to the preset HDFS component.

[0094] The first monitoring component determines whether there are any new business scenarios. If there are new business scenarios, the HDFS component accesses the file data in the new business scenario, and then the MapReduce component calculates the file data in the new business scenario and transfers the file data to the Hive data warehouse in the form of a stream.

[0095] Step 204: Calculate the file data in the newly added business scenario according to the preset MapReduce component, and transfer the calculation results to the Hive data warehouse to realize the automatic transfer of file data to the Hive data warehouse.

[0096] In this embodiment, after performing the step of automatically transferring the calculation results to the Hive data warehouse to realize the transfer of file data to the Hive data warehouse, the method further includes: obtaining the newly stored data in the Hive data warehouse; replacing the source data in step 301 with the newly stored data, combining them into Hive execution statements, and executing steps 302 to 304 to complete the construction of the business data table corresponding to the new business scenario; adding the table name of the business data table corresponding to the new business scenario to the target array to complete the update operation of the first batch of business data tables.

[0097] By replacing the source data in step 301 with file data that is transferred to the Hive data warehouse in the form of a stream, Hive execution statements are combined, and steps 302 to 304 are executed to complete the construction of the business data table corresponding to the new business scenario and realize the transfer of file data to the newly constructed business data table.

[0098] Step 205: Based on the second monitoring component preset in the Hadoop infrastructure, identify whether the source data in the Hive data warehouse has been updated under the original business application scenario.

[0099] In this embodiment, the second monitoring component monitors the source data in the Hive data warehouse.

[0100] Step 206: If the source data in the Hive data warehouse is updated in the original business application scenario, the updated data is calculated according to the MapReduce component, and the updated data is replaced in the corresponding first batch of business data tables. This completes the automatic update and transfer of the updated data from the Hive data warehouse to the first batch of business data tables, so as to update and transfer medical data in a timely manner and facilitate patients to obtain medical data information in a timely manner.

[0101] Continue to refer to Figure 6 , Figure 6 yes Figure 2 A flowchart of a specific embodiment of step 206 shown includes:

[0102] Step 601: Obtain the updated data from the Hive data warehouse;

[0103] Step 602: Based on a preset log file, determine the data before the update corresponding to the updated data, wherein the log file includes at least the changes in the data before and after the update in the data warehouse;

[0104] Step 603: Identify the Hive execution statement corresponding to the data before it was updated from the preset Hive execution statement cache area, wherein the Hive execution statement cache area caches all the Hive execution statements combined in step 301;

[0105] Step 604: Identify the SQL execution statement corresponding to the data before the update based on the Hive execution statement;

[0106] Step 605: Replace the unupdated data in the SQL execution statement with the updated data to generate a new SQL execution statement;

[0107] Step 606: Execute the new SQL statement to complete the automated update and transfer of the updated data from the Hive data warehouse to the first batch of business data tables.

[0108] The second monitoring component monitors whether the Hive data warehouse has been updated in the original business application scenario. This facilitates timely updates to the relational data tables corresponding to the original business application scenario based on the data updates in the Hive data warehouse. The transfer of file data to the Hive data warehouse and the transfer of data from the Hive data warehouse to relational data tables are automated, reducing manual transfer steps and saving time. It also avoids the poor treatment experience for patients caused by excessively long manual transfer times.

[0109] The specific monitoring methods of the first and second monitoring components mentioned above can be achieved by monitoring log files to monitor the logic program of the original business application scenario and the source data in the Hive data warehouse.

[0110] This application obtains source data from a pre-defined Hive data warehouse based on the original business application scenario and constructs the first batch of business data tables. Based on a pre-defined first monitoring component in the Hadoop infrastructure, it identifies whether a new business scenario exists within the original business application scenario. If a new business scenario exists, it accesses the file data in the new business scenario using a pre-defined HDFS component. It calculates the file data in the new business scenario using a pre-defined MapReduce component and transfers the calculation results to the Hive data warehouse, achieving automated transfer of file data to the Hive data warehouse. Based on a pre-defined second monitoring component in the Hadoop infrastructure, it identifies whether the source data in the Hive data warehouse is updated under the original business application scenario. If the source data in the Hive data warehouse is updated under the original business application scenario, it calculates the updated data using the MapReduce component and replaces the updated data in the corresponding first batch of business data tables, completing the automated update and transfer of the updated data from the Hive data warehouse to the first batch of business data tables. The process of transferring file data to the Hive data warehouse and then transferring data from the Hive data warehouse to relational data tables is automated, reducing manual transfer steps and saving time. It also avoids the poor treatment experience caused by excessive time spent on manual transfer.

[0111] The embodiments of this application can acquire and process relevant data based on artificial intelligence technology. Artificial intelligence (AI) refers to the theories, methods, technologies, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to obtain optimal results.

[0112] Foundational technologies for artificial intelligence generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive systems, and mechatronics. AI software technologies mainly encompass computer vision, robotics, biometrics, speech processing, natural language processing, and machine learning / deep learning.

[0113] In this embodiment, file data is transferred to the Hive data warehouse and data in the Hive data warehouse is transferred to relational data tables in an automated manner. This reduces the manual transfer steps, automates the manual transfer, saves time, and avoids the poor service experience for patients caused by untimely data updates due to manual transfer.

[0114] Further reference Figure 7 As a response to the above Figure 2 To implement the method shown, this application provides an embodiment of an automated data transfer device, which is similar to... Figure 2 Corresponding to the method embodiments shown, this device can be specifically applied to various electronic devices.

[0115] like Figure 7 As shown, the automated data transfer device 700 described in this embodiment includes: a source table construction module 701, a first monitoring module 702, a file data access module 703, a first transfer module 704, a second monitoring module 705, and a second transfer module 706. Wherein:

[0116] The source table construction module 701 is used to obtain source data from the preset Hive data warehouse according to the original business application scenario and construct the first batch of business data tables.

[0117] The first monitoring module 702 is used to identify whether there are any new business scenarios in the original business application scenario based on the first monitoring component preset in the Hadoop infrastructure.

[0118] The file data access module 703 is used to access the file data in the new business scenario according to the preset HDFS component if there is a new business scenario in the original business application scenario.

[0119] The first transfer module 704 is used to calculate the file data in the new business scenario according to the preset MapReduce component, and transfer the calculation result to the Hive data warehouse to realize the automatic transfer of file data to the Hive data warehouse.

[0120] The second monitoring module 705 is used to identify whether the source data in the Hive data warehouse has been updated under the original business application scenario, based on the second monitoring component preset in the Hadoop infrastructure.

[0121] The second transfer module 706 is used to calculate the updated data according to the MapReduce component if the source data in the Hive data warehouse is updated in the original business application scenario, and replace the updated data into the corresponding first batch of business data tables, thereby completing the automatic update and transfer of the updated data from the Hive data warehouse to the first batch of business data tables.

[0122] Continue to refer to Figure 8 , Figure 8 yes Figure 7 A schematic diagram of a specific embodiment of module 701 is shown, wherein the source table construction module 701 includes a first combination submodule 801, a first execution submodule 802, a second combination submodule 803, and a second execution submodule 804, wherein:

[0123] The first combination submodule 801 is used to combine a Hive execution statement based on a pre-set Hive template and the source data, wherein the Hive template is composed of non-replaced fields in the Hive execution statement and replaceable example fields in the Hive execution statement.

[0124] The first execution submodule 802 is used to execute all combined Hive execution statements, obtain the execution results, and cache the execution results in a preset middleware, wherein the middleware can be a distributed caching component in the Hadoop infrastructure.

[0125] The second combination submodule 803 is used to combine an SQL execution statement based on a pre-set SQL template and the execution result cached in the middleware, wherein the SQL template consists of non-replaced fields in the SQL execution statement and replaceable example fields in the SQL execution statement.

[0126] The second execution submodule 804 is used to execute all the combined SQL execution statements, obtain the execution results and output them to the preset database table, thereby completing the construction of the first batch of business data tables.

[0127] In some specific embodiments of this application, the first combination submodule 801 includes a first loop combination unit. The first loop combination unit is used to sequentially obtain each target field in the source data to replace the replaceable example field in the Hive execution statement, and combine them to form the Hive execution statement corresponding to the target field; it is also used to repeatedly replace the target field and repeatedly combine the Hive execution statement corresponding to the target field according to the original business application scenario, until all the combined Hive execution statements meet all data operation requirements in the original business application scenario, then the combination of the Hive execution statements corresponding to the original business application scenario is completed.

[0128] In some specific embodiments of this application, the second combination submodule 803 also includes a second loop combination unit. The second loop combination unit is used to sequentially obtain each result field in the execution result to replace the replaceable example field in the SQL execution statement, and combine the SQL execution statement corresponding to the result field; and to complete the combination of the SQL execution statements corresponding to all result fields in the execution result based on each result field in the execution result.

[0129] This application obtains source data from a pre-defined Hive data warehouse based on the original business application scenario and constructs the first batch of business data tables. Based on a pre-defined first monitoring component in the Hadoop infrastructure, it identifies whether a new business scenario exists in the original business application scenario. If a new business scenario exists, it accesses the file data in the new business scenario using a pre-defined HDFS component. It calculates the file data in the new business scenario using a pre-defined MapReduce component and transfers the calculation results to the Hive data warehouse, achieving automated file data transfer to the Hive data warehouse. Based on a pre-defined second monitoring component in the Hadoop infrastructure, it identifies whether the source data in the Hive data warehouse is updated under the original business application scenario. If the source data in the Hive data warehouse is updated under the original business application scenario, it calculates the updated data using the MapReduce component and replaces the updated data in the corresponding first batch of business data tables, completing the automated update and transfer of updated data from the Hive data warehouse to the first batch of business data tables. The process of transferring file data to the Hive data warehouse and then transferring data from the Hive data warehouse to relational data tables is automated, reducing manual transfer steps and saving time. This avoids the poor patient experience caused by excessively long manual transfer times.

[0130] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by instructing related hardware through computer-readable instructions. These computer-readable instructions can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the methods described above. The aforementioned storage medium can be a non-volatile storage medium such as a magnetic disk, optical disk, or read-only memory (ROM), or random access memory (RAM).

[0131] It should be understood that although the steps in the flowcharts of the accompanying figures are shown sequentially as indicated by the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the accompanying figures may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the sub-steps or stages of other steps.

[0132] To address the aforementioned technical problems, embodiments of this application also provide a computer device. Please refer to [link / reference needed]. Figure 9 , Figure 9 This is a basic structural block diagram of the computer device in this embodiment.

[0133] The computer device 9 includes a memory 9a, a processor 9b, and a network interface 9c that are interconnected via a system bus. It should be noted that only the computer device 9 with components 9a-9c is shown in the figure; however, it should be understood that it is not required to implement all the shown components, and more or fewer components can be implemented alternatively. Those skilled in the art will understand that the computer device described here is a device capable of automatically performing numerical calculations and / or information processing according to pre-set or stored instructions, and its hardware includes, but is not limited to, microprocessors, application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), digital signal processors (DSPs), embedded devices, etc.

[0134] The computer device can be a desktop computer, laptop, handheld computer, or cloud server, etc. The computer device can interact with the user via a keyboard, mouse, remote control, touchpad, or voice control.

[0135] The memory 9a includes at least one type of readable storage medium, including flash memory, hard disk, multimedia card, card-type memory (e.g., SD or DX memory), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the memory 9a may be an internal storage unit of the computer device 9, such as the hard disk or memory of the computer device 9. In other embodiments, the memory 9a may also be an external storage device of the computer device 9, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc., equipped on the computer device 9. Of course, the memory 9a may include both the internal storage unit and its external storage device of the computer device 9. In this embodiment, the memory 9a is typically used to store the operating system and various application software installed on the computer device 9, such as computer-readable instructions for automated data transfer methods. In addition, the memory 9a can also be used to temporarily store various types of data that have been output or will be output.

[0136] In some embodiments, the processor 9b may be a central processing unit (CPU), controller, microcontroller, microprocessor, or other data processing chip. The processor 9b is typically used to control the overall operation of the computer device 9. In this embodiment, the processor 9b is used to execute computer-readable instructions stored in the memory 9a or to process data, for example, to execute computer-readable instructions of the automated data transfer method.

[0137] The network interface 9c may include a wireless network interface or a wired network interface, which is typically used to establish communication connections between the computer device 9 and other electronic devices.

[0138] The computer device proposed in this embodiment belongs to the field of data transfer and digital medical technology, and is applied to the field of automated medical data transfer. It obtains source data from a preset Hive data warehouse based on the original business application scenario, constructing the first batch of business data tables. Based on a preset first monitoring component in the Hadoop infrastructure, it identifies whether a new business scenario exists in the original business application scenario. If a new business scenario exists, it accesses the file data in the new business scenario using a preset HDFS component. It calculates the file data in the new business scenario using a preset MapReduce component and transfers the calculation results to the Hive data warehouse, achieving automated transfer of file data to the Hive data warehouse. Based on a preset second monitoring component in the Hadoop infrastructure, it identifies whether the source data in the Hive data warehouse is updated under the original business application scenario. If the source data in the Hive data warehouse is updated under the original business application scenario, it calculates the updated data using the MapReduce component and replaces the updated data in the corresponding first batch of business data tables, completing the automated update and transfer of the updated data from the Hive data warehouse to the first batch of business data tables. The process of transferring file data to the Hive data warehouse and then transferring data from the Hive data warehouse to relational data tables is automated, reducing manual transfer steps and saving time. This avoids the poor patient experience caused by excessively long manual transfer times.

[0139] This application also provides another embodiment, namely, providing a computer-readable storage medium storing computer-readable instructions that can be executed by a processor to cause the processor to perform the steps of the data automatic transfer method described above.

[0140] The computer-readable storage medium proposed in this embodiment belongs to the field of data transfer and digital healthcare technology, and is applied to the field of automated medical data transfer. Based on the original business application scenario, source data is obtained from a preset Hive data warehouse to construct the first batch of business data tables. Based on a preset first monitoring component in the Hadoop infrastructure, it is identified whether a new business scenario exists in the original business application scenario. If a new business scenario exists, the file data in the new business scenario is accessed according to a preset HDFS component. The file data in the new business scenario is calculated according to a preset MapReduce component, and the calculation result is transferred to the Hive data warehouse, realizing the automated transfer of file data to the Hive data warehouse. Based on a preset second monitoring component in the Hadoop infrastructure, it is identified whether the source data in the Hive data warehouse is updated under the original business application scenario. If the source data in the Hive data warehouse is updated under the original business application scenario, the updated data is calculated according to the MapReduce component, and the updated data is replaced in the corresponding first batch of business data tables, completing the automated update and transfer of the updated data from the Hive data warehouse to the first batch of business data tables. The process of transferring file data to the Hive data warehouse and then transferring data from the Hive data warehouse to relational data tables is automated, reducing manual transfer steps and saving time. This avoids the poor patient experience caused by excessively long manual transfer times.

[0141] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0142] Obviously, the embodiments described above are only some embodiments of this application, not all embodiments. The accompanying drawings show preferred embodiments of this application, but do not limit the patent scope of this application. This application can be implemented in many different forms; rather, the purpose of providing these embodiments is to provide a more thorough and comprehensive understanding of the disclosure of this application. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing specific embodiments, or make equivalent substitutions for some of the technical features. Any equivalent structures made using the content of this application's specification and drawings, directly or indirectly applied to other related technical fields, are similarly within the scope of patent protection of this application.

Claims

1. A method for automated data transfer, characterized in that, Includes the following steps: Based on the original business application scenario, source data is obtained from a pre-set Hive data warehouse to construct the first batch of business data tables. Specifically, the step of obtaining source data from the pre-set Hive data warehouse and constructing the first batch of business data tables includes: Step 301: Combine the pre-set Hive template and the source data to generate a Hive execution statement, wherein the Hive template is composed of non-replaced fields in the Hive execution statement and replaceable example fields in the Hive execution statement; Step 302: Execute all combined Hive execution statements, obtain the execution results, and cache the execution results in a preset middleware, wherein the middleware can be a distributed caching component in the Hadoop infrastructure; Step 303: Combine the pre-set SQL template and the execution result cached in the middleware to generate an SQL execution statement, wherein the SQL template consists of non-replaced fields in the SQL execution statement and replaceable example fields in the SQL execution statement; Step 304: Execute all combined SQL statements, obtain the execution results and output them to the preset database table to complete the construction of the first batch of business data tables; Based on the first monitoring component pre-set in the Hadoop infrastructure, identify whether there are any new business scenarios in the original business application scenario; If a new business scenario exists in the original business application scenario, the file data in the new business scenario is accessed according to the preset HDFS component; The file data in the newly added business scenario is calculated according to the preset MapReduce component, and the calculation results are transferred to the Hive data warehouse to realize the automatic transfer of file data to the Hive data warehouse; Based on the second monitoring component pre-set in the Hadoop infrastructure, it is used to identify whether the source data in the Hive data warehouse has been updated under the original business application scenario; If the source data in the Hive data warehouse is updated in the original business application scenario, the updated data is calculated based on the MapReduce component, and the updated data is used to replace the corresponding first batch of business data tables. This completes the automated update and transfer of the updated data from the Hive data warehouse to the first batch of business data tables. Specifically, the step of replacing the updated data in the corresponding first batch of business data tables to complete the automated update and transfer of the updated data from the Hive data warehouse to the first batch of business data tables includes: Retrieve the updated data from the Hive data warehouse; Based on a preset log file, determine the data before the update that corresponds to the updated data; Identify the Hive execution statement corresponding to the data before it was updated from the preset Hive execution statement cache; Based on the Hive execution statement, identify the SQL execution statement corresponding to the data before it was updated; Replace the unupdated data in the SQL execution statement with the updated data to generate a new SQL execution statement; Execute the new SQL statement to complete the automated update and transfer of the updated data from the Hive data warehouse to the first batch of business data tables.

2. The automated data transfer method according to claim 1, characterized in that, The step of generating Hive execution statements based on a pre-set Hive template and the source data specifically includes: Step 3011: Sequentially obtain each target field in the source data to replace the replaceable example field in the Hive execution statement, and combine them to form the Hive execution statement corresponding to the target field; Step 3012: Repeatedly change the target field and repeat step 3011 according to the original business application scenario until all the combined Hive execution statements meet all data operation requirements in the original business application scenario. Then the combination of Hive execution statements corresponding to the original business application scenario is completed.

3. The automated data transfer method according to claim 1, characterized in that, The step of generating an SQL execution statement based on a pre-set SQL template and the execution results cached in the middleware specifically includes: Step 3031: Sequentially obtain each result field in the execution result and replace the replaceable example field in the SQL execution statement to combine the result fields into the corresponding SQL execution statement; Step 3032: Repeat step 3031 for each result field in the execution result to complete the combination of SQL execution statements corresponding to all result fields in the execution result.

4. The automated data transfer method according to claim 1, characterized in that, After executing all the combined SQL statements, obtaining the execution results, and outputting them to a preset database table to complete the construction of the first batch of business data tables, the method further includes: Obtain the table names of each table in the first batch of business data tables; Construct a target array using the table names of each table as array elements, and set an array name for the target array.

5. The automated data transfer method according to claim 4, characterized in that, After performing the step of transferring the calculation results to the Hive data warehouse to automate the transfer of file data to the Hive data warehouse, the method further includes: Retrieve newly stored data from the Hive data warehouse; Replace the source data in step 301 with the newly stored data, combine them to form Hive execution statements, and execute steps 302 to 304 to complete the construction of the business data table corresponding to the new business scenario; Add the table name of the business data table corresponding to the newly added business scenario to the target array to complete the update operation of the first batch of business data tables.

6. A data automated transfer device, characterized in that, The automated data transfer device implements the steps of the automated data transfer method as described in any one of claims 1 to 5, including: The source table construction module is used to obtain source data from a preset Hive data warehouse based on the original business application scenario and construct the first batch of business data tables. The first monitoring module is used to identify whether there are any new business scenarios in the original business application scenario based on the first monitoring component preset in the Hadoop infrastructure. The file data access module is used to access the file data in the new business scenario according to the preset HDFS component if there is a new business scenario in the original business application scenario. The first transfer module is used to calculate the file data in the new business scenario according to the preset MapReduce component, and transfer the calculation results to the Hive data warehouse to realize the automatic transfer of file data to the Hive data warehouse. The second monitoring module is used to identify whether the source data in the Hive data warehouse has been updated under the original business application scenario, based on the second monitoring component preset in the Hadoop infrastructure. The second transfer module is used to calculate the updated data according to the MapReduce component if the source data in the Hive data warehouse is updated in the original business application scenario, and replace the updated data into the corresponding first batch of business data tables, thereby completing the automatic update and transfer of the updated data from the Hive data warehouse to the first batch of business data tables.

7. A computer device comprising a memory and a processor, wherein the memory stores computer-readable instructions, and the processor, when executing the computer-readable instructions, implements the steps of the data automatic transfer method as described in any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-readable instructions, which, when executed by a processor, implement the steps of the data automatic transfer method as described in any one of claims 1 to 5.

Images