Data transmission method, device and equipment based on cloud data warehouse, and storage medium

Abstract

The application provides a data transmission method and device based on a cloud data warehouse, equipment and a storage medium. The method comprises the following steps: obtaining to-be-transmitted data in a preset application server; wherein the application server is used for storing the to-be-transmitted data before data transmission; determining a port number of the cloud data warehouse according to preset database connection information; wherein the database connection information is used for indicating information for establishing a connection between the application server and the cloud data warehouse, the database connection information comprises the port number of the cloud data warehouse, and a node port corresponding to the port number is used for receiving the to-be-transmitted data; based on a computing layer of the cloud data warehouse, the node port corresponding to the port number loads the to-be-transmitted data to the computing layer; wherein the cloud data warehouse comprises the computing layer and a storage layer, and the to-be-transmitted data is used for being transmitted from the computing layer to the storage layer for storage. The application directly loads the to-be-transmitted data to the cloud data warehouse, thereby improving the data transmission efficiency.

Description

Technical Field

[0001] This application relates to communication technology, and more particularly to a data transmission method, apparatus, device, and storage medium based on a cloud data warehouse. Background Technology

[0002] A cloud data warehouse is a new generation of data warehouses deployed and operated in a cloud environment. A cloud data warehouse can be divided into a three-layer architecture: a cloud service layer, a computing layer, and a storage layer. The management layer provides services such as tenant access points, resource management, metadata management, transaction management, parsing, and execution scheduling; the computing layer provides and manages computing resources of different specifications and offers caching acceleration support; the storage layer provides object storage services and supports the persistence of result set caching.

[0003] In existing technologies, users need to select files from the application server, upload the selected files to the object storage of the cloud data warehouse, and then execute loading commands through the cloud data warehouse front-end environment to complete the data loading. That is, when transferring data to the cloud data warehouse, it is necessary to import and then load the data, and users need to manually select files through a page, resulting in low data transfer efficiency. Summary of the Invention

[0004] This application provides a data transmission method, apparatus, device, and storage medium based on a cloud data warehouse to improve data transmission efficiency.

[0005] In a first aspect, this application provides a data transmission method based on a cloud data warehouse, the method being applied to an electronic device, comprising:

[0006] Obtain the data to be transmitted from a preset application server; wherein, the application server is used to store the data to be transmitted before data transmission;

[0007] Based on preset database connection information, the port number of the cloud data warehouse is determined; wherein, the database connection information is used to indicate the information for establishing a connection between the application server and the cloud data warehouse, the database connection information includes the port number of the cloud data warehouse, and the node port corresponding to the port number is used to receive the data to be transmitted;

[0008] Based on the computing layer of the cloud data warehouse, the node port corresponding to the port number loads the data to be transmitted into the computing layer; wherein, the cloud data warehouse includes a computing layer and a storage layer, and the data to be transmitted is used to be transmitted from the computing layer to the storage layer for storage.

[0009] Secondly, this application provides a data transmission device based on a cloud data warehouse, the device being applied to an electronic device, comprising:

[0010] A data acquisition module is used to acquire data to be transmitted from a preset application server; wherein, the application server is used to store the data to be transmitted before data transmission.

[0011] The port number determination module is used to determine the port number of the cloud data warehouse based on preset database connection information; wherein, the database connection information is used to indicate the information for establishing a connection between the application server and the cloud data warehouse, the database connection information includes the port number of the cloud data warehouse, and the node port corresponding to the port number is used to receive the data to be transmitted;

[0012] The data loading module is used to load the data to be transmitted onto the computing layer of the cloud data warehouse based on the node port corresponding to the port number; wherein the cloud data warehouse includes a computing layer and a storage layer, and the data to be transmitted is used to be transmitted from the computing layer to the storage layer for storage.

[0013] Thirdly, this application provides an electronic device, including: a processor, and a memory communicatively connected to the processor;

[0014] The memory stores computer-executed instructions;

[0015] The processor executes computer execution instructions stored in the memory to implement the data transmission method based on a cloud data warehouse as described in the first aspect of this application.

[0016] Fourthly, this application provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement the data transmission method based on a cloud data warehouse as described in the first aspect of this application.

[0017] Fifthly, this application provides a computer program product, including a computer program that, when executed by a processor, implements the data transmission method based on a cloud data warehouse as described in the first aspect of this application.

[0018] This application provides a data transmission method, apparatus, device, and storage medium based on a cloud data warehouse. It receives data to be transmitted through the node port of the cloud data warehouse's computing layer and can directly load the data from the application server into the cloud data warehouse, eliminating the need for prior import and loading. This achieves data loading during the import process, avoiding the process in existing technologies where the data to be transmitted is first imported into object storage and then loaded via a loading command executed by the cloud data warehouse front-end environment, effectively improving data transmission efficiency. Attached Figure Description

[0019] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0020] Figure 1 A flowchart illustrating a data transmission method based on a cloud data warehouse, provided as an embodiment of this application;

[0021] Figure 2 This is a schematic diagram illustrating data transmission between an application server and a cloud data warehouse, provided in an embodiment of this application.

[0022] Figure 3 A flowchart illustrating a data transmission method based on a cloud data warehouse, provided as an embodiment of this application;

[0023] Figure 4 A structural block diagram of a data transmission device based on a cloud data warehouse provided in an embodiment of this application;

[0024] Figure 5 A structural block diagram of a data transmission device based on a cloud data warehouse provided in an embodiment of this application;

[0025] Figure 6 A structural block diagram of an electronic device provided in an embodiment of this application;

[0026] Figure 7 This is a structural block diagram of an electronic device provided in an embodiment of this application.

[0027] The accompanying drawings illustrate specific embodiments of this application, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concept of this application to those skilled in the art through reference to particular embodiments. Detailed Implementation

[0028] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings.

[0029] It should be understood that the described embodiments are merely some, not all, of the embodiments in this application. All other embodiments obtained by those skilled in the art based on the embodiments in this application without inventive effort are within the scope of protection of this application.

[0030] In the following description, when referring to the accompanying drawings, the same numbers in different drawings denote the same or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0031] In the description of this application, it should be understood that the terms "first," "second," "third," etc., are used only to distinguish similar objects and are not necessarily used to describe a specific order or sequence, nor should they be construed as indicating or implying relative importance. Those skilled in the art can understand the specific meaning of the above terms in this application according to the specific circumstances. Furthermore, in the description of this application, unless otherwise stated, "multiple" refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. The character " / " generally indicates that the preceding and following related objects have an "or" relationship.

[0032] It should be noted that, due to space limitations, this application specification does not exhaustively list all possible implementation methods. Those skilled in the art, after reading this application specification, should be able to deduce that, as long as the technical features do not contradict each other, any combination of technical features can constitute an optional implementation method. The following provides a detailed description of each embodiment.

[0033] A cloud data warehouse, hereinafter referred to as a cloud data warehouse, is a new generation of data warehouses deployed and operated in a cloud environment. Cloud data warehouses allow users to create and use data warehouse services quickly, focusing on business operations at a lower cost. It has three key features: elastic resource management, multi-tenant secure isolation, and high service availability. A cloud data warehouse consists of a cloud service layer, a compute layer, and a storage layer. The cloud service layer provides tenant access, resource management, metadata management, transaction management, parsing, and execution scheduling services. The compute layer provides and manages compute resources of different specifications and offers caching acceleration support. The storage layer provides object storage services and supports persistent result set caching. The storage layer is deployed at the IaaS (Infrastructure as a Service) layer, and the compute layer runs at the PaaS (Platform as a Service) layer, achieving separation of storage and compute. The compute nodes in the compute layer are deployed as containers in a Kubernetes environment, which cannot permanently store data and cannot adequately meet the needs of large-scale batch loading of local data. Kubernetes (K8S) is a portable container orchestration and management tool designed for container services.

[0034] Application servers primarily provide cloud data warehouses with environments for batch script deployment, data import, data loading, and batch program execution. In a cloud data warehouse environment, if batch data loading is required, the data must first be imported into object storage, and then the loading command is executed through the cloud data warehouse front-end environment to complete the data loading. Furthermore, users need to manually select the data to be transferred through a preset page, making batch transfer impossible and resulting in low data transfer efficiency.

[0035] This application provides a data transmission method, apparatus, device, and storage medium based on a cloud data warehouse, aiming to solve the above-mentioned technical problems in the prior art.

[0036] The technical solution of this application and how the technical solution of this application solves the above-mentioned technical problems are described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will now be described with reference to the accompanying drawings.

[0037] Figure 1 This is a flowchart illustrating a data transmission method based on a cloud data warehouse according to an embodiment of this application. The method is applied to an electronic device and executed by a data transmission device for a cloud data warehouse. Figure 1 As shown, the method includes the following steps:

[0038] S101. Obtain the data to be transmitted from the preset application server; wherein, the application server is used to store the data to be transmitted before data transmission.

[0039] For example, the application server primarily provides the cloud data warehouse with a batch script deployment environment, a data import environment, a data loading environment, and a batch program execution environment. When users conduct transactions, they can store transaction data and user information in the application server. For instance, when a user conducts a transaction through a trading application, the data in the application can be transferred to the application server. The application server stores data that needs to be transferred to the cloud data warehouse; this data is called "data to be transferred." In other words, the application server is used to store data to be transferred before data transmission.

[0040] When data needs to be uploaded to the cloud data warehouse, the data is retrieved from the application server as the data to be transferred. Users can manually select the folder containing the data to be transferred from the application server; for example, users can select the folder through a visual interface. Alternatively, the application server can automatically or periodically determine the data it stores to be transferred; for example, it can retrieve the data to be transferred from the application server every 24 hours. The data to be transferred can be the data stored by the application server that day.

[0041] In this embodiment, obtaining the data to be transmitted from the preset application server includes: determining the amount of data to be transmitted in the preset data storage path of the application server; if the amount of data exceeds the preset data amount threshold, then obtaining the data to be transmitted in batches.

[0042] Specifically, the location where data is stored on the application server, i.e., the data storage path, is predetermined. When the application server receives data, it can store the received data in the preset data storage path, for example, in a preset folder. The data stored in the preset data storage path can be identified as the data to be transmitted. The application server can determine the amount of data in the data storage path, i.e., the amount of data to be transmitted, in real time or periodically. For example, the application server can count the data volume every 10 minutes.

[0043] A pre-set data volume threshold is used. After obtaining the data volume to be transferred, it is compared with the threshold. If the data volume exceeds the threshold, it is determined that data needs to be imported into the cloud data warehouse, and this volume of data can be retrieved in batches. If the data volume does not exceed the threshold, it is determined that data does not need to be imported into the cloud data warehouse, and the process continues to check whether the data volume exceeds the threshold. There can be one or more data storage paths. Each time the data volume is checked, the data to be transferred from multiple storage paths can be counted together, meaning that the data to be transferred from multiple storage paths can be retrieved simultaneously, enabling batch transfer of the data to the cloud data warehouse.

[0044] The advantages of this setup are that it automatically determines whether data needs to be transferred to the cloud data warehouse and controls the amount of data transferred each time, preventing errors caused by excessive data transfer and improving data transmission accuracy and efficiency. Each time data to be transferred is retrieved in batches, users don't need to manually select files, reducing user operations and improving transmission efficiency.

[0045] In this embodiment, obtaining the data to be transmitted from the preset application server includes: obtaining the data to be transmitted in batches from the preset data storage path of the application server according to the preset data transmission period.

[0046] Specifically, data to be transmitted can be scheduled to the cloud data warehouse at regular intervals. A data transmission cycle can be preset, for example, one hour, meaning data to be transmitted to the cloud data warehouse once per hour. Data on the application server is stored in a preset data storage path; therefore, data in this path can be transmitted to the cloud data warehouse at regular intervals. For example, 10 PM can be set as the preset data transmission time each day, allowing batch retrieval of data received within the 24 hours prior to 10 PM.

[0047] There can be one or more data storage paths. Each time the data volume is determined, the data to be transmitted in multiple data storage paths can be counted together. That is, the data to be transmitted in multiple data storage paths can be obtained in batches.

[0048] The advantages of this setup are that it automatically determines whether data needs to be transferred to the cloud data warehouse and controls the frequency of data transfer, avoiding too many or too few transfers, thus improving the accuracy and efficiency of data transmission. Each time data to be transferred is retrieved, it can be acquired in batches, eliminating the need for manual file selection by the user, reducing user operations, and improving transmission efficiency.

[0049] S102. Determine the port number of the cloud data warehouse according to the preset database connection information; wherein, the database connection information is used to indicate the information for establishing a connection between the application server and the cloud data warehouse, and the database connection information includes the port number of the cloud data warehouse, and the node port corresponding to the port number is used to receive the data to be transmitted.

[0050] For example, database connection information is pre-configured. This database connection information represents the information used to establish a connection between the application server and the cloud data warehouse. That is, a transmission channel between the application server and the cloud data warehouse can be established based on the database connection information. The database connection information may include a username, password, database name, and port number. The username and password can be used to determine whether the user has permission to control the application server to transmit data to the cloud data warehouse; the database name can refer to the name of the database in the cloud data warehouse where the data to be transmitted needs to be stored; and the port number can refer to the port number on which the cloud data warehouse receives the data to be transmitted.

[0051] The database connection information is pre-configured and does not need to be set every time the application server uploads data. When the application server needs to transmit data to the cloud data warehouse, it can obtain the preset port number from the preset database connection information, that is, determine the port number of the cloud data warehouse. The cloud data warehouse can receive the data to be transmitted through the Node Port corresponding to the port number.

[0052] S103. On the computing layer based on the cloud data warehouse, the node port corresponding to the port number loads the data to be transmitted into the computing layer; wherein, the cloud data warehouse includes a computing layer and a storage layer, and the data to be transmitted is used to be transmitted from the computing layer to the storage layer for storage.

[0053] For example, after determining the port number of the cloud data warehouse, the application server transmits the data to be transferred to the cloud data warehouse through the NodePort corresponding to that port number. The data to be transferred is loaded during the transfer process, meaning that the data to be transferred is already loaded when it is imported into the cloud data warehouse, and there is no need to perform the process of importing first and then loading.

[0054] A cloud data warehouse can include a cloud service layer, a computing layer, and a storage layer. NodePort is located in the computing layer. Data to be transmitted can first be loaded into the computing layer of the cloud data warehouse through NodePort, and then sent from the computing layer to the storage layer for storage. Figure 2 This is a schematic diagram illustrating data transmission between the application server and the cloud data warehouse in an embodiment of this application. Figure 2 In the cloud service layer, application servers can load data to be transmitted into containers in the compute layer via node ports. The compute layer can include multiple containers, which can be used to perform compute services. Each container can have its own container port to receive data transmitted from the node port. After performing compute services, the containers in the compute layer can transmit the computed data to the storage layer for storage. Multiple tenants can register in the cloud service layer, and each tenant can retrieve and view data from the storage layer.

[0055] In this embodiment, the computing layer of the cloud data warehouse is deployed in a preset cloud environment; based on the node port corresponding to the port number on the computing layer of the cloud data warehouse, the data to be transmitted is loaded into the computing layer, including: according to a preset data transmission protocol, based on the external exposed service of the node port corresponding to the port number in the cloud environment, the data to be transmitted is loaded in batches into the computing layer of the cloud data warehouse.

[0056] Specifically, the computing layer of a cloud data warehouse can be pre-deployed on Kubernetes (K8S). By deploying a K8S environment, the cloud data warehouse can utilize K8S's NodePort technology, which exposes services via static ports. Exposing services via static ports enables containers to access external application servers, thereby allowing batch import of data to be transferred from external application servers into the cloud data warehouse environment, meeting the requirements for batch loading.

[0057] Pre-configure the data transfer protocol for the data to be transferred. For example, the default data transfer protocol could be FTP (File Transfer Protocol). Using the FTP protocol, enable the NodePort service and load the data to be transferred in batches to the computing layer of the cloud data warehouse through the NodePort corresponding to the port number.

[0058] The advantage of this setup is that it creates a NodePort service in the K8S cloud environment and deploys the data transmission protocol on the application server, enabling data loading and batch script execution in the cloud data warehouse, thereby improving data transmission efficiency.

[0059] This application provides a data transmission method based on a cloud data warehouse. The method receives data to be transmitted through the node port of the cloud data warehouse's computing layer and can directly load the data from the application server into the cloud data warehouse, eliminating the need for prior import and loading. This allows data loading to be completed during the import process, avoiding the prior art method of first importing the data to be transmitted into object storage and then executing loading commands through the cloud data warehouse front-end environment, thus effectively improving data transmission efficiency.

[0060] Figure 3 This is a flowchart illustrating a data transmission method based on a cloud data warehouse, which is an optional embodiment based on the above embodiments.

[0061] In this embodiment, obtaining the data to be transmitted from the preset application server can be further refined as follows: batch obtaining data marked with a preset first identifier in the preset data storage path of the application server, which is the data to be transmitted; wherein, the first identifier is used to indicate that the data has not been transmitted to the cloud data warehouse.

[0062] like Figure 3 As shown, the method includes the following steps:

[0063] S301. Batch acquire data marked with a preset first identifier in the preset data storage path of the application server, which is the data to be transmitted; wherein, the first identifier is used to indicate that the data has not been transmitted to the cloud data warehouse.

[0064] For example, when an application server receives data such as transaction data or user information, it can add a preset first identifier to the received data. If the data has a first identifier, it indicates that the data has not yet been transmitted to the cloud data warehouse.

[0065] The application server can check in real-time or periodically whether a first identifier exists in any of its stored data. If the first identifier is present, the data is determined to be data to be imported into the cloud data warehouse. It can check whether the first identifier exists in data within a preset data storage path, without needing to check all data on the application server. It retrieves all data marked with a preset first identifier from the data storage path as data to be transferred, enabling batch retrieval of data to be transferred.

[0066] S302. Determine the port number of the cloud data warehouse according to the preset database connection information; wherein, the database connection information is used to indicate the information for establishing a connection between the application server and the cloud data warehouse, and the database connection information includes the port number of the cloud data warehouse, and the node port corresponding to the port number is used to receive the data to be transmitted.

[0067] For example, this step can refer to step S102 above, and will not be repeated here.

[0068] S303. On the computing layer based on the cloud data warehouse, the node port corresponding to the port number loads the data to be transmitted into the computing layer; wherein, the cloud data warehouse includes a computing layer and a storage layer, and the data to be transmitted is used to be transmitted from the computing layer to the storage layer for storage.

[0069] For example, according to a preset data transmission protocol, based on the externally exposed services of the node port corresponding to the port number in the cloud environment, the data to be transmitted is loaded in batches into the computing layer of the cloud data warehouse.

[0070] In this embodiment, on the computing layer based on the cloud data warehouse, after the node port corresponding to the port number loads the data to be transmitted into the computing layer, it further includes: replacing the first identifier of the data to be transmitted in the preset data storage path with a preset second identifier; wherein, the second identifier is used to indicate that the data has been transmitted to the cloud data warehouse.

[0071] Specifically, after the application server sends the data to be transmitted with a first identifier to the cloud data warehouse, it can change the first identifier of the data to be transmitted within the application server. A second identifier is pre-set, which can be used to indicate that the data has been transmitted to the cloud data warehouse. Replacing the first identifier of the data to be transmitted to the cloud data warehouse with the second identifier indicates that the data transmission has been completed. For example, preset characters can be added to the end of the data message as the first and second identifiers.

[0072] The advantage of this setup is that when the application server checks whether there is data to be transmitted with the first identifier, it can avoid identifying data that has already been transmitted as data to be transmitted, thus avoiding duplicate data transmission. It also eliminates the need for users to manually identify data to be transmitted, thereby improving the efficiency and accuracy of data transmission.

[0073] This application provides a data transmission method based on a cloud data warehouse. The method receives data to be transmitted through the node port of the cloud data warehouse's computing layer and can directly load the data from the application server into the cloud data warehouse, eliminating the need for prior import and loading. This allows data loading to be completed during the import process, avoiding the prior art method of first importing the data to be transmitted into object storage and then executing loading commands through the cloud data warehouse front-end environment, thus effectively improving data transmission efficiency.

[0074] Figure 4 This is a structural block diagram of a data transmission device based on a cloud data warehouse, provided as an embodiment of this application. This device is applied to an electronic device. For ease of explanation, only the parts relevant to the embodiments of this disclosure are shown. (Refer to...) Figure 4 The device includes: a data acquisition module 401, a port number determination module 402, and a data loading module 403.

[0075] The data acquisition module 401 is used to acquire data to be transmitted from a preset application server; wherein, the application server is used to store the data to be transmitted before data transmission.

[0076] The port number determination module 402 is used to determine the port number of the cloud data warehouse according to preset database connection information; wherein, the database connection information is used to indicate the information for establishing a connection between the application server and the cloud data warehouse, the database connection information includes the port number of the cloud data warehouse, and the node port corresponding to the port number is used to receive the data to be transmitted;

[0077] The data loading module 403 is used to load the data to be transmitted into the computing layer based on the node port corresponding to the port number on the computing layer of the cloud data warehouse; wherein, the cloud data warehouse includes a computing layer and a storage layer, and the data to be transmitted is used to be transmitted from the computing layer to the storage layer for storage.

[0078] Figure 5 This application provides a structural block diagram of a data transmission device based on a cloud data warehouse, in which... Figure 4 Based on the illustrated embodiments, as Figure 5 As shown, the data acquisition module 401 includes a data volume determination unit 4011 and a data volume comparison unit 4012.

[0079] The data volume determination unit 4011 is used to determine the data volume of the data to be transmitted in the preset data storage path of the application server;

[0080] The data volume comparison unit 4012 is used to acquire the data to be transmitted in batches if the data volume exceeds a preset data volume threshold.

[0081] In one example, data acquisition module 401 is specifically used for:

[0082] According to the preset data transmission cycle, the data to be transmitted is obtained in batches from the preset data storage path of the application server.

[0083] In one example, data acquisition module 401 is specifically used for:

[0084] Data marked with a preset first identifier in a preset data storage path on the application server is batch retrieved as the data to be transmitted; wherein, the first identifier is used to indicate that the data has not been transmitted to the cloud data warehouse.

[0085] In one example, the device also includes:

[0086] The identifier replacement module is used to replace the first identifier of the data to be transmitted in the preset data storage path with a preset second identifier after the node port corresponding to the port number loads the data to be transmitted into the computing layer based on the cloud data warehouse; wherein the second identifier is used to indicate that the data has been transmitted to the cloud data warehouse.

[0087] In one example, the compute layer of the cloud data warehouse is deployed in a pre-defined cloud environment;

[0088] Data loading module 403 is specifically used for:

[0089] According to the preset data transmission protocol, based on the external exposed services of the node port corresponding to the port number in the cloud environment, the data to be transmitted is loaded in batches into the computing layer of the cloud data warehouse.

[0090] Figure 6 A structural block diagram of an electronic device provided in an embodiment of this application, such as... Figure 6 As shown, the electronic device includes: a memory 61 and a processor 62; the memory 61 is a memory used to store instructions executable by the processor 62.

[0091] The processor 62 is configured to perform the methods provided in the above embodiments.

[0092] The electronic device also includes a receiver 63 and a transmitter 64. The receiver 63 is used to receive instructions and data sent by other devices, and the transmitter 64 is used to send instructions and data to external devices.

[0093] Figure 7This is a block diagram illustrating an electronic device according to an exemplary embodiment. The device may be a mobile phone, computer, digital broadcasting terminal, messaging device, game console, tablet device, medical device, fitness equipment, personal digital assistant, vehicle, etc.

[0094] Device 700 may include one or more of the following components: processing component 702, memory 704, power supply component 706, multimedia component 708, audio component 710, input / output (I / O) interface 712, sensor component 714, and communication component 716.

[0095] Processing component 702 typically controls the overall operation of device 700, such as operations associated with display, telephone calls, data communication, camera operation, and recording. Processing component 702 may include one or more processors 720 to execute instructions to perform all or part of the steps of the methods described above. Furthermore, processing component 702 may include one or more modules to facilitate interaction between processing component 702 and other components. For example, processing component 702 may include a multimedia module to facilitate interaction between multimedia component 708 and processing component 702.

[0096] Memory 704 is configured to store various types of data to support the operation of device 700. Examples of this data include instructions for any application or method operating on device 700, contact data, phonebook data, messages, pictures, videos, etc. Memory 704 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0097] Power supply component 706 provides power to various components of device 700. Power supply component 706 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to device 700.

[0098] Multimedia component 708 includes a screen that provides an output interface between the device 700 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touchscreen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touches, swipes, and gestures on the touch panel. The touch sensors may sense not only the boundaries of the touch or swipe action but also the duration and pressure associated with the touch or swipe operation. In some embodiments, multimedia component 708 includes a front-facing camera and / or a rear-facing camera. When the device 700 is in an operating mode, such as a shooting mode or a video mode, the front-facing camera and / or the rear-facing camera may receive external multimedia data. Each front-facing camera and rear-facing camera may be a fixed optical lens system or have focal length and optical zoom capabilities.

[0099] Audio component 710 is configured to output and / or input audio signals. For example, audio component 710 includes a microphone (MIC) configured to receive external audio signals when device 700 is in an operating mode, such as call mode, recording mode, and voice recognition mode. The received audio signals may be further stored in memory 704 or transmitted via communication component 716. In some embodiments, audio component 710 also includes a speaker for outputting audio signals.

[0100] I / O interface 712 provides an interface between processing component 702 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include, but are not limited to, home buttons, volume buttons, power buttons, and lock buttons.

[0101] Sensor assembly 714 includes one or more sensors for providing state assessments of various aspects of device 700. For example, sensor assembly 714 may detect the on / off state of device 700, the relative positioning of components such as the display and keypad of device 700, changes in the position of device 700 or a component of device 700, the presence or absence of user contact with device 700, the orientation or acceleration / deceleration of device 700, and temperature changes of device 700. Sensor assembly 714 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. Sensor assembly 714 may also include a light sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, sensor assembly 714 may also include an accelerometer, a gyroscope, a magnetometer, a pressure sensor, or a temperature sensor.

[0102] Communication component 716 is configured to facilitate wired or wireless communication between device 700 and other devices. Device 700 can access wireless networks based on communication standards, such as WiFi, 2G, or 3G, or combinations thereof. In one exemplary embodiment, communication component 716 receives broadcast signals or broadcast-related information from an external broadcast management system via a broadcast channel. In one exemplary embodiment, communication component 716 also includes a near-field communication (NFC) module to facilitate short-range communication. For example, the NFC module may be implemented based on radio frequency identification (RFID) technology, Infrared Data Association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology, and other technologies.

[0103] In an exemplary embodiment, device 700 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the methods described above.

[0104] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 704 including instructions, which can be executed by a processor 720 of device 700 to perform the above-described method. For example, the non-transitory computer-readable storage medium may be a ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, and optical data storage device, etc.

[0105] A non-transitory computer-readable storage medium, wherein when instructions in the storage medium are executed by a processor of a terminal device, the terminal device is able to execute the aforementioned data transmission method based on a cloud data warehouse.

[0106] This application also discloses a computer program product, including a computer program that, when executed by a processor, implements the method described in this embodiment.

[0107] Various embodiments of the systems and technologies described above in this application can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include: implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0108] The program code used to implement the methods of this application may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing device, such that when executed by the processor or controller, the functions / operations specified in the flowcharts and / or block diagrams are implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or electronic device.

[0109] In the context of this application, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. Machine-readable media can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0110] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0111] The systems and technologies described herein can be implemented in computing systems that include back-end components (e.g., as data electronic devices), or computing systems that include middleware components (e.g., application electronic devices), or computing systems that include front-end components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with implementations of the systems and technologies described herein), or any combination of such back-end, middleware, or front-end components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0112] Computer systems can include client and electronic devices. Clients and electronic devices are generally geographically separated and typically interact via communication networks. The client-electronic device relationship is created by computer programs running on the respective computers and having a client-electronic device relationship with each other. The electronic device can be a cloud electronic device, also known as a cloud computing electronic device or cloud host, a host product within the cloud computing service system, addressing the shortcomings of traditional physical hosts and VPS services ("Virtual Private Server," or simply "VPS") in terms of management difficulty and weak business scalability. The electronic device can also be an electronic device in a distributed system or an electronic device incorporating blockchain technology. It should be understood that the various forms of processes shown above can be used, with steps reordered, added, or deleted. For example, the steps described in this application can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this application is achieved, and this is not limited herein.

[0113] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this application are indicated by the following claims.

[0114] It should be understood that this application is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this application is limited only by the appended claims.

Claims

1. A data transmission method based on a cloud data warehouse, characterized in that, The method is applied to an electronic device, and the method includes: Data marked with a preset first identifier in a preset data storage path on the application server is batch retrieved as data to be transmitted; wherein, the first identifier is used to indicate that the data has not been transmitted to the cloud data warehouse, and the application server is used to store the data to be transmitted before data transmission. Based on preset database connection information, the port number of the cloud data warehouse is determined; wherein, the database connection information is used to indicate the information for establishing a connection between the application server and the cloud data warehouse, the database connection information includes the port number of the cloud data warehouse, and the node port corresponding to the port number is used to receive the data to be transmitted; Based on the computing layer of the cloud data warehouse, the node port corresponding to the port number loads the data to be transmitted into the computing layer; wherein, the cloud data warehouse includes a computing layer and a storage layer, and the data to be transmitted is used to be transmitted from the computing layer to the storage layer for storage; The first identifier of the data to be transmitted in the preset data storage path is replaced with a preset second identifier; wherein the second identifier is used to indicate that the data has been transmitted to the cloud data warehouse; the computing layer of the cloud data warehouse is deployed in a preset cloud environment, and based on the node port corresponding to the port number on the computing layer of the cloud data warehouse, the data to be transmitted is loaded into the computing layer, including: According to a preset data transmission protocol, based on the external exposed services of the node port corresponding to the port number in the cloud environment, the data to be transmitted is loaded in batches into the computing layer of the cloud data warehouse. The preset data transmission protocol is the FTP file transfer protocol, and the cloud environment is a K8S environment.

2. The method according to claim 1, characterized in that, Retrieve the data to be transmitted from the preset application server, including: Determine the amount of data to be transmitted in the preset data storage path of the application server; If the amount of data exceeds a preset data volume threshold, the data to be transmitted will be acquired in batches.

3. The method according to claim 1, characterized in that, Retrieve the data to be transmitted from the preset application server, including: According to the preset data transmission cycle, the data to be transmitted is obtained in batches from the preset data storage path of the application server.

4. The method according to claim 1, characterized in that, The computing layer of the cloud data warehouse is deployed in a pre-defined cloud environment; Based on the computing layer of the cloud data warehouse, the node port corresponding to the port number loads the data to be transmitted into the computing layer, including: According to the preset data transmission protocol, based on the external exposed services of the node port corresponding to the port number in the cloud environment, the data to be transmitted is loaded in batches into the computing layer of the cloud data warehouse.

5. A data transmission device based on a cloud data warehouse, characterized in that, The device is used in an electronic device, and the device includes: The data acquisition module is used to acquire in batches data marked with a preset first identifier in a preset data storage path on the application server, which is the data to be transmitted; wherein, the first identifier is used to indicate that the data has not been transmitted to the cloud data warehouse, and the application server is used to store the data to be transmitted before data transmission; The port number determination module is used to determine the port number of the cloud data warehouse based on preset database connection information; wherein, the database connection information is used to indicate the information for establishing a connection between the application server and the cloud data warehouse, the database connection information includes the port number of the cloud data warehouse, and the node port corresponding to the port number is used to receive the data to be transmitted; A data loading module is used to load the data to be transmitted into the computing layer based on the node port corresponding to the port number on the computing layer of the cloud data warehouse; wherein, the cloud data warehouse includes a computing layer and a storage layer, and the data to be transmitted is used to be transmitted from the computing layer to the storage layer for storage; The identifier replacement module is used to replace the first identifier of the data to be transmitted in the preset data storage path with a preset second identifier after the data to be transmitted is loaded into the computing layer of the node port corresponding to the port number based on the cloud data warehouse; wherein, the second identifier is used to indicate that the data has been transmitted to the cloud data warehouse. The computing layer of the cloud data warehouse is deployed in a preset cloud environment. The data loading module is specifically used to load the data to be transmitted into the computing layer of the cloud data warehouse in batches according to a preset data transmission protocol and based on the external exposed services of the node port corresponding to the port number in the cloud environment. The preset data transmission protocol is the FTP file transfer protocol and the cloud environment is a K8S environment.

6. An electronic device, characterized in that, include: A processor, and a memory communicatively connected to the processor; The memory stores computer-executed instructions; The processor executes computer execution instructions stored in the memory to implement the data transmission method based on a cloud data warehouse as described in any one of claims 1-4.

7. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the data transmission method based on a cloud data warehouse as described in any one of claims 1-4.

8. A computer program product, characterized in that, It includes a computer program that, when executed by a processor, implements the data transmission method based on a cloud data warehouse as described in any one of claims 1-4.

Images