Database system, data processing method and computing device

Abstract

The invention provides a database system, the database system comprises a first node and at least two second nodes, the first node is used for obtaining a first Python statement and a second Python statement, converting the first Python statement into a first SQL statement and sending the first SQL statement to the at least two second nodes, and the first SQL statement and the first Python statement have the same data processing function; the at least two second nodes are used for executing the first SQL statement in a distributed manner; the first node is also used for acquiring a first UDF having the same data processing function as the second Python statement, and transmitting the first UDF to the at least two second nodes; the at least two second nodes execute the first UDF in a distributed mode, and a first API with the same data processing function as the second Python statement in the Python tripartite library is called through the first UDF. The database can support full Python statements, and the data processing capacity is improved.

Description

[0001] This application claims priority to the Chinese patent application No. 202410430578.4, filed on April 10, 2024, with the State Intellectual Property Office of China, with the title of "A Request Response Method, Device and Computing Device Cluster", the whole content of which is incorporated herein by reference. TECHNICAL FIELD

[0002] The present application relates to the field of cloud computing, and more particularly, to a database system, a data processing method and a computing device. BACKGROUND

[0003] Artificial intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. Any AI model needs to be trained before being used to solve a specific technical problem. A database usually provides training data for the training process of an AI model as a data warehouse.

[0004] In some scenarios, before providing training data to an AI model, the data stored in the database needs to be processed, and the processed data is used as the training data of the AI model, so as to train the AI model. For example, a data engineer uses structured query language (SQL) to operate the database to complete filtering, cleaning and other operations on the data in the database; uses Python language to operate Python three-party libraries (such as Pandas data analysis tool) to complete data feature extraction and type conversion. Since the data engineer needs to master both SQL language and Python language, the human cost and learning cost are increased.

[0005] In some mainstream databases, Python application programming interface (API) is developed to enable the database to support SQL language and Python language. One solution is to translate original Python statements into matching SQL statements and then execute them by the database SQL engine, but since some original Python statements cannot be translated into matching SQL statements, this way cannot support full-quantity Python statements. Another solution is to continue processing data by using the interface of original Python Pandas for Python statements that cannot be translated into matching SQL statements, but since the engine inside Python three-party libraries uses single thread, the processing capacity for data is limited and cannot handle large data volume. SUMMARY

[0006] The application provides a database system which can support full Python statements and improve data processing capability.

[0007] In a first aspect, a database system is provided, which includes a first node and at least two second nodes. The first node is configured to obtain a first Python statement and a second Python statement, convert the first Python statement into a first structured query language (SQL) statement, and send the first SQL statement to the at least two second nodes. The first SQL statement has the same data processing function as the first Python statement. The at least two second nodes are configured to execute the first SQL statement in a distributed manner. The first node is further configured to obtain a first user-defined function (UDF), which has the same data processing function as the second Python statement. The at least two second nodes are further configured to execute the first UDF in a distributed manner, call a first application programming interface (API) in a Python third-party library through the first UDF, and the first API has the same data processing function as the second Python statement.

[0008] In the above technical solution, if a native Python statement can be converted into an SQL statement, the database converts it into a corresponding SQL statement and sends it to an SQL backend engine for execution. If the native Python statement cannot be converted into an SQL statement, the distributed execution capability of a UDF in the database is utilized to call a native Python Pandas operator in the UDF, so that the database can support full Python statements, and the distributed computing capability of the database can be used to execute the Python statements, thereby improving the data processing efficiency.

[0009] With reference to the first aspect, in some implementations of the first aspect, the first node is specifically configured to receive the first UDF input by a user.

[0010] With reference to the first aspect, in some implementations of the first aspect, the first node stores a plurality of UDFs, and the first UDF is determined by a user from the plurality of UDFs according to the data processing function of the second Python statement.

[0011] With reference to the first aspect, in some implementations of the first aspect, the first UDF is written by a user according to the data processing function of the second Python statement.

[0012] With reference to the first aspect, in some implementations of the first aspect, the database system stores a plurality of APIs in the Python third-party library, and the plurality of APIs include the first API.

[0013] With reference to the first aspect, in some implementations of the first aspect, the first node is further configured to merge at least two UDFs in the plurality of UDFs into one UDF, the at least two UDFs containing operators of the same data processing function.

[0014] In the above technical solution, the internal logic of the UDF can be opened, and fusion optimization can be implemented on the UDF, so as to accelerate the performance of the UDF.

[0015] With reference to the first aspect, in some implementations of the first aspect, the database system is a cloud database system, the cloud database system includes the plurality of nodes, and the cloud database system is deployed in cloud computing resources managed by a cloud management platform, and the cloud computing resources have a cloud storage service deployed therein.

[0016] The second aspect provides a data processing method, which is characterized in that the method is applied to a database system, and the database system includes a plurality of nodes, including a first node and at least two second nodes. The first node obtains a first Python statement and a second Python statement, converts the first Python statement into a first structured query language (SQL) statement, and sends the first SQL statement to the at least two second nodes. The first SQL statement has the same data processing function as the first Python statement. The at least two second nodes distribute and execute the first SQL statement. The first node obtains a first user-defined function (UDF), and the first UDF has the same data processing function as the second Python statement. The at least two second nodes distribute and execute the first UDF, call a first application programming interface (API) in a Python third-party library through the first UDF, and the first API has the same data processing function as the second Python statement.

[0017] With reference to the second aspect, in some implementations of the second aspect, the first node obtaining the first user-defined function (UDF) includes that the first node receiving the first UDF input by a user.

[0018] With reference to the second aspect, in some implementations of the second aspect, the first node stores a plurality of UDFs, and the first UDF is determined by a user from the plurality of UDFs according to a data processing function of the second Python statement.

[0019] With reference to the second aspect, in some implementations of the second aspect, the first UDF is written by a user according to a data processing function of the second Python statement.

[0020] With reference to the second aspect, in some implementations of the second aspect, the database system stores a plurality of APIs in the Python third-party library, and the plurality of APIs includes the first API.

[0021] With reference to the second aspect, in some implementations of the second aspect, the method further includes: merging, by the first node, at least two UDFs in the plurality of UDFs into one UDF, the at least two UDFs including operators with the same data processing function.

[0022] With reference to the second aspect, in some implementations of the second aspect, the database system is a cloud database system, the cloud database system includes the plurality of nodes, and the cloud database system is deployed in cloud computing resources managed by a cloud management platform, and the cloud computing resources have a cloud storage service deployed therein.

[0023] In a third aspect, a computing device is provided, including a processor and a memory, and optionally, an input / output interface. The processor is configured to control the input / output interface to receive and send information, and the memory is configured to store a computer program. The processor is configured to call and run the computer program from the memory, so as to execute the method in the second aspect or any possible implementation manner of the second aspect.

[0024] Optionally, the processor can be a general-purpose processor, which can be implemented by hardware or software. When implemented by hardware, the processor can be a logic circuit, an integrated circuit, or the like. When implemented by software, the processor can be a general-purpose processor, which reads software codes stored in the memory to implement the processor. The memory can be integrated in the processor or exist independently outside the processor.

[0025] In a fourth aspect, a computing device cluster is provided, including at least one computing device, each computing device including a processor and a memory. The processor of the at least one computing device is configured to execute instructions stored in the memory of the at least one computing device, so as to execute the method in the second aspect or any possible implementation manner of the second aspect.

[0026] In a fifth aspect, a chip is provided, which acquires instructions and executes the instructions to implement the method in the second aspect and any possible implementation manner of the second aspect.

[0027] Optionally, as an implementation manner, the chip includes a processor and a data interface. The processor reads instructions stored in the memory through the data interface, and executes the method in the second aspect and any possible implementation manner of the second aspect.

[0028] Optionally, as an implementation form, the chip further comprises a memory, and the memory stores instructions, and the processor is configured to execute the instructions stored in the memory, and the processor is configured to execute the method in the second aspect and any implementation form of the second aspect when the instructions are executed.

[0029] In a sixth aspect, a computer program product is provided, which comprises instructions, and when the instructions are executed by a computing device, the computing device executes the method in the second aspect and any implementation form of the second aspect.

[0030] In a seventh aspect, a computer readable storage medium is provided, which comprises computer program instructions, and when the computer program instructions are executed by a computing device, the computing device executes the method in the second aspect and any implementation form of the second aspect.

[0031] As an example, the computer readable storage includes, but is not limited to, one or more of the following: read-only memory (ROM), programmable ROM (PROM), erasable PROM (EPROM), Flash memory, electrically EPROM (EEPROM), and hard drive.

[0032] Optionally, as an implementation form, the storage medium can be a non-volatile storage medium. BRIEF DESCRIPTION OF DRAWINGS

[0033] Figure 1 is a schematic diagram of a scenario applied to an embodiment of the present application.

[0034] Figure 2 is a schematic block diagram of a cloud scenario suitable for an embodiment of the present application.

[0035] Figure 3 is a schematic block diagram of a database system provided by an embodiment of the present application.

[0036] Figure 4 is a schematic flowchart of a data processing method provided by an embodiment of the present application.

[0037] Figure 5 is a schematic diagram of data to be processed.

[0038] Figure 6 is a schematic diagram of data obtained after processing the data shown in Figure 5

[0039] Figure 7 is a schematic diagram of data obtained after processing the data shown in​Figure 6 a schematic diagram of data obtained after processing the illustrated data.

[0040] Figure 8 is a schematic block diagram of a front end and a back end of a database system provided by an embodiment of the present application.

[0041] Figure 9 is a schematic block diagram of a fusion optimization UDF provided by an embodiment of the present application.

[0042] Figure 10 is a schematic block diagram of an apparatus 1000 for data processing provided by an embodiment of the present application.

[0043] Figure 11 is an architectural schematic diagram of a computing device 1500 provided by an embodiment of the present application.

[0044] Figure 12 is an architectural schematic diagram of a computing device cluster provided by an embodiment of the present application.

[0045] Figure 13 is a schematic diagram of connection between the computing devices 1500A and 1500B through a network provided by an embodiment of the present application. DETAILED DESCRIPTION

[0046] The technical solutions in the present application will be described below with reference to the drawings.

[0047] The present application will present various aspects, embodiments or features around a system including a plurality of devices, components, modules, etc. It should be understood and appreciated that each system can include additional devices, components, modules, etc., and / or can not include all the devices, components, modules, etc. discussed in connection with the drawings. In addition, combinations of these solutions can also be used.

[0048] In addition, in the embodiments of the present application, the words "example", "for example", etc. are used to mean by way of example, illustration or description. Any embodiment or design solution described as "example" in the present application should not be interpreted as more preferred or more advantageous than other embodiments or design solutions. Rather, the word "example" is intended to present the concept in a specific manner.

[0049] In the embodiments of the present application, "corresponding" and "relevant" can be used interchangeably at times. It should be pointed out that when the distinction is not emphasized, the meanings expressed are consistent.

[0050] The service scenarios described in the embodiments of the present application are used to more clearly illustrate the technical solutions of the embodiments of the present application, and do not constitute a limitation on the technical solutions provided by the embodiments of the present application. Those skilled in the art can know that, with the evolution of network architecture and the appearance of new service scenarios, the technical solutions provided by the embodiments of the present application are also applicable to similar technical problems.

[0051] In this specification, reference to “one embodiment” or “some embodiments” means that a particular feature, structure, or characteristic described in connection with the embodiment is included in at least one embodiment of the application. The appearances of the phrases “in one embodiment”, “in some embodiments”, “in other embodiments”, “in additional embodiments”, and so on, in various places in the specification are not necessarily all referring to the same embodiment, unless otherwise specifically specified. The terms “comprising”, “including”, “having” and their variants mean “including but not limited to”, unless otherwise specifically specified.

[0052] In this specification, “at least one” means one or more, and “multiple” means two or more. “And / or” describes the association relationship of associated objects, which means that there can be three relationships, for example, A and / or B can represent the following cases: A exists alone, A and B exist together, and B exists alone, where A and B can be singular or plural. The character “ / ” generally represents an “or” relationship between the associated objects before and after it. “At least one of the following” or similar expressions means any combination of these items, including any combination of single or multiple items. For example, at least one of a, b, or c can represent a, b, c, a-b, a-c, b-c, or a-b-c, where a, b, and c can be single or multiple.

[0053] Artificial intelligence (AI) is the use of digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use knowledge to obtain the best results. In other words, artificial intelligence is a branch of computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can react in a similar way to human intelligence. Artificial intelligence is the design principle and implementation method of various intelligent machines, so that machines have the functions of perception, reasoning, and decision-making. The research in the field of artificial intelligence includes robots, natural language processing, computer vision, decision-making and reasoning, human-computer interaction, recommendation and search, AI basic theory, etc.

[0054] Before being used to solve a specific technical problem, any AI model needs to be trained. The training of an AI model refers to a process of using a specified initial model to calculate training data, adjusting the parameters in the initial model according to the calculated results, and making the model gradually learn certain rules and have specific functions. The AI model with stable functions after training can be used for inference. The inference of an AI model refers to a process of using a trained AI model to calculate input data and obtain a predicted inference result.

[0055] In the field of AI, a database is usually used as a data warehouse to provide training data for the training process of an AI model. For ease of understanding, the related concepts of a database are introduced as follows.

[0056] A database can be understood as a warehouse that organizes, stores and manages data according to a data structure. Specifically, a database is a collection of a large amount of data stored in a computer, organized, shareable and uniformly managed. A database can include multiple table spaces.

[0057] A database system refers to a system composed of a database and its management software. A database can be controlled by a database management system (DBMS) to realize operations such as addition (i.e., insertion), deletion, query or modification (i.e., update) of data in the database. In reality, the data in the database and the DBMS together are referred to as a database system, which is usually simply referred to as a database.

[0058] A cloud DB refers to a database deployed into a virtual computing environment, which can realize advantages such as pay-as-you-go, on-demand expansion, high availability and storage integration. The virtual computing environment can be a cloud platform. A cloud platform is a short name for a cloud computing platform, which can provide computing, network and storage capabilities based on hardware resources and software resources. Through a network "cloud", huge data computing is processed and analyzed in a remote end and then returned to a user, with characteristics such as large scale, distribution, virtualization, high availability, scalability, on-demand service and security. A cloud platform can realize rapid delivery and release of configurable computing resources with small management cost or low interactive complexity of a user and a business provider.

[0059] The cloud database is described in detail below. Figure 1

[0060] Figure 1 is a scenario schematic diagram applied to an embodiment of the present application. The scenario can include a terminal 101 and a server 102. The terminal 101 is directly or indirectly connected to the server 102 through a wireless network or a wired network.​

[0061] The terminal 101 can be at least one of a smartphone, a desktop computer, an augmented reality terminal, a tablet computer, an electronic book reader, and a laptop computer. The terminal 101 is installed and runs an application program supporting access to the cloud database. The application program can be a client of the cloud database or a browser, and the present application does not limit the same. For example, the application program is a client, and relevant personnel (such as an operation and maintenance personnel) can obtain abnormal analysis information of the database through the client, and perform corresponding operations based on the received abnormal analysis information to give an exception avoidance scheme or repair system problems.

[0062] The server 102 is a server of the cloud database system, and is configured to manage the cloud database, for example, to execute a parsing query request for the cloud database, to provide data communication, and the like. The server 102 is a physical server, or a server cluster or a distributed file system composed of multiple physical servers, or a cloud server providing cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms, and the like. In some embodiments, the server 102 is taken as an example of a cloud server, and the cloud server refers to a service based on hardware resources and software resources, providing computing, network, and storage capabilities. Through a network “cloud”, huge data computing is processed and analyzed in a remote end and returned to a user, and has characteristics of large scale, distribution, virtualization, high availability, scalability, on-demand service, and security. The cloud server can realize rapid delivery and release of configurable computing resources with small management cost or low interaction complexity between the user and the service provider. Illustratively, the server 102 is configured to provide database services for the client of the cloud database running on the terminal 101.

[0063] It should be noted that the terminal 101 can generally refer to one of multiple terminals, or a set of multiple terminals; and the server 102 can generally refer to one of multiple servers, or a set of multiple servers. The present application does not limit the number and types of devices in the implementation environment.

[0064] In some embodiments, the aforementioned wireless or wired networks utilize standard communications technologies and / or protocols. The networks include but are not limited to data center networks, storage area networks (SANs), local area networks (LANs), metropolitan area networks (MANs), wide area networks (WANs), mobile, wired or wireless networks, any combination thereof, or any combination of networks. In some implementations, technologies and / or formats including hyper text markup language (HTML), extensible markup language (XML), etc. are used to represent data exchanged over the networks. In addition, conventional encryption technologies such as secure sockets layer (SSL), transport layer security (TLS), virtual private networks (VPNs), internet protocol security (IPsec), etc. can be used to encrypt all or portions of links. In other embodiments, custom and / or proprietary data communications technologies can be used in place of or in combination with the aforementioned data communications technologies.

[0065] In some scenarios, before providing the training data to the AI model, the data stored in the database needs to be processed, and the processed data is used as the training data of the AI model, so as to train the AI model.

[0066] As an example, a data engineer can process the data stored in the database to obtain processed data. An algorithm engineer uses an AI framework to execute the training process of the AI model by using the processed data as the training data of the AI model.

[0067] For example, as shown in FIG. 1, the process of processing the data stored in the database by the data engineer can include but is not limited to the following processes: Figure 1

[0068] 1. Use structured query language (SQL) to operate the database to complete filtering, cleaning, etc. of the data in the database.

[0069] ​It should be appreciated that SQL is a standard programming language used to manage DBMS. It allows users to query, update, and manage data in a database through a series of commands.

[0070] 2. Use Python language to operate Python third-party libraries (such as Pandas, NumPy, Matplotlib, etc. data analysis tools) to complete data feature extraction and type conversion, such as converting data into a data type that can be recognized by an AI framework.

[0071] It should be appreciated that Python is an interpreted, interactive, object-oriented programming language. Python language emphasizes code readability and allows developers to express ideas with less code rather than using a large amount of complex code. Python supports multiple programming paradigms, including procedural, object-oriented, and functional programming. Python is one of the main languages in the field of data analysis, and it has libraries like pandas and numpy that make data processing and analysis very easy.

[0072] It should also be appreciated that Pandas is an open-source Python data analysis library that provides a large number of functions and methods that allow us to quickly and conveniently process data, making data processing and visualization more simple and efficient. It should be noted that the following embodiments of the method and device are described using the pandas library as an example of a python third-party library, but those skilled in the art should understand that the embodiments of the method and device are still applicable when the pandas library is replaced with other python third-party libraries in actual applications.

[0073] As an example, the main functions of Pandas include but are not limited to: data cleaning (for example, Pandas provides a wealth of data cleaning tools such as missing value handling, duplicate value detection, and data type conversion), data conversion (for example, Pandas can easily convert data into different formats such as importing data from a Comma-Separated Values (CSV) file into a DataFrame, or converting from one shape to another, generating derived variables, etc.), data analysis (for example, Pandas provides a wealth of functions to perform descriptive statistics, aggregation, and calculations on data sets), data visualization (for example, Pandas can be used in combination with other libraries for data visualization), time series analysis (for example, Pandas has built-in support for time series analysis, which can easily perform time series analysis).

[0074] In the process of the above data engineer processing data stored in the database, the data engineer needs to master both SQL language and Python language, thus increasing the human cost and learning cost. Moreover, since the engine inside the Python third-party library adopts single thread, the data processing capacity is limited, and distributed processing cannot be supported, and large data volume cannot be processed.

[0075] At present, in order to reduce the learning cost of the data engineer and improve the efficiency of data analysis, some mainstream databases develop Python application programming interface (API), so that the database can support SQL language and Python language. In this way, the data engineer can directly operate the database by using Python language, which not only reduces the learning cost, but also can use the database to perform data feature extraction and other data analysis work.

[0076] In the prior art, the response modes of some mainstream databases to Python statements (or Python functions) are as follows:

[0077] One existing scheme is that the mainstream database translates the received Python statement into a SQL statement matched with the Python statement, the SQL statement and the Python statement have the same data processing function. The SQL statement matched with the Python statement is sent to the SQL engine, and the corresponding operator in the SQL engine executes the SQL statement. In this scheme, since part of the original Python statement cannot be translated into a SQL statement matched with the Python statement, this mode cannot support full Python statements.

[0078] Another existing scheme is that the mainstream database provides a to_pandas API, which can convert the data structure of the SQL database into the data structure of the original Python Pandas, so as to continue to process the data by using the interface of the original Python Pandas. However, since the engine inside the Python Pandas adopts single thread, the data processing capacity is limited, and distributed processing cannot be supported, and large data volume cannot be processed.

[0079] Therefore, the embodiments of the present application provide a database system, which can support full Python statements, and can also execute Python statements based on the distributed computing capacity of the database, so as to improve the efficiency of data processing.

[0080] In one example, the database system provided by the embodiments of the present application can be applied to a cloud service scenario. First, the cloud service scenario is described in detail. Figure 2 The cloud service scenario is described in detail.

[0081] Figure 2 is a schematic block diagram of a cloud scenario applicable to embodiments of the present application. As shown in Figure 2 , the cloud scenario can include a cloud management platform 110, the Internet 120, and a client 130.

[0082] As shown in Figure 2 , the cloud management platform 110 is configured to manage an infrastructure that provides a plurality of cloud services. The infrastructure includes a plurality of cloud data centers, each of which includes a plurality of servers, each of which includes a cloud service resource that provides a corresponding cloud service for a tenant. In embodiments of the present application, the cloud service resource can be a cloud database.

[0083] The cloud management platform 110 can be located in a cloud data center, which can provide an access interface (such as an interface or an application program interface (API)). A tenant can remotely access the access interface to register a cloud account and a password with the cloud management platform 110 using the client 130, and log in to the cloud management platform 110. After the cloud management platform 110 successfully authenticates the cloud account and the password, the tenant can further select and purchase a virtual machine of a specific specification (processor, memory, disk) at the cloud management platform 110 by paying a fee. After the payment and purchase are successful, the cloud management platform 110 provides a remote login account and password of the purchased virtual machine, and the client 130 can remotely log in to the virtual machine, install and run an application of the tenant in the virtual machine. Therefore, the tenant can create, manage, log in to, and operate a virtual machine in the cloud data center through the cloud management platform 110. The virtual machine can also be referred to as an elastic compute service (ECS) or an elastic instance (different cloud service providers have different names).

[0084] It should be understood that a tenant of a cloud service can be an individual, an enterprise, a school, a hospital, an administrative organ, etc.

[0085] The functions of the cloud management platform 110 include, but are not limited to, a user console, a computing management service, a network management service, a storage management service, an authentication service, and an image management service. The user console provides an interface or an API to interact with a tenant, the computing management service is configured to manage servers running virtual machines and containers and bare metal servers, the network management service is configured to manage network services (such as gateways and firewalls), the storage management service is configured to manage storage services (such as data bucket services), the authentication service is configured to manage account and password of a tenant, and the image management service is configured to manage virtual machine images. A tenant uses the client 130 to log in to the cloud management platform 110 through the Internet 120 to manage the rented cloud service.

[0086] Figure 3is a schematic block diagram of a database system provided by an embodiment of the present application. As shown in Figure 3 The database system can include a first node 310 and at least two second nodes 320, where the first node 310 and the second nodes 320 are each configured to deploy a database instance. Taking the first node as an example, the node can be a server, a server cluster, a virtual machine, or a container. The first node 310 and the second nodes 320 can physically correspond to the same server or virtual machine or container, or can correspond to different servers or virtual machines or containers. The functions of the first node 310 and the at least two second nodes 320 are described in detail below.

[0087] 1. The first node 310:

[0088] The first node 310 is configured to obtain a first Python statement and a second Python statement, convert the first Python statement into a first SQL statement, and send the first SQL statement to the at least two second nodes 320, where the first SQL statement has the same data processing function as the first Python statement.

[0089] The first node 310 is further configured to obtain a first user-defined function (UDF) and pass the first UDF to the at least two second nodes 320, where the first UDF has the same data processing function as the second Python statement.

[0090] As an example, the first node 310 obtains the first UDF in the following manner: the first node 310 receives a first UDF input by a user.

[0091] For example, the first node 310 stores a plurality of UDFs, and the first UDF is determined by the user from the plurality of UDFs according to the data processing function of the second Python statement, and the UDF has the same data processing function as the first UDF.

[0092] For another example, the first UDF is written by the user according to the data processing function of the second Python statement.

[0093] In some embodiments, if at least two UDFs contain operators with the same data processing function, the at least two UDFs can also be combined into one UDF.

[0094] 2. The at least two second nodes 320:

[0095] The at least two second nodes 320 are configured to execute the first SQL statement in a distributed manner.

[0096] The at least two second nodes 320 are further used to execute the first UDF in a distributed manner, calling a first API in a Python third-party library through the first UDF. The first API has the same data processing function as the second Python statement.

[0097] In the above technical solution, if the native Python statement can be converted into an SQL statement, the database will convert it into the corresponding SQL statement and send it to the SQL backend engine for execution. If it cannot be converted into an SQL statement, the distributed execution capability of the UDF in the database is utilized to call the native Python Pandas operator within the UDF. This allows the database to support a full range of Python statements and execute Python statements based on the distributed computing capability of the database, thereby improving data processing efficiency.

[0098] The following combination Figure 4 ,by Figures 5-7 Taking the data processing process of as an example, the data processing method provided by the embodiment of the present application is described in detail. It should be understood that Figure 4 The examples are only intended to help those skilled in the art understand the embodiments of the present application, and are not intended to limit the embodiments of the present application to Figure 4 The specific numerical values ​​or specific scenarios shown in the examples. Figure 4 It is obvious that various equivalent modifications or changes can be made to the following examples, and such modifications and changes also fall within the scope of the embodiments of the present application.

[0099] For the convenience of description, the following Figures 5-7 , explaining the data processing process.

[0100] 1. Acquisition Figure 5 The data shown;

[0101] 2. Extract the data in the column named "name" from the data shown in 5 to obtain Figure 6 The data shown;

[0102] 3. Transpose the data shown in 6 to obtain Figure 7 The data shown.

[0103] As an example, to achieve Figures 5-7 The data processing process shown requires executing the following Python statements:

[0104] "df = read_csv(xxx)

[0105] df=df.filter(items=['name'])

[0106] df_transposed = df.T

[0107] The above "df = read_csv(xxx)" is a code snippet in Python programming language for reading a CSV file and storing its content into a DataFrame object. This line of code usually uses the read_csv function in pandas library to perform the reading. The "xxx" here is the path or file name of the CSV file to be read, and df is a variable name to store the data read from the CSV file, which can be, for example, Figure 5 the data shown.

[0108] It should be understood that in pandas, DataFrame object is used to store tabular data, so df (usually stands for "DataFrame") is a common naming choice.

[0109] The above "df = df.filter(items=['name'])" is a code snippet in Python programming language for selecting a specific column from the data stored in the original df. This line of code usually uses the filter method in pandas library, which is used to select columns or rows based on labels, for example, selecting the column named "name", and returns the data of this column to a new DataFrame object, which is df.filter, containing the data of the column named "name" in the original df, which can be, for example, Figure 6 the data shown.

[0110] The above "df_transposed = df.T" is a code snippet in Python programming language for transposing the data stored in the original df and storing the transposed data in a new DataFrame (e.g. df_transposed). This line of code usually uses an attribute "df.T" of the DataFrame object, which is used to transpose the data stored in the original df. The transposition operation is to exchange the rows and columns of the data stored in the original df, that is, the rows of the data in the original df will become the columns of the data in the new DataFrame (e.g. df_transposed), and the columns of the data in the original df will become the rows of the data in the new DataFrame (e.g. df_transposed). For example, the data shown in Figure 6 is transposed to the data shown in Figure 7 .

[0111] Figure 4 is a schematic flow chart of a data processing method provided by an embodiment of the present application. As shown in Figure 4As shown, the method can include steps 410-440, which are described in detail below.

[0112] Step 410: The database receives a data processing request message including a first Python statement and a second Python statement.

[0113] As an example, in an embodiment of the present application, the database can receive a data processing request message including a first Python statement and / or a second Python statement.

[0114] It should be understood that the above Python statement can also be referred to as a Python function, or a Python API, or a Python operator.

[0115] The above first Python statement is, for example, a statement that the database can directly translate into an SQL statement with the same data processing function.

[0116] The above second Python statement is, for example, a statement that the database cannot directly translate into an SQL statement with the same data processing function.

[0117] For example, taking the process of data processing of Figures 5-7 As an example, taking the process of data processing of

[0118] Step 420: The database converts the first Python statement into a corresponding first SQL statement.

[0119] In an embodiment of the present application, for the first Python statement included in the data processing request message, the database can directly convert the first Python statement into a first SQL statement with the same data processing function.

[0120] As an example, taking the first Python statement including "df = read_csv(xxx)", and "df = df.filter(items=['name'])" as an example, since the SQL engine also supports the "read_csv" and "filter" capabilities, the functions of reading data and selecting specific columns of data can be implemented through the "select" statement. Therefore, as shown in Figure 8As shown, the database can convert the first Python statement into a matching first SQL statement "select table.name from table" to achieve the same data processing function as the first Python statement.

[0121] Step 430: The database executes the first SQL statement.

[0122] In the embodiments of the present application, after the database obtains the first SQL statement matched with the first Python statement, the database can send the first SQL statement to the backend SQL engine, and execute the first SQL statement by calling the corresponding operator in the SQL engine.

[0123] For example, the first SQL statement is "select table.name from table", therefore, the Scan operator and the Project operator in the SQL engine can be called to execute the select statement, so as to realize the "read_csv" and "filter" operations in the Pandas data analysis tool.

[0124] It should be understood that the operators in the SQL engine refer to various operations or functions used in the SQL query execution process, which are used to process and convert data. These operators can be regarded as building blocks of SQL query statements during execution, and they work together to complete the query task.

[0125] The Scan operator described above is an operator in the SQL engine for performing scanning operations, which supports the "read_csv" capability described above. As an example, the Scan operator can be divided into a TableScan operator and an IndexScan operator. The TableScan operator is responsible for reading data pages in the form of continuous blocks from the disk, and in the SQL query, there are several tables, so there are several TableScan operations. When an index is created in the data table and the index field is used for query, the IndexScan operator is responsible for index scanning.

[0126] The Project operator described above is an operator in the SQL engine for performing projection operations, which supports the "filter" capability described above. For example, in the select statement, the Project operator is responsible for selecting relevant columns from the table according to the query field.

[0127] In the embodiments of this application, the first SQL statement can also be executed by multiple data nodes (DNs) in a distributed manner based on the distributed computing capability of the database. It should be understood by those skilled in the related art that the distributed computing capability is a basic operation capability of the distributed database, and the meaning of distributed computing is briefly described herein. When the SQL statement is used to request to perform a target operation on the to-be-processed data, in the case that the database has multiple computing nodes, the target operation can be performed on a part of the to-be-processed data by each node respectively, so as to realize the distributed execution of the SQL statement.

[0128] In the following, two DNs (DN1 and DN2) are taken as an example, and the distributed execution of the first SQL statement is described in combination with the distributed computing capability of the database. Figure 8 The execution result of the first SQL statement on DN1 and DN2 is exemplarily described.

[0129] 1. DN1 executes the first SQL statement "select table.name from table" by executing a Scan operator and a Project operator in sequence.

[0130] Firstly, DN1 scans the data by executing the Scan operator, and obtains the data in the table stored on DN1, wherein the execution result of the Scan operator on DN1 is as follows:

[0131] Alice 9.5 male 0

[0132] Bob 8.0 male 0

[0133] Then, DN1 takes the above execution result of the Scan operator as the input of the Project operator, and selects the data of the name column in the table stored on DN1 by executing the Project operator, wherein the execution result of the Project operator on DN1 is as follows:

[0134] Alice

[0135] Bob

[0136] 2. DN2 executes the first SQL statement "select table.name from table" by executing a Scan operator and a Project operator in sequence.

[0137] Firstly, DN2 scans the data by executing the Scan operator, and obtains the data in the table stored on DN2, wherein the execution result of the Scan operator on DN2 is as follows:

[0138] LiLei 4.0 male 1

[0139] Jesse 9.0 male 2

[0140] After that, DN2 takes the above result obtained after executing the Scan operator as the input of the Project operator, and selects the data of the name column in the table stored in DN2 by executing the Project operator, wherein the result of DN2 after executing the Project operator is as follows:

[0141] LiLei

[0142] Jesse

[0143] Step 440: The database calls the transpose operator of native Python Pandas by executing the first UDF, which is a UDF with the same data processing capability as the second Python statement selected from multiple UDFs.

[0144] In the embodiments of the present application, when the database cannot be directly translated into an SQL statement with the same data processing function as the second Python statement, a first UDF with the same data processing capability as the second Python statement can be selected from multiple UDFs.

[0145] For example, taking the second Python statement "df_transposed=df.T" as an example, since the second Python statement performs a transpose operation on the data, and the database SQL engine does not support the data transpose operation, the second Python statement cannot be translated into a corresponding SQL statement. Therefore, as shown in Figure 8 The database provides a first UDF "udf_T(partition)" that matches the function of the second Python statement, which calls the transpose operator of native Python Pandas internally to achieve the same transpose function as the second Python statement.

[0146] In the embodiments of the present application, the database can execute the first UDF, which calls the transpose operator of native Python Pandas to achieve the same transpose function as the second Python statement.

[0147] For example, the database can invoke a "map_reduce" sub-function (also referred to as a map_reduce operator) in a first UDF "udf_T(partition)", and invoke a transpose operator of native Python Pandas in the "map_reduce" sub-function. The "map_reduce" sub-function includes a map function and a reduce function, i.e., a map operator and a reduce operator of the database. The main function of the map function is to apply each part of the data to be processed to a function or operator and generate a new data set in the case that the data set to be processed is divided into several parts. The main function of the reduce function is to aggregate and calculate the data to generate the final output result.

[0148] In the embodiments of the present application, the first UDF can also be executed by the plurality of data nodes DN in a distributed manner based on the distributed computing capability of the database. It can be understood that the UDF is also used to request to perform the target operation on the data to be processed, which is similar to the distributed execution of the SQL statement. In the case that the database has a plurality of computing nodes, the target operation corresponding to the UDF can be performed on a part of the data to be processed by each node, so as to realize the distributed execution of the UDF. Hereinafter, two DN (DN1 and DN2) are taken as an example, and the distributed execution of the first UDF is described in combination with the following examples. Figure 8 The execution results of the first UDF on DN1 and DN2 are taken as an example.

[0149] 1. The execution result of the map (Pandas, T, partition) function in the first UDF executed by DN1 is as follows:

[0150] Alice Bob

[0151] It should be understood that DN1 executes the map (Pandas, T, partition) function, and the transpose T operator of Python Pandas is invoked in the map function, which is used to perform the transpose operation on the result of the execution of the Project operator of DN1 to obtain the execution result.

[0152] 2. The execution result of the map (Pandas, T, partition) function in the first UDF executed by DN2 is as follows:

[0153] LiLei Jesse

[0154] It should be understood that the DN2 executes the map (Pandas, T, partition) function, in which the transpose T operator of the Python Pandas is called, which is used to transpose the result of the execution of the Project operator of the DN2.

[0155] 3. After obtaining the result executed by the DN1 and the result executed by the DN2, the DN3 executes the reduce function in the first UDF to aggregate the result executed by the DN1 and the result executed by the DN2, and the result aggregated by the DN3 is as follows:

[0156] Alice Bob

[0157] LiLei Jesse

[0158] It should be understood that the DN3 executing the reduce function in the first UDF can be the DN1, or can be the DN2, or can be another DN other than the ND1 and the ND2, which is not limited in the application.

[0159] It should be noted that in the embodiments of the application, the operators (which can also be referred to as functions or APIs) of the native Python Pandas can be stored in the database.

[0160] In the above technical solution, the native Python statement that can be converted into a SQL statement is converted into a SQL statement and sent to a SQL backend engine for execution; the native Python statement that cannot be converted into a SQL statement is used to call the operator of the native Python Pandas in the UDF, so that full Python statements can be supported in the SQL database, and the data processing efficiency can be improved based on the distributed computing capability of the database.

[0161] Considering that the number of native Python Pandas operators is large, the database generates a corresponding UDF for each native Python Pandas operator, which is a large workload. In the embodiments of the application, on the one hand, the functions of commonly used operators in Python Pandas can be implemented by calling the original operators of the SQL engine and the internal UDF (the UDF directly generated by the database, which directly calls the native Python Pandas operator); on the other hand, the subOperator can be opened for advanced users, so that the advanced users can directly call the corresponding function and freely combine the functions in the database, thereby reducing the workload of the database developers and quickly implementing the distributed execution of the native Python Pandas operator.

[0162] In the present application, since the UDF runs the exact native Python Pandas API, in one possible embodiment, the internal logic of the UDF can also be opened to implement fusion optimization for the UDF, thereby accelerating the performance of the UDF.

[0163] For example, as shown in FIG. 6, the OP1, OP2, OP3, OP4, and OP5 operators form a logical Plan, wherein the OP3 is a UDF and calls the map_reduce sub-function to perform distributed execution of the Python Pandas API. Since the OP3 is followed by the OP5 and both perform map operations, the reduce operation in the OP3 can be eliminated, and the database can fuse the OP3 and the OP5 into an OP6 operator. Figure 9

[0164] The above describes in detail the database and the data processing method provided by the embodiments of the present application, and the following will describe in detail the embodiments of the device of the present application in conjunction with Figures 1 to 9 The description of the method embodiments corresponds to the description of the device embodiments, and thus, the parts not described in detail can be referred to the foregoing method embodiments. Figures 10-13

[0165] Figure 10 FIG. 10 is a schematic block diagram of a data processing device 1000 provided by the embodiments of the present application. The device 1000 can be implemented by software, hardware, or a combination of both.

[0166] The device 1000 includes an acquisition module 1010 and a processing module 1020, wherein the acquisition module 1010 is configured to acquire a first Python statement and a second Python statement by a first node, convert the first Python statement into a first structured query language (SQL) statement, and send the first SQL statement to at least two second nodes, the first SQL statement having the same data processing function as the first Python statement; the processing module 1020 is configured to perform the first SQL statement in a distributed manner by the at least two second nodes; the acquisition module 1010 is further configured to acquire a first user-defined function (UDF) by the first node, the first UDF having the same data processing function as the second Python statement; and the processing module 1020 is further configured to perform the first UDF in a distributed manner by the at least two second nodes, and call a first application programming interface (API) in a Python third-party library through the first UDF, the first API having the same data processing function as the second Python statement.

[0167] Optionally, the acquisition module 1010 is specifically configured to receive the first UDF input by a user by the first node. ​​

[0168] Optionally, the first node stores a plurality of UDFs, and the first UDF is determined from the plurality of UDFs according to a function of data processing of the second Python statement.

[0169] Optionally, the first UDF is written by a user according to a function of data processing of the second Python statement.

[0170] Optionally, the database system stores a plurality of APIs in the Python third-party library, and the plurality of APIs includes the first API.

[0171] Optionally, the processing module 1020 is further configured to combine at least two UDFs in the plurality of UDFs into one UDF by the first node, and the at least two UDFs contain operators with the same data processing function.

[0172] Optionally, the database system is a cloud database system, the cloud database system includes the plurality of nodes, the cloud database system is deployed in cloud computing resources managed by a cloud management platform, and the cloud computing resources are deployed with a cloud storage service.

[0173] The apparatus 1000 herein can be embodied in the form of functional modules. The term “module” herein can be implemented by software and / or hardware, and no specific limitation is made thereto.

[0174] For example, the “module” can be a software program, a hardware circuit, or a combination of both, which implements the above functions. For example, the implementation of the obtaining module 1010 is described below. Similarly, the implementation of other modules, such as the processing module 1020, can refer to the implementation of the obtaining module 1010.

[0175] As an example of a software functional unit, the obtaining module 1010 can include code running on a computing instance. The computing instance can include at least one of a physical host (computing device), a virtual machine, and a container. Further, the computing instance can be one or more. For example, the obtaining module 1010 can include code running on multiple hosts / virtual machines / containers. It should be noted that the multiple hosts / virtual machines / containers used to run the code can be distributed in the same region (region), or in different regions. Further, the multiple hosts / virtual machines / containers used to run the code can be distributed in the same availability zone (AZ), or in different AZs, each AZ including a data center or multiple data centers with similar geographical locations. Generally, one region can include multiple AZs.

[0176] Likewise, the multiple hosts / virtual machines / containers used to run the code can be distributed in the same virtual private cloud (VPC) or in multiple VPCs. Among them, usually one VPC is set in one region, and communication between two VPCs in the same region or between VPCs in different regions needs to set a communication gateway in each VPC to realize the interconnection between VPCs through the communication gateway.

[0177] The acquisition module 1010 as an example of a hardware functional unit can include at least one computing device, such as a server, etc. Alternatively, the acquisition module 1010 can also be a device implemented by an application-specific integrated circuit (ASIC) or a programmable logic device (PLD), etc. Among them, the above-mentioned PLD can be implemented by a complex programmable logic device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), or any combination thereof.

[0178] The multiple computing devices included in the acquisition module 1010 can be distributed in the same region or in different regions. The multiple computing devices included in the acquisition module 1010 can be distributed in the same AZ or in different AZs. Likewise, the multiple computing devices included in the acquisition module 1010 can be distributed in the same VPC or in multiple VPCs. Among them, the multiple computing devices can be any combination of servers, ASICs, PLDs, CPLDs, FPGAs, and GALs, etc.

[0179] Therefore, the modules of the various examples described in the embodiments of the present application can be implemented in electronic hardware or a combination of computer software and electronic hardware. Whether the functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Professional technicians can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0180] It should be noted that the apparatus provided by the above embodiments is only used for executing the above method, and the above division of the functional modules is used for example, and in actual application, the above functions can be completed by different functional modules according to needs, that is, the internal structure of the apparatus is divided into different functional modules to complete all or part of the above described functions. For example, the acquisition module 1010 can be used to execute any step in the above method, the processing module 1020 can be used to execute any step in the above method, and the processing module 1020 can be used to execute any step in the above method. The steps implemented by the acquisition module 1010 and the processing module 1020 can be specified as needed, and the entire function of the above apparatus can be implemented by the acquisition module 1010 and the processing module 1020 respectively implementing different steps in the above method.

[0181] In addition, the apparatus and method embodiments provided by the above embodiments belong to the same concept, and the specific implementation process is described in the method embodiments above, which will not be repeated here.

[0182] The method provided by the embodiments of the present application can be executed by a computing device, which can also be referred to as a computer system. It includes a hardware layer, an operating system layer running on the hardware layer, and an application layer running on the operating system layer. The hardware layer includes hardware such as processing units, memories, and memory control units, and the functions and structures of the hardware will be described in detail later. The operating system is any one or more computer operating systems that implement business processing through processes, such as Linux operating system, Unix operating system, Android operating system, iOS operating system, or windows operating system, etc. The application layer contains browser, address book, word processing software, instant messaging software, etc. application program. And optionally, the computer system is a handheld device such as a smart phone, or a terminal device such as a personal computer, and the present application is not particularly limited as long as it can execute the method provided by the embodiments of the present application. The execution subject of the method provided by the embodiments of the present application can be a computing device, or a functional module in the computing device that can call and execute a program.

[0183] The following will be described in combination with Figure 11 A computing device provided by the embodiments of the present application will be described in detail.

[0184] Figure 11 is a schematic diagram of the architecture of a computing device 1500 provided by the embodiments of the present application. The computing device 1500 can be a server or a computer or other device with computing capability. Figure 11 The computing device 1500 shown includes at least one processor 1510 and a memory 1520.

[0185] It should be appreciated that the number of processors and memories in the computing device 1500 is not limited.

[0186] The processor 1510 executes instructions in the memory 1520, so that the computing device 1500 implements the method provided by the present application. Alternatively, the processor 1510 executes instructions in the memory 1520, so that the computing device 1500 implements the functional modules provided by the present application, thereby implementing the method provided by the present application.

[0187] Optionally, the computing device 1500 further includes a communication interface 1530. The communication interface 1530 uses a transceiver module such as, but not limited to, a network interface card and a transceiver, to implement the communication between the computing device 1500 and other devices or communication networks.

[0188] Optionally, the computing device 1500 further includes a system bus 1540, wherein the processor 1510, the memory 1520 and the communication interface 1530 are connected with the system bus 1540 respectively. The processor 1510 can access the memory 1520 through the system bus 1540, for example, the processor 1510 can read and write data in the memory 1520 or execute code in the memory 1520 through the system bus 1540. The system bus 1540 is a peripheral component interconnect express (PCI) bus or an extended industry standard architecture (EISA) bus, etc. The system bus 1540 is divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 11 In the figure, only one thick line is used to represent the system bus 1540, but it does not mean that there is only one bus or only one type of bus.

[0189] In one possible implementation, the function of the processor 1510 is mainly to interpret the instructions (or code) of the computer program and process the data in the computer software. The instructions of the computer program and the data in the computer software can be saved in the memory 1520 or the cache 1516.

[0190] Optionally, the processor 1510 is a chip that has a processing capability of signals. By way of example, and not limitation, the processor 1510 is a general purpose processor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), or other programmable logic device, discrete gate or transistor logic, discrete hardware components. The general purpose processor is a microprocessor, etc. For example, the processor 1510 is a central processing unit (CPU).

[0191] Optionally, each processor 1510 includes at least one processing unit 1512 and a memory control unit 1514.

[0192] Optionally, the processing unit 1512 is also called a core or a kernel, and is the most important component of the processor. The processing unit 1512 is manufactured by single crystal silicon through a certain production process, and all calculations, command reception, command storage, and data processing of the processor are performed by the core. The processing units respectively independently run program instructions, and use the parallel computing capability to speed up the running of the program. Various processing units have fixed logic structures, for example, the processing unit includes logic units such as a first-level cache, a second-level cache, an execution unit, an instruction-level unit, and a bus interface.

[0193] Optionally, the memory control unit 1514 is used to control the data interaction between the memory 1520 and the processing unit 1512. Specifically, the memory control unit 1514 receives a memory access request from the processing unit 1512, and controls the access to the memory based on the memory access request. By way of example, and not limitation, the memory control unit is a memory management unit (MMU) or the like.

[0194] Optionally, each memory control unit 1514 addresses the memory 1520 through a system bus. An arbiter (not shown in the figure) is configured in the system bus, and is responsible for processing and coordinating the competitive access of the plurality of processing units 1512. Figure 11

[0195] Optionally, the processing unit 1512 and the memory control unit 1514 are communicatively connected through an internal connection line of the chip, for example, an address line, so as to realize the communication between the processing unit 1512 and the memory control unit 1514.

[0196] ​Optionally, each processor 1510 also includes a cache 1516, which provides temporary storage of often-used data and instructions. Recent use patterns of the data can be tracked by the cache control 1518 in order to maximize the effectiveness of the cache 1516. The cache 1516 is typically more expensive per byte of storage than the main memory 1520, but is faster. Therefore, the cache 1516 acts as a bridge between the main memory 1520 and the processor 1510, allowing the processor 1510 to run faster while the more expensive cache 1516 does the work of fetching data and instructions and storing the results of access intensive workloads.

[0197] The memory 1520 can provide a space for processes in the computing device 1500 to run, for example, the memory 1520 stores computer programs (in particular, the codes of the programs) for generating the processes. After the computer programs are run by the processor to generate the processes, the processor allocates corresponding storage spaces in the memory 1520 for the processes. Further, the storage spaces further include a text segment, an initialized data segment, a bit initialized data segment, a stack segment, a heap segment, and the like. The memory 1520 stores data generated during the running of the processes, for example, intermediate data, or process data, and the like, in the storage spaces corresponding to the processes.

[0198] Optionally, the memory is also referred to as the internal memory, which is used to temporarily store the data for operation in the processor 1510, and exchange the data with the external memory such as the hard disk. As long as the computer is running, the processor 1510 will call the data needed for operation to the internal memory for operation, and the processing unit 1512 will transmit the results after the operation is completed.

[0199] By way of example, and not limitation, memory 1520 is volatile memory or nonvolatile memory, or can include both volatile and nonvolatile memory. By way of example, and not limitation, nonvolatile memory can be read-only memory (ROM), programmable ROM (PROM), erasable PROM (EPROM), electrically EPROM (EEPROM), or flash memory. Volatile memory is random access memory (RAM), which acts as external cache. By way of example, and not limitation, many forms of RAM are available, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double-data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), Synchlink DRAM (SLDRAM), and direct rambus RAM (DRRAM). The system and method described herein can be stored on or transmitted across one or more of these and any other suitable types of memory.

[0200] The structure of the computing device 1500 listed above is only exemplary, and the present application is not limited thereto. The computing device 1500 of the embodiments of the present application includes various hardware in the prior art computer system, for example, the computing device 1500 also includes other memories in addition to the memory 1520, such as disk memories and the like. Those skilled in the art should understand that the computing device 1500 can also include other devices necessary for normal operation. Meanwhile, according to specific needs, those skilled in the art should understand that the above-mentioned computing device 1500 can also include hardware devices for realizing other additional functions. In addition, those skilled in the art should understand that the above-mentioned computing device 1500 can also only include devices necessary for realizing the embodiments of the present application, and it is not necessary to include all the devices shown in the prior art. Figure 11 The computing device 1500 of the embodiments of the present application can also include other devices necessary for normal operation. Meanwhile, according to specific needs, those skilled in the art should understand that the above-mentioned computing device 1500 can also include hardware devices for realizing other additional functions. In addition, those skilled in the art should understand that the above-mentioned computing device 1500 can also only include devices necessary for realizing the embodiments of the present application, and it is not necessary to include all the devices shown in the prior art.

[0201] The embodiments of the present application also provide a computing device cluster. The computing device cluster includes at least one computing device. The computing device can be a server. In some embodiments, the computing device can also be a desktop computer, a notebook computer, or a terminal device such as a smart phone.

[0202] like Figure 12 As shown, the computing device cluster includes at least one computing device 1500. The memory 1520 in one or more computing devices 1500 in the computing device cluster may store the same instructions for executing the above method.

[0203] In some possible implementations, the memory 1520 of one or more computing devices 1500 in the computing device cluster may also store some instructions for executing the above method. In other words, the combination of one or more computing devices 1500 can jointly execute the instructions of the above method.

[0204] It should be noted that the memory 1520 in different computing devices 1500 in the computing device cluster can store different instructions, each for performing part of the functions of the above-mentioned apparatus. In other words, the instructions stored in the memory 1520 in different computing devices 1500 can implement the functions of one or more modules in the above-mentioned apparatus.

[0205] In some possible implementations, one or more computing devices in a computing device cluster may be connected via a network, which may be a wide area network or a local area network. Figure 13 A possible implementation is shown. Figure 13 As shown, two computing devices 1500A and 1500B are connected via a network. Specifically, the connection to the network is achieved through a communication interface in each computing device.

[0206] It should be understood that Figure 13 The functionality of computing device 1500A shown in FIG. 15 may also be implemented by multiple computing devices 1500. Similarly, the functionality of computing device 1500B may also be implemented by multiple computing devices 1500.

[0207] This embodiment also provides a computer program product including instructions. The computer program product may be software or a program product including instructions that can be run on a computing device or stored in any available medium. When the computer program product is run on a computing device, it causes the computing device to perform the method provided above, or causes the computing device to implement the functions of the apparatus provided above.

[0208] In this embodiment, a computer readable storage medium is also provided. The computer readable storage medium can be any available media or data storage device that can be accessed by the computing device and includes one or more of the available media. The available media can be magnetic media, (e.g., floppy disks, hard disks, tapes), optical media (e.g., CD-ROMs, DVDs), or semiconductor media (e.g., solid state hard drives), etc. The computer readable storage medium includes instructions that, when executed on the computing device, cause the computing device to perform the method provided above.

[0209] It should be understood that the size of the sequence number of each process described above does not mean the order of execution, and the execution order of each process should be determined according to its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0210] Those skilled in the art can realize that the units and algorithm steps of each example described in combination with the embodiments disclosed herein can be realized by electronic hardware or a combination of computer software and electronic hardware. Whether the functions are realized in hardware or software depends on the specific application and design constraints of the technical solution. The skilled person can use different methods to realize the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0211] Those skilled in the art can clearly understand that, for the convenience and brevity of the description, the specific working process of the system, device and unit described above can refer to the corresponding process in the foregoing method embodiments, which will not be repeated here.

[0212] In several embodiments provided in the present application, it should be understood that the disclosed system, device and method can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division, and actual implementation can have another division manner. For example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the units shown or discussed can be indirect coupling or communication connection through some interface, device or unit, and can be electrical, mechanical or other forms.

[0213] The units described as separate components can or can not be physically separate, and the components shown as units can or can not be physical units, i.e. they can be located in one place, or can be distributed on a plurality of network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the embodiment.

[0214] In addition, each function unit in each embodiment of the present application can be integrated in one processing unit, or each unit can be physically present separately, or two or more units can be integrated in one unit.

[0215] If the functions are realized in the form of software function units and sold or used as independent products, they can be stored in a computer readable storage medium. Based on this understanding, the technical solutions of the present application or the parts of the technical solutions that essentially contribute to the prior art can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes a plurality of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in each embodiment of the present application. The aforementioned storage medium includes a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and various media that can store program codes.

[0216] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art can easily think of changes or replacements within the technical scope disclosed in the present application, which should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A database system, characterized in that: The database system is a cloud database system, which includes a first node and at least two second nodes. The cloud database system is deployed in cloud computing resources managed by a cloud management platform, and a cloud storage service is deployed in the cloud computing resources, wherein: The first node and the at least two second nodes are both used to deploy a database instance; The first node is configured to obtain a first Python statement and a second Python statement, convert the first Python statement into a first structured query language SQL statement, and send the first SQL statement to the at least two second nodes, wherein the first SQL statement and the first Python statement have the same data processing function; The at least two second nodes are used for distributed execution of the first SQL statement; The first node is further configured to obtain a first user-defined function (UDF) and pass the first UDF to the at least two second nodes, wherein the first UDF and the second Python statement have the same data processing function; The at least two second nodes are further used to execute the first UDF in a distributed manner, calling a first application programming interface API in a Python third-party library through the first UDF, where the first API and the second Python statement have the same data processing function.

2. The database system according to claim 1, wherein: The first node is specifically configured to: The first UDF receives user input.

3. The database system according to claim 2, characterized in that The first node stores a plurality of UDFs, and the first UDF is determined by a user from the plurality of UDFs according to a data processing function of the second Python statement.

4. The database system according to claim 2, characterized in that The first UDF is a data processing function written by a user according to the second Python statement.

5. The database system according to any one of claims 1 to 4, characterized in that: The database system stores multiple APIs in the Python third-party library, and the multiple APIs include the first API.

6. The database system according to any one of claims 3 to 5, characterized in that: The first node is further configured to merge at least two UDFs among the multiple UDFs into one UDF, where the at least two UDFs include operators with the same data processing function.

7. The database system according to claim 1, characterized in that The first UDF calls the map_reduce operator of the database, and the map_reduce operator calls the first API.

8. A data processing method, characterized in that: The method is applied to a database system, which is a cloud database system. The cloud database system includes a first node and at least two second nodes. The cloud database system is deployed in cloud computing resources managed by a cloud management platform. A cloud storage service is deployed in the cloud computing resources. The database system includes a first node and at least two second nodes. The first node and the at least two second nodes are both used to deploy a database instance. The first node obtains a first Python statement and a second Python statement, converts the first Python statement into a first structured query language SQL statement, and sends the first SQL statement to the at least two second nodes, where the first SQL statement and the first Python statement have the same data processing function; The at least two second nodes execute the first SQL statement in a distributed manner; The first node obtains a first user-defined function (UDF) and transmits the first UDF to the at least two second nodes, wherein the first UDF and the second Python statement have the same data processing function; The at least two second nodes execute the first UDF in a distributed manner, and call a first application programming interface API in a Python tripartite library through the first UDF. The first API and the second Python statement have the same data processing function.

9. The method according to claim 8, characterized in that The first node obtains a first user-defined function (UDF), including: The first node receives the first UDF input by a user.

10. The method according to claim 9, characterized in that The first node stores a plurality of UDFs, and the first UDF is determined by a user from the plurality of UDFs according to a data processing function of the second Python statement.

11. The method according to claim 9, characterized in that The first UDF is a data processing function written by a user according to the second Python statement.

12. The method according to any one of claims 8 to 11, characterized in that The database system stores multiple APIs in the Python third-party library, and the multiple APIs include the first API.

13. The method according to any one of claims 10 to 12, characterized in that The method further comprises: The first node merges at least two UDFs among the multiple UDFs into one UDF, where the at least two UDFs include operators with the same data processing function.

14. The method according to claim 8, characterized in that The first UDF calls the map_reduce operator of the database, and the map_reduce operator calls the first API.

15. A computing device cluster, characterized in that: comprising at least one computing device, each computing device including a processor and a memory; The processor of the at least one computing device is configured to execute instructions stored in a memory of the at least one computing device, so that the computing device cluster executes the method according to any one of claims 8 to 14.

16. A computer program product comprising instructions, characterized in that When the instructions are executed by a computing device cluster, the computing device cluster is caused to perform the method according to any one of claims 8 to 14.

17. A computer-readable storage medium, characterized in that The method comprises computer program instructions which, when executed by a computing device cluster, cause the computing device cluster to perform the method according to any one of claims 8 to 14.

Images