Data processing method, data system, electronic device, and storage medium

Abstract

A data processing method, a data system, an electronic device and a storage medium. The data processing unit is configured to execute one or more jobs in the project, and the database unit is configured to execute database operations. The method comprises: in response to a data usage demand for the database generated in the process of job execution, the data processing unit calls the database unit to perform corresponding database operations; and in response to a calling demand of a first target job generated in the process of database operation, the database unit calls the data processing unit to perform a data processing operation corresponding to the first target job on the data in the database, and the data processing unit returns the result of the data processing operation corresponding to the first target job to the database unit. The method enables the task execution operation in the data processing unit and the database operation in the database unit to be more deeply integrated, enables the database related functions to be conveniently combined with the tasks executed by the data processing unit, and improves the efficiency of job execution.

Description

Technical Field

[0001] Embodiments of this disclosure relate to a data processing method, a data system, an electronic device, and a storage medium. Background Technology

[0002] A growing number of data processing tasks, such as big data analytics, model training, and model inference—all artificial intelligence tasks—rely on massive amounts of data. In data science and AI applications, there is an urgent need to manage high-dimensional vector data, a trend driven by the explosive growth of unstructured data (e.g., images, videos, and audio) and machine learning. Unstructured data analysis is widely used in practical applications. Many database systems are beginning to incorporate unstructured data analysis to meet this need, and machine learning models typically transform unstructured data into feature vectors for data analysis. Summary of the Invention

[0003] At least one embodiment of this disclosure provides a data processing method for a data system including a data processing unit and a database unit. The data processing unit is configured to execute one or more jobs in a project, and the database unit is configured to perform database operations. The data processing method includes: in response to a data usage request for the database generated during job execution, the data processing unit calls the database unit to perform a corresponding database operation; and in response to a request to call a first target job generated during database operations, the database unit calls the data processing unit to perform a data processing operation corresponding to the first target job on the data in the database, and the data processing unit returns the result of the data processing operation corresponding to the first target job to the database unit.

[0004] For example, in at least one embodiment of the data processing method provided in this disclosure, the data processing unit calls the database unit to perform corresponding database operations, including: the data processing unit calls the database unit using the database retrieval module and transmits a data query request to the database unit; in response to receiving the data query request, the database unit performs a data query operation on the database based on the data query request and returns the queried data to the data processing unit; and / or, the data processing unit calls the database unit using the database retrieval module and transmits a data update request to the database unit; in response to receiving the data update request, the database unit performs a data update operation on the database based on the data update request.

[0005] For example, in the data processing method provided in at least one embodiment of this disclosure, the data query operation includes an approximate query operation, which includes a vector-based approximate query operation and / or a hybrid query operation based on labels and vectors.

[0006] For example, in at least one embodiment of the data processing method provided in this disclosure, the data processing unit calls the database unit to perform corresponding database operations, including: the data processing unit calls the database unit using the database retrieval module and passes a first function invocation request to the database unit; the database unit responds to the first function invocation request by running a first user-defined function to process the data in the database, wherein the first user-defined function is configured to call at least one task process of the data processing unit to perform data processing operations corresponding to a second target job on the data in the database, and obtain the result of the data processing operation corresponding to the second target job returned by the data processing unit.

[0007] For example, in the data processing method provided in at least one embodiment of this disclosure, the database retrieval module includes a database software development kit, which runs in the deployment environment of the data processing unit.

[0008] For example, in the data processing method provided in at least one embodiment of this disclosure, the invocation request of the first target job includes a second function invocation request for a second user-defined function corresponding to the first target job; the database unit calls the data processing unit to perform data processing operations corresponding to the first target job on the data in the database, and the data processing unit returns the result of the data processing operation corresponding to the first target job to the database unit, including: the database unit responds to the second function invocation request by running the second user-defined function, the second user-defined function being configured to call at least one task process of the data processing unit to perform data processing operations corresponding to the first target job on the data in the database, and obtaining the result of the data processing operation corresponding to the first target job returned by the data processing unit.

[0009] For example, in the data processing method provided in at least one embodiment of this disclosure, the first target job is a job that can be called after it is in the running state. The data processing method further includes: the data processing unit runs the first target job and generates a first call port, and the database unit registers the first call port as a second user-defined function in the database unit.

[0010] For example, at least one embodiment of the data processing method provided in this disclosure further includes: in response to a call request for a second call port generated during job execution, the data processing unit calls the second call port to perform data processing operations corresponding to the third target job on the data specified in the call request, wherein the second call port is generated after the data processing unit executes the third target job that is callable after it is in the running state.

[0011] At least one embodiment of this disclosure provides a data processing method for a data system including a data processing unit and a database unit. The data processing unit is configured to execute one or more jobs in a project, and the database unit is configured to execute database operations. The one or more jobs include a first job, which is a model training job. The data processing method includes: in response to a first data query request being generated during the execution of the first job, the data processing unit calls the database unit and passes the first data query request to the database unit; in response to receiving the first data query request, the database unit performs a data query operation corresponding to the first data query request on the database and returns the queried first sample data to the data processing unit; the data processing unit uses the first sample data to train a model.

[0012] For example, in at least one embodiment of the data processing method provided in this disclosure, the data processing unit trains a model using first sample data to obtain a model after the Nth round of training; the data processing method further includes: the data processing unit performs model inference on the first sample data using the model after the Nth round of training to obtain a model inference result for the first sample data, the model inference result for the first sample data including a vector; in response to a first data update request generated during the execution of a first job, the database unit updates the model inference result for the first sample data to the database, the model inference result for the first sample data including a model inference result for the target sample data; in response to a second data query request generated during the execution of a first job, the data processing unit calls the database unit and transmits the second data query request to the database unit; in response to receiving the second data query request, the database unit processes the data... The library executes the data query operation corresponding to the second data query request, and returns the second sample data obtained from the first sample data to the data processing unit. The second data query request is a hybrid query request, and the query result of the hybrid query request satisfies both distance and attribute conditions. The distance condition in the second data query request includes that the distance between the vector corresponding to the query result and the vector corresponding to the target sample data is greater than a first preset condition, and the attribute condition in the second data query request includes that the query result is a positive sample of the target sample data; or, the distance condition in the second data query request includes that the distance between the vector corresponding to the query result and the vector corresponding to the target sample data is less than a second preset condition, and the attribute condition in the second data query request includes that the query result is a negative sample of the target sample data. The data processing unit uses the second sample data to train the model after the Nth round of training.

[0013] For example, in at least one embodiment of the data processing method provided in this disclosure, the data processing unit trains the model using first sample data to obtain the model after the Nth round of training; the data processing method further includes: the data processing unit uses the model after the Nth round of training to perform model inference on unlabeled data to obtain the model inference result of unlabeled data, the model inference result of unlabeled data including vectors, and the unlabeled data being obtained from a database query; in response to a second data update request generated during the execution of the first job, the database unit updates the model inference result of unlabeled data to the database, the model inference result of unlabeled data including the model inference result of the target unlabeled data; in response to a third data query request generated during the execution of the first job, the data processing unit calls the database unit and passes the third data query request to the database unit; in response to receiving the third data query request, the database unit performs a data query operation corresponding to the third data query request on the database, and returns the first unlabeled data obtained from the unlabeled data to the data processing unit; the third data query request is a mixed query request. The query results satisfy both distance and attribute conditions; the distance condition in the third data query request includes that the distance between the vector corresponding to the query result and the vector corresponding to the target unlabeled data is less than a third preset condition, and the attribute condition in the third data query request includes that the query result is unlabeled data; or, the distance condition in the third data query request includes that the distance between the vector corresponding to the query result and the vector corresponding to the target unlabeled data is greater than a fourth preset condition, and the attribute condition in the third data query request includes that the query result is unlabeled data; in response to the distance between the vector corresponding to the first unlabeled data and the vector corresponding to the target unlabeled data being less than the third preset condition, the first unlabeled data and the target unlabeled data are labeled as positive samples; and / or, in response to the distance between the vector corresponding to the first unlabeled data and the vector corresponding to the target unlabeled data being greater than the fourth preset condition, the first unlabeled data and the target unlabeled data are labeled as negative samples, and the unlabeled data labeled as positive samples and / or negative samples are used as new sample data; the data processing unit uses the new sample data to train the model after the Nth round of training.

[0014] For example, in the data processing method provided in at least one embodiment of this disclosure, the attribute conditions in the third data query request also include auxiliary conditions.

[0015] For example, in the data processing method provided in at least one embodiment of this disclosure, one or more jobs further include a second job, the second job being a model service job, and the data processing method further includes: declaring the trained model as a second job that can be called after it is in a running state, running the second job, and generating a calling port corresponding to the second job.

[0016] For example, in at least one embodiment of the data processing method provided in this disclosure, one or more jobs further include a third job. The data processing method further includes: in response to a model inference request for first specified data in the database generated during the execution of the third job, the data processing unit calls the database unit and passes a third function call request to the database unit; the database unit runs a third user-defined function to perform model inference operations on the first specified data in response to the third function call request, and obtains a first model inference result returned by the data processing unit; the database unit stores the first model inference result in the database or returns it to the data processing unit; the third user-defined function is configured to call at least one task process of the data processing unit to perform data processing operations corresponding to the second job on the data in the database, and obtain the result of the data processing operation corresponding to the second job returned by the data processing unit; the calling port corresponding to the second job is registered as the third user-defined function in the database unit.

[0017] For example, in at least one embodiment of the data processing method provided in this disclosure, one or more operations further include a third operation, and the data processing method further includes: in response to a fourth data query request for second specified data in the database being generated during the execution of the third operation, the data processing unit calls the database unit and transmits the fourth data query request to the database unit; in response to receiving the fourth data query request, the database unit performs a data query operation corresponding to the fourth data query request on the database based on the data query request, and returns the first data obtained by the query to the data processing unit; in response to a call request for a call port corresponding to the second operation being generated during the execution of the third operation, the data processing unit calls the call port corresponding to the second operation, performs a model inference operation corresponding to the second operation on the second specified data, and obtains a second model inference result; in response to a third data update request for the second model inference result being generated during the execution of the third operation, the data processing unit calls the database unit and transmits the third data update request to the database unit; in response to receiving the third data update request, the database unit updates the second model inference result to the database.

[0018] For example, at least one embodiment of the data processing method provided in this disclosure further includes: the database unit running a third user-defined function to perform data processing operations on second specified data in the database to obtain a third model inference result returned by the data processing unit; the database unit storing the third model inference result in the database; the third user-defined function being configured to call at least one task process of the data processing unit to perform model inference operations corresponding to a second job on the data in the database to obtain the result of the data processing operation corresponding to the second job returned by the data processing unit, and the calling port corresponding to the second job being registered as the third user-defined function in the database unit.

[0019] For example, in the data processing method provided in at least one embodiment of this disclosure, the model is a feature extraction model, the result of model inference is the extracted features, one or more jobs further include a fourth job, and the data processing method further includes: in response to the generation of an approximate query request for the features extracted from the database during the execution of the fourth job, the data processing unit calls the database unit and passes the approximate query request to the database unit; in response to receiving the approximate query request, the database unit performs an approximate query operation on the extracted features based on the approximate query request, and returns the first data corresponding to the features that satisfy the distance conditions contained in the approximate query request to the data processing unit, and the data processing unit uses the first data as the execution result of the fourth job or performs subsequent processing on the first data to obtain the execution result of the fourth job.

[0020] For example, in the data processing method provided in at least one embodiment of this disclosure, one or more jobs further include a fifth job, and the subsequent processing of the first data further includes: in response to a call request generated during the execution of the fourth job for a call port corresponding to the fifth job, the data processing unit calls the call port corresponding to the fifth job to perform the data processing operation corresponding to the fifth job on the first data, obtains the data processing result corresponding to the fifth job, and determines the execution result of the fourth job by the data processing result corresponding to the fifth job; the fifth job is a job that is in a callable state after running.

[0021] For example, in the data processing method provided in at least one embodiment of this disclosure, the model is a feature extraction model, the result of model inference is the extracted feature, and the data processing method further includes: the database unit performs an approximate query operation on the features extracted from the database to obtain the second data corresponding to the features that satisfy the distance conditions contained in the approximate query operation.

[0022] For example, in at least one embodiment of the data processing method provided in this disclosure, one or more jobs further include a fifth job, and the data processing method further includes: the database unit calls a fourth user-defined function to perform data processing operations on the second data, and obtains the data processing result corresponding to the fifth job returned by the data processing unit; the fourth user-defined function is configured to call at least one task process of the data processing unit to perform the data processing operation corresponding to the fifth job on the data in the database, and obtain the result of the data processing operation corresponding to the fifth job returned by the data processing unit; the fifth job is in a callable state after running, and the calling port corresponding to the fifth job has been registered as a user-defined function in the database unit.

[0023] At least one embodiment of this disclosure also provides a data system, including: a data processing unit and a database unit. The data processing unit is configured to call the database unit to perform corresponding database operations in response to a data usage request for the database generated during job execution. The database unit is configured to call the data processing unit to perform data processing operations corresponding to the first target job on the data in the database in response to a request for a first target job generated during database operations. The data processing unit returns the result of the data processing operation corresponding to the first target job to the database unit.

[0024] At least one embodiment of this disclosure also provides an electronic device, including the data system provided in at least one embodiment of this disclosure.

[0025] At least one embodiment of this disclosure also provides an electronic device, including: a processor; a memory including one or more computer program modules; the one or more computer program modules are stored in the memory and configured to be executed by the processor, the one or more computer program modules being used to implement the data processing method provided in at least one embodiment of this disclosure.

[0026] At least one embodiment of this disclosure also provides a storage medium storing non-transitory computer-readable instructions that, when executed by a computer, implement the data processing method provided in at least one embodiment of this disclosure. Attached Figure Description

[0027] To more clearly illustrate the technical solutions of the embodiments of this disclosure, the accompanying drawings of the embodiments will be briefly described below. Obviously, the drawings described below only relate to some embodiments of this disclosure and are not intended to limit this disclosure.

[0028] Figure 1 A schematic flowchart of a data processing method provided in at least one embodiment of the present disclosure is shown;

[0029] Figure 2 A schematic flowchart illustrating an example of step S101 provided in at least one embodiment of this disclosure is shown;

[0030] Figure 3 A schematic flowchart illustrating another example of step S101 provided in at least one embodiment of this disclosure is shown;

[0031] Figure 4 A schematic flowchart of a data processing method provided in at least one embodiment of the present disclosure is shown;

[0032] Figure 5 A schematic block diagram of a data system provided by at least one embodiment of the present disclosure is shown;

[0033] Figure 6A A schematic block diagram of an electronic device provided for at least one embodiment of this disclosure;

[0034] Figure 6B A schematic block diagram of another electronic device provided for some embodiments of this disclosure;

[0035] Figure 7 A schematic block diagram of yet another electronic device provided for some embodiments of this disclosure;

[0036] Figure 8 This is a schematic diagram of a storage medium provided for some embodiments of this disclosure. Detailed Implementation

[0037] To make the objectives, technical solutions, and advantages of the embodiments of this disclosure clearer, the technical solutions of the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this disclosure. All other embodiments obtained by those skilled in the art based on the described embodiments of this disclosure without creative effort are within the scope of protection of this disclosure.

[0038] Unless otherwise defined, the technical or scientific terms used in this disclosure shall have the ordinary meaning understood by one of ordinary skill in the art to which this disclosure pertains. The terms “first,” “second,” and similar terms used in this disclosure do not indicate any order, quantity, or importance, but are merely used to distinguish different components. Similarly, the terms “an,” “a,” or “the,” and similar terms do not indicate a quantity limitation, but rather indicate the presence of at least one. The terms “including,” “comprising,” or “containing,” and similar terms mean that the element or object preceding the word encompasses the elements or objects listed following the word and their equivalents, without excluding other elements or objects. The terms “connected,” “linked,” or similar terms are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. The terms “upper,” “lower,” “left,” and “right,” etc., are used only to indicate relative positional relationships, and these relative positional relationships may change accordingly when the absolute position of the described objects changes.

[0039] Performing data processing tasks such as artificial intelligence and big data-related tasks requires the use of massive amounts of structured and unstructured data. In some data processing systems, the data processing system itself is not integrated with the data storage system. On the one hand, the data processing system does not fully utilize the efficient storage, management, and query capabilities of the data storage system for massive amounts of data. On the other hand, the data storage system does not fully utilize the diverse data processing capabilities of the data processing units, making it difficult to leverage these capabilities to perform diverse data processing on the data in the data storage system.

[0040] At least one embodiment of this disclosure provides a data processing method for a data system including a data processing unit and a database unit. The data processing unit is configured to execute one or more jobs in a project, and the database unit is configured to perform database operations. The data processing method includes: in response to a data usage request for the database generated during job execution, the data processing unit calls the database unit to perform a corresponding database operation; and in response to a request to call a first target job generated during database operations, the database unit calls the data processing unit to perform a data processing operation corresponding to the first target job on the data in the database, and the data processing unit returns the result of the data processing operation corresponding to the first target job to the database unit.

[0041] The data processing method provided in the above embodiments of this disclosure enables a deeper integration of job execution operations in the data processing unit and database operations in the database unit. It can conveniently combine database-related functions with the tasks executed by the data processing unit. This not only allows the data processing unit to conveniently utilize the massive data storage, management, and query capabilities of the database, but also allows the database unit to conveniently utilize the functions of the data processing unit to expand its data processing capabilities. This improves the efficiency of job execution in the data system and makes the data system more adaptable to big data processing tasks and artificial intelligence tasks.

[0042] At least one embodiment of this disclosure also provides a data system corresponding to the above-described data processing method, an electronic device including the data system, and a storage medium for non-transitory storage of instructions for implementing the above-described data processing method.

[0043] The embodiments of this disclosure will now be described in detail with reference to the accompanying drawings, but this disclosure is not limited to these specific embodiments.

[0044] Figure 1 A schematic flowchart illustrating a data processing method according to at least one embodiment of the present disclosure is shown. This data processing method is used in a data system comprising a data processing unit and a database unit, wherein the data processing unit is configured to execute one or more jobs in a process, and the database unit is configured to perform database operations.

[0045] For example, data systems are used to perform data processing tasks. These processing tasks can be artificial intelligence tasks such as model training and model inference, or big data analysis and processing tasks. Performing these processing tasks typically requires massive amounts of data, such as unstructured data and / or structured data.

[0046] For example, the data processing unit includes an artificial intelligence unit that can perform artificial intelligence tasks, such as training, inference, or other processing of artificial intelligence models. Of course, the embodiments of this disclosure are not limited to this; the data processing unit can also be other units with processing functions, and is not limited to an artificial intelligence unit. The embodiments of this disclosure do not impose any limitations on this.

[0047] A project is created by the user to implement a data processing task. A data processing task can be a complete project or one or more jobs within a project. For example, a project can be a complete data processing task, and a project can include at least one job, such as one or more jobs with execution order and dependencies.

[0048] like Figure 1 As shown, the data processing method includes the following steps S101 to S102.

[0049] Step S101: In response to a data usage request for the database generated during job execution, the data processing unit calls the database unit to perform the corresponding database operation.

[0050] For example, the database can be a relational database, which can be operated using Structured Query Language (SQL).

[0051] Databases can store structured data, which can correspond to unstructured data (e.g., images including pictures and / or videos, text, audio, gene sequences, protein sequences, molecular formulas, molecular structures, etc.). Unstructured data can be stored in third-party unstructured data storage systems such as object data storage systems or file systems. The structured data corresponding to unstructured data includes, for example, the ID of the unstructured data, the date of its creation, the tags of the unstructured data, the representation of the unstructured data (e.g., vectors or graphs), and the storage address (url) of the unstructured data in the unstructured storage system. This information about the same unstructured data can be stored in a corresponding row in a database table. Table 1 shows an example of information storage for unstructured data (e.g., images).

[0052] Table 1: Example of information storage of structured data corresponding to images in a database table

[0053]

[0054] For example, when the data processing unit executes a job, it may generate data usage requirements for the database (such as data querying, updating, approximate data querying, and data processing). In such cases, the database unit needs to be invoked to perform the corresponding database operations. In some examples, when the data processing unit is developed using Python, when Python parses an SQL statement, it indicates that SQL statements are needed to perform database-related operations, thus generating data usage requirements. For instance, when an SQL statement is parsed during job execution, the database service client (db client) is invoked to access the database and perform data queries and updates using SQL statements. For example, data queries can be performed using the "SELECT" statement in SQL, and data updates can be performed using the "UPDATE" statement in SQL.

[0055] Database operations refer to a series of operations performed on data in a database, including querying data, updating data (including writing data, modifying data, deleting data, etc.). For example, SQL can be used to implement database operations.

[0056] Therefore, by performing database operations using SQL statements, such as using "SELECT" and "UPDATE" to query and update data, the data processing unit can utilize the database unit's SQL data engine to efficiently store, manage, and query massive amounts of data when executing jobs, thereby improving the processing efficiency of data processing tasks.

[0057] Step S102: In response to the request to call the first target job during the database operation, the database unit calls the data processing unit to perform the data processing operation corresponding to the first target job on the data in the database, and the data processing unit returns the result of the data processing operation corresponding to the first target job to the database unit.

[0058] For example, the first target job is a job that can be invoked after execution. After running the first target job in the data processing unit, a corresponding call port is generated (at this point, the first target job becomes a callable service). In the database unit, this call port (i.e., the program corresponding to the first target job) can be registered as a UDF (user-defined function). Then, the data processing unit can be invoked to perform data processing operations corresponding to the first target job on the data in the database by calling the UDF. After execution, the data processing unit returns the processing result. The functions and operations that the UDF can implement depend on the functions of the first target job corresponding to the registered call port, which can be determined according to actual needs. The embodiments of this disclosure do not impose any limitations on this. Thus, the database unit can use the services provided by the data processing unit by utilizing the UDF, thereby expanding the database unit's data processing capabilities.

[0059] It is understood that "data in the database" in this disclosure can be interpreted broadly, including not only the data itself in the database, but also data stored in a third-party storage system, where the address of the data in the third-party storage system is stored in the database.

[0060] For example, a job A can be set up in the data processing unit and invoked after execution. After job A is run in the data processing unit, a corresponding process is started, and a call port for that process is generated. This call port can be registered as a User-Defined Function (UDF) in the database unit. Subsequently, when the database unit calls this UDF, it further calls the call port. The database unit sends data from its database to the data processing unit through this call port. The data processing unit uses the previously started process, or additional processes (more processes can be started if the data processing unit has elastic computing capabilities), to perform the data processing operations corresponding to job A on the data obtained through the call port. Finally, the data processing unit returns the processing result corresponding to job A to the database unit, thus completing this call.

[0061] During job execution, the data processing unit and the database unit can call each other multiple times. For example, when the database unit calls a UDF, it calls the aforementioned call port through the UDF, thereby invoking the data processing unit to perform the corresponding data processing for the job corresponding to the call port. During job execution, if the data processing unit parses and obtains an SQL statement, it will then call the database unit based on the SQL statement to retrieve, query, and update data. This process can occur multiple times, enabling the data processing unit and the database unit to call each other repeatedly, achieving deep interaction between the two units and enhancing their deep integration.

[0062] Step S101 is an exemplary method for the data processing unit to call the database unit, and step S102 is an exemplary method for the database unit to call the data processing unit. The data processing unit and the database unit can call each other. There are several scenarios for the data processing unit to call the database unit, which will be discussed below. Figure 2 and Figure 3 Please provide a detailed explanation.

[0063] Figure 2 A schematic flowchart illustrating an example of step S101 provided in at least one embodiment of this disclosure is shown.

[0064] like Figure 2 As shown, step S101 includes the following steps S201 to S204.

[0065] Step S201: The data processing unit uses the database retrieval module to call the database unit and transmits a data query request to the database unit.

[0066] For example, in some embodiments of this disclosure, the database retrieval module includes a Database Software Development Kit (DB SDK), which runs in the deployment environment of the data processing unit. Taking a code file written in Python as an example, when an SQL instruction is parsed from the Python program, Python calls the DataBase SDK (pre-installed in the Python runtime environment, also known as the DB SDK). The DB SDK issues SQL instructions to the database, thereby executing the corresponding database operations. The SDK can refer to a software development kit. Step S202: In response to receiving a data query request, the database unit performs a data query operation on the database based on the data query request and returns the retrieved data to the data processing unit.

[0067] Step S203: The data processing unit uses the database retrieval module to call the database unit and sends a data update request to the database unit.

[0068] Step S204: In response to receiving a data update request, the database unit performs a data update operation on the database based on the data update request.

[0069] Data querying and updating are common database operations. For example, the "SELECT" statement is used to query data, and the "UPDATE" statement is used to update data. Querying allows for the rapid retrieval of data that meets specific criteria from the database; updating allows the results obtained by the data processing unit to be written to the database unit in a timely manner. This facilitates the use of this data in subsequent steps of the job or in other tasks, and also allows for further processing of the data using the database unit's massive storage, management, and querying capabilities. Therefore, by calling the database unit from within the data processing unit, the database unit's massive data storage, management, and querying capabilities can be fully utilized.

[0070] For example, in some examples, the data processing unit needs to perform a model training task. Therefore, the data processing unit can call the database unit to perform a data query operation, which retrieves specific data from the database for model training. In other examples, the data processing unit generates new data while running a task and needs to update the database with this new data. Therefore, the data processing unit calls the database unit to perform a data update operation. This data update operation can be writing the data processing results (e.g., object classification) of a task (e.g., object categories) into the database for use by subsequent steps of the same task or by other tasks, or it can be used by the database itself (e.g., checking in the database whether the object category is correct).

[0071] Data query operations can include exact queries, that is, filtering data from the database that exactly matches the query criteria. For example, in some examples, a data query operation might retrieve the `id` and `label` columns from the `test` table in the database, with the SQL statement: `Select id label from 'test'`. Alternatively, a data query operation might find the rows in the `test` table where `label = cat` and retrieve their corresponding `id` and `label` columns, with the SQL statement: `Select id label from 'test' where label = "cat"`.

[0072] For example, a data update operation includes writing target data into a specified database table. The target data may be new data generated by the data processing unit during job execution, or it may be the data processing result obtained from the data processing corresponding to the job execution. The data update operation can replace the existing data in the specified database table with the target data.

[0073] Data query operations and data update operations can be used in combination. For example, a query can be performed first, followed by an update, and then another query can be performed to complete the filtering, updating, and use of the data. For example, the second query could be initiated by the data processing unit or by the database unit itself. Of course, the embodiments of this disclosure are not limited to this, and other applicable methods can be used to combine data query operations and data update operations, depending on actual needs.

[0074] For example, in some examples, the data processing unit sends a data query request to the database unit, requesting images tagged as "cats." The database unit receives this request, performs a data query operation on its database, and returns the results (images tagged as "cats," possibly stored in a third-party storage system) to the data processing unit. Then, the data processing unit uses a color classification model to classify the results (images tagged as "cats") by color, obtaining the corresponding color labels (i.e., the color labels for each cat image). The data processing unit then sends a data update request to the database unit to write the color labels to the database. The database unit receives this request and performs a data update operation. Next, the data processing unit sends a data query request to the database unit to retrieve the "id" and "label" columns with the color label "white." The database unit receives this request, performs a data query operation on its database, and returns the retrieved "id" and "label" columns to the data processing unit. In this example, the data processing unit makes three calls to the database unit, which is a query-update-query combination.

[0075] For example, in other examples, the data processing unit sends a data query request to the database unit, requesting an image tagged as "cat." The database unit receives the request, performs a data query operation on its database, and returns the results to the data processing unit. Then, the data processing unit uses a color classification model to obtain a color label and sends a data update request to the database unit to write the color label to the database. The database unit receives the update request and performs the update operation. Next, the database unit checks all the labels. For example, if the color label "blue" for an image tagged as "cat" seems problematic, the database unit queries the URL of the ID with the color label "blue," reads the image from a third-party storage system (e.g., an unstructured data storage system) according to the URL, and checks if the classification is correct. In this example, the data processing unit makes two calls to the database unit, which is a combination of query-update and database-specific usage requirements.

[0076] It should be noted that multiple data query requests, data update requests, and requests to process data using UDFs can be contained within the same SQL command. For example, the data processing unit issues the following SQL command to the database unit: query column m in table test, process the data in this column using a user-defined function UDF1 (UDF1 is, for example, a job that can be called by the data processing unit and has been registered as a UDF in the database unit), and write the processing result to column n in test.

[0077] The above methods allow for the combination of data query and update operations, providing diverse combination options to adapt to complex data processing tasks.

[0078] For example, in some embodiments of this disclosure, the data query operation includes an approximate query operation, which includes a vector-based approximate query operation and / or a hybrid query operation based on labels and vectors.

[0079] Data query operations include approximate query operations for unstructured data representations, which can be vectors. This means retrieving vectors from the database that approximately match the query vector. For example, the vectors obtained from an approximate query could be vectors with a similarity greater than a threshold to the query vector and / or the top K vectors with the closest similarity.

[0080] Hybrid queries based on attributes and vectors refer to situations where a single data record contains both attributes (such as tags, which we will use as an example here) and vectors. In this case, the query performs an exact query on the tag corresponding to the data record and an approximate query on the vector corresponding to the data record. Through hybrid query operations, we can obtain data records where the tag is a specific tag value and the vector approximately matches the vector to be queried.

[0081] For example, many application scenarios require approximation / mixed query methods. For instance, when a graphic trademark is represented as a vector, an approximation query using vectors can be used to retrieve similar trademarks in the trademark field. As another example, when a graphic trademark's image is represented as a vector and the trademark category is represented as a label, a mixed query using vectors and labels can be used to retrieve trademarks similar to a specific trademark within a particular category.

[0082] For example, in some examples, the query statement for vector approximation is: Select label, distance('topK=10')(vector, [1.0, 1.0, 1.0]) from test. This query statement means to retrieve the 10 vectors that are closest to the vector [1.0, 1.0, 1.0] from the vector column of the test table in the database.

[0083] For example, in some examples, the query statement for a vector approximation query is: Select label, distance('topK=10')(vector, [1.0,1.0,1.0]) as d from test where d<10. This query statement means to query the vectors that are 10 distances from the vector [1.0,1.0,1.0] from the vector column vectors that are less than 10 distances from the database table test.

[0084] For example, in some cases, the database stores labels corresponding to unstructured data. Therefore, when performing a mixed query, the labels can be matched precisely, while the vectors can be matched approximately. For example, the query statement for a mixed vector query is: `Select label, distance('topK=10')(vector,[1.0,1.0,1.0]) from test where label=“cat”`. This query statement selects the data records in the table named "test" whose label is "cat", and further selects the data records corresponding to the top 10 vectors with the distance to the vector [0.1,0.1,0.1], returns their labels, and the distance to the vector [0.1,0.1,0.1].

[0085] Here, `distance` is a custom distance operator in SQL used to calculate the distance between vectors.

[0086] Approximate queries can be combined with other operations. For example, during the execution of an AI project, a sample image (with a sample label indicating it is a sample, corresponding to an exact query operation) is retrieved from the database. A feature extraction model is trained using the sample image. The trained feature extraction model is declared as a UDF in the database unit. The UDF is then called in the database unit to extract features from the base image stored in the database (or the image is stored in a third-party storage system with its storage address stored in the database). The base image features are written to the database (corresponding to a data update operation). Finally, an approximate query of the input image features is performed within the range of the base image features (corresponding to an approximate query operation).

[0087] Figure 3 A schematic flowchart illustrating another example of step S101 provided in at least one embodiment of this disclosure is shown.

[0088] like Figure 3 As shown, step S101 includes the following steps S301 to S302.

[0089] Step S301: The data processing unit uses the database retrieval module to call the database unit and passes the first function wake-up request to the database unit.

[0090] Step S302: The database unit responds to the first function call request by running the first user-defined function to process the data in the database.

[0091] For example, during the execution of a program process corresponding to a certain job, when it reaches a certain code segment (which indicates a need to call the database unit to invoke the first user-defined function), it will call the database retrieval module based on the instructions of that code segment. This will then invoke the first user-defined function within the database retrieval module, thus realizing the invocation of the first user-defined function. It can be understood that the first user-defined function corresponds to a callable job in the data processing unit that is in a callable state after execution. This callable job can be registered as a UDF in the database unit and can be invoked by the database. Simultaneously, this callable job can also be directly invoked by another job in the data processing unit. For example, if the callable job is A, and it is registered as a UDFa in the database unit, in the data processing unit, if job B wants to call job A, there are two ways: 1. By invoking UDFa in the database; 2. By directly invoking job A in job B through a request such as RPC (Remote Procedure Call). Of these two methods, compared to job B directly calling job A in the data processing unit, if job B calls job A by calling the user-defined function UDFa from the database unit, the parallel processing capability of the SQL engine can be utilized.

[0092] For example, the first user-defined function is configured to call at least one task process of the data processing unit to perform data processing operations corresponding to the second target job on the data in the database, and obtain the result of the data processing operation corresponding to the second target job returned by the data processing unit.

[0093] For example, the database unit can run a first user-defined function to pass data to the data processing unit. The data processing unit performs the corresponding data processing according to the job of the first user-defined function, uses the distributed computing capabilities of the data processing unit to perform data analysis and processing, and then writes the processed data back to the database unit.

[0094] For example, in some examples, the first user-defined function is a feature extraction function. The data processing unit calls the database unit, which then calls the first user-defined function to perform feature extraction. The data processing unit then returns the extracted features to the database unit. Of course, the embodiments of this disclosure are not limited to this; the first user-defined function can be any function with applicable functionality, depending on actual needs.

[0095] By registering the callable jobs running in the data processing unit as user-defined functions in the database, and then using these user-defined functions to enable the database unit to use the services provided by the data processing unit, the database unit's ability to process data in the database can be expanded. For example, it can use the pre-trained models in the database unit to perform AI-related processing such as inference.

[0096] It is understandable that the database managed by the database unit can be a relational database, while unstructured data is stored in a third-party unstructured data storage system (such as an object storage system). The database managed by the database unit stores the storage addresses of the unstructured data in the third-party storage system. When the database unit processes data using user-defined functions, the database unit pulls data from the third-party storage system (or it is directly pulled by the data processing unit) and passes it to the corresponding job running in the data processing unit that corresponds to the user-defined function. The corresponding process then performs the processing on the data according to the user-defined function.

[0097] Return to Figure 1 In some embodiments of this disclosure, the invocation request of the first target job includes a second function invocation request for the second user-defined function corresponding to the first target job, and step S102 may include: the database unit responding to the second function invocation request to run the second user-defined function.

[0098] For example, the second user-defined function is configured to call at least one task process of the data processing unit to perform data processing operations corresponding to the first target job on the data in the database, and obtain the result of the data processing operation corresponding to the first target job returned by the data processing unit.

[0099] Calls to user-defined functions can include calls initiated from the data processing unit, or SQL calls initiated directly from the database unit. Calls initiated from the data processing unit have already been... Figure 3 The above has been explained in detail and will not be repeated here. For SQL calls initiated directly by the database unit, the database unit receives the SQL command from the SQL client, executes the SQL command, and runs the user-defined function directly through the database unit.

[0100] For example, in some examples, the second user-defined function is a feature extraction function. The database unit directly calls the feature extraction function, thereby calling at least one task process of the data processing unit to perform feature extraction on the data in the database. The data processing unit returns the extracted features to the database unit, and the database unit stores the extracted features in the database or returns them to the data processing unit for further processing.

[0101] It should be noted that user-defined functions in the database unit are obtained by registering the callable jobs that run the data processing unit.

[0102] For example, the jobs executed in the data processing unit can include callable jobs and non-callable jobs. Jobs that can be registered as user-defined functions are callable jobs that are executed after running. Non-callable jobs are those that terminate after execution (e.g., a job execution request is submitted, the job starts running, and terminates after completion). Once a callable job is executed, a call port is generated. This call port is always in a listening state (consuming very few resources while listening). Each time it detects that it has been called, the corresponding process processes the data passed to the called job. The number of processes used can be flexibly allocated based on the number of call requests during job data processing.

[0103] For example, in some embodiments of this disclosure, the first target job is a job that can be called after it is in a running state, and the data processing method may further include: the data processing unit runs the first target job and generates a first call port, and the database unit registers the first call port as a second user-defined function in the database unit.

[0104] For example, after the first target job runs, a corresponding first call port is generated. This first call port is always in a listening state. When it detects a call, the first target job is executed. The first call port can be selected by the user and registered as a second user-defined function in the database unit. Calling the first target job can be done by the data processing unit through the first call port (e.g., by issuing an RPC request in Python code), or by the database unit calling the second user-defined function registered on the first call port, or by the data processing unit calling the database unit and the database unit then calling the second user-defined function registered on the first call port. For example, the first call port can be a domain name or port in the form of Hypertext Transfer Protocol (HTTP), or it can be other types of ports. The embodiments of this disclosure are not limited in this regard.

[0105] For example, in some embodiments of this disclosure, the data processing method may further include: in response to a call request for a second call port generated during job execution, the data processing unit calls the second call port to perform data processing operations corresponding to the third target job on the data specified in the call request.

[0106] For example, the second call port is generated after the data processing unit executes the third target job that can be called after it is in the running state.

[0107] In the preceding description, the data processing unit can call the database unit, which in turn calls the user-defined function. However, in this embodiment, the data processing unit can directly call the second call port to perform the data processing operation corresponding to the third target job.

[0108] For example, if the third target job is a feature extraction job, this feature extraction job can be run, generating a corresponding second call port. Then, when another job, different from the third target job, generates a feature extraction requirement, this second call port can be called to run the feature extraction job and perform the feature extraction operation, such as performing feature extraction on data specified in the call requirement (e.g., a specified image). Thus, the data processing unit can directly call the second call port to perform feature extraction operations. For example, another job in the data processing unit, different from the third target job, can call the second call port via an RPC (Remote Procedure Call) request (e.g., an HTTP request).

[0109] In this way, the data processing unit directly calls the call port corresponding to the callable job to perform the corresponding operation, so that the job can be reused and called without relying on the database, avoiding repeated development, and facilitating the repeated running of the operation corresponding to a certain job.

[0110] A typical AI task may include model training, model inference (e.g., if the model is a feature extraction model, then model inference is feature extraction), approximate query using the extracted features, and further model inference or other post-processing using the approximate query results. In the data system disclosed herein, a typical task can be a complete project or one or more jobs within a project, with model training, model inference, etc., being different jobs. The model training step is completed in the data processing unit; the model inference step can be initiated by the data processing unit (e.g., registering the trained model as a UDF and calling the UDF from the database unit; or directly calling the port corresponding to the model service job from the data processing unit), or by the database unit (e.g., registering the trained model as a UDF and calling the UDF from the database unit); the approximate query step can be initiated by the data processing unit (e.g., by including SQL instructions for approximate queries in the code corresponding to the job), or by the database unit (e.g., running SQL instructions in the database unit); post-processing of the query results can be initiated by the data processing unit, or similarly by the database unit (by calling the corresponding UDF for post-processing). The project is described in detail below.

[0111] Figure 4 A schematic flowchart illustrating a data processing method according to at least one embodiment of the present disclosure is shown. This data processing method is used in a data system comprising a data processing unit and a database unit. The data processing unit is configured to execute one or more jobs in a process, and the database unit is configured to perform database operations. The one or more jobs include a first job, which is a model training job.

[0112] like Figure 4 As shown, the data processing method includes the following steps S401 to S403.

[0113] Step S401: In response to the first data query request generated during the execution of the first job, the data processing unit calls the database unit and passes the first data query request to the database unit.

[0114] Step S402: In response to receiving the first data query request, the database unit performs the data query operation corresponding to the first data query request on the database and returns the first sample data obtained from the query to the data processing unit.

[0115] Step S403: The data processing unit trains the model using the first sample data.

[0116] It is understandable that in step S402, the first data query request can be a precise query for data containing sample labels.

[0117] This data processing method enables interaction between database units and data processing units in a data system, enhances the integration between them, and facilitates the use of database functions by the data processing unit when running a certain job (such as a model training job) multiple times. This helps improve the processing efficiency of data processing (such as model training) and enhances the ease of using database functions.

[0118] For example, in some embodiments of this disclosure, after N rounds of training (e.g., coarse model tuning), in order to improve training efficiency, difficult samples can be selected for subsequent training (e.g., fine model tuning).

[0119] At this point, through steps S401-S403, the data processing unit uses the first sample data to train the model and obtain the model after the Nth round of training.

[0120] Figure 4 The data processing methods shown also include:

[0121] Step S404: The data processing unit uses the model trained in the Nth round to perform model inference on the first sample data to obtain the model inference result of the first sample data, which includes a vector.

[0122] Step S405: In response to the first data update request generated during the execution of the first job, the database unit updates the model inference result of the first sample data to the database, wherein the model inference result of the first sample data includes the model inference result of the target sample data;

[0123] For example, a column can be set up in a database table to store the model inference results obtained by performing model inference using an untrained model. For instance, this column stores the inference results of the model after the Nth training round. When the inference results of the model after the N+1Lth training round are generated, they can be written into this column, overwriting the inference results of the model after the Nth training round.

[0124] For example, the target sample data can be any sample data, or the target sample data can be any sample data in which a positive sample exists.

[0125] Understandably, multiple sample data that meet the conditions can be used as target sample data in sequence to filter out multiple difficult samples.

[0126] Step S406: In response to the second data query request generated during the execution of the first job, the data processing unit calls the database unit and passes the second data query request to the database unit;

[0127] Step S407: In response to receiving the second data query request, the database unit performs the data query operation corresponding to the second data query request on the database and returns the second sample data obtained from the first sample data to the data processing unit.

[0128] The second data query request is a hybrid query request, and the query results of the hybrid query request satisfy both distance conditions and attribute conditions. The distance conditions in the second data query request include that the distance between the vector corresponding to the query result and the vector corresponding to the target sample data is greater than the first preset condition, and the attribute conditions in the second data query request include that the query result is a positive sample of the target sample data; or, the distance conditions in the second data query request include that the distance between the vector corresponding to the query result and the vector corresponding to the target sample data is less than the second preset condition, and the attribute conditions in the second data query request include that the query result is a negative sample of the target sample data.

[0129] For example, given a model trained for N rounds, we want to further train it using difficult samples. For instance, two samples are mutually positive, and their corresponding inference results should theoretically be the same. However, if the distance between their inference results (e.g., vectors) is greater than a first preset condition (e.g., threshold a), then they are considered difficult samples. Similarly, two samples are mutually negative, and their corresponding inference results should theoretically be different. However, if the distance between their corresponding inference results (e.g., vectors) is less than a second preset condition (e.g., threshold b), then they are considered difficult samples. Through mixed queries, difficult samples can be filtered out from the sample data.

[0130] Step S408: The data processing unit uses the second sample data to train the model after the Nth round of training.

[0131] It is understandable that in step S408, the difficult samples selected in step S407 can be used for one or several rounds of training to obtain a model after N+M rounds of training. Then, the model can be used to perform model inference on the first sample data, and the model inference results can be written into the same column in step S405. Then, difficult samples for the model after N+M rounds of training can be selected through steps S406 and S407, and the model after N+M rounds of training can be further trained.

[0132] For example, in some embodiments of this disclosure, after N rounds of training with labeled data (e.g., coarse model tuning), subsequent training is performed with unlabeled data (e.g., fine model tuning), thereby achieving weakly supervised learning.

[0133] At this point, through steps S401-S403, the data processing unit uses the first sample data to train the model and obtain the model after the Nth round of training.

[0134] Figure 4 The data processing methods shown also include:

[0135] Step S409: The data processing unit uses the model trained in the Nth round to perform model inference on the unlabeled data and obtains the model inference result of the unlabeled data. The model inference result of the unlabeled data includes vectors. The unlabeled data is obtained from the database.

[0136] For example, by using a mixed query, data with an empty or null "label" column attribute can be retrieved as unlabeled data.

[0137] For example, a column can be set up in a database table to store the model inference results obtained by performing model inference using an untrained model.

[0138] Step S410: In response to the second data update request generated during the execution of the first job, the database unit updates the model inference results of the unlabeled data to the database, wherein the model inference results of the unlabeled data include the model inference results of the target unlabeled data;

[0139] The target unlabeled data can be any unlabeled data. Understandably, multiple unlabeled data can be used sequentially as the target sample data.

[0140] Step S411: In response to a third data query request generated during the execution of the first job, the data processing unit calls the database unit and passes the third data query request to the database unit;

[0141] Step S412: In response to receiving a third data query request, the database unit performs a data query operation corresponding to the third data query request on the database and returns the first unlabeled data obtained from the unlabeled data to the data processing unit.

[0142] The third data query request is a hybrid query request. The query results of the hybrid query request satisfy both distance and attribute conditions. The distance condition in the third data query request includes that the distance between the vector corresponding to the query result and the vector corresponding to the target unlabeled data is less than a third preset condition. The attribute condition in the third data query request includes that the query result is unlabeled data. Alternatively, the distance condition in the third data query request includes that the distance between the vector corresponding to the query result and the vector corresponding to the target unlabeled data is greater than a fourth preset condition. The attribute condition in the third data query request includes that the query result is unlabeled data.

[0143] Taking the model as a feature extraction model as an example, when the model after the Nth round of training determines that the feature vector distance between the unlabeled image A and the target unlabeled image is extremely small (e.g., less than the threshold c), they are likely to be positive samples of each other (e.g., they contain the same object). When the model after the Nth round of training determines that the feature vector distance between the unlabeled image A and the target unlabeled image is extremely large (e.g., greater than the threshold d), they are likely to be negative samples of each other (e.g., they do not contain the same object).

[0144] Understandably, besides the distance between vectors, other auxiliary conditions can help determine whether unlabeled sample data are positive or negative samples. For example, if two images were taken at similar times and locations, they are more likely to contain the same object; if two images are consecutive frames from the same video, they are more likely to contain the same object. These auxiliary conditions can be used as attribute conditions in mixed queries.

[0145] Step S413: In response to the distance between the vector corresponding to the first unlabeled data and the vector corresponding to the target unlabeled data being less than a third preset condition, the first unlabeled data and the target unlabeled data are marked as positive samples; and / or, in response to the distance between the vector corresponding to the first unlabeled data and the vector corresponding to the target unlabeled data being greater than a fourth preset condition, the first unlabeled data and the target unlabeled data are marked as negative samples, and the unlabeled data marked as positive samples and / or negative samples are used as new sample data;

[0146] Labeling the first unlabeled data and the target unlabeled data as negative samples can be achieved by updating the "Label" column corresponding to the first unlabeled data and the target unlabeled data to positive samples, so that they can be filtered out as sample data later by attribute conditions.

[0147] Step S414: The data processing unit uses the newly added sample data to train the model after the Nth round of training.

[0148] For example, the data processing unit uses existing sample data and newly added sample data for further training.

[0149] Understandably, in step S414, the newly added sample data selected in step S413 can be used for one or several rounds of training to obtain a model after N+M rounds of training. Then, the model can be used to perform model inference on the first sample data, and the model inference result can be written into the same column in step S409. More new samples can be selected through steps S412 and S413 using the model after N+M rounds of training, and the new samples can be used to further train the model after N+M rounds of training.

[0150] In this way, only a small amount of labeled data is needed to start training, and data that can be used for subsequent training can be selected from the unlabeled data after the Nth round of training, which greatly reduces the labeling cost of training samples.

[0151] For example, in some embodiments of this disclosure, one or more jobs may further include a second job, which is a model service job. Figure 4 The data processing method shown may further include: declaring the trained model as a second job that can be called after it is in a running state, running the second job and generating the calling port corresponding to the second job.

[0152] For example, a model service job can be a pre-trained model. When invoked, the model service job can use the pre-trained model to perform inference on the data corresponding to the call request and return the inference result. As mentioned above, a callable model service job will generate a corresponding call port after it runs for invocation.

[0153] This data processing method allows users to share trained or debugged jobs or projects within the data system, such as trained model service jobs, so that other jobs can call the model for inference, thereby achieving model sharing and reuse.

[0154] For example, one or more assignments may include a third assignment. Figure 4 The data processing method shown may further include: in response to a model inference request for first specified data in the database generated during the execution of the third job, the data processing unit calls the database unit and passes a third function call request to the database unit; the database unit responds to the third function call request by running a third user-defined function to perform model inference operation on the first specified data, obtains the first model inference result returned by the data processing unit, and stores the first model inference result in the database or returns it to the data processing unit.

[0155] In other words, in this embodiment, in the third job which is different from the second job, the service corresponding to the second job can be called through the database unit calling UDF.

[0156] For example, the third user-defined function is configured to call at least one task process of the data processing unit to perform data processing operations corresponding to the second job on the data in the database, and obtain the result of the data processing operation corresponding to the second job returned by the data processing unit; the calling port corresponding to the second job is registered as the third user-defined function in the database unit.

[0157] For example, when the program process corresponding to the third task runs, when it reaches a certain code segment (which indicates that the database unit needs to be called, and then the database unit calls the third user-defined function to perform model inference), it will call the database retrieval module based on the instructions of the code segment. In the database retrieval module, the third user-defined function is called. The data processing unit performs model inference on the data and returns the model inference results to the database unit. The database unit stores these results in the database table.

[0158] During the execution of corresponding processes by the data processing unit, the data processing unit can call the database unit, which in turn calls a User-Defined Function (UDF). The database unit then calls one or more processes within the data processing unit based on the UDF, thereby completing the required operation. For example, if the data processing unit parses an SQL statement, it will call the database unit based on the SQL statement. The database unit will then call the data processing unit through the UDF to execute the corresponding processing. This process allows the data processing unit and the database unit to call each other multiple times, achieving deep interaction between them. This fully utilizes the data processing capabilities of the callable jobs within the data processing unit, its flexible computing capabilities, and the parallel data processing capabilities of the database unit, thereby improving the efficiency and convenience of data processing.

[0159] For example, Figure 4 The data processing method shown may further include: in response to a fourth data query request for second specified data in the database generated during the execution of the third job, the data processing unit calls the database unit and passes the fourth data query request to the database unit; in response to receiving the fourth data query request, the database unit performs a data query operation corresponding to the fourth data query request on the database based on the data query request, and returns the first data obtained by the query to the data processing unit; in response to a call request for a call port corresponding to the second job generated during the execution of the third job, the data processing unit calls the call port corresponding to the second job, performs a model inference operation corresponding to the second job on the second specified data, and obtains a second model inference result; in response to a third data update request for the second model inference result generated during the execution of the third job, the data processing unit calls the database unit and passes the third data update request to the database unit; in response to receiving the third data update request, the database unit updates the second model inference result to the database.

[0160] In other words, in this embodiment, the service corresponding to the second job can be called directly in the third job, which is different from the second job.

[0161] In the first embodiment, the data processing unit invokes the second job by calling a UDF through the database unit. Specifically, this can be achieved through a single SQL command, including data querying, UDF invocation (using the queried data as input), and data update (updating the data table with the UDF's return value). In the second embodiment, the data processing unit can also directly invoke the second job within itself. This can be done by querying data using SQL statements, then invoking the second job via an RPC request, and finally updating the data using SQL statements. Both implementations utilize the data processing unit's flexible computing capabilities when invoking the second job. Furthermore, compared to the latter, the former better leverages the parallel processing capabilities of the SQL engine, resulting in higher data processing efficiency.

[0162] By using the methods described above, data query operations, model inference operations, and data update operations can be combined to adapt to complex data processing tasks.

[0163] For example, Figure 4 The data processing method shown may further include: the database unit running a third user-defined function to perform data processing operations on the second specified data in the database, and obtaining the third model inference result returned by the data processing unit; the database unit storing the third model inference result into the database.

[0164] In other words, in this embodiment, the service corresponding to the second job can be invoked in the database unit by calling a UDF.

[0165] For example, the third user-defined function is configured to call at least one task process of the data processing unit to perform model inference operations on the data in the database corresponding to the second job, and obtain the results of the data processing operations corresponding to the second job returned by the data processing unit.

[0166] For example, the calling port corresponding to the second task is registered as a third user-defined function in the database unit.

[0167] For example, as mentioned above, the call to the third user-defined function includes a direct SQL call, which generates the execution requirement of the model service job in the database unit. The database unit can directly call the third user-defined function to perform model inference operations. The data processing unit returns the model inference results to the database unit, and the database unit stores the results in the database table.

[0168] For example, in some examples, the UDF is called in the database unit to extract features from the base image stored in the database (or the base image is stored in a third-party storage system and its address in the third-party storage system is stored in the database). The data processing unit returns the extracted features to the database unit, and the database unit stores the extracted features in a database table.

[0169] In this way, user-defined functions can be used in the database unit to enable the database unit to use the services provided by the data processing unit, thereby expanding the processing capabilities of the data in the database.

[0170] For example, in some embodiments of this disclosure, the model is a feature extraction model, the result of model inference is the extracted features, and one or more tasks further include a fourth task. Figure 4 The data processing method shown may further include: in response to an approximate query request for features extracted from the database generated during the execution of the fourth task, the data processing unit calls the database unit and passes the approximate query request to the database unit; in response to receiving the approximate query request, the database unit performs an approximate query operation on the extracted features based on the approximate query request, and returns the first data corresponding to the features that satisfy the distance conditions contained in the approximate query request to the data processing unit, and the data processing unit uses the first data as the execution result of the fourth task or performs subsequent processing on the first data to obtain the execution result of the fourth task.

[0171] After feature extraction, the results can be further utilized, such as for approximate queries.

[0172] The results of feature extraction can be further utilized in a fourth task. The fourth task can be the same as the third task, that is, in the same task, feature extraction is performed first, and then the features obtained from feature extraction are used for approximate query; or, the fourth task can be different from the third task, with feature extraction performed in one task and approximate query performed in another task.

[0173] For example, by invoking a feature extraction job, the database stores the extracted features. The data processing unit calls the database unit to perform an approximate query operation on the extracted features, returning the first data that meets the conditions to the data processing unit. For instance, in some examples, after feature extraction is performed on an image, an approximate query request for the extracted features arises during the execution of the fourth job. The data processing unit returns images with a similarity of 90% or more to the target image, or the attributes (such as tags, addresses, etc.) corresponding to images with a similarity of 90% or more to the data processing unit. The data processing unit can use the first data as the execution result of the fourth job or perform further processing on the first data to obtain the execution result of the fourth job.

[0174] It should be noted that the distance conditions referred to in this disclosure may be, for example, the distance being greater than or less than a certain threshold, or the distance being among the top K largest or smallest distances, or other distance conditions. This disclosure does not limit these conditions.

[0175] For example, one or more assignments may also include a fifth assignment. Figure 4 The data processing method shown may further include: in response to a call request for the call port corresponding to the fifth job generated during the execution of the fourth job, the data processing unit calls the call port corresponding to the fifth job, performs the data processing operation corresponding to the fifth job on the first data, obtains the data processing result corresponding to the fifth job, and determines the execution result of the fourth job by the data processing result corresponding to the fifth job.

[0176] For example, the fifth job is a job that is in a callable state after it has been run.

[0177] The approximate query result of the fourth task is either a feature vector whose distance to the query vector is within a certain distance condition, or the first data (e.g., image data) corresponding to a feature vector whose distance is within a certain distance condition. Further processing can then be performed on the approximate query result. For example, the approximate query result can be used as input to another callable task (e.g., the fifth task) to obtain the return result of that task. For instance, the fifth task is a model service task, and its corresponding model is a classification model. Compared to using only image a as input to the classification model, using image a along with its approximate images, image b and image c, as input might yield a more accurate classification result for image a. In this case, the approximate query result can be used as input to the fifth task to obtain its data processing result, which is then used as the execution result of the fourth task.

[0178] In this way, the approximate query results obtained from the database can be conveniently processed in the data processing unit, for example, the approximate query results can be used as input for callable jobs.

[0179] For example, in some embodiments of this disclosure, Figure 4 The data processing method shown may further include: the database unit performs an approximate query operation on the features extracted from the database to obtain the second data corresponding to the features that satisfy the distance conditions contained in the approximate query operation.

[0180] For example, the model is a feature extraction model, and the result of the model inference is the extracted features. These results are stored in a database. Database units can directly perform approximate query operations or label vector mixed query operations on the extracted features in the database to obtain the second data corresponding to the features whose distance is within the distance condition.

[0181] This allows database units to be more closely integrated with data processing tasks (such as artificial intelligence tasks), writing data generated by the data processing unit, such as vectors, into the database unit, and then performing approximate queries of vectors, mixed queries of label vectors, etc. in the database unit, which greatly expands the data processing capabilities of the database, allowing users to perform approximate queries of unstructured data representations through simple SQL statements.

[0182] For example, Figure 4 The data processing method shown may also include: the database unit calls the fourth user-defined function to perform data processing operations on the second data, and obtains the data processing result corresponding to the fifth job returned by the data processing unit.

[0183] For example, the fourth user-defined function is configured to call at least one task process of the data processing unit to perform the data processing operation corresponding to the fifth job on the data in the database, and obtain the result of the data processing operation corresponding to the fifth job returned by the data processing unit.

[0184] For example, the fifth task is in a callable state after it is run, and the calling port corresponding to the fifth task has been registered as a user-defined function in the database unit.

[0185] As mentioned earlier, further processing of approximate query results can also be performed within the database unit. This can be achieved by registering the callable job as a UDF in the database unit beforehand, and then calling the UDF within the database unit. In this way, the database unit can use SQL to call the callable job that has been registered as a UDF in the database by the data processing unit, thereby performing further complex data processing on the approximate query results within the database. This expands the data processing capabilities of the database and provides diversity and flexibility in data processing.

[0186] The following example illustrates how the above data processing method works.

[0187] This example utilizes a data processing method provided in at least one embodiment of this disclosure, which includes the following four steps.

[0188] Step 1: Preprocess the training data in the database.

[0189] The database stores structured and unstructured data (or the addresses corresponding to unstructured data), from which labeled data can be selected as training data. Preprocessing such as resizing and rotation of the training data can be performed in the database.

[0190] For example, a user-defined function for the preprocessing step `resize` has been registered in the database. Data that meets the specified conditions can be selected from the database as training data. The database unit calls this user-defined function to preprocess the data (the data processing unit's flexible computing capabilities are used during the preprocessing process).

[0191] Step 2: Train the model.

[0192] First, the data processing unit reads the data required for model training from the database (the preprocessed training data from step one). In this step, the code corresponding to the model training job is executed. When the code encounters a statement that requires calling the database retrieval module, the data processing unit calls the database unit through the database retrieval module and passes the SQL statement to the database unit. The database unit executes the statement "SELECT label,image from landmarks" to retrieve the preprocessed training data from step one and returns the query result to the data processing unit.

[0193] Step 3: Perform model inference in the database (extract features using the trained model).

[0194] First, the trained feature extraction model is declared as a callable model service job, which is then run to generate a corresponding call port. Next, the call port corresponding to this model service job can be registered as a feature extraction function (user-defined function) in the database, which can then be called subsequently. Then, the feature extraction function is called from the database. Data is pushed into the data processing unit, where its elastic computing capabilities are used for parallel feature extraction, and the extracted features are stored in a database table. Compared to directly calling the call port corresponding to the model service job in the data processing unit, this method fully utilizes the parallel processing capabilities of the database engine.

[0195] Step 4: Perform approximate query operations in the database based on the extracted features.

[0196] For example, a vector approximation query operation can be performed directly in the database, or the data processing unit can send a vector approximation query instruction to the database unit to perform an approximation query operation.

[0197] Figure 5 A schematic block diagram of a data system 500 provided in at least one embodiment of the present disclosure is shown, which can be used to perform... Figure 1 The data processing method shown.

[0198] like Figure 5As shown, the data system 500 includes a data processing unit 501 and a database unit 502.

[0199] The data processing unit 501 is configured to call the database unit to perform corresponding database operations in response to data usage requests for the database generated during job execution.

[0200] For example, data processing unit 501 executes... Figure 1 Step S101.

[0201] Database unit 502 is configured to respond to a request for a first target job generated during database operations, call the data processing unit to perform data processing operations on the data in the database corresponding to the first target job, and the data processing unit returns the result of the data processing operations corresponding to the first target job to the database unit.

[0202] For example, database unit 502 executes Figure 1 Step S102.

[0203] For example, the data system 500 can be implemented using hardware, software, firmware, or any feasible combination thereof, and this disclosure does not impose any limitations on this. For example, the data processing unit 501 and the database unit 502 can be dedicated or general-purpose circuits, chips, or devices, or they can be a combination of a processor and memory. Regarding the specific implementation of the data processing unit 501 and the database unit 502, the embodiments of this disclosure do not impose any limitations on this.

[0204] It should be noted that in the embodiments of this disclosure, each unit of the data system 500 corresponds to each step of the aforementioned data processing method. For the specific functions of the data system 500, please refer to the relevant description of the data processing method above, which will not be repeated here. Figure 5 The components and structure of the data system 500 shown are merely exemplary and not limiting. The data system 500 may also include other components and structures as needed. The technical effects of the above-described data system are... Figure 1 The data processing methods shown have the same technical effect, and will not be described in detail here.

[0205] At least one embodiment of this disclosure also provides an electronic device that includes the data system provided in at least one embodiment of this disclosure.

[0206] Figure 6A This is a schematic block diagram of an electronic device 600 provided for at least one embodiment of the present disclosure. Figure 6A As shown, the electronic device 600 includes a data system, which can be... Figure 5The data system 500 is shown. The electronic device 600 is, for example, suitable for implementing the data processing methods provided in the embodiments of this disclosure. The electronic device 600 may be a terminal device or a server, etc., and this disclosure does not limit it.

[0207] At least one embodiment of this disclosure also provides an electronic device including a processor and a memory. The memory stores computer-executable instructions, which, when executed by the processor, can implement the data processing method provided in at least one embodiment of this disclosure.

[0208] Figure 6B This is a schematic block diagram of another electronic device 601 provided in some embodiments of this disclosure. For example... Figure 6B As shown, the electronic device 601 includes a processor 610 and a memory 620. The memory 620 is used to store computer-executable instructions (e.g., one or more computer program modules). The processor 610 is used to run the computer-executable instructions, which, when run by the processor 610, can perform one or more steps in the data processing method described above. The memory 620 and the processor 610 can be interconnected via a bus system and / or other forms of connection mechanism (not shown).

[0209] For example, processor 610 may be a central processing unit (CPU), a graphics processing unit (GPU), or other form of processing unit with data processing and / or program execution capabilities. For example, the central processing unit (CPU) may be an x86 or ARM architecture. Processor 610 may be a general-purpose processor or a special-purpose processor, capable of controlling other components in electronic device 601 to perform desired functions.

[0210] For example, memory 620 may include any combination of one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. Volatile memory may include, for example, random access memory (RAM) and / or cache memory. Non-volatile memory may include, for example, read-only memory (ROM), hard disk, erasable programmable read-only memory (EPROM), portable compact disc read-only memory (CD-ROM), USB memory, flash memory, etc. One or more computer program modules may be stored on the computer-readable storage medium, and processor 610 may run one or more computer program modules to implement various functions of electronic device 601. Various application programs and various data, as well as various data used and / or generated by the application programs, may also be stored in the computer-readable storage medium.

[0211] It should be noted that, in the embodiments of this disclosure, the specific functions and technical effects of the electronic device 601 can be referred to the description of the data processing method above, and will not be repeated here.

[0212] Figure 7 This is a schematic block diagram of another electronic device provided in some embodiments of the present disclosure. The electronic device 700 is, for example, suitable for implementing the data processing method provided in the embodiments of the present disclosure. The electronic device 700 may be a terminal device, a server, or a computer system, etc. It should be noted that... Figure 7 The illustrated electronic device 700 is merely an example and does not impose any limitation on the functionality and scope of use of the embodiments of this disclosure.

[0213] like Figure 7 As shown, the electronic device 700 may include a processing unit (e.g., a central processing unit, a graphics processor, etc.) 710, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 720 or a program loaded from a storage device 780 into a random access memory (RAM) 730. The RAM 730 also stores various programs and data required for the operation of the electronic device 700. The processing unit 710, the ROM 720, and the RAM 730 are interconnected via a bus 740. An input / output (I / O) interface 750 is also connected to the bus 740.

[0214] Typically, the following devices can be connected to the I / O interface 750: input devices 760 including, for example, a touchscreen, touchpad, keyboard, mouse, camera, microphone, accelerometer, gyroscope, etc.; output devices 770 including, for example, a liquid crystal display (LCD), speaker, vibrator, etc.; storage devices 780 including, for example, magnetic tape, hard disk, etc.; and communication devices 790. The communication device 790 allows the electronic device 700 to communicate wirelessly or wiredly with other electronic devices to exchange data. Although Figure 7 An electronic device 700 with various devices is shown, but it should be understood that it is not required to implement or have all of the devices shown, and the electronic device 700 may alternatively implement or have more or fewer devices.

[0215] For example, according to embodiments of this disclosure, the above-described data processing method can be implemented as a computer software program. For instance, embodiments of this disclosure include a computer program product comprising a computer program carried on a non-transitory computer-readable medium, the computer program including program code for performing the above-described data processing method. In such embodiments, the computer program can be downloaded and installed from a network via a communication device 790, or installed from a storage device 780, or installed from a ROM 720. When the computer program is executed by the processing device 710, the functions defined in the data processing method provided by embodiments of this disclosure can be implemented.

[0216] At least one embodiment of this disclosure provides a computer-readable storage medium that non-temporarily stores computer-executable instructions, which, when executed by a processor, implement the data processing method provided in at least one embodiment of this disclosure.

[0217] Figure 8 This is a schematic diagram of a storage medium provided for some embodiments of this disclosure. For example... Figure 8 As shown, the storage medium 800 non-transitory stores computer-executable instructions 810. For example, when the computer-executable instructions 810 are executed by a computer, one or more steps in the data processing method described above can be performed. Detailed descriptions and technical effects of the storage medium 800 can be found in the description of the data processing method above, and will not be repeated here.

[0218] The following points need to be explained:

[0219] (1) The accompanying drawings of the embodiments of this disclosure only involve the structures involved in the embodiments of this disclosure. Other structures can be referred to the general design.

[0220] (2) Where there is no conflict, the embodiments of this disclosure and the features in the embodiments can be combined with each other to obtain new embodiments.

[0221] The above description is merely a specific embodiment of this disclosure, but the scope of protection of this disclosure is not limited thereto. The scope of protection of this disclosure should be determined by the scope of protection of the claims.

Claims

1. A data processing method for a data system comprising a data processing unit and a database unit, wherein, The data processing unit is configured to execute one or more jobs in the project, the database unit is configured to execute database operations, the one or more jobs include a first job, the first job being a model training job, and the data processing method includes: In response to a first data query request generated during the execution of the first job, the data processing unit calls the database unit and transmits the first data query request to the database unit; In response to receiving the first data query request, the database unit performs a data query operation corresponding to the first data query request on the database and returns the first sample data obtained by the query to the data processing unit. The data processing unit uses the first sample data to train the model to obtain the model after the Nth round of training; The data processing unit uses the model trained in the Nth round to perform model inference on the first sample data to obtain the model inference result of the first sample data, which includes a vector. In response to a first data update request generated during the execution of the first job, the database unit updates the model inference result of the first sample data to the database, wherein the model inference result of the first sample data includes the model inference result of the target sample data; In response to a second data query request generated during the execution of the first job, the data processing unit calls the database unit and transmits the second data query request to the database unit. In response to receiving the second data query request, the database unit performs a data query operation corresponding to the second data query request on the database, and returns the second sample data obtained from the first sample data to the data processing unit; the second data query request is a hybrid query request, and the query result of the hybrid query request satisfies distance conditions and attribute conditions; the distance condition in the second data query request includes that the distance between the vector corresponding to the query result and the vector corresponding to the target sample data is greater than a first preset condition, and the attribute condition in the second data query request includes that the query result is a positive sample of the target sample data; or, the distance condition in the second data query request includes that the distance between the vector corresponding to the query result and the vector corresponding to the target sample data is less than a second preset condition, and the attribute condition in the second data query request includes that the query result is a negative sample of the target sample data; The data processing unit uses the second sample data to train the model after the Nth round of training.

2. The data processing method according to claim 1 further includes: The data processing unit uses the model trained in the Nth round to perform model inference on the unlabeled data to obtain the model inference result of the unlabeled data, which includes a vector. The unlabeled data is obtained from the database. In response to a second data update request generated during the execution of the first job, the database unit updates the model inference results of the unlabeled data to the database, wherein the model inference results of the unlabeled data include the model inference results of the target unlabeled data; In response to a third data query request generated during the execution of the first job, the data processing unit calls the database unit and transmits the third data query request to the database unit. In response to receiving the third data query request, the database unit performs a data query operation corresponding to the third data query request on the database, and returns the first unlabeled data obtained from the unlabeled data to the data processing unit; the third data query request is a hybrid query request, and the query result of the hybrid query request satisfies distance conditions and attribute conditions; the distance condition in the third data query request includes that the distance between the vector corresponding to the query result and the vector corresponding to the target unlabeled data is less than a third preset condition, and the attribute condition in the third data query request includes that the query result is unlabeled data; or, the distance condition in the third data query request includes that the distance between the vector corresponding to the query result and the vector corresponding to the target unlabeled data is greater than a fourth preset condition, and the attribute condition in the third data query request includes that the query result is unlabeled data; In response to the distance between the vector corresponding to the first unlabeled data and the vector corresponding to the target unlabeled data being less than the third preset condition, the first unlabeled data and the target unlabeled data are marked as positive samples; and / or, in response to the distance between the vector corresponding to the first unlabeled data and the vector corresponding to the target unlabeled data being greater than the fourth preset condition, the first unlabeled data and the target unlabeled data are marked as negative samples, and the unlabeled data marked as positive samples and / or negative samples are used as new sample data; The data processing unit uses the newly added sample data to train the model after the Nth round of training.

3. The data processing method according to claim 2, wherein, The attribute conditions in the third data query request also include auxiliary conditions.

4. The data processing method according to any one of claims 1-3, wherein, The one or more jobs further include a second job, the second job being a model service job, and the data processing method further includes: The trained model is declared as the second job that can be called after it is in a running state. The second job is run and the corresponding calling port is generated.

5. The data processing method according to claim 4, wherein, The one or more tasks further include a third task, and the data processing method further includes: In response to a model inference request for first specified data in the database generated during the execution of the third job, the data processing unit calls the database unit and passes a third function invocation request to the database unit. The database unit, in response to the third function invocation request, runs a third user-defined function to perform model inference on the first specified data, obtaining a first model inference result returned by the data processing unit. The database unit stores the first model inference result in the database or returns it to the data processing unit. The third user-defined function is configured to call at least one task process of the data processing unit to perform data processing operations corresponding to the second job on the data in the database, obtaining the result of the data processing operation corresponding to the second job returned by the data processing unit. The calling port corresponding to the second job is registered as the third user-defined function in the database unit.

6. The data processing method according to claim 4, wherein, The one or more tasks further include a third task, and the data processing method further includes: In response to a fourth data query request for second specified data in the database generated during the execution of the third job, the data processing unit calls the database unit and transmits the fourth data query request to the database unit. In response to receiving the fourth data query request, the database unit performs a data query operation corresponding to the fourth data query request on the database based on the data query request, and returns the first data obtained by the query to the data processing unit; In response to a call request generated during the execution of the third job for the call port corresponding to the second job, the data processing unit calls the call port corresponding to the second job to perform model inference operation on the second specified data corresponding to the second job, and obtains the second model inference result; In response to a third data update request generated during the execution of the third task for the reasoning result of the second model, the data processing unit calls the database unit and transmits the third data update request to the database unit. In response to receiving the third data update request, the database unit updates the second model inference result to the database.

7. The data processing method according to claim 4 further includes: The database unit runs a third user-defined function to perform data processing operations on the second specified data in the database, and obtains the third model inference result returned by the data processing unit; The database unit stores the reasoning results of the third model into the database; The third user-defined function is configured to call at least one task process of the data processing unit to perform model inference operation on the data in the database corresponding to the second job, and obtain the result of the data processing operation corresponding to the second job returned by the data processing unit. The calling port corresponding to the second job is registered as the third user-defined function in the database unit.

8. The data processing method according to any one of claims 5-7, wherein, The model is a feature extraction model, and the result of the model's inference is the extracted feature. The one or more tasks also include a fourth task, and the data processing method further includes: In response to an approximate query request for the extracted features in the database generated during the execution of the fourth task, the data processing unit calls the database unit and passes the approximate query request to the database unit. In response to receiving the approximate query request, the database unit performs an approximate query operation on the extracted features based on the approximate query request, and returns the first data corresponding to the features that satisfy the distance conditions contained in the approximate query request to the data processing unit. The data processing unit uses the first data as the execution result of the fourth job or performs subsequent processing on the first data to obtain the execution result of the fourth job.

9. The data processing method according to claim 8, wherein, The one or more jobs also include a fifth job, and the subsequent processing of the first data further includes: In response to a call request generated during the execution of the fourth job for the call port corresponding to the fifth job, the data processing unit calls the call port corresponding to the fifth job to perform the data processing operation corresponding to the fifth job on the first data, obtains the data processing result corresponding to the fifth job, and determines the execution result of the fourth job by the data processing result corresponding to the fifth job; wherein, the fifth job is a job that is in a callable state after running.

10. The data processing method according to any one of claims 5-7, wherein, The model is a feature extraction model, and the result of the model's inference is the extracted feature. The data processing method further includes: The database unit performs an approximate query operation on the extracted features in the database to obtain second data corresponding to the features that satisfy the distance conditions included in the approximate query operation.

11. The data processing method according to claim 10, wherein, The one or more tasks further include a fifth task, and the data processing method further includes: The database unit calls the fourth user-defined function to perform data processing operations on the second data, and obtains the data processing result corresponding to the fifth job returned by the data processing unit; The fourth user-defined function is configured to call at least one task process of the data processing unit to perform the data processing operation corresponding to the fifth job on the data in the database, and obtain the result of the data processing operation corresponding to the fifth job returned by the data processing unit. The fifth job is a job that is in a callable state after it has been run, and the calling port corresponding to the fifth job has been registered as a user-defined function in the database unit.

12. An electronic device, comprising: processor; Memory, including one or more computer program modules; The one or more computer program modules are stored in the memory and configured to be run by the processor, and the one or more computer program modules are used to implement the data processing method according to any one of claims 1-11.

13. A storage medium storing non-transitory computer-readable instructions that, when executed by a computer, implement the data processing method according to any one of claims 1-11.

Images