Data processing method, system, device, and storage medium

Abstract

One or more embodiments of the present specification provide a data processing method, system, device and storage medium, the method is applied to a computing platform; a user-defined function corresponding to a machine learning task is deployed in the computing platform; wherein the user-defined function contains initialization logic and data processing logic corresponding to the machine learning task; the method comprises: obtaining to-be-processed data corresponding to the machine learning task; calling the initialization logic in the user-defined function to obtain a machine learning model corresponding to the machine learning task stored in a storage platform connected to the computing platform; further calling the data processing logic in the user-defined function to use the machine learning model to process the to-be-processed data, and storing the data processing result to a database connected to the computing platform.

Description

Technical Field

[0001] This specification relates to one or more embodiments in the field of computer application technology, and more particularly to a data processing method, system, device and storage medium. Background Technology

[0002] Today, machine learning technology is increasingly widely used in various fields. Generally, machine learning can be divided into two phases: model training and model usage. In the training phase, a machine learning model is trained using training samples; in the usage phase, the trained model is used to make predictions on given samples. Machine learning models can include regression models, decision tree models, K-means clustering models, neural networks, and more.

[0003] For example, when using machine learning technology in the field of equipment anomaly management, historical operating data of equipment labeled with equipment anomaly events can be used as training samples. Based on these training samples, a machine learning model for classification can be trained in a supervised manner. After training, the trained machine learning model can be used to predict newly generated operating data during equipment operation to obtain possible anomaly events during equipment operation. This allows personnel responsible for equipment anomaly management to maintain the equipment based on the predicted anomaly events.

[0004] Therefore, how to build a system that can provide machine learning services for various application fields has become a highly concerned issue. Summary of the Invention

[0005] This specification provides one or more embodiments of the following technical solutions:

[0006] This specification provides a data processing method applied to a computing platform; the computing platform deploys user-defined functions corresponding to machine learning tasks; wherein, the user-defined functions include initialization logic and data processing logic corresponding to the machine learning task; the method includes:

[0007] Obtain the data to be processed corresponding to the machine learning task;

[0008] The user-defined function is invoked to retrieve the machine learning model corresponding to the machine learning task stored in the storage platform that interfaces with the computing platform;

[0009] The system further calls the data processing logic in the user-defined function, uses the machine learning model to process the data to be processed, and stores the data processing results in a database that is connected to the computing platform.

[0010] This specification also provides a data processing system, including:

[0011] Computing platform;

[0012] Storage platform and database that interface with the computing platform;

[0013] The computing platform is used to implement the steps of the method as described in any of the above.

[0014] This specification also provides an electronic device, including:

[0015] processor;

[0016] Memory used to store processor-executable instructions;

[0017] The processor executes the executable instructions to implement the steps of the method as described in any of the preceding descriptions.

[0018] This specification also provides a computer-readable storage medium having computer instructions stored thereon that, when executed by a processor, implement the steps of the method as described in any of the preceding claims.

[0019] In the above technical solution, a user-defined function corresponding to the machine learning task can be deployed in the computing platform. When the computing platform obtains the data to be processed corresponding to the machine learning task, it calls the initialization logic in the user-defined function to obtain the machine learning model corresponding to the machine learning task stored in the storage platform connected to the computing platform. Then, it calls the data processing logic in the user-defined function to use the machine learning model to process the data to be processed and stores the data processing result in the database connected to the computing platform.

[0020] Using the above approach, a data processing system consisting of a computing platform, a storage platform interfacing with the computing platform, and a database can provide machine learning services for various application domains. On one hand, the computing platform within this data processing system can meet the demand for massive computing power. On the other hand, because user-defined functions can be deployed within this computing platform, the platform can call the initialization logic within these user-defined functions to obtain machine learning models from outside the platform, thus avoiding cross-platform data transfer between the computing platform and the machine learning model. Furthermore, since the machine learning model is deployed outside the computing platform, it allows for efficient and continuous updates to the machine learning models used by the data processing system. Attached Figure Description

[0021] Figure 1 This is a schematic diagram illustrating a data processing system according to an exemplary embodiment of this specification.

[0022] Figure 2 This is a flowchart illustrating a data processing method in an exemplary embodiment of this specification.

[0023] Figure 3 This is a flowchart illustrating another data processing method as shown in an exemplary embodiment of this specification.

[0024] Figure 4 This is a schematic diagram illustrating a streaming data model as shown in an exemplary embodiment of this specification.

[0025] Figure 5 This is a schematic diagram of the structure of a device shown in an exemplary embodiment of this specification. Detailed Implementation

[0026] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numerals in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with one or more embodiments of this specification. Rather, they are merely examples of apparatuses and methods consistent with some aspects of one or more embodiments of this specification as detailed in the appended claims.

[0027] It should be noted that the steps of the corresponding methods are not necessarily performed in the order shown and described in this specification in other embodiments. In some other embodiments, the methods may include more or fewer steps than described in this specification. Furthermore, a single step described in this specification may be broken down into multiple steps in other embodiments; and multiple steps described in this specification may be combined into a single step in other embodiments.

[0028] In practical applications, on the one hand, applications such as equipment anomaly management have high requirements for real-time performance to avoid serious losses. Therefore, the use of machine learning technology in these applications usually requires a lot of computing power, which is a significant challenge for systems that provide machine learning services. On the other hand, in order to maintain the effectiveness of the machine learning technology, the machine learning model is usually updated when its computational accuracy decreases, so that a new machine learning model can be used in the future. Therefore, the machine learning model usually needs to undergo multiple updates, which to some extent increases the complexity of the design and implementation of systems that provide machine learning services.

[0029] In related technologies, to meet the high computing power requirements of systems providing machine learning services, powerful computing platforms are typically used to build these systems. To ensure easy updates to the machine learning models used in these systems, the models are usually deployed outside the computing platform. For example, the computing platform can be deployed on a server cluster consisting of multiple independent physical hosts, while the machine learning model can be deployed on other independent servers connected to this cluster. If the machine learning model were deployed directly within the computing platform, updating the model would require significant modifications to the platform itself. Therefore, deploying the machine learning model outside the computing platform avoids frequent and extensive modifications to the platform.

[0030] However, in a system that includes a computing platform and a machine learning model deployed outside of that computing platform, data transmission between the computing platform and the machine learning model is a cross-platform data transmission, and it requires multiple and large-scale data transmissions, resulting in significant communication overhead and impacting system performance.

[0031] This specification aims to provide a technical solution for data processing, enabling the construction of a data processing system capable of providing machine learning services for various application areas. This system meets the high computing power requirements of the data processing system, ensures easy updates to the machine learning models used, and improves system performance. In this solution, user-defined functions corresponding to machine learning tasks can be deployed in the computing platform. When the computing platform obtains the data to be processed corresponding to the machine learning task, it calls the initialization logic in the user-defined function to retrieve the machine learning model corresponding to the task stored in the storage platform connected to the computing platform. Furthermore, it calls the data processing logic in the user-defined function to process the data using the machine learning model and stores the processing results in the database connected to the computing platform.

[0032] In practical implementation, the aforementioned computing platform can deploy user-defined functions (UDFs) corresponding to machine learning tasks. These UDFs can contain initialization logic and data processing logic specific to the machine learning task; the initialization logic can trigger the computing platform to initialize for the machine learning task, and the data processing logic can trigger the computing platform to perform data processing corresponding to the machine learning task.

[0033] The aforementioned computing platform can acquire the data to be processed corresponding to the aforementioned machine learning task, and thus perform data processing corresponding to the machine learning task based on the data to be processed, thereby completing the machine learning task.

[0034] The aforementioned computing platform can call the initialization logic in the user-defined function to initialize the aforementioned machine learning task. Specifically, the initialization for the machine learning task may include retrieving the machine learning model corresponding to the machine learning task stored in the storage platform that interfaces with the computing platform.

[0035] After obtaining the data to be processed and the machine learning model, the computing platform can further invoke the data processing logic in the user-defined function to process the data using the machine learning model. Subsequently, the computing platform can store the data processing results in a database connected to the platform.

[0036] Using the above approach, a data processing system consisting of a computing platform, a storage platform interfacing with the computing platform, and a database can provide machine learning services for various application domains. On one hand, the computing platform within this data processing system can meet the demand for massive computing power. On the other hand, because user-defined functions can be deployed within this computing platform, the platform can call the initialization logic within these user-defined functions to obtain machine learning models from outside the platform, thus avoiding cross-platform data transfer between the computing platform and the machine learning model. Furthermore, since the machine learning model is deployed outside the computing platform, it allows for efficient and continuous updates to the machine learning models used by the data processing system.

[0037] Please refer to Figure 1 , Figure 1 This is a schematic diagram illustrating a data processing system according to an exemplary embodiment of this specification.

[0038] like Figure 1 As shown, the aforementioned data processing system may include a computing platform, a storage platform that interfaces with the computing platform, and a database that interfaces with the computing platform.

[0039] The aforementioned computing platform, storage platform, or database may be deployed on a server containing a single physical host, or on a server cluster consisting of multiple independent physical hosts; or, the aforementioned computing platform, storage platform, or database may be deployed on a server built on cloud computing services.

[0040] In practical applications, the aforementioned computing platform, storage platform, and database can be deployed on the same physical or virtual device, or on different physical or virtual devices. This specification does not impose any special restrictions on this.

[0041] In this embodiment, the storage platform can store the machine learning model corresponding to the machine learning task. The computing platform can deploy user-defined functions corresponding to the machine learning task, allowing the computing platform to retrieve the machine learning model stored in the storage platform by calling the user-defined functions, and then use the machine learning model to perform corresponding data processing to complete the machine learning task. The database can store the data processing results from the computing platform corresponding to the machine learning task.

[0042] In addition, firstly, the user client can interface with the aforementioned storage platform, allowing users to store the aforementioned machine learning models on the storage platform through the user client.

[0043] Secondly, the user client can interface with the aforementioned computing platform, allowing users to deploy the user-defined functions to the computing platform through the user client.

[0044] Thirdly, the user client can interface with the aforementioned database, allowing users to retrieve and view the data processing results from the database through the user client.

[0045] The aforementioned user client can correspond to terminal devices such as smartphones, tablets, laptops, PCs (Personal Computers), PDAs (Personal Digital Assistants), wearable devices (e.g., smart glasses, smartwatches), smart in-vehicle devices, or game consoles.

[0046] In practical applications, the three user clients mentioned above may refer to the same user client or different user clients; the users of these three user clients may be the same or different; this manual does not impose any special restrictions on this.

[0047] Please combine Figure 1 ,refer to Figure 2 , Figure 2 This is a flowchart illustrating a data processing method in an exemplary embodiment of this specification.

[0048] like Figure 2 The data processing method shown can be applied to, for example... Figure 1 The computing platform shown.

[0049] In this embodiment, the computing platform described above can deploy user-defined functions corresponding to machine learning tasks. These user-defined functions may include initialization logic and data processing logic corresponding to the machine learning task; the initialization logic can be used to trigger the computing platform to initialize for the machine learning task, and the data processing logic can be used to trigger the computing platform to perform data processing corresponding to the machine learning task.

[0050] In practical applications, the aforementioned machine learning task can correspond to the application of machine learning techniques in a specific application domain. Specifically, this machine learning task can correspond to the model training phase and / or the model usage phase in machine learning techniques. For example, assuming the application domain is equipment anomaly management, the machine learning task could specifically include using a trained machine learning model to predict possible abnormal events that may occur during equipment operation based on the operational data generated during equipment operation.

[0051] The above data processing method may include the following steps:

[0052] Step 202: Obtain the data to be processed corresponding to the machine learning task.

[0053] In this embodiment, the computing platform can acquire the data to be processed corresponding to the machine learning task, and then perform data processing corresponding to the machine learning task based on the data to be processed, thereby completing the machine learning task.

[0054] Since the aforementioned machine learning tasks can correspond to the application of machine learning techniques in a specific application domain, the data to be processed for that machine learning task can include data that needs to be processed within that application domain. For example, assuming the application domain is equipment anomaly management, the data to be processed could specifically include operational data generated during equipment operation.

[0055] Step 204: Call the initialization logic in the user-defined function to obtain the machine learning model corresponding to the machine learning task stored in the storage platform that interfaces with the computing platform.

[0056] In this embodiment, the computing platform can call the initialization logic in the user-defined function to initialize the machine learning task. Specifically, the initialization of the machine learning task may include retrieving the machine learning model corresponding to the machine learning task stored in the storage platform that interfaces with the computing platform.

[0057] Since the aforementioned machine learning tasks correspond to the application of machine learning techniques in a specific application domain, the machine learning model corresponding to that task can be a machine learning model trained on training samples composed of data that needs to be processed in that application domain. For example, assuming the application domain is equipment anomaly management, the machine learning model can specifically be a machine learning model obtained through supervised training based on training samples composed of equipment operation data labeled with equipment anomaly events.

[0058] In other words, although the machine learning model itself is not directly deployed on the computing platform, user-defined functions can be deployed on the platform. This allows the platform to call the initialization logic within these user-defined functions to retrieve the machine learning model from outside the platform, thus avoiding cross-platform data transfer between the computing platform and the machine learning model. Furthermore, because the machine learning model is deployed outside the computing platform, it allows for efficient and continuous updates to the model used by that platform.

[0059] Step 206: Further call the data processing logic in the user-defined function, use the machine learning model to process the data to be processed, and store the data processing result in the database that is connected to the computing platform.

[0060] In this embodiment, after obtaining the data to be processed and the machine learning model, the computing platform can further call the data processing logic in the user-defined function to process the data using the machine learning model. Subsequently, the computing platform can store the data processing results in a database connected to the platform, allowing users to retrieve and view the results from the database.

[0061] In the above technical solution, a user-defined function corresponding to the machine learning task can be deployed in the computing platform. When the computing platform obtains the data to be processed corresponding to the machine learning task, it calls the initialization logic in the user-defined function to obtain the machine learning model corresponding to the machine learning task stored in the storage platform connected to the computing platform. Then, it calls the data processing logic in the user-defined function to use the machine learning model to process the data to be processed and stores the data processing result in the database connected to the computing platform.

[0062] Using the above approach, a data processing system consisting of a computing platform, a storage platform interfacing with the computing platform, and a database can provide machine learning services for various application domains. On one hand, the computing platform within this data processing system can meet the demand for massive computing power. On the other hand, because user-defined functions can be deployed within this computing platform, the platform can call the initialization logic within these user-defined functions to obtain machine learning models from outside the platform, thus avoiding cross-platform data transfer between the computing platform and the machine learning model. Furthermore, since the machine learning model is deployed outside the computing platform, it allows for efficient and continuous updates to the machine learning models used by the data processing system.

[0063] In some embodiments, the computing platform may specifically include the Flink real-time computing platform, thereby directly utilizing the powerful computing capabilities of the Flink real-time computing platform to meet the large computing power requirements of the data processing system and reducing the complexity of building the data processing system.

[0064] Real-time computing deals with real-time and unbounded streaming data, which is subscribed to and consumed in real-time according to time sequence. For example, it could be a website's access log stream; as long as the website remains open, the access log stream will continuously generate and enter the real-time computing platform. Because the data entering the real-time computing platform is streaming, the entire computing process is continuous. The results of real-time computing can be directly written to the target data store. For example, the calculated report data can be directly written to a relational database for report display. The calculation results of the streaming data, like the streaming data itself, can be continuously written to the target data store.

[0065] In the above scenario, users can encapsulate the data processing logic corresponding to the aforementioned machine learning task into a user-defined function through a user client that interfaces with the Flink real-time computing platform. The user-defined function can then be configured to retrieve the machine learning model corresponding to the machine learning task from the aforementioned storage platform, thereby uploading and registering the user-defined function to the Flink real-time computing platform.

[0066] In some embodiments, the Flink real-time computing platform supports the development of Python UDFs (User-defined Functions, i.e., User-defined Functions in Python), a feature often referred to as PyFlink. Therefore, users can use a user client that interfaces with the Flink real-time computing platform to encapsulate the data processing logic corresponding to the aforementioned machine learning tasks into Python-based user-defined functions using PyFlink. The initialization logic within these user-defined functions can be configured to retrieve the machine learning model corresponding to the machine learning task from the aforementioned storage platform, allowing the user-defined function to be uploaded and registered with the Flink real-time computing platform. However, the Flink real-time computing platform itself is Java-based. Since the Flink real-time computing platform needs to call Python-based user-defined functions, an environment capable of running Python code needs to be configured within the Flink real-time computing platform as the runtime environment for the Python-based user-defined functions. This ensures that the Flink real-time computing platform can correctly call the user-defined functions deployed within it.

[0067] Therefore, users can not only upload the user-defined function to the computing platform through a user client that interfaces with the platform, but also upload the runtime environment that the user-defined function depends on through the same user client. In other words, the computing platform can obtain the user-defined function uploaded by the user client, as well as the runtime environment that the user-defined function depends on, and thus deploy the user-defined function to that runtime environment for invocation.

[0068] In some embodiments, the storage platform may specifically include OSS object storage service, thereby enabling the data processing system to directly utilize the storage capacity of OSS object storage service and reducing the complexity of building the data processing system.

[0069] In the above scenario, users can use their client, which interfaces with the OSS object storage service's cloud storage, to push the machine learning model corresponding to the aforementioned machine learning task to the OSS object storage service's cloud storage via the OSS object storage service's file upload interface. Correspondingly, the computing platform can call the initialization logic in the user-defined function to download the machine learning model corresponding to the aforementioned machine learning task from the OSS object storage service's file download interface.

[0070] In such Figure 2 Based on the embodiments, with reference to Figure 3 , Figure 3 This is a flowchart illustrating another data processing method as shown in an exemplary embodiment of this specification.

[0071] The above data processing method may include the following steps:

[0072] Step 302: Obtain the data to be processed corresponding to the machine learning task.

[0073] The specific implementation of step 302 can be referred to the aforementioned step 202, and will not be repeated here.

[0074] Step 304: Call the initialization logic in the user-defined function to obtain the machine learning model corresponding to the machine learning task and the domain knowledge of the application field corresponding to the machine learning task stored in the storage platform that interfaces with the computing platform.

[0075] In this embodiment, the computing platform can call the initialization logic in the user-defined function to initialize the machine learning task. Specifically, the initialization of the machine learning task may include obtaining the machine learning model corresponding to the machine learning task stored in the storage platform connected to the computing platform, and obtaining domain knowledge of the application domain corresponding to the machine learning task stored in the storage platform.

[0076] The aforementioned domain knowledge can be used to pre-screen the data to be processed, identifying more valuable data segments, and then processing these segments based on the aforementioned machine learning model. For example, assuming the application domain is equipment anomaly management, this domain knowledge can represent patterns in the equipment operation data corresponding to equipment anomaly events, such as the continuous and repeated occurrence of the character "ERROR" in the equipment operation data. Therefore, data segments conforming to this pattern can be screened from the data to be processed, and these segments can be processed based on the aforementioned machine learning model.

[0077] Step 306: Further call the data processing logic in the user-defined function, preprocess the data to be processed based on the domain knowledge, use the machine learning model to process the preprocessed data to be processed, and store the data processing results in the database connected to the computing platform.

[0078] In this embodiment, after obtaining the data to be processed and the machine learning model, the computing platform can further call the data processing logic in the user-defined function to preprocess the data based on the domain knowledge, and then use the machine learning model to process the preprocessed data. Subsequently, the computing platform can store the data processing results in a database connected to the platform, allowing users to retrieve and view the results from the database.

[0079] In some embodiments, the above application areas may include device anomaly management. In this case, the machine learning model described above can be used to predict abnormal events generated during device operation; the data to be processed may include operation logs generated during device operation; and the data processing results may include abnormal events generated during device operation predicted by the machine learning model.

[0080] In some embodiments, by storing the data processing results in a database that interfaces with the computing platform, a user client connected to the database can retrieve the data processing results stored in the database and output the results to the user. That is, the user can retrieve and view the data processing results from the database through a user client connected to the database.

[0081] In practical applications, continuously generated data can be divided into two types: bounded data and unbounded data.

[0082] Bounded data refers to data with a defined start and end time. For bounded data, processing can be performed after acquiring all data within the timeframe from the start to the end time. Bounded data can be sorted, so ordered acquisition is not required. Processing bounded data is often referred to as batch processing. In this case, all data within the timeframe from the start to the end time can be considered as a single batch of data.

[0083] Unbounded data refers to data whose start time is defined, but whose end time is not. In other words, unbounded data is considered to be data that is generated endlessly. Therefore, continuous data processing is required for unbounded data. Unbounded data typically requires acquisition in a specific order, such as chronological order, to ensure the completeness of the data processing results. Processing unbounded data is often referred to as stream processing.

[0084] In some embodiments, the data to be processed may be batch data related to batch processing. Specifically, the data to be processed is batch data from the same batch.

[0085] In the above scenario, for each batch of data, the computing platform can first call the initialization logic in the user-defined function to obtain the machine learning model corresponding to the machine learning task stored in the storage platform, and then further call the data processing logic in the user-defined function to process the batch of data using the machine learning model and store the data processing results in the database.

[0086] For example, assuming that data generated between 12:00 and 13:00 on a certain day is defined as the first batch of data, and data generated between 14:00 and 15:00 on the same day is defined as the second batch of data, then when the computing platform obtains the first batch of data as the data to be processed, it can first call the initialization logic in the user-defined function to obtain the machine learning model corresponding to the machine learning task stored in the storage platform, and then further call the data processing logic in the user-defined function to use the machine learning model to process the first batch of data and store the data processing results in the database. When the computing platform obtains the second batch of data as the data to be processed, it can also first call the initialization logic in the user-defined function to obtain the machine learning model corresponding to the machine learning task stored in the storage platform, and then further call the data processing logic in the user-defined function to use the machine learning model to process the second batch of data and store the data processing results in the database.

[0087] In some embodiments, the data to be processed is streaming data related to stream processing.

[0088] In the above scenario, for streaming data, the computing platform can call the initialization logic in the user-defined function once upon obtaining the starting point of the streaming data to retrieve the machine learning model corresponding to the machine learning task stored in the storage platform. Subsequently, the computing platform can continuously process the streaming data without needing to call the initialization logic in the user-defined function again to retrieve the machine learning model. Specifically, it can further call the data processing logic in the user-defined function to process all data (i.e., a portion of the streaming data) acquired within the latest time period according to a preset time cycle. This time cycle can be set by the user according to actual needs or it can be a system default value; this specification does not impose any special restrictions on it.

[0089] For example Figure 4Taking the streaming data shown as an example, assuming the streaming data begins at 12:00 on a certain day, and the time period is 1 hour, then when the computing platform obtains the starting point of the streaming data at 12:00 on that day, it can call the initialization logic in the user-defined function once to obtain the machine learning model corresponding to the machine learning task stored in the storage platform. The computing platform can further call the data processing logic in the user-defined function. At 13:00 on that day, it uses the machine learning model to process all the data obtained between 12:00 and 13:00 on that day; at 14:00 on that day, it uses the machine learning model to process all the data obtained between 13:00 and 14:00 on that day; at 15:00 on that day, it uses the machine learning model to process all the data obtained between 14:00 and 15:00 on that day; and so on.

[0090] In some embodiments, since users can retrieve and view the data processing results from the database through a user client connected to the database, they can statistically analyze the data processing results to determine whether the computational accuracy of the machine learning model has significantly decreased. If so, the user can update the machine learning model and store the updated model in the storage platform through the user client connected to the storage platform. After the machine learning model stored in the storage platform is updated, the computing platform needs to retrieve the updated model from the storage platform again to use it for data processing, thereby ensuring the accuracy of the data processing results.

[0091] When the data to be processed is batch data related to batch processing, the computing platform can first call the initialization logic in the user-defined function for each batch of batch data to obtain the machine learning model corresponding to the machine learning task stored in the storage platform. Therefore, the machine learning task executed by the computing platform does not need to be restarted.

[0092] In the case where the data to be processed is streaming data related to stream processing, since the computing platform performs continuous data processing on the streaming data, the machine learning task executed by the computing platform can be restarted to enable the computing platform to re-acquire the updated machine learning model stored in the storage platform.

[0093] Specifically, in response to the task restart event triggered after the machine learning model stored in the storage platform is updated, the computing platform can re-invoke the initialization logic in the user-defined function to obtain the updated machine learning model stored in the storage platform, and further invoke the data processing logic in the user-defined function to process the data to be processed using the updated machine learning model, and store the data processing result in the database connected to the computing platform.

[0094] It should be noted that the computing platform will only reacquire the newly generated streaming data, which serves as the data to be processed, after the aforementioned machine learning task restarts. Therefore, upon acquiring the starting point of the newly generated streaming data, the computing platform can re-invoke the initialization logic in the user-defined function, retrieve the updated machine learning model stored in the storage platform, and further invoke the data processing logic in the user-defined function to process the newly generated streaming data using the updated machine learning model, and store the data processing results in the database connected to the computing platform.

[0095] Please continue to refer to this. Figure 1 An exemplary embodiment of this specification illustrates a data processing system, including: a computing platform; a storage platform and a database that interface with the computing platform;

[0096] The computing platform deploys user-defined functions corresponding to machine learning tasks; wherein, the user-defined functions contain initialization logic and data processing logic corresponding to the machine learning tasks;

[0097] The computing platform is used for:

[0098] Obtain the data to be processed corresponding to the machine learning task;

[0099] The user-defined function is invoked to retrieve the machine learning model corresponding to the machine learning task stored in the storage platform that interfaces with the computing platform;

[0100] The system further calls the data processing logic in the user-defined function, uses the machine learning model to process the data to be processed, and stores the data processing results in a database that is connected to the computing platform.

[0101] Optionally, the computing platform includes the Flink real-time computing platform.

[0102] Optionally, the storage platform includes OSS object storage service.

[0103] Optionally, the computing platform is specifically used for:

[0104] Obtain the machine learning model corresponding to the machine learning task, and the domain knowledge of the application field corresponding to the machine learning task, stored in the storage platform that interfaces with the computing platform;

[0105] The data to be processed is preprocessed based on the domain knowledge, and the preprocessed data to be processed is then processed using the machine learning model.

[0106] Optionally, the application area includes equipment anomaly management; the machine learning model is used to predict abnormal events generated during equipment operation; the data to be processed includes operation logs generated during equipment operation; and the data processing result includes abnormal events generated during equipment operation predicted by the machine learning model.

[0107] Optionally, the computing platform is specifically used for:

[0108] The data processing results are stored in a database that interfaces with the computing platform, so that user clients that interface with the database can obtain the data processing results stored in the database and output the data processing results to the user.

[0109] Optionally, the data to be processed is batch data related to batch processing.

[0110] Optionally, the data to be processed is streaming data related to stream processing;

[0111] The computing platform is specifically used for:

[0112] According to a preset time period, the machine learning model is used to process the data to be processed obtained in the latest time period.

[0113] Optionally, the computing platform is also used for:

[0114] In response to the task restart event corresponding to the machine learning task triggered after the machine learning model is updated, the initialization logic in the user-defined function is called again to obtain the updated machine learning model stored in the storage platform. The data processing logic in the user-defined function is then called to process the data to be processed using the updated machine learning model, and the data processing result is stored in the database connected to the computing platform.

[0115] Optionally, the computing platform is also used for:

[0116] Obtain the user-defined function uploaded by the user client, and the runtime environment that the user-defined function depends on;

[0117] The user-defined function is deployed to the runtime environment so that it can be called within the runtime environment.

[0118] The system implementation is basically the same as the method implementation; therefore, relevant details can be found in the description of the method implementation.

[0119] Figure 5 This is a schematic diagram illustrating the structure of a device according to an exemplary embodiment of this specification. Please refer to... Figure 5 At the hardware level, the device includes a processor 502, an internal bus 504, a network interface 506, memory 508, and non-volatile memory 510, and may also include other necessary hardware. One or more embodiments of this specification can be implemented in software, such as the processor 502 reading the corresponding computer program from the non-volatile memory 510 into memory 508 and then running it. Of course, in addition to software implementation, one or more embodiments of this specification do not exclude other implementation methods, such as logic devices or a combination of hardware and software, etc. That is to say, the execution subject of the following processing flow is not limited to each logic module, but can also be hardware or logic devices.

[0120] The systems, devices, modules, or units described in the above embodiments can be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer, which can take the form of a personal computer, laptop computer, cellular phone, camera phone, smartphone, personal digital assistant, media player, navigation device, email sending and receiving device, game console, tablet computer, wearable device, or any combination of these devices.

[0121] In a typical configuration, a computer includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0122] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0123] Computer-readable media, including both permanent and non-permanent, removable and non-removable media, can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, disk storage, quantum memory, graphene-based storage media or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0124] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0125] The foregoing has described specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired result. In some embodiments, multitasking and parallel processing are possible or may be advantageous.

[0126] The terminology used in one or more embodiments of this specification is for the purpose of describing particular embodiments only and is not intended to limit the scope of one or more embodiments of this specification. The singular forms “a,” “described,” and “the” used in one or more embodiments of this specification and in the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any or all possible combinations of one or more associated listed items.

[0127] It should be understood that although the terms first, second, third, etc., may be used to describe various information in one or more embodiments of this specification, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, first information may also be referred to as second information without departing from the scope of one or more embodiments of this specification, and similarly, second information may also be referred to as first information. Depending on the context, the word "if" as used herein may be interpreted as "when," "in response to a determination," or "when," or "in the event of a determination."

[0128] The above description is merely a preferred embodiment of one or more embodiments of this specification and is not intended to limit the scope of one or more embodiments of this specification. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of one or more embodiments of this specification should be included within the protection scope of one or more embodiments of this specification.

[0129] The user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this manual are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation portals are provided for users to choose to authorize or refuse.

Claims

1. A data processing method, said method being applied to a computing platform; wherein, the computing platform deploys user-defined functions corresponding to machine learning tasks; wherein, The user-defined function contains initialization logic and data processing logic corresponding to the machine learning task; the method includes: Obtain the data to be processed corresponding to the machine learning task; The user-defined function is invoked to retrieve the machine learning model corresponding to the machine learning task and the domain knowledge of the application field corresponding to the machine learning task stored in the storage platform that interfaces with the computing platform. The system further calls the data processing logic in the user-defined function to preprocess the data to be processed based on the domain knowledge, uses the machine learning model to process the preprocessed data to be processed, and stores the data processing results in a database that is connected to the computing platform.

2. The method according to claim 1, wherein the application field includes equipment anomaly management; the machine learning model is used to predict abnormal events generated during equipment operation; the data to be processed includes operation logs generated during equipment operation; and the data processing result includes abnormal events generated during equipment operation predicted by the machine learning model.

3. The method according to claim 2, wherein storing the data processing results in a database interfaced with the computing platform comprises: The data processing results are stored in a database that interfaces with the computing platform, so that user clients that interface with the database can obtain the data processing results stored in the database and output the data processing results to the user.

4. The method according to claim 1, wherein the data to be processed is batch data related to batch processing.

5. The method according to claim 1, wherein the data to be processed is streaming data related to stream processing; The process of using the machine learning model to process the data to be processed includes: According to a preset time period, the machine learning model is used to process the data to be processed obtained in the latest time period.

6. The method according to claim 5, further comprising: In response to the task restart event corresponding to the machine learning task triggered after the machine learning model is updated, the initialization logic in the user-defined function is called again to obtain the updated machine learning model stored in the storage platform. The data processing logic in the user-defined function is then called to process the data to be processed using the updated machine learning model, and the data processing result is stored in the database connected to the computing platform.

7. The method according to claim 1, further comprising: Obtain the user-defined function uploaded by the user client, and the runtime environment that the user-defined function depends on; The user-defined function is deployed to the runtime environment so that it can be called within the runtime environment.

8. A data processing system, comprising: Computing platform; Storage platform and database that interface with the computing platform; The computing platform is used to implement the method as described in any one of claims 1 to 7.

9. An electronic device, comprising: processor; Memory used to store processor-executable instructions; The processor implements the method as described in any one of claims 1 to 7 by executing the executable instructions.

10. A computer-readable storage medium having stored thereon computer instructions that, when executed by a processor, implement the method as described in any one of claims 1 to 7.

Images