Content processing method and related product

Abstract

The embodiment of the invention discloses a content processing method and related products, and the method comprises the steps: receiving an asynchronous task from a management platform, the asynchronous task comprises a resource bundle of a target task, the resource bundle is downloaded from a task warehouse by the management platform, and the target task is used for processing a plurality of contents; and executing the resource packet by using a first processing resource of the at least one first target working node and a second processing resource of the at least one second target working node through the at least one first target working node and the at least one second target working node so as to process the plurality of contents. According to the embodiment of the invention, the content processing efficiency is improved.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a content processing method and related products. Background Art

[0002] Thanks to the increase in machine learning computing power and algorithm improvements, in recent years, user-generated content (UGC) has increasingly used machine learning models to understand content, and the understanding results have been used in recommendation systems, review systems, and other artificial intelligence systems to improve user experience.

[0003] In actual applications, models are constantly iterated to improve their effectiveness. After each iteration, a new version of the model needs to be used to re-understand a large amount of UGC. The process of re-understanding UGC using a new version of the model is called algorithm data retracement. At present, how to use a new version of the model to efficiently re-understand a batch of UGC is a technical problem that needs to be solved urgently. Summary of the invention

[0004] The embodiments of the present application provide a content processing method and related products. When performing algorithm data retrieval on content, the first processing resource and the second processing resource in the cluster resources are used to process the content simultaneously, that is, the content is retrieval, thereby improving the retrieval efficiency.

[0005] In a first aspect, an embodiment of the present application provides a content processing method, the method being applied to a target master node, the target master node being a master node in a target cluster, the target cluster further comprising at least one first target working node having a first processing resource and at least one second target working node having a second processing resource; the method comprising:

[0006] Receiving an asynchronous task from a management platform, wherein the asynchronous task includes a resource package of a target task, the resource package is downloaded by the management platform from a task repository, and the target task is used to process a plurality of contents;

[0007] The resource package is executed through the at least one first target working node and the at least one second target working node, utilizing the first processing resources of the at least one first target working node and the second processing resources of the at least one second target working node, so as to process the multiple contents.

[0008] In a second aspect, an embodiment of the present application provides a content processing method, which is applied to a management platform, and the method includes:

[0009] Acquire a target task, wherein the target task is used to process multiple contents;

[0010] Download the resource package corresponding to the target task from the task warehouse;

[0011] Submit an asynchronous task to a target master node, wherein the target master node is a node in a target cluster, and the target cluster also includes at least one first target work node having a first processing resource and at least one second target work node having a second processing resource, and the asynchronous task includes the resource package, and the asynchronous task is used to request the target master node to execute the resource package through the at least one first target work node and the at least one second target work node, using the first processing resources of the at least one first target work node and the second processing resources of the at least one second target work node, so as to process the multiple contents.

[0012] In a third aspect, an embodiment of the present application provides a content processing method, which is applied to a content processing system, wherein the content processing system includes: a management platform and a target cluster; the target cluster includes a target master node, at least one first target working node having a first processing resource, and at least one second target working node having a second processing resource; the method includes:

[0013] The management platform acquires a target task, wherein the target task includes a task identifier, and the target task is used to process multiple contents;

[0014] The management platform downloads the resource package of the target task from the task warehouse based on the task identifier;

[0015] The management platform submits an asynchronous task to the target master node, wherein the asynchronous task includes the resource package;

[0016] The target master node executes the resource package through the at least one first target working node and the at least one second target working node, utilizing the first processing resources of the at least one first target working node and the second processing resources of the at least one second target working node, so as to process the multiple contents.

[0017] In a fourth aspect, an embodiment of the present application provides a node, including: an acquisition module and a processing module;

[0018] The acquisition module is used to receive an asynchronous task from the management platform, wherein the asynchronous task includes a resource package of a target task, the resource package is downloaded by the management platform from a task repository, and the target task is used to process multiple contents;

[0019] The processing module is used to execute the resource package through the at least one first target working node and the at least one second target working node, using the first processing resources of the at least one first target working node and the second processing resources of the at least one second target working node, so as to process the multiple contents.

[0020] In a fifth aspect, an embodiment of the present application provides a management platform, including: an acquisition module and a processing module;

[0021] The acquisition module is used to acquire a target task, wherein the target task is used to process multiple contents;

[0022] The processing module is used to download the resource package corresponding to the target task from the task warehouse;

[0023] Submit an asynchronous task to a target master node, wherein the target master node is a node in a target cluster, and the target cluster also includes at least one first target work node having a first processing resource and at least one second target work node having a second processing resource, and the asynchronous task includes the resource package, and the asynchronous task is used to request the target master node to execute the resource package through the at least one first target work node and the at least one second target work node, using the first processing resources of the at least one first target work node and the second processing resources of the at least one second target work node, so as to process the multiple contents.

[0024] In a sixth aspect, an embodiment of the present application provides a content processing system, the content processing system comprising: a management platform and a target cluster; the target cluster comprises a target master node, at least one first target working node having a first processing resource, and at least one second target working node having a second processing resource;

[0025] The management platform is used to obtain a target task, wherein the target task includes a task identifier, and the target task is used to process multiple contents;

[0026] The management platform is used to download the resource package of the target task from the task warehouse based on the task identifier;

[0027] The management platform is used to submit an asynchronous task to the target master node, wherein the asynchronous task includes the resource package;

[0028] The target master node is used to execute the resource package through the at least one first target working node and the at least one second target working node, utilizing the first processing resources of the at least one first target working node and the second processing resources of the at least one second target working node, so as to process the multiple contents.

[0029] In a seventh aspect, an embodiment of the present application provides a content rewinding method, including:

[0030] The management platform obtains a retrace task, wherein the retrace task is used to retrace multiple user-generated contents;

[0031] The management platform downloads a resource package corresponding to the retrace task from the task warehouse;

[0032] The management platform parses the resource package to determine the cluster resources required for the retrace task;

[0033] The management platform sends a cluster creation request to the k8s cluster, wherein the cluster creation request includes cluster resources;

[0034] The k8s cluster creates a ray cluster on the k8s based on the cluster resources;

[0035] The management platform submits a ray job to the ray cluster, wherein the ray job includes the resource package;

[0036] The ray cluster executes the ray job based on the resource package to retrace the plurality of user-generated contents.

[0037] In an eighth aspect, an embodiment of the present application provides an electronic device, comprising: a processor and a memory, wherein the processor is connected to the memory, the memory is used to store a computer program, and the processor is used to execute the computer program stored in the memory, so that the electronic device executes the method described in any one of aspects 1 to 3.

[0038] In a ninth aspect, an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method described in any one of the first aspect to the third aspect is implemented.

[0039] In a tenth aspect, an embodiment of the present application provides a computer program product, wherein the computer program product includes a computer program, and when the computer program is executed by a processor, the method described in any one of the first aspect to the third aspect is implemented.

[0040] Implementing the embodiments of the present application has the following beneficial effects:

[0041] It can be seen that in an embodiment of the present application, when multiple contents need to be processed, the user can submit a target task to the management platform; then the management platform downloads the resource package corresponding to the target task from the task warehouse, and then the management platform submits the resource package to the target cluster in the form of an asynchronous task; since the target cluster has a first target working node with a first processing resource and a second target working node with a second processing resource, the first processing resource and the second processing resource in the target cluster can be used to execute the resource package, so that the logical processing power of the CPU and the parallel power of the GPU can be used to process multiple contents, thereby achieving efficient content processing. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0043] Figure 1 A schematic diagram of a k8s cluster provided in an embodiment of the present application;

[0044] Figure 2 A schematic diagram of a ray cluster provided in an embodiment of the present application;

[0045] Figure 3 A schematic diagram of a content processing system provided in an embodiment of the present application;

[0046] Figure 4 A schematic diagram of another content processing system provided in an embodiment of the present application;

[0047] Figure 5 A flowchart of a content processing method provided in an embodiment of the present application;

[0048] Figure 6 A schematic diagram of content processing provided in an embodiment of the present application;

[0049] Figure 7 A flowchart of a content rewinding method provided in an embodiment of the present application;

[0050] Figure 8 A schematic diagram of a node provided in an embodiment of the present application;

[0051] Fig. 9 A schematic diagram of a management platform provided in an embodiment of the present application;

[0052] Fig.10A schematic diagram of a content processing system provided in an embodiment of the present application;

[0053] Fig.11 A schematic diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0054] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0055] The terms "first", "second", "third" and "fourth" etc. in the specification and claims of the present application and the drawings are used to distinguish different objects, rather than to describe a specific order. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not limited to the listed steps or units, but optionally includes steps or units that are not listed, or optionally includes other steps or units inherent to these processes, methods, products or devices.

[0056] Reference to "embodiments" herein means that a particular feature, result, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0057] In order to facilitate understanding of the technical solution of the present application, the relevant technologies involved in the present application will be introduced below in conjunction with the accompanying drawings.

[0058] See also Figure 1 , Figure 1 A schematic diagram of a k8s (Kubernetes) cluster provided in an embodiment of the present application.

[0059] like Figure 1 As shown in the figure, the k8s cluster includes a master node (Master) and one or more worker nodes (Workernode). In the k8s cluster, the Master is responsible for the scheduling, management and operation and maintenance of the k8s cluster. Therefore, the Master serves as the control plane of the entire k8s cluster, while the Worker node carries the actual workload.

[0060] like Figure 1 As shown, the components deployed in the Master include but are not limited to the API server (Server), scheduler (Scheduler), controller manager (controller-manager) and distributed key-value storage system (Etcd);

[0061] APl Server: As the gateway of Kubernetes, it receives and processes all command requests;

[0062] Scheduler: Schedules the resource requests of containerized applications to suitable worker nodes according to predefined scheduling policies. The scheduler performs scheduling based on factors such as resource utilization, health status, and affinity of worker nodes.

[0063] Controller-Manager: The management controller in the k8s cluster, used to manage and control the life cycle of resource objects.

[0064] Etcd: It is a distributed key-value storage system for k8s clusters, used to store cluster configuration data and status information.

[0065] Furthermore, for user-defined resources (Custom Resource Definitions, CRD), similar to built-in resources such as Pod, they can also be stored in Etcd, so as to facilitate the subsequent implementation of user customization requirements based on the CRD. The CRD of this application is mainly a custom resource related to creating a ray cluster. Specifically, the CRD is a custom yaml file.

[0066] like Figure 1 As shown in the figure, the components deployed on the Worker node are: Kubelet, Kube-proxy, Docker, Pod, and Container.

[0067] Among them, Pod is the smallest deployable unit in the k8s cluster. A Pod can contain one or more containers that share the same network and storage resources and run on the same node. Pod provides an abstract layer for encapsulating the operating environment of an application. It can contain all the components required by the application, such as the main application container, auxiliary containers, and shared storage volumes. Specifically, Pod has the following characteristics:

[0068] Network sharing: All containers in a Pod share the same network namespace and IP address;

[0069] Storage sharing: All containers in a Pod can access shared storage volumes, which allows them to share data and files;

[0070] Lifecycle: Pod has its own lifecycle, it can be created, started, stopped and destroyed. When a Pod is destroyed, all containers in it will also be terminated;

[0071] Scheduling and deployment: Pods are temporary and can be created and destroyed as needed. Pods can be scheduled and deployed on k8s worker nodes by the Master's Scheduler. The Scheduler will consider the resources and constraints of the worker nodes and select the appropriate worker node to run the Pod. Figure 1 As shown, multiple Pods can be deployed on each worker node.

[0072] Kubelet: An agent program running on each worker node, responsible for managing containers and containerized applications on the worker node. Kubelet communicates with the APl Server, receives instructions and performs corresponding operations, such as creating, destroying, and monitoring containers.

[0073] Kube-proxy: responsible for implementing service discovery and load balancing in the k8s cluster.

[0074] Docker and Container are both Container Runtime, which provides functions such as container lifecycle management, resource isolation and security. Docker is an open source container platform that provides a complete set of container solutions, including building, packaging, distributing and running containers. It includes multiple components such as Docker Engine, Docker Swarm, DockerCompose, etc., and provides advanced functions such as image building and container orchestration, which is suitable for users who need complex container management functions. Docker is not only a container engine, but also includes components such as network and storage. It can easily create, stop, start, delete and other operations of containers, with higher security and stability. Container is a lightweight container runtime that focuses on the underlying container lifecycle management, such as container creation, start, stop and destruction. It is lighter than Docker, can quickly start and run containers, and has better performance and scalability. Containerd is designed to be simpler and more stable, and pays more attention to container lifecycle management and basic image operations.

[0075] The following describes the Pod scheduling and deployment process.

[0076] First, the operation and maintenance personnel send a Pod creation request to the API Server. The API Server responds to the Pod creation request, and after authentication and authorization, stores the Pod creation request in Etcd and notifies the Controller Manager. The ControllerManager reads the request in Etcd through the API Server and creates the Pod object according to the preset template. After the creation is completed, the Pod object is written to Etcd, and the Scheduler is notified through the API Server, requesting the Scheduler to select the most suitable Worker node for the newly created Pod object. The Scheduler selects the best Worker node from all Worker nodes according to the scheduling strategy. After the Scheduler selects the best Worker node, it returns the selected best Worker node to the Controller Manager through the API Server. The Controller Manager writes the selection result to Etcd through the APIServer and updates the status of the Pod stored in Etcd, for example, updating the node information of the newly created Pod, and so on. The Kubelet on the optimal worker node can obtain the updated Pod status through the API Server and perform corresponding operations based on the updated Pod status, such as pulling container images, creating containers, and starting containers, thereby providing a container environment for the operation of the newly created Pod, thereby realizing the creation of the Pod and scheduling the Pod to the optimal Worker node.

[0077] Furthermore, a controller (Operator) is pre-installed in the k8s cluster, where the Operator is actually the controller of the CRD, which is used to monitor changes in custom resources in Etcd. For example, after detecting that a custom resource object has been created in Etcd, the Operator can read the properties of the custom resource and then create a Pod to run a specific program, and bind the Pod to the custom resource object to complete the creation of the custom resource object. This application mainly uses the Operator as a cluster controller as an example for explanation. The cluster controller is mainly used to create a target cluster on a k8s cluster. Furthermore, if Figure 1 As shown, this application takes the cluster controller as Ray Operator as an example to facilitate the creation of a ray cluster on a k8s cluster through Ray Operator. Optionally, the deployment of Ray Operator spans the entire k8s cluster, and Ray Operator runs on any available node according to the cluster's scheduling strategy.

[0078] See also Figure 2 , Figure 2 A schematic diagram of a ray cluster provided in an embodiment of the present application. Figure 2 As shown, raycluster includes a head node and one or more worker nodes.

[0079] Each Worker node consists of the following processes:

[0080] 1) One or more Worker processes;

[0081] The one or more Worker processes are responsible for task submission and execution. Each Worker process is either stateless or stateful. A stateless Worker process can be understood as a Pod, and a stateful Worker process can be understood as a ray actor. The number of initial Worker processes on each Worker node is determined by the number of CPUs on the Workernode.

[0082] 2) Raylet;

[0083] Raylet is the local scheduler on each Worker node, responsible for scheduling tasks and managing the Worker process on each Workernode:

[0084] 3) Object Store;

[0085] The shared memory storage in a Worker node is also called Plasma Object Store. The Object Stores in each Worker node in Raycluster together build the distributed object storage of Ray cluster.

[0086] 4) Scheduler;

[0087] Responsible for resource management, task scheduling, and storing task parameters in the distributed Object Store.

[0088] Compared with Worker node, Head node includes Driver process, Global Control Store (GCS) and Autoscaling in addition to the above-mentioned Worker node related processes.

[0089] The Driver process is a node designated for running the top-level code application. It can submit tasks but cannot execute them on itself. Although the Driver can run on any node, it only runs on the Head node by default. GCS is a server that manages the metadata of the ray cluster, such as the location of the actor, the key-value pairs stored by the Worker process, etc. Autoscaling is an independent process running on the master node, responsible for increasing or decreasing the number of Worker nodes in the cluster ray cluster according to certain conditions.

[0090] In order to facilitate understanding of the technical solution of the present application, the scenarios involved in the present application are first described.

[0091] First, the multiple contents of the present application may be any type of content, such as content generated by a machine during operation, or user-generated content, and the present application does not limit this. For ease of understanding, the present application mainly takes the multiple contents as multiple user-generated content as an example for explanation;

[0092] Secondly, since algorithmic data retrace is to re-understand the content using the new version of the model, algorithmic data retrace can also be called data retrace, content retrace or retrace, and the algorithmic data retrace task corresponding to the algorithmic data retrace is also called data retrace task, content retrace task or retrace task. Therefore, for the sake of unified description, this application uniformly refers to algorithmic data retrace as content retrace, and algorithmic data retrace task as retrace task;

[0093] Finally, the target task of this application can be any task that meets the following three conditions:

[0094] 1. The amount of data that needs to be processed is relatively large; 2. The processing logic is flexible, requiring a large amount of calculation and input and output; 3. The calculation mode is complex, and CPU resources and GPU resources are required during the calculation process.

[0095] Optionally, since the retrace task is to use the new version of the model to re-understand multiple user-generated contents, the amount of data that needs to be processed is relatively large, and a large amount of reading and writing is required, such as reading multiple user-generated contents from a specified location, and writing the re-understood multiple user-generated contents to a specified location; and the re-understanding process involves a variety of data preprocessing, the logic of data preprocessing is relatively flexible, and the amount of calculation is relatively large; and in the re-understanding process, the user-generated content needs to be re-reasoned, which requires the use of CPU resources and GPU resources for reasoning. Therefore, the retrace task meets the above three conditions, so the target task of this application can be a retrace task;

[0096] Optionally, for big data analysis tasks, the amount of data that needs to be analyzed is relatively large, and it will involve a large number of reading and writing processes, that is, there are a large number of input and output requirements; moreover, during the analysis process, it will be necessary to batch process multiple data, there will be a large number of computing requirements and flexible processing logic, such as the cross-use of parallel processing and serial processing, etc.; and during the analysis process, the corresponding model will be called to explore the potential connections between the data, so CPU resources and GPU resources are also needed in the big data analysis process. Therefore, the big data analysis task also meets the above three conditions, so the target task of this application can also be a big data analysis task;

[0097] Optionally, for log analysis to perform system warning, it is necessary to collect log information of thousands of systems, so there is a large amount of reading and writing requirements; during the log analysis process, the logs can be flexibly processed according to the analysis requirements, for example, parallel analysis and serial analysis can be used interchangeably; and during the analysis process, the corresponding model will be called to explore the potential connections between different logs, so as to analyze whether there are risks in the operation of the system. Therefore, CPU resources and GPU resources are also needed in the log analysis process. Therefore, the log analysis to perform system warning tasks also meets the above three conditions, so the target task of this application can also be the log analysis to perform system warning tasks.

[0098] In order to facilitate understanding of the technical solution of the present application, the present application mainly uses the target task as a retrace task for illustration, that is, the present application mainly describes how to use ray cluster to retrace multiple user contents. Therefore, the target tasks mentioned later can all be understood as retrace tasks, and the processing of multiple contents can all be understood as retracement of multiple contents, and no repeated description will be given.

[0099] For the convenience of description, the original cluster mentioned in this application is described by taking the k8s cluster as an example, but the type of the original cluster is not limited. The target cluster is described by taking the ray cluster as an example, but the type of the target cluster is not limited. For the convenience of distinction, this application refers to the master node in the original cluster as the original master node, and the working node in the original cluster as the original working node. In this application, the first processing resource and / or the second processing resource can be configured for each original node. As for whether the first processing resource is configured on each original node or the second processing resource is configured according to actual needs, it is not limited to this. This application mainly describes the configuration of the first processing resource and the second processing resource for the original node as an example. And, in order to facilitate the distinction between the original cluster and the target cluster, the original working node with the first processing resource is referred to as the first original working node in this application, and the original working node with the second processing resource is referred to as the second original working node. The master node in the target cluster is referred to as the target master node, the working node with the first processing resource in the target cluster is referred to as the first target working node, and the working node with the second processing resource is referred to as the second target working node.

[0100] Optionally, this application mainly uses the first processing resource as a CPU resource and the second processing resource as a GPU resource as an example for illustration, but this application does not limit the types of the first processing resource and the second processing resource. The first processing resource mentioned later can be understood as a CPU resource, and the second processing resource can be understood as a GPU resource, and no repeated description is given.

[0101] See also Figure 3 , Figure 3 A schematic diagram of a content processing system provided in an embodiment of the present application. Figure 3 As shown, the content processing system includes a management platform 301 , an original cluster 302 , a target cluster 303 , a meta-service platform 304 and a task warehouse 305 .

[0102] The target cluster 303 is created based on the original cluster 302. The original cluster includes an original master node, a first original working node with a first processing resource, and a second original working node with a second processing resource, wherein the number of the first original working node and the second original working node can be one or more, and the present application does not limit this; the target cluster 303 includes a target master node, a first target working node with a first processing resource, and a second target working node with a second processing resource, wherein the number of the first target working node and the second target working node can be one or more.

[0103] First, for each target task, the task code and configuration file of the target task are packaged and compressed to obtain a resource package of each target task, and the resource package of each target task is uploaded to the task warehouse 305 .

[0104] Therefore, when the user submits a target task for processing multiple contents to the management platform 301, the management platform 301 obtains the task identifier of the target task, and then downloads the resource package of the target task from the task warehouse 305 based on the task identifier. Then, the management platform 301 submits an asynchronous task to the target master node, and the asynchronous task includes a resource package. The target master node executes the resource package through at least one first target working node and at least one second target working node, using the first processing resources of the at least one first target working node and the second processing resources of the at least one second target working node, so as to process the multiple contents.

[0105] The meta-service platform 304 is used to record the status information of the target task, such as the cluster address of the target cluster 303 , the cluster resources of the target cluster 303 , the task identifier of the asynchronous task, the execution progress of the asynchronous task, and so on.

[0106] See also Figure 4 , Figure 4 A schematic diagram of another content processing system provided in an embodiment of the present application. Figure 4 As shown, the content processing system includes a task execution system 401 and a task management system 402 .

[0107] Optionally, the task execution system 401 can be abstracted into multiple layers, such as Figure 4 The application layer 403, core layer 404 and hardware layer 405 are shown. The hardware layer 405 is a layer abstracted from the hardware resources in the k8s cluster, including storage devices, video memory, and heterogeneous processing resources, where the heterogeneous processing resources include CPU and GPU; the core layer 404 is a node or virtual machine in the k8s cluster, etc.; the application layer 403 is a system layer that provides applications for users. For example, the applications that can be provided to users include model batch reasoning, data batch calculation, and content batch retracement. Each upper-layer application implements the corresponding upper-layer application through a software development kit (SDK).

[0108] Optionally, the task management system 402 includes the above-mentioned management platform 301, meta-service platform 304 and task warehouse 305. The management platform provides a RESTful API interface, so that users can submit tasks, view tasks, stop tasks, etc. through the RESTful API interface.

[0109] based on Figure 4 In the content processing system shown, the management platform 301 obtains the target task through the Restful API interface; then, the management platform 301 downloads the resource package corresponding to the target task from the task warehouse 305 based on the task identifier of the target task. Then, the management platform 301 submits the asynchronous task to the core layer 404 through the SDK, wherein the asynchronous task includes the resource package; then, the core layer 404 processes multiple contents by calling the heterogeneous resources in the hardware layer 405, thereby realizing the upper layer application, that is, realizing content retrieval of batches of user-generated content.

[0110] See also Figure 5 , Figure 5 A flow chart of a content processing method provided in an embodiment of the present application. The method is applied to the above-mentioned content processing system. The method includes but is not limited to the following steps:

[0111] 501: The management platform obtains a target task, where the target task is used to process multiple contents.

[0112] First, it is explained that the multiple contents involved in this embodiment are described by taking multiple user-generated contents and the target task being a retrace task as an example.

[0113] Exemplarily, the management platform provides a Restful API interface, through which users can submit target tasks to the management platform. Specifically, a corresponding task identifier can be set for each target task. For example, the task identifier of the target task can be the identifier of the new version of the model used by the target task. For example, the task identifier is composed of the name and version number of the new version of the model. In this way, the user can upload a processing request through the Restful API interface, wherein the processing request includes the task identifier, and the processing request is used to request the management platform to execute the target task corresponding to the task identifier.

[0114] It should be noted that for each target task, the present application first packages and compresses the task code of the target task and the configuration file of the target task to obtain the resource package of the target task, and then uploads the resource package to the task warehouse. Among them, the task code is the entire functional code of the target task, and the configuration file can be various files required in the process of executing the task code. Each file records some configuration parameters required to execute the task code. For example, the configuration file includes a yaml file for creating a ray cluster, and the yaml file records the cluster resources required to create the ray cluster. For example, the configuration file includes an executable file of a new version of the model, and the executable file includes the script code of the new version of the model.

[0115] 502: The management platform downloads a resource package corresponding to the target task from the task repository.

[0116] Based on the task identifier of the target task, the management platform downloads the resource package corresponding to the task identifier from the task warehouse.

[0117] 503: The management platform submits an asynchronous task to the target master node, wherein the asynchronous task includes the resource package.

[0118] The target master node is a node in a target cluster, and the target cluster also includes at least one first target working node having a first processing resource and at least one second target working node having a second processing resource.

[0119] It should be noted that before submitting an asynchronous task to the target master node, the target cluster is first created in the original cluster. This application mainly uses the original cluster as a k8s cluster and the target cluster as a ray cluster as an example for explanation.

[0120] First, it is explained that in order to process the content through the heterogeneous first processing resources and the second processing resources, the first processing resources and the second processing resources need to be deployed on the original cluster. Therefore, based on the functions and resources owned by the nodes, the nodes in the original cluster can be divided into the original master node, the first original working node with the first processing resource, and the second original working node with the second processing resource, wherein the number of the first original working node and the second original working node can be one or more, and this application does not limit this.

[0121] Furthermore, in order to create a target cluster, a cluster controller needs to be deployed in the original cluster in advance, corresponding to the ray cluster. This application takes the cluster controller as Ray Operator as an example for explanation.

[0122] Exemplarily, the management platform first parses the resource package to obtain the task code and configuration file of the target task, and then reads the cluster configuration file for creating the target cluster from the configuration file, wherein the cluster configuration file describes the cluster resources required to create the target cluster. Optionally, the cluster configuration file can be a yaml file.

[0123] Optionally, the cluster resources include but are not limited to: the number of nodes in the target cluster, the number of first target working nodes, the number of second target working nodes, defining the resource requirements of the Head node in the target cluster, defining the resource requirements of each first target working node in the target cluster, defining the resource requirements of each second target working node in the target cluster, and specifying the k8s API version to be used, etc.; wherein, the resource requirements of the Head node include but are not limited to: the number of CPUs and memory size of the Head node, etc.; the resource requirements of each first target working node include but are not limited to: the number of CPUs and memory size, etc.; the resource requirements of each second target working node include but are not limited to: the number of GPUs and memory size, etc.

[0124] Then, the management platform sends a cluster creation request to the original master node, where the cluster creation request includes the above cluster configuration file; the original master node receives the cluster creation request through the API Server. After receiving the cluster creation request, the API Server of the original master node stores the cluster creation request in Etcd and notifies the cluster controller to read the cluster creation request. The cluster controller reads the cluster creation request from Etcd, and then creates the above target cluster on the original cluster based on the cluster configuration file.

[0125] Exemplarily, the cluster controller parses the cluster configuration file to obtain the cluster resources, and schedules the minimum unit (Pod) deployed on the first original working node and the minimum unit (Pod) deployed on the second original working node based on the cluster resources to obtain the target cluster. Specifically, the cluster controller sends the cluster resources required for the target cluster to the Scheduler of the original master node through the API Server of the original master node, and then the Scheduler of the original master node schedules the minimum unit deployed on the first original working node and the minimum unit deployed on the second original working node based on the scheduling policy to obtain the target cluster, that is, the Pod deployed on the first original working node and the Pod deployed on the second original working node are scheduled to obtain the target cluster. Specifically, the Scheduler of the original master node schedules each Pod deployed to the first original work node according to the preset strategy based on the resource requirements of the Head node, and selects the optimal Pod corresponding to the Head node; based on the resource requirements of each first target work node, schedule each Pod deployed to the first original work node, and select the optimal Pod corresponding to each first target work node; for example, for a first target work node that requires 2 first processing resources, select Pods with at least 2 first processing resources from the first original work node, and then score these Pods, and match the Pod with the highest score to the optimal Pod corresponding to the first target work node. Based on the resource requirements of each second target work node, schedule each Pod deployed to the second original work node, and select the optimal Pod corresponding to each second target work node.

[0126] Finally, the Scheduler of the original master node returns the optimal Pod corresponding to the Head node, the optimal Pod corresponding to each first target working node, and the optimal Pod corresponding to each second target working node to the cluster controller through the API Server; the cluster controller writes the optimal Pod information to the CRD and updates the information in the CRD; in this way, the Kubelet of the original working node corresponding to the optimal Pod can obtain the optimal Pod information from the CRD, and divide the optimal Pod into corresponding nodes, thereby dividing the Head node, at least one first target working node, and at least one second target working node. It can be understood that after scheduling the Pod on the first original working node and the Pod on the second original working node, the Head node, at least one first target working node, and at least one second target working node that meet the requirements can be divided, so as to build a target cluster in the original cluster.

[0127] Furthermore, after creating the target cluster, the management platform submits an asynchronous task to the target master node in the target cluster, wherein the asynchronous task includes the resource package, so that the target cluster processes the multiple contents based on the resource package.

[0128] In one embodiment of the present application, after creating the target cluster, the cluster controller returns the cluster address of the target cluster to the management platform, so that the management platform can submit asynchronous tasks to the target master node through the cluster address. Exemplarily, the cluster controller stores the cluster address in the Etcd of the original master node, updates the state of the CRD in the Etcd, and then the API Server obtains the cluster address from the Etcd and returns the cluster address to the management platform.

[0129] Optionally, corresponding to the ray cluster, this application mainly uses the asynchronous task as a ray job as an example for explanation.

[0130] In one embodiment of the present application, before creating a target cluster, the management platform sends the cluster resources required by the target cluster to the meta-service platform (MetaServer), and the meta-service platform records the cluster resources required by the target task, so that for each target task, the meta-service platform records the cluster resources occupied by each target task. Accordingly, after receiving the cluster resources, the meta-service platform sends resource indication information to the management platform based on the remaining free resources of the original cluster, wherein the remaining free resources are determined by the meta-service platform based on the cluster resources occupied by the currently executing target task and all available cluster resources of the original cluster. Optionally, if the remaining idle resources meet the demand for the cluster resources, the resource indication information is used to indicate that the remaining idle cluster resources meet the above-mentioned cluster resources. Accordingly, the management platform sends the above-mentioned cluster creation request to the original master node in response to the resource indication information; optionally, if the remaining idle resources do not meet the demand for the above-mentioned cluster resources, for example, the remaining idle resources are less than the cluster resources, then the resource indication information is used to indicate that the remaining idle cluster resources of the original cluster do not meet the above-mentioned cluster resources. In response to the resource indication information, the management platform will continue to wait until the meta-service platform indicates that the remaining idle cluster resources of the original cluster meet the cluster resources, and then send the cluster creation request to the original master node.

[0131] In one implementation of the present application, the management platform may submit the cluster address to the meta-service platform for recording, so as to facilitate the subsequent management of the target cluster.

[0132] In one embodiment of the present application, after the management platform submits an asynchronous task to the target cluster, the target cluster will assign a task identifier to the asynchronous task and then return the task identifier of the asynchronous task to the management platform. The management platform will submit the task identifier of the asynchronous task to the meta-service platform for recording, so as to facilitate the management of the asynchronous task later.

[0133] In one embodiment of the present application, the management platform returns the cluster address of the target cluster and the task identifier of the asynchronous task to the user end, so that the user can manage the target task based on the cluster address and the task identifier of the asynchronous task.

[0134] 504: The target master node executes the resource package through the at least one first target working node and the at least one second target working node, using the first processing resources of the at least one first target working node and the second processing resources of the at least one second target working node, so as to process the multiple contents.

[0135] The above-mentioned target tasks include multiple task stages. Since using a model to process data generally includes five steps: data reading, data preprocessing, reasoning, data post-processing, and data writing, this application mainly uses the multiple stages including the data reading stage, data preprocessing stage, reasoning stage, data post-processing stage, and data writing stage as an example for explanation.

[0136] Exemplarily, the resource package includes a task code and a configuration file corresponding to the target task, and the target master node determines the processing resources required for each task stage based on the task code and the configuration file. Optionally, the configuration file includes multiple sub-configurations, wherein the multiple sub-configurations are configuration files required to execute the multiple task stages.

[0137] Specifically, the target master node understands the task code line by line, obtains the function of each line of code in the task code, determines the code segment corresponding to each task stage based on the function of each line of code, and then determines the processing resources required for each task stage based on the code segment corresponding to each task stage. Optionally, if the code segment specifies the processing resources required for the task stage, the processing resources required for the task stage are directly read from the code segment; if the code segment defines the processing resources required for the task stage by referencing a sub-configuration file, the sub-configuration file referenced by the code segment is obtained, and the processing resources required for the task stage are read from the sub-configuration file. For example, the target master node parses the resource package through the Driver process to obtain the processing resources required for each task stage.

[0138] The processing resources required for each task stage include, but are not limited to: the number of required working nodes, the number of required working processes, the number of CPUs or GPUs required for each working process, and so on.

[0139] Exemplarily, the target working node creates a working process for each task stage based on the processing resources required for each task stage, that is, based on the number of working processes required for the task stage, a corresponding number of working processes are created for the task stage, wherein the working process can be a stateful working process or a stateless working process. This application mainly illustrates the working process as a stateful working process and taking the working process as a ray actor as an example.

[0140] Exemplarily, the target master node determines the target working nodes corresponding to each task stage in the at least one first target working node and the at least one second target working node based on the processing resources required for each task stage, the processing resources of each working node in the at least one first target working node, and the processing resources of each working node in the at least one second target working node. Specifically, the scheduler of the target master node determines the working nodes that meet the requirements of each task stage based on the processing resources required for each task stage and the current processing resources of each working node, and uses the best working node among the working nodes that meet the requirements as the target working node for each task stage.

[0141] For example, the processing resources required for a task stage are 4 work processes, and each work process requires 0.5 CPU. In this way, the scheduler first selects candidate work nodes with 0.5 CPU idle resources from at least one first target work node, and then scores the candidate nodes, and selects the candidate node with the highest score as the target work node for the task stage. Then, the 4 work processes are scheduled to the target work node.

[0142] Then, the target master node schedules the work process of each task stage to the target work node corresponding to each task stage, and sends the task code and the configuration file to the target work node. The target work node can temporarily store the task code and the configuration file in the Object Store of the target work node. Optionally, for each task stage, the target master node can parse the code segment corresponding to each task stage from the task code and obtain the sub-configuration file corresponding to each task stage from the configuration file. In this way, the target work node can only transmit the code segment and sub-configuration file corresponding to each task stage to the target work node corresponding to each task stage to reduce data transmission.

[0143] Finally, the work process of the target work node corresponding to each task stage utilizes the processing resources of the target work node to execute each task stage based on the task code and the configuration file to process the multiple contents, that is, to rewind the multiple user-generated contents, that is, to re-understand the multiple user-generated contents.

[0144] Exemplarily, if the task stage is a data reading stage, the processing resources required for the task stage include a first processing resource, and the target working node corresponding to the data reading stage is a working node among the at least one target working node. However, as to which working nodes among the at least one target working node are used to execute the data reading stage, the scheduler will actually schedule according to the processing resources required by the task stage; then, the working process on the target working node corresponding to the data reading stage parses the task code to obtain a reading path for multiple contents, that is, the reading path is described in the task code, and then the working process uses the first processing resource of the target working node to read the multiple contents based on the reading path. Specifically, an access key is obtained from the task code, and a storage system is connected to the access key, and then based on the reading path, the bit stream corresponding to the multiple contents is read from the storage system, and the bit stream is obtained by serializing the contents; then, the bit stream of each content is deserialized to obtain each content.

[0145] It should be noted that if there are multiple work processes in the data reading stage, each work process can use the first processing resource of the target work node where the work process is located to read the content separately, thereby realizing batch reading of multiple contents.

[0146] In one embodiment of the present application, since when the content is serialized, in addition to serializing the actual content, the related information of each content, such as the author of the content, interactive information, etc., is also serialized, therefore, when the storage system stores each content, what is stored is the entire serialized bit stream of the content, and only a part of the bit stream is the bit stream that is actually obtained by serializing the content, however, task processing only needs to process the actual content, and other information does not need to be processed, and for each business scenario, the position of the content in the bit stream is related to the scenario, therefore, for each business scenario, the position of the content in the bit stream under the business scenario is pre-determined, and an index of the content under each business scenario is constructed, wherein the index is used to indicate the position of the content in the bit stream. In this way, for the above-mentioned target task, the business scenario corresponding to the target task can be obtained; then, the index corresponding to the business scenario is obtained, wherein the index is used to indicate the position of the above-mentioned multiple contents in the bit stream; then, when reading each content, the work process on the target working node corresponding to the data reading stage uses the first processing resource of the target working node to read the bit stream indicated by the index from the bit stream corresponding to each content, obtain the target bit stream corresponding to each content, and deserialize the target bit stream corresponding to each content to obtain the multiple contents. In this way, in the data reading stage, only the target bit stream related to the content is read, so there is no need to read too many invalid bit streams, and only the target bit stream needs to be deserialized, and there is no need to deserialize the invalid bit stream, thereby improving the data reading efficiency and then improving the content processing efficiency.

[0147] The above business scenarios include but are not limited to: search scenarios, recommendation scenarios, etc.

[0148] Optionally, the content of this application can be any type of storage object, such as S3 storage object, OSS storage object, etc.

[0149] Exemplarily, if the task stage is a data preprocessing stage, the processing resources required for the task stage include a first processing resource, and the target working node is a working node in the at least one first target working node. Then, the working process dispatched to the target working node parses the task code to obtain the first processing logic of the data preprocessing stage, and obtains the first processing parameter of the data preprocessing stage based on the configuration file. Specifically, the working process parses the task code, determines the sub-configuration file referenced by the task stage, then reads the sub-configuration file from the configuration file, and parses the sub-configuration file to obtain the first processing parameter. For example, if the data preprocessing stage is embedded processing, the first processing logic is the embedded logic, and the first processing parameter is an embedded table. Finally, the working process uses the first processing resource of the target working node to preprocess the multiple contents according to the processing logic and the first processing parameter.

[0150] It should be noted that if there are multiple work processes in the data preprocessing stage, each work process can use the first processing resource of the target work node where the work process is located to preprocess the content in parallel, thereby achieving batch preprocessing of multiple contents.

[0151] Exemplarily, if the task stage is the reasoning stage, the processing resources required for the task stage include the second processing resources, and the target working node is a working node in the at least one second target working node. Then, the working process scheduled to the target working node downloads the executable file of the new version model based on the task code and the configuration file. Specifically, the working process parses the task code to create a model downloader object; and obtains a sub-configuration file corresponding to the reasoning stage from the configuration file, and obtains download parameters corresponding to the new version model from the sub-configuration file, wherein the download parameters include but are not limited to: download path, authentication information, model name, model version number, etc. Then, the working process on the target working node downloads the executable file of the new version model based on the download parameters through the model downloader object. Then, the working process uses the preprocessed multiple contents as input data of the new version model, and uses the second processing resource of the target working node to execute the executable file, thereby reasoning the multiple contents and obtaining multiple reasoning results.

[0152] It should be noted that if multiple work processes are deployed in the reasoning stage, each work process can use the second processing resource of the target work node where the work process is located to reason about the preprocessed content in parallel, thereby realizing batch reasoning of multiple contents.

[0153] Exemplarily, if the task stage is a data post-processing stage, the processing resources required for the task stage include first processing resources, and the target working node is a working node in the at least one first target working node. Then, the working process of the target working node parses the task code to obtain the second processing logic of the data post-processing stage, that is, the task code is functionally analyzed to parse out the second processing logic of post-processing; and the second processing parameters of the data post-processing stage are obtained based on the configuration file, that is, the sub-configuration file corresponding to the post-processing stage is obtained from the configuration file, and the sub-configuration file is parsed to obtain the second processing parameters of the post-processing stage. Finally, the working process uses the first processing resource of the target working node to post-process the multiple inference results according to the second processing logic and the second processing parameters.

[0154] It should be noted that if multiple work processes are deployed in the data post-processing stage, each work process can use the first processing resource of the target work node where the work process is located to post-process the inference results in parallel, so as to achieve batch post-processing of multiple inference results and obtain multiple post-processed inference results.

[0155] Exemplarily, if the task stage is a data writing stage, the processing resources required for the task stage include a first processing resource, and the target work node is a work node in the at least one first target work node. Then, the work process deployed to the target work node parses the task code to obtain the writing path of the multiple contents, and uses the first processing resource of the target work node to write the multiple inference results after post-processing to the storage location indicated by the writing path based on the writing path.

[0156] It should be noted that if multiple work processes are deployed in the data writing stage, each work process can use the first processing resource of the target work node where the work process is located to write the post-processed reasoning results in parallel, thereby realizing batch writing of multiple post-processed reasoning results.

[0157] It should be noted that if in the data reading stage, only the target bit stream indicated by the index is read for each content, then when the post-processed reasoning results are written, the post-processed reasoning results will be serialized to obtain the bit stream to be written, and then the bit stream to be written will be written, that is, the target bit stream is replaced by the bit stream to be written, thereby completing the data writing.

[0158] It can be understood that since the CPU is generally used to process complex logical and sequential tasks, and for the data reading stage, data preprocessing stage, data post-processing stage and data writing stage, a large number of I / O operations, data cleaning, format conversion and other tasks are involved, these stages generally require logical processing and sequential execution, so the present application uses the first processing resource to execute these stages, which can make full use of the capabilities of the CPU, execute these stages more efficiently, and improve processing efficiency; however, the GPU has a large number of cores and is very suitable for parallel processing of simple but repetitive tasks. For the model reasoning stage, a large number of matrix operations are involved. These tasks can be decomposed into many small, independent calculation processes, which will involve a large number of parallel calculation processes. Therefore, the present application uses the second processing resource for model reasoning, which can make full use of the capabilities of the GPU, execute the model reasoning stage more efficiently, and improve processing efficiency.

[0159] In one embodiment of the present application, when the number of work processes in any task stage is multiple, the number of target nodes corresponding to any task stage is multiple, that is, the multiple work processes of each task stage are scheduled to different target work nodes; when different task stages have dependencies and the types of processing resources required are the same, the target work nodes corresponding to the different task stages are the same, that is, the work processes that have dependencies and use the same type of processing resources are scheduled to the same work node.

[0160] It should be noted that the number of target working nodes in each task stage can be one or more, which is scheduled by the scheduler according to actual needs. However, in actual applications, the scheduler will try to schedule multiple working processes of the same task stage to different working nodes for execution. For example, the working processes of the control task stage correspond to the number of target nodes one by one, so that different working nodes can execute the task stage in parallel, so that the task stage can be executed in parallel for multiple contents, so that multiple contents can be processed in batches in parallel, and will not be restricted by the resources or network environment of a certain node, so as to improve processing efficiency. For example, for the data reading stage, 4 working processes are created, so that the 4 working processes are scheduled to execute on four nodes, so that the 4 working processes on the four nodes can read the multiple contents in parallel, so that the working node on a certain node can send the user-generated content to the working process of the data preprocessing stage for processing after reading a content. Then there will be no interference and competition between the four working processes, and the content can be read in batches in parallel.

[0161] Furthermore, the above-mentioned ray cluster is used to process the multiple user generated content, and the work processes of each task stage will not affect each other. In this way, when processing multiple contents, if a task stage of a certain content is completed, and the task stage of other contents has not been completed, the work process of the next task stage can be used to continue to execute the next task stage of the content, without waiting for all the multiple contents to complete the task stage before executing the next task stage. In this way, the processing processes of the content will not restrict and affect each other, thereby improving processing efficiency.

[0162] For example, some content has completed the data preprocessing stage, while some content has not yet completed the data preprocessing stage. For those contents that have completed the data preprocessing stage, they can be transmitted to the work process of the reasoning stage for reasoning on these contents. There is no need to wait for all the content to complete the data preprocessing stage before reasoning on the content.

[0163] Furthermore, for task stages that have dependencies and require the same type of processing resources, for example, if they all require the first processing resource, the work processes of these task stages can be scheduled to the same work node, so that after the previous task stage is completed, the next stage task can be directly executed on the work node, which does not require data transmission between nodes, but only requires data transmission between node processes, thereby avoiding data transmission across nodes and improving processing efficiency. For example, for the data preprocessing stage that depends on the data reading and writing stage, and both task stages require the first processing resource, the work process of the data preprocessing stage and the work process of the data reading and writing stage can be scheduled to the same work node as much as possible, so that when the engineering process corresponding to the data reading and writing stage scheduled to the work node completes the data reading and writing, the engineering process corresponding to the data preprocessing stage scheduled to the work node can obtain the read and written content for preprocessing, so that the two work processes only need to communicate within the work node, without the need for cross-node transmission, thereby improving processing efficiency.

[0164] For example, if Figure 6As shown in the figure, two working processes are deployed for the data reading (Read) stage, three working processes are deployed for the data preprocessing (Preprocess) stage, four working processes are deployed for the inference (Inference) stage, three working processes are deployed for the data postprocessing (Postprocess) stage, and two working processes are deployed for the data writing (Write) stage. Then, the two working processes in the data reading stage and the three working processes in the data preprocessing stage are scheduled to the same working node. Since the resources required by these working processes are CPU resources, this working node can be called CPUNode, and then the two working processes in the data reading stage can use the first processing resource of the CPU Node to read the content; the three working processes in the data preprocessing stage communicate within the CPU Node, obtain the content from the two working processes in the data reading stage, and use the first processing resource of the CPU Node to preprocess the read content; the four working processes in the inference stage are deployed to one node. Since the resources required for the inference stage are GPU resources, this node can be called GPU Node, so that the four working processes can use GPU The second processing resource of the Node performs reasoning on the preprocessed content to obtain the reasoning result. The three work processes in the data post-processing stage and the two work processes in the data writing stage are deployed on the same work node. Similarly, the resources required by these work processes are CPU resources, so this work node can be called a CPU Node. In this way, the three work processes in the data post-processing stage can use the first processing resource of the CPU Node to post-process the reasoning result and obtain the post-processed reasoning result. Finally, the two work processes in the data writing stage use the first processing resource of the CPU Node to write the post-processed reasoning result.

[0165] It can be seen that in an embodiment of the present application, when multiple contents need to be processed, the user can submit a target task to the management platform; then the management platform downloads the resource package corresponding to the target task from the task warehouse, and then the management platform submits the resource package to the target cluster in the form of an asynchronous task; since the target cluster has a first target working node with a first processing resource and a second target working node with a second processing resource, the first processing resource and the second processing resource in the target cluster can be used to execute the resource package, so that the logical processing power of the CPU and the parallel power of the GPU can be used to process multiple contents, thereby achieving efficient content processing.

[0166] In one embodiment of the present application, the target master node will count the execution progress of the target task during the execution of the target task, and send the execution progress to the meta-service platform for recording. Accordingly, the user can send a query request to the management platform through the user terminal, wherein the query request includes the cluster address and the task identifier of the asynchronous task. For example, the user terminal provides a query entry, and the user can enter the cluster address and the task identifier of the asynchronous task through the query entry, and generate a query request based on the cluster address and the task identifier of the asynchronous task. Accordingly, the management platform submits the query request to the meta-service platform, and the meta-service platform queries the task progress of the target task corresponding to the cluster address and the task identifier of the asynchronous task based on the query request, and returns the task progress to the management platform; the management platform returns the task progress to the user terminal.

[0167] In one embodiment of the present application, when the target master node determines that the asynchronous task has ended, it can send a status notification to the management platform, and the status notification is used to indicate the end of the asynchronous task. After receiving the status notification, the management platform can send information that the asynchronous task has been completed to the meta-service platform, so that the meta-service platform can record the end of the asynchronous task and facilitate the recording of resources. Accordingly, the management platform will request the target master node to delete the asynchronous task. After receiving a successful deletion response, the management platform will also request the cluster controller to delete the target cluster.

[0168] See also Figure 7 , Figure 7 A flowchart of a content rewinding method provided in an embodiment of the present application. The method is applied to the above-mentioned content processing system. The implementation of each step in this embodiment can refer to the above-mentioned implementation, and will not be described again here; the method includes but is not limited to the contents in the following steps:

[0169] 701: The user terminal submits a scan back task to the management platform, wherein the scan back task is used to scan back multiple user-generated contents.

[0170] 702: The management platform downloads a resource package corresponding to the retrace task from the task repository based on the task identifier of the retrace task.

[0171] 703: The management platform parses the resource package to obtain cluster resources required to create a target cluster.

[0172] 704: The management platform registers the cluster resources with the meta-service platform.

[0173] 705: The management platform sends a cluster creation request to the k8s cluster, where the cluster creation request includes cluster resources.

[0174] 706: The k8s cluster responds to the cluster creation request and creates a ray cluster on the k8s cluster.

[0175] 707: The k8s cluster returns the cluster address of the ray cluster to the management platform.

[0176] 708: The management platform registers the cluster address of the ray cluster with the meta-service platform.

[0177] 709: The management platform submits a ray job to the ray cluster, wherein the ray job includes a resource package and is used to scan back multiple user-generated contents.

[0178] 710: The ray cluster initializes the ray job.

[0179] 711: The ray cluster returns the task identifier of the ray job to the management platform.

[0180] 712: ray cluster executes ray job.

[0181] 713: The ray cluster submits the execution progress of the ray job to the meta-service platform.

[0182] 714: The management platform returns the cluster address of the ray cluster and the task identifier of the ray job to the user end.

[0183] 715: The user terminal sends a query request to the management platform, where the query request includes the cluster address of the ray cluster and the task identifier of the ray job.

[0184] 716: The management platform sends a query request to the meta-service platform.

[0185] 717: The meta-service platform sends the execution progress of the ray job to the management platform, wherein the execution progress of the ray job is obtained by querying the meta-service platform based on the cluster address of the ray cluster and the task identifier of the ray job.

[0186] 718: The management platform sends the execution progress of the ray job to the user end.

[0187] 719: In response to the completion of the ray job execution, the ray cluster sends a status notification to the management platform, where the status notification is used to indicate the completion of the ray job execution.

[0188] 720: The management platform registers the completion of the ray job execution with the meta-service platform.

[0189] 721: The management platform requests the ray cluster to delete the ray job.

[0190] 722: The ray cluster sends a response to the management platform indicating that the ray job has been successfully deleted.

[0191] 723: The management platform requests the k8s cluster to delete the ray cluster.

[0192] 724: The k8s cluster sends a response to the management platform indicating that the ray cluster has been successfully deleted.

[0193] See also Figure 8 , Figure 8 A schematic diagram of a node provided in an embodiment of the present application. The node 800 includes: an acquisition module 801 and a processing module 802;

[0194] The acquisition module 801 is used to receive an asynchronous task from a management platform, wherein the asynchronous task includes a resource package of a target task, the resource package is downloaded by the management platform from a task repository, and the target task is used to process multiple contents;

[0195] The processing module 802 is used to execute the resource package through the at least one first target working node and the at least one second target working node, using the first processing resources of the at least one first target working node and the second processing resources of the at least one second target working node, so as to process the multiple contents.

[0196] In one embodiment of the present application, the resource package includes a task code and a configuration file of the target task; the target task includes multiple task stages; in terms of executing the resource package through the at least one first target working node and the at least one second target working node, using the first processing resource of the at least one first target working node and the second processing resource of the at least one second target working node to process the multiple contents, the processing module 802 is specifically used to:

[0197] Determining the processing resources required for each task stage based on the task code and the configuration file;

[0198] Create a worker process for each task stage based on the processing resources required for each task stage;

[0199] Based on the processing resources required for each task stage, the processing resources of each working node in the at least one first target working node, and the processing resources of each working node in the at least one second target working node, determine the target working node corresponding to each task stage in the at least one first target working node and the at least one second target working node;

[0200] Scheduling the work process of each task stage to the target work node corresponding to each task stage, and sending the task code and the configuration file to the target work node;

[0201] The work process of the target work node corresponding to each task stage utilizes the processing resources of the target work node to execute each task stage based on the task code and the configuration file to process the multiple contents.

[0202] In one embodiment of the present application, the multiple task stages include a data reading stage, a data preprocessing stage, an inference stage, a data post-processing stage, and a data writing stage;

[0203] If the task stage is a data reading stage, the processing resources required for the task stage include first processing resources, and the target working node is a working node in the at least one first target working node;

[0204] If the task stage is a data preprocessing stage, the processing resources required for the task stage include first processing resources, and the target working node is a working node in the at least one first target working node;

[0205] If the task stage is the reasoning stage, the processing resources required for the task stage include the second processing resources, and the target working node is a working node in the at least one second target working node;

[0206] If the task stage is a data post-processing stage, the processing resources required for the task stage include first processing resources, and the target working node is a working node in the at least one first target working node;

[0207] If the task stage is a data writing stage, the processing resources required for the task stage include first processing resources, and the target working node is a working node among the at least one first target working node.

[0208] See also Fig. 9 , Fig. 9 A schematic diagram of a management platform provided in an embodiment of the present application. The management platform 900 includes: an acquisition module 901 and a processing module 902;

[0209] An acquisition module 901 is used to acquire a target task, wherein the target task is used to process multiple contents;

[0210] Processing module 902, used to download a resource package corresponding to the target task from a task repository;

[0211] Submit an asynchronous task to a target master node, wherein the target master node is a node in a target cluster, and the target cluster also includes at least one first target work node having a first processing resource and at least one second target work node having a second processing resource, and the asynchronous task includes the resource package, and the asynchronous task is used to request the target master node to execute the resource package through the at least one first target work node and the at least one second target work node, using the first processing resources of the at least one first target work node and the second processing resources of the at least one second target work node, so as to process the multiple contents.

[0212] In one embodiment of the present application, the target cluster is an upper cluster of the original cluster; the original cluster includes an original master node and multiple first original working nodes, and the original master node is deployed with a cluster controller; before submitting the asynchronous task to the target master node, the processing module 902 is further used to:

[0213] Obtaining a cluster address of the target cluster from the cluster controller;

[0214] Submit the asynchronous task to the target master node via the cluster address.

[0215] In one embodiment of the present application, before sending the cluster creation request to the original master node, the processing module 902 is further configured to:

[0216] Sending the cluster resources to a meta-service platform;

[0217] Acquire resource indication information from the meta-service platform;

[0218] If the resource indication information indicates that the remaining idle cluster resources of the original cluster satisfy the cluster resources, sending the cluster creation request to the original master node;

[0219] If the resource indication information indicates that the remaining idle cluster resources of the original cluster do not satisfy the cluster resources, continue to wait until the meta-service platform indicates that the remaining idle cluster resources of the original cluster satisfy the cluster resources, and then send the cluster creation request to the original master node.

[0220] In one embodiment of the present application, the processing module 902 is further configured to:

[0221] Receiving a query request for the asynchronous task from a user terminal;

[0222] Sending the query request to the meta-service platform;

[0223] A query response is received from the meta-service platform, wherein the query response includes a task progress of the asynchronous task, the task progress is queried by the meta-service platform based on the query identifier, and the task progress is submitted to the meta-service platform by the target master node.

[0224] See also Fig.10 , Fig.10 A schematic diagram of a content processing system provided in an embodiment of the present application. The content processing system 1000 includes a management platform 1001 and a target cluster 1002; the target cluster 1002 includes a target master node 1003, at least one first target working node 1004 having a first processing resource, and at least one second target working node 1005 having a second processing resource;

[0225] The management platform 1001 is used to obtain a target task, wherein the target task includes a task identifier, and the target task is used to process multiple contents;

[0226] The management platform 1001 is used to download the resource package of the target task from the task warehouse based on the task identifier;

[0227] A management platform 1001 is used to submit an asynchronous task to the target master node, wherein the asynchronous task includes the resource package;

[0228] The target master node 1003 is used to execute the resource package through the at least one first target working node 1004 and the at least one second target working node 1005, utilizing the first processing resources of the at least one first target working node and the second processing resources of the at least one second target working node, so as to process the multiple contents.

[0229] See also Fig.11 , Fig.11 Schematic diagram of an electronic device provided in an embodiment of the present application. Electronic device 1100 may be the above-mentioned target master node or management platform. Electronic device 1100 includes memory 1101, processor 1102, communication interface 1103 and bus 1104. Among them, memory 1101, processor 1102, communication interface 1103 realize communication connection with each other through bus 1104.

[0230] The memory 1101 may be a read-only memory (ROM), a static storage device, a dynamic storage device or a random access memory (RAM). The memory 1101 may store programs; when the electronic device 1100 is the above-mentioned target master node, when the program stored in the memory 1101 is executed by the processor 1102, the processor 1102 and the communication interface 1103 cooperate with each other to execute the various steps executed by the above-mentioned target master node in the embodiment of the present application; when the electronic device 1100 is the above-mentioned management platform, when the program stored in the memory 1101 is executed by the processor 1102, the processor 1102 and the communication interface 1103 cooperate with each other to execute the various steps executed by the above-mentioned management platform in the embodiment of the present application.

[0231] The processor 1102 can adopt a general central processing unit (CPU), a microprocessor, an application specific integrated circuit (ASIC), a graphics processing unit (GPU) or one or more integrated circuits to execute relevant programs to implement the content processing method of the embodiment of the present application.

[0232] The processor 1102 may also be an integrated circuit chip with signal processing capabilities. In the implementation process, each step in the data stream transmission method of the present application may be completed by an integrated logic circuit of hardware or software instructions in the processor 1102. The above-mentioned processor 1102 may also be a general-purpose processor, a digital signal processor (DigitalSignal Processing, DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (Field ProgrammableGate Array, FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The disclosed methods, steps and logic block diagrams in the embodiments of the present application may be implemented or executed. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc. The steps of the method disclosed in conjunction with the embodiments of the present application may be directly embodied as a hardware decoding processor for execution, or may be executed by a combination of hardware and software modules in a decoding processor. The software module may be located in a mature storage medium in the art such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory 1101, and the processor 1102 reads the information in the memory 1101 to execute each step in the content processing method of the present application.

[0233] The communication interface 1103 can be a transceiver or other transceiver-like device to realize communication between the electronic device 1100 and other devices or a communication network; the communication interface 1103 can also be an input-output interface to realize data transmission between the electronic device 1100 and input-output devices, wherein the input-output devices include but are not limited to keyboards, mice, display screens, USB flash drives, and hard disks.

[0234] The bus 1104 may include a path for transmitting information between various components of the device electronic device 1100 (eg, the memory 1101 , the processor 1102 , and the communication interface 1103 ).

[0235] It should be noted that although Fig.11 The electronic device 1100 shown only shows a memory, a processor, and a communication interface, but in the specific implementation process, those skilled in the art should understand that the electronic device 1100 also includes other devices necessary for normal operation. At the same time, according to specific needs, those skilled in the art should understand that the electronic device 1100 may also include hardware devices for implementing other additional functions. In addition, those skilled in the art should understand that the electronic device 1100 may also only include the devices necessary to implement the embodiments of the present application, and does not necessarily include Fig.11 All devices shown in .

[0236] An embodiment of the present application further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and the computer program is executed by a processor to implement part or all of the steps of any content processing method recorded in the above method embodiments.

[0237] An embodiment of the present application also provides a computer program product, which includes a non-transitory computer-readable storage medium storing a computer program, and the computer program is operable to cause a computer to execute part or all of the steps of any content processing method recorded in the above method embodiments.

[0238] It should be noted that, for the aforementioned method embodiments, for the sake of simplicity, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the present application is not limited by the described order of actions, because according to the present application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all optional embodiments, and the actions and modules involved are not necessarily required by the present application.

[0239] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0240] In the several embodiments provided in the present application, it should be understood that the disclosed device can be implemented in other ways. For example, the device embodiments described above are only schematic, such as the division of the units, which is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, and the indirect coupling or communication connection of the device or unit can be electrical or other forms.

[0241] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0242] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above integrated unit may be implemented in the form of hardware or in the form of a software program module.

[0243] If the integrated unit is implemented in the form of a software program module and sold or used as an independent product, it can be stored in a computer-readable memory. Based on this understanding, the technical solution of the present application, in essence, or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, which is stored in a memory and includes several instructions for a computer device (which can be a personal computer, a server or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned memory includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, disk or optical disk and other media that can store program codes.

[0244] A person skilled in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by instructing related hardware through a program, and the program can be stored in a computer-readable memory, and the memory can include: a flash drive, a read-only memory (English: Read-Only Memory, abbreviated as: ROM), a random access memory (English: Random Access Memory, abbreviated as: RAM), a magnetic disk or an optical disk, etc.

[0245] The embodiments of the present application are introduced in detail above. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method and core idea of ​​the present application. At the same time, for general technical personnel in this field, according to the idea of ​​the present application, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. A content processing method, characterized in that: The method is applied to a target master node, the target master node is a master node in a target cluster, the target cluster also includes at least one first target working node with a first processing resource and at least one second target working node with a second processing resource; the method includes: Receiving an asynchronous task from a management platform, wherein the asynchronous task includes a resource package of a target task, the resource package is downloaded by the management platform from a task repository, and the target task is used to process a plurality of contents; The resource package is executed through the at least one first target working node and the at least one second target working node, utilizing the first processing resources of the at least one first target working node and the second processing resources of the at least one second target working node, so as to process the multiple contents.

2. The method according to claim 1, characterized in that The resource package includes a task code and a configuration file of the target task; the target task includes a plurality of task stages; the resource package is executed by the at least one first target working node and the at least one second target working node, using a first processing resource of the at least one first target working node and a second processing resource of the at least one second target working node to process the plurality of contents, including: Determining the processing resources required for each task stage based on the task code and the configuration file; Create a worker process for each task stage based on the processing resources required for each task stage; Based on the processing resources required for each task stage, the processing resources of each working node in the at least one first target working node, and the processing resources of each working node in the at least one second target working node, determine the target working node corresponding to each task stage in the at least one first target working node and the at least one second target working node; Scheduling the work process of each task stage to the target work node corresponding to each task stage, and sending the task code and the configuration file to the target work node; The work process of the target work node corresponding to each task stage utilizes the processing resources of the target work node to execute each task stage based on the task code and the configuration file to process the multiple contents.

3. The method according to claim 2, characterized in that The multiple task stages include a data reading stage, a data preprocessing stage, an inference stage, a data post-processing stage, and a data writing stage; If the task stage is a data reading stage, the processing resources required for the task stage include first processing resources, and the target working node is a working node in the at least one first target working node; If the task stage is a data preprocessing stage, the processing resources required for the task stage include first processing resources, and the target working node is a working node in the at least one first target working node; If the task stage is the reasoning stage, the processing resources required for the task stage include the second processing resources, and the target working node is a working node in the at least one second target working node; If the task stage is a data post-processing stage, the processing resources required for the task stage include first processing resources, and the target working node is a working node in the at least one first target working node; If the task stage is a data writing stage, the processing resources required for the task stage include first processing resources, and the target working node is a working node among the at least one first target working node.

4. The method according to claim 3, characterized in that: The working process of the target working node corresponding to each task stage uses the processing resources of the target working node to execute each task stage based on the task code and the configuration file to process the multiple contents, including: If the task stage is a data reading stage, the work process of the target work node parses the task code to obtain a reading path of the multiple contents, and uses the first processing resource of the target work node to read the multiple contents based on the reading path; If the task stage is a data preprocessing stage, the work process of the target work node parses the task code to obtain a first processing logic of the data preprocessing stage, obtains a first processing parameter of the data preprocessing stage based on the configuration file, and uses a first processing resource of the target work node to preprocess the multiple contents according to the processing logic and the first processing parameter; If the task stage is the reasoning stage, the working process of the target working node downloads the executable file of the new version model based on the task code and the configuration file, and uses the preprocessed multiple contents as input data of the new version model, and uses the second processing resource of the target working node to execute the executable file to reason on the preprocessed multiple contents to obtain multiple reasoning results; If the task stage is a data post-processing stage, the work process of the target work node parses the task code to obtain a second processing logic of the data post-processing stage, obtains a second processing parameter of the data post-processing stage based on the configuration file, and uses the first processing resource of the target work node to post-process the multiple inference results according to the second processing logic and the second processing parameter; If the task stage is the data writing stage, the working process of the target working node parses the task code to obtain the writing path of the multiple contents, and uses the first processing resource of the target working node to write the post-processed multiple inference results to the storage location indicated by the writing path based on the writing path.

5. The method according to claim 4, characterized in that The using the first processing resource of the target working node to read the plurality of contents based on the reading path includes: Obtaining the business scenario corresponding to the target task; Obtaining an index corresponding to the business scenario; Utilizing the first processing resource of the target working node, based on the reading path, the bit stream indicated by the index is read from the bit stream corresponding to each content to obtain the target bit stream corresponding to each content, and the target bit stream corresponding to each content is deserialized to obtain the multiple contents.

6. The method according to any one of claims 1 to 5, characterized in that: The target cluster is created based on the original cluster, wherein the original cluster is deployed with an original master node, a first original working node with a first processing resource, a second original working node with a second processing resource, and a cluster controller; the method further includes: The original master node receives a cluster creation request from the management platform, wherein the cluster creation request includes a cluster configuration file, the cluster configuration file is used to describe cluster resources for creating the target cluster, and the cluster configuration file is obtained by the management platform parsing the resource package; The cluster controller obtains the cluster creation request from the original master node, and in response to the cluster creation request, schedules the minimum unit deployed to the first original working node and the minimum unit deployed to the second original working node based on the cluster resources to obtain the target cluster.

7. A content processing method, characterized in that: The method is applied to a management platform, and the method comprises: Acquire a target task, wherein the target task is used to process multiple contents; Downloading a resource package corresponding to the target task from a task repository; Submit an asynchronous task to a target master node, wherein the target master node is a node in a target cluster, and the target cluster also includes at least one first target work node having a first processing resource and at least one second target work node having a second processing resource, and the asynchronous task includes the resource package, and the asynchronous task is used to request the target master node to execute the resource package through the at least one first target work node and the at least one second target work node, using the first processing resources of the at least one first target work node and the second processing resources of the at least one second target work node, so as to process the multiple contents.

8. The method according to claim 7, characterized in that The target cluster is an upper cluster of the original cluster; the original cluster includes an original master node and a plurality of first original working nodes, and the original master node is deployed with a cluster controller; Before submitting the asynchronous task to the target master node, the method further includes: Parsing the resource package to obtain the cluster resources required to execute the target task; A cluster creation request is sent to the original master node, wherein the cluster creation request includes the cluster resources, so that the cluster controller creates the target cluster on the original cluster based on the cluster resources.

9. The method according to claim 7 or 8, characterized in that: Before submitting the asynchronous task to the target master node, the method further includes: Obtaining a cluster address of the target cluster from the cluster controller; Submit the asynchronous task to the target master node via the cluster address.

10. The method according to claim 8 or 9, characterized in that: Before sending the cluster creation request to the original master node, the method further includes: Sending the cluster resources to a meta-service platform; Acquire resource indication information from the meta-service platform; If the resource indication information indicates that the remaining idle cluster resources of the original cluster satisfy the cluster resources, sending the cluster creation request to the original master node; If the resource indication information indicates that the remaining idle cluster resources of the original cluster do not satisfy the cluster resources, continue to wait until the meta-service platform indicates that the remaining idle cluster resources of the original cluster satisfy the cluster resources, and then send the cluster creation request to the original master node.

11. The method according to any one of claims 7 to 10, characterized in that: The method further comprises: Receiving a query request for the asynchronous task from a user terminal; Sending the query request to the meta-service platform; receiving a query response from the meta-service platform, wherein the query response includes a task progress of the asynchronous task, the task progress is queried by the meta-service platform based on the query identifier, and the task progress is submitted to the meta-service platform by the target master node; Return the task progress to the user terminal.

12. A content processing method, characterized in that: Applied to a content processing system, the content processing system comprises: a management platform and a target cluster; the target cluster comprises a target master node, at least one first target working node having a first processing resource, and at least one second target working node having a second processing resource; the method comprises: The management platform acquires a target task, wherein the target task includes a task identifier, and the target task is used to process multiple contents; The management platform downloads the resource package of the target task from the task warehouse based on the task identifier; The management platform submits an asynchronous task to the target master node, wherein the asynchronous task includes the resource package; The target master node executes the resource package through the at least one first target working node and the at least one second target working node, utilizing the first processing resources of the at least one first target working node and the second processing resources of the at least one second target working node, so as to process the multiple contents.

13. A node, characterized in that: include: Acquisition module and processing module; The acquisition module is used to receive an asynchronous task from the management platform, wherein the asynchronous task includes a resource package of a target task, the resource package is downloaded by the management platform from a task repository, and the target task is used to process multiple contents; The processing module is used to execute the resource package through the at least one first target working node and the at least one second target working node, using the first processing resources of the at least one first target working node and the second processing resources of the at least one second target working node, so as to process the multiple contents.

14. A management platform, characterized in that: include: Acquisition module and processing module; The acquisition module is used to acquire a target task, wherein the target task is used to process multiple contents; The processing module is used to download the resource package corresponding to the target task from the task warehouse; Submit an asynchronous task to a target master node, wherein the target master node is a node in a target cluster, and the target cluster also includes at least one first target work node having a first processing resource and at least one second target work node having a second processing resource, and the asynchronous task includes the resource package, and the asynchronous task is used to request the target master node to execute the resource package through the at least one first target work node and the at least one second target work node, using the first processing resources of the at least one first target work node and the second processing resources of the at least one second target work node, so as to process the multiple contents.

15. A content processing system, characterized in that: The content processing system includes: a management platform and a target cluster; the target cluster includes a target master node, at least one first target working node with a first processing resource, and at least one second target working node with a second processing resource; The management platform is used to obtain a target task, wherein the target task includes a task identifier, and the target task is used to process multiple contents; The management platform is used to download the resource package of the target task from the task warehouse based on the task identifier; The management platform is used to submit an asynchronous task to the target master node, wherein the asynchronous task includes the resource package; The target master node is used to execute the resource package through the at least one first target working node and the at least one second target working node, utilizing the first processing resources of the at least one first target working node and the second processing resources of the at least one second target working node, so as to process the multiple contents.

16. An electronic device, characterized in that: include: A processor and a memory, the processor is connected to the memory, the memory is used to store a computer program, and the processor is used to execute the computer program stored in the memory, so that the electronic device executes the method according to any one of claims 1 to 12.

17. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and the computer program is executed by a processor to implement the method according to any one of claims 1 to 12.

Images