Streaming media data processing system, method, device, storage medium and program product

Abstract

The embodiment of the present application provides a data processing system, method, device, storage medium and program product for streaming media, and relates to the field of artificial intelligence technology. In the embodiment of the present application, the tensor computing engine is responsible for performing tensor computing, and the machine learning library corresponding to multiple subtasks in the video streaming task is constructed based on the tensor computing engine. Since different machine learning libraries are built based on the tensor computing engine, they can support the same data structure, so that multiple subtasks do not need to convert the data structure during the interpretation and execution process, thereby improving the running performance of the application. And, the runtime scheduler is responsible for allocating multiple subtasks to appropriate target hardware computing resources, and scheduling multiple subtasks in a pipeline parallelized manner to improve computing efficiency and resource utilization.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a streaming media data processing system, method, device, storage medium and program product. Background Art

[0002] With the emergence of AR (Augmented Reality) technology, users can use terminal devices such as smartphones, tablets and AR glasses to interact with digital content in real-time visual experience. It has been widely used in e-commerce, education, entertainment and other fields, so various AR applications such as AR makeup trial and AR classroom have emerged.

[0003] The workflow of AR applications: First, real-world environmental information is collected through sensors such as cameras to obtain multimedia streams; based on the multimedia streams, multimedia processing on video frames (such as image scaling, etc.), model reasoning based on machine learning tasks (such as face detection, etc.), and rendering and display between virtual content and real-world environmental information based on model reasoning results can be performed in sequence.

[0004] In the existing AR application design, in order to simplify the design difficulty, based on the computing resources such as CPU (Central Processing Unit) and GPU (Graphics Processing Unit) provided by the terminal equipment, the three parts of streaming media processing, model reasoning and rendering display are independently designed and implemented. For example, a computer vision library can be used to solve streaming media processing problems, a neural network reasoning engine can be used to implement model reasoning capabilities, and a rendering engine can be used to solve rendering display problems.

[0005] This type of AR application design is flexible and has low design difficulty. However, since one AR process requires interaction between multiple engines, there are problems with poor AR application performance, such as low operating efficiency and large AR response delay. Summary of the invention

[0006] Multiple aspects of the present application provide a streaming media data processing system, method, device, storage medium and program product to improve the operating performance of AR applications.

[0007] An embodiment of the present application provides a streaming media data processing system, the data processing system is deployed between an operating system and an application of a terminal device, the application supports at least one video stream task, the video stream task includes multiple subtasks acting serially on the same video frame, and the data processing system includes: a runtime scheduler, a dynamic programming language interpreter, multiple machine learning libraries and a tensor computing engine; the multiple machine learning libraries correspond to the multiple subtasks, and the subtasks are written in a dynamic programming language based on library functions in the corresponding machine learning libraries; the library functions in the multiple machine learning libraries are constructed based on tensor operators provided by the tensor computing engine; the runtime scheduler is used to allocate multiple subtasks in a triggered target video stream task to multiple target hardware computing resources of the terminal device during the running of the application, and obtain the video stream corresponding to the target video stream task, and schedule the multiple subtasks acting on different video frames in the video stream in a pipeline parallel manner; the dynamic language interpreter is used to interpret and execute the scheduled subtasks based on the multiple machine learning libraries and the tensor computing engine according to the scheduling order of the multiple subtasks acting on different video frames by the runtime scheduler.

[0008] An embodiment of the present application also provides a streaming media data processing method, which is applied to an application on a terminal device, wherein the application supports at least one video stream task, and the video stream task includes multiple subtasks that act serially on the same video frame; the method includes: during the running of the application, allocating multiple subtasks in the triggered target video stream task to multiple target hardware computing resources of the terminal device; obtaining a video stream corresponding to the target video stream task, wherein the video stream includes multiple video frames; scheduling the multiple subtasks that act on different video frames in the video stream in a pipeline parallel manner; interpreting and executing the scheduled subtasks based on multiple machine learning libraries and tensor computing engines according to the scheduling order of the multiple subtasks that act on different video frames; wherein the multiple machine learning libraries correspond to the multiple subtasks, and the subtasks are written in a dynamic programming language based on the library functions in the corresponding machine learning libraries; and the library functions in the multiple machine learning libraries are constructed based on tensor operators provided by the tensor computing engine.

[0009] An embodiment of the present application also provides an electronic device, including a memory and a processor, wherein the memory is used to store a computer program, and the processor is coupled to the memory and is used to execute the computer program to implement the steps in each method provided in the embodiment of the present application.

[0010] An embodiment of the present application also provides a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, the processor is enabled to implement the steps in the above method.

[0011] An embodiment of the present application also provides a computer program product, which includes a computer program / instructions. When the computer program / instructions are executed by a processor, the processor is enabled to implement the steps in the above method embodiment.

[0012] In the embodiment of the present application, the tensor computing engine is responsible for performing tensor computing, and the machine learning libraries corresponding to multiple subtasks in the video streaming task are constructed based on the tensor computing engine. Since different machine learning libraries are constructed based on the tensor computing engine, they can support the same data structure, so that multiple subtasks do not need to convert the data structure during the interpretation and execution process, thereby improving the running performance of the application. In addition, the runtime scheduler is responsible for allocating multiple subtasks to appropriate target hardware computing resources, and scheduling multiple subtasks in a pipeline parallel manner to improve computing efficiency and resource utilization.

[0013] In addition, when the application is implemented as an AR application, in response to the fragmentation of the AR development ecosystem, in an embodiment of the present application, the video data acquisition interface supported by multiple operating systems is repackaged to obtain a standardized interface, thereby shielding the software configuration differences between different operating systems and achieving interface unification. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0015] Figure 1 A schematic diagram of the structure of a video stream task development system and a data processing system included therein provided in an embodiment of the present application;

[0016] Figure 2 A schematic diagram of a scheduling process of a scheduling module provided in an embodiment of the present application;

[0017] Figure 3 A schematic diagram of the structure of a neural rendering engine provided in an embodiment of the present application;

[0018] Figure 4 A flow chart of a method for processing streaming media data provided by an embodiment of the present application;

[0019] Figure 5 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0020] In order to make the purpose, technical solution and advantages of the present application clearer, the technical solution of the present application will be clearly and completely described below in combination with the specific embodiments of the present application and the corresponding drawings. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present application.

[0021] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of relevant countries and regions, and provide corresponding operation portals for users to choose to authorize or refuse. In addition, the various models involved in this application (including but not limited to language models or large models) are in compliance with relevant laws and standards.

[0022] In the embodiment of the present application, an application is running on the terminal device. The terminal device can be an AR device, such as AR glasses, AR helmets, etc.; it can also be a VR device, such as VR glasses, VR helmets, etc.; it can also be but not limited to mobile phones, tablet computers, laptops, and smart bracelets, etc., and there is no limitation on this. In addition, there is no limitation on the types of applications, such as shopping applications, video applications, live broadcast applications, game applications, etc. The application supports at least one video stream task, wherein the video stream task refers to a task that uses machine learning technology to perform at least one processing operation on the video stream. Depending on the application scenario, the processing of the video stream will be different. For example, in the AR scenario, the processing of the video stream includes but is not limited to: image processing, model reasoning and image rendering, etc.; in the live streaming media scenario, the processing of the video stream task includes: encoding, image processing, model reasoning, image rendering, subtitle generation and image quality enhancement, etc. In the embodiment of the present application, according to the processing operation of the video stream, the video stream task is divided into multiple subtasks, and a subtask represents a processing operation on the video stream. After that, the multiple subtasks are interpreted and executed in sequence, and finally the output display of the video stream can be realized.

[0023] In the workflow of the application, different types of hardware computing resources are provided by the terminal device to execute video stream tasks, so as to perform a series of operation steps such as image processing, model reasoning and image rendering on the video stream generated by the application, so as to realize the output display of the video stream. Hardware computing resources are various hardware resources with computing capabilities provided by the terminal device, such as but not limited to: CPU, DPU (Data Processing Unit), NPU (Neural network Processing Unit) and GPU, etc. The embodiment of the present application does not limit the type and quantity of hardware computing resources provided by the terminal device. For example, it may include one or more types of hardware computing resources, and the number of each type of hardware computing resources may be one or more. If the terminal device includes multiple different types of hardware computing resources, these different types of hardware computing resources may be referred to as heterogeneous hardware computing resources.

[0024] In application design, if the subtasks included in the video streaming task are designed and implemented independently, there may be conflicts between different hardware computing resources when interpreting and executing multiple subtasks using the hardware computing resources of the terminal device, and the computing engines used to implement different subtasks need to interact with each other. The interaction process involves conversion processing of data structure, semantic information, etc., which will lead to low operating efficiency and large response delay of the upper-layer application. These problems are particularly evident on heterogeneous devices. Taking AR applications as an example, due to the significant differences in computing resources and software configurations of heterogeneous terminal devices, the AR development ecosystem is fragmented. On the one hand, the software configurations of heterogeneous terminal devices are inconsistent, resulting in different AR SDKs (Software Development Kits). For example, iOS and Android operating systems implement different AR SDKs. On the other hand, the hardware computing resources of heterogeneous terminal devices are of different and limited types. Different subtasks may consume different types of hardware computing resources. And subject to the limited hardware computing resources of the terminal device, how to allocate multiple subtasks to the appropriate hardware computing resources on the terminal device has become the key to improving efficiency and reducing latency when AR applications are running. In addition, the iteration of AR applications needs to consider the development of multiple engines, which makes it impossible to iterate quickly.

[0025] In an embodiment of the present application, the tensor computing engine is responsible for performing tensor computing, and a machine learning library corresponding to multiple subtasks in the video streaming task is constructed based on the tensor computing engine. Since different machine learning libraries are built based on the tensor computing engine, they can support the same data structure, so that multiple subtasks do not need to convert data structures, operators, etc. during the interpretation and execution process, thereby improving the running performance of the application. Moreover, in the iterative process, as long as the development of the tensor computing engine is considered, fast iteration can be achieved. And, the runtime scheduler is responsible for allocating multiple subtasks to appropriate target hardware computing resources, and scheduling multiple subtasks in a pipeline parallelized manner to improve computing efficiency and resource utilization.

[0026] In addition, when the application is implemented as an AR application, in response to the fragmentation of the AR development ecosystem, in an embodiment of the present application, the video data acquisition interface supported by multiple operating systems is re-encapsulated to obtain a standardized interface, thereby shielding the software configuration differences of heterogeneous devices and achieving interface unification.

[0027] The technical solutions provided by various embodiments of the present application are described in detail below in conjunction with the accompanying drawings.

[0028] Figure 1 A schematic diagram of the structure of a video streaming task development system provided in an embodiment of the present application. The development system 10 involves the development, deployment and runtime stages of the video streaming task, which respectively correspond to the development workbench 113, the cloud deployment platform 112 and the streaming media data processing system 100 in the development system 10, wherein the streaming media data processing system 100 can be referred to as the data processing system 100. In actual applications, the data processing system 100 is deployed on the side of the terminal device, so it can be called the end-side data processing system. Further, as Figure 1 As shown, the data processing system 100 is deployed between the operating system 107 and the application 111 of the terminal device, and is used to process the video stream task in the application 111 on the terminal side.

[0029] See also Figure 1 In this embodiment, the development system 10 includes a development workbench 113 for developers to develop and test video stream tasks based on the development workbench 113. The development workbench 113 relies on the execution environment of the video stream task by the data processing system 100, so that the video stream task can be run on the developer's desktop and other terminal devices. The developer desktop is the development tools 115 and development environment 114 provided by the development workbench 113 for developers, which are intended to help developers write, test and deploy video stream tasks. With the help of the development workbench 113, developers can debug and display the target video stream output by the video stream task on the desktop, and push the target video stream to the terminal device for end-to-end testing.

[0030] In this embodiment, a video stream task that meets the quality requirements can be obtained based on the development test of the development workbench 113. Each video stream task corresponds to identification information. The video stream task output by the development workbench 113 can be sent to the cloud deployment platform 112. In this way, the terminal device can pull the corresponding video stream task from the cloud deployment platform 112 according to the identification information of the video stream task, and use the data processing system 100 to execute the video stream task. Alternatively, the cloud deployment platform 112 obtains the identification information of the video stream task, and actively pushes the video stream task corresponding to the identification information to the terminal device, so that the terminal device can use the data processing system 100 to execute the video stream task.

[0031] It is explained here that the video streaming task can be developed through the above-mentioned development workbench 113 and cloud deployment platform 112, and the data processing system 100 can also be developed and deployed, and the data processing system 100 can be updated and deployed. The deployment method of the data processing system 100 is the same or similar to the deployment method of the video streaming task, and will not be repeated here.

[0032] In this embodiment, the video stream task can be executed during the running of the application. Among them, the processing of the video stream task can be performed by the data processing system 100. The data processing system 100 can be developed as an independent framework. In the case of developing the data processing system 100, in one implementation, the data processing system 100 is deployed in the cloud, and as the application is started, the data processing system 100 is downloaded from the cloud, and the data processing system 100 is run between the operating system 107 and the application 111, and is used to provide the application 111 with a processing service for the video stream task. In another implementation, the data processing system 100 is integrated in the installation package of the application 111, and the data processing system 100 can be installed together when the application 111 is installed; when the application 111 is started, the data processing system 100 can be locally loaded and run between the operating system 107 and the application 111, and is used to provide the application 111 with a processing service for the video stream task. An operation interface can be provided in the application 111, and the user can interact with the operation interface, so that different video stream tasks can be triggered.

[0033] In the embodiment of the present application, the application 111 supports at least one video streaming task, and the types of video streaming tasks are not limited, for example, they may include but are not limited to: AR (Augmented Reality) tasks, VR tasks, MR tasks and live streaming, etc. Among them, AR tasks refer to enhancing the user's perception of the real environment by superimposing the generated virtual content on the view of the real world environment, such as but not limited to AR makeup trial, AR product display and AR game, etc.; VR (Virtual Reality) tasks, VR tasks refer to simulating real feelings and experiences through the generated virtual environment, such as VR games and VR videos, etc.; it can also be MR (Mixed Reality) tasks, MR tasks refer to seamless integration of virtual environment and real environment, which builds an interactive feedback loop between the virtual environment, the real environment and the user to enhance the realism of the user experience, such as MR games, etc.; and live streaming, live streaming refers to the use of the network to transmit audio content and / or video content, such as live game broadcasts, live event broadcasts, etc. It should be understood that the above-mentioned types of video stream tasks are only examples, and any task that uses machine learning technology to perform a series of processing operations on the video stream generated by the application program falls within the scope of the video stream task defined in this embodiment.

[0034] Different video stream tasks supported by the same application 111 may correspond to different application portals and may be triggered in different ways. For example, an e-commerce application may support video stream tasks such as AR makeup trial, AR furniture arrangement, and AR clothing trial. Among them, e-commerce applications can provide AR fitting or AR makeup trying tasks in the corresponding product details page. Users click on the corresponding controls or issue fitting or makeup trying instructions through voice or trigger the fitting or makeup trying function through head movement, eye movement or other interactive methods, which can trigger entry into the corresponding AR scene. At this time, the e-commerce application can call the camera to collect the user's body image or facial image to form a video stream, and then merge it with the clothing or makeup products viewed by the user on the basis of the video stream to perform AR fitting or AR makeup trying, etc.; or, provide a real-scene scanning function in the e-commerce application. The user clicks the real-scene scanning control or issues a real-scene scanning instruction through voice or triggers the real-scene scanning function through head movement, eye movement or other interactive methods. The e-commerce application can call the camera to collect the real-scene image of the user's three-dimensional space to form a video stream, and merge the three-dimensional model of the product currently browsed by the user with the scanned real-scene three-dimensional space to view the layout effect of the product in the three-dimensional real-scene space; and so on.

[0035] In this embodiment, the video stream task includes multiple subtasks acting serially on the same frame in the video stream, and the subtasks are machine learning tasks, which may be, but are not limited to, image processing subtasks, model reasoning subtasks, and image rendering subtasks, etc. The processing system is responsible for parsing and executing the multiple subtasks acting on each video frame included in the video stream task to complete the output display of the target video stream.

[0036] It should be understood that different video stream tasks include different subtasks. For example, when the video stream task is an AR task, the video stream task may include an image processing subtask, a model reasoning subtask, and an image rendering subtask, etc.; when the video stream task is a live streaming media, the video stream task may include an image processing subtask, a model reasoning subtask, an image rendering subtask, a subtitle generation subtask, and a picture quality enhancement subtask, etc. In subsequent embodiments, the process of the processing system executing a video stream task will be illustrated by taking the video stream task including an image processing subtask, a model reasoning subtask, and an image rendering subtask, but it is not limited to this.

[0037] In this embodiment, the data processing system 100 includes a tensor computing engine 106. The main work of the tensor computing engine is: tensor definition, tensor computing implementation and optimization. The execution process of the tensor computing engine is generally: (1) using operators to build a computing graph; (2) tensor computing graph; (3) given an input tensor, the tensor computing engine executes the computing graph to obtain and return the output tensor. In this embodiment, a tensor is a vector or matrix of any dimension, and tensor computing is a computing process with tensors as input and output. The tensor computing engine is mainly used to provide basic tensor computing capabilities to the outside world using tensor operators (Operator, Op) as units. The tensor computing engine 106 can be, but is not limited to, an MNN (Mobile Neural Network, neural network) engine. The MNN engine is a lightweight deep learning end-side inference engine, including the optimization, conversion and inference of deep neural network models, with high generality and high performance. The tensor computing engine has optimized a large number of tensor operators for a variety of hardware computing resources, including but not limited to arithmetic operators, transformation operators, composite operators, and control flow operators. These tensor operators can be adapted to different hardware computing resources through different hardware backends, and can more widely support the construction of various machine learning libraries.

[0038] In this embodiment, the data processing system 100 includes multiple machine learning libraries 105, which are constructed based on tensor operators provided by the tensor computing engine 106 in the data processing system 100. Different integrated designs are implemented for machine learning libraries that different subtasks rely on, which is conducive to promoting the unification of data types, data structures and underlying operators, and realizing zero copy of data between library functions (or modules) of different machine learning libraries. Each subtask can be decomposed into one or more tensor operators, etc.

[0039] In this embodiment, multiple machine learning libraries 105 correspond to multiple subtasks in the video stream task. For example, when the multiple subtasks are image processing subtasks, model reasoning subtasks, and image rendering subtasks, the machine learning libraries corresponding to the multiple subtasks are image processing library, model reasoning library, and image rendering library. In other words, the machine learning library constructed based on the tensor computing engine includes at least one of the following: image processing library, model reasoning library, and image rendering library.

[0040] Each subtask is written in a dynamic programming interpretation language based on the library function in the corresponding machine learning library 105. The machine learning library 105 provides an application programming interface of the library function for video stream task developers or application developers, so that the video stream task developers or application developers can write each subtask based on the application programming interface to obtain the video stream task.

[0041] In this embodiment, there is no limitation on the dynamic programming interpretation language for writing subtasks, and it may include, for example, Python (a widely used interpreted, high-level and general programming language), JavaScript (a lightweight, interpreted or just-in-time compiled programming language with function-first characteristics), and Ruby (a simple and fast object-oriented programming scripting language), etc.

[0042] In this embodiment, the processing system includes a runtime scheduler 101. During the running of the application 111, the runtime scheduler 101 can allocate multiple subtasks in the triggered target video stream task to multiple target hardware computing resources of the terminal device (such as Figure 1 Target hardware computing resource 1-target hardware computing resource N). In this embodiment, the hardware resources 109 of the terminal device include target hardware computing resources 1-N, N is a natural number, and N>1. It should be noted that, Figure 1The types of target hardware computing resources, such as CPU, GPU or NPU, are only examples and do not mean that the terminal device only contains or must contain CPU, GPU or NPU type hardware computing resources. Different subtasks will be assigned to appropriate target hardware computing resources to avoid conflicts between target hardware computing resources, and the types of target hardware computing resources for different subtasks can be the same or different, depending on the characteristics of the subtasks and the types of hardware computing resources supported by the terminal device. It should be noted that the above-mentioned multiple target hardware computing resources can be part or all of the hardware computing resources included in the terminal device.

[0043] In an optional embodiment, the types of multiple target hardware computing resources are the same, for example, they may be CPU, GPU, or NPU, etc. In this case, since there is no difference in the types of target hardware computing resources allocated to the subtasks, in the allocation process, the operating performance, occupancy rate, throughput, etc. of each of the multiple hardware computing resources on the terminal device can be combined to allocate multiple subtasks to suitable target hardware computing resources to improve resource utilization. In another optional embodiment, the types of multiple target hardware computing resources may be different, for example, CPU, GPU, and NPU, etc., respectively, then in the process of allocating target hardware computing resources to subtasks, the degree of matching between subtasks and hardware computing resources may be considered. That is to say, any subtask has its own degree of matching with different types of hardware computing resources. In the process of allocating target hardware computing resources to any subtask, any subtask is allocated to a type of hardware computing resource with a higher degree of matching, and the hardware computing resource is used as the target hardware computing resource. For example, if the subtask is an image processing subtask, it can be assigned to the CPU for interpretation and execution; if the subtask is a model reasoning subtask, it can be assigned to the NPU or CPU for interpretation and execution; if the subtask is an image rendering subtask, it can be assigned to the GPU for interpretation and execution. Among them, assigning different subtasks to different hardware computing resources is conducive to reducing the burden of a single hardware computing resource, providing a basis for parallel scheduling of multiple subtasks, and improving the execution efficiency of subtasks while improving the utilization of hardware computing resources.

[0044] In this embodiment, the runtime scheduler 101 is also used to obtain the video stream corresponding to the target video stream task, and schedule multiple subtasks acting on different video frames in the video stream in a pipeline parallel manner. Optionally, the video stream data associated with the video stream task can be collected by the camera of the terminal device and various other sensors, and the video stream data includes multiple video frames. For multiple subtasks in the target video stream task, it can act on each video frame or part of the video frame with the video frame as the granularity, and the part of the video frame can be a sampled or selected video frame. Among them, pipeline parallel refers to scheduling and executing multiple subtasks acting on the same video frame in a pipeline manner, and multiple subtasks acting on different video frames can be executed in parallel, thereby improving the overall execution efficiency of the subtasks. For example, for video stream tasks, the processing process is relatively repetitive, including multiple video frames. The processing process of each video frame can be divided into several subtasks executed sequentially, such as image processing, model reasoning, and image rendering. Each subtask can be allocated independent hardware computing resources, and subtasks on different video frames will be allocated to different hardware computing resources. Each hardware computing resource can independently execute the subtask it is responsible for. Multiple hardware computing resources responsible for multiple subtasks on the same video frame can run sequentially, and multiple hardware computing resources responsible for subtasks on different video frames can run in parallel, thereby achieving efficient processing of video stream tasks in a resource-constrained environment.

[0045] In this embodiment, the subtask is a section of program code written in a dynamic programming language. A dynamic programming language interpreter 104 in the processing system is adapted thereto. The dynamic programming language interpreter 104 refers to an interpreter that can interpret and execute program codes programmed in a dynamic programming language, and can also be referred to as a virtual machine for a dynamic programming language. Wherein, the dynamic programming language used in writing the subtask is different, and the dynamic programming language interpreter 104 will also be different. The dynamic programming language interpreter 104 includes, but is not limited to, a Python interpreter, a JavaScript interpreter, and a Ruby interpreter.

[0046] In this embodiment, the dynamic programming language interpreter 104 in the processing system is used to interpret and execute the scheduled subtasks based on multiple machine learning libraries 105 and tensor computing engines 106 in the data processing system 100 according to the scheduling order of multiple subtasks acting on different video frames by the runtime scheduler.

[0047] In the embodiment of the present application, a machine learning library corresponding to multiple subtasks in the video stream task is constructed based on the tensor computing engine, so that multiple subtasks do not need to interact between multiple engines during the interpretation and execution process, thereby improving the running performance of the application. In addition, the runtime scheduler is responsible for allocating multiple subtasks to appropriate target hardware computing resources and scheduling multiple subtasks in a pipeline parallel manner to improve computing efficiency and resource utilization.

[0048] In an optional embodiment, the tensor operator provided by the tensor computing engine supports multiple types of hardware computing resources; the runtime scheduler includes a scheduling module, which can select a target resource allocation strategy whose operating performance meets the performance requirements from a plurality of preset resource allocation strategies based on the types of hardware computing resources supported by the terminal device, and this method of allocating target resources is referred to as static resource scheduling. In the case where the scheduling module adopts static resource scheduling, performance tests can be performed in an offline state according to multiple subtasks and multiple groups of hardware computing resources to obtain the operating performance of each resource allocation strategy, and each group of hardware computing resources includes different types and / or quantities of hardware computing resources. Optionally, in the estimation process, multiple subtasks are allocated to multiple groups of hardware computing resources for simulated execution. Among them, the operating performance of the hardware computing resources is measured by the running time of the hardware computing resources to interpret and execute the subtasks, and the sum of the running times of the multiple subtasks included in the video stream task is smaller as the goal, thereby obtaining multiple preset resource allocation strategies.

[0049] When multiple subtasks are scheduled in a pipelined parallel manner, the frequency of the pipeline outputting video frames is limited by the processing frequency of the more time-consuming subtasks. That is to say, for the video stream task, the inter-frame interval of the output video frame is determined by the delay of the slower subtask on the pipeline. In this embodiment, based on a given terminal device and video stream task, the optimization goal of the resource allocation strategy is to reduce the execution time of the more time-consuming subtasks among multiple subtasks when multiple target hardware computing resources execute multiple subtasks respectively. The execution time can be characterized by a cost function, which is defined as follows:

[0050]

[0051] Among them, Represents the computational workload of the subtask, which can be measured using floating-point operations; Indicates the operating performance of the target hardware computing resources, which can be expressed by the number of floating-point operations per second (FLOPS); Represents the scheduling cost of a subtask, for example, for an image rendering subtask on a CPU or NPU, = .

[0052] Each resource allocation strategy includes a set of resource types corresponding to multiple subtasks, and the types of hardware computing resources corresponding to multiple subtasks in different resource allocation strategies are different and have different operating performance. In one example, when the multiple subtasks are image processing subtasks, model reasoning subtasks, and image rendering subtasks, the first resource allocation strategy includes the target hardware computing resources (such as CPU) corresponding to the image processing subtask, the target hardware computing resources (such as NPU) corresponding to the model reasoning subtask, and the target hardware computing resources (such as GPU) corresponding to the image rendering subtask. The second resource allocation strategy includes the target hardware computing resources (such as CPU) corresponding to the image processing subtask, the target hardware computing resources (such as CPU) corresponding to the model reasoning subtask, and the target hardware computing resources (such as GPU) corresponding to the image rendering subtask. The first and second resource allocation strategies are allocation strategies for image processing subtasks, model reasoning subtasks, and image rendering subtasks. Assuming that the first resource allocation strategy has better operating performance than the second resource allocation strategy due to the participation of NPU, the first resource allocation strategy can be used as the target resource strategy of the terminal device when the terminal device includes NPU.

[0053] It should be understood that the above-mentioned use of the running time of the hardware computing resource running the subtask as a standard to measure its running performance is only an example. In this embodiment, the standard for measuring the running performance is not limited, and it can be the running time in the above embodiment, and can also be but not limited to the energy efficiency of the hardware computing resource, rendering quality, etc. Optionally, the running performance can be measured in combination with multiple measurement standards.

[0054] In an optional embodiment, a method for determining a target resource allocation strategy includes: based on the type of hardware computing resources supported by the terminal device, selecting at least one candidate resource allocation strategy from a plurality of preset resource allocation strategies; a group of resource types corresponding to multiple subtasks in any candidate resource allocation strategy all belong to the type of hardware computing resources supported by the terminal device; and selecting, from at least one candidate resource allocation strategy, a candidate resource allocation strategy whose operating performance meets the set performance requirements as the target resource allocation strategy.

[0055] In this embodiment, the method for the terminal device to obtain the target resource allocation policy is not limited. For example, the target resource allocation policy may be preset in the terminal device; or, multiple preset resource allocation policies may be preset in the terminal device so that when the terminal device executes the corresponding video streaming task, the target resource allocation policy can be selected from multiple preset policies; or, the target resource allocation policy may be pulled from multiple preset resource allocation policies in the cloud.

[0056] In this embodiment, according to the target resource allocation strategy, a group of resource types corresponding to multiple subtasks are included, and according to the resource type corresponding to each subtask, multiple subtasks are allocated to corresponding multiple target hardware computing resources, and the video stream corresponding to the target video stream task is obtained. As mentioned above, the AR SDKs of heterogeneous terminal devices are different, and the AR SDK can provide a video data acquisition interface 108 for obtaining video streams, such as Figure 1 As shown in the figure, iOS provides AR kit; Android provides AR Core; and HarmonyOS provides AR Engine ( Figure 1 The difference in SDK results in different video data acquisition interfaces 108 supported by different operating systems, and the data formats of the video streams acquired by different video data acquisition interfaces 108 are also different.

[0057] In this embodiment, the video streams collected by different video data acquisition interfaces 108 are called non-standardized video streams, and the corresponding standardized video streams realize the format unification of video streams in different data formats. In order to achieve the output of standardized video streams, in an optional embodiment, the processing system also includes a video data encoder 110, and the video data encoder 110 includes a standardized interface obtained by secondary packaging of video data acquisition interfaces 108 supported by multiple operating systems. The standardized interface is used to convert the non-standardized video streams collected by each video data acquisition interface 108 into a standardized video stream and output it, thereby shielding the differences between the video data acquisition interfaces 108 of different operating systems to provide development efficiency. Among them, in the case where the video stream task is an AR task, the video data encoder can be implemented as AR Unity.

[0058] In this embodiment, the runtime scheduler 101 includes a data acquisition module 103, which can acquire the normalized video stream corresponding to the target video stream task output by the normalized interface of the video data encoder 110, and store the normalized video stream in the memory space of the application 111.

[0059] Further, the runtime scheduler 101 schedules multiple subtasks acting on different video frames in the video stream in a pipeline parallelized manner. Optionally, the scheduling module 102 in the runtime scheduler 101 generates a task workflow diagram for multiple subtasks acting on different video frames. For example, the task workflow diagram can be implemented as a directed acyclic graph. Among them, the task workflow diagram includes multiple nodes, one of which represents a subtask, which is used to record the execution status and planned execution time of the subtask in different video frames. The execution status includes not executed, executing and completed, and the execution status changes dynamically with the execution of the subtask.

[0060] In this embodiment, in order to generate a task workflow graph for multiple subtasks acting on different video frames, the task workflow graph is implemented as a graph consisting of dependency edges and subtasks, each subtask corresponds to a node, and the dependency edge is a directed edge used to indicate the dependency relationship between subtasks. A dependency relationship refers to a constraint relationship between multiple subtasks, that is, the execution of one subtask depends on the result of another subtask. For example, if the dependency edge points from subtask A to subtask B, then subtask A and subtask B have a dependency relationship, and the execution of subtask B depends on the execution result of subtask A. When the execution status of subtask A is completed, subtask B can be executed.

[0061] Combine the following Figure 2 , the scheduling module is introduced to schedule multiple subtasks in a pipelined and parallel manner based on the task workflow graph. Figure 2 It includes two parts, one part is the task workflow diagram generated by the scheduling module 102, and the other part is the scheduling process of the scheduling module 102 for multiple subtasks in a pipeline parallel manner based on the task workflow diagram.

[0062] For ease of description and understanding, Figure 2 In the example, multiple subtasks are image processing subtask, model reasoning subtask and image rendering subtask, but not limited to this. Figure 2 In the text, the image processing subtask is referred to as image processing; the model reasoning subtask is referred to as model reasoning; and the image rendering subtask is referred to as image rendering. Figure 2 As shown in the figure, it includes video stream tasks acting on 4 different video frames (such as video frame 1 to video frame 4). The video stream tasks of the same video frame include image processing subtasks, model reasoning subtasks and image rendering subtasks. The video stream tasks of video frames 1 to video frames 4 include image processing subtasks 1-4, model reasoning subtasks 1-4 and image rendering subtasks respectively. Each subtask corresponds to a node, and there is a dependency relationship between the nodes. Figure 2 ,It can be seen that for multiple subtasks of the same video frame, the image rendering subtask depends on the model reasoning subtask, and the model reasoning subtask depends on the image rendering subtask. Multiple subtasks are executed sequentially, and the execution order can be expressed as image processing subtask → model reasoning subtask → image rendering subtask.

[0063] like Figure 2As shown, in this embodiment, the task workflow diagram includes the execution status and planned execution time of the subtask in different video frames, and the execution status includes not executed, executing and completed. For the execution status of each subtask, see the figure. And, for the planned execution time of different subtasks, the planned execution time of the model reasoning subtask> the planned execution time of the image rendering subtask> the planned execution time of the image processing subtask is taken as an example, but it is not limited to this. Figure 2 Each rectangle in the figure represents a subtask, the length of the rectangle represents the planned execution time of the subtask, and the pattern filled in the rectangle represents the execution status of the subtask.

[0064] like Figure 2 As shown in Figure 1, assuming that the execution time of the model reasoning subtask is longer than that of the image rendering subtask and the image processing subtask, the model reasoning subtask is used as the target subtask, and the target subtask is used as the anchor point. Combined with the planned execution time of the target subtask on different video frames recorded in the task workflow diagram, the target subtasks acting on different video frames are scheduled sequentially. Figure 2 As shown, the model reasoning subtasks of four different video frames are arranged in sequence, namely, model reasoning subtasks 1-4, and these model reasoning subtasks are executed sequentially.

[0065] In this embodiment, the model reasoning subtask is used as the target subtask, and relative to the model reasoning subtask, the other subtasks refer to the image processing subtask and the image rendering subtask. Further, according to the task dependency between the other subtasks and the target subtask in the multiple subtasks and the auxiliary information used for task scheduling, the other subtasks acting on different video frames are scheduled in parallel relative to the target subtask; the auxiliary information at least includes the execution status of the multiple subtasks on different video frames.

[0066] In this embodiment, parallel scheduling refers to the target subtask, which belongs to parallel scheduling. That is, when executing the target subtask of video frame 1, since the execution time of the target subtask is longer and other subtasks corresponding to the target subtask have been completed, other subtasks of the next video frame can be executed first. Figure 2 , and introduces parallel scheduling.

[0067] like Figure 2As shown, for multiple subtasks of the same video frame, image rendering subtask 1 depends on model reasoning subtask 1, and model reasoning subtask 1 depends on image rendering subtask 1. Further, taking model reasoning subtask 1 as the anchor point, when the image processing subtask is completed, model reasoning subtask 1 can be executed. However, model reasoning subtask 1 does not have a dependency relationship with image processing subtask 2, so during the execution of model reasoning subtask 1, image processing subtask 2 can be executed in parallel. When model reasoning subtask 1 is in the process of execution completion, image rendering subtask 1 can be executed. Similarly, image rendering subtask 1 does not have a dependency relationship with model reasoning subtask 2, so during the execution of image rendering subtask 1, model reasoning subtask 2 can be executed in parallel. And so on, until all subtasks in the video stream task are executed.

[0068] Optionally, the auxiliary information may also include the operating status and load conditions of the target hardware computing resources.

[0069] Further optionally, if Figure 2 As shown, in order to schedule the subtasks more accurately, the scheduling module can also monitor the operating status of multiple target hardware computing resources and / or the execution performance of multiple subtasks during the execution of corresponding subtasks by multiple target hardware computing resources; wherein the operating status may include the utilization rate, temperature and energy consumption of the hardware computing resources, and the execution performance may include the running time, throughput and delay, etc. Further, according to the operating status of the multiple target hardware computing resources and / or the execution performance of the multiple subtasks, different types of hardware computing resources are reallocated to at least some of the multiple subtasks, and the target resource allocation strategy is updated.

[0070] In an embodiment of the present application, a dynamic programming language interpreter can interpret and execute the scheduled subtasks based on multiple machine learning libraries and tensor computing engines according to the scheduling order of multiple subtasks acting on different video frames by the runtime scheduler. Optionally, for any subtask scheduled, the any subtask is interpreted as a machine instruction, and the machine instruction points to the target library function in the machine learning library corresponding to the any subtask. Among them, the any subtask can be a subtask acting on any video frame. Further, the tensor computing engine is called to execute the target tensor operator corresponding to the target library function on the target hardware computing resource allocated to the any subtask to execute the any subtask. The tensor computing engine executes the target tensor operator to generate tensor data as the output result. After the tensor operators corresponding to all library functions corresponding to any subtask are executed, the generated tensor data is stored in the memory space of the application as the execution result data of the subtask.

[0071] In this embodiment, multiple subtasks act on the same video frame, and the multiple subtasks are executed serially. During this process, the video frame acted on by the first subtask is input into the first task as the input data of the first task, and the execution result data is obtained after being processed by multiple subtasks including the first subtask. Among them, data sharing is involved between different subtasks. In order to achieve data sharing, in an optional embodiment, the dynamic programming language interpreter obtains the address pointer of the execution result data of any subtask in the memory space of the application program when any subtask is not the last subtask among multiple subtasks; the address pointer is passed to the next subtask acting on the same video frame as the any subtask, so that the next subtask uses the execution result data as its input data. This method of passing address pointers is suitable for high-throughput transmission.

[0072] Further optionally, the dynamic programming language interpreter creates a thread-level virtual machine based on the dynamic coding language for the target video stream task, each dynamic programming language interpreter corresponds to a thread-level virtual machine, and the dynamic programming language interpreter and its corresponding target video stream task run in the corresponding thread-level virtual machine. Thus, each thread-level virtual machine is an independent execution thread, which can not only run the program code of the target video stream task, but also has its own dynamic programming language interpreter, so that each target video stream task runs in an independent environment without interfering with each other. Then, the thread-level virtual machine is run to interpret any subtask in the target video stream task into a machine instruction, call the tensor computing engine, and execute the target tensor operator corresponding to the target library function on the target hardware computing resources allocated to any subtask to execute the any subtask.

[0073] Optionally, in the case where the target video stream task is a video stream task including multiple display windows, the multiple subtasks include an image rendering subtask, and one display window corresponds to one image rendering subtask. The dynamic language interpreter creates multiple thread-level virtual machines for the target video stream task according to the number of display windows, and one thread virtual machine is responsible for one image subtask, so that each image rendering subtask and its corresponding dynamic programming language interpreter are respectively run in an execution thread, realizing multi-threaded concurrent execution, and speeding up the execution efficiency of the image rendering subtask. In combination with the above, data sharing between different thread virtual machines can be realized by passing address pointers, which is suitable for high-throughput transmission. Optionally, for low-latency messages with lower throughput, it can be implemented based on the Actor model. The Actor model is a concurrent computing model that uses Actor as a unit-based unit, represents an entity that can execute subtasks, can receive messages, process messages and communicate with other Actors. In this optional embodiment, each thread-level virtual machine acts as an Actor, each subtask is processed by a thread-level virtual machine, and different thread-level virtual machines can communicate.

[0074] Further optionally, the multiple subtasks include an image processing subtask, a model reasoning subtask, and an image rendering subtask; the page corresponding to the target video stream task provided by the application includes multiple display areas, one display area corresponds to an image rendering subtask, and the image rendering subtask is a subtask that needs to be repeatedly executed multiple times on the same video frame, and the number of display areas is the number of repeated executions. Taking AR makeup trial as an example, the same interface includes 4 makeup trial windows of different lipstick colors, one makeup trial window corresponds to an image rendering subtask, and the image rendering subtask is repeated 4 times on the same video frame to achieve the effect of displaying 4 different lipstick colors on the same interface. It is explained here that an independent thread-level virtual machine can be created to carry and run image processing subtasks and model reasoning subtasks, or any subtask can be selected from the above multiple subtasks, and the selected subtask carries and runs the image processing subtask and model reasoning subtask, without limitation.

[0075] In this embodiment, the implementation method of the relevant engines provided by each machine learning library for performing image processing subtasks, model reasoning subtasks, and image rendering subtasks is not limited. For those skilled in the art, the scheme for building a machine learning library based on tensor operators of a tensor computing engine disclosed in the embodiment of this application can be used to flexibly build a machine learning library for implementing different functions as needed. In this embodiment, the image rendering engine is responsible for rendering the image subtask. The image rendering engine can be implemented as a neural rendering engine (Neural Network Rendering Engine, NNR), which creates a dedicated context to manage and use rendering resources, thereby avoiding global variables and achieving thread safety.

[0076] Combine the following Figure 3 The structural diagram of the neural rendering engine shown in the figure introduces the working principle of the neural rendering engine.

[0077] like Figure 3 As shown, in this embodiment, the neural rendering engine 300 includes a compiler 301 and an executor 302; the compiler 301 is used to separate the data processing in the rendering from the graphics interface call, and convert them into a machine learning model and a rendering image respectively, and further package them into a composite NNR file, i.e., a file to be rendered; the executor 302 is used to load and run the NNR file, and can control the data flow between model reasoning and image rendering. The neural rendering engine 300 adopts the architecture of the compiler 301 and the executor 302, which significantly reduces the rendering burden of the terminal device and builds a lightweight rendering environment.

[0078] In this embodiment, the executor 302 includes a data processing module 303 and a real-time rendering module 304; the data processing module 303 is used to load models and data from NNR files, perform model reasoning, script execution and data synchronization; the real-time rendering module 304 is used to execute pre-compiled renderings according to the dependencies between nodes. In this embodiment, the above-mentioned functional modules can be constructed by tensor operators in the tensor computing engine, and the various functions implemented by them can eventually be parsed into one or more tensor operators.

[0079] In the above embodiments, a streaming media data processing system is introduced. In the following embodiments, a streaming media data processing method is introduced. The details of the method can be referred to the system embodiment and will not be repeated in the following embodiments.

[0080] Figure 4 A flow chart of a streaming media data processing method provided in an embodiment of the present application. The method is applied to an application on a terminal device, the application supports at least one video streaming task, and the video streaming task includes multiple subtasks acting serially on the same video frame. Figure 4 As shown, the method includes:

[0081] S401: during the running of the application, allocating multiple subtasks in the triggered target video stream task to multiple target hardware computing resources of the terminal device;

[0082] S402: Acquire a video stream corresponding to a target video stream task, where the video stream includes a plurality of video frames;

[0083] S403: Scheduling multiple subtasks acting on different video frames in the video stream in a pipeline parallel manner;

[0084] S404: According to the scheduling order of multiple subtasks acting on different video frames, the scheduled subtasks are interpreted and executed based on multiple machine learning libraries and tensor computing engines; wherein the multiple machine learning libraries correspond to the multiple subtasks, and the subtasks are written in a dynamic programming language based on the library functions in the corresponding machine learning libraries; and the library functions in the multiple machine learning libraries are constructed based on the tensor operators provided by the tensor computing engine.

[0085] In an optional embodiment, the tensor operator provided by the tensor computing engine supports multiple types of hardware computing resources; multiple subtasks in the triggered target video stream task are allocated to multiple target hardware computing resources of the terminal device, including: according to a group of resource types corresponding to the multiple subtasks included in the target resource allocation strategy, the multiple subtasks are allocated to the multiple target hardware computing resources; wherein the target resource allocation strategy is based on the type of hardware computing resources supported by the terminal device, and a resource allocation strategy whose operating performance meets the performance requirements is selected from a plurality of preset resource allocation strategies; the resource allocation strategy includes a group of resource types corresponding to the multiple subtasks, and the resource types corresponding to the multiple subtasks in different resource allocation strategies are different and have different operating performance.

[0086] In an optional embodiment, the method further includes: monitoring the operating status of the multiple target hardware computing resources and / or the execution performance of the multiple subtasks during the process of the multiple target hardware computing resources executing corresponding subtasks; reallocating different types of hardware computing resources to at least some of the multiple subtasks based on the operating status of the multiple target hardware computing resources and / or the execution performance of the multiple subtasks, and updating the target resource allocation strategy.

[0087] In an optional embodiment, obtaining the video stream corresponding to the target video stream task includes: obtaining a normalized video stream corresponding to the target video stream task output by a normalization interface, and storing the normalized video stream in the memory space of the application.

[0088] In an optional embodiment, the multiple subtasks acting on different video frames in the video stream are scheduled in a pipeline parallel manner, including: generating a task workflow diagram for the multiple subtasks acting on different video frames, the task workflow diagram including multiple nodes, one of which represents a subtask, and is used to record the execution status and planned execution time of the subtask in different video frames, and the execution status changes dynamically with the execution of the subtask; taking the target subtask among the multiple subtasks as an anchor point, combining the planned execution time of the target subtask on different video frames recorded in the task workflow diagram, sequentially scheduling the target subtasks acting on different video frames; and scheduling other subtasks acting on different video frames in parallel relative to the target subtask based on the task dependency relationship between other subtasks among the multiple subtasks and the target subtask and auxiliary information used for task scheduling; the auxiliary information at least includes the execution status of the multiple subtasks on different video frames.

[0089] In an optional embodiment, the scheduled subtasks are interpreted and executed based on the multiple machine learning libraries and the tensor computing engine, including: for any scheduled subtask, interpreting the any subtask into a machine instruction, the machine instruction pointing to a target library function in the machine learning library corresponding to the any subtask; calling the tensor computing engine to execute the target tensor operator corresponding to the target library function on the target hardware computing resources allocated to the any subtask, so as to execute the any subtask.

[0090] In an optional embodiment, the method further includes: when any subtask is not the last subtask among the multiple subtasks, obtaining an address pointer of the execution result data of any subtask in the memory space of the application; passing the address pointer to the next subtask that acts on the same video frame as any subtask, so that the next subtask uses the execution result data as its input data; and, when any subtask is the first subtask among the multiple subtasks, using the video frame to be acted upon by any subtask as the input data of any subtask.

[0091] In an optional embodiment, the method further includes: creating a thread-level virtual machine implemented based on a dynamic coding language for the target video stream task; running the thread-level virtual machine to interpret any subtask in the target video stream task into machine instructions, calling the tensor computing engine, and executing the target tensor operator corresponding to the target library function on the target hardware computing resources allocated to any subtask to execute any subtask.

[0092] In an optional embodiment, when the target video stream task is a video stream task including multiple display windows, the multiple subtasks include an image rendering subtask, and one display window corresponds to one image rendering subtask; the method further includes: according to the number of the display windows, creating multiple thread-level virtual machines for the target video stream task, and one thread-level virtual machine is responsible for one image rendering subtask.

[0093] The detailed implementation and beneficial effects of each step in the method of this embodiment have been described in detail in the aforementioned embodiments and will not be elaborated here.

[0094] It should be noted that the execution subject of each step of the method provided in the above embodiment can be the same device, or the method can be executed by different devices. For example, the execution subject of steps 401 to 403 can be device A; for another example, the execution subject of steps 401 and 402 can be device A, and the execution subject of step 403 can be device B; and so on.

[0095] In addition, in some of the processes described in the above embodiments and the accompanying drawings, multiple operations that appear in a specific order are included, but it should be clearly understood that these operations may not be executed in the order in which they appear in this article or executed in parallel, and the sequence numbers of the operations, such as 401, 402, etc., are only used to distinguish between different operations, and the sequence numbers themselves do not represent any execution order. In addition, these processes may include more or fewer operations, and these operations may be executed in sequence or in parallel. It should be noted that the descriptions of "first", "second", etc. in this article are used to distinguish different messages, devices, modules, etc., do not represent the order of precedence, and do not limit the "first" and "second" to be different types.

[0096] Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. Figure 5 As shown, the electronic device includes: a memory 54 and a processor 55.

[0097] The memory 54 is used to store computer programs and can be configured to store various other data to support operations on the electronic device. Examples of such data include instructions for any application or method operating on the electronic device, contact data, phone book data, messages, pictures, videos, etc.

[0098] The memory 54 may be implemented by any type of volatile or nonvolatile memory device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0099] The processor 55 is coupled to the memory 54 and is used to execute the computer program in the memory 54, so as to: during the running of the application, allocate multiple subtasks in the triggered target video stream task to multiple target hardware computing resources of the terminal device; obtain the video stream corresponding to the target video stream task, and the video stream includes multiple video frames; schedule the multiple subtasks acting on different video frames in the video stream in a pipeline parallel manner; interpret and execute the scheduled subtasks based on multiple machine learning libraries and tensor computing engines according to the scheduling order of the multiple subtasks acting on different video frames; wherein the multiple machine learning libraries correspond to the multiple subtasks, and the subtasks are written in a dynamic programming language based on the library functions in the corresponding machine learning libraries; and the library functions in the multiple machine learning libraries are constructed based on the tensor operators provided by the tensor computing engine.

[0100] In an optional embodiment, the tensor operator provided by the tensor computing engine supports multiple types of hardware computing resources; when the processor 55 allocates multiple subtasks in the triggered target video stream task to the multiple target hardware computing resources of the terminal device, it is specifically used to: allocate the multiple subtasks to the multiple target hardware computing resources according to a group of resource types corresponding to the multiple subtasks included in the target resource allocation strategy; wherein the target resource allocation strategy is based on the type of hardware computing resources supported by the terminal device, and selects a resource allocation strategy whose operating performance meets the performance requirements from a plurality of preset resource allocation strategies; the resource allocation strategy includes a group of resource types corresponding to the multiple subtasks, and the resource types corresponding to the multiple subtasks in different resource allocation strategies are different and have different operating performance.

[0101] In an optional embodiment, the processor 55 is further used to: monitor the operating status of the multiple target hardware computing resources and / or the execution performance of the multiple subtasks during the process of the multiple target hardware computing resources executing corresponding subtasks; reallocate different types of hardware computing resources to at least some of the multiple subtasks based on the operating status of the multiple target hardware computing resources and / or the execution performance of the multiple subtasks, and update the target resource allocation strategy.

[0102] In an optional embodiment, when the processor 55 obtains the video stream corresponding to the target video stream task, it is specifically used to: obtain the normalized video stream corresponding to the target video stream task output by the normalization interface, and store the normalized video stream in the memory space of the application.

[0103] In an optional embodiment, when the processor 55 schedules the multiple subtasks acting on different video frames in the video stream in a pipeline parallel manner, it is specifically used to: generate a task workflow diagram for the multiple subtasks acting on different video frames, the task workflow diagram including multiple nodes, one of which represents a subtask, and is used to record the execution status and planned execution time of the subtask in different video frames, and the execution status changes dynamically with the execution of the subtask; taking the target subtask among the multiple subtasks as an anchor point, combining the planned execution time of the target subtask on different video frames recorded in the task workflow diagram, sequentially scheduling the target subtasks acting on different video frames; and according to the task dependency relationship between other subtasks among the multiple subtasks and the target subtask and the auxiliary information used for task scheduling, parallel scheduling other subtasks acting on different video frames relative to the target subtask; the auxiliary information at least includes the execution status of the multiple subtasks on different video frames.

[0104] In an optional embodiment, when the processor 55 interprets and executes the scheduled subtasks based on the multiple machine learning libraries and the tensor computing engine, it is specifically used to: interpret any scheduled subtask into a machine instruction, wherein the machine instruction points to a target library function in the machine learning library corresponding to the any subtask; and call the tensor computing engine to execute the target tensor operator corresponding to the target library function on the target hardware computing resources allocated to the any subtask, so as to execute the any subtask.

[0105] In an optional embodiment, the processor 55 is further used to: when any subtask is not the last subtask among the multiple subtasks, obtain the address pointer of the execution result data of any subtask in the memory space of the application; pass the address pointer to the next subtask that acts on the same video frame as any subtask, so that the next subtask uses the execution result data as its input data; and, when any subtask is the first subtask among the multiple subtasks, use the video frame to be acted upon by any subtask as the input data of any subtask.

[0106] In an optional embodiment, the processor 55 is also used to: create a thread-level virtual machine based on a dynamic coding language for the target video stream task; run the thread-level virtual machine to interpret any subtask in the target video stream task into machine instructions, call the tensor computing engine, and execute the target tensor operator corresponding to the target library function on the target hardware computing resources allocated to any subtask to execute any subtask.

[0107] In an optional embodiment, when the target video stream task is a video stream task including multiple display windows, the multiple subtasks include an image rendering subtask, and one display window corresponds to one image rendering subtask; the processor 55 is further used to: create multiple thread-level virtual machines for the target video stream task according to the number of display windows, and one thread-level virtual machine is responsible for one image rendering subtask.

[0108] Further, if Figure 5 As shown, the electronic device also includes: a communication component 56, a display 57, a power component 58, an audio component 59 and other components. Figure 5 Only some components are shown schematically, which does not mean that the electronic device only includes Figure 5 In addition, Figure 5The components in the dashed box are optional components, not mandatory components, and the specific components depend on the product form of the electronic device. The electronic device of this embodiment can be implemented as a terminal device such as a desktop computer, a laptop computer, a smart phone, or an IOT device, or a server device such as a conventional server, a cloud server, or a server array. If the electronic device of this embodiment is implemented as a terminal device such as a desktop computer, a laptop computer, a smart phone, etc., it can include Figure 5 If the electronic device of this embodiment is implemented as a server device such as a conventional server, a cloud server or a server array, it may not include Figure 5 Components within the dashed box.

[0109] Accordingly, an embodiment of the present application further provides a computer-readable storage medium storing a computer program, and when the computer program is executed by a processor, the processor is enabled to implement the steps in the above method.

[0110] An embodiment of the present application also provides a computer program product, which includes a computer program / instructions. When the computer program / instructions are executed by a processor, the processor is enabled to implement the steps in the above method embodiment.

[0111] The above-mentioned memory can be implemented by any type of volatile or non-volatile memory device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read only memory (EEPROM), erasable programmable read only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0112] The above-mentioned communication component is configured to facilitate wired or wireless communication between the device where the communication component is located and other devices. The device where the communication component is located can access a wireless network based on a communication standard, such as WiFi, 2G, 3G, 4G / LTE, 5G and other mobile communication networks, or a combination thereof. In an exemplary embodiment, the communication component receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on Radio Frequency Identification (RFID) technology, Infrared Data Association (IrDA) technology, Ultra Wide Band (UWB) technology, Bluetooth (BT) technology and other technologies.

[0113] The display includes a screen, and the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from a user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor may not only sense the boundary of a touch or slide action, but also detect the duration and pressure associated with the touch or slide operation.

[0114] The power supply assembly provides power to various components of the device where the power supply assembly is located. The power supply assembly may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the device where the power supply assembly is located.

[0115] The above-mentioned audio component can be configured to output and / or input audio signals. For example, the audio component includes a microphone (Microphone, MIC), and when the device where the audio component is located is in an operating mode, such as a call mode, a recording mode, and a speech recognition mode, the microphone is configured to receive an external audio signal. The received audio signal can be further stored in a memory or sent via a communication component. In some embodiments, the audio component also includes a speaker for outputting an audio signal.

[0116] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present application may take the form of a computer program product implemented on one or more computer-readable storage media (including but not limited to disk storage, compact disc read-only memory (CD-ROM), optical storage, etc.) containing computer-usable program codes.

[0117] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0118] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0119] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0120] In a typical configuration, a computing device includes one or more processors (Central Processing Unit, CPU), input / output interface, network interface and memory.

[0121] Memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. Memory is an example of a computer-readable medium.

[0122] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase-change random access memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined in this article, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0123] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0124] The above are only embodiments of the present application and are not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included within the scope of the claims of the present application.

Claims

1. A streaming media data processing system, characterized in that: The data processing system is deployed between an operating system and an application of a terminal device, the application supports at least one video stream task, the video stream task includes multiple subtasks acting serially on the same video frame, and the data processing system includes: a runtime scheduler, a dynamic programming language interpreter, multiple machine learning libraries, and a tensor computing engine; The multiple machine learning libraries correspond to the multiple subtasks, and the subtasks are written in a dynamic programming language based on library functions in the corresponding machine learning libraries; the library functions in the multiple machine learning libraries are constructed based on tensor operators provided by the tensor computing engine; The runtime scheduler is used to allocate multiple subtasks in the triggered target video stream task to multiple target hardware computing resources of the terminal device during the running of the application, and obtain the video stream corresponding to the target video stream task, and schedule the multiple subtasks acting on different video frames in the video stream in a pipeline parallel manner, including: taking the target subtask among the multiple subtasks as an anchor point, combining the planned execution time of the target subtask on different video frames, sequentially scheduling the target subtasks acting on different video frames; and, according to the task dependency relationship between other subtasks among the multiple subtasks and the target subtask and the auxiliary information used for task scheduling, parallel scheduling other subtasks acting on different video frames relative to the target subtask, wherein the auxiliary information at least includes the execution status of the multiple subtasks on different video frames; The dynamic programming language interpreter is used to interpret and execute the scheduled subtasks based on the multiple machine learning libraries and the tensor computing engine according to the scheduling order of the multiple subtasks acting on different video frames by the runtime scheduler.

2. The data processing system according to claim 1, characterized in that The tensor operators provided by the tensor computing engine support various types of hardware computing resources; the runtime scheduler includes a scheduling module, and the scheduling module is used to: Allocate the multiple subtasks to the multiple target hardware computing resources according to a group of resource types corresponding to the multiple subtasks included in the target resource allocation strategy; Among them, the target resource allocation strategy is based on the type of hardware computing resources supported by the terminal device, and a resource allocation strategy whose operating performance meets the performance requirements is selected from a plurality of preset resource allocation strategies; the resource allocation strategy includes a group of resource types corresponding to the multiple subtasks, and the resource types corresponding to the multiple subtasks in different resource allocation strategies are different and have different operating performance.

3. The data processing system according to claim 2, characterized in that: The scheduling module is also used for: During the process of the plurality of target hardware computing resources executing corresponding subtasks, monitoring the operating status of the plurality of target hardware computing resources and / or the execution performance of the plurality of subtasks; According to the running states of the multiple target hardware computing resources and / or the execution performances of the multiple subtasks, different types of hardware computing resources are reallocated to at least some of the multiple subtasks, and the target resource allocation strategy is updated.

4. The data processing system according to claim 1, characterized in that: It also includes a video data encoder, which includes a standardized interface obtained by repackaging video data acquisition interfaces supported by multiple operating systems, and the standardized interface is used to convert the non-standardized video streams collected by each video data acquisition interface into a standardized video stream and output it; The runtime scheduler includes a data acquisition module, which is used to: obtain the normalized video stream corresponding to the target video stream task output by the normalized interface of the video data encoder, and store the normalized video stream in the memory space of the application.

5. The data processing system according to claim 1, characterized in that: The runtime scheduler includes a scheduling module, and the scheduling module is further used for: A task workflow diagram is generated for the multiple subtasks acting on different video frames, wherein the task workflow diagram includes multiple nodes, one of which represents a subtask and is used to record the execution status and planned execution time of the subtask in different video frames, and the execution status changes dynamically with the execution of the subtask.

6. The data processing system according to any one of claims 1 to 5, characterized in that: The dynamic programming language interpreter is specifically used for: For any subtask that is scheduled, interpret the subtask into a machine instruction, where the machine instruction points to a target library function in a machine learning library corresponding to the subtask; The tensor computing engine is called to execute the target tensor operator corresponding to the target library function on the target hardware computing resource allocated to any subtask, so as to execute any subtask.

7. The data processing system according to claim 6, characterized in that: The dynamic programming language interpreter is also used to: In a case where the any subtask is not the last subtask among the multiple subtasks, obtaining an address pointer of the execution result data of the any subtask in the memory space of the application program; Passing the address pointer to a next subtask acting on the same video frame as any of the subtasks, so that the next subtask uses the execution result data as its input data; as well as In the case that any subtask is the first subtask among the multiple subtasks, the video frame to be acted upon by any subtask is used as input data of the any subtask.

8. The data processing system according to claim 6, characterized in that: The dynamic programming language interpreter is specifically used for: For the target video stream task, creating a thread-level virtual machine implemented based on a dynamic coding language; Run the thread-level virtual machine to interpret any subtask in the target video stream task into machine instructions, call the tensor computing engine, and execute the target tensor operator corresponding to the target library function on the target hardware computing resources allocated to any subtask to execute any subtask.

9. The data processing system according to claim 8, characterized in that: In the case where the target video stream task is a video stream task including multiple display windows, the multiple subtasks include an image rendering subtask, and one display window corresponds to one image rendering subtask; The dynamic programming language interpreter is specifically used to create a plurality of the thread-level virtual machines for the target video stream task according to the number of the display windows, and one thread-level virtual machine is responsible for one image rendering subtask.

10. A method for processing streaming media data, characterized in that: An application program applied to a terminal device, the application program supporting at least one video stream task, the video stream task comprising a plurality of subtasks acting serially on the same video frame; the method comprising: During the running of the application, multiple subtasks in the triggered target video stream task are allocated to multiple target hardware computing resources of the terminal device; Acquire a video stream corresponding to the target video stream task, wherein the video stream includes multiple video frames; The plurality of subtasks acting on different video frames in the video stream are scheduled in a pipeline parallel manner, including: taking a target subtask among the plurality of subtasks as an anchor point, sequentially scheduling the target subtasks acting on different video frames in combination with the planned execution time of the target subtask on different video frames; and parallelly scheduling the other subtasks acting on different video frames relative to the target subtask according to the task dependency between other subtasks among the plurality of subtasks and the target subtask and auxiliary information used for task scheduling; the auxiliary information at least includes the execution status of the plurality of subtasks on different video frames; According to the scheduling order of the multiple subtasks acting on different video frames, the scheduled subtasks are interpreted and executed based on multiple machine learning libraries and tensor computing engines; Among them, the multiple machine learning libraries correspond to the multiple subtasks, and the subtasks are written in a dynamic programming language based on the library functions in the corresponding machine learning libraries; the library functions in the multiple machine learning libraries are constructed based on the tensor operators provided by the tensor computing engine.

11. The method according to claim 10, characterized in that The tensor operator provided by the tensor computing engine supports multiple types of hardware computing resources; multiple subtasks in the triggered target video stream task are allocated to multiple target hardware computing resources of the terminal device, including: Allocate the multiple subtasks to the multiple target hardware computing resources according to a group of resource types corresponding to the multiple subtasks included in the target resource allocation strategy; Among them, the target resource allocation strategy is based on the type of hardware computing resources supported by the terminal device, and a resource allocation strategy whose operating performance meets the performance requirements is selected from a plurality of preset resource allocation strategies; the resource allocation strategy includes a group of resource types corresponding to the multiple subtasks, and the resource types corresponding to the multiple subtasks in different resource allocation strategies are different and have different operating performance.

12. The method according to claim 11, characterized in that The method further comprises: During the process of the plurality of target hardware computing resources executing corresponding subtasks, monitoring the operating status of the plurality of target hardware computing resources and / or the execution performance of the plurality of subtasks; According to the running states of the multiple target hardware computing resources and / or the execution performances of the multiple subtasks, different types of hardware computing resources are reallocated to at least some of the multiple subtasks, and the target resource allocation strategy is updated.

13. The method according to claim 10, characterized in that Obtaining the video stream corresponding to the target video stream task includes: The normalized video stream corresponding to the target video stream task output by the normalization interface is obtained, and the normalized video stream is stored in the memory space of the application program.

14. The method according to claim 10, characterized in that Before sequentially scheduling the target subtasks acting on different video frames in combination with the planned execution time of the target subtasks on different video frames, the method further includes: A task workflow diagram is generated for the multiple subtasks acting on different video frames, wherein the task workflow diagram includes multiple nodes, one of which represents a subtask and is used to record the execution status and planned execution time of the subtask in different video frames, and the execution status changes dynamically with the execution of the subtask.

15. The method according to any one of claims 10 to 14, characterized in that: The method further comprises: interpreting and executing the scheduled subtasks based on the multiple machine learning libraries and the tensor computing engine, including: For any subtask that is scheduled, interpret the subtask into a machine instruction, where the machine instruction points to a target library function in a machine learning library corresponding to the subtask; The tensor computing engine is called to execute the target tensor operator corresponding to the target library function on the target hardware computing resource allocated to any subtask, so as to execute any subtask.

16. The method according to claim 15, characterized in that The method further comprises: In a case where the any subtask is not the last subtask among the multiple subtasks, obtaining an address pointer of the execution result data of the any subtask in the memory space of the application program; Passing the address pointer to a next subtask acting on the same video frame as any one of the subtasks, so that the next subtask uses the execution result data as its input data; and In the case that any subtask is the first subtask among the multiple subtasks, the video frame to be acted upon by any subtask is used as input data of the any subtask.

17. The method according to claim 15, characterized in that The method further comprises: For the target video stream subtask, creating a thread-level virtual machine implemented based on a dynamic coding language; Run the thread-level virtual machine to interpret any subtask in the target video stream task into machine instructions, call the tensor computing engine, and execute the target tensor operator corresponding to the target library function on the target hardware computing resources allocated to any subtask to execute any subtask.

18. An electronic device, characterized in that: The invention comprises a memory and a processor, wherein the memory is used to store a computer program, and the processor is coupled to the memory and is used to execute the computer program to implement the steps in the method according to any one of claims 10 to 17.

19. A computer-readable storage medium storing a computer program / instruction, characterized in that: When the computer program is executed by a processor, the processor is enabled to implement the steps in the method according to any one of claims 10 to 17.

20. A computer program product, characterized in that include: A computer program / instruction, when executed by a processor, causes the processor to implement the steps of the method according to any one of claims 10 to 17.

Images