Video stream processing method, system, electronic device and storage medium

Abstract

The present disclosure provides a video stream processing method, a video stream processing device, a video stream task management device, a media server, a video stream processing system, an electronic device, a storage medium and a program product, which relate to the field of image processing technology, and in particular to the field of artificial intelligence and computer vision technology. The specific implementation scheme is: the method is applied to a video stream processing device of a video stream processing system, the video stream processing system also includes a video stream task management device, the method includes: loading a calculation graph, wherein the calculation graph includes pre-arranged operators for processing video streams and scheduling logic between operators; obtaining video stream processing tasks distributed by the video stream task management device; processing the video stream processing tasks based on the calculation graph. The present disclosure can improve system performance and development efficiency.

Description

Technical Field

[0001] The present disclosure relates to the field of image processing technology, in particular to the field of artificial intelligence and computer vision technology, and specifically to a video stream processing method, a video stream processing device, a video stream task management device, a media server, a video stream processing system, an electronic device, a storage medium and a program product. Background Art

[0002] Video content is everywhere in our lives, in our smartphones, desktops, smart TVs, and all types of cameras, with huge amounts of data streaming over the Internet every day.

[0003] In the era of artificial intelligence, computer vision technology is one of its research branches. It can establish a mapping relationship between images and image descriptions, so that the content of the video can be understood through digital image processing and analysis. Therefore, the combination of video stream and computer vision has gradually become one of the main application methods of video stream processing. Summary of the invention

[0004] The present disclosure provides a video stream processing method, a video stream processing device, a video stream task management device, a media server, a video stream processing system, an electronic device, a storage medium and a program product.

[0005] According to one aspect of the present disclosure, a video stream processing method is provided, which is applied to a video stream processing device of a video stream processing system, wherein the video stream processing system further includes a video stream task management device, and the method includes:

[0006] Loading a computation graph, wherein the computation graph includes pre-arranged operators for processing video streams and scheduling logic between operators;

[0007] Obtaining a video stream processing task distributed by the video stream task management device;

[0008] The video stream processing task is processed based on the computation graph.

[0009] According to another aspect of the present disclosure, a video stream processing method is provided, which is applied to a video stream task management device of a video stream processing system, wherein the video stream processing system further includes a video stream processing device, and the method includes:

[0010] The video stream processing task is distributed to the video stream processing device, wherein the video stream processing device is loaded with a calculation graph, the calculation graph includes pre-arranged operators for processing video streams and scheduling logic between operators, and the video stream processing device is used to process the video stream processing task based on the calculation graph.

[0011] According to another aspect of the present disclosure, a video stream processing method is provided, which is applied to a media server of a video stream processing system, wherein the video stream processing system further includes a video stream task management device and a video stream processing device, and the method includes:

[0012] Acquire registration information from the video stream task management device, where the registration information is used to register a video stream address corresponding to a video stream processing task;

[0013] Acquire the video stream corresponding to the video stream processing task according to the video stream address;

[0014] Generate an intranet address of the video stream corresponding to the video stream processing task, and send the intranet address to the video stream processing device;

[0015] Among them, the video stream processing device is used to obtain the video stream processing task distributed by the video stream task management device, obtain the video stream corresponding to the video stream processing task according to the intranet address, and use the video stream processing device to process the video stream corresponding to the video stream processing task based on the loaded calculation graph. The calculation graph is loaded by the video stream processing device, including pre-arranged operators for processing video streams and scheduling logic between operators.

[0016] According to another aspect of the present disclosure, a video stream processing device is provided, which is applied to a video stream processing system. The video stream processing system further includes a video stream task management device. The video stream processing device includes:

[0017] A loading module, used to load a computation graph, wherein the computation graph includes pre-arranged operators for processing video streams and scheduling logic between operators;

[0018] A task acquisition module, used to acquire the video stream processing task distributed by the video stream task management device;

[0019] A processing module is used to process the video stream processing task based on the calculation graph.

[0020] According to another aspect of the present disclosure, a video stream task management device is provided, which is applied to a video stream processing system, wherein the video stream processing system further includes a video stream processing device, and the video stream task management device includes:

[0021] A task distribution module is used to distribute the video stream processing task to the video stream processing device, wherein the video stream processing device is loaded with a calculation graph, the calculation graph includes pre-arranged operators for processing video streams and scheduling logic between operators, and the video stream processing device is used to process the video stream processing task based on the calculation graph.

[0022] According to another aspect of the present disclosure, a media server is provided, which is applied to a video stream processing system, wherein the video stream processing system further includes a video stream task management device and a video stream processing device, and the media server includes:

[0023] A registration information acquisition module, used to acquire registration information from the video stream task management device, wherein the registration information is used to register a video stream address corresponding to a video stream processing task;

[0024] A video stream acquisition module, used for acquiring the video stream corresponding to the video stream processing task according to the video stream address;

[0025] An intranet address generation module, used to generate an intranet address of the video stream corresponding to the video stream processing task, and send the intranet address to the video stream processing device;

[0026] Among them, the video stream processing device is used to obtain the video stream processing task distributed by the video stream task management device, obtain the video stream corresponding to the video stream processing task according to the intranet address, and process the video stream corresponding to the video stream processing task based on the loaded calculation graph, wherein the calculation graph includes pre-arranged operators for processing video streams and scheduling logic between operators.

[0027] According to another aspect of the present disclosure, a video stream processing system is provided, comprising a video stream task management device, a video stream processing device and a media server, wherein:

[0028] The video stream task management device is used to send registration information to the media server, wherein the registration information is used to register the video stream address corresponding to the video stream processing task;

[0029] The media server is used to obtain the video stream corresponding to the video stream processing task according to the video stream address, generate the intranet address of the video stream corresponding to the video stream processing task, and send the intranet address to the video stream processing device;

[0030] The video stream task management device is further used to distribute the video stream processing task to the video stream processing device;

[0031] The video stream processing device is used to obtain the video stream processing task distributed by the video stream task management device, obtain the video stream corresponding to the video stream processing task according to the intranet address, and process the video stream corresponding to the video stream processing task based on the loaded calculation graph, wherein the calculation graph includes pre-arranged operators for processing video streams and scheduling logic between operators.

[0032] According to another aspect of the present disclosure, there is provided an electronic device, comprising:

[0033] at least one processor; and

[0034] a memory communicatively connected to the at least one processor; wherein,

[0035] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the video stream processing method described in any embodiment of the present disclosure.

[0036] According to another aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable a computer to execute the video stream processing method described in any embodiment of the present disclosure.

[0037] According to another aspect of the present disclosure, a computer program product is provided, including a computer program, wherein when the computer program is executed by a processor, the video stream processing method described in any embodiment of the present disclosure is implemented.

[0038] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure.

[0040] Figure 1 is a schematic diagram of a video stream processing method according to an embodiment of the present disclosure;

[0041] Figure 2 is a schematic diagram of a video stream processing method according to an embodiment of the present disclosure;

[0042] Figure 3 is a schematic diagram of a video stream processing method according to an embodiment of the present disclosure;

[0043] Figure 4 is a schematic diagram of a video stream processing method according to an embodiment of the present disclosure;

[0044] Figure 5 is a schematic diagram of a video stream processing method according to an embodiment of the present disclosure;

[0045] Figure 6 is a structural schematic diagram of a video stream processing device according to an embodiment of the present disclosure;

[0046] Figure 7 is a structural diagram of a video stream task management device according to an embodiment of the present disclosure;

[0047] Figure 8 is a structural diagram of a media server according to an embodiment of the present disclosure;

[0048] Fig. 9 is a structural schematic diagram of a video stream processing system according to an embodiment of the present disclosure;

[0049] Fig.10a is an architecture diagram of a video stream processing system according to an embodiment of the present disclosure;

[0050] Fig.10b is a flowchart corresponding to the architecture diagram of the video stream processing system according to an embodiment of the present disclosure;

[0051] Fig.11 is a schematic diagram of an abstract processing flow of a data processing mode of a pipeline according to an embodiment of the present disclosure;

[0052] Fig.12 is an interaction diagram of a video stream processing device according to an embodiment of the present disclosure;

[0053] Fig.13 The block diagram is a block diagram of an electronic device for implementing the video stream processing method according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0054] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0055] Figure 1 This is a flowchart of a video stream processing method according to an embodiment of the present disclosure. This embodiment is applicable to situations where real-time processing and analysis of video streams generated in real time is performed, and relates to the field of image processing technology, and in particular to the field of artificial intelligence and computer vision technology. The method can be executed by a video stream processing device of a video stream processing system, and the video stream processing device is implemented in software and / or hardware, and is preferably configured in an electronic device, such as a computer device or a server. The video stream processing method of this embodiment is applied to a video stream processing device of a video stream processing system, and the video stream processing system also includes a video stream task management device. Figure 1 As shown, the method specifically includes the following:

[0056] S101. Load a computation graph, wherein the computation graph includes pre-arranged operators for processing video streams and scheduling logic between operators.

[0057] S102: Obtain a video stream processing task distributed by a video stream task management device.

[0058] S103. Process the video stream processing task based on the computational graph.

[0059] The video stream processing device and the video stream task management device cooperate with each other to complete the video stream task processing. The video stream task management device can be used for task distribution and scheduling, and the video stream processing device is used to implement specific calculations and video analysis processing, that is, to achieve decoupling of the scheduling system and the computing system.

[0060] The video stream processing task refers to a task to be processed for processing and analyzing the video stream, for example, to achieve environmental safety monitoring, face detection or crowd counting through the analysis of the video stream. After the video stream processing device obtains the video stream processing task distributed by the video stream task management device, it uses the loaded computational graph to process the video stream processing task.

[0061] The calculation graph includes pre-arranged operators for processing video streams and scheduling logic between operators. An operator is a reusable reasoning and processing component that can be abstracted as a specific function or operation. In the embodiment of the present disclosure, an operator refers to a calculation function or algorithm for processing video streams, which can be abstracted into four aspects: pre-processing engineering operators, AI operators, business logic operators, and post-processing engineering operators, and the corresponding operator warehouses are formed through accumulation. Among them, pre-processing engineering operators can include basic operators such as media stream access, encoding / decoding, frame extraction, and image processing (cropping, scaling, image format conversion); AI operators can include models generated by machine learning or deep learning, such as AI algorithms such as face recognition, image classification, and text recognition; business logic operators are operators developed by users according to business logic requirements, such as current limiting, clock frequency synchronization, channel merging, and back pressure operators; post-processing engineering operators can include operators such as structured information transmission, text frame diagram overlay, push media streams, and image rendering that meet the current external output requirements. The technicians configure the calculation graph according to the actual processing requirements of the video stream. The operators in the calculation graph can indicate what processing needs to be performed on the video stream, and the scheduling logic between the operators can indicate the complete processing flow. Therefore, the calculation graph can show the complete process of processing the video stream. Developers can freely develop the required operators according to the specifications, form an operator warehouse, and form a calculation graph for processing the video stream through arrangement according to the scene requirements. After loading the calculation graph, the video stream processing device can call the corresponding operator from the operator warehouse and use the calculation graph to complete the complete task processing process of the video stream.

[0062] It should be noted that the processing of video streams in related technologies is usually based on a big data streaming processing framework, which is composed of different clusters. Each cluster performs its own function relatively independently, and completes the video stream processing through combination. In other words, the process of processing video streams is implemented in different clusters. Since data transmission between clusters is also required through the network, a large amount of network I / O will be generated in the related technology, and the data cannot be saved in the memory during this process. A large amount of disk I / O will also occur, resulting in insufficient system performance and insufficient throughput.

[0063] In the technical solution of the embodiment of the present disclosure, the video stream task management device realizes task distribution and scheduling, and the video stream processing device completes the specific calculation and processing, so that the scheduling system and the computing system are decoupled, the scheduling system is responsible for distributing tasks, and the computing system is responsible for task processing. Therefore, a complete set of video stream processing processes is completed in a video stream processing device, rather than being scattered in different clusters as in the related art. Therefore, the embodiment of the present disclosure does not involve the problem of insufficient system performance and insufficient throughput caused by a large number of network I / Os and disk I / Os appearing in different clusters and between clusters, so that the system performance is improved. At the same time, the video stream processing device of the embodiment of the present disclosure processes the video stream based on the loaded calculation graph. The developer only needs to configure and arrange the corresponding calculation graph based on different scenarios and requirements, and then load the calculation graph through the video stream processing device, so that the complete video stream processing process can be realized based on the calculation graph, which not only improves the development efficiency of video stream processing, but also facilitates migration in different scenarios, and improves applicability and convenience.

[0064] Figure 2 1 is a flow chart of a video stream processing method according to an embodiment of the present disclosure. This embodiment is further optimized on the basis of the above embodiment. In this embodiment, the video stream processing system further includes a media server. Figure 2 As shown, the method specifically includes the following:

[0065] S201. Load a computation graph, where the computation graph includes pre-arranged operators for processing video streams and scheduling logic between operators.

[0066] S202: Obtain a video stream processing task distributed by a video stream task management device.

[0067] S203. Obtain the video stream corresponding to the video stream processing task through the intranet address provided by the media server, and process the video stream corresponding to the video stream processing task based on the calculation graph.

[0068] The intranet address is generated by the media server after obtaining the video stream corresponding to the video stream processing task. The video stream corresponding to the video stream processing task is obtained by the media server according to the video stream address, and the video stream address is registered to the media server by the video stream task management device. In other words, the media server plays the role of external video stream aggregation, and then through the converted intranet address, the video stream processing device can obtain the video stream corresponding to the video stream processing task.

[0069] In one embodiment, the video stream processing device of the embodiment of the present disclosure processes the video stream processing task in a single instance single process based on the calculation graph. Specifically, based on virtualization and containerization technologies such as k8s, a video stream processing device can start multiple instances, each instance runs in its own process, and some resources (CPU, memory, disk and operating system, etc.) are allocated to each instance as needed, so that the video stream processing of a task can be completed in a single instance single process. In a single instance single process, the scheduling between the operators involved in the calculation graph is implemented in thread mode, and the scheduling process between the operators only involves simple memory data exchange, which improves the speed, thereby further improving the system performance and concurrency, and reducing the system delay. At the same time, multiple instances can run in parallel to process video streams in different tasks, thereby improving the task processing efficiency. Of course, the same task can also be divided to obtain multiple different subtasks, which are processed simultaneously by different instances. The embodiment of the present disclosure does not make any limitation to this, and can be configured based on needs.

[0070] In another embodiment, the video stream processing device of the embodiment of the present disclosure processes the video stream processing task based on the calculation graph and in accordance with the data processing mode of the pipeline. Specifically, when the video stream processing device is started, the calculation graph corresponding to the current task will be loaded, and after completion, the video stream data will be waited for input. After the video stream data is input, the video stream processing device will realize the efficient processing of the video stream data in the process through mechanisms such as thread pool, state machine, asynchronous processing, timestamp, etc. Among them, the data processing mode of the pipeline, its core is to divide the entire data processing into multiple modules, and the modules are connected in series by related data channels. When processing data, each module performs parallel processing, and when each module is processed, the processed data is passed to the downstream module through the channel to achieve efficient data processing. In the video stream task processing scenario of the embodiment of the present disclosure, each operator is a module, and the scheduling logic between each operator is the data channel between each module. Specifically, for video streaming scenarios, the entire data processing process can be abstracted into pre-processing (stream access, decoding, frame extraction, image processing, etc.), AI algorithms, business logic, post-processing and other modules. After orchestration, a data processing calculation graph that can cope with different scenarios can be formed. Each data module processes video or image data in parallel, thereby achieving smooth and efficient data processing.

[0071] In the related art, since its framework is composed of different clusters, different clusters have different requirements for algorithms and pre- and post-processing in different processing scenarios, it is necessary to develop different processing logics for each scene for each different cluster, which makes the overall development efficiency low. In contrast, the technical solution of the embodiment of the present disclosure only needs to configure and arrange the calculation graph, and a complete set of video stream processing processes can be completed in a single instance and a single process under the processing mode of the pipeline. This not only realizes programmability, but also avoids unnecessary I / O and improves system performance. In addition, it can also quickly iterate according to the changes in the scene, accumulate operator warehouses, and achieve rapid reuse and iteration of scenes to improve development efficiency.

[0072] In another embodiment, the video stream processing system of the embodiment of the present disclosure further includes a video stream processing cluster, and the video stream processing cluster includes multiple video stream processing devices; accordingly, the method further includes reporting the number of tasks currently being processed to the video stream task management device. The video stream task management device is used to determine the idle state of the corresponding video stream processing device according to the processing capacity of each video stream processing device in the video stream processing cluster and the number of tasks currently being processed, and distribute the video stream processing tasks according to the idle state.

[0073] Specifically, a cluster of multiple video stream processing devices can be formed in the video stream processing system, and the video stream task management device distributes tasks according to the idle state of each video stream processing device. The idle state can be determined according to the processing capacity of the video stream processing device and the number of tasks currently being processed. Therefore, the video stream processing device also needs to report the number of tasks currently being processed to the video stream task management device. Among them, the capacity of the video stream processing device refers to the number of video streams that can be processed simultaneously. For example, if the capacity is 10 channels, it means that 10 video streams can be processed in one process. Moreover, some operators can be processed in batches to improve performance. For example, GPU (NVIDIA Jetson) supports batch processing mechanism, which can perform AI processing on 16 pictures together, instead of serial processing one by one, thereby improving performance.

[0074] In other implementations, developers can also develop required operators based on GPU or CPU according to the specifications of the video stream processing device. When all operators are developed based on GPU, full GPU processing of the operators can be achieved, thereby improving the processing efficiency of video images.

[0075] Exemplarily, the processing of the video stream may include the following steps, wherein ①②③ are processed by the pre-processing engineering operator, ④ is processed by the AI ​​operator, ⑤ is processed by the business logic operator, and ⑥ is processed by the post-processing engineering operator:

[0076] ① Pull stream: Pull the video stream corresponding to the video stream processing task according to the intranet address;

[0077] ② Video stream decapsulation: Decapsulate the pulled video stream to obtain the original video stream. The streaming media protocol supports common rtsp, rtmp, hls and GB28181, etc.

[0078] ③ Video stream decoding: decode the original video stream, and then extract the corresponding frame images at the configured time interval to form a frame image sequence. Video encoding formats such as h.264, h.265, VP9, ​​etc. are supported;

[0079] ④ Image processing: It can perform general processing on the decoded images, such as frame extraction, cropping, compression, format conversion, etc., to facilitate integration into the input format required by subsequent AI operators;

[0080] ⑤AI processing: The acquired frame sequence or sampled image is input into the encapsulated AI operator. This part mainly performs AI processing and returns the output of the algorithm;

[0081] ⑥Business logic processing: including the application side performing relevant business logic processing based on the obtained algorithm results;

[0082] ⑦ Post-processing: For some forms of external output after being processed by the application business logic, it mainly includes support for structured information transmission or text box overlay, which facilitates the development of business system monitoring alarms and rendered video stream playback.

[0083] Figure 3 This is a flowchart of a video stream processing method according to an embodiment of the present disclosure. This embodiment is applicable to situations where real-time processing and analysis of video streams generated in real time is performed, and relates to the field of image processing technology, and in particular to the field of artificial intelligence and computer vision technology. The method can be executed by a video stream task management device of a video stream processing system, and the video stream task management device is implemented in software and / or hardware, and is preferably configured in an electronic device, such as a computer device or a server. The video stream processing method of this embodiment is applied to a video stream task management device of a video stream processing system, and the video stream processing system also includes a video stream processing device. Figure 3 As shown, the method specifically includes the following:

[0084] S301. Distribute the video stream processing task to the video stream processing device, wherein the video stream processing device is loaded with a calculation graph, the calculation graph includes pre-arranged operators for processing video streams and scheduling logic between operators, and the video stream processing device is used to process the video stream processing task based on the calculation graph.

[0085] Specifically, the video stream processing device and the video stream task management device cooperate with each other to complete the video stream task processing. The video stream task management device can be used for task distribution and scheduling, and the video stream processing device is used to implement specific calculations and video analysis processing, that is, to achieve decoupling of the scheduling system and the computing system. After the video stream task management device obtains the video stream processing task, it distributes the video stream processing task to the video stream processing device, and the video stream processing device processes the video stream processing task based on the loaded calculation graph. In this process, a complete set of video stream processing processes is completed in a video stream processing device, rather than being dispersed in different clusters as in the related art. Therefore, the embodiments of the present disclosure do not involve the problems of insufficient system performance and insufficient throughput caused by a large number of network I / Os and disk I / Os in different clusters and between clusters, so that the system performance can be improved. At the same time, the video stream processing device of the embodiment of the present disclosure processes the video stream based on the loaded calculation graph. The developer only needs to configure and arrange the corresponding calculation graph based on different scenarios and requirements, and then load the calculation graph through the video stream processing device to realize the complete video stream processing process based on the calculation graph. This not only improves the development efficiency of video stream processing, but also facilitates migration in different scenarios, thereby improving applicability and convenience.

[0086] Figure 4 1 is a flow chart of a video stream processing method according to an embodiment of the present disclosure. This embodiment is further optimized on the basis of the above embodiment, wherein the video stream processing system further includes a video stream processing cluster and a media server, and the video stream processing cluster includes a plurality of the video stream processing devices. Figure 4 As shown, the method specifically includes the following:

[0087] S401. Create a video stream processing task according to the acquired task information, and put the video stream processing task into a task queue.

[0088] The task information includes, for example, the start time, end time, or frame extraction interval of the task processing, etc. In addition, it may also include information of hardware devices such as cameras for collecting video streams.

[0089] S402: Obtain the video stream address of the video stream processing task, and register the video stream address to the media server.

[0090] Specifically, the video stream address is registered to the media server, and the media server is allowed to obtain the video stream corresponding to the video stream processing task according to the video stream address, and generate the intranet address of the video stream. It can be seen that the media server plays the role of external data stream convergence, obtains the video stream from the outside, and then sends the intranet address to the video stream processing device, so that the video stream processing device can obtain the video stream corresponding to the video stream processing task through the intranet address.

[0091] S403: Determine a target video stream processing device according to the idle state of each video stream processing device in the video stream processing cluster.

[0092] The idle state is determined based on the processing capacity of the video stream processing device and the number of tasks currently being processed. The processing capacity of each video stream processing device will be recorded in the video stream task management device, and each video stream processing device can also feed back the status of its own processing tasks to the video stream task management device in a certain time period. Specifically, the video stream task management device can first obtain the number of tasks currently being processed by each video stream processing device in the video stream processing cluster, and then determine the idle state of the corresponding video stream processing device based on the processing capacity of each video stream processing device and the number of tasks currently being processed, and determine the target video stream processing device based on the idle state, which is the currently most idle optimal video stream processing device for processing the current task.

[0093] S404: Distribute the video stream processing task to a target video stream processing device, wherein the target video stream processing device is loaded with a computation graph, the computation graph including pre-arranged operators for processing video streams and scheduling logic between operators, and the target video stream processing device is used to process the video stream processing task based on the computation graph.

[0094] In one embodiment, the video stream processing task is processed by the video stream processing device in a single instance and a single process based on a computation graph. In another embodiment, the video stream processing task is processed by the video stream processing device based on a computation graph and in accordance with a pipeline data processing mode. Of course, the video stream processing device can also be used to process the task in a single instance and a single process based on a computation graph and in accordance with a pipeline data processing mode.

[0095] The technical solution of the disclosed embodiment not only decouples the scheduling system and the computing system, but also the computing system is a cluster composed of multiple video stream processing devices. The video stream task management device can distribute tasks according to the idle state of each video stream processing device to achieve load balancing and improve processing efficiency. In addition, each video stream processing device abstracts the stream processing process into a specific pipeline data processing mode. Developers can configure and arrange the required calculation graphs based on different scenarios, and complete a complete set of video stream processing processes in a single instance and a single process. This not only achieves programmability, but also avoids unnecessary I / O and improves system performance. In addition, it is also possible to quickly iterate according to changes in the scene, accumulate operator warehouses, and achieve rapid reuse and iteration of scenes to improve development efficiency.

[0096] Figure 5This is a flowchart of a video stream processing method according to an embodiment of the present disclosure. This embodiment is applicable to situations where real-time processing and analysis of video streams generated in real time is performed, and relates to the field of image processing technology, and in particular to the field of artificial intelligence and computer vision technology. The method can be executed by a media server of a video stream processing system, and the media server is implemented in software and / or hardware, and is preferably configured in an electronic device, such as a computer device or a server. The video stream processing method of this embodiment is applied to a media server of a video stream processing system, and the video stream processing system also includes a video stream task management device and a video stream processing device. Figure 5 As shown, the method specifically includes the following:

[0097] S501. Acquire registration information from a video stream task management device, where the registration information is used to register a video stream address corresponding to a video stream processing task.

[0098] S502: Obtain the video stream corresponding to the video stream processing task according to the video stream address.

[0099] S503: Generate an intranet address of the video stream corresponding to the video stream processing task, and send the intranet address to the video stream processing device.

[0100] Among them, the video stream processing device is used to obtain the video stream processing task distributed by the video stream task management device, obtain the video stream corresponding to the video stream processing task according to the intranet address, and process the video stream corresponding to the video stream processing task based on the loaded calculation graph, wherein the calculation graph includes pre-arranged operators for processing video streams and scheduling logic between operators.

[0101] The technical solution of the disclosed embodiment is that the media server is mainly used for external video stream aggregation, adapting to different streaming media protocols, converting the video stream into a specified streaming media protocol, and providing an intranet address for the video stream processing device to pull the stream, so that subsequent processing can rely on a unified streaming media protocol without adapting to multiple protocols. At the same time, after the streaming media format conversion of the media server, the user can also directly view the video stream through the media server, which provides convenience for the user. In addition, when the same video stream needs to be processed in multiple channels, the media server can also copy the video stream so that it can be processed by multiple video stream processing devices at the same time.

[0102] Figure 61 is a schematic diagram of the structure of a video stream processing device according to an embodiment of the present disclosure. The video stream processing device of this embodiment is applied to a video stream processing system, and the video stream processing system also includes a video stream task management device. This embodiment can be applied to the situation where real-time processing and analysis of video streams generated in real time are performed, and it relates to the field of image processing technology, especially to the field of artificial intelligence and computer vision technology. The video stream processing device can implement the video stream processing method described in any embodiment of the present disclosure. Figure 7 As shown, the video stream processing device 600 specifically includes:

[0103] A loading module 601, used to load a computation graph, wherein the computation graph includes pre-arranged operators for processing video streams and scheduling logic between operators;

[0104] The task acquisition module 602 is used to acquire the video stream processing task distributed by the video stream task management device.

[0105] The processing module 603 is used to process the video stream processing task based on the calculation graph.

[0106] Optionally, the video stream processing system further includes a video stream processing cluster, and the video stream processing cluster includes a plurality of the video stream processing devices; accordingly, the device further includes:

[0107] A reporting module, used for reporting the number of tasks currently being processed to the video stream task management device;

[0108] The video stream task management device is used to determine the idle state of the corresponding video stream processing device according to the processing capacity of each video stream processing device in the video stream processing cluster and the number of tasks currently being processed, and distribute the video stream processing tasks according to the idle state.

[0109] Optionally, the video stream processing system further includes a media server; accordingly, the processing module 603 is specifically used for:

[0110] Obtaining the video stream corresponding to the video stream processing task through the intranet address provided by the media server, and processing the video stream corresponding to the video stream processing task based on the calculation graph;

[0111] Among them, the intranet address is generated after the media server obtains the video stream corresponding to the video stream processing task, the video stream corresponding to the video stream processing task is obtained by the media server according to the video stream address, and the video stream address is registered to the media server by the video stream task management device.

[0112] Optionally, the processing module 603 includes:

[0113] The first processing unit is used to process the video stream processing task in a single instance and a single process based on the calculation graph.

[0114] Optionally, the processing module 603 includes:

[0115] The second processing unit is used to process the video stream processing task based on the calculation graph and in accordance with the pipeline data processing mode.

[0116] Figure 7 1 is a schematic diagram of the structure of a video stream task management device according to an embodiment of the present disclosure. The video stream task management device of this embodiment is applied to a video stream processing system, and the video stream processing system also includes a video stream processing device. This embodiment can be applied to the situation where real-time processing and analysis of video streams generated in real time are performed, and it relates to the field of image processing technology, especially to the field of artificial intelligence and computer vision technology. The video stream task management device can implement the video stream processing method described in any embodiment of the present disclosure. Figure 7 As shown, the video stream task management device 700 specifically includes:

[0117] The task distribution module 701 is used to distribute the video stream processing task to the video stream processing device, wherein the video stream processing device is loaded with a calculation graph, the calculation graph includes pre-arranged operators for processing video streams and scheduling logic between operators, and the video stream processing device is used to process the video stream processing task based on the calculation graph.

[0118] Optionally, the video stream processing system further includes a video stream processing cluster, and the video stream processing cluster includes a plurality of the video stream processing devices; accordingly, the task distribution module 701 includes:

[0119] a target determination unit, configured to determine a target video stream processing device according to an idle state of each video stream processing device in the video stream processing cluster;

[0120] A task distribution unit is used to distribute the video stream processing task to the target video stream processing device, and the target video stream processing device is used to process the video stream processing task based on the calculation graph.

[0121] Optionally, the target determination unit includes:

[0122] A task quantity acquisition subunit, used to acquire the number of tasks currently processed by each video stream processing device in the video stream processing cluster;

[0123] The target determination subunit is used to determine the idle state of the corresponding video stream processing device according to the processing capacity of each video stream processing device and the number of tasks currently being processed, and determine the target video stream processing device according to the idle state.

[0124] Optionally, the video stream processing system further includes a media server; correspondingly, the device further includes:

[0125] An address registration module, used for obtaining the video stream address of the video stream processing task and registering the video stream address with the media server before the task distribution module 701 distributes the video stream processing task to the video stream processing device;

[0126] Among them, the media server is used to obtain the video stream corresponding to the video stream processing task according to the video stream address, and generate the intranet address of the video stream corresponding to the video stream processing task, and the video stream processing device is used to obtain the video stream corresponding to the video stream processing task according to the intranet address.

[0127] Optionally, the device further comprises:

[0128] The queue management module is used to create the video stream processing task according to the acquired task information and put the video stream processing task into the task queue before the task distribution module 701 distributes the video stream processing task to the video stream processing device.

[0129] Optionally, the video stream processing task is processed by the video stream processing device in a single instance and a single process based on the computation graph.

[0130] Optionally, the video stream processing task is processed by the video stream processing device based on the calculation graph and in accordance with a pipeline data processing mode.

[0131] Figure 8 1 is a schematic diagram of the structure of a media server according to an embodiment of the present disclosure. The media server of this embodiment is applied to a video stream processing system, and the video stream processing system also includes a video stream task management device and a video stream processing device. This embodiment can be applied to the situation where real-time processing and analysis of video streams generated in real time are performed, and it relates to the field of image processing technology, especially to the field of artificial intelligence and computer vision technology. The media server can implement the video stream processing method described in any embodiment of the present disclosure. Figure 8 As shown, the media server 800 specifically includes:

[0132] A registration information acquisition module 801 is used to acquire registration information from the video stream task management device, where the registration information is used to register the video stream address corresponding to the video stream processing task;

[0133] A video stream acquisition module 802 is used to acquire the video stream corresponding to the video stream processing task according to the video stream address;

[0134] An intranet address generating module 803, used to generate an intranet address of the video stream corresponding to the video stream processing task, and send the intranet address to the video stream processing device;

[0135] Among them, the video stream processing device is used to obtain the video stream processing task distributed by the video stream task management device, obtain the video stream corresponding to the video stream processing task according to the intranet address, and process the video stream corresponding to the video stream processing task based on the loaded calculation graph, wherein the calculation graph includes pre-arranged operators for processing video streams and scheduling logic between operators.

[0136] Fig. 9 1 is a schematic diagram of the structure of a video stream processing system according to an embodiment of the present disclosure. This embodiment can be applied to the case of real-time processing and analysis of real-time video streams, and relates to the field of image processing technology, especially to the field of artificial intelligence and computer vision technology. The video stream processing system can implement the video stream processing method described in any embodiment of the present disclosure. Fig. 9 As shown, the video stream processing system 900 includes a video stream task management device 901, a video stream processing device 902 and a media server 903, wherein:

[0137] The video stream task management device 901 is used to send registration information to the media server, wherein the registration information is used to register the video stream address corresponding to the video stream processing task;

[0138] The media server 903 is used to obtain the video stream corresponding to the video stream processing task according to the video stream address, generate the intranet address of the video stream corresponding to the video stream processing task, and send the intranet address to the video stream processing device;

[0139] The video stream task management device 901 is further used to distribute the video stream processing task to the video stream processing device;

[0140] The video stream processing device 902 is used to obtain the video stream processing task distributed by the video stream task management device, obtain the video stream corresponding to the video stream processing task according to the intranet address, and process the video stream corresponding to the video stream processing task based on the loaded calculation graph, wherein the calculation graph includes pre-arranged operators for processing video streams and scheduling logic between operators.

[0141] The above-mentioned product can execute the method provided by any embodiment of the present disclosure, and has the corresponding functional modules and beneficial effects of the execution method.

[0142] Fig.10a1 is an architecture diagram of a video stream processing system according to an embodiment of the present disclosure. As shown in the figure, the video stream processing system mainly includes a video stream task management device, a video stream processing cluster and multiple media servers. Fig.10b is a flowchart corresponding to the architecture diagram of the video stream processing system according to an embodiment of the present disclosure. Fig.10a and Fig.10b , describe the working process of the video stream processing system:

[0143] S1: Camera access. The user accesses the camera through the front-end console and submits relevant information about the camera and edge box, such as media stream address, hardware device information, and regional information.

[0144] S2: Task submission. The front-end console submits the task to the video stream task management device, and includes the configured task information, including the task start time, end time and frame extraction interval.

[0145] S3: Media stream registration. After receiving the camera information, the controller Master in the video stream task processing device will first register the media stream address to the media server. The media server cluster is mainly responsible for external video stream access, adapting to different streaming media protocols, and uniformly converting them into the specified streaming media protocol and providing an internal video stream address to facilitate other processing by the internal cluster.

[0146] S4: Task distribution: After receiving the task, the Master creates a scheduling task in the internal JobCreater and enters the task queue of the Task Manager. The Task Manager is responsible for obtaining the relevant tasks and the task information such as the camera of the task from the queue, and selects the most idle video stream processing device (Worker) from the media stream processing cluster for task distribution.

[0147] S5: Task processing. After receiving the task, the designated Worker of the media stream cluster pulls the video data according to the intranet video stream address and starts the arranged pipeline model to process the video stream in real time.

[0148] S6: Post-processing. It mainly performs some operations defined by users, such as video streaming, video storage, monitoring alarm, docking with the application's business system, and other process operations until the results are output.

[0149] Fig.11 FIG. 1 is a schematic diagram of an abstract processing flow of a pipeline data processing mode according to an embodiment of the present disclosure. Fig.11As shown, in the embodiment of the present disclosure, the stream processing process is abstracted into a specific pipeline, which mainly includes four parts, namely pre-processing, AI reasoning, business logic and post-processing, which specifically includes the process of stream access-decoding-frame extraction-image processing-AI algorithm-business logic-post-processing. It can be seen that the embodiment of the present disclosure provides pipeline support and related basic engineering operator support, among which the basic engineering operator includes general video processing capabilities, such as video access, hardware / software encoding and decoding, image processing (cropping, scaling, image format conversion, etc.), rendering, playback, etc. It also includes commonly used operators in the process of streaming data processing, such as current limiting, clock frequency synchronization, channel merging, back pressure operators, etc. Developers can freely develop AI operators according to the specifications, and arrange the required scenarios based on the pipeline mechanism, and finally provide external services.

[0150] Fig.12 This is an interactive diagram of a video stream processing device according to an embodiment of the present disclosure. In Worker, the pipeline processes pending tasks in a process processing mode. The processing of media streams is performed in a single instance and a single process. The scheduling between operators adopts a thread mode, which avoids a large amount of I / O work and only performs simple memory data exchange, greatly improving performance and concurrency and reducing system latency. Specifically, the Worker processing flow is as follows:

[0151] Load the calculation graph configured by the user, and obtain the required operators by parsing the configuration;

[0152] Pull relevant operators from the operator warehouse according to the required operators and combine them into a processing pipeline according to the calculation graph;

[0153] After the combination is completed, the pipeline is started through the built-in scheduler of the worker, and resources are allocated to related operators. After completion, it waits for the media stream to be pulled and then processed;

[0154] When the external media stream is input, it will be processed according to the calculation graph arranged by the developer, and finally return structured data or rendered media stream as needed for post-processing by the business system.

[0155] The video stream processing system disclosed in the present invention has the following beneficial effects:

[0156] (1) Distributed automatic load balancing and configurable stream processing system. By decoupling the scheduling system from the computing system, the scheduling system is responsible for the load balancing, deployment management, resource management and other functions of all video stream processing tasks. The pipeline system abstracts the stream processing process into a specific pipeline processing model to facilitate performance estimation and pipeline orchestration. Finally, a distributed and configurable video stream processing system is realized.

[0157] (2) The data processing mode of the pipeline supports the programmable characteristics of intelligent video stream processing. This feature can help business lines quickly orchestrate the required pipelines according to actual user scenarios, and can quickly iterate according to scenario changes to improve support efficiency. At the same time, this solution can be used to accumulate operator warehouses, so that business lines can use more operators in the future, and the development efficiency is higher. Finally, various proven and effective video stream processing scenarios can be precipitated to achieve rapid reuse and iteration of scenarios.

[0158] (3) High-performance real-time video stream processing framework, using process processing mode. The media stream is processed in a single instance and a single process, and the scheduling between operators adopts thread mode, avoiding a large amount of I / O work and only performing simple memory data exchange, which greatly improves performance and concurrency and reduces system latency. At the same time, for the application of acceleration cards such as GPU, the data decoded after the video stream is connected can be fully processed by GPU to improve system performance.

[0159] (4) Based on the programmable computation graph and the performance information of each operator, it is possible to accurately evaluate how many video streams a processing instance can process, and then perform accurate scheduling and load balancing. However, the existing solution cannot accurately evaluate performance due to the separation of various modules, large image data volume, and different grouping sizes.

[0160] (5) The pipeline based on the process mode can process the video stream frame by frame or frame by frame. All frame data are exchanged through the memory or video memory. It has high real-time performance and can cover all existing video stream scenarios. In the existing solution, different modules are deployed as a cluster service and use the network for communication. Therefore, due to the large amount of frame data after the video stream is decoded, only the frame extraction processing mode can be used. At the same time, the frequency of extracting frames cannot be too high, otherwise it will cause the performance and real-time performance to decrease, and the applicable scenarios are narrow.

[0161] In the technical solution of the present disclosure, the collection, storage, use, processing, transmission, provision and disclosure of user personal information involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0162] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium and a computer program product.

[0163] Fig.13A schematic block diagram of an example electronic device 1300 that can be used to implement an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0164] like Fig.13 As shown, the device 1300 includes a computing unit 1301, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 1302 or a computer program loaded from a storage unit 1308 into a random access memory (RAM) 1303. In the RAM 1303, various programs and data required for the operation of the device 1300 can also be stored. The computing unit 1301, the ROM 1302, and the RAM 1303 are connected to each other via a bus 1304. An input / output (I / O) interface 1305 is also connected to the bus 1304.

[0165] A number of components in the device 1300 are connected to the I / O interface 1305, including: an input unit 1306, such as a keyboard, a mouse, etc.; an output unit 1307, such as various types of displays, speakers, etc.; a storage unit 1308, such as a disk, an optical disk, etc.; and a communication unit 1309, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 1309 allows the device 1300 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0166] The computing unit 1301 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 1301 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 1301 performs the various methods and processes described above, such as a video stream processing method applied to a video stream processing device. For example, in some embodiments, the video stream processing method applied to a video stream processing device may be implemented as a computer software program, which is tangibly included in a machine-readable medium, such as a storage unit 1308. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 1300 via the ROM 1302 and / or the communication unit 1309. When the computer program is loaded into the RAM 1303 and executed by the computing unit 1301, one or more steps of the video stream processing method applied to the video stream processing device described above may be performed. Alternatively, in other embodiments, the computing unit 1301 may be configured in any other appropriate manner (for example, by means of firmware) to execute the video stream processing method applied to the video stream processing apparatus.

[0167] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0168] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0169] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0170] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0171] The systems and techniques described herein may be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0172] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship between the client and the server is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or a cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services. The server may also be a server of a distributed system, or a server combined with a blockchain.

[0173] Artificial intelligence is a discipline that studies how to use computers to simulate certain human thought processes and intelligent behaviors (such as learning, reasoning, thinking, planning, etc.). It includes both hardware-level and software-level technologies. Artificial intelligence hardware technologies generally include technologies such as sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, and big data processing; artificial intelligence software technologies mainly include computer vision technology, speech recognition technology, natural language processing technology, machine learning / deep learning technology, big data processing technology, knowledge graph technology, and other major directions.

[0174] Cloud computing refers to a technology system that uses network access to elastically scalable shared physical or virtual resource pools. Resources can include servers, operating systems, networks, software, applications, and storage devices, and can be deployed and managed on demand and in a self-service manner. Cloud computing technology can provide efficient and powerful data processing capabilities for technical applications such as artificial intelligence and blockchain, as well as model training.

[0175] In addition, according to an embodiment of the present disclosure, the present disclosure also provides another electronic device, another readable storage medium and another computer program product, which are used to execute one or more steps of the video stream processing method applied to the video stream task management device and the video stream processing method applied to the media server described in any embodiment of the present disclosure. The specific structure and program code thereof can be found in Fig.13 The description of the contents of the illustrated embodiment will not be repeated here.

[0176] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions provided by this disclosure can be achieved, and this document does not limit this.

[0177] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A video stream processing method, applied to a video stream processing device of a video stream processing system, wherein the video stream processing system further comprises a video stream task management device, the method comprising: Loading a computation graph, wherein the computation graph includes pre-arranged operators for processing video streams and scheduling logic between operators; Obtaining a video stream processing task distributed by the video stream task management device; Processing the video stream processing task based on the computation graph; The video stream processing system further includes a media server; accordingly, processing the video stream processing task based on the computational graph includes: Obtaining the video stream corresponding to the video stream processing task through the intranet address provided by the media server, and processing the video stream corresponding to the video stream processing task based on the calculation graph; Among them, the intranet address is generated after the media server obtains the video stream corresponding to the video stream processing task, the video stream corresponding to the video stream processing task is obtained by the media server according to the video stream address, and the video stream address is registered to the media server by the video stream task management device; the video stream task management device is used for task distribution and scheduling.

2. According to the method of claim 1, the video stream processing system further comprises a video stream processing cluster, and the video stream processing cluster comprises a plurality of the video stream processing devices; accordingly, the method further comprises: Reporting the number of tasks currently being processed to the video stream task management device; The video stream task management device is used to determine the idle state of the corresponding video stream processing device according to the processing capacity of each video stream processing device in the video stream processing cluster and the number of tasks currently being processed, and distribute the video stream processing tasks according to the idle state.

3. The method according to claim 1, wherein: The processing of the video stream processing task based on the computation graph includes: Based on the computation graph, the video stream processing task is processed in a single instance and a single process.

4. The method according to claim 1, wherein: The processing of the video stream processing task based on the computation graph includes: Based on the computation graph, the video stream processing task is processed according to a pipeline data processing mode.

5. A video stream processing method, applied to a video stream task management device of a video stream processing system, wherein the video stream processing system further comprises a video stream processing device and a media server, the method comprising: Obtaining a video stream address of a video stream processing task, and registering the video stream address to the media server; The media server is used to obtain the video stream corresponding to the video stream processing task according to the video stream address, and generate the intranet address of the video stream corresponding to the video stream processing task, and the video stream processing device is used to obtain the video stream corresponding to the video stream processing task according to the intranet address; the video stream processing task is distributed to the video stream processing device, wherein the video stream processing device is loaded with a calculation graph, the calculation graph includes pre-arranged operators for processing video streams and scheduling logic between operators, and the video stream processing device is used to process the video stream processing task based on the calculation graph.

6. The method according to claim 5, wherein: The video stream processing system further comprises a video stream processing cluster, wherein the video stream processing cluster comprises a plurality of the video stream processing devices; Accordingly, distributing the video stream processing task to the video stream processing device includes: Determining a target video stream processing device according to an idle state of each video stream processing device in the video stream processing cluster; The video stream processing task is distributed to the target video stream processing device, and the target video stream processing device is used to process the video stream processing task based on the calculation graph.

7. The method according to claim 6, wherein: The step of determining a target video stream processing device according to the idle state of each video stream processing device in the video stream processing cluster comprises: Obtaining the number of tasks currently being processed by each video stream processing device in the video stream processing cluster; According to the processing capacity of each of the video stream processing devices and the number of tasks currently being processed, the idle state of the corresponding video stream processing device is determined, and the target video stream processing device is determined according to the idle state.

8. The method according to claim 5, wherein: Before distributing the video stream processing task to the video stream processing device, the method further includes: The video stream processing task is created according to the acquired task information, and the video stream processing task is put into a task queue.

9. The method according to claim 5, wherein: The video stream processing task is processed by the video stream processing device in a single instance and a single process based on the calculation graph.

10. The method according to claim 5, wherein: The video stream processing task is processed by the video stream processing device based on the calculation graph and in accordance with the pipeline data processing mode.

11. A video stream processing method, applied to a media server of a video stream processing system, wherein the video stream processing system further comprises a video stream task management device and a video stream processing device, the method comprising: Acquire registration information from the video stream task management device, where the registration information is used to register a video stream address corresponding to a video stream processing task; Acquire the video stream corresponding to the video stream processing task according to the video stream address; Generate an intranet address of the video stream corresponding to the video stream processing task, and send the intranet address to the video stream processing device; The video stream processing device is used to obtain the video stream processing task distributed by the video stream task management device, obtain the video stream corresponding to the video stream processing task according to the intranet address, and process the video stream corresponding to the video stream processing task based on the loaded calculation graph, wherein the calculation graph includes pre-arranged operators for processing video streams and scheduling logic between operators; The video stream task management device is used to determine the idle state of the corresponding video stream processing device according to the processing capacity of each video stream processing device in the video stream processing cluster and the number of tasks currently being processed, and distribute the video stream processing tasks according to the idle state.

12. A video stream processing device, applied to a video stream processing system, the video stream processing system further comprising a video stream task management device, the video stream processing device comprising: A loading module, used to load a computation graph, wherein the computation graph includes pre-arranged operators for processing video streams and scheduling logic between operators; A task acquisition module, used to acquire the video stream processing task distributed by the video stream task management device; A processing module, used for processing the video stream processing task based on the calculation graph; Wherein, the video stream processing system further includes a media server; accordingly, the processing module is specifically used for: Obtaining the video stream corresponding to the video stream processing task through the intranet address provided by the media server, and processing the video stream corresponding to the video stream processing task based on the calculation graph; Among them, the intranet address is generated after the media server obtains the video stream corresponding to the video stream processing task, the video stream corresponding to the video stream processing task is obtained by the media server according to the video stream address, and the video stream address is registered to the media server by the video stream task management device; the video stream task management device is used for task distribution and scheduling.

13. A video stream task management device, applied to a video stream processing system, the video stream processing system further comprising a video stream processing device and a media server, the video stream task management device comprising: A task distribution module, used to distribute the video stream processing task to the video stream processing device, wherein the video stream processing device is loaded with a calculation graph, the calculation graph includes pre-arranged operators for processing video streams and scheduling logic between operators, and the video stream processing device is used to process the video stream processing task based on the calculation graph; The address registration module is used to obtain the video stream address of the video stream processing task and register the video stream address to the media server before the task distribution module distributes the video stream processing task to the video stream processing device; wherein the media server is used to obtain the video stream corresponding to the video stream processing task according to the video stream address, and generate the intranet address of the video stream corresponding to the video stream processing task, and the video stream processing device is used to obtain the video stream corresponding to the video stream processing task according to the intranet address.

14. A media server, applied to a video stream processing system, the video stream processing system further comprising a video stream task management device and a video stream processing device, the media server comprising: A registration information acquisition module, used to acquire registration information from the video stream task management device, wherein the registration information is used to register a video stream address corresponding to a video stream processing task; A video stream acquisition module, used for acquiring the video stream corresponding to the video stream processing task according to the video stream address; An intranet address generation module, used to generate an intranet address of the video stream corresponding to the video stream processing task, and send the intranet address to the video stream processing device; The video stream processing device is used to obtain the video stream processing task distributed by the video stream task management device, obtain the video stream corresponding to the video stream processing task according to the intranet address, and process the video stream corresponding to the video stream processing task based on the loaded calculation graph, wherein the calculation graph includes pre-arranged operators for processing video streams and scheduling logic between operators; The video stream task management device is used to determine the idle state of the corresponding video stream processing device according to the processing capacity of each video stream processing device in the video stream processing cluster and the number of tasks currently being processed, and distribute the video stream processing tasks according to the idle state.

15. A video stream processing system, comprising a video stream task management device, a video stream processing device and a media server, wherein: The video stream task management device is used to send registration information to the media server, wherein the registration information is used to register the video stream address corresponding to the video stream processing task; The media server is used to obtain the video stream corresponding to the video stream processing task according to the video stream address, generate the intranet address of the video stream corresponding to the video stream processing task, and send the intranet address to the video stream processing device; The video stream task management device is further used to distribute the video stream processing task to the video stream processing device; The video stream processing device is used to obtain the video stream processing task distributed by the video stream task management device, obtain the video stream corresponding to the video stream processing task according to the intranet address, and process the video stream corresponding to the video stream processing task based on the loaded calculation graph, wherein the calculation graph includes pre-arranged operators for processing video streams and scheduling logic between operators.

16. The system of claim 15, wherein: The number of the video stream processing devices is multiple, forming a video stream processing cluster; Correspondingly, the video stream task management device is also used to determine the target video stream processing device according to the idle state of each video stream processing device in the video stream processing cluster, and distribute the video stream processing task to the target video stream processing device.

17. An electronic device comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the video stream processing method according to any one of claims 1-4, the video stream processing method according to any one of claims 5-10, or the video stream processing method according to claim 11.

18. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to enable a computer to execute the video stream processing method according to any one of claims 1 to 4, the video stream processing method according to any one of claims 5 to 10, or the video stream processing method according to claim 11.

19. A computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the computer program implements the video stream processing method according to any one of claims 1 to 4, the video stream processing method according to any one of claims 5 to 10, or the video stream processing method according to claim 11.

Images