Video data processing method and device, equipment, storage medium and edge equipment

By optimizing tensor data processing and allocation to graphics processing units, the method addresses resource inefficiencies in edge computing, enabling efficient video data processing and deployment of edge devices.

CN119071529APending Publication Date: 2024-12-03BOE TECHNOLOGY GROUP CO LTD +1
View PDF 14 Cites 0 Cited by

Patent Information

Application Number
CN202310626368.8
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2023-05-30
Publication Date
2024-12-03

AI Technical Summary

Technical Problem

Existing video data processing methods require significant computing and storage resources, making it difficult to deploy electronic equipment at the edge due to the need for separate processing modules and multiple deep learning models, which leads to inefficient resource utilization and high real-time processing demands.

Method used

A method that involves obtaining multiple tensor data based on video frame data within a preset time window, splicing sub-tensor data to form second tensor data, and allocating these data to graphics processing units for inference, optimizing resource utilization and reducing the number of deep learning models required.

Benefits of technology

This approach enhances the utilization rate of graphics processors, allowing for efficient video data processing at the edge by reducing resource consumption and enabling deployment of small edge devices.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119071529A_ABST
    Figure CN119071529A_ABST
Patent Text Reader

Abstract

The invention provides a video data processing method and device, equipment, a storage medium and edge equipment, and relates to the technical field of artificial intelligence, in particular to the technical field of computer vision and edge computing. The video data processing method comprises the steps that multiple pieces of first tensor data are obtained based on multiple pieces of video frame data received in a preset time window, and the first tensor data comprise multiple pieces of sub-tensor data corresponding to multiple deep learning models respectively; splicing a plurality of sub-tensor data included in each of the plurality of first tensor data to obtain a plurality of second tensor data corresponding to each of the plurality of deep learning models; and distributing the plurality of second tensor data to the plurality of graphic processing units based on deep learning models deployed by the plurality of graphic processing units included in the graphic processor, so as to perform data reasoning on the plurality of second tensor data through the graphic processor.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence technology, in particular to the field of computer vision and edge computing technology, and specifically to a video data processing method, apparatus, device, storage medium and edge device. Background Art

[0002] Edge computing can be the process of moving the operation of applications, data and services from the central node of the network to the edge node of the network logic for processing. For example, in the scenario of video surveillance analysis, edge computing can be an electronic device deployed on the edge node to process the video data collected by multiple cameras connected to the edge node. Summary of the invention

[0003] The present disclosure provides a video data processing method, apparatus, device, storage medium and edge device.

[0004] According to one aspect of the present disclosure, a video data processing method is provided, including: obtaining multiple first tensor data based on multiple video frame data received within a preset time window, wherein the above-mentioned first tensor data include multiple sub-tensor data corresponding to multiple deep learning models respectively; splicing the multiple sub-tensor data included in each of the above-mentioned multiple first tensor data to obtain multiple second tensor data corresponding to each of the above-mentioned multiple deep learning models; and based on the above-mentioned deep learning models respectively deployed by multiple graphics processing units included in a graphics processor, allocating the above-mentioned multiple second tensor data to the above-mentioned multiple graphics processing units, so as to perform data inference on the above-mentioned multiple second tensor data through the above-mentioned graphics processor.

[0005] According to another aspect of the present disclosure, a video data processing device is provided, including: a first processing module, used to obtain multiple first tensor data based on multiple video frame data received within a preset time window, wherein the above-mentioned first tensor data include multiple sub-tensor data corresponding to each of the multiple deep learning models; a second processing module, used to splice the multiple sub-tensor data included in each of the above-mentioned multiple first tensor data, to obtain multiple second tensor data corresponding to each of the above-mentioned multiple deep learning models; and a distribution module, used to distribute the above-mentioned multiple second tensor data to the above-mentioned multiple graphics processing units based on the above-mentioned deep learning models respectively deployed by the multiple graphics processing units included in the graphics processor, so as to perform data inference on the above-mentioned multiple second tensor data through the above-mentioned graphics processor.

[0006] According to another aspect of the present disclosure, an electronic device is provided, including a memory and a processor, wherein instructions executable by the processor are stored in the memory, and when the instructions are executed by the processor, the processor executes the method described above.

[0007] According to another aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable a computer to execute and implement the above-mentioned method.

[0008] According to another aspect of the present disclosure, a computer program product is provided, including a computer program, and when the computer program is executed by a processor, the computer program implements the above method.

[0009] According to another aspect of the present disclosure, an edge device is provided, including: a frame multiplexer, a business plug-in manager, an inference manager and a model manager; wherein the business plug-in manager is configured to: obtain multiple first tensor data based on multiple video frame data received within a preset time window, wherein the first tensor data includes multiple sub-tensor data corresponding to each of the multiple deep learning models; the inference manager is configured to: splice the multiple sub-tensor data included in each of the multiple first tensor data to obtain multiple second tensor data corresponding to each of the multiple deep learning models; and based on the deep learning models deployed by each of the multiple graphics processing units included in the graphics processor, distribute the multiple second tensor data to the multiple graphics processing units; and the model manager is configured to: perform data inference on the multiple second tensor data through the graphics processor.

[0010] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure.

[0012] Figure 1 The figure schematically shows a video data processing method in the related art.

[0013] Figure 2 An exemplary system architecture to which the video data processing method according to an embodiment of the present disclosure can be applied is schematically shown.

[0014] Figure 3 The flowchart of the video data processing method according to the embodiment of the present disclosure is schematically shown.

[0015] Figure 4 The diagram schematically shows a video data processing method according to an embodiment of the present disclosure.

[0016] Figure 5 A schematic diagram of an edge device according to an embodiment of the present disclosure is schematically shown.

[0017] Fig. 6A A schematic diagram of a frame multiplexer according to an embodiment of the present disclosure is schematically shown.

[0018] Figure 6B A schematic diagram schematically shows a video frame multiplexing workflow of a frame multiplexer according to an embodiment of the present disclosure.

[0019] Figure 7 The diagram schematically shows a service plug-in manager according to an embodiment of the present disclosure.

[0020] Figure 8 The following is a schematic diagram schematically showing a communication process between a business plug-in manager and an inference manager according to an embodiment of the present disclosure.

[0021] Fig. 9 The figure schematically shows a schematic diagram of a model manager according to an embodiment of the present disclosure.

[0022] Fig.10 The block diagram schematically shows a video processing device according to an embodiment of the present disclosure.

[0023] Fig.11 A block diagram of an electronic device suitable for implementing a video data processing method according to an embodiment of the present disclosure is schematically shown. DETAILED DESCRIPTION

[0024] In order to make the purpose, technical scheme and advantages of the embodiments of the present disclosure clearer, the technical scheme in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are part of the embodiments of the present disclosure, not all of them. Based on the described embodiments of the present disclosure, all other embodiments obtained by ordinary technicians in this field without creative work belong to the scope of protection of the present disclosure. It should be noted that throughout the drawings, the same elements are represented by the same or similar figure marks. In the following description, some specific embodiments are only used for descriptive purposes and should not be understood as any limitation to the present disclosure, but are only examples of the embodiments of the present disclosure. Conventional structures or configurations will be omitted when they may cause confusion in the understanding of the present disclosure. It should be noted that the shapes and sizes of the components in the figures do not reflect the actual size and proportion, but only illustrate the contents of the embodiments of the present disclosure.

[0025] Unless otherwise defined, the technical terms or scientific terms used in the embodiments of the present disclosure should have the usual meanings understood by those skilled in the art. The words "first", "second" and similar words used in the embodiments of the present disclosure do not indicate any order, quantity or importance, but are only used to distinguish different components.

[0026] In the embodiments of the present disclosure, the collection, updating, analysis, processing, use, transmission, provision, disclosure, storage, etc. of the data involved (for example, including but not limited to user personal information) are in compliance with the provisions of relevant laws and regulations, are used for legitimate purposes, and do not violate public order and good morals. In particular, necessary measures are taken for user personal information to prevent illegal access to user personal information data and maintain the security of user personal information, network security, and national security.

[0027] In the embodiments of the present disclosure, the user's authorization or consent is obtained before obtaining or collecting the user's personal information.

[0028] With the development of Internet technology, the centralized processing mode of cloud computing has gradually failed to meet the real-time requirements of business processing. In this regard, businesses with high real-time requirements can be deployed on edge nodes, and the edge nodes complete the processing operations of the business to achieve rapid response of the business. For example, in the video surveillance analysis scenario, the server equipment deployed on the edge node can be used to process the various videos collected by the edge node.

[0029] However, in the related art, for each video frame data in each channel of collected video, a separate business processing module is generally used to process the video frame data, and when processing the video frame data, the business processing module needs to separately apply for the computing resources of the graphics processing unit (GPU) to process the inference request. The inference request can be, for example, as shown in Table 1, and the inference request can include Meta metadata and Data data. Meta metadata can include the name of the deep learning model involved, such as yolov5, and the input interface standard of the deep learning model, such as 684,684,3, which means that the data input to the deep learning model should be a 3-channel 684×684 image data. Meta metadata can be an explanation of Data data. Data data can be represented as video frame data, such as 108,255,0,88 in Table 1 can be represented as part of the video frame data.

[0030] Table 1

[0031]

[0032] Figure 1 The figure schematically shows a video data processing method in the related art.

[0033] like Figure 1As shown, multiple business processing modules may be deployed in the electronic device, for example, a face recognition module, a restricted area recognition module, an action recognition module, etc. Before processing the video frame data, the multiple business processing modules need to decode the collected video respectively to obtain the video frame data. After the business processing module completes the pre-processing of the video frame data, it can assign the pre-processed video frame data to the computing module of the electronic device, and use the computing resources of the computing module to complete the processing of the video frame data. The computing module may include multiple GPU cards, such as GPU card 1, GPU card 2, etc. The functions implemented by the business processing module generally require multiple deep learning models to implement, and the deep learning models required by each business processing module may be different. Therefore, multiple GPU cards need to deploy a full amount of deep learning models.

[0034] Therefore, in the related art, taking the number of business processing modules as M and the total number of deep learning modules as P as an example, to complete the processing of N video frame data, it is necessary to complete M×N decoding operations and deploy N×M×P deep learning models in the computing module. Therefore, the realization of video data processing requires more computing and storage resources, which puts forward higher requirements on the performance of electronic equipment, making it difficult to deploy the electronic equipment at the edge.

[0035] In view of this, an embodiment of the present disclosure provides a video data processing method, by which the usage of computing resources can be effectively reduced, the utilization rate of the graphics processor can be improved, and the device applying the video data processing method can be deployed at the edge. Specifically, the method includes: based on multiple video frame data received within a preset time window, obtaining multiple first tensor data, wherein the first tensor data includes multiple sub-tensor data corresponding to each of the multiple deep learning models; splicing the multiple sub-tensor data included in each of the multiple first tensor data to obtain multiple second tensor data corresponding to each of the multiple deep learning models; and based on the deep learning models deployed by each of the multiple graphics processing units included in the graphics processor, allocating the multiple second tensor data to the multiple graphics processing units, so as to perform data inference on the multiple second tensor data through the graphics processor.

[0036] Figure 2 The exemplary system architecture to which the video data processing method according to the embodiment of the present disclosure can be applied is schematically shown. It should be noted that: Figure 2 What is shown is merely an example of a system architecture to which the embodiments of the present disclosure can be applied, in order to help those skilled in the art understand the technical content of the present disclosure, but it does not mean that the embodiments of the present disclosure cannot be used in other devices, systems, environments or scenarios.

[0037] like Figure 2As shown, the system architecture 200 according to this embodiment may include edge clusters 201 , 202 , a network 203 and a computing center 204 .

[0038] The edge clusters 201 and 202 may include edge devices and one or more camera devices, wherein the one or more camera devices may be used to acquire video streams, and the edge devices may be used to process the video streams acquired by the one or more camera devices. The edge devices may be various electronic devices, including but not limited to smart phones, tablet computers, laptop computers, and desktop computers.

[0039] The network 203 is used to provide a medium for communication links between the edge clusters 201, 202 and the computing center 204. The network 203 may include various connection types, such as wired and / or wireless communication links, etc. The network 203 may also be configured with a gateway and a firewall for access control between the edge clusters 201, 202 and the computing center 204.

[0040] The computing center 204 may include various types of server devices, such as a GPU server, which may be used to provide computing and storage resources for edge devices in the edge clusters 201 and 202 .

[0041] It should be noted that the video data processing method provided in the embodiment of the present disclosure can generally be executed by the computing center 204. Accordingly, the video data processing device provided in the embodiment of the present disclosure can also be arranged in the computing center 204.

[0042] For example, the camera devices included in the edge clusters 201 and 202 can perform video acquisition, and the acquired video stream can be transmitted in real time by the edge devices included in the edge clusters 201 and 202 to the computing center 204. The computing center 204 can decode the video stream to obtain multiple video frame data, and convert the multiple video frame data and splice them based on the deep learning model dimension to obtain multiple second tensor data. The multiple second tensor data can be allocated based on the resource consumption of the GPU cards of the multiple servers deployed in the computing center 204, so as to complete the processing of the second tensor data using the GPU card, thereby realizing the processing of multiple video frame data.

[0043] Alternatively, the video data processing method provided by the embodiment of the present disclosure may also be executed by edge devices in the edge clusters 201 and 202 . Accordingly, the video data processing method provided by the embodiment of the present disclosure may also be set in the edge devices in the edge clusters 201 and 202 .

[0044] It should be understood that Figure 2 The number of edge clusters, computing centers, and networks in the embodiment is only for illustration purposes. Any number of edge clusters, computing centers, and networks may be provided as required.

[0045] Figure 3 The flowchart of the video data processing method according to the embodiment of the present disclosure is schematically shown.

[0046] like Figure 3 As shown, the video data processing method may include operations S310 to S330.

[0047] In operation S310, a plurality of first tensor data are obtained based on a plurality of video frame data received within a preset time window, wherein the tensor data includes a plurality of sub-tensor data corresponding to respective ones of the plurality of deep learning models.

[0048] In operation S320, the multiple sub-tensor data included in each of the multiple first tensor data are spliced ​​to obtain multiple second tensor data corresponding to each of the multiple deep learning models.

[0049] In operation S330, based on the deep learning model respectively deployed by the plurality of graphics processing units included in the graphics processor, the plurality of second tensor data are distributed to the plurality of graphics processing units, so as to perform data inference on the plurality of second tensor data through the graphics processor.

[0050] According to an embodiment of the present disclosure, the preset time window can be represented as a time period with a preset duration, and the window size of the preset time window is the preset duration. The preset duration and the step size of the preset time window can be set according to a specific application scenario. For example, the preset duration can be set to be equal to the step size of the preset time window to avoid repeated processing of video frame data, which is not limited here.

[0051] According to an embodiment of the present disclosure, multiple video frame data may include video frame data from different video streams. For example, video frame data 1 may be one of multiple video frame data obtained by decoding video stream a captured by device A, and video frame data 2 may be one of multiple video frame data obtained by decoding video stream b captured by device B.

[0052] According to an embodiment of the present disclosure, for each video frame data, the video frame data can be processed based on the input requirements of multiple deep learning models to obtain multiple sub-tensor data. Each sub-tensor data can be represented as a matrix. For example, the deep learning model can be an image recognition model that processes RGB images, and the sub-tensor data corresponding to the deep learning model can be represented as a 3×H×W matrix, where 3 represents the value of the channel dimension, H can represent the height of the RGB image, and W can represent the width of the RGB image. For another example, the received video frame can be a grayscale image, and accordingly, its video frame data can be represented as a 1×H×W matrix. Deep learning model A can be a model that processes grayscale graphics, and the video frame data can be directly used as sub-tensor data adapted to the input of deep learning model A. Deep learning model B can be a model that processes RGB images. When generating sub-tensor data adapted to the input of the deep learning model B, the video frame data can be converted into a color domain, and the value of each pixel in the video frame data is converted from a grayscale value to an RGB value to achieve the conversion from the grayscale color domain to the RGB color domain, thereby obtaining the sub-tensor data represented as a 3×H×W matrix.

[0053] According to an embodiment of the present disclosure, multiple sub-tensor data can be spliced ​​to obtain a first tensor data. During splicing, the missing dimensional data in the multiple sub-tensor data can be filled, and the sub-tensor data can be spliced ​​based on an additional dimension. For example, sub-tensor data A can be represented as a 3×H1×W1 matrix, sub-tensor data B can be represented as a 1×H2×W2 matrix, and H1 is greater than H2, W2 is greater than W1, and sub-tensor data A and sub-tensor data B can be filled respectively. The filled sub-tensor data A and the filled sub-tensor data B can both be expressed as a 3×H1×W2 matrix. After splicing the filled sub-tensor data A and the filled sub-tensor data B, the first tensor data obtained can be represented as a 2×3×H1×W2 matrix.

[0054] According to an embodiment of the present disclosure, a video data processing method can be used to implement a variety of business functions, and each business function can be implemented using one or more of a plurality of deep learning models. For each business function, each frame of video frame data can be processed based on the input requirements of one or more deep learning models required to implement the business function to obtain one or more sub-tensor data, and the one or more sub-tensor data can be spliced ​​to obtain a first tensor data. That is, a plurality of first tensor data may include a plurality of first tensor data belonging to different business functions, and the first tensor data corresponding to each business function may include one or more sub-tensor data corresponding to one or more deep learning models. Taking the number of video frame data as N and the number of types of business functions as 2 as an example, the implementation of the first business function requires the support of P1 deep learning models, and the implementation of the second business function requires the support of P2 deep learning models. Based on N video frame data, 2×N first tensor data can be obtained. Each of the N first tensor data corresponding to the first business function can include P1 sub-tensor data, and each of the N first tensor data corresponding to the second business function can include P2 sub-tensor data.

[0055] According to an embodiment of the present disclosure, the multiple sub-tensor data included in each of the multiple first tensor data are spliced, and the multiple sub-tensor data included in each of the multiple first tensor data can be classified based on the deep learning model as a dimension. Each sub-tensor data can carry a model label, and the deep learning model associated with the sub-tensor data can be determined through the model label, thereby completing the classification of the sub-tensor data. The multiple sub-tensor data included in each category can be suitable for processing by a single deep learning model. The multiple sub-tensor data included in each category can be spliced ​​to obtain second tensor data. That is, the second tensor data obtained after classification can be processed by a single deep learning model.

[0056] According to an embodiment of the present disclosure, a graphics processor may include multiple image processing units, each of which may be represented as a GPU card, and allocating multiple second tensor data may be allocating multiple second tensor data to multiple GPU cards for processing. One or more deep learning models may be deployed in each GPU card, and after receiving the second tensor data, the GPU card may use the corresponding deep learning model deployed in the GPU card to process the second tensor data to obtain a processing result of the second tensor data.

[0057] According to an embodiment of the present disclosure, the manner of distributing the plurality of second tensor data to the plurality of image processing units may include but is not limited to average distribution, low-fragmentation distribution, and the like. Average distribution may be a distribution manner that makes the loads of the plurality of GPU cards after distribution substantially consistent. Low-fragmentation distribution may be a distribution manner that preferentially uses non-idle GPU cards for data processing.

[0058] According to an embodiment of the present disclosure, each sub-tensor data in the second tensor data may also carry a video frame label. Through the video frame label, the video frame data associated with the sub-tensor data can be determined, so that the processing result of the video frame data can be determined based on the processing result of the second tensor data.

[0059] According to an embodiment of the present disclosure, all video frame data received within a time window can be converted into a plurality of first tensor data, each of which can include a plurality of sub-tensor data, and each sub-tensor data can be obtained by processing according to the input requirements of the corresponding deep learning model. The plurality of sub-tensor data included in each of the plurality of first tensor data can be spliced ​​to obtain a plurality of second tensor data. When processing data, the plurality of second tensor data can be task-assigned in the image processor based on a load balancing strategy to complete the processing of the video frame data. By splicing the sub-tensor data belonging to the same deep learning model into a second tensor data, the batch size of the data can be increased, so that the batch processing capability of the graphics processor can be effectively utilized, the throughput of the graphics processor can be increased, and the utilization rate of the graphics processor can be improved. At the same time, the number of deep learning models that need to be deployed in the memory of the graphics processor can be reduced, and the resource consumption of the graphics processor can be reduced, so that small edge devices can also realize the processing of video data.

[0060] Reference below Figure 4 , combined with specific embodiments Figure 3 The video data processing method shown is further explained.

[0061] According to an embodiment of the present disclosure, in response to receiving a video stream, the video stream may be decoded to obtain video frame data.

[0062] According to an embodiment of the present disclosure, a video stream can be obtained by a camera device after acquisition and encoding. The video stream can be represented as a segment of video, and correspondingly, the video frame data is obtained by decoding the segment of video. There can be multiple camera devices, and correspondingly, there can be multiple video streams, and the multiple video frame data can include video frame data obtained by respectively decoding multiple video streams.

[0063] According to an embodiment of the present disclosure, for each of the multiple video frame data received within a preset time window, the video frame data can be converted into at least one first tensor data based on at least one loaded dynamic link library. Based on the at least one first tensor data converted from each of the multiple video frame data, multiple first tensor data are obtained.

[0064] According to an embodiment of the present disclosure, a dynamic link library can be represented as a decoupled library file, for example, it can be represented as a file with a .so suffix on Linux. The dynamic link library can support various business interface methods, and the business interface methods may include init, stop, start, pause, resume, etc. During the initialization phase, the dynamic link library can be loaded into the memory. By loading the dynamic link library into the memory of the device, various business interface methods can be used to implement the call of the business function represented by the dynamic link library.

[0065] According to an embodiment of the present disclosure, a dynamic link library may be associated with at least one target deep learning model, and at least one target deep learning model may belong to multiple deep learning models. Specifically, at least one target deep learning model may be called in the dynamic link library to implement the business function represented by the dynamic link library.

[0066] According to an embodiment of the present disclosure, converting the video frame data into the first tensor data may be to convert the video frame data according to the input interface standards of the respective deep learning models to obtain at least one first tensor data. Specifically, for each video frame data, based on at least one loaded dynamic link library, converting the video frame data into at least one first tensor data may include the following operations:

[0067] For each video frame data among multiple video frame data received within a preset time window, generate at least one business instance based on at least one dynamic link library; run at least one business instance, wherein the business instance is configured to load the video frame data, and process the video frame data based on the input interface standard of each of at least one target deep learning model related to the business instance to obtain at least one sub-tensor data; and obtain at least one first tensor data based on at least one sub-tensor data corresponding to each of the at least one business instance.

[0068] According to the embodiments of the present disclosure, a business instance can be represented as a task, and by executing the business instance, a business function corresponding to the dynamic link library can be realized. Accordingly, generating a business instance based on a dynamic link library can be to create a new task according to the requirements described in the dynamic link library.

[0069] According to an embodiment of the present disclosure, taking the deep learning model as an image processing model as an example, the input interface standard of the image processing model can indicate that the input data is single-channel grayscale image data. The video frame data to be processed can be three-channel RGB image data. The video frame data can be processed based on the input interface standard of the image processing model, and the color space transformation can be performed on the video frame data to be processed, and the RGB value of each pixel in the video frame data can be converted into a grayscale value to obtain the video frame data to be processed, that is, the sub-tensor data corresponding to the image processing model.

[0070] According to the embodiments of the present disclosure, by completing video data processing based on a dynamic link library, decoupling between business functions and business processes can be achieved. When faced with new business function requirements, developers write the new business function requirements into a separate dynamic link library according to a template, and then write the dynamic link library into the memory to realize the launch of the new business function, thereby effectively improving business development efficiency.

[0071] According to an embodiment of the present disclosure, the decoded video frame data can be temporarily stored so as to realize the multiplexing of the video frame data by at least one service instance. Specifically, in response to receiving the video frame data, the video frame data can be written into a storage unit. After the generation and startup of at least one service instance are completed, the service instance can load the video frame data from the storage unit.

[0072] According to the embodiments of the present disclosure, the storage unit may refer to a memory, a hard disk, a database or the like, which is not limited here.

[0073] According to an embodiment of the present disclosure, the multiplexing of the video frame data by at least one service instance can be realized by using a reference counter. Specifically, after the video frame data is written into a storage unit, a data packet can be obtained, the video frame data can be stored in the body of the data packet, and the header of the data packet can record fields related to the video frame data. The count field of the video frame data can be configured based on the number of at least one dynamic link library, and the count field is the reference counter. For example, a count field can be added to a field in the body of the data packet corresponding to the video frame data, or a count field can be added to the header of the data packet corresponding to the video frame data. In response to detecting that the service instance loads the video frame data from the storage unit, the value of the count field of the video frame data can be updated. When the value of the count field of the video frame data is a preset value, the video frame data is released from the storage unit.

[0074] According to an embodiment of the present disclosure, updating the value of the count field of the video frame data may be to increase or decrease the value of the count field according to the data update step. For example, the data update step may be set to 1, and updating the value of the count field of the video frame data may be to reduce the value of the original count field by 1.

[0075] According to an embodiment of the present disclosure, the preset value may be set according to a specific application scenario, for example, may be set to 0. When the value of the count field of the video frame data is the preset value, it may indicate that the video frame data has been called a predetermined number of times.

[0076] According to an embodiment of the present disclosure, releasing the video frame data from the storage unit may be performed by physically deleting the video frame data from the storage unit.

[0077] According to an embodiment of the present disclosure, configuring the count field of the video frame data may be to add a count field to the video frame data, and the count field may be represented as a parameter whose value is empty. The initial value of the count field may be determined based on the number of at least one dynamic link library. Specifically, the initial value of the count field may be determined based on the data update step, the total number of dynamic link libraries, and the preset value. For example, if the number of dynamic link libraries is a, the data update step is b, and the preset value is c, then the initial value of the count field may be determined to be c+a×b.

[0078] For example, after receiving the decoded video frame data, the video frame data can be packaged into a data packet, the data packet is written into the memory, and a field ref_counter is added to the data packet, and the initial value of the field ref_counter is set according to the number of generated business instances. If the number of generated business instances is 3, the initial value of the field ref_counter can be set to 3. After starting the business instance, a thread of the business processor can be used to run a business instance. A smart pointer can be configured in each thread to point to the data packet in the memory. Alternatively, a smart pointer can be configured in the business processor, and the business instance in each thread of the business processor can call the smart pointer configured in the business processor. After the business instance is started, the business instance can determine the address of the data packet in the memory by calling the smart pointer, thereby realizing the loading of the data packet. After waiting for the business instance to load the data packet to reuse the video frame data, the value of the field ref_counter can be controlled to be reduced by 1. When the value of the field ref_counter is 0, the data packet in the memory can be released to destroy the corresponding video frame data.

[0079] According to an embodiment of the present disclosure, by storing video frame data in a storage unit, at least one service instance can reuse the video frame data through a pointer. By adding a reference counter to the video frame data, when the service instance uses the video frame data through a pointer, the value of the reference counter can change accordingly, and when the value of the reference counter is a preset value, the video frame data can be released. Thus, the reuse of video frame data can be achieved, and each service instance does not need to repeatedly decode the video stream when using the video frame data, thereby reducing the number of decoding operations and saving computing resources. At the same time, the storage unit does not need to store the video frame data generated by repeated decoding, saving storage resources.

[0080] According to an embodiment of the present disclosure, at least one first tensor data may be generated for each video frame data using at least one business instance, and the multiple first tensor data may include at least one first tensor data of each of the multiple video frame data. Specifically, the at least one first tensor data associated with each of the multiple video frame data may be combined to obtain multiple first tensor data.

[0081] According to an embodiment of the present disclosure, the preset time window may include multiple sub-windows, and within each sub-window, one or more first tensor data output by all completed business instances may be packaged into a request, the request may be packaged into a message, and the one or more first tensor data may be used as message body data of the message. The request may be sent to a management module of a graphics processor so that the management module performs resource scheduling to give full play to the processing capability of the graphics processor for large batches of data.

[0082] According to an embodiment of the present disclosure, the multiple sub-tensor data included in each first tensor data can be respectively suitable for processing by different deep learning models. The sub-tensor data suitable for processing by the deep learning model in the multiple first tensor data can be classified into one category with the deep learning model as the dimension. All the sub-tensor data included in each category can be combined to obtain a large-size, high-dimensional second tensor data. The management module of the image processor can perform resource scheduling for the large-size, high-dimensional second tensor data. Compared with the related art, for each received video frame data, the video frame data needs to be packaged into a message, and the message is sent to the management module of the graphics processor, and the management module needs to schedule each video frame data separately. Since the graphics processor needs to generate processing tasks for the deployed deep learning models after receiving the data to be processed, and in view of the fact that the graphics processor has a strong large-batch data processing capability, the processing time and resource occupancy of each processing task are poorly correlated with the amount of data to be processed. That is, in each processing task, the time and resources consumed for processing small-sized single video frame data and large-sized, high-dimensional second tensor data are basically the same. Therefore, the method of splicing multiple groups of first tensor data to obtain multiple second tensor data and performing resource scheduling on the multiple second tensor data can give full play to the graphics processor's processing capabilities for large batches of data.

[0083] According to an embodiment of the present disclosure, the steps performed by the management module of the graphics processor to distribute the plurality of second tensor data to the plurality of graphics processing units based on the deep learning models respectively deployed by the plurality of graphics processing units included in the graphics processor may include the following operations:

[0084] For each second tensor data, based on the deep learning model corresponding to the second tensor data and the deep learning models respectively deployed by the plurality of graphics processing units, at least one target graphics processing unit is determined from the plurality of graphics processing units, wherein the at least one deep learning model deployed by the target graphics processing unit includes the deep learning model corresponding to the second tensor data. The second tensor data is distributed to the at least one target graphics processing unit.

[0085] According to an embodiment of the present disclosure, for example, GPU A, GPU B, and GPU C may be deployed with model a, and when allocating the second tensor data related to model a, GPU A, GPU B, and GPU C may be the determined target GPUs. Accordingly, the second tensor data related to model a may be allocated to GPU A, GPU B, and GPU C for processing.

[0086] According to an embodiment of the present disclosure, allocating the second tensor data to at least one target graphics processing unit may include the following operations:

[0087] Based on the resource margin of at least one target graphics processing unit, the second tensor data is segmented with sub-tensor data as the granularity to obtain at least one sub-data, and the at least one sub-data is allocated to at least one target graphics processing unit.

[0088] According to an embodiment of the present disclosure, segmenting the second tensor data with sub-tensor data as the granularity can be to use the sub-tensor data as the smallest unit for segmentation, that is, the sub-tensor data included in the second tensor data after segmentation are all complete, and each sub-data can include one or more sub-tensor data.

[0089] For example, at least one target graphics processing unit includes graphics processing unit A, graphics processing unit B, graphics processing unit C, and graphics processing unit D. The resource margins of graphics processing unit A, graphics processing unit B, graphics processing unit C, and graphics processing unit D are 70%, 80%, 100%, and 50%, respectively. The second tensor data may include 100 sub-tensor data. The processing capabilities of graphics processing unit A, graphics processing unit B, graphics processing unit C, and graphics processing unit D may be the same, and the processing of each sub-tensor data requires 1% of the resources of the graphics processing unit.

[0090] When the second tensor data is split based on the balanced allocation strategy, the second tensor data can be split into three sub-data, namely sub-data a, sub-data b and sub-data c, sub-data a can include 20 sub-tensor data, sub-data b can include 30 sub-tensor data, and sub-data c can include 50 sub-tensor data. Sub-data a can be allocated to graphics processing unit A for processing, sub-data b can be allocated to graphics processing unit B for processing, sub-data c can be allocated to graphics processing unit C for processing, and graphics processing unit D is not allocated. After completing the data allocation, the resource margins of graphics processing unit A, graphics processing unit B, graphics processing unit C and graphics processing unit D are all 50%, so that the load of each graphics processing unit can be guaranteed to be basically consistent.

[0091] When the second tensor data is split based on the anti-fragmentation allocation strategy, the second tensor data can be split into 2 sub-data, namely sub-data d and sub-data e, and sub-data d and sub-data e can each include 50 sub-tensor data. Sub-data d can be allocated to graphics processing unit D, sub-data e can be allocated to graphics processing unit A, and graphics processing unit B and image processing unit C are not allocated. After the data allocation is completed, the resource margin of graphics processing unit A is 20%, the resource margin of graphics processing unit B is 80%, the resource margin of graphics processing unit C is 100%, and the resource margin of graphics processing unit D is 0%, thereby ensuring the integrity of the resources of the graphics processing unit, so that when a large batch of data is received, the graphics processing unit with complete resources can be used to process the large batch of data to improve the data reasoning efficiency of the image processing unit.

[0092] Figure 4 The diagram schematically shows a video data processing method according to an embodiment of the present disclosure.

[0093] like Figure 4 As shown, the video data processing method can decode the video stream from the cameras configured in various business systems to obtain video frame data. Various business systems may include, for example, access control systems, confidential area management systems, road violation detection systems, etc. Taking the access control system as an example, the access control system may be a system for identifying and verifying the identities of people entering and leaving various types of parks. The camera configured in the access control system can be used to obtain information such as portraits and identity credentials. Accordingly, the decoded video frame data may be image data containing information such as portraits and identity credentials. By using the video data processing method of the embodiment of the present disclosure to process the video acquired by the access control system, the video frame data acquired by each of the multiple cameras may be processed in parallel to improve the processing efficiency of the video frame data.

[0094] According to an embodiment of the present disclosure, the video frame data received within the preset time window may include video frame data video1 and video frame data video2.

[0095] According to the embodiments of the present disclosure, the business functions that the access control system needs to implement may include a face recognition function and a restricted area management function. Accordingly, for each video frame data, two business instances corresponding to the face recognition function and the restricted area management function may be generated. Specifically, instance A1 and instance B1 corresponding to the video frame data video1 may be generated, and instance A2 and instance B2 corresponding to the video frame data video2 may be generated.

[0096] According to an embodiment of the present disclosure, each instance can process the corresponding video frame data based on the deep learning model required to realize the corresponding business function to obtain the first tensor data. For example, the realization of the face recognition function requires the application of model α and model β, and the realization of the restricted area management function requires the application of model γ. Accordingly, instance A1 processes the video frame data video1 to obtain the first tensor data a1, which may include sub-tensor data α_a1 determined based on the input interface standard of model α and sub-tensor data β_a1 determined based on the input interface standard of model β. Instance A2 processes the video frame data video2 to obtain the first tensor data a2, which may include sub-tensor data α_a2 determined based on the input interface standard of model α and sub-tensor data β_a2 determined based on the input interface standard of model β. Instance B1 processes the video frame data video1 to obtain the first tensor data b1, which may include sub-tensor data γ_b1 determined based on the input interface standard of model γ. Example B2 processes the video frame data video2 to obtain the first tensor data b2, which may include sub-tensor data γ_b2 determined based on the model γ.

[0097] According to an embodiment of the present disclosure, the sub-tensor data can be aggregated and spliced ​​based on the dimension of the deep learning model to obtain the second tensor data. Specifically, the sub-tensor data α_a1 and the sub-tensor data α_a2 can be spliced ​​to obtain the second tensor data α_a, the sub-tensor data β_a1 and the sub-tensor data β_a2 can be spliced ​​to obtain the second tensor data β_a, and the sub-tensor data γ_b1 and the sub-tensor data γ_b2 can be spliced ​​to obtain the second tensor data γ_b.

[0098] According to an embodiment of the present disclosure, a graphics processor may include a GPU card 1 and a GPU card 2, a model α, a model β, and a model γ may be deployed in the GPU card 1, and a model γ may be deployed in the GPU card 2. Therefore, the target graphics processing unit corresponding to the second tensor data α_a is the GPU card 1, the target graphics processing unit corresponding to the second tensor data β_a is the GPU card 1, and the target graphics processing units corresponding to the second tensor data γ_b are the GPU card 1 and the GPU card 2.

[0099] According to an embodiment of the present disclosure, when allocating the second tensor data, the second tensor data α_a and the second tensor data β_a may be allocated to GPU card 1. Based on the resource margins of GPU card 1 and GPU card 2, the second tensor data γ_b may be split into two sub-data, and the two sub-data may be allocated to GPU card 1 and GPU card 2, respectively.

[0100] According to an embodiment of the present disclosure, the video data processing method as described above may be implemented using an edge device.

[0101] Figure 5 A schematic diagram of an edge device according to an embodiment of the present disclosure is schematically shown.

[0102] like Figure 5 As shown, the edge device 500 may include a business plug-in manager 510, an inference manager 520, and a model manager 530. The edge device 500 may be represented as a single electronic device, such as a GPU server. Alternatively, the edge device 500 may also include multiple electronic devices, such as multiple GPU servers, the business plug-in manager 510 may be deployed on one or more GPU servers, and the inference manager 520 and the model manager 530 may be deployed on another GPU server.

[0103] According to an embodiment of the present disclosure, the business plug-in manager 510 can connect multiple cameras through the external interface of the edge device. For each video stream generated by the camera, the business plug-in manager 510 can be configured to decode the video stream in response to receiving the video stream to obtain video frame data.

[0104] According to an embodiment of the present disclosure, the service plug-in manager 510 may be configured to obtain a plurality of first tensor data based on a plurality of video frame data received within a preset time window. The first tensor data includes a plurality of sub-tensor data corresponding to each of the plurality of deep learning models.

[0105] According to an embodiment of the present disclosure, the inference manager 520 may be configured to: concatenate multiple sub-tensor data included in each of the multiple first tensor data to obtain multiple second tensor data corresponding to each of the multiple deep learning models.

[0106] According to an embodiment of the present disclosure, the service plug-in manager 510 may be configured with multiple service plug-ins, such as a face recognition service plug-in 511, a restricted area recognition service plug-in 512, a motion recognition service plug-in 513, etc. The service plug-in may be represented as a dynamic link library written into the memory.

[0107] According to an embodiment of the present disclosure, the inference manager 520 may be configured to distribute a plurality of second tensor data to a plurality of graphics processing units included in the graphics processor based on at least one deep learning model deployed by each of the plurality of graphics processing units.

[0108] According to an embodiment of the present disclosure, the model manager 530 may be configured to: perform data reasoning on the plurality of second tensor data through a graphics processor.

[0109] According to an embodiment of the present disclosure, a model manager 530 may be used to manage deep learning models that need to be deployed in an image processor. Deep learning models may include, for example, a yolov5m model 531, a resnet model 532, a face model 533, a keypoint model 534, a restrict model 535, a motion model 536, a mask model 537, a quality module 538, and the like. A combination of one or more of the above deep learning models may be used to implement different business functions. For example, a face recognition function in an access control system may require a resnet model 532 and a face model 533 to implement. The number of deep learning models that a model manager may be able to manage may be set according to a specific application scenario and is not limited here.

[0110] According to an embodiment of the present disclosure, the model manager 530 may also be used to manage a plurality of graphics processing units 541 included in the graphics processor 540 , such as GPU card 1 , GPU card 2 , and the like.

[0111] According to an embodiment of the present disclosure, one or more deep learning models may be pre-deployed in each graphics processing unit 541. For example, a yolov5m model 531, a resnet model 532, and a keypoint model 534 may be deployed in GPU card 1, and a resnet model 532, a motion model 536, and a mask model 537 may be deployed in GPU card 1. The model manager 530 may maintain a mapping table, which may include a mapping relationship between a graphics processing unit 541 and a deep learning model, that is, a deep learning model that needs to be deployed in the graphics processing unit 541. After the edge device 500 starts working, the model manager 530 may deploy a deep learning model in each graphics processing unit 541 according to the mapping table.

[0112] According to an embodiment of the present disclosure, when allocating the second tensor data, the inference manager 520 may allocate the second tensor data to the corresponding graphics processing unit based on the deep learning model corresponding to the second tensor data. For example, if the deep learning model corresponding to the second tensor data A is the yolov5m model 541, the inference manager 520 may allocate the second tensor data A to GPU card 1. For another example, if the deep learning model corresponding to the second tensor data B is the resnet model 542, the inference manager 520 may allocate the second tensor data B to GPU card 1 and GPU card 2 respectively based on the resource margins of GPU card 1 and GPU card 2.

[0113] According to an embodiment of the present disclosure, by splicing sub-tensor data belonging to the same deep learning model into a second tensor data, the batch size of the data can be increased, so that the batch processing capability of the graphics processor can be effectively utilized, the throughput of the graphics processor can be increased, and the utilization rate of the graphics processor can be improved. At the same time, the number of deep learning models that need to be deployed in the memory of the graphics processor can be reduced, and the resource consumption of the graphics processor can be reduced. In turn, the edge device 500 can be deployed at the edge end.

[0114] Reference below Fig. 6A , Figure 6B and Figure 7 to Figure 9 , combined with specific embodiments Figure 5 The edge device shown is further explained.

[0115] According to an embodiment of the present disclosure, the edge device 500 may further include a frame multiplexer 550. The frame multiplexer 550 may be disposed between the external interface of the edge device 500 and the service plug-in manager 510. The external interface of the edge device 500 may be connected to multiple cameras, and each camera may provide a video stream to the edge device 500. The number of cameras connected to the external interface of the edge device 500 may be set according to the specific application scenario, and is not limited here. For example, in the application scenario of a campus access control system, the campus may include 4 entrances and exits, and each entrance and exit may be provided with 2 cameras, and the number of cameras connected to the external interface of the edge device 500 may be 8.

[0116] Fig. 6A A schematic diagram of a frame multiplexer according to an embodiment of the present disclosure is schematically shown.

[0117] like Fig. 6A As shown, the frame multiplexer 550 can be configured to: in response to receiving a video stream, decode the video stream to obtain video frame data. Specifically, the frame multiplexer 550 can decode the video stream from camera 1 into video frame data frame1, can decode the video stream from camera 2 into video frame data frame2, and can decode the video stream from camera 3 into video frame data frame3.

[0118] According to an embodiment of the present disclosure, the frame multiplexer 550 may be configured to: in response to receiving the video frame data, write the video frame data into the storage unit 551 .

[0119] According to an embodiment of the present disclosure, the frame multiplexer 550 may be configured to: configure a count field of the video frame data based on the number of at least one dynamic link library. Specifically, the frame multiplexer 550 may be configured to: add a count field to the video frame data. Determine an initial value of the count field based on the number of at least one dynamic link library. For example, the number of dynamic link libraries may be 3, and a count field count may be configured for each video frame data, respectively. The initial values ​​of the count fields count of the video frame data frame1, the video frame data frame2, and the video frame data frame3 may all be set to 3.

[0120] According to an embodiment of the present disclosure, the service plug-in manager 510 can generate service instances based on the configured service plug-ins. For example, the face recognition service plug-in 511 configured by the service plug-in manager 510 can be used to generate a face recognition service instance, the restricted area recognition service plug-in 512 configured by the service plug-in manager 510 can be used to generate a restricted area recognition service instance, and the motion recognition service plug-in 511 configured by the service plug-in manager 510 can be used to generate a motion recognition service instance. The face recognition service instance, the restricted area recognition service instance, and the motion recognition service instance can load video frame data frame1, video frame data frame2, and video frame data frame3 from the storage unit, respectively.

[0121] According to an embodiment of the present disclosure, the frame multiplexer 550 can be configured to: update the value of the count field of the video frame data in response to detecting that a business instance loads video frame data from a storage unit; and release the video frame data from the storage unit when the value of the count field of the video frame data is a preset value.

[0122] Figure 6B A schematic diagram schematically shows a video frame multiplexing workflow of a frame multiplexer according to an embodiment of the present disclosure.

[0123] like Figure 6BAs shown, the video stream collected by the camera can enter the frame multiplexer 550 through the gateway. After the frame multiplexer 550 parses the video stream, video frame data can be obtained. The frame multiplexer 550 can store the video frame data in the memory and generate a data structure corresponding to the video frame data in the memory. The data structure may include two parts, one of which includes the memory address and reference count field refcounter of the video frame data, and the other part is the memory data of the video frame data. The service instance generated by the service plug-in manager 510 can call the data structure and find the reference count field refcounter and memory address of the video frame data according to the data structure. Based on the memory address, the service plug-in manager 510 can implement the call and processing of the video frame data. Specifically, each service instance can complete the call of the video frame data through a separate thread. For example, service instance 1 calls frame1 through thread 1, and service instance 2 calls frame1 through thread 2. After detecting that the business plug-in manager 510 has completed the call of the video frame data, the frame multiplexer 550 can reduce the value of the reference count field refcounter in the data structure by 1. When the value of the reference count field refcounter is less than 1, the frame multiplexer 510 can recycle the data structure, that is, delete the video frame data from the memory.

[0124] Figure 7 The diagram schematically shows a service plug-in manager according to an embodiment of the present disclosure.

[0125] like Figure 7 As shown, the service plug-in manager 510 can be configured with multiple service interfaces. By calling the service interface, the service instance can be processed using the service interface method corresponding to the service interface. The multiple service interfaces can include an init interface, a stop interface, a start interface, a pause interface, a resume interface, etc. For example, when the stop interface is called, the service instance can be controlled to stop running. For another example, when the start interface is called, the service instance can be controlled to start running.

[0126] According to an embodiment of the present disclosure, the service plug-in manager 510 can manage multiple service plug-ins, which can include a face recognition service plug-in 511, a restricted area recognition service plug-in 512, a motion recognition service plug-in 513, etc. Each service plug-in can be represented as a loaded dynamic link library.

[0127] According to an embodiment of the present disclosure, the service plug-in manager 510 may be configured to: for each video frame data of a plurality of video frame data received within a preset time window, based on at least one loaded dynamic link library, convert the video frame data into at least one first tensor data. Based on at least one first tensor data converted from each of the plurality of video frame data, a plurality of first tensor data are obtained. Specifically, each dynamic link library may be associated with at least one target deep learning model, and the at least one target deep learning model belongs to a plurality of deep learning models.

[0128] According to an embodiment of the present disclosure, the business plug-in manager 510 can be specifically configured to: generate at least one business instance based on at least one dynamic link library in response to receiving video frame data; run at least one business instance, wherein the business instance is configured to load video frame data, specifically, the video frame data can be loaded from the storage unit 551, and the video frame data can be processed based on the input interface standard of each of at least one target deep learning model related to the business instance to obtain at least one sub-tensor data; and obtain at least one first tensor data based on at least one sub-tensor data corresponding to each of the at least one business instance.

[0129] According to an embodiment of the present disclosure, a mapping table may be maintained for each dynamic link library in the service plug-in manager 510, and the mapping table may include at least one deep learning model associated with the dynamic link library and the input interface standard of at least one deep learning model. When the service instance is running, for each video frame data, the service instance may be configured to process the video frame data based on the input interface standard of at least one deep learning model included in the mapping table, obtain at least one sub-tensor data, and splice the at least one sub-tensor data to obtain the first tensor data.

[0130] According to an embodiment of the present disclosure, a service instance can be represented as a program that can be run based on a service plug-in. The service instance can be run in a thread of the service plug-in manager 510. After the service instance is run, the service instance can be released from the thread of the service plug-in manager 510.

[0131] For example, for video frame data frame1, video frame data frame2 and video frame data frame3, the service plug-in manager 510 can generate face recognition service instance 1, face recognition service instance 2 and face recognition service instance 3 corresponding to the face recognition service plug-in, generate restricted area recognition service instance 1, restricted area recognition service instance 2 and restricted area recognition service instance 3 corresponding to the restricted area recognition service plug-in, and generate motion recognition service instance 1, motion recognition service instance 2 and motion recognition service instance 3 corresponding to the motion recognition service plug-in.

[0132] According to an embodiment of the present disclosure, the service plug-in manager 510 and the frame multiplexer 550 may be configured to be set in the CPU device of the edge device, and the reasoning manager 520, the model manager 530 and the graphics processor 540 may be configured to be set in the GPU device of the edge device. The CPU device and the GPU device of the edge device may communicate in the form of a request message.

[0133] Figure 8 The following is a schematic diagram schematically showing a communication process between a business plug-in manager and an inference manager according to an embodiment of the present disclosure.

[0134] like Figure 8 As shown, with time as the axis, the preset time window may include multiple sub-windows, and in each sub-window, the business plug-in manager 510 may package all first tensor data generated in the sub-window into one request.

[0135] For example, in sub-window 1, only face recognition service instance 1 in service plug-in manager 510 has completed operation, then service plug-in manager 510 can package the first tensor data output by face recognition service instance 1 as request req1. In sub-window 2, face recognition service instance 2 and face recognition service instance 3 in service plug-in manager 510 have completed operation, then service plug-in manager 510 can package the first tensor data output by face recognition service instance 2 and face recognition service instance 3 as request req2. In sub-window 3, restricted area recognition service instance 1, restricted area recognition service instance 2, restricted area recognition service instance 3, motion recognition service instance 1, motion recognition service instance 2 and motion recognition service instance 3 in service plug-in manager 510 have completed operation, then service plug-in manager 510 can package the first tensor data output by restricted area recognition service instance 1, restricted area recognition service instance 2, restricted area recognition service instance 3, motion recognition service instance 1, motion recognition service instance 2 and motion recognition service instance 3 as request req3.

[0136] According to an embodiment of the present disclosure, the inference manager 520 can be configured to use the deep learning model as a dimension, and for requests generated by each sub-window of a preset time window, the multiple sub-tensor data included in each request can be classified. Each sub-tensor data can carry a model label, through which the deep learning model associated with the sub-tensor data can be determined, thereby completing the classification of the sub-tensor data. The multiple sub-tensor data included in each category can be suitable for processing by a single deep learning model. The multiple sub-tensor data included in each category can be spliced ​​to obtain second tensor data. That is, the second tensor data obtained after classification can be processed by a single deep learning model.

[0137] For example, the multiple sub-tensor data included in the request req1 can be respectively classified into the second tensor data batch1 and the second tensor data batch3. The multiple sub-tensor data included in the request req2 can be respectively classified into the second tensor data batch1 and the second tensor data batch3. The multiple sub-tensor data included in the request req3 can be respectively classified into the second tensor data batch2 and the second tensor data batch3.

[0138] According to an embodiment of the present disclosure, the inference manager 520 may be configured to: for each second tensor data, based on a deep learning model corresponding to the second tensor data and deep learning models respectively deployed by a plurality of graphics processing units, determine at least one target graphics processing unit from a plurality of graphics processing units, wherein the deep learning model deployed by the target graphics processing unit includes the deep learning model corresponding to the second tensor data; and allocate the second tensor data to the at least one target graphics processing unit. Specifically, the inference manager 520 may be configured to: based on the respective resource margins of the at least one target graphics processing unit, split the second tensor data with sub-tensor data as the granularity to obtain at least one sub-data; and allocate the at least one sub-data to the at least one target graphics processing unit respectively.

[0139] For example, the inference manager 520 may be configured to distribute the second tensor data batch1 to GPU card 1 , distribute the second tensor data batch2 to GPU card 2 , and distribute the second tensor data batch3 to GPU card 3 and GPU card 4 .

[0140] Fig. 9 The figure schematically shows a schematic diagram of a model manager according to an embodiment of the present disclosure.

[0141] like Fig. 9 As shown, the model manager 530 can be used to manage various deep learning models and deploy the deep learning models on various graphics processing units in the graphics processor. After the deep learning model is deployed on the graphics processing unit, the graphics processing unit can generate an instance corresponding to the deep learning model. After receiving the second tensor data, the graphics processing unit can run the instance corresponding to the deep learning model in the thread of the graphics processing unit to perform data reasoning on the second tensor data.

[0142] For example, four deep learning models, namely restrict, motion, mask and quality, may be deployed in the graphics processing unit GPU1. After the deployment is completed, the graphics processing unit GPU1 may generate instances instance1 and instance2 of the restrict model, instances instance1 and instance2 of the motion model, instances instance1 and instance2 of the mask model, and instances instance1 and instance2 of the quality model. The second tensor data may be data that needs to be processed by the motion model. After the second tensor data is input into the graphics processing unit GPU1, the two threads of the graphics processing unit GPU1 corresponding to the instances instance1 and instance2 of the motion model may load the second tensor data respectively, and run the method of the motion model to process the second tensor data.

[0143] Fig.10 The block diagram schematically shows a video processing device according to an embodiment of the present disclosure.

[0144] like Fig.10 As shown, the video processing device 1000 may include a first processing module 1010 , a second processing module 1020 and an allocation module 1030 .

[0145] The first processing module 1010 is used to obtain multiple first tensor data based on multiple video frame data received within a preset time window, wherein the first tensor data includes multiple sub-tensor data corresponding to each of the multiple deep learning models.

[0146] The second processing module 1020 is used to splice the multiple sub-tensor data included in each of the multiple first tensor data to obtain multiple second tensor data corresponding to each of the multiple deep learning models.

[0147] The allocation module 1030 is used to allocate multiple second tensor data to multiple graphics processing units based on the deep learning models deployed by the multiple graphics processing units included in the graphics processor, so as to perform data inference on the multiple second tensor data through the graphics processor.

[0148] According to an embodiment of the present disclosure, the video processing apparatus 1000 may further include a conversion module and a third processing module.

[0149] The conversion module is used to convert each video frame data among multiple video frame data received within a preset time window into at least one first tensor data based on at least one loaded dynamic link library.

[0150] The third processing module is used to obtain multiple first tensor data based on at least one first tensor data converted from each of the multiple video frame data.

[0151] According to an embodiment of the present disclosure, the dynamic link library is related to at least one target deep learning model, and the at least one target deep learning model belongs to multiple deep learning models.

[0152] According to an embodiment of the present disclosure, the conversion module includes a first conversion unit, a second conversion unit and a third conversion unit.

[0153] The first conversion unit is used to generate at least one service instance for each video frame data based on at least one dynamic link library.

[0154] The second conversion unit is used to run at least one business instance, wherein the business instance is configured to load video frame data, process the video frame data based on the input interface standard of at least one target deep learning model related to the business instance, and obtain at least one sub-tensor data.

[0155] The third conversion unit is used to obtain at least one first tensor data based on at least one sub-tensor data corresponding to at least one business instance.

[0156] According to an embodiment of the present disclosure, the video processing apparatus 1000 may further include a writing module.

[0157] The writing module is used for writing the video frame data into the storage unit in response to receiving the video frame data.

[0158] According to an embodiment of the present disclosure, the service instance is configured to load video frame data from a storage unit.

[0159] According to an embodiment of the present disclosure, the video processing apparatus 1000 may further include a configuration module, an update module, and a release module.

[0160] The configuration module is used to configure the count field of the video frame data based on the quantity of at least one dynamic link library.

[0161] The updating module is used to update the value of the count field of the video frame data in response to detecting that the service instance loads the video frame data from the storage unit.

[0162] The release module is used to release the video frame data from the storage unit when the value of the count field of the video frame data is a preset value.

[0163] According to an embodiment of the present disclosure, the configuration module includes a first configuration unit and a second configuration unit.

[0164] The first configuration unit is used to add a count field to the video frame data.

[0165] The second configuration unit is used to determine an initial value of the count field based on the number of at least one dynamic link library.

[0166] According to an embodiment of the present disclosure, the allocation module 1030 includes a first allocation unit and a second allocation unit.

[0167] The first allocation unit is used to determine, for each second tensor data, at least one target graphics processing unit from a plurality of graphics processing units based on a deep learning model corresponding to the second tensor data and deep learning models respectively deployed by the plurality of graphics processing units, wherein the deep learning model deployed by the target graphics processing unit includes the deep learning model corresponding to the second tensor data.

[0168] The second distribution unit is configured to distribute the second tensor data to at least one target graphics processing unit.

[0169] According to an embodiment of the present disclosure, the second allocating unit includes a first allocating subunit and a second allocating subunit.

[0170] The first allocation subunit is used to divide the second tensor data into sub-tensor data based on the resource margin of each of the at least one target graphics processing units to obtain at least one sub-data.

[0171] The second allocating sub-unit is used to allocate at least one sub-data to at least one target graphics processing unit.

[0172] According to an embodiment of the present disclosure, the video processing apparatus 1000 may further include a decoding module.

[0173] The decoding module is used for decoding the video stream in response to receiving the video stream to obtain a plurality of video frame data.

[0174] According to the embodiments of the present invention, any one or more of the modules, submodules, units, and subunits, or at least part of the functions of any one of them can be implemented in one module. According to the embodiments of the present invention, any one or more of the modules, submodules, units, and subunits can be split into multiple modules for implementation. According to the embodiments of the present invention, any one or more of the modules, submodules, units, and subunits can be at least partially implemented as hardware circuits, such as field programmable gate arrays (Field Programmable Gate Array, FPGA), programmable logic arrays (Programmable Logic Arrays, PLA), systems on chips, systems on substrates, systems on packages, application specific integrated circuits (Application Specific Integrated Circuit, ASIC), or can be implemented by hardware or firmware of any other reasonable way of integrating or packaging the circuit, or by any one of the three implementation methods of software, hardware, and firmware or by a suitable combination of any of them. Alternatively, according to the embodiments of the present invention, one or more of the modules, submodules, units, and subunits can be at least partially implemented as computer program modules, and when the computer program modules are run, the corresponding functions can be performed.

[0175] For example, any multiple of the first processing module 1010, the second processing module 1020, and the allocation module 1030 can be combined in one module / unit / subunit for implementation, or any one of the modules / units / subunits can be split into multiple modules / units / subunits. Alternatively, at least part of the functions of one or more of these modules / units / subunits can be combined with at least part of the functions of other modules / units / subunits and implemented in one module / unit / subunit. According to an embodiment of the present disclosure, at least one of the first processing module 1010, the second processing module 1020, and the allocation module 1030 can be at least partially implemented as a hardware circuit, such as a field programmable gate array (FPGA), a programmable logic array (PLA), a system on a chip, a system on a substrate, a system on a package, an application specific integrated circuit (ASIC), or can be implemented by hardware or firmware such as any other reasonable way of integrating or packaging the circuit, or by any one of the three implementation methods of software, hardware, and firmware, or by a suitable combination of any of them. Alternatively, at least one of the first processing module 1010, the second processing module 1020, and the allocation module 1030 may be at least partially implemented as a computer program module, and when the computer program module is executed, the corresponding function may be performed.

[0176] It should be noted that the video data processing device part in the embodiment of the present disclosure corresponds to the video data processing method part in the embodiment of the present disclosure. The description of the video data processing device part specifically refers to the video data processing method part, which will not be repeated here.

[0177] Fig.11 A block diagram of an electronic device suitable for implementing a video data processing method according to an embodiment of the present disclosure is schematically shown. Fig.11 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0178] like Fig.11 As shown, the electronic device 1100 according to an embodiment of the present disclosure includes a processor 1101, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1102 or a program loaded from a storage part 1108 to a random access memory (RAM) 1103. The processor 1101 may include, for example, a general-purpose microprocessor (e.g., a CPU), an instruction set processor and / or a related chipset and / or a special-purpose microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 1101 may also include an onboard memory for caching purposes. The processor 1101 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of the present disclosure.

[0179] In RAM 1103, various programs and data required for the operation of electronic device 1100 are stored. Processor 1101, ROM 1102 and RAM 1103 are connected to each other through bus 1104. Processor 1101 performs various operations of the method flow according to the embodiment of the present disclosure by executing the program in ROM 1102 and / or RAM 1103. It should be noted that the program can also be stored in one or more memories other than ROM 1102 and RAM 1103. Processor 1101 can also perform various operations of the method flow according to the embodiment of the present disclosure by executing the program stored in one or more memories.

[0180] According to an embodiment of the present disclosure, the electronic device 1100 may further include an input / output (I / O) interface 1105, which is also connected to the bus 1104. The electronic device 1100 may further include one or more of the following components connected to the input / output (I / O) interface 1105: an input portion 1106 including a keyboard, a mouse, etc.; an output portion 1107 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage portion 1108 including a hard disk, etc.; and a communication portion 1109 including a network interface card such as a LAN card, a modem, etc. The communication portion 1109 performs communication processing via a network such as the Internet. A drive 1110 is also connected to the input / output (I / O) interface 1105 as needed. A removable medium 1111, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 1110 as needed, so that a computer program read therefrom is installed into the storage portion 1108 as needed.

[0181] The present disclosure also provides a non-transient computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments; or may exist independently without being assembled into the device / apparatus / system. The above computer-readable storage medium carries one or more programs, and when the above one or more programs are executed, the method according to the embodiment of the present disclosure is implemented.

[0182] According to an embodiment of the present disclosure, a computer-readable storage medium may be a non-volatile computer-readable storage medium, for example, may include but is not limited to: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program, which may be used by or in combination with an instruction execution system, an apparatus, or a device. For example, according to an embodiment of the present disclosure, a computer-readable storage medium may include the ROM 1102 and / or RAM 1103 described above and / or one or more memories other than ROM 1102 and RAM 1103.

[0183] The embodiment of the present disclosure also includes a computer program product, which includes a computer program, and the computer program contains program code for executing the method shown in the flowchart. When the computer program product is run in a computer system, the program code is used to enable the computer system to implement the verification method of process data of the product provided by the embodiment of the present disclosure.

[0184] The above functions defined in the system / device of the embodiment of the present disclosure are performed when the computer program is executed by the processor 1101. According to the embodiment of the present disclosure, the system, device, module, unit, etc. described above can be implemented by a computer program module.

[0185] In one embodiment, the computer program may rely on tangible storage media such as optical storage devices, magnetic storage devices, etc. In another embodiment, the computer program may also be transmitted and distributed in the form of signals on a network medium, and downloaded and installed through the communication part 1109, and / or installed from a removable medium 1111. The program code contained in the computer program may be transmitted using any appropriate network medium, including but not limited to: wireless, wired, etc., or any suitable combination of the above.

[0186] In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 1109, and / or installed from the removable medium 1111. When the computer program is executed by the processor 1101, the above functions defined in the system of the embodiment of the present disclosure are performed. According to the embodiment of the present disclosure, the system, device, means, module, unit, etc. described above can be implemented by a computer program module.

[0187] According to an embodiment of the present disclosure, the program code for executing the computer program provided by the embodiment of the present disclosure can be written in any combination of one or more programming languages. Specifically, these computing programs can be implemented using high-level process and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, Java, C++, python, "C" language or similar programming languages. The program code can be executed entirely on the user computing device, partially on the user device, partially on the remote computing device, or entirely on the remote computing device or server. In the case of a remote computing device, the remote computing device can be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computing device (for example, using an Internet service provider to connect through the Internet).

[0188] The flow charts and block diagrams in the accompanying drawings illustrate the possible architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each box in the flow chart or block diagram can represent a module, a program segment, or a part of a code, and the above-mentioned module, program segment, or a part of a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flow chart, and the combination of the boxes in the block diagram or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0189] It will be appreciated by those skilled in the art that the features described in the various embodiments and / or claims of the present disclosure may be combined and / or combined in a variety of ways, even if such combinations and / or combinations are not explicitly described in the present disclosure. In particular, the features described in the various embodiments and / or claims of the present disclosure may be combined and / or combined in a variety of ways without departing from the spirit and teachings of the present disclosure. All of these combinations and / or combinations fall within the scope of the present disclosure.

[0190] The embodiments of the present disclosure are described above. However, these embodiments are only for illustrative purposes and are not intended to limit the scope of the present disclosure. Although the embodiments are described above separately, this does not mean that the measures in the various embodiments cannot be used in combination to advantage. The scope of the present disclosure is defined by the attached claims and their equivalents. Without departing from the scope of the present disclosure, those skilled in the art may make a variety of substitutions and modifications, which should all fall within the scope of the present disclosure.

Claims

1. A video data processing method, comprising: Based on a plurality of video frame data received within a preset time window, a plurality of first tensor data are obtained, wherein the first tensor data includes a plurality of sub-tensor data corresponding to each of the plurality of deep learning models; splicing the multiple sub-tensor data included in each of the multiple first tensor data to obtain multiple second tensor data corresponding to each of the multiple deep learning models; and Based on the deep learning models respectively deployed by the multiple graphics processing units included in the graphics processor, the multiple second tensor data are distributed to the multiple graphics processing units so as to perform data inference on the multiple second tensor data through the graphics processor.

2. The method according to claim 1, wherein: The method of obtaining a plurality of first tensor data based on a plurality of video frame data received within a preset time window includes: For each of the plurality of video frame data received within the preset time window, converting the video frame data into at least one first tensor data based on at least one loaded dynamic link library; and The plurality of first tensor data are obtained based on at least one first tensor data obtained by converting each of the plurality of video frame data.

3. The method according to claim 2, wherein: The dynamic link library is associated with at least one target deep learning model, and the at least one target deep learning model belongs to the multiple deep learning models; The step of converting each of the plurality of video frame data received within the preset time window into at least one first tensor data based on at least one loaded dynamic link library comprises: For each of the video frame data, generating at least one service instance based on the at least one dynamic link library; Running the at least one business instance, wherein the business instance is configured to load the video frame data, process the video frame data based on the input interface standard of each of the at least one target deep learning model related to the business instance, and obtain at least one of the sub-tensor data; and Based on the at least one sub-tensor data corresponding to each of the at least one business instance, the at least one first tensor data is obtained.

4. The method according to claim 3, further comprising: In response to receiving the video frame data, writing the video frame data into a storage unit; Wherein, the service instance is configured to load the video frame data from the storage unit.

5. The method according to claim 4, further comprising: Based on the number of the at least one dynamic link library, configuring a count field of the video frame data; In response to detecting that the service instance loads the video frame data from the storage unit, updating a value of a count field of the video frame data; as well as When the value of the count field of the video frame data is a preset value, the video frame data is released from the storage unit.

6. The method according to claim 5, wherein: The configuring the count field of the video frame data based on the number of the at least one dynamic link library comprises: adding the count field to the video frame data; and An initial value of the count field is determined based on the number of the at least one dynamic link library.

7. The method according to claim 1, wherein: The method of allocating the plurality of second tensor data to the plurality of graphics processing units based on the deep learning model respectively deployed by the plurality of graphics processing units included in the graphics processor comprises: For each of the second tensor data, determining at least one target graphics processing unit from the plurality of graphics processing units based on the deep learning model corresponding to the second tensor data and the deep learning models respectively deployed by the plurality of graphics processing units, wherein the deep learning model deployed by the target graphics processing unit includes the deep learning model corresponding to the second tensor data; and The second tensor data is distributed to the at least one target graphics processing unit.

8. The method according to claim 7, wherein: The allocating the second tensor data to the at least one target graphics processing unit comprises: Based on the resource margin of each of the at least one target graphics processing unit, the second tensor data is segmented with the sub-tensor data as a granularity to obtain at least one sub-data; and The at least one sub-data is distributed to the at least one target graphics processing unit.

9. The method according to claim 1, further comprising: In response to receiving a video stream, the video stream is decoded to obtain the video frame data.

10. A video data processing device, comprising: A first processing module, configured to obtain a plurality of first tensor data based on a plurality of video frame data received within a preset time window, wherein the first tensor data includes a plurality of sub-tensor data corresponding to each of the plurality of deep learning models; A second processing module is used to splice the multiple sub-tensor data included in each of the multiple first tensor data to obtain multiple second tensor data corresponding to each of the multiple deep learning models; and A distribution module is used to distribute the multiple second tensor data to the multiple graphics processing units included in the graphics processor based on at least one of the deep learning models deployed by each of the multiple graphics processing units, so as to perform data inference on the multiple second tensor data through the graphics processor.

11. An electronic device comprising a memory and a processor, wherein the memory stores instructions executable by the processor, and when the instructions are executed by the processor, the processor executes the method according to any one of claims 1 to 9.

12. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1 to 9.

13. A computer program product comprising a computer program, which, when executed by a processor, implements the method according to any one of claims 1 to 9.

14. An edge device comprising: Business plugin manager, inference manager, and model manager; Wherein, the service plug-in manager is configured as follows: Based on a plurality of video frame data received within a preset time window, a plurality of first tensor data are obtained, wherein the first tensor data includes a plurality of sub-tensor data corresponding to each of the plurality of deep learning models; The inference manager is configured to: splicing the multiple sub-tensor data included in each of the multiple first tensor data to obtain multiple second tensor data corresponding to each of the multiple deep learning models; and Based on the deep learning model respectively deployed by the plurality of graphics processing units included in the graphics processor, the plurality of second tensor data are distributed to the plurality of graphics processing units; and The model manager is configured to: Data inference is performed on the plurality of second tensor data by the graphics processor.

15. The edge device according to claim 14, wherein: The service plug-in manager is configured to: For each of the plurality of video frame data received within the preset time window, converting the video frame data into at least one first tensor data based on at least one loaded dynamic link library; as well as The plurality of first tensor data are obtained based on at least one first tensor data obtained by converting each of the plurality of video frame data.

16. The edge device according to claim 15, wherein: The dynamic link library is associated with at least one target deep learning model, and the at least one target deep learning model belongs to the multiple deep learning models; Wherein, the service plug-in manager is configured as follows: In response to receiving the video frame data, generating at least one service instance based on the at least one dynamic link library; Running the at least one business instance, wherein the business instance is configured to load the video frame data, process the video frame data based on the input interface standard of each of the at least one target deep learning model related to the business instance, and obtain at least one of the sub-tensor data; and Based on the at least one sub-tensor data corresponding to each of the at least one business instance, the at least one first tensor data is obtained.

17. The edge device according to claim 16, wherein: The edge device also includes a frame multiplexer; The frame multiplexer is configured as follows: In response to receiving the video frame data, writing the video frame data into a storage unit; Wherein, the service plug-in manager is configured as follows: The video frame data is loaded from the storage unit through the service instance.

18. The edge device according to claim 17, wherein: The frame multiplexer is configured as follows: Based on the number of the at least one dynamic link library, configuring a count field of the video frame data; In response to detecting that the service instance loads the video frame data from the storage unit, updating a value of a count field of the video frame data; as well as When the value of the count field of the video frame data is a preset value, the video frame data is released from the storage unit.

19. The edge device according to claim 18, wherein: The frame multiplexer is configured as follows: adding the count field to the video frame data; and An initial value of the count field is determined based on the number of the at least one dynamic link library.

20. The edge device according to claim 14, wherein: The inference manager is configured to: For each of the second tensor data, determining at least one target graphics processing unit from the plurality of graphics processing units based on the deep learning model corresponding to the second tensor data and the deep learning models respectively deployed by the plurality of graphics processing units, wherein the deep learning model deployed by the target graphics processing unit includes the deep learning model corresponding to the second tensor data; and The second tensor data is distributed to the at least one target graphics processing unit.

21. The edge device according to claim 20, wherein: The inference manager is configured to: Based on the resource margin of each of the at least one target graphics processing unit, the second tensor data is segmented with the sub-tensor data as a granularity to obtain at least one sub-data; as well as The at least one sub-data is distributed to the at least one target graphics processing unit.

22. The edge device of claim 14, wherein: The service plug-in manager is configured to: In response to receiving a video stream, the video stream is decoded to obtain the video frame data.

Citation Information

Patent Citations

  • Method and device for optimizing performance of image processing server end

    CN108920274A

  • Video memory control method and device of deep learning framework and storage medium

    CN110413408A

  • Multi-video-stream processing method and system, computer equipment and storage medium

    CN112001351A

  • Method for determining tensor information of deep learning model and computing device

    CN113313241A

  • Data processing method and device thereof

    CN113449859A