Video data processing method and apparatus, device, storage medium, and edge device
In the video surveillance analysis scenario, by receiving and splicing video frame data within the preset time window and allocating it to the graphics processing unit for inference, the problem of high resource consumption when edge nodes process video data is solved, and efficient video data at the edge is achieved deal with.
Patent Information
- Application Number
- PCT/CN2024/088029
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2023-05-30
- Filing Date
- 2024-04-16
- Publication Date
- 2024-12-05
AI Technical Summary
In the video surveillance analysis scenario, the edge node needs to decode and deploy deep learning models multiple times when processing video frame data, resulting in high computing and storage resource consumption and making it difficult to deploy at the edge.
By receiving multiple video frame data within the preset time window, splicing it into multiple tensor data, and distributing it to the graphics processing unit for data inference, reducing the number of deployments and resource consumption of deep learning models.
It effectively reduces the amount of computing resources, improves the utilization rate of graphics processors, and enables small edge devices to perform video data processing.
Smart Images

Figure CN2024088029_05122024_PF_FP_ABST
Abstract
Description
Video data processing method, device, equipment, storage medium and edge device Technical Field
[0001] The present disclosure relates to the field of artificial intelligence technology, in particular to the field of computer vision and edge computing technology, and specifically to a video data processing method, apparatus, device, storage medium, and edge device. Background Art
[0002] Edge computing involves moving the computation of applications, data, and services from central network nodes to logically edge nodes. For example, in a video surveillance analysis scenario, edge computing could involve electronic devices deployed at edge nodes processing video data collected by multiple cameras connected to the edge nodes.
[0003] Summary of the Invention
[0004] The present disclosure provides a video data processing method, apparatus, device, storage medium, and edge device.
[0005] According to one aspect of the present disclosure, a video data processing method is provided, including: obtaining a plurality of first tensor data based on a plurality of video frame data received within a preset time window, wherein the first tensor data include a plurality of sub-tensor data corresponding to a plurality of deep learning models; splicing the plurality of sub-tensor data included in each of the plurality of first tensor data to obtain a plurality of second tensor data corresponding to each of the plurality of deep learning models; and distributing the plurality of second tensor data to the plurality of graphics processing units included in a graphics processor based on the plurality of deep learning models deployed therein, so as to perform data inference on the plurality of second tensor data through the graphics processor.
[0006] According to another aspect of the present disclosure, a video data processing device is provided, including: a first processing module for obtaining multiple first tensor data based on multiple video frame data received within a preset time window, wherein the above-mentioned first tensor data include multiple sub-tensor data corresponding to each of the multiple deep learning models; a second processing module for splicing the multiple sub-tensor data included in each of the above-mentioned multiple first tensor data to obtain multiple second tensor data corresponding to each of the above-mentioned multiple deep learning models; and a distribution module for distributing the above-mentioned multiple second tensor data to the above-mentioned multiple graphics processing units included in the graphics processor based on the above-mentioned deep learning models respectively deployed by the multiple graphics processing units, so as to perform data inference on the above-mentioned multiple second tensor data through the above-mentioned graphics processor.
[0007] According to another aspect of the present disclosure, an electronic device is provided, including a memory and a processor, wherein instructions executable by the processor are stored in the memory, and when the instructions are executed by the processor, the processor executes the method described above.
[0008] According to another aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable a computer to execute and implement the above method.
[0009] According to another aspect of the present disclosure, a computer program product is provided, including a computer program, which implements the above method when executed by a processor.
[0010] According to another aspect of the present disclosure, an edge device is provided, including: a frame multiplexer, a business plug-in manager, an inference manager and a model manager; wherein the business plug-in manager is configured to: obtain multiple first tensor data based on multiple video frame data received within a preset time window, wherein the first tensor data includes multiple sub-tensor data corresponding to each of the multiple deep learning models; the inference manager is configured to: splice the multiple sub-tensor data included in each of the multiple first tensor data to obtain multiple second tensor data corresponding to each of the multiple deep learning models; and based on the deep learning models deployed by each of the multiple graphics processing units included in the graphics processor, distribute the multiple second tensor data to the multiple graphics processing units; and the model manager is configured to: perform data inference on the multiple second tensor data through the graphics processor.
[0011] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS
[0012] The accompanying drawings are provided to facilitate a better understanding of the present invention and do not constitute a limitation of the present disclosure.
[0013] FIG1 schematically shows a diagram of a video data processing method in related art.
[0014] FIG2 schematically shows an exemplary system architecture to which the video data processing method according to an embodiment of the present disclosure can be applied.
[0015] FIG3 schematically shows a flow chart of a video data processing method according to an embodiment of the present disclosure.
[0016] FIG4 schematically shows a schematic diagram of a video data processing method according to an embodiment of the present disclosure.
[0017] FIG5 schematically shows a schematic diagram of an edge device according to an embodiment of the present disclosure.
[0018] FIG6A schematically shows a schematic diagram of a frame multiplexer according to an embodiment of the present disclosure.
[0019] FIG6B schematically shows a schematic diagram of a video frame multiplexing workflow of a frame multiplexer according to an embodiment of the present disclosure.
[0020] FIG7 schematically shows a schematic diagram of a service plug-in manager according to an embodiment of the present disclosure.
[0021] FIG8 schematically shows a schematic diagram of a communication process between a service plug-in manager and an inference manager according to an embodiment of the present disclosure.
[0022] FIG9 schematically shows a schematic diagram of a model manager according to an embodiment of the present disclosure.
[0023] FIG10 schematically shows a block diagram of a video processing apparatus according to an embodiment of the present disclosure.
[0024] FIG11 schematically shows a block diagram of an electronic device suitable for implementing a video data processing method according to an embodiment of the present disclosure. DETAILED DESCRIPTION
[0025] In order to make the purpose, technical solutions and advantages of the embodiments of the present disclosure clearer, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are part of the embodiments of the present disclosure, not all of them. Based on the described embodiments of the present disclosure, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present disclosure. It should be noted that throughout the drawings, the same elements are represented by the same or similar figure marks. In the following description, some specific embodiments are only for descriptive purposes and should not be understood as any limitation to the present disclosure, but are only examples of the embodiments of the present disclosure. Conventional structures or configurations will be omitted when they may cause confusion in the understanding of the present disclosure. It should be noted that the shapes and sizes of the components in the figures do not reflect the actual size and proportion, but only illustrate the contents of the embodiments of the present disclosure.
[0026] Unless otherwise defined, technical or scientific terms used in the embodiments of the present disclosure shall have the same general meaning as those skilled in the art. The terms "first," "second," and similar terms used in the embodiments of the present disclosure do not denote any order, quantity, or importance, but are merely used to distinguish different components.
[0027] In the embodiments of the present disclosure, the collection, updating, analysis, processing, use, transmission, provision, disclosure, and storage of the data involved (for example, including but not limited to user personal information) all comply with the provisions of relevant laws and regulations, are used for legitimate purposes, and do not violate public order and good morals. In particular, necessary measures are taken with respect to user personal information to prevent unauthorized access to user personal information data and to maintain the security of user personal information, network security, and national security.
[0028] In the embodiments of the present disclosure, the user's authorization or consent is obtained before obtaining or collecting the user's personal information.
[0029] With the development of internet technology, the centralized processing model of cloud computing is no longer able to meet the real-time demands of business processing. To address this, services with high real-time requirements can be deployed on edge nodes, where they can complete processing operations and achieve rapid response. For example, in video surveillance analysis scenarios, servers deployed on edge nodes can be used to process the various videos collected by the edge nodes.
[0030] However, in the related art, for each video frame data in each video channel collected, a separate business processing module is generally used to process the video frame data, and when processing the video frame data, the business processing module needs to separately apply for the computing resources of the graphics processing unit (GPU) to process the inference request. The inference request can be shown in Table 1, for example, and the inference request can include Meta metadata and Data data. Meta metadata can include the name of the deep learning model involved, such as yolov5, and the input interface standard of the deep learning model, such as 684, 684, 3, which means that the data input to the deep learning model should be a 3-channel 684×684 image data. Meta metadata can be an explanation of the Data data. The Data data can be represented as video frame data, such as 108, 255, 0, 88 in Table 1 can be represented as part of the video frame data.
[0031] Table 1
[0032] FIG1 schematically shows a diagram of a video data processing method in related art.
[0033] As shown in Figure 1, multiple business processing modules can be deployed in the electronic device, for example, a face recognition module, a restricted area recognition module, a motion recognition module, etc. Before processing the video frame data, the multiple business processing modules need to decode the collected video separately to obtain the video frame data. After completing the pre-processing of the video frame data, the business processing module can assign the pre-processed video frame data to the computing module of the electronic device, and use the computing resources of the computing module to complete the processing of the video frame data. The computing module may include multiple GPU cards, such as GPU card 1, GPU card 2, etc. The functions implemented by the business processing module generally require multiple deep learning models to implement, and the deep learning models required by each business processing module may be different. Therefore, multiple GPU cards need to deploy a full range of deep learning models.
[0034] Therefore, in related technologies, taking the number of business processing modules as M and the total number of deep learning modules as P as an example, to complete the processing of N video frames, it is necessary to complete M×N decoding operations and deploy N×M×P deep learning models in the computing modules. Therefore, video data processing consumes a lot of computing and storage resources, places high demands on the performance of electronic devices, and makes such electronic devices difficult to deploy at the edge.
[0035] In view of this, an embodiment of the present disclosure provides a video data processing method, by which the usage of computing resources can be effectively reduced, the utilization rate of the graphics processor can be improved, and the device applying the video data processing method can be deployed at the edge. Specifically, the method includes: based on multiple video frame data received within a preset time window, obtaining multiple first tensor data, wherein the first tensor data includes multiple sub-tensor data corresponding to each of the multiple deep learning models; splicing the multiple sub-tensor data included in each of the multiple first tensor data to obtain multiple second tensor data corresponding to each of the multiple deep learning models; and based on the deep learning models deployed by each of the multiple graphics processing units included in the graphics processor, allocating the multiple second tensor data to the multiple graphics processing units, so as to perform data inference on the multiple second tensor data through the graphics processor.
[0036] FIG2 schematically illustrates an exemplary system architecture to which the video data processing method according to an embodiment of the present disclosure may be applied. It should be noted that FIG2 merely illustrates an example of a system architecture to which the embodiment of the present disclosure may be applied, to help those skilled in the art understand the technical content of the present disclosure, and does not imply that the embodiment of the present disclosure may not be applied to other devices, systems, environments, or scenarios.
[0037] As shown in FIG. 2 , a system architecture 200 according to this embodiment may include edge clusters 201 and 202 , a network 203 and a computing center 204 .
[0038] Edge clusters 201 and 202 may include edge devices and one or more cameras. The one or more cameras may be used to capture video streams, and the edge devices may be used to process the video streams captured by the one or more cameras. Edge devices may be various electronic devices, including but not limited to smartphones, tablets, laptops, and desktop computers.
[0039] Network 203 is used to provide a medium for communication links between edge clusters 201 and 202 and computing center 204. Network 203 can include various connection types, such as wired and / or wireless communication links. Network 203 can also be configured with a gateway and firewall for access control between edge clusters 201 and 202 and computing center 204.
[0040] The computing center 204 may include various types of server devices, such as GPU servers, which may be used to provide computing and storage resources for edge devices in the edge clusters 201 and 202 .
[0041] It should be noted that the video data processing method provided in the embodiment of the present disclosure can generally be executed by the computing center 204. Accordingly, the video data processing device provided in the embodiment of the present disclosure can also be set in the computing center 204.
[0042] For example, the camera devices included in the edge clusters 201 and 202 can perform video capture, and the captured video stream can be transmitted in real time by the edge devices included in the edge clusters 201 and 202 to the computing center 204. The computing center 204 can decode the video stream to obtain multiple video frame data, and convert the multiple video frame data and splice them based on the dimensions of the deep learning model to obtain multiple second tensor data. The multiple second tensor data can be allocated based on the resource consumption of the GPU cards of the multiple servers deployed in the computing center 204, so as to use the GPU cards to complete the processing of the second tensor data, thereby realizing the processing of multiple video frame data.
[0043] Alternatively, the video data processing method provided by the embodiment of the present disclosure may also be executed by edge devices in the edge clusters 201 and 202 . Accordingly, the video data processing method provided by the embodiment of the present disclosure may also be set in the edge devices in the edge clusters 201 and 202 .
[0044] It should be understood that the number of edge clusters, computing centers, and networks in FIG2 is merely illustrative and any number of edge clusters, computing centers, and networks may be used as needed.
[0045] FIG3 schematically shows a flow chart of a video data processing method according to an embodiment of the present disclosure.
[0046] As shown in FIG. 3 , the video data processing method may include operations S310 to S330 .
[0047] In operation S310, a plurality of first tensor data are obtained based on a plurality of video frame data received within a preset time window, wherein the tensor data includes a plurality of sub-tensor data corresponding to respective ones of the plurality of deep learning models.
[0048] In operation S320, the multiple sub-tensor data included in each of the multiple first tensor data are spliced to obtain multiple second tensor data corresponding to each of the multiple deep learning models.
[0049] In operation S330 , based on the deep learning model deployed by each of the plurality of graphics processing units included in the graphics processor, the plurality of second tensor data are distributed to the plurality of graphics processing units, so that data inference is performed on the plurality of second tensor data through the graphics processor.
[0050] According to an embodiment of the present disclosure, the preset time window can be represented as a time period having a preset duration, and the window size of the preset time window is the preset duration. The preset duration and the step size of the preset time window can be set according to the specific application scenario. For example, the preset duration can be set to be equal to the step size of the preset time window to avoid repeated processing of video frame data. This is not limited here.
[0051] According to an embodiment of the present disclosure, multiple video frame data may include video frame data from different video streams. For example, video frame data 1 may be one of multiple video frame data obtained by decoding video stream a captured by device A, and video frame data 2 may be one of multiple video frame data obtained by decoding video stream b captured by device B.
[0052] According to an embodiment of the present disclosure, for each video frame data, the video frame data can be processed based on the input requirements of multiple deep learning models to obtain multiple sub-tensor data. Each sub-tensor data can be represented as a matrix. For example, the deep learning model can be an image recognition model that processes RGB images. The sub-tensor data corresponding to the deep learning model can be represented as a 3×H×W matrix, where 3 represents the value of the channel dimension, H can represent the height of the RGB image, and W can represent the width of the RGB image. For another example, the received video frame can be a grayscale image, and accordingly, its video frame data can be represented as a 1×H×W matrix. Deep learning model A can be a model that processes grayscale graphics, and the video frame data can be directly used as sub-tensor data for adapting the input of deep learning model A. Deep learning model B can be a model that processes RGB images. When generating sub-tensor data adapted to the input of the deep learning model B, the video frame data can be converted into a color domain, and the value of each pixel in the video frame data can be converted from a grayscale value to an RGB value to achieve the conversion from the grayscale color domain to the RGB color domain, thereby obtaining the sub-tensor data represented as a 3×H×W matrix.
[0053] According to an embodiment of the present disclosure, multiple sub-tensor data can be spliced to obtain a first tensor data. During splicing, the missing dimensional data in the multiple sub-tensor data can be filled, and the sub-tensor data can be spliced based on an additional dimension. For example, sub-tensor data A can be represented as a 3×H1×W1 matrix, sub-tensor data B can be represented as a 1×H2×W2 matrix, and H1 is greater than H2, and W2 is greater than W1. Sub-tensor data A and sub-tensor data B can be padded respectively, and the padded sub-tensor data A and the padded sub-tensor data B can both be represented as a 3×H1×W2 matrix. After splicing the padded sub-tensor data A and the padded sub-tensor data B, the first tensor data obtained can be represented as a 2×3×H1×W2 matrix.
[0054] According to an embodiment of the present disclosure, a video data processing method can be used to implement a variety of business functions, and each business function can be implemented using one or more of a plurality of deep learning models. For each business function, each frame of video frame data can be processed based on the input requirements of one or more deep learning models required to implement the business function to obtain one or more sub-tensor data, and the one or more sub-tensor data are spliced to obtain a first tensor data. That is, a plurality of first tensor data may include a plurality of first tensor data belonging to different business functions, and the first tensor data corresponding to each business function may include one or more sub-tensor data corresponding to one or more deep learning models. Taking the number of video frame data as N and the number of types of business functions as 2 as an example, the implementation of the first business function requires the support of P1 deep learning models, and the implementation of the second business function requires the support of P2 deep learning models. Based on N video frame data, 2×N first tensor data can be obtained. Each of the N first tensor data corresponding to the first business function can include P1 sub-tensor data, and each of the N first tensor data corresponding to the second business function can include P2 sub-tensor data.
[0055] According to an embodiment of the present disclosure, the multiple sub-tensor data included in each of the multiple first tensor data are spliced, and the multiple sub-tensor data included in each of the multiple first tensor data can be classified based on the deep learning model as the dimension. Each sub-tensor data can carry a model label, and through the model label, the deep learning model associated with the sub-tensor data can be determined, thereby completing the classification of the sub-tensor data. The multiple sub-tensor data included in each category can be suitable for processing by a single deep learning model. The multiple sub-tensor data included in each category can be spliced to obtain second tensor data. That is, the second tensor data obtained after classification can be processed by a single deep learning model.
[0056] According to an embodiment of the present disclosure, a graphics processor may include multiple image processing units, each of which may be represented as a GPU card. Distributing the multiple second tensor data may include distributing the multiple second tensor data to multiple GPU cards for processing. One or more deep learning models may be deployed in each GPU card. After receiving the second tensor data, the GPU card may use the corresponding deep learning model deployed in the GPU card to process the second tensor data to obtain a processing result for the second tensor data.
[0057] According to an embodiment of the present disclosure, the manner of distributing the plurality of second tensor data to the plurality of image processing units may include, but is not limited to, equal distribution, low-fragmentation distribution, and the like. Equal distribution may be a distribution method that makes the loads of the plurality of GPU cards substantially uniform after distribution. Low-fragmentation distribution may be a distribution method that preferentially uses non-idle GPU cards for data processing.
[0058] According to an embodiment of the present disclosure, each sub-tensor data in the second tensor data may also carry a video frame label. Through the video frame label, the video frame data associated with the sub-tensor data may be determined, thereby determining the processing result of the video frame data based on the processing result of the second tensor data.
[0059] According to an embodiment of the present disclosure, all video frame data received within a time window can be converted into multiple first tensor data, each of which can include multiple sub-tensor data, and each sub-tensor data can be obtained by processing according to the input requirements of the corresponding deep learning model. The multiple sub-tensor data included in each of the multiple first tensor data can be spliced to obtain multiple second tensor data. When processing data, the multiple second tensor data can be task-assigned in the image processor based on a load balancing strategy to complete the processing of the video frame data. By splicing the sub-tensor data belonging to the same deep learning model into one second tensor data, the batch size of the data can be increased, thereby effectively utilizing the batch processing capability of the graphics processor, increasing the throughput of the graphics processor, and improving the utilization rate of the graphics processor. At the same time, the number of deep learning models that need to be deployed in the memory of the graphics processor can be reduced, reducing the resource consumption of the graphics processor, so that small edge devices can also realize video data processing.
[0060] 4 , the video data processing method shown in FIG. 3 will be further described in conjunction with a specific embodiment.
[0061] According to an embodiment of the present disclosure, in response to receiving a video stream, the video stream may be decoded to obtain video frame data.
[0062] According to embodiments of the present disclosure, a video stream can be obtained by capturing and encoding a video stream using a camera. The video stream can be represented as a segment of video, and accordingly, the video frame data is obtained by decoding the segment of video. There can be multiple cameras, and accordingly, there can be multiple video streams. The multiple video frame data can include video frame data obtained by decoding the multiple video streams.
[0063] According to an embodiment of the present disclosure, for each of a plurality of video frame data received within a preset time window, the video frame data can be converted into at least one first tensor data based on at least one loaded dynamic link library. A plurality of first tensor data are obtained based on the at least one first tensor data converted from each of the plurality of video frame data.
[0064] According to an embodiment of the present disclosure, a dynamic link library can be represented as a decoupled library file, for example, it can be represented as a file with a .so suffix on Linux. The dynamic link library can support various business interface methods, and the business interface methods may include init (initialization), stop (stop), start (start), pause (pause), resume (resume), etc. During the initialization phase, the dynamic link library can be loaded into the memory. By loading the dynamic link library into the memory of the device, various business interface methods can be used to implement the call of the business functions represented by the dynamic link library.
[0065] According to an embodiment of the present disclosure, a dynamic link library can be associated with at least one target deep learning model, and at least one target deep learning model can belong to multiple deep learning models. Specifically, the dynamic link library can call at least one target deep learning model to implement the business function represented by the dynamic link library.
[0066] According to an embodiment of the present disclosure, converting video frame data into first tensor data may include performing data conversion on the video frame data according to the input interface standards of each of the multiple deep learning models to obtain at least one first tensor data. Specifically, for each video frame data, based on at least one loaded dynamic link library, converting the video frame data into at least one first tensor data may include the following operations:
[0067] For each video frame data among multiple video frame data received within a preset time window, at least one business instance is generated based on at least one dynamic link library; at least one business instance is run, wherein the business instance is configured to load the video frame data, and process the video frame data based on the input interface standard of each of at least one target deep learning model related to the business instance to obtain at least one sub-tensor data; and at least one first tensor data is obtained based on at least one sub-tensor data corresponding to each of the at least one business instance.
[0068] According to an embodiment of the present disclosure, a business instance can be represented as a task. By executing the business instance, the business function corresponding to the dynamic link library can be realized. Accordingly, generating a business instance based on the dynamic link library can be creating a new task based on the requirements described in the dynamic link library.
[0069] According to an embodiment of the present disclosure, taking the deep learning model as an image processing model as an example, the input interface standard of the image processing model may indicate that the input data is single-channel grayscale image data. The video frame data to be processed may be three-channel RGB image data. Processing the video frame data based on the input interface standard of the image processing model may involve performing a color space transformation on the video frame data to convert the RGB value of each pixel in the video frame data into a grayscale value to obtain the video frame data to be processed, i.e., the sub-tensor data corresponding to the image processing model.
[0070] According to the embodiments of the present disclosure, by completing video data processing based on a dynamic link library, the decoupling between business functions and business processes can be achieved. When faced with new business function requirements, developers write the new business function requirements into a separate dynamic link library according to a template, and then write the dynamic link library into the memory to realize the launch of the new business function, thereby effectively improving business development efficiency.
[0071] According to embodiments of the present disclosure, decoded video frame data can be temporarily stored to facilitate reuse of the video frame data by at least one service instance. Specifically, in response to receiving the video frame data, the video frame data can be written to a storage unit. After generating and starting at least one service instance, the service instance can load the video frame data from the storage unit.
[0072] According to an embodiment of the present disclosure, the storage unit may refer to a memory, a hard disk, a database or other storage device, which is not limited here.
[0073] According to an embodiment of the present disclosure, the multiplexing of the video frame data by at least one business instance can be achieved by using a reference counter. Specifically, after the video frame data is written to the storage unit, a data packet can be obtained, the video frame data can be stored in the body portion of the data packet, and the header portion of the data packet can record fields related to the video frame data. The count field of the video frame data can be configured based on the number of at least one dynamic link library, and the count field is the reference counter. For example, a count field can be added to a field in the body portion of the data packet corresponding to the video frame data, or a count field can be added to the header portion of the data packet corresponding to the video frame data. In response to detecting that the business instance loads the video frame data from the storage unit, the value of the count field of the video frame data can be updated. When the value of the count field of the video frame data is a preset value, the video frame data is released from the storage unit.
[0074] According to an embodiment of the present disclosure, updating the value of the count field of the video frame data may be performed by increasing or decreasing the value of the count field according to the data update step size. For example, the data update step size may be set to 1, and updating the value of the count field of the video frame data may be performed by subtracting 1 from the original value of the count field.
[0075] According to an embodiment of the present disclosure, the preset value can be set according to a specific application scenario, for example, it can be set to 0. When the value of the count field of the video frame data is the preset value, it can indicate that the video frame data has been called a predetermined number of times.
[0076] According to an embodiment of the present disclosure, releasing the video frame data from the storage unit may be performed by physically deleting the video frame data from the storage unit.
[0077] According to an embodiment of the present disclosure, configuring a count field for video frame data may involve adding a count field to the video frame data. The count field may be represented as a parameter with an empty value. The initial value of the count field may be determined based on the number of at least one dynamic link library. Specifically, the initial value of the count field may be determined based on the data update step size, the total number of dynamic link libraries, and a preset value. For example, if the number of dynamic link libraries is a, the data update step size is b, and the preset value is c, then the initial value of the count field may be determined to be c+a×b.
[0078] For example, after receiving decoded video frame data, the video frame data can be packaged into a data packet and written to memory. A field called ref_counter is added to the data packet, and the initial value of the ref_counter field is set based on the number of generated service instances. For example, if three service instances are generated, the initial value of the ref_counter field can be set to 3. After starting a service instance, each service instance can be run using a thread of the service processor. A smart pointer can be configured in each thread to point to the data packet in memory. Alternatively, a smart pointer can be configured in the service processor, and each service instance in the service processor's thread can call the smart pointer configured in the service processor. After starting a service instance, the service instance can call the smart pointer to determine the address of the data packet in memory, thereby loading the data packet. After waiting for the service instance to load the data packet to reuse the video frame data, the value of the ref_counter field can be decremented by 1. When the value of the ref_counter field reaches 0, the data packet in memory can be released, destroying the corresponding video frame data.
[0079] According to an embodiment of the present disclosure, by storing video frame data in a storage unit, at least one business instance can reuse the video frame data through a pointer. By adding a reference counter to the video frame data, when a business instance uses the video frame data through a pointer, the value of the reference counter can change accordingly, and when the value of the reference counter reaches a preset value, the video frame data can be released. In this way, the multiplexing of video frame data can be achieved. When each business instance uses the video frame data, there is no need to repeatedly decode the video stream, thereby reducing the number of decoding operations and saving computing resources. At the same time, the storage unit no longer needs to store the video frame data generated by repeated decoding, saving storage resources.
[0080] According to an embodiment of the present disclosure, at least one first tensor data can be generated for each video frame data using at least one service instance, and the multiple first tensor data can include at least one first tensor data associated with each of the multiple video frames data. Specifically, the at least one first tensor data associated with each of the multiple video frames data can be combined to obtain the multiple first tensor data.
[0081] According to an embodiment of the present disclosure, a preset time window may include multiple subwindows. Within each subwindow, one or more first tensor data output by all completed business instances may be packaged into a request, which may be packaged into a message, and the one or more first tensor data may serve as the message body of the message. This request may be sent to the management module of the graphics processor, so that the management module can perform resource scheduling to fully utilize the graphics processor's ability to process large batches of data.
[0082] According to an embodiment of the present disclosure, the multiple sub-tensor data included in each first tensor data can be respectively suitable for processing by different deep learning models. The sub-tensor data suitable for processing by the deep learning model in the multiple first tensor data can be classified into one category based on the deep learning model. All the sub-tensor data included in each category can be combined to obtain a large-size, high-dimensional second tensor data. The management module of the image processor can perform resource scheduling for the large-size, high-dimensional second tensor data. Compared with the related art, for each received video frame data, the video frame data needs to be packaged into a message and sent to the management module of the graphics processor. The management module needs to schedule each video frame data separately. Since the graphics processor needs to generate processing tasks for the deployed deep learning model after receiving the data to be processed, and given that the graphics processor has a strong large-batch data processing capability, the processing time and resource usage of each processing task are poorly correlated with the amount of data to be processed. That is, in each processing task, the time and resources consumed in processing small-sized single video frame data and large-sized, high-dimensional second tensor data are basically the same. Therefore, splicing multiple groups of first tensor data to obtain multiple second tensor data and performing resource scheduling on the multiple second tensor data can fully utilize the graphics processor's processing capabilities for large batches of data.
[0083] According to an embodiment of the present disclosure, the steps performed by the management module of the graphics processor to distribute the plurality of second tensor data to the plurality of graphics processing units included in the graphics processor based on the deep learning models deployed by the plurality of graphics processing units may include the following operations:
[0084] For each second tensor data, based on a deep learning model corresponding to the second tensor data and deep learning models deployed on each of the plurality of graphics processing units, at least one target graphics processing unit is determined from the plurality of graphics processing units, wherein the at least one deep learning model deployed on the target graphics processing unit includes the deep learning model corresponding to the second tensor data. The second tensor data is distributed to the at least one target graphics processing unit.
[0085] According to an embodiment of the present disclosure, for example, GPU A, GPU B, and GPU C may be deployed with model a. When allocating second tensor data associated with model a, GPU A, GPU B, and GPU C may be determined target GPUs. Accordingly, the second tensor data associated with model a may be allocated to GPU A, GPU B, and GPU C for processing.
[0086] According to an embodiment of the present disclosure, distributing the second tensor data to at least one target graphics processing unit may include the following operations:
[0087] Based on the resource margin of at least one target graphics processing unit, the second tensor data is divided into sub-tensor data as a granularity to obtain at least one sub-data, and the at least one sub-data is allocated to at least one target graphics processing unit.
[0088] According to an embodiment of the present disclosure, segmenting the second tensor data with sub-tensor data as the granularity can be to use the sub-tensor data as the smallest unit for segmentation, that is, the sub-tensor data included in the second tensor data after segmentation are all complete, and each sub-data can include one or more sub-tensor data.
[0089] For example, the at least one target GPU includes GPU A, GPU B, GPU C, and GPU D. The resource margins of GPU A, GPU B, GPU C, and GPU D are 70%, 80%, 100%, and 50%, respectively. The second tensor data may include 100 sub-tensor data. The processing capabilities of GPU A, GPU B, GPU C, and GPU D may be the same, and processing each sub-tensor data requires 1% of the GPU's resources.
[0090] When splitting the second tensor data based on a balanced allocation strategy, the second tensor data can be split into three sub-data: sub-data a, sub-data b, and sub-data c. Sub-data a can include 20 sub-tensor data, sub-data b can include 30 sub-tensor data, and sub-data c can include 50 sub-tensor data. Sub-data a can be assigned to GPU A for processing, sub-data b can be assigned to GPU B for processing, sub-data c can be assigned to GPU C for processing, and GPU D can remain unassigned. After data allocation is completed, the resource margins of GPUs A, B, C, and D are all 50%, thereby ensuring that the loads of each GPU are substantially consistent.
[0091] When the second tensor data is split based on the anti-fragmentation allocation strategy, the second tensor data can be split into two sub-data, namely sub-data d and sub-data e, and sub-data d and sub-data e can each include 50 sub-tensor data. Sub-data d can be allocated to graphics processing unit D, and sub-data e can be allocated to graphics processing unit A. Graphics processing unit B and image processing unit C are not allocated. After the data allocation is completed, the resource margin of graphics processing unit A is 20%, the resource margin of graphics processing unit B is 80%, the resource margin of graphics processing unit C is 100%, and the resource margin of graphics processing unit D is 0%, thereby ensuring the integrity of the resources of the graphics processing unit, so that when a large batch of data is received, the graphics processing unit with complete resources can be used to process the large batch of data, thereby improving the data reasoning efficiency of the image processing unit.
[0092] FIG4 schematically shows a schematic diagram of a video data processing method according to an embodiment of the present disclosure.
[0093] As shown in Figure 4, the video data processing method can decode the video streams from cameras configured in various business systems to obtain video frame data. Various business systems may include, for example, access control systems, confidential area management systems, road violation detection systems, etc. Taking the access control system as an example, the access control system may be a system for identifying and verifying the identities of people entering and leaving various types of parks. The cameras configured in the access control system can be used to obtain information such as portraits and identity credentials. Accordingly, the decoded video frame data may be image data containing information such as portraits and identity credentials. By using the video data processing method of the embodiment of the present disclosure to process the video obtained by the access control system, the video frame data obtained by each of the multiple cameras can be processed in parallel to improve the processing efficiency of the video frame data.
[0094] According to an embodiment of the present disclosure, the video frame data received within the preset time window may include video frame data video1 and video frame data video2.
[0095] According to embodiments of the present disclosure, the access control system may implement services that include facial recognition and restricted area management. Accordingly, for each video frame, two service instances may be generated, corresponding to the facial recognition and restricted area management functions, respectively. Specifically, instances A1 and B1 may be generated for video frame data video1, while instances A2 and B2 may be generated for video frame data video2.
[0096] According to the embodiments of the present disclosure, each instance can process the corresponding video frame data based on the deep learning model required to implement the corresponding business function to obtain first tensor data. For example, the implementation of the face recognition function requires the application of model α and model β, and the implementation of the restricted area management function requires the application of model γ. Accordingly, instance A1 processes the video frame data video1 to obtain first tensor data a1, which can include sub-tensor data α_a1 determined based on the input interface standard of model α and sub-tensor data β_a1 determined based on the input interface standard of model β. Instance A2 processes the video frame data video2 to obtain first tensor data a2, which can include sub-tensor data α_a2 determined based on the input interface standard of model α and sub-tensor data β_a2 determined based on the input interface standard of model β. Instance B1 processes the video frame data video1 to obtain first tensor data b1, which can include sub-tensor data γ_b1 determined based on the input interface standard of model γ. Example B2 processes the video frame data video2 to obtain first tensor data b2, which may include sub-tensor data γ_b2 determined based on the model γ.
[0097] According to an embodiment of the present disclosure, sub-tensor data can be aggregated and spliced based on the dimension of the deep learning model to obtain second tensor data. Specifically, sub-tensor data α_a1 and sub-tensor data α_a2 can be spliced to obtain second tensor data α_a, sub-tensor data β_a1 and sub-tensor data β_a2 can be spliced to obtain second tensor data β_a, and sub-tensor data γ_b1 and sub-tensor data γ_b2 can be spliced to obtain second tensor data γ_b.
[0098] According to an embodiment of the present disclosure, a graphics processor may include GPU card 1 and GPU card 2. Model α, model β, and model γ may be deployed in GPU card 1, and model γ may be deployed in GPU card 2. Therefore, the target graphics processing unit corresponding to the second tensor data α_a is GPU card 1, the target graphics processing unit corresponding to the second tensor data β_a is GPU card 1, and the target graphics processing units corresponding to the second tensor data γ_b are GPU card 1 and GPU card 2.
[0099] According to an embodiment of the present disclosure, when allocating the second tensor data, the second tensor data α_a and the second tensor data β_a may be allocated to GPU card 1. Based on the resource margins of GPU card 1 and GPU card 2, the second tensor data γ_b may be split into two sub-data, and the two sub-data may be allocated to GPU card 1 and GPU card 2, respectively.
[0100] According to an embodiment of the present disclosure, the video data processing method described above may be implemented using an edge device.
[0101] FIG5 schematically shows a schematic diagram of an edge device according to an embodiment of the present disclosure.
[0102] As shown in Figure 5, edge device 500 may include a service plug-in manager 510, an inference manager 520, and a model manager 530. Edge device 500 may be represented as a single electronic device, such as a GPU server. Alternatively, edge device 500 may include multiple electronic devices, such as multiple GPU servers. Service plug-in manager 510 may be deployed on one or more GPU servers, and inference manager 520 and model manager 530 may be deployed on another GPU server.
[0103] According to an embodiment of the present disclosure, the business plug-in manager 510 can connect multiple cameras through the external interface of the edge device. For the video stream generated by each camera, the business plug-in manager 510 can be configured to decode the video stream in response to receiving the video stream to obtain video frame data.
[0104] According to an embodiment of the present disclosure, the service plug-in manager 510 may be configured to obtain a plurality of first tensor data based on a plurality of video frame data received within a preset time window. The first tensor data includes a plurality of sub-tensor data corresponding to each of the plurality of deep learning models.
[0105] According to an embodiment of the present disclosure, the inference manager 520 may be configured to: concatenate multiple sub-tensor data included in each of the multiple first tensor data to obtain multiple second tensor data corresponding to each of the multiple deep learning models.
[0106] According to an embodiment of the present disclosure, the service plug-in manager 510 may be configured with multiple service plug-ins, such as a face recognition service plug-in 511, a restricted area recognition service plug-in 512, and a motion recognition service plug-in 513. The service plug-in may be represented as a dynamic link library written into the memory.
[0107] According to an embodiment of the present disclosure, the inference manager 520 may be configured to distribute the plurality of second tensor data to the plurality of graphics processing units included in the graphics processor based on at least one deep learning model deployed by each of the plurality of graphics processing units.
[0108] According to an embodiment of the present disclosure, the model manager 530 may be configured to perform data inference on the plurality of second tensor data through a graphics processor.
[0109] According to an embodiment of the present disclosure, the model manager 530 can be used to manage deep learning models that need to be deployed in the image processor. Deep learning models may include, for example, a yolov5m model 531, a resnet model 532, a face model 533, a keypoint model 534, a restrict model 535, a motion model 536, a mask model 537, a quality module 538, and the like. A combination of one or more of the above-mentioned deep learning models can be used to implement different business functions. For example, the face recognition function in the access control system may require a resnet model 532 and a face model 533 to implement. The number of deep learning models that the model manager can manage can be set according to the specific application scenario and is not limited here.
[0110] According to an embodiment of the present disclosure, the model manager 530 may also be used to manage multiple graphics processing units 541 included in the graphics processor 540 , such as GPU card 1 , GPU card 2 , and the like.
[0111] According to an embodiment of the present disclosure, one or more deep learning models may be pre-deployed in each graphics processing unit 541. For example, a yolov5m model 531, a resnet model 532, and a keypoint model 534 may be deployed in GPU card 1, and a resnet model 532, a motion model 536, and a mask model 537 may be deployed in GPU card 1. The model manager 530 may maintain a mapping table, which may include a mapping relationship between the graphics processing unit 541 and the deep learning model, that is, the deep learning model that needs to be deployed in the graphics processing unit 541. After the edge device 500 starts working, the model manager 530 may deploy the deep learning model in each graphics processing unit 541 according to the mapping table.
[0112] According to an embodiment of the present disclosure, when allocating the second tensor data, the inference manager 520 may allocate the second tensor data to the corresponding graphics processing unit based on the deep learning model corresponding to the second tensor data. For example, if the deep learning model corresponding to the second tensor data A is the yolov5m model 541, the inference manager 520 may allocate the second tensor data A to GPU card 1. For another example, if the deep learning model corresponding to the second tensor data B is the resnet model 542, the inference manager 520 may allocate the second tensor data B to GPU card 1 and GPU card 2 respectively based on the resource margins of GPU card 1 and GPU card 2.
[0113] According to the embodiments of the present disclosure, by concatenating sub-tensor data belonging to the same deep learning model into a second tensor data, the batch size of the data can be increased, thereby effectively utilizing the batch processing capability of the graphics processor, increasing the throughput of the graphics processor, and improving the utilization rate of the graphics processor. At the same time, the number of deep learning models that need to be deployed in the memory of the graphics processor can be reduced, reducing the resource consumption of the graphics processor. This allows the edge device 500 to be deployed at the edge.
[0114] The edge device shown in FIG5 will be further described below with reference to FIG6A , FIG6B and FIG7 to FIG9 in combination with specific embodiments.
[0115] According to an embodiment of the present disclosure, the edge device 500 may further include a frame multiplexer 550. The frame multiplexer 550 may be arranged between the external interface of the edge device 500 and the service plug-in manager 510. The external interface of the edge device 500 may be connected to multiple cameras, and each camera may provide one video stream to the edge device 500. The number of cameras connected to the external interface of the edge device 500 may be set according to the specific application scenario and is not limited here. For example, in the application scenario of a campus access control system, the campus may include 4 entrances and exits, and each entrance and exit may be provided with 2 cameras, then the number of cameras connected to the external interface of the edge device 500 may be 8.
[0116] FIG6A schematically shows a schematic diagram of a frame multiplexer according to an embodiment of the present disclosure.
[0117] As shown in FIG6A , the frame multiplexer 550 may be configured to, in response to receiving a video stream, decode the video stream to obtain video frame data. Specifically, the frame multiplexer 550 may decode the video stream from camera 1 into video frame data frame1, the video stream from camera 2 into video frame data frame2, and the video stream from camera 3 into video frame data frame3.
[0118] According to an embodiment of the present disclosure, the frame multiplexer 550 may be configured to, in response to receiving the video frame data, write the video frame data into the storage unit 551 .
[0119] According to an embodiment of the present disclosure, the frame multiplexer 550 may be configured to configure a count field for the video frame data based on the number of at least one dynamic link library. Specifically, the frame multiplexer 550 may be configured to add a count field to the video frame data. An initial value of the count field may be determined based on the number of at least one dynamic link library. For example, the number of dynamic link libraries may be three, and a count field (count) may be configured for each video frame data. The initial value of the count field (count) for the video frame data frame1, the video frame data frame2, and the video frame data frame3 may all be set to 3.
[0120] According to an embodiment of the present disclosure, the service plug-in manager 510 can generate service instances based on the configured service plug-ins. For example, the face recognition service plug-in 511 configured by the service plug-in manager 510 can be used to generate a face recognition service instance, the restricted area recognition service plug-in 512 configured by the service plug-in manager 510 can be used to generate a restricted area recognition service instance, and the motion recognition service plug-in 511 configured by the service plug-in manager 510 can be used to generate a motion recognition service instance. The face recognition service instance, the restricted area recognition service instance, and the motion recognition service instance can load video frame data frame1, video frame data frame2, and video frame data frame3 from the storage unit, respectively.
[0121] According to an embodiment of the present disclosure, the frame multiplexer 550 can be configured to: update the value of the count field of the video frame data in response to detecting that a business instance loads video frame data from a storage unit; and release the video frame data from the storage unit when the value of the count field of the video frame data is a preset value.
[0122] FIG6B schematically shows a schematic diagram of a video frame multiplexing workflow of a frame multiplexer according to an embodiment of the present disclosure.
[0123] As shown in Figure 6B, the video stream captured by the camera can enter the frame multiplexer 550 through the gateway. After parsing the video stream, the frame multiplexer 550 can obtain video frame data. The frame multiplexer 550 can store the video frame data in the memory and generate a data structure corresponding to the video frame data in the memory. This data structure may include two parts: one part includes the memory address and reference count field refcounter of the video frame data, and the other part is the memory data of the video frame data. The service instance generated by the service plug-in manager 510 can call this data structure and find the reference count field refcounter and memory address of the video frame data based on this data structure. Based on this memory address, the service plug-in manager 510 can call and process the video frame data. Specifically, each service instance can complete the call of video frame data through a separate thread. For example, service instance 1 calls frame1 through thread 1, and service instance 2 calls frame1 through thread 2. After detecting that the service plug-in manager 510 has completed the call of the video frame data, the frame multiplexer 550 can reduce the value of the reference count field refcounter in the data structure by 1. When the value of the reference count field refcounter is less than 1, the frame multiplexer 510 can recycle the data structure, that is, delete the video frame data from the memory.
[0124] FIG7 schematically shows a schematic diagram of a service plug-in manager according to an embodiment of the present disclosure.
[0125] As shown in Figure 7, the service plug-in manager 510 can be configured with multiple service interfaces. By calling a service interface, the service instance can be processed using the service interface method corresponding to the service interface. The multiple service interfaces can include an init interface, a stop interface, a start interface, a pause interface, and a resume interface. For example, calling the stop interface can cause the service instance to stop running. For another example, calling the start interface can cause the service instance to start running.
[0126] According to an embodiment of the present disclosure, the service plug-in manager 510 can manage multiple service plug-ins, including a face recognition service plug-in 511, a restricted area recognition service plug-in 512, a motion recognition service plug-in 513, etc. Each service plug-in can be represented as a loaded dynamic link library.
[0127] According to an embodiment of the present disclosure, the service plug-in manager 510 can be configured to: for each video frame data among multiple video frame data received within a preset time window, based on at least one loaded dynamic link library, convert the video frame data into at least one first tensor data. Based on the at least one first tensor data converted from each of the multiple video frame data, multiple first tensor data are obtained. Specifically, each dynamic link library can be associated with at least one target deep learning model, and the at least one target deep learning model can belong to multiple deep learning models.
[0128] According to an embodiment of the present disclosure, the business plug-in manager 510 can be specifically configured to: generate at least one business instance based on at least one dynamic link library in response to receiving video frame data; run at least one business instance, wherein the business instance is configured to load video frame data, specifically, the video frame data can be loaded from the storage unit 551, and the video frame data can be processed based on the input interface standard of each of the at least one target deep learning model related to the business instance to obtain at least one sub-tensor data; and obtain at least one first tensor data based on the at least one sub-tensor data corresponding to each of the at least one business instance.
[0129] According to an embodiment of the present disclosure, the service plug-in manager 510 may maintain a mapping table for each dynamic link library. This mapping table may include at least one deep learning model associated with the dynamic link library and the input interface standard of the at least one deep learning model. When a service instance is running, the service instance may be configured to process each video frame based on the input interface standard of the at least one deep learning model included in the mapping table, obtain at least one sub-tensor data, and concatenate the at least one sub-tensor data to obtain the first tensor data.
[0130] According to an embodiment of the present disclosure, a service instance can be represented as a program that can be run based on a service plug-in. The service instance can be run in a thread of the service plug-in manager 510. After the service instance is completed, the service instance can be released from the thread of the service plug-in manager 510.
[0131] For example, for video frame data frame1, video frame data frame2 and video frame data frame3, the service plug-in manager 510 can generate face recognition service instance 1, face recognition service instance 2 and face recognition service instance 3 corresponding to the face recognition service plug-in, generate restricted area recognition service instance 1, restricted area recognition service instance 2 and restricted area recognition service instance 3 corresponding to the restricted area recognition service plug-in, and generate motion recognition service instance 1, motion recognition service instance 2 and motion recognition service instance 3 corresponding to the motion recognition service plug-in.
[0132] According to an embodiment of the present disclosure, the service plug-in manager 510 and the frame multiplexer 550 can be configured to be set in the CPU device of the edge device, and the inference manager 520, the model manager 530, and the graphics processor 540 can be configured to be set in the GPU device of the edge device. The CPU device and the GPU device of the edge device can communicate with each other in the form of request messages.
[0133] FIG8 schematically shows a schematic diagram of a communication process between a service plug-in manager and an inference manager according to an embodiment of the present disclosure.
[0134] As shown in FIG8 , with time as the axis, the preset time window may include multiple sub-windows, and within each sub-window, the service plug-in manager 510 may package all first tensor data generated within the sub-window into one request.
[0135] For example, in sub-window 1, only face recognition service instance 1 has completed execution in service plug-in manager 510. In this case, service plug-in manager 510 may package the first tensor data output by face recognition service instance 1 into request req1. In sub-window 2, face recognition service instance 2 and face recognition service instance 3 have completed execution in service plug-in manager 510. In this case, service plug-in manager 510 may package the first tensor data output by face recognition service instance 2 and face recognition service instance 3 into request req2. In sub-window 3, restricted area recognition service instance 1, restricted area recognition service instance 2, restricted area recognition service instance 3, motion recognition service instance 1, motion recognition service instance 2, and motion recognition service instance 3 have completed execution in service plug-in manager 510. In this case, service plug-in manager 510 may package the first tensor data output by restricted area recognition service instance 1, restricted area recognition service instance 2, restricted area recognition service instance 3, motion recognition service instance 1, motion recognition service instance 2, and motion recognition service instance 3 into request req3.
[0136] According to an embodiment of the present disclosure, the inference manager 520 can be configured to use the deep learning model as a dimension, and for the requests generated by each sub-window of the preset time window, the multiple sub-tensor data included in each request can be classified. Each sub-tensor data can carry a model label, and through the model label, the deep learning model associated with the sub-tensor data can be determined, thereby completing the classification of the sub-tensor data. The multiple sub-tensor data included in each category can be suitable for processing by a single deep learning model. The multiple sub-tensor data included in each category can be spliced to obtain second tensor data. That is, the second tensor data obtained after classification can be processed by a single deep learning model.
[0137] For example, the multiple sub-tensor data included in the request req1 can be respectively included in the second tensor data batch1 and the second tensor data batch3. The multiple sub-tensor data included in the request req2 can be respectively included in the second tensor data batch1 and the second tensor data batch3. The multiple sub-tensor data included in the request req3 can be respectively included in the second tensor data batch2 and the second tensor data batch3.
[0138] According to an embodiment of the present disclosure, the inference manager 520 may be configured to: for each second tensor data, determine at least one target graphics processing unit from a plurality of graphics processing units based on a deep learning model corresponding to the second tensor data and deep learning models deployed by each of the plurality of graphics processing units, wherein the deep learning model deployed by the target graphics processing unit includes the deep learning model corresponding to the second tensor data; and allocate the second tensor data to the at least one target graphics processing unit. Specifically, the inference manager 520 may be configured to: based on the resource margin of each of the at least one target graphics processing unit, split the second tensor data into sub-tensor data as a granularity to obtain at least one sub-data; and allocate the at least one sub-data to the at least one target graphics processing unit.
[0139] For example, the inference manager 520 may be configured to distribute the second tensor data batch 1 to GPU card 1 , distribute the second tensor data batch 2 to GPU card 2 , and distribute the second tensor data batch 3 to GPU cards 3 and 4 .
[0140] FIG9 schematically shows a schematic diagram of a model manager according to an embodiment of the present disclosure.
[0141] As shown in FIG9 , the model manager 530 can be used to manage various deep learning models and deploy the deep learning models on various graphics processing units (GPUs) in the graphics processor. After deploying the deep learning model on the GPU, the GPU can generate an instance corresponding to the deep learning model. After receiving the second tensor data, the GPU can run the instance corresponding to the deep learning model in the thread of the GPU to perform data inference on the second tensor data.
[0142] For example, four deep learning models, namely restrict, motion, mask and quality, can be deployed in the graphics processing unit GPU1. After the deployment is completed, the graphics processing unit GPU1 can generate instances instance1 and instance2 of the restrict model, instances instance1 and instance2 of the motion model, instances instance1 and instance2 of the mask model, and instances instance1 and instance2 of the quality model. The second tensor data can be, for example, data that needs to be processed by the motion model. After the second tensor data is input into the graphics processing unit GPU1, the two threads of the graphics processing unit GPU1 corresponding to the instances instance1 and instance2 of the motion model can load the second tensor data respectively and run the method of the motion model to process the second tensor data.
[0143] FIG10 schematically shows a block diagram of a video processing apparatus according to an embodiment of the present disclosure.
[0144] As shown in FIG. 10 , the video processing apparatus 1000 may include a first processing module 1010 , a second processing module 1020 , and an allocation module 1030 .
[0145] The first processing module 1010 is used to obtain multiple first tensor data based on multiple video frame data received within a preset time window, wherein the first tensor data includes multiple sub-tensor data corresponding to each of the multiple deep learning models.
[0146] The second processing module 1020 is used to splice the multiple sub-tensor data included in each of the multiple first tensor data to obtain multiple second tensor data corresponding to each of the multiple deep learning models.
[0147] The allocation module 1030 is used to allocate multiple second tensor data to multiple graphics processing units based on the deep learning models deployed by the multiple graphics processing units included in the graphics processor, so as to perform data inference on the multiple second tensor data through the graphics processor.
[0148] According to an embodiment of the present disclosure, the video processing apparatus 1000 may further include a conversion module and a third processing module.
[0149] The conversion module is used to convert each video frame data among multiple video frame data received within a preset time window into at least one first tensor data based on at least one loaded dynamic link library.
[0150] The third processing module is configured to obtain a plurality of first tensor data based on at least one first tensor data obtained by converting each of the plurality of video frame data.
[0151] According to an embodiment of the present disclosure, a dynamic link library is associated with at least one target deep learning model, and the at least one target deep learning model belongs to multiple deep learning models.
[0152] According to an embodiment of the present disclosure, the conversion module includes a first conversion unit, a second conversion unit, and a third conversion unit.
[0153] The first conversion unit is configured to generate at least one service instance for each video frame data based on at least one dynamic link library.
[0154] The second conversion unit is used to run at least one business instance, wherein the business instance is configured to load video frame data, process the video frame data based on the input interface standard of at least one target deep learning model related to the business instance, and obtain at least one sub-tensor data.
[0155] The third conversion unit is used to obtain at least one first tensor data based on at least one sub-tensor data corresponding to each of the at least one business instances.
[0156] According to an embodiment of the present disclosure, the video processing apparatus 1000 may further include a writing module.
[0157] The writing module is configured to write the video frame data into the storage unit in response to receiving the video frame data.
[0158] According to an embodiment of the present disclosure, the service instance is configured to load video frame data from a storage unit.
[0159] According to an embodiment of the present disclosure, the video processing apparatus 1000 may further include a configuration module, an update module, and a release module.
[0160] The configuration module is used to configure the count field of the video frame data based on the number of at least one dynamic link library.
[0161] The updating module is configured to update the value of the count field of the video frame data in response to detecting that the service instance loads the video frame data from the storage unit.
[0162] The releasing module is used to release the video frame data from the storage unit when the value of the count field of the video frame data is a preset value.
[0163] According to an embodiment of the present disclosure, the configuration module includes a first configuration unit and a second configuration unit.
[0164] The first configuration unit is configured to add a count field to the video frame data.
[0165] The second configuration unit is configured to determine an initial value of the count field based on the number of at least one dynamic link library.
[0166] According to an embodiment of the present disclosure, the allocation module 1030 includes a first allocation unit and a second allocation unit.
[0167] The first allocation unit is configured to determine, for each second tensor data, at least one target graphics processing unit from a plurality of graphics processing units based on a deep learning model corresponding to the second tensor data and deep learning models deployed by each of the plurality of graphics processing units, wherein the deep learning model deployed by the target graphics processing unit includes the deep learning model corresponding to the second tensor data.
[0168] The second distribution unit is configured to distribute the second tensor data to at least one target graphics processing unit.
[0169] According to an embodiment of the present disclosure, the second allocating unit includes a first allocating sub-unit and a second allocating sub-unit.
[0170] The first allocating sub-unit is configured to divide the second tensor data into sub-tensor data based on the resource margin of each of the at least one target graphics processing units to obtain at least one sub-data.
[0171] The second allocating sub-unit is configured to allocate at least one sub-data to at least one target graphics processing unit.
[0172] According to an embodiment of the present disclosure, the video processing apparatus 1000 may further include a decoding module.
[0173] The decoding module is used for decoding the video stream in response to receiving the video stream to obtain a plurality of video frame data.
[0174] According to the module of the embodiment of the present disclosure, submodule, unit, subunit, any multiple, or at least part of the function of any multiple thereof can be realized in a module. According to the module of the embodiment of the present disclosure, submodule, unit, subunit, any one or more can be split into multiple modules to realize. According to the module of the embodiment of the present disclosure, submodule, unit, subunit, any one or more can be at least partially realized as a hardware circuit, such as a field programmable gate array (FPGA), a programmable logic array (PLA), a system on a chip, a system on a substrate, a system on a package, an application specific integrated circuit (ASIC), or can be realized by hardware or firmware of any other reasonable way of integrating or encapsulating the circuit, or by any one of the three implementation modes of software, hardware and firmware or by a suitable combination of any of them. Or, according to the module of the embodiment of the present disclosure, submodule, unit, subunit, one or more can be at least partially realized as a computer program module, which can execute the corresponding function when the computer program module is run.
[0175] For example, any multiple of the first processing module 1010, the second processing module 1020, and the allocation module 1030 can be combined into one module / unit / sub-unit for implementation, or any one of the modules / units / sub-units can be split into multiple modules / units / sub-units. Alternatively, at least part of the functions of one or more of these modules / units / sub-units can be combined with at least part of the functions of other modules / units / sub-units and implemented in one module / unit / sub-unit. According to an embodiment of the present disclosure, at least one of the first processing module 1010, the second processing module 1020, and the allocation module 1030 can be at least partially implemented as a hardware circuit, such as a field programmable gate array (FPGA), a programmable logic array (PLA), a system on a chip, a system on a substrate, a system on a package, an application specific integrated circuit (ASIC), or can be implemented by hardware or firmware such as any other reasonable way of integrating or packaging the circuit, or implemented in any one of the three implementation modes of software, hardware, and firmware, or in an appropriate combination of any of them. Alternatively, at least one of the first processing module 1010 , the second processing module 1020 , and the allocation module 1030 may be at least partially implemented as a computer program module, and when the computer program module is executed, the corresponding function may be performed.
[0176] It should be noted that the video data processing device part in the embodiment of the present disclosure corresponds to the video data processing method part in the embodiment of the present disclosure. The description of the video data processing device part specifically refers to the video data processing method part and will not be repeated here.
[0177] Figure 11 schematically shows a block diagram of an electronic device suitable for implementing the video data processing method according to an embodiment of the present disclosure. The electronic device shown in Figure 11 is only an example and should not bring any limitation to the functions and scope of use of the embodiment of the present disclosure.
[0178] As shown in Figure 11, the electronic device 1100 according to an embodiment of the present disclosure includes a processor 1101, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1102 or a program loaded from a storage portion 1108 into a random access memory (RAM) 1103. The processor 1101 may, for example, include a general-purpose microprocessor (e.g., a CPU), an instruction set processor and / or a related chipset and / or a dedicated microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 1101 may also include an onboard memory for caching purposes. The processor 1101 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of the present disclosure.
[0179] Various programs and data required for the operation of the electronic device 1100 are stored in the RAM 1103. The processor 1101, the ROM 1102, and the RAM 1103 are connected to each other via a bus 1104. The processor 1101 executes the various operations of the method flow according to the embodiment of the present disclosure by executing the programs in the ROM 1102 and / or the RAM 1103. It should be noted that the programs may also be stored in one or more memories other than the ROM 1102 and the RAM 1103. The processor 1101 may also execute the various operations of the method flow according to the embodiment of the present disclosure by executing the programs stored in the one or more memories.
[0180] According to an embodiment of the present disclosure, electronic device 1100 may further include an input / output (I / O) interface 1105, which is also connected to bus 1104. Electronic device 1100 may further include one or more of the following components connected to I / O interface 1105: an input section 1106 including a keyboard, mouse, etc.; an output section 1107 including devices such as a cathode ray tube (CRT), a liquid crystal display (LCD), and speakers; a storage section 1108 including a hard disk; and a communication section 1109 including a network interface card such as a LAN card or a modem. Communication section 1109 performs communication processing via a network such as the Internet. A drive 1110 is also connected to I / O interface 1105 as needed. Removable media 1111, such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory, is installed in drive 1110 as needed, so that computer programs read from the removable media can be installed into storage section 1108 as needed.
[0181] The present disclosure also provides a non-transitory computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments, or may exist independently and not be incorporated into the device / apparatus / system. The computer-readable storage medium carries one or more programs, and when executed, implements the method according to the embodiments of the present disclosure.
[0182] According to an embodiment of the present disclosure, a computer-readable storage medium may be a non-volatile computer-readable storage medium, such as but not limited to: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, apparatus, or device. For example, according to an embodiment of the present disclosure, a computer-readable storage medium may include the ROM 1102 and / or RAM 1103 described above and / or one or more memories other than ROM 1102 and RAM 1103.
[0183] Embodiments of the present disclosure also include a computer program product comprising a computer program containing program code for executing the method shown in the flowchart. When the computer program product is executed in a computer system, the program code is used to cause the computer system to implement the method for verifying process data of a product provided by the embodiments of the present disclosure.
[0184] The computer program executes the above functions defined in the system / device of the embodiment of the present disclosure when the computer program is executed by the processor 1101. According to the embodiment of the present disclosure, the system, device, module, unit, etc. described above can be implemented by a computer program module.
[0185] In one embodiment, the computer program may be stored on a tangible storage medium such as an optical storage device or a magnetic storage device. In another embodiment, the computer program may be transmitted and distributed in the form of a signal on a network medium, downloaded and installed via the communication portion 1109, and / or installed from removable media 1111. The program code contained in the computer program may be transmitted using any appropriate network medium, including but not limited to wireless, wired, or any suitable combination thereof.
[0186] In such an embodiment, the computer program can be downloaded and installed from a network via the communication section 1109 and / or installed from the removable medium 1111. When the computer program is executed by the processor 1101, the above-described functions defined in the system of the embodiment of the present disclosure are performed. According to the embodiment of the present disclosure, the systems, devices, means, modules, units, etc. described above can be implemented by computer program modules.
[0187] According to an embodiment of the present disclosure, the program code for executing the computer program provided by the embodiment of the present disclosure can be written in any combination of one or more programming languages. Specifically, these computer programs can be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages include, but are not limited to, languages such as Java, C++, python, "C" or similar programming languages. The program code can be executed entirely on the user computing device, partially on the user device, partially on a remote computing device, or entirely on a remote computing device or server. In cases involving a remote computing device, the remote computing device can be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computing device (for example, using an Internet service provider to connect via the Internet).
[0188] The flowcharts and block diagrams in the accompanying drawings illustrate the possible implementation architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each box in the flowchart or block diagram can represent a module, program segment, or a part of code, and the above-mentioned module, program segment, or a part of code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in an order different from that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flowchart, and the combination of boxes in the block diagram or flowchart, can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.
[0189] Those skilled in the art will appreciate that the features described in the various embodiments and / or claims of this disclosure may be combined and / or coupled in various ways, even if such combinations and / or couplings are not explicitly described in this disclosure. In particular, the features described in the various embodiments and / or claims of this disclosure may be combined and / or coupled in various ways without departing from the spirit and teachings of this disclosure. All such combinations and / or couplings are intended to fall within the scope of this disclosure.
[0190] The embodiments of the present disclosure are described above. However, these embodiments are for illustrative purposes only and are not intended to limit the scope of the present disclosure. Although each embodiment has been described separately above, this does not mean that the measures in each embodiment cannot be used in combination to advantage. The scope of the present disclosure is defined by the appended claims and their equivalents. Without departing from the scope of the present disclosure, those skilled in the art may make various substitutions and modifications, which should all fall within the scope of the present disclosure.
Claims
1. A video data processing method, comprising: Based on a plurality of video frame data received within a preset time window, a plurality of first tensor data are obtained, wherein the first tensor data includes a plurality of sub-tensor data corresponding to each of the plurality of deep learning models; splicing the multiple sub-tensor data included in each of the multiple first tensor data to obtain multiple second tensor data corresponding to each of the multiple deep learning models; and Based on the deep learning models respectively deployed by the multiple graphics processing units included in the graphics processor, the multiple second tensor data are distributed to the multiple graphics processing units so as to perform data inference on the multiple second tensor data through the graphics processor.
2. The method according to claim 1, wherein: The method of obtaining a plurality of first tensor data based on a plurality of video frame data received within a preset time window includes: For each of the plurality of video frame data received within the preset time window, converting the video frame data into at least one first tensor data based on at least one loaded dynamic link library; and The plurality of first tensor data are obtained based on at least one first tensor data obtained by converting each of the plurality of video frame data.
3. The method according to claim 2, wherein: The dynamic link library is associated with at least one target deep learning model, and the at least one target deep learning model belongs to the multiple deep learning models; The step of converting each of the plurality of video frame data received within the preset time window into at least one first tensor data based on at least one loaded dynamic link library comprises: For each of the video frame data, generating at least one service instance based on the at least one dynamic link library; Running the at least one business instance, wherein the business instance is configured to load the video frame data, process the video frame data based on the input interface standard of each of the at least one target deep learning model related to the business instance, and obtain at least one of the sub-tensor data; and Based on the at least one sub-tensor data corresponding to each of the at least one business instance, the at least one first tensor data is obtained.
4. The method according to claim 3, further comprising: In response to receiving the video frame data, writing the video frame data into a storage unit; Wherein, the service instance is configured to load the video frame data from the storage unit.
5. The method according to claim 4, further comprising: Based on the number of the at least one dynamic link library, configuring a count field of the video frame data; In response to detecting that the service instance loads the video frame data from the storage unit, updating a value of a count field of the video frame data; as well as When the value of the count field of the video frame data is a preset value, the video frame data is released from the storage unit.
6. The method according to claim 5, wherein: The configuring the count field of the video frame data based on the number of the at least one dynamic link library comprises: adding the count field to the video frame data; and An initial value of the count field is determined based on the number of the at least one dynamic link library.
7. The method according to claim 1, wherein: The method of allocating the plurality of second tensor data to the plurality of graphics processing units based on the deep learning model respectively deployed by the plurality of graphics processing units included in the graphics processor comprises: For each of the second tensor data, determining at least one target graphics processing unit from the plurality of graphics processing units based on the deep learning model corresponding to the second tensor data and the deep learning models respectively deployed by the plurality of graphics processing units, wherein the deep learning model deployed by the target graphics processing unit includes the deep learning model corresponding to the second tensor data; and The second tensor data is distributed to the at least one target graphics processing unit.
8. The method according to claim 7, wherein: The allocating the second tensor data to the at least one target graphics processing unit comprises: Based on the resource margin of each of the at least one target graphics processing unit, the second tensor data is segmented with the sub-tensor data as a granularity to obtain at least one sub-data; and The at least one sub-data is distributed to the at least one target graphics processing unit.
9. The method according to claim 1, further comprising: In response to receiving a video stream, the video stream is decoded to obtain the video frame data.
10. A video data processing device, comprising: A first processing module, configured to obtain a plurality of first tensor data based on a plurality of video frame data received within a preset time window, wherein the first tensor data includes a plurality of sub-tensor data corresponding to each of the plurality of deep learning models; A second processing module is used to splice the multiple sub-tensor data included in each of the multiple first tensor data to obtain multiple second tensor data corresponding to each of the multiple deep learning models; and An allocation module is used to allocate the multiple second tensor data to the multiple graphics processing units included in the graphics processor based on at least one of the deep learning models deployed by each of the multiple graphics processing units, so as to perform data inference on the multiple second tensor data through the graphics processor.
11. An electronic device, comprising a memory and a processor, wherein the memory stores instructions executable by the processor, and when the instructions are executed by the processor, the processor executes the method according to any one of claims 1 to 9.
12. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1 to 9.
13. A computer program product comprising a computer program, which, when executed by a processor, implements the method according to any one of claims 1 to 9.
14. An edge device, comprising: Business plugin manager, inference manager, and model manager; Wherein, the service plug-in manager is configured as follows: Based on a plurality of video frame data received within a preset time window, a plurality of first tensor data are obtained, wherein the first tensor data includes a plurality of sub-tensor data corresponding to each of the plurality of deep learning models; The inference manager is configured to: splicing the multiple sub-tensor data included in each of the multiple first tensor data to obtain multiple second tensor data corresponding to each of the multiple deep learning models; and Based on the deep learning model respectively deployed by the plurality of graphics processing units included in the graphics processor, the plurality of second tensor data are distributed to the plurality of graphics processing units; and The model manager is configured to: Data inference is performed on the plurality of second tensor data by the graphics processor.
15. The edge device according to claim 14, wherein: The service plug-in manager is configured to: For each of the plurality of video frame data received within the preset time window, converting the video frame data into at least one first tensor data based on at least one loaded dynamic link library; as well as The plurality of first tensor data are obtained based on at least one first tensor data obtained by converting each of the plurality of video frame data.
16. The edge device according to claim 15, wherein: The dynamic link library is associated with at least one target deep learning model, and the at least one target deep learning model belongs to the multiple deep learning models; Wherein, the service plug-in manager is configured as follows: In response to receiving the video frame data, generating at least one service instance based on the at least one dynamic link library; Running the at least one business instance, wherein the business instance is configured to load the video frame data, process the video frame data based on the input interface standard of each of the at least one target deep learning model related to the business instance, and obtain at least one of the sub-tensor data; and Based on the at least one sub-tensor data corresponding to each of the at least one business instance, the at least one first tensor data is obtained.
17. The edge device according to claim 16, wherein: The edge device also includes a frame multiplexer; The frame multiplexer is configured as follows: In response to receiving the video frame data, writing the video frame data into a storage unit; Wherein, the service plug-in manager is configured as follows: The video frame data is loaded from the storage unit through the service instance.
18. The edge device according to claim 17, wherein: The frame multiplexer is configured as follows: Based on the number of the at least one dynamic link library, configuring a count field of the video frame data; In response to detecting that the service instance loads the video frame data from the storage unit, updating a value of a count field of the video frame data; as well as When the value of the count field of the video frame data is a preset value, the video frame data is released from the storage unit.
19. The edge device according to claim 18, wherein: The frame multiplexer is configured as follows: adding the count field to the video frame data; and An initial value of the count field is determined based on the number of the at least one dynamic link library.
20. The edge device according to claim 14, wherein: The inference manager is configured to: For each of the second tensor data, determining at least one target graphics processing unit from the plurality of graphics processing units based on the deep learning model corresponding to the second tensor data and the deep learning models respectively deployed by the plurality of graphics processing units, wherein the deep learning model deployed by the target graphics processing unit includes the deep learning model corresponding to the second tensor data; and The second tensor data is distributed to the at least one target graphics processing unit.
21. The edge device according to claim 20, wherein: The inference manager is configured to: Based on the resource margin of each of the at least one target graphics processing unit, the second tensor data is segmented with the sub-tensor data as a granularity to obtain at least one sub-data; as well as The at least one sub-data is distributed to the at least one target graphics processing unit.
22. The edge device of claim 14, wherein: The service plug-in manager is configured to: In response to receiving a video stream, the video stream is decoded to obtain the video frame data.
Citation Information
Patent Citations
Data processing method and device, and storage medium
CN107563512A
Cigarette packet missing detection method based on deep learning
CN114092741A
Data processing method and device, processor, electronic equipment and storage medium
CN115269003A
Neural network model training method and device, electronic equipment and storage medium
CN115688917A
Avoid system restart after applying fixes
US20230091512A1
