Methods, devices, terminals and media for scheduling video analytics tasks in heterogeneous application systems

Abstract

This invention discloses a method, apparatus, terminal, and medium for scheduling video analysis tasks in a heterogeneous application system. The method includes: acquiring the computing power parameters and video decoding parameters required for the video analysis task; determining the scheduling strategy and corresponding computing modules for the video analysis task based on the required computing power parameters, video decoding parameters, and the computing power parameters and video encoding / decoding parameters of each computing module; and scheduling the video decoding and computing tasks within the video analysis task to the corresponding computing modules according to the scheduling strategy to execute the corresponding tasks. This invention utilizes video data communication transmission between multiple computing modules in a heterogeneous application system to schedule multiple computing modules to complete video analysis tasks, maximizing the utilization of the computing power and video encoding / decoding capabilities of the heterogeneous application system, increasing the number of video analysis task processing paths supported by the system, and improving the efficiency of video analysis.

Description

Technical Field

[0001] This invention relates to the field of video processing technology, and in particular to a method, apparatus, terminal and medium for scheduling video analysis tasks in heterogeneous application systems. Background Technology

[0002] In fields such as smart transportation, smart security, smart construction sites, and intelligent industrial manufacturing, a large amount of camera video data is accessed. Edge computing devices then analyze and process this multi-channel video data. This includes tasks such as detecting traffic congestion at intersections, vehicle feature detection (e.g., vehicle count, license plate number, vehicle type, and vehicle color), dump truck detection and tracking, personnel feature detection (e.g., headcount, gender, age, clothing, and mask detection), facial recognition, construction helmet detection, and product appearance anomaly detection. Typically, a large number of cameras are connected, requiring each edge computing device to analyze and process a significant amount of video data. Since the processing power of a single hardware platform's computing modules is limited, edge computing devices may employ multiple computing modules. Furthermore, depending on specific business needs and cost requirements, the multiple computing modules of an edge computing device may be designed heterogeneously, forming a heterogeneous application system.

[0003] For video detection and analysis tasks, industry edge computing devices or heterogeneous application systems typically schedule a video analysis task to a specific computing module based on the computing power and video decoding capabilities of each built-in computing module, as well as the computing power and video resources required for the video analysis task. If the remaining computing power and video resources of all computing modules in the edge computing device are insufficient, the scheduling of that video analysis task may fail. At this point, some computing modules may have sufficient computing power, while others may have sufficient video processing capabilities, resulting in a waste of hardware resources in the edge computing device.

[0004] Furthermore, regarding the computing modules on different hardware platforms currently available in the industry, some modules have high computing power but lower video encoding and decoding capabilities, while others have strong video encoding and decoding capabilities but lower computing power. Different computing power configurations and video decoding processing capability parameters result in varying costs and power consumption for the computing modules. To handle video detection and analysis tasks from multiple cameras, both high computing power and significant video decoding processing capabilities may be required simultaneously, potentially leading to very high costs for the entire edge computing device. Therefore, for computing modules on different hardware platforms, the issue of low processing efficiency for video analysis tasks also exists.

[0005] Therefore, existing technologies still need improvement. Summary of the Invention

[0006] The technical problem to be solved by the present invention is to provide a method, device, terminal and medium for scheduling video analysis tasks in heterogeneous application systems, in order to solve the technical problem of low processing efficiency of existing video analysis tasks, in view of the defects of the prior art.

[0007] The technical solution adopted by this invention to solve the technical problem is as follows:

[0008] In a first aspect, the present invention provides a method for scheduling video analytics tasks in a heterogeneous application system, comprising:

[0009] Obtain the computing power parameters and video decoding parameters required for the video analysis task;

[0010] Based on the required computing power parameters, video decoding parameters, and the computing power parameters and video encoding / decoding parameters of each computing module, determine the scheduling strategy for the video analysis task and the corresponding computing modules;

[0011] According to the scheduling strategy, the video decoding task and the computing task in the video analysis task are scheduled to the corresponding computing modules to execute the corresponding tasks.

[0012] In one implementation, the step of obtaining the computing power parameters and video decoding parameters required for the video analysis task includes, prior to:

[0013] Configure the computing power parameters and video encoding / decoding parameters of each computing module according to the preset parameters.

[0014] In one implementation, the step of obtaining the computing power parameters and video decoding parameters required for the video analysis task further includes:

[0015] Obtain the computing power parameters and video encoding / decoding parameters reported by each computing module.

[0016] In one implementation, obtaining the computing power parameters and video decoding parameters required for the video analysis task includes:

[0017] Acquire video data information collected by each video acquisition device, and configure the video analysis task based on the video data information;

[0018] The computational power parameters and video decoding parameters required for the video analysis task are analyzed.

[0019] In one implementation, determining the scheduling strategy and corresponding computing modules for the video analysis task based on the required computing power parameters, video decoding parameters, and the computing power parameters and video encoding / decoding parameters of each computing module includes:

[0020] Based on the required computing power parameters and video decoding parameters, as well as the computing power parameters and video encoding / decoding parameters of each computing module, determine in turn whether each computing module simultaneously meets the video decoding requirements and computing requirements;

[0021] If each computing module cannot simultaneously satisfy the video decoding requirement and the computing requirement, then a first computing module that satisfies the video decoding requirement and a second computing module that satisfies the computing requirement are determined.

[0022] In one implementation, scheduling the video decoding task and the computation task in the video analysis task to the corresponding computation module according to the scheduling strategy includes:

[0023] The video decoding task in the video analysis task is scheduled to the first computing module for decoding to obtain the decoded video data;

[0024] The computational task in the video analysis task is scheduled to the second computation module, and the decoded video data is sent to the second computation module for analysis and detection, and the analysis results of the video analysis task are output.

[0025] In one implementation, determining the scheduling strategy and corresponding computing modules for the video analysis task based on the required computing power parameters, video decoding parameters, and the computing power parameters and video encoding / decoding parameters of each computing module further includes:

[0026] Based on the required computing power parameters and video decoding parameters, as well as the computing power parameters and video encoding / decoding parameters of each computing module, determine in turn whether each computing module meets the video decoding requirements;

[0027] If the computing modules cannot meet the video decoding requirements, the corresponding video acquisition device is controlled via protocol to convert the video data compression format to a preset compression format.

[0028] A third calculation module that determines the decoding requirements of video data that meets the preset format;

[0029] The fourth calculation module that meets the calculation requirements is identified.

[0030] In one implementation, scheduling the video decoding task and the computation task in the video analysis task to the corresponding computation module according to the scheduling strategy further includes:

[0031] The video decoding task in the video analysis task is scheduled to the third computing module for decoding to obtain the decoded video data;

[0032] The computational tasks in the video analysis task are scheduled to the fourth computation module, and the decoded video data is sent to the fourth computation module for analysis and detection, and the analysis results of the video analysis task are output.

[0033] In one implementation, the decoded video data is the original YUV or RGB format data, or the decoded video data is data that meets the size requirements of the analysis algorithm, or the decoded video data is recompressed H264 / H265 / JPGE compressed image data.

[0034] In one implementation, scheduling the video decoding task and the computation task in the video analysis task to the corresponding computation module according to the scheduling strategy further includes:

[0035] Determine the degradation scheduling strategy based on the minimum processing frame rate and minimum computing power requirement in the task parameters;

[0036] According to the degradation scheduling strategy, the processing frame rate and computing power requirements are gradually reduced until the task scheduling is successful or the minimum processing frame rate and minimum computing power requirements are reached.

[0037] One implementation also includes:

[0038] Receive batch video analysis tasks and obtain the computing power parameters and video decoding parameters required for each video analysis task;

[0039] Obtain the computing power parameters and video encoding / decoding parameters of each computing module, and obtain the video encoding and compression parameters supported by each video acquisition device;

[0040] Based on the computing power parameters and video decoding parameters required for each video analysis task, the computing power parameters and video encoding and decoding parameters of each computing module, and the video encoding and compression parameters supported by each video acquisition device, the optimal scheduling strategy parameters are calculated comprehensively.

[0041] The corresponding computing modules are scheduled according to the optimal scheduling strategy parameters to execute the video decoding and computing tasks of the batch video analysis task.

[0042] Secondly, the present invention provides a video analysis task scheduling device for heterogeneous application systems, comprising:

[0043] The task parameter acquisition module is used to acquire the computing power parameters and video decoding parameters required for the video analysis task;

[0044] The scheduling strategy module is used to determine the scheduling strategy and corresponding computing modules for the video analysis task based on the required computing power parameters, video decoding parameters, and the computing power parameters and video encoding / decoding parameters of each computing module.

[0045] The task execution module is used to schedule the video decoding task and the computing task in the video analysis task to the corresponding computing module according to the scheduling strategy, so as to execute the corresponding task.

[0046] Thirdly, the present invention provides a terminal, comprising: a processor and a memory, wherein the memory stores a heterogeneous application system video analysis task scheduler, and the heterogeneous application system video analysis task scheduler, when executed by the processor, is used to implement the operation of the heterogeneous application system video analysis task scheduling method as described in the first aspect.

[0047] Fourthly, the present invention also provides a medium, which is a computer-readable storage medium, storing a heterogeneous application system video analysis task scheduler, which, when executed by a processor, is used to implement the operation of the heterogeneous application system video analysis task scheduling method as described in the first aspect.

[0048] The present invention, by employing the above technical solution, has the following effects:

[0049] This invention obtains the computing power parameters and video decoding parameters required for video analysis tasks. Based on these parameters, along with the computing power parameters and video encoding / decoding parameters of each computing module, it determines the scheduling strategy and corresponding computing modules for the video analysis tasks. According to the scheduling strategy, the video decoding and computing tasks within the video analysis task are scheduled to the corresponding computing modules for execution. Furthermore, this invention utilizes video data communication and transmission between multiple computing modules in a heterogeneous application system to schedule multiple computing modules to complete video analysis tasks, maximizing the utilization of the computing power and video encoding / decoding capabilities of the heterogeneous application system, increasing the number of video analysis task processing paths supported by the system, and improving the efficiency of video analysis. Attached Figure Description

[0050] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the structures shown in these drawings without creative effort.

[0051] Figure 1 This is a flowchart of a video analysis task scheduling method for a heterogeneous application system in one implementation of the present invention.

[0052] Figure 2 This is a schematic diagram of the first heterogeneous application system in one implementation of the present invention.

[0053] Figure 3 This is a schematic diagram of a second heterogeneous application system in one implementation of the present invention.

[0054] Figure 4 This is a schematic diagram of the first method for obtaining computing power parameters and video encoding / decoding parameters in one implementation of the present invention.

[0055] Figure 5 This is a schematic diagram of a second method for obtaining computing power parameters and video encoding / decoding parameters in one implementation of the present invention.

[0056] Figure 6 This is a schematic diagram of the scheduling of a single video analysis task in one implementation of the present invention.

[0057] Figure 7 This is a schematic diagram of the first transmission of decoded video data in one implementation of the present invention.

[0058] Figure 8 This is a schematic diagram of a second transmission method for decoded video data in one implementation of the present invention.

[0059] Figure 9 This is a schematic diagram illustrating the configuration of video encoding and compression parameters of a video acquisition device in one implementation of the present invention.

[0060] Figure 10 This is a schematic diagram illustrating the reporting of video encoding and compression parameters by a video acquisition device in one implementation of the present invention.

[0061] Figure 11 This is a schematic diagram illustrating the switching of encoding and compression formats via protocol in one implementation of the present invention.

[0062] Figure 12 This is a schematic diagram of the scheduling of batch video analysis tasks in one implementation of the present invention.

[0063] Figure 13 This is a functional schematic diagram of the terminal in one implementation of the present invention.

[0064] The realization of the objective, functional features and advantages of the present invention will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0065] To make the objectives, technical solutions, and advantages of this invention clearer and more explicit, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.

[0066] Exemplary methods

[0067] Currently, the computing modules on different hardware platforms in the industry vary in performance. Some modules have high computing power but lower video encoding / decoding capabilities, while others have strong video encoding / decoding capabilities but lower computing power. Different computing power and video decoding capabilities result in different costs and power consumption for the computing modules. To handle video detection and analysis tasks from multiple cameras, both high computing power and significant video decoding capabilities may be required simultaneously. This can lead to very high costs for the entire edge computing device and prevent the full utilization of each computing module's capabilities. Therefore, the low processing efficiency of video analysis tasks remains a problem for computing modules on different hardware platforms.

[0068] To address the aforementioned technical issues, this embodiment provides a video analysis task scheduling method for heterogeneous application systems. This method utilizes video data communication and transmission between multiple computing modules within the heterogeneous application system to schedule multiple computing modules to complete video analysis tasks. This maximizes the utilization of the computing power and video encoding / decoding capabilities of the heterogeneous application system, increases the number of video analysis task processing paths supported by the system, and improves the efficiency of video analysis.

[0069] like Figure 1 As shown, this embodiment of the invention provides a method for scheduling video analytics tasks in a heterogeneous application system, comprising the following steps:

[0070] Step S100: Obtain the computing power parameters and video decoding parameters required for the video analysis task.

[0071] In this embodiment, the heterogeneous application system video analysis task scheduling method is applied to a terminal, which includes, but is not limited to, devices such as computers.

[0072] In this embodiment, the terminal can be an independent control module (i.e., management module) within the entire heterogeneous application system, such as... Figure 2 As shown, the control module is connected to multiple computing modules and is used to control these modules. Within this heterogeneous application system, there are multiple computing modules and one control module. Each computing module possesses a certain computing power or video encoding / decoding capability. The control module is responsible for scheduling all video analysis tasks. Data communication channels, such as network channels, exist between all computing modules and the control module, as well as between all computing modules themselves. All video acquisition devices, such as cameras, communicate with the heterogeneous application system through these network channels.

[0073] In this embodiment, the terminal can also be a computing module within the entire heterogeneous application system, such as... Figure 3 As shown, the control module is one of the computing modules, used by the other computing modules; in this system, the functions of the control module can be performed by one of the computing modules.

[0074] In this embodiment, as Figure 4 As shown, the computing power and video encoding / decoding format capabilities of all computing modules in a heterogeneous application system can be configured to the control module side via parameters.

[0075] Specifically, in one implementation of this embodiment, the following steps are included before step S100:

[0076] Step S101a: Configure the computing power parameters and video encoding / decoding parameters of each computing module according to preset parameters.

[0077] In this embodiment, during the process of configuring the computing power parameters and video encoding / decoding parameters of each computing module, the hardware information of each computing module, such as CPU information and GPU information, can be obtained. Then, based on the hardware information of each computing module, preset parameters are determined to obtain the computing power parameters and video encoding / decoding parameters of each computing module.

[0078] Specifically, in one implementation of this embodiment, the following steps are included before step S100:

[0079] Step S101b: Obtain the computing power parameters and video encoding / decoding parameters reported by each computing module.

[0080] In this embodiment, as Figure 5 As shown, the computing power and video encoding / decoding capabilities of all computing modules in the heterogeneous application system can also be reported by each computing module to the control module. The control module can then obtain the computing power parameters and video encoding / decoding parameters of each computing module.

[0081] Specifically, in one implementation of this embodiment, step S100 includes the following steps:

[0082] Step S101: Obtain video data information collected by each video acquisition device, and configure the video analysis task according to the video data information;

[0083] Step S102: Analyze the computing power parameters and video decoding parameters required for the video analysis task.

[0084] In this embodiment, the video data information acquired by the control module comes from the video acquisition devices connected to each computing module, or it can come from the video acquisition devices directly connected to the control module. After acquiring the video data information, the size, resolution, and format of the video data acquired by each video acquisition device are determined, and a video analysis task is configured based on this information. The computing power parameters and video decoding parameters required for the video analysis task are also analyzed to determine the corresponding task scheduling strategy based on these required parameters.

[0085] like Figure 1 As shown, in one implementation of this invention, the heterogeneous application system video analysis task scheduling method further includes the following steps:

[0086] Step S200: Based on the required computing power parameters, video decoding parameters, and the computing power parameters and video encoding / decoding parameters of each computing module, determine the scheduling strategy for the video analysis task and the corresponding computing modules.

[0087] In this embodiment, the control module obtains the computing power and video decoding capability requirements of each video analysis task based on parameters such as the computing power and video encoding / decoding capability of each computing module in the system, and schedules the video analysis task to a computing module that meets the computing power and video decoding capability requirements; the specific scheduling strategy includes, but is not limited to, the following three methods.

[0088] Specifically, in one implementation of this embodiment, step S200 includes the following steps:

[0089] Step S211: Based on the required computing power parameters and video decoding parameters, as well as the computing power parameters and video encoding / decoding parameters of each computing module, determine in turn whether each computing module simultaneously meets the video decoding requirements and computing requirements.

[0090] Step S212: If each calculation module cannot simultaneously meet the video decoding requirements and the calculation requirements, then determine the first calculation module that meets the video decoding requirements and the second calculation module that meets the calculation requirements.

[0091] In this embodiment, the first scheduling strategy is as follows: If the control module finds that the computing power and video decoding capability required for a certain video analysis task cannot be simultaneously met by all computing modules in the system, then it separately checks the computing modules that meet the computing power and video decoding capability requirements. If computing module A (i.e., the first computing module) that meets the video decoding capability requirements and computing module B (i.e., the second computing module) that meets the computing power requirements are found, then computing module A is set as the execution module for the video decoding task, and computing module B is set as the execution module for the computing task.

[0092] Specifically, in one implementation of this embodiment, step S200 further includes the following steps:

[0093] Step S221: Based on the required computing power parameters and video decoding parameters, as well as the computing power parameters and video encoding / decoding parameters of each computing module, determine in turn whether each computing module meets the video decoding requirements;

[0094] Step S222: If each computing module cannot meet the video decoding requirements, the corresponding video acquisition device is controlled through the protocol to convert the video data compression format to a preset compression format.

[0095] Step S223: Determine the third calculation module that meets the decoding requirements of the video data in the preset format;

[0096] Step S224: Determine the fourth calculation module that meets the calculation requirements.

[0097] In this embodiment, the second scheduling strategy is as follows: If the control module finds that the computing power and video decoding capabilities required for a certain video analysis task are insufficient for all computing modules in the system to simultaneously meet the requirements, and there is no computing module that meets the video decoding capabilities, but a computing module that meets the computing power requirements is detected, then the control module can control the corresponding video acquisition device or other video acquisition device via a protocol to switch the video data compression format of the video acquisition device to a preset compression format; wherein, the preset compression format is the format that meets the current system's video decoding capability requirements. If the video compression format of the video acquisition device is successfully switched, the control module detects one or more computing modules (i.e., the third computing module) that meet the new video decoding capability requirements and sets them as the execution module for the video decoding task; and detects computing modules (i.e., the fourth computing module) that meet the requirements for the video analysis task and sets them as the execution module for the computing task.

[0098] In the second scheduling strategy mentioned above, the video data encoding format of the video acquisition device can be modified to switch between various video image formats such as H264, H265, and JPEG. Alternatively, lower resolution and lower frame rate video or raw image data can be sent directly as needed by the algorithm.

[0099] In this embodiment, the third scheduling strategy is as follows: if the control module finds that one or more computing modules in the system can simultaneously meet the computing power and video decoding capabilities required by a certain video analysis task, then this computing module is set as the execution module of the video analysis task, or the computing module with the best capabilities is set as the execution module of the video analysis task based on the strength of computing power and video decoding capabilities.

[0100] In this embodiment, video data communication and transmission between multiple computing modules in a heterogeneous application system are used to schedule multiple computing modules to complete a video analysis task, thereby maximizing the utilization of the computing power and video encoding and decoding capabilities of the heterogeneous application system and increasing the number of video analysis task processing paths supported by the system.

[0101] like Figure 1 As shown, in one implementation of this invention, the heterogeneous application system video analysis task scheduling method further includes the following steps:

[0102] Step S300: According to the scheduling strategy, the video decoding task and the computing task in the video analysis task are scheduled to the corresponding computing modules to execute the corresponding tasks.

[0103] In this embodiment, different tasks are executed in the corresponding computing modules for the three different scheduling strategies mentioned above. The specific execution process is as follows.

[0104] Specifically, in one implementation of this embodiment, step S300 includes the following steps:

[0105] Step S311: Schedule the video decoding task in the video analysis task to the first computing module for decoding to obtain the decoded video data;

[0106] Step S312: Schedule the computation task in the video analysis task to the second computation module, send the decoded video data to the second computation module for analysis and detection, and output the analysis result of the video analysis task.

[0107] In this embodiment, for the first scheduling strategy, the control module controls the computing module A to perform video decoding. After obtaining the decoded video data, the video data is sent to the computing module B through the communication channel between the computing modules. The computing module B then performs video analysis and detection to obtain the final video analysis result.

[0108] Specifically, in one implementation of this embodiment, step S300 further includes the following steps:

[0109] Step S321: Schedule the video decoding task in the video analysis task to the third computing module for decoding to obtain the decoded video data;

[0110] Step S322: Schedule the computation task in the video analysis task to the fourth computation module, send the decoded video data to the fourth computation module for analysis and detection, and output the analysis result of the video analysis task.

[0111] In this embodiment, for the second scheduling strategy, the control module will control the third computing module to complete the video decoding. After obtaining the decoded video data, the video data will be sent to the fourth computing module through the communication channel between the computing modules. The fourth computing module will then perform video analysis and detection to obtain the final video analysis result.

[0112] In this embodiment, for the third scheduling strategy, the control module will control the computing module that simultaneously meets the conditions to directly execute the video decoding task and the video analysis task, or control the computing module with the best control capability to directly execute the video decoding task and the video analysis task.

[0113] In this embodiment, during the execution of the first or second scheduling strategy, the decoded video data is the original YUV or RGB format data, or the decoded video data is data that meets the size requirements of the analysis algorithm, or the decoded video data is recompressed H264 / H265 / JPGE compressed image data.

[0114] Taking the first scheduling strategy as an example, such as Figure 7 As shown, the transmission of video data between computing module A and computing module B can utilize the original YUV or RGB data (including other image data formats such as RGBA and ARGB) after video decoding. Alternatively, it can be compressed to the required size (416*416 or 224*224) according to the needs of the video analysis algorithm before transmission over the network to reduce the amount of data transmitted. Video data sent from computing module A to computing module B can be transmitted at the frame rate required by the video analysis algorithm; it is not necessary to send every frame.

[0115] like Figure 8 As shown, the transmission of video data between computing module A and computing module B can be optimized based on the encoding and decoding capabilities of the two computing modules for various video / image formats such as H.264, H.265, and JPEG. Compressing can be performed on computing module A according to a specific encoding format that meets the capabilities of computing module B. In other words, computing module A compresses the transmitted video / image data before transmitting it to computing module B. Computing module B then decompresses the received video / image data and performs video analysis calculations. When compressing video / image data, computing module A can compress it to a size of 416*416 or 224*224, and adjust the frame rate required by the video analysis algorithm, depending on the algorithm's needs.

[0116] like Figure 9 As shown, for the second scheduling strategy, the video encoding and compression capability parameters supported by each video acquisition device can be uniformly configured on the control module through parameters.

[0117] like Figure 10 As shown, for the second scheduling strategy, the video encoding compression capability parameters supported by each video acquisition device can also be actively reported to the control module by each video acquisition device.

[0118] like Figure 11 As shown, for the second scheduling strategy, the control module can set the video data of a certain video acquisition device to a specified encoding and compression format via a protocol. The communication protocol can refer to standard protocols such as GB / T-28181 or Digital Retina, or provide corresponding extended support.

[0119] Specifically, in one implementation of this embodiment, step S300 further includes the following steps:

[0120] Step S331: Determine the degradation scheduling strategy based on the minimum processing frame rate and minimum computing power requirement in the task parameters;

[0121] Step S332: Gradually reduce the processing frame rate and computing power parameter requirements according to the downgrade scheduling strategy until the task scheduling is successful or the minimum processing frame rate and minimum computing power requirements are reached.

[0122] In another implementation of this invention, for the aforementioned video analysis task, if configured as a degradeable scheduling task, the task parameters may include parameters such as the task's normal processing frame rate, normal computing power requirements, minimum processing frame rate, and minimum computing power requirements. If task scheduling fails under normal processing frame rate and computing power requirements, especially if the computing power cannot meet the requirements, degraded scheduling can be performed gradually from high to low. By gradually reducing the processing frame rate and computing power requirements, task scheduling is re-attempted until task scheduling succeeds or the minimum processing frame rate and computing power requirements are reached.

[0123] This embodiment addresses a specific video analysis task. If no computing module in the system can meet the video decoding requirements, but a computing module can meet the computing power requirements, then a certain communication protocol can be used to control the connected video acquisition devices, such as cameras, and modify the encoding format of the video data sent by the video acquisition devices to a format that the system can decode. This ultimately completes the scheduling of the video analysis task. By modifying the protocol, the hardware requirements for video decoding on the entire system are reduced.

[0124] In one implementation of this invention, the video analysis task scheduling method for heterogeneous application systems further includes the following steps:

[0125] Step S400: Receive batch video analysis tasks and obtain the computing power parameters and video decoding parameters required for each video analysis task;

[0126] Step S500: Obtain the computing power parameters and video encoding / decoding parameters of each computing module, and obtain the video encoding and compression parameters supported by each video acquisition device;

[0127] Step S600: Calculate the optimal scheduling strategy parameters based on the computing power parameters and video decoding parameters required for each video analysis task, the computing power parameters and video encoding / decoding parameters of each computing module, and the video encoding and compression parameters supported by each video acquisition device.

[0128] Step S700: Schedule the corresponding computing module according to the optimal scheduling strategy parameters to execute the video decoding task and computing task of the batch video analysis task.

[0129] In this embodiment, for calculating the optimal strategy during batch scheduling, the video decoding capabilities and computing power requirements for all video analysis tasks can be evaluated. Based on the maximum video decoding capabilities supported by the heterogeneous system, priority is given to switching the video acquisition devices to the video decoding capabilities supported by the system. Secondly, based on the computing power and video decoding capability requirements, video analysis tasks are scheduled to the same computing module whenever possible. Finally, multiple computing modules are used to jointly complete a single video analysis task.

[0130] This embodiment can comprehensively calculate the optimal scheduling strategy parameters for batch video analysis tasks, thereby maximizing the scheduling of video analysis tasks and reducing the consumption of system resources.

[0131] The following sections illustrate single video analysis tasks and batch video analysis tasks through practical application scenarios.

[0132] like Figure 6 As shown, in a single video analytics task scenario, the following steps are included:

[0133] Step S11: Obtain parameters such as computing power and video decoding capabilities required for the video analysis task;

[0134] Step S12: Determine if there is a computing module that meets the requirements for computing power and video decoding capabilities; if yes, proceed to step S13; if no, proceed to step S15.

[0135] Step S13: Schedule the video analysis task to a computing module that meets the requirements for computing power and video decoding capabilities;

[0136] Step S14: Task scheduling successful.

[0137] Steps S13 to S14 above represent the third scheduling strategy described in this embodiment.

[0138] Step S15: Determine whether there is a computing module that meets the computing power requirements and a computing module that meets the video decoding capability requirements; if yes, proceed to step S16; if no, proceed to step S18.

[0139] Step S16: The video decoding function is scheduled to be executed on computing module A, which meets the video decoding capability requirements; the video analysis calculation task is scheduled to be executed on computing module B, which meets the computing power requirements; computing module A sends the video data to computing module B.

[0140] Step S17: Task scheduling successful.

[0141] Steps S15 to S17 above represent the first scheduling strategy described in this embodiment.

[0142] Step S18: Determine if there is a computing module that meets the computing power requirements; if yes, proceed to step S20; if no, proceed to step S19.

[0143] Step S19: Task scheduling failed.

[0144] Step S20: Switch the video data format of the video acquisition device to a video decoding format that meets the system's current requirements via protocol control.

[0145] Step S21: Is the video format switching of the video acquisition device successful? If yes, return to step S12; if no, proceed to step S22.

[0146] Step S22, scheduling task failed.

[0147] Steps S20 to S21 described above represent the second scheduling strategy in this embodiment.

[0148] like Figure 12 As shown, the batch video analysis task scenario includes the following steps:

[0149] Step S31: Receive batch video analysis tasks;

[0150] Step S32: Obtain parameters such as computing power and video decoding capabilities required for each video analysis task;

[0151] Step S33: Obtain the computing power and video encoding / decoding capabilities of each computing module in the current system;

[0152] Step S34: Obtain the video encoding and compression capability parameters supported by each video acquisition device;

[0153] Step S35: Calculate the optimal scheduling strategy parameters.

[0154] Step S36: Complete the control and function scheduling of each computing module and video acquisition device.

[0155] This embodiment achieves the following technical effects through the above technical solution:

[0156] This embodiment obtains the computing power parameters and video decoding parameters required for the video analysis task. Based on these parameters, along with the computing power parameters and video encoding / decoding parameters of each computing module, it determines the scheduling strategy and corresponding computing modules for the video analysis task. According to the scheduling strategy, the video decoding and computing tasks within the video analysis task are scheduled to the corresponding computing modules for execution. This embodiment utilizes video data communication and transmission between multiple computing modules in a heterogeneous application system to schedule multiple computing modules to complete the video analysis task, maximizing the utilization of the computing power and video encoding / decoding capabilities of the heterogeneous application system, increasing the number of video analysis task processing paths supported by the system, and improving the efficiency of video analysis.

[0157] Exemplary device

[0158] Based on the above embodiments, the present invention also provides a video analysis task scheduling device for heterogeneous application systems, comprising:

[0159] Receive batch video analysis tasks and obtain the computing power parameters and video decoding parameters required for each video analysis task;

[0160] Obtain the computing power parameters and video encoding / decoding parameters of each computing module, and obtain the video encoding and compression parameters supported by each video acquisition device;

[0161] Based on the computing power parameters and video decoding parameters required for each video analysis task, the computing power parameters and video encoding and decoding parameters of each computing module, and the video encoding and compression parameters supported by each video acquisition device, the optimal scheduling strategy parameters are calculated comprehensively.

[0162] The corresponding computing modules are scheduled according to the optimal scheduling strategy parameters to execute the video decoding and computing tasks of the batch video analysis task.

[0163] Based on the above embodiments, the present invention also provides a terminal, the principle block diagram of which can be as follows: Figure 13 As shown.

[0164] The terminal includes: a processor, a memory, an interface, a display screen, and a communication module connected via a system bus; wherein, the processor of the terminal provides computing and control capabilities; the memory of the terminal includes a storage medium and internal memory; the storage medium stores the operating system and computer programs; the internal memory provides an environment for the operation of the operating system and computer programs in the storage medium; the interface is used to connect to external devices, such as mobile terminals and computers; the display screen is used to display relevant information; and the communication module is used to communicate with a cloud server or mobile terminal.

[0165] When executed by a processor, this computer program is used to implement a video analysis task scheduling method for a heterogeneous application system.

[0166] It will be understood by those skilled in the art that Figure 13 The schematic diagram shown is merely a partial structural diagram related to the present invention and does not constitute a limitation on the terminal to which the present invention is applied. A specific terminal may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0167] In one embodiment, a terminal is provided, comprising: a processor and a memory, the memory storing a heterogeneous application system video analysis task scheduler, which, when executed by the processor, is used to implement the operation of the heterogeneous application system video analysis task scheduling method described above.

[0168] In one embodiment, a storage medium is provided, wherein the storage medium stores a heterogeneous application system video analysis task scheduler, which, when executed by a processor, is used to implement the operation of the heterogeneous application system video analysis task scheduling method described above.

[0169] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile storage medium, and when executed, it can include the processes of the embodiments of the methods described above. Any references to memory, storage, databases, or other media used in the embodiments provided by this invention can include non-volatile and / or volatile memory.

[0170] In summary, this invention provides a method, apparatus, terminal, and medium for scheduling video analysis tasks in a heterogeneous application system. The method includes: acquiring the computing power parameters and video decoding parameters required for the video analysis task; determining the scheduling strategy and corresponding computing modules for the video analysis task based on the required computing power parameters, video decoding parameters, and the computing power parameters and video encoding / decoding parameters of each computing module; and scheduling the video decoding task and computing tasks in the video analysis task to the corresponding computing modules according to the scheduling strategy to execute the corresponding tasks. This invention schedules multiple computing modules to complete video analysis tasks through video data communication and transmission between multiple computing modules in a heterogeneous application system, maximizing the utilization of the computing power and video encoding / decoding capabilities of the heterogeneous application system, increasing the number of video analysis task processing paths supported by the system, and improving the efficiency of video analysis.

[0171] It should be understood that the application of the present invention is not limited to the examples above. Those skilled in the art can make improvements or modifications based on the above description, and all such improvements and modifications should fall within the protection scope of the appended claims.

Claims

1. A method for scheduling video analysis tasks in a heterogeneous application system, characterized in that, include: Obtain the computing power parameters and video decoding parameters required for the video analysis task; Based on the required computing power parameters, video decoding parameters, and the computing power parameters and video encoding / decoding parameters of each computing module, determine the scheduling strategy and corresponding computing modules for the video analysis tasks; obtain the computing power and video decoding capability requirements for each video analysis task, and schedule the video analysis tasks to a computing module that meets the computing power and video decoding capability requirements; According to the scheduling strategy, the video decoding task and the computing task in the video analysis task are scheduled to the corresponding computing modules to execute the corresponding tasks; The computing power parameters and video decoding parameters required for the video analysis task include: Acquire video data information collected by each video acquisition device, and configure the video analysis task based on the video data information; Analyze the computing power parameters and video decoding parameters required for the video analysis task; The step of determining the scheduling strategy and corresponding computing modules for the video analysis task based on the required computing power parameters, video decoding parameters, and the computing power parameters and video encoding / decoding parameters of each computing module includes: Based on the required computing power parameters and video decoding parameters, as well as the computing power parameters and video encoding / decoding parameters of each computing module, determine in turn whether each computing module simultaneously meets the video decoding requirements and computing requirements; If each computing module cannot simultaneously meet the video decoding requirements and the computing requirements, then a first computing module that meets the video decoding requirements and a second computing module that meets the computing requirements are determined. The step of scheduling the video decoding and computation tasks in the video analysis task to the corresponding computation modules according to the scheduling strategy includes: The video decoding task in the video analysis task is scheduled to the first computing module for decoding to obtain the decoded video data; The computational task in the video analysis task is scheduled to the second computation module, and the decoded video data is sent to the second computation module for analysis and detection, and the analysis results of the video analysis task are output.

2. The video analysis task scheduling method for heterogeneous application systems according to claim 1, characterized in that, The acquisition of the computing power parameters and video decoding parameters required for the video analysis task includes, prior to: Configure the computing power parameters and video encoding / decoding parameters of each computing module according to the preset parameters.

3. The heterogeneous application system video analysis task scheduling method according to claim 1, characterized in that, The process of obtaining the computing power parameters and video decoding parameters required for the video analysis task also previously included: Obtain the computing power parameters and video encoding / decoding parameters reported by each computing module.

4. The video analysis task scheduling method for heterogeneous application systems according to claim 1, characterized in that, The step of determining the scheduling strategy and corresponding computing modules for the video analysis task based on the required computing power parameters, video decoding parameters, and the computing power parameters and video encoding / decoding parameters of each computing module further includes: Based on the required computing power parameters and video decoding parameters, as well as the computing power parameters and video encoding / decoding parameters of each computing module, determine in turn whether each computing module meets the video decoding requirements; If the computing modules cannot meet the video decoding requirements, the corresponding video acquisition device is controlled via protocol to convert the video data compression format to a preset compression format. A third calculation module is used to determine the decoding requirements of video data that meets the preset compression format. The fourth calculation module that meets the calculation requirements is identified.

5. The video analysis task scheduling method for heterogeneous application systems according to claim 4, characterized in that, The step of scheduling the video decoding task and the computing task in the video analysis task to the corresponding computing module according to the scheduling strategy further includes: The video decoding task in the video analysis task is scheduled to the third computing module for decoding to obtain the decoded video data; The computational tasks in the video analysis task are scheduled to the fourth computation module, and the decoded video data is sent to the fourth computation module for analysis and detection, and the analysis results of the video analysis task are output.

6. The video analysis task scheduling method for heterogeneous application systems according to claim 1 or 5, characterized in that, The decoded video data is either in the original YUV or RGB format, or it is data that meets the size requirements of the analysis algorithm, or it is recompressed H264 / H265 / JPGE compressed image data.

7. The video analysis task scheduling method for heterogeneous application systems according to claim 1, characterized in that, The step of scheduling the video decoding task and the computing task in the video analysis task to the corresponding computing module according to the scheduling strategy further includes: Determine the degradation scheduling strategy based on the minimum processing frame rate and minimum computing power requirement in the task parameters; According to the degradation scheduling strategy, the processing frame rate and computing power requirements are gradually reduced until the task scheduling is successful or the minimum processing frame rate and minimum computing power requirements are reached.

8. The video analysis task scheduling method for heterogeneous application systems according to claim 1, characterized in that, Also includes: Receive batch video analysis tasks and obtain the computing power parameters and video decoding parameters required for each video analysis task; Obtain the computing power parameters and video encoding / decoding parameters of each computing module, and obtain the video encoding and compression parameters supported by each video acquisition device; Based on the computing power parameters and video decoding parameters required for each video analysis task, the computing power parameters and video encoding and decoding parameters of each computing module, and the video encoding and compression parameters supported by each video acquisition device, the optimal scheduling strategy parameters are calculated comprehensively. The corresponding computing modules are scheduled according to the optimal scheduling strategy parameters to execute the video decoding and computing tasks of the batch video analysis task.

9. A heterogeneous application system video analysis task scheduling device, used to implement the heterogeneous application system video analysis task scheduling method as described in any one of claims 1-8, characterized in that, include: The task parameter acquisition module is used to acquire the computing power parameters and video decoding parameters required for the video analysis task; The scheduling strategy module is used to determine the scheduling strategy and corresponding computing modules for the video analysis task based on the required computing power parameters, video decoding parameters, and the computing power parameters and video encoding / decoding parameters of each computing module. The task execution module is used to schedule the video decoding task and the computing task in the video analysis task to the corresponding computing module according to the scheduling strategy, so as to execute the corresponding task.

10. A terminal, characterized in that, include: The processor and memory, wherein the memory stores a heterogeneous application system video analysis task scheduler, which, when executed by the processor, is used to implement the operation of the heterogeneous application system video analysis task scheduling method as described in any one of claims 1-8.

11. A medium, characterized in that, The medium is a computer-readable storage medium that stores a heterogeneous application system video analysis task scheduler. When executed by a processor, the heterogeneous application system video analysis task scheduler is used to implement the operation of the heterogeneous application system video analysis task scheduling method as described in any one of claims 1-8.

Images