Video analysis method and device, electronic equipment, storage medium and program product

Abstract

The embodiment of the invention provides a video analysis method and device, electronic equipment, a storage medium and a program product. The video analysis method comprises the following steps: issuing a video analysis task based on preset task arrangement; selecting a target calculation engine from the calculation engines based on the convergence information of the scheduling module and the task arrangement information corresponding to the video analysis task; and performing feature analysis on the video stream based on the target calculation engine. The scheme of the embodiment can be suitable for different scenes and environments, the maximum utilization of environment resources is realized by determining the target calculation engine, and a series of problems of low video analysis capability, single task scheduling capability, low resource utilization rate and the like in an articulated naturality web analysis system are solved.

Description

Technical Field

[0001] The embodiments of the present disclosure relate to the field of visual networking technology, and in particular to a video analysis method, device, electronic device, storage medium, and program product. Background Art

[0002] Driven by the cross-domain integration of technologies such as 5G (fifth-generation mobile communication technology), AI (artificial intelligence), and IoT (Internet of Things), there is a growing demand for video as the main information transmission medium and functional carrier. Video traffic will become the main part of network traffic, and video digital applications will evolve from traditional video surveillance to a more ubiquitous, diversified, basic, and capable visual network. In the era of visual networking where everything is a screen, traditional video analysis solutions can no longer meet objective needs. The main problems are as follows:

[0003] First, traditional video surveillance systems can usually only analyze a single type of detection item or algorithm and cannot comprehensively and accurately analyze the video stream status and video content characteristics.

[0004] Second, the task scheduling capabilities are single and usually cannot adapt to different scenarios and environments.

[0005] Third, resource utilization is low and environmental resources cannot be maximized. Summary of the invention

[0006] Embodiments of the present disclosure provide a video analysis method, device, electronic device, storage medium, and program product.

[0007] In a first aspect, an embodiment of the present disclosure provides a video analysis method, which includes:

[0008] Dispatching video analysis tasks based on preset task scheduling;

[0009] Selecting a target computing engine from computing engines based on the aggregated information of the scheduling module and the task scheduling information corresponding to the video analysis task;

[0010] The video stream is subjected to feature analysis based on the target computing engine.

[0011] In a second aspect, the present disclosure also provides a video analysis device, which includes:

[0012] Task management module, used to dispatch video analysis tasks based on preset task scheduling;

[0013] A scheduling module, configured to select a target computing engine from computing engines based on the aggregated information of the scheduling module and the task scheduling information corresponding to the video analysis task;

[0014] A feature analysis module is used to perform feature analysis on the video stream based on the target computing engine.

[0015] In a third aspect, an embodiment of the present disclosure further provides an electronic device, comprising:

[0016] one or more processors;

[0017] A memory having one or more programs stored thereon, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the video analysis method;

[0018] One or more input / output I / O interfaces are connected between the processor and the memory and are configured to implement information interaction between the processor and the memory.

[0019] In a fourth aspect, an embodiment of the present disclosure further provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the video analysis method is implemented.

[0020] In a fifth aspect, an embodiment of the present disclosure further provides a computer program product, which includes a computer program, and the computer program implements the video analysis method when executed by a processor.

[0021] The disclosed embodiment scheme issues video analysis tasks based on preset task scheduling; selects a target computing engine from computing engines based on the converged information of the scheduling module and the task scheduling information corresponding to the video analysis task; and performs feature analysis on the video stream based on the target computing engine. The scheme can process multiple video analysis tasks at the same time through task scheduling, so as to adapt to different scenes and environments; and after issuing the analysis task, the target computing engine is determined by comprehensively considering the task information of the analysis task and the converged information of the scheduling module, so as to maximize the use of environmental resources and solve a series of problems in the visual network analysis system, such as weak video analysis capability, single task scheduling capability, and low resource utilization. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] In the accompanying drawings of the embodiments of the present disclosure:

[0023] Figure 1 A flow chart of a video analysis method provided by an embodiment of the present disclosure;

[0024] Figure 2 A schematic diagram of the structure of a video analysis system provided in an embodiment of the present disclosure;

[0025] Figure 3 A schematic diagram of detecting a video stream state provided by an embodiment of the present disclosure;

[0026] Figure 4A flow chart of a method for acquiring a target computing engine provided by an embodiment of the present disclosure;

[0027] Figure 5 A schematic diagram of a scheduling strategy provided by an embodiment of the present disclosure;

[0028] Figure 6 A flow chart of a method for performing feature analysis on a video stream based on a target computing engine provided in an embodiment of the present disclosure;

[0029] Figure 7 A schematic diagram of a video content feature detection method provided by an embodiment of the present disclosure;

[0030] Figure 8 A block diagram of the video analysis device provided in the embodiment of the present disclosure;

[0031] Fig. 9 A block diagram of the electronic device provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0032] In order to enable those skilled in the art to better understand the technical solution of the present disclosure, the communication perception data processing method and computer-readable storage medium provided by the embodiments of the present disclosure are described in detail below in conjunction with the accompanying drawings.

[0033] The present disclosure will be described more fully below with reference to the accompanying drawings, but the embodiments shown may be embodied in different forms, and the present disclosure should not be construed as being limited to the embodiments set forth below. On the contrary, the purpose of providing these embodiments is to make the present disclosure thorough and complete, and will enable those skilled in the art to fully understand the scope of the present disclosure.

[0034] The accompanying drawings of the embodiments of the present disclosure are used to provide a further understanding of the embodiments of the present disclosure and constitute a part of the specification. Together with the detailed embodiments, they are used to explain the present disclosure and do not constitute a limitation of the present disclosure. By describing the detailed embodiments with reference to the accompanying drawings, the above and other features and advantages will become more apparent to those skilled in the art.

[0035] The present disclosure may be described with reference to plan views and / or cross-sectional views by means of ideal schematic views of the present disclosure. Therefore, the exemplary illustrations may be modified according to manufacturing techniques and / or tolerances.

[0036] In the absence of conflict, the various embodiments of the present disclosure and the various features therein may be combined with each other.

[0037] The terms used in the present disclosure are only used to describe specific embodiments and are not intended to limit the present disclosure. The term "and / or" as used in the present disclosure includes any and all combinations of one or more related enumerated items. The singular forms "one" and "the" as used in the present disclosure are also intended to include plural forms, unless the context clearly indicates otherwise. The terms "including", "made of..." as used in the present disclosure specify the presence of the features, wholes, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or groups thereof.

[0038] Unless otherwise defined, all terms (including technical and scientific terms) used in this disclosure have the same meaning as commonly understood by those of ordinary skill in the art. It will also be understood that terms such as those defined in commonly used dictionaries should be interpreted as having a meaning consistent with their meaning in the context of the relevant art and this disclosure, and will not be interpreted as having an idealized or overly formal meaning unless explicitly defined in this disclosure.

[0039] Traditional video analysis solutions have the following problems:

[0040] First, traditional video surveillance systems can usually only analyze a single type of detection item or algorithm and cannot comprehensively and accurately analyze the video stream status and video content characteristics.

[0041] Second, the task scheduling capabilities are single and usually cannot adapt to different scenarios and environments.

[0042] Third, resource utilization is low and environmental resources cannot be maximized.

[0043] The disclosed embodiment scheme issues video analysis tasks based on preset task scheduling; selects a target computing engine from computing engines based on the converged information of the scheduling module and the task scheduling information corresponding to the video analysis task; and performs feature analysis on the video stream based on the target computing engine. This scheme can process multiple video analysis tasks at the same time through task scheduling, so that it can adapt to different scenes and environments; and after issuing the analysis task, it comprehensively considers the task information of the analysis task and the converged information of intelligent scheduling to determine the target computing engine, thereby maximizing the use of environmental resources and solving a series of problems in the visual network analysis system, such as weak video analysis capabilities, single task scheduling capabilities, and low resource utilization.

[0044] The embodiments of the present disclosure can be applied to any system with a video surveillance function, for example, including but not limited to related products of video Internet of Things (video surveillance), video middle platform, video cloud platform, machine vision and other systems.

[0045] The following is a detailed introduction to the embodiments of the present disclosure.

[0046] The present disclosure provides a video analysis method. Figure 1 As shown, steps S11-S13 are included:

[0047] S11. Dispatching video analysis tasks based on preset task scheduling.

[0048] In the embodiment of the present disclosure, the preset task scheduling refers to at least one type of video analysis task scheduled in advance based on preset task scheduling information according to different requirements or different application scenarios.

[0049] In the embodiments of the present disclosure, Figure 2 As shown, the overall structure of the video analysis system for video analysis can be divided into a video aggregation component, a video analysis component and a task management module according to logic.

[0050] In the disclosed embodiment, the task management module is used to schedule and issue video surveillance analysis tasks (which may be referred to as video analysis tasks). Among them, the video analysis tasks can be pre-scheduled to obtain a variety of task schedules, each of which can set task types, task execution events, corresponding camera channels and other information, so as to pull the video stream of the corresponding camera channel at the corresponding time according to this information to perform the video analysis task of this task type. When pulling the video stream, it is necessary to perform pre-detection on the video stream, which is mainly completed by the video aggregation component. After successfully pulling the video stream, it is necessary to perform video analysis on the video stream, which is mainly completed by the video analysis component.

[0051] In the disclosed embodiment, the scheme for arranging video analysis tasks by the video management module is first described in detail below.

[0052] In the embodiment of the present disclosure, before issuing the video analysis task based on the preset task scheduling, the method may further include:

[0053] Based on the task scheduling information, various types of video analysis tasks are scheduled to obtain various preset task schedules.

[0054] In the disclosed embodiment, the task scheduling information may include, but is not limited to, any one or more of the following: task execution times type, task execution time parameters, task analysis type, task algorithm parameters corresponding to the task analysis type, frame extraction method, and camera channel. Task algorithm parameters refer to execution parameters set for the detection package that executes the task analysis type in order to execute the corresponding task analysis type. Each task analysis type is executed based on the corresponding detection package in the computing engine according to the task algorithm parameters.

[0055] In the embodiment of the present disclosure, the types of task execution times may include, but are not limited to: single execution or periodic execution;

[0056] Task analysis types may include, but are not limited to, any one or more of the following: video quality diagnosis, person or object recognition, and smoke and fire detection.

[0057] In the disclosed embodiment, the task management module can support various types of video analysis tasks such as video stream status detection, video quality diagnosis (including video stream status), various AI algorithm analysis (including video stream status) through pre-set task scheduling information, support periodic and single task delivery, support setting task algorithm parameters, support setting frame extraction mode and other functions. A visualization page can be provided to configure the above-mentioned task execution number type, task execution time parameters, task analysis type, task algorithm parameters, frame extraction mode, camera channel and other task scheduling information.

[0058] In the embodiments of the present disclosure, required task scheduling information may be selected according to requirements or different application scenarios to perform task scheduling, thereby obtaining a variety of task scheduling that meets the requirements and application scenarios.

[0059] In the embodiments of the present disclosure, for example, Table 1, Table 2, and Table 3 provide three task scheduling embodiments, namely, task scheduling 1, task scheduling 2, and task scheduling 3.

[0060] Table 1 Task arrangement 1

[0061]

[0062]

[0063] Table 2 Task Scheduling 2

[0064]

[0065] Table 3 Task Scheduling 3

[0066]

[0067] In the embodiment of the present disclosure, after the video analysis task is issued based on the preset task arrangement, the method further includes:

[0068] Pull the video stream based on the video analysis task and perform status detection on the video stream;

[0069] In response to the state of the video stream being normal, a process of selecting a target computing engine from computing engines based on task scheduling information corresponding to the video analysis task and aggregated information of the scheduling module is entered.

[0070] In the disclosed embodiment, each video analysis task is established on the premise that the video stream state is normal. Therefore, when pulling the stream based on task scheduling, it is necessary to perform a pre-detection of the video stream state.

[0071] In the embodiment of the present disclosure, the status detection of the video stream may include but is not limited to detecting any one or more of the following information:

[0072] Whether the camera channel is online;

[0073] Whether the pull stream address is obtained successfully; and,

[0074] Whether the video stream is pulled successfully.

[0075] In the embodiment of the present disclosure, the method may further include: when it is detected that the camera channel is online, the stream pulling address is successfully obtained, and the video stream is successfully pulled, determining that there is no abnormality in the status of the video stream.

[0076] In the disclosed embodiment, in order to realize the video monitoring analysis task, the camera channel needs to be online normally, the streaming address can be obtained, and the video stream can be pulled normally. If the video stream status is abnormal, for example, the camera channel is offline, the streaming address fails to be obtained, the video stream fails to be pulled, etc., the video stream cannot be pulled, decoded, framed, etc., and the video content features cannot be detected.

[0077] In the disclosed embodiment, before the video analysis phase, that is, before the detection of the content features of the video stream, a sufficiently reliable video stream status detection can be performed in the video aggregation component.

[0078] In the embodiments of the present disclosure, Figure 2 As shown, the video convergence components may include but are not limited to: pre-detection package, media management, streaming media and other modules.

[0079] In the disclosed embodiment, the pre-detection package is responsible for detecting and analyzing the video stream status. The video stream status detection algorithm can be made into a pre-detection package and integrated into the video aggregation component to perform fully reliable pre-detection on each video analysis task so that normal scheduling and analysis can be performed later. The pre-detection package can detect and analyze the video stream itself by interacting with modules such as media management and streaming media.

[0080] In the disclosed embodiment, media management is responsible for detecting and managing the status of cameras and streaming media. Media management records the online and offline status of camera channels by interacting with camera devices; and records streaming address information by interacting with streaming media.

[0081] In the disclosed embodiment, the streaming media is responsible for pulling the video stream of the camera device and forwarding the video stream to the computing engine in the video analysis component for analysis and processing.

[0082] In the disclosed embodiment, based on the video aggregation component, the video stream state detection capability can be integrated into the video aggregation link as a pre-detection function. The pre-detection package interacts with the media management and streaming media modules to detect anomalies in the video stream itself. The detection algorithms included in the pre-detection package may include but are not limited to camera channel detection, acquisition of pull stream address detection, pull stream detection and other algorithms.

[0083] In the embodiment of the present disclosure, the detection schematic diagram of the video stream state can be as follows: Figure 3 shown.

[0084] In the embodiments of the present disclosure, the camera channel detection, the acquisition of the streaming address detection, and the streaming detection schemes are respectively introduced in detail below.

[0085] In the disclosed embodiment, the camera channel detection may be performed first. The pre-detection package triggers the camera channel detection algorithm, and obtains the online and offline status data of the camera channel from the media management. The media management may directly interact with the camera device to record whether the camera channel is online or offline, and synchronize the online or offline status data of the camera channel to the pre-detection package. Based on the online or offline status data of the camera channel, if it is determined that the camera channel is online, the next step is performed, for example, obtaining the pull stream address detection; if it is determined that the camera channel is offline, an alarm may be issued, and the next step will not be performed.

[0086] In the embodiment of the present disclosure, after detecting that the camera channel is online, the step of obtaining the streaming address detection can be entered. The pre-detection package can trigger the detection algorithm for obtaining the streaming address, and obtain the streaming address from the media management. If the streaming address is successfully obtained, the next step is performed, for example, streaming detection; if the streaming address fails to be obtained (for example, it may include but is not limited to not finding an available streaming media, the protocol between the streaming media and the camera has not established a connection, etc.), an alarm is issued, and the next step will not be performed. The media management records the streaming address information by directly interacting with the streaming media, and synchronizes the streaming address information to the pre-detection package.

[0087] In the disclosed embodiment, after detecting that the camera channel is online and the streaming address is successfully obtained, streaming detection can be performed. The pre-detection package can trigger the detection algorithm for obtaining streaming, and briefly pull the video stream to the streaming media; if the streaming is successful, the next step is performed, for example, determining that there is no abnormality in the video stream status and issuing a video analysis task; if the streaming fails (for example, it may include but is not limited to parsing failure, playback failure, etc.), an alarm is issued and the next step will not be performed.

[0088] S12: Select a target computing engine from computing engines based on the aggregated information of the scheduling module and the task scheduling information corresponding to the video analysis task.

[0089] In the disclosed embodiment, after the aforementioned steps, when the task management module determines that there is no abnormality in the status of the video stream, the video monitoring analysis task enters the video analysis stage and can be completed by the video analysis component.

[0090] In the embodiments of the present disclosure, Figure 2 As shown, the video analysis components may include but are not limited to: intelligent scheduling (i.e., scheduling module, or AI scheduling), computing engine and other modules.

[0091] In the disclosed embodiment, the aggregated information may include but is not limited to: basic information of the computing engine actively collected by the scheduling module and task information of the completed video analysis task actively reported to the scheduling module by the computing engine after completing the video analysis task.

[0092] In the embodiments of the present disclosure, intelligent scheduling can communicate with computing engines (for example, computing engine 1...computing engine n, where n is a positive integer; each computing engine can include multiple detection packages for processing different types of analysis tasks, for example, detection package 1...detection package n, etc.) on a regular basis (the timing period can be in seconds), and obtain in real time the basic information of the computing engines and the task information of the completed video analysis tasks reported by the computing engines, so as to select the target computing engine from the computing engines based on the basic information, the reported task information of the completed video analysis tasks, and the task scheduling information corresponding to the video analysis task to complete the video analysis task.

[0093] In the embodiments of the present disclosure, the basic information may include, but is not limited to, any one or more of the following: available detection packages (one detection package is used to analyze one type of video analysis task), remaining resource load, and information of the currently executed video analysis task;

[0094] The task information of the completed video analysis task may include, but is not limited to: the task analysis result of the completed video analysis task.

[0095] In the disclosed embodiment, after each video analysis task is executed, the computing engine executing the video analysis task will immediately report the analysis result of the video analysis task to the intelligent scheduling. The analysis result may include information about the video analysis task that has been executed by the computing engine. Therefore, based on the reported analysis result, the intelligent scheduling can obtain information about the video analysis task that has been executed in the computing engine and update the basic information corresponding to the computing engine.

[0096] In the embodiments of the present disclosure, for example, Table 4 and Table 5 respectively give examples of basic information of the computing engine 1 and analysis results of the executed video analysis tasks.

[0097] Table 4 Basic information of computing engine

[0098]

[0099]

[0100] Table 5 Information table of analysis results

[0101]

[0102] In the embodiment of the present disclosure, the method may further include:

[0103] After each computing engine completes executing any video analysis task, the aggregation information of the scheduling module is updated.

[0104] In the embodiment of the present disclosure, before performing feature analysis on the video stream based on the target computing engine, the method may further include:

[0105] Configure the resource consumption data of a single run of multiple algorithms;

[0106] A resource consumption table is generated based on the single-run resource consumption data of multiple algorithms, and the resource consumption table is used to determine the remaining resource load of each computing engine and the resource value required to be consumed by the video analysis task.

[0107] In an embodiment of the present disclosure, during the system initialization phase, it is possible to support the configuration of single-run resource consumption data of various algorithms (for example, including but not limited to detection algorithms, task analysis algorithms, etc.), and these data will be aggregated into a resource consumption table and preset into the intelligent scheduling.

[0108] In the embodiments of the present disclosure, the single-run resource consumption data of various algorithms can be obtained through stress testing and the like. For example, only one algorithm is run under limited resources, and the maximum number of paths that can be run is recorded. The total resource data is divided by the number of paths to obtain the single-run resource consumption.

[0109] In the embodiments of the present disclosure, when configuring the single-run resource consumption data of various algorithms, in addition to configuring through a page, it can also be implemented by calling an interface, and it can also be implemented by directly executing SQL (Structured Query Language) statements to operate the database.

[0110] In the disclosed embodiment, since the resource consumption table records the resource consumption values ​​of each algorithm (for example, including but not limited to CPU consumption value, GPU consumption value, memory consumption value, etc.), when any video analysis task is generated, the detection package required (i.e., the required algorithm) will be recorded in the video analysis task, so that the resource consumption value of the corresponding algorithm can be found according to the resource consumption table to determine the resource value required to be consumed by the video analysis task. Similarly, when any video analysis task is generated, one or more video analysis tasks may be executed in the computing engine at this time, and the video analysis tasks being executed will also consume certain resource values ​​(i.e., have certain consumed resource values). Therefore, according to the resource load initially configured by each computing engine (i.e., the configured resource value), and the consumed resource value of the video analysis task being executed, the remaining resource load of each current computing engine can be obtained by difference calculation, so that the target computing engine can be selected based on the remaining resource load to achieve reasonable scheduling of the video analysis task.

[0111] In the embodiment of the present disclosure, selecting a target computing engine from computing engines based on the aggregated information of the scheduling module and the task scheduling information corresponding to the video analysis task may include:

[0112] The target computing engine is selected from the computing engines based on the task analysis type and task algorithm parameters of the video analysis task, as well as the remaining resource load of each computing engine, the available detection packages, the information of the currently executed video analysis task and the task analysis results of the completed video analysis task.

[0113] In the embodiments of the present disclosure, Figure 4 As shown, based on the task analysis type and task algorithm parameters of the video analysis task, as well as the remaining resource load of each computing engine, the detection packages available, the information of the currently executed video analysis task, and the task analysis results of the completed video analysis task, selecting a target computing engine from the computing engines may include steps S21-S24:

[0114] S21. Based on the task analysis type of the video analysis task and the detection packages of each computing engine, a first group of computing engines whose detection packages include detection packages corresponding to the task analysis type are selected from the computing engines.

[0115] S22. Determine the resource value required to be consumed by the video analysis task based on the task analysis type and task algorithm parameters corresponding to the video analysis task.

[0116] S23. Based on the remaining resource load corresponding to each computing engine in the first group of computing engines and the resource value required to be consumed by the analysis task, the first group of computing engines is screened for the second time to obtain a second group of computing engines whose remaining resource load is greater than or equal to the resource value required to be consumed by the video analysis task.

[0117] S24. Based on the information of the video analysis task currently being executed by each computing engine in the second group of computing engines and the task analysis results of the completed video analysis tasks, predict the computing engine in the second group of computing engines with the shortest execution time required to execute the issued video analysis task as the target computing engine.

[0118] In the embodiments of the present disclosure, the execution order between the schemes in the above steps S21, S22, S23 and S24 can be defined according to needs, and is not limited to being executed only in the order of S21, S22, S23 and S24. Any executable order is within the protection scope of the embodiments of the present disclosure.

[0119] In the embodiment of the present disclosure, based on the information of the video analysis task currently being executed by each computing engine in the second group of computing engines and the task analysis results of the completed video analysis tasks, predicting the computing engine in the second group of computing engines with the shortest execution time required to execute the issued video analysis task as the target computing engine may include:

[0120] Based on the task analysis results of the completed video analysis tasks corresponding to each computing engine in the second group of computing engines, determine the execution time required for each computing engine to perform tasks of the same task analysis type as the video analysis task multiple times within a preset time period before the current moment;

[0121] Based on the execution time required for each computing engine, the information of all video analysis tasks currently being executed, and the pre-trained AI model, the computing engine in the second group of computing engines with the shortest execution time required to execute the issued video analysis task is predicted as the target computing engine.

[0122] In the disclosed embodiment, intelligent scheduling utilizes AI scheduling strategies based on task scheduling information, combined with task analysis algorithms (i.e., available detection packages), computing engine remaining resource load, task execution status and other data to maximize resource utilization, adapt to different scenarios and environments, achieve efficient task collaboration, and facilitate the fastest processing of video analysis tasks.

[0123] In the embodiment of the present disclosure, intelligent scheduling identifies the task analysis type (or detection package type) of the task arrangement corresponding to the video analysis task to be processed, and combines the detection packages provided in the basic information of the computing engine to screen out the computing engines (i.e., the first group of computing engines) that have detection packages capable of processing the video analysis task to be processed.

[0124] In the disclosed embodiment, intelligent scheduling identifies the task analysis type and task algorithm parameters of the task scheduling, and in combination with the above-mentioned resource consumption table, obtains the resource value required for the video analysis task, for example, including but not limited to: CPU (central processing unit) consumption value, GPU (graphics processing unit) consumption value, memory consumption value, etc., and then in combination with the remaining resource load of the computing engine (the first group of computing engines), screens out the computing engines (i.e., the second group of computing engines) whose remaining resource load can handle the video analysis task to be processed.

[0125] In the disclosed embodiment, intelligent scheduling can obtain the execution time required for the algorithm to process this type of video analysis task several times in the recent period (e.g., within a preset time period before the current moment) based on the task analysis result information of the video analysis task that has been executed by the screened computing engine (the second group of computing engines), and then combine the execution status of other video analysis tasks currently being executed by the computing engine to predict the execution time of the video analysis task to be executed in each computing engine in the second group of computing engines through a preset AI model, take the computing engine with the shortest execution time as the target computing engine, and schedule the video analysis task to the target computing engine for processing.

[0126] In the disclosed embodiment, the AI ​​model may be a pre-trained intelligent model with the ability to predict execution time.

[0127] In the embodiment of the present disclosure, a detailed embodiment of determining the target computing engine is given below based on the above scheme. Figure 5As shown in Table 1, if the task arrangement is as shown in Table 1, the intelligent scheduling can first determine the computing engine capable of executing the video quality diagnosis algorithm from many computing engines. If it is known through detection that computing engine 1 contains detection package 1 (for video quality diagnosis VQD detection), detection package 2 (for face detection), detection package 3 (for vehicle detection), ..., detection package n (for flame detection), computing engine n contains detection package 1 (for video quality diagnosis VQD detection), detection package 2 (for face detection), detection package 3 (for smoke detection), ..., detection package n (for helmet detection), then the computing engines that meet the requirements are computing engine 1 and computing engine n, because computing engine 1 and computing engine n are All computing engines n contain detection package 1 (for video quality diagnosis VQD detection); then, according to the remaining resource load of computing engine 1 and computing engine n, the computing engine that can handle the current video analysis task to be processed can be screened out. If it can be seen from the screening results that computing engine 1 and computing engine n both meet the requirements; finally, according to the execution status of each algorithm in computing engine 1 and computing engine n and the duration of the recent execution of the video quality diagnosis algorithm, the AI ​​model can be used to predict that the computing engine with the shortest execution time of executing the video quality diagnosis algorithm is computing engine 1. Therefore, computing engine 1 can be determined as the target computing engine, and the video analysis task can be scheduled to computing engine 1 for analysis.

[0128] In the disclosed embodiment, during the operation of the system, the intelligent scheduling will also combine the initialized resource consumption table and the real-time data of the computing engine (such as the execution duration of the task, the task execution status, the remaining resource load, etc.) to dynamically adjust the resource consumption value corresponding to each algorithm and improve the resource consumption table of the algorithm.

[0129] S13. Perform feature analysis on the video stream based on the target computing engine.

[0130] In the embodiment of the present disclosure, the video stream is a video stream pulled according to a video analysis task.

[0131] In the embodiments of the present disclosure, Figure 6 As shown, performing feature analysis on the video stream based on the target computing engine may include steps S31-S34:

[0132] S31. Decode the video stream.

[0133] S32: Extract frames from the decoded video stream based on the frame extraction method in the task scheduling information corresponding to the video stream.

[0134] S33, sending the data obtained by extracting frames to the detection package that matches the task analysis type corresponding to the video stream.

[0135] S34. The detection package performs feature analysis on the data based on the task algorithm parameters corresponding to the task analysis type.

[0136] In the disclosed embodiment, after intelligent scheduling, the algorithm of the video analysis task will be executed in the target computing engine.

[0137] In the disclosed embodiment, before analyzing the video stream, stream processing can be performed first. According to the intelligent scheduling arrangement, the stream processing of the target computing engine will obtain the real-time video stream of the camera channel from the streaming media, decode the video stream, extract frames of the video stream according to the frame extraction method corresponding to the task scheduling information, and send the extracted frame data to the detection package corresponding to the analysis task type in the task scheduling information. The detection package may include but is not limited to CPU algorithms (video quality diagnosis), GPU algorithms (video AI analysis), etc.

[0138] In the embodiments of the present disclosure, Figure 7 As shown, for example, if the task scheduling is as shown in Table 1, the video analysis task is scheduled to computing engine 1 (target computing engine) according to the intelligent scheduling; and according to the streaming media function in the overall structure diagram, the video stream of the camera channel can be obtained through the streaming media. Therefore, the stream processing can obtain the video stream of channel 1 of the camera from the streaming media according to the task scheduling information in Table 1, and convert the video stream into multiple pictures according to the preset frame extraction strategy of "extracting 5 frames per second, a total of 30 frames", and pass them together with the detection package parameters (i.e., the task algorithm parameters corresponding to the task analysis type of video quality diagnosis) to the video quality diagnosis (VQD) detection package.

[0139] In the disclosed embodiment, the following algorithm can be executed by the detection package. After receiving the frame extraction data, the detection package can use the frame extraction data and the task algorithm parameters of the video analysis task (i.e., the detection package parameters) as algorithm inputs, and the input algorithm generates diagnosis and analysis results after reasoning based on the above frame extraction data, and the target computing engine reports the algorithm diagnosis and analysis results as the results of the video analysis task to the intelligent scheduling.

[0140] In the disclosed embodiment, for example, according to the video quality diagnosis detection package in computing engine 1 (target computing engine), after receiving the picture information and detection package parameter information after the frame data is extracted, the video quality diagnosis algorithm is executed to generate an inference result. Computing engine 1 uploads the inference result to intelligent scheduling, as one of the big data sources of intelligent scheduling, refer to the aforementioned task information table (as shown in Table 4 and Table 5). Intelligent scheduling can also perform AI task scheduling based on the latest basic data of computing engine 1.

[0141] In the embodiments of the present disclosure, at least the following advantages are included:

[0142] 1. The embodiment scheme of the present disclosure is based on the visual network analysis technology and system integrating multiple types of detection, which solves a series of problems in the visual network analysis system, such as weak video analysis capability, single task scheduling capability, and low resource utilization.

[0143] 2. By integrating different types of video detection and analysis technologies such as video stream status inspection, video quality diagnosis, and video AI analysis into the system, a deep integration of video aggregation and video analysis capabilities is achieved, which can accurately and comprehensively analyze the video stream status and video stream content characteristics.

[0144] 3. Through intelligent task scheduling technology, arrange scheduling strategies, perceive resource changes, maximize resource utilization, adapt to different scenarios and environments, and achieve efficient task collaboration.

[0145] The disclosed embodiment also provides a video analysis device 100, such as Figure 8 As shown, it includes:

[0146] The task management module 101 is used to dispatch video analysis tasks based on preset task scheduling;

[0147] A scheduling module 102, configured to select a target computing engine from computing engines based on the aggregated information of the scheduling module and the task scheduling information corresponding to the video analysis task;

[0148] The feature analysis module 103 is used to perform feature analysis on the video stream based on the target computing engine.

[0149] In the disclosed embodiment, the task management module 101 may be implemented by the aforementioned video management, the scheduling module 102 may be implemented by the aforementioned intelligent scheduling, and the feature analysis module 103 may be implemented by the aforementioned computing engine.

[0150] In the embodiment of the present disclosure, a pre-detection module may also be included, which is used to pull the video stream based on the video analysis task and detect the state of the video stream. The pre-detection module may be implemented by the pre-detection package in the aforementioned video convergence component.

[0151] In the embodiments of the present disclosure, any embodiment of the aforementioned video analysis method is applicable to the embodiment of the video analysis device 100 and will not be described in detail here.

[0152] The present disclosure also provides an electronic device 200, such as Fig. 9 As shown, including:

[0153] One or more processors 201;

[0154] A memory 202 having one or more programs stored thereon, and when the one or more programs are executed by the one or more processors 201, the one or more processors implement the video analysis method;

[0155] One or more input / output I / O interfaces 203 are connected between the processor 201 and the memory 202 and are configured to implement information interaction between the processor 201 and the memory 202 .

[0156] The embodiment of the present disclosure further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the video analysis method is implemented.

[0157] The embodiment of the present disclosure also provides a computer program product, which includes a computer program, and when the computer program is executed by a processor, the video analysis method is implemented.

[0158] In the embodiments of the present disclosure, any embodiment of the aforementioned video analysis method is applicable to the electronic device 200, storage medium, and program product embodiments, and will not be described one by one here.

[0159] Those skilled in the art will appreciate that all or some of the functional modules / units disclosed above may be implemented as software, firmware, hardware, or a suitable combination thereof.

[0160] In hardware implementations, the division between functional modules / units mentioned in the above description does not necessarily correspond to the division of physical components; for example, one physical component may have multiple functions, or one function or step may be performed by several physical components in cooperation.

[0161] Some physical components or all physical components may be implemented as software executed by a processor, such as a central processing unit (CPU), a digital signal processor or a microprocessor, or implemented as hardware, or implemented as an integrated circuit, such as an application-specific integrated circuit. Such software may be distributed on a computer-readable medium, which may include a computer storage medium (or non-temporary medium) and a communication medium (or temporary medium). As known to those of ordinary skill in the art, the term computer storage medium includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules or other data). Computer storage media include, but are not limited to, random access memory (RAM, more specifically SDRAM, DDR, etc.), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory (FLASH) or other disk storage; read-only compact disk (CD-ROM), digital versatile disk (DVD) or other optical disk storage; magnetic cassettes, magnetic tapes, disk storage or other magnetic storage; any other medium that can be used to store desired information and can be accessed by a computer. Furthermore, it is well known to those skilled in the art that communication media typically embodies computer readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave or other transport mechanism, and may include any information delivery media.

[0162] The present disclosure has disclosed example embodiments, and although specific terms are employed, they are used and should be interpreted only in a general illustrative sense and not for limiting purposes. In some instances, it will be apparent to those skilled in the art that, unless otherwise expressly stated, features, characteristics, and / or elements described in conjunction with a particular embodiment may be used alone or in combination with features, characteristics, and / or elements described in conjunction with other embodiments. Therefore, those skilled in the art will appreciate that various changes in form and detail may be made without departing from the scope of the present disclosure as set forth in the appended claims.

Claims

1. A video analysis method, comprising: Dispatching video analysis tasks based on preset task scheduling; Selecting a target computing engine from computing engines based on the aggregated information of the scheduling module and the task scheduling information corresponding to the video analysis task; The video stream is subjected to feature analysis based on the target computing engine.

2. The video analysis method according to claim 1, wherein: After the video analysis task is dispatched based on the preset task arrangement, the method further includes: Pulling a video stream based on the video analysis task, and performing status detection on the video stream; In response to the state of the video stream being normal, entering the process of selecting a target computing engine from computing engines based on the task scheduling information corresponding to the video analysis task and the converged information of the scheduling module.

3. The video analysis method according to claim 2, wherein: The detecting the state of the video stream includes detecting any one or more of the following information: Whether the camera channel is online; Whether the pull stream address is obtained successfully; and, Whether the video stream is pulled successfully.

4. The video analysis method according to claim 1, wherein: The method further comprises: Based on the task scheduling information, multiple types of video analysis tasks are scheduled to obtain multiple preset task schedules.

5. The video analysis method according to claim 1 or 4, wherein: The task scheduling information includes any one or more of the following: Task execution times type, task execution time parameters, task analysis type, task algorithm parameters corresponding to the task analysis type, frame extraction method, and camera channel.

6. The video analysis method according to claim 1, wherein: The aggregation information includes: The scheduling module actively collects basic information about the computing engine and the task information about the completed video analysis task that the computing engine actively reports to the scheduling module after completing the video analysis task.

7. The video analysis method according to claim 6, wherein: The basic information of the computing engine includes any one or more of the following: available detection packages, remaining resource load, and information of the currently executed video analysis task; The task information of the completed video analysis task includes: the task analysis result of the completed video analysis task.

8. The video analysis method according to claim 7, wherein: The task scheduling information corresponding to the video analysis task includes the task analysis type and task algorithm parameters corresponding to the video analysis task; the task analysis type is executed based on the corresponding detection package in the computing engine according to the task algorithm parameters; The selecting a target computing engine from computing engines based on the aggregated information of the scheduling module and the task scheduling information corresponding to the video analysis task includes: The target computing engine is selected from the computing engines based on the task analysis type and task algorithm parameters of the video analysis task, as well as the remaining resource load of each computing engine, the available detection packages, the information of the currently executing video analysis task and the task analysis results of the completed video analysis task.

9. The video analysis method according to claim 8, wherein: The selecting the target computing engine from the computing engines based on the task analysis type and task algorithm parameters of the video analysis task, the remaining resource load of each computing engine, the available detection packages, the information of the currently executed video analysis task, and the task analysis results of the completed video analysis task, comprises: Based on the task analysis type of the video analysis task and the detection packages possessed by each computing engine, a first group of computing engines including detection packages corresponding to the task analysis type are screened out from the computing engines; Determining a resource value required to be consumed by the video analysis task based on the task analysis type and the task algorithm parameters corresponding to the video analysis task; Based on the remaining resource load corresponding to each computing engine in the first group of computing engines and the resource value required to be consumed by the video analysis task, the first group of computing engines is screened for the second time to obtain a second group of computing engines whose remaining resource load is greater than or equal to the resource value required to be consumed by the video analysis task; Based on the information of the video analysis task currently being executed by each computing engine in the second group of computing engines and the task analysis results of the completed video analysis task, the computing engine in the second group of computing engines with the shortest execution time required to execute the issued video analysis task is predicted as the target computing engine.

10. The video analysis method according to claim 9, wherein: The method of predicting, based on the information of the video analysis task currently being executed by each computing engine in the second group of computing engines and the task analysis result of the completed video analysis task, the computing engine in the second group of computing engines having the shortest execution time required to execute the issued video analysis task as the target computing engine, includes: Based on the task analysis result of the completed video analysis task corresponding to each computing engine in the second group of computing engines, determine the execution time required for each computing engine to perform tasks of the same task analysis type as the video analysis task multiple times within a preset time period before a current moment; Based on the execution time required for each time determined for each computing engine, information about all video analysis tasks currently being executed, and a pre-trained AI model, the computing engine in the second group of computing engines with the shortest execution time required to execute the issued video analysis task is predicted as the target computing engine.

11. The video analysis method according to claim 1, wherein: The performing feature analysis on the video stream based on the target computing engine includes: Decoding the video stream; Extract frames from the decoded video stream based on the frame extraction method in the task scheduling information corresponding to the video stream; Sending the data obtained by extracting frames to a detection package matching the task analysis type corresponding to the video stream; The detection package performs feature analysis on the data based on the task algorithm parameters corresponding to the task analysis type.

12. The video analysis method according to claim 1, wherein: The method further comprises: After each computing engine completes executing any video analysis task, the aggregation information of the scheduling module is updated.

13. The video analysis method according to claim 1, wherein: The method further comprises: Configure the resource consumption data of a single run of multiple algorithms; A resource consumption table is generated based on the single-run resource consumption data of the multiple algorithms, and the resource consumption table is used to determine the remaining resource load of each computing engine and the resource value required to be consumed by the video analysis task.

14. A video analysis device, comprising: Task management module, used to dispatch video analysis tasks based on preset task scheduling; A scheduling module, configured to select a target computing engine from computing engines based on the aggregated information of the scheduling module and the task scheduling information corresponding to the video analysis task; A feature analysis module is used to perform feature analysis on the video stream based on the target computing engine.

15. An electronic device, comprising: one or more processors; A memory having one or more programs stored thereon, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the video analysis method according to any one of claims 1 to 13; One or more input / output I / O interfaces are connected between the processor and the memory and are configured to implement information interaction between the processor and the memory.

16. A computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the video analysis method according to any one of claims 1 to 13.

17. A computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the video analysis method according to any one of claims 1 to 13 is implemented.

Images