An intelligent analysis method, system and device based on video monitoring

Abstract

The application discloses an intelligent analysis method, system and device based on video monitoring. The method comprises the following steps: acquiring a device list corresponding to each analysis task in multiple types of analysis tasks, and generating a preselected queue corresponding to each analysis task based on the device list corresponding to each analysis task, wherein the preselected queue comprises one or more devices to be analyzed; acquiring the concurrent processing capacity of each analysis server cluster in a multiple-analysis server cluster, and generating an analysis queue to be analyzed based on the concurrent processing capacity of each analysis server cluster and the preselected queue corresponding to each analysis task; and performing video monitoring analysis on each device to be analyzed in the analysis queue.

Description

Technical Field

[0001] This application relates to the field of business analysis technology, and in particular to a method, system and device for determining intelligent analysis methods based on video surveillance. Background Technology

[0002] With the development of internet and video technologies, and the continuous improvement of people's living standards, the application scenarios of the video surveillance industry have gradually diversified, increasingly being used in small and medium-sized enterprises and home security. Furthermore, as artificial intelligence (AI) technology gradually permeates video front-end and back-end equipment, video surveillance no longer presents users with simply large amounts of video data. Instead, it can achieve intelligent analysis functions such as video quality diagnosis, fire and smoke detection, safety helmet detection, facial recognition, human tracking, intrusion alarms, and traffic management, and feed the analysis results back to the user.

[0003] Different application scenarios require different analysis tasks to be configured for video surveillance equipment. Each analysis task needs to pull the live stream, extract frames, and then analyze the extracted images. For video surveillance equipment configured with multiple analysis tasks, there may be situations where multiple analysis tasks pull multiple live streams at the same time, which poses challenges to the interface performance, network resources, and frame extraction computing resources of the video surveillance system. Summary of the Invention

[0004] This application provides an intelligent analysis method, apparatus, device, chip, and computer-readable storage medium based on video surveillance.

[0005] The intelligent analysis method based on video surveillance provided in this application includes:

[0006] Obtain a list of devices corresponding to each type of analysis task in the multi-type analysis task, and generate a pre-selection queue corresponding to each type of analysis task based on the list of devices corresponding to each type of analysis task, wherein the pre-selection queue includes one or more devices to be analyzed;

[0007] Obtain the concurrent processing capability of each type of analysis server cluster in the multi-type analysis server cluster, and generate a queue to be analyzed based on the concurrent processing capability of each type of analysis server cluster and the pre-selected queue corresponding to each type of analysis task.

[0008] Video surveillance analysis is performed on each device in the queue to be analyzed.

[0009] The intelligent analysis system based on video surveillance provided in this application includes: a task scheduling module, a resource management module, a video frame extraction module, and an intelligent analysis module; wherein,

[0010] The task scheduling module is configured to obtain a device list corresponding to each type of analysis task in a multi-type analysis task, and generate a pre-selected queue corresponding to each type of analysis task based on the device list, wherein the pre-selected queue includes one or more devices to be analyzed; and to call the resource management module to obtain the concurrent processing capacity of each type of analysis server cluster in the multi-type analysis server cluster; and generate a queue to be analyzed based on the concurrent processing capacity of each type of analysis server cluster and the pre-selected queue corresponding to each type of analysis task.

[0011] The resource management module is used to accept the call from the task scheduling module and obtain the concurrent processing capacity of each type of analysis server cluster in the multi-type analysis server cluster.

[0012] The video frame extraction module is used to perform frame extraction operations on the video of each device to be analyzed in the queue to be analyzed.

[0013] The intelligent analysis module is used to perform video monitoring analysis on the frame-extracted images corresponding to each device in the analysis queue.

[0014] The electronic device provided in this application includes a processor and a memory. The memory is used to store computer programs, and the processor is used to call and run the computer programs stored in the memory to execute any of the above-described intelligent analysis methods based on video surveillance.

[0015] The chip provided in this application includes a processor for calling and running a computer program from a memory, causing a device on which the chip is installed to perform any of the methods described above.

[0016] The computer-readable storage medium provided in this application embodiment is used to store a computer program that causes a computer to execute any of the methods described above.

[0017] In the technical solution of this application embodiment, on the one hand, a device list corresponding to each type of analysis task in multiple analysis tasks is obtained, and a pre-selected queue corresponding to each type of analysis task is generated based on the device list corresponding to each type of analysis task. In this way, using multiple pre-selected queues can reduce database operations and improve the polling efficiency of devices. On the other hand, the concurrent processing capacity of each type of analysis server cluster in multiple analysis server clusters is obtained, and a queue to be analyzed is generated based on the concurrent processing capacity of each type of analysis server cluster and the pre-selected queue corresponding to each type of analysis task. In this way, a unified queue to be analyzed is established for all analysis tasks, which facilitates unified management of analysis tasks and system deployment and maintenance. Furthermore, video monitoring analysis is performed on each device to be analyzed in the queue to be analyzed through a shared video frame extraction module and intelligent analysis module. In this way, the access pressure on the existing monitoring system interface can be reduced, and the network and computing resource consumption can be reduced. Attached Figure Description

[0018] Figure 1 This is a schematic diagram of the structural composition of the intelligent analysis system based on video surveillance provided in the embodiments of this application. Figure 1 ;

[0019] Figure 2 This is a flowchart illustrating the intelligent analysis method based on video surveillance provided in the embodiments of this application. Figure 1 ;

[0020] Figure 3 This is a flowchart illustrating the intelligent analysis method based on video surveillance provided in the embodiments of this application. Figure 2 ;

[0021] Figure 4 This is a schematic diagram of the structural composition of the intelligent analysis system based on video surveillance provided in the embodiments of this application. Figure 2 ;

[0022] Figure 5 This is a schematic structural diagram of an electronic device provided in an embodiment of this application;

[0023] Figure 6 This is a schematic structural diagram of the chip according to an embodiment of this application. Detailed Implementation

[0024] The technical solutions of the embodiments of this application will now be described with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are within the scope of protection of this application.

[0025] It should be noted that, in the embodiments of this application, the term "and / or" is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. Additionally, in the embodiments of this application, the character " / " generally indicates that the preceding and following related objects have an "or" relationship.

[0026] It should be understood that the term "instruction" mentioned in the embodiments of this application can be a direct instruction, an indirect instruction, or an indication of a relationship. For example, A instructing B can mean that A directly instructs B, such as B being able to obtain information through A; it can also mean that A indirectly instructs B, such as A instructing C, so B can obtain information through C; or it can mean that there is a relationship between A and B.

[0027] In the description of the embodiments of this application, the term "correspondence" may indicate that there is a direct or indirect correspondence between two things, or that there is an association between two things, or that there is a relationship of instruction and being instructed, configuration and being configured, etc.

[0028] To facilitate understanding of the technical solutions of the embodiments of this application, the relevant technologies of the embodiments of this application are described below. The following relevant technologies are optional solutions and can be combined with the technical solutions of the embodiments of this application in any way, and they all fall within the protection scope of the embodiments of this application.

[0029] Different application scenarios require different analysis tasks to be configured for video surveillance equipment. Each analysis task needs to pull the live stream, extract frames, and then analyze the extracted images. For video surveillance equipment configured with multiple analysis tasks, there may be situations where multiple analysis tasks pull multiple live streams at the same time, which poses challenges to the interface performance, network resources, and frame extraction computing resources of the video surveillance system.

[0030] It should be noted that the "video surveillance equipment" in this application embodiment can also be simply referred to as "monitoring equipment," "detection equipment," or "equipment." For example, a video surveillance equipment is a camera.

[0031] In one implementation scheme, different types of intelligent analysis (i.e., intelligent analysis corresponding to different analysis tasks) can be performed on common video frame sequences of the same device according to certain rules. For example, a multi-target management feature matrix can be constructed based on the actual management needs of multi-target cameras in key areas. This allows for the scientific invocation of multi-target intelligent analysis algorithms and the rational allocation and use of the image computing server's computing resources, enabling multi-target visualization management to achieve the highest real-time performance while meeting practical application requirements. In this scheme, the image computing server caches the video streams from various video surveillance devices and generates a video frame queue. Then, it invokes different intelligent analysis algorithms for analysis based on the values ​​of the multi-target management feature vectors. This approach deeply binds the monitoring devices to the image computing server and is only suitable for smaller-scale monitoring scenarios. It is not suitable for polling intelligent analysis in large-scale monitoring systems that require large-scale concurrent processing.

[0032] Therefore, the following technical solutions are proposed in the embodiments of this application. The technical solutions in the embodiments of this application can realize the management of intelligent analysis server clusters (also referred to as analysis server clusters) of monitoring systems with a capacity of 100,000 or more, adapt to the polling requirements of different types of intelligent analysis, and quickly respond to new intelligent analysis requests from users. For the management of analysis server clusters in monitoring systems with over 100,000 users, a Kafka messaging mechanism is used to group intelligent analysis messages, ensuring that the same intelligent analysis message can only be consumed and processed by one analysis server cluster, facilitating dynamic system expansion. For different types of intelligent analysis needs, a pre-selection queue is established for each type. When the task scheduling module starts, it first accesses the resource management module to obtain the concurrent processing capacity of each type of analysis server cluster, and reads the database to obtain the intelligent analysis needs (i.e., the analysis tasks configured for each device) corresponding to each device. For devices with multiple intelligent analysis needs, they are placed in the pre-selection queue with the shortest polling interval, achieving the goal of multiple intelligent analyses sharing a single video stream frame extraction process, thereby saving network and computing resources. At specific intervals, a specific number of devices are selected from each pre-selection queue according to the concurrent processing capacity of each type of analysis server cluster and placed into the analysis queue. At specific intervals, the video streams of the devices are retrieved from the analysis queue and intelligent analysis is performed on the video streams. To quickly respond to new intelligent analysis requests from users, when a new intelligent analysis request is received, the pre-selected queues are adjusted in real time according to the polling interval of each type of intelligent analysis after the new request is added, so as to ensure that the new analysis request can be processed as soon as possible.

[0033] To facilitate understanding of the technical solutions of the embodiments of this application, the technical solutions of this application are described in detail below through specific embodiments. The above-mentioned related technologies are optional solutions and can be arbitrarily combined with the technical solutions of the embodiments of this application, all of which fall within the protection scope of the embodiments of this application. The embodiments of this application include at least some of the following contents.

[0034] It should be noted that the "analysis task" described in the embodiments of this application can also be called "intelligent analysis task", "AI analysis task" or "detection task".

[0035] It should be noted that the "queue to be analyzed" described in the embodiments of this application can also be called a "task queue".

[0036] Figure 1 This is a schematic diagram of the structural composition of the intelligent analysis system based on video surveillance provided in the embodiments of this application, as shown below. Figure 1 As shown, the intelligent analysis system includes a task scheduling module, a resource management module, a video frame extraction module, and an intelligent analysis module. Upon startup, the task scheduling module reads the database to obtain a list of devices corresponding to each type of analysis task, and generates a pre-selected queue for each type of analysis task based on this list. Then, at specific time intervals, the task scheduling module calls the resource management module to obtain the concurrent processing capacity of each type of analysis server cluster. Based on the concurrent processing capacity of each type of analysis server cluster and the pre-selected queue for each type of analysis task, a queue to be analyzed is generated (i.e., orchestrated), thereby enabling polling analysis of devices in the video surveillance system. In some optional implementations, for devices manually initiated by the user that require continuous analysis, the device is placed at the front of the queue to be analyzed to ensure timely response. The resource management module manages the hardware resource configuration of each type of analysis server cluster and calculates the concurrent processing capacity of each type of analysis server cluster in real time. The video frame extraction module is responsible for receiving video frame extraction messages (referred to as frame extraction messages) sent by the task scheduling module, and encapsulating the extracted images into topic messages corresponding to various analysis tasks according to the analysis task list in the frame extraction message; the intelligent analysis module is responsible for receiving topic messages corresponding to its own analysis tasks, obtaining the extracted images according to the topic messages, analyzing them, and reporting the analysis results.

[0037] Figure 2 This is a flowchart illustrating the intelligent analysis method based on video surveillance provided in the embodiments of this application. Figure 1 ,like Figure 2 As shown, the intelligent analysis method based on video surveillance includes:

[0038] Step 201: Obtain the device list corresponding to each type of analysis task in the multi-type analysis task, and generate a pre-selection queue corresponding to each type of analysis task based on the device list corresponding to each type of analysis task, wherein the pre-selection queue includes one or more devices to be analyzed.

[0039] In this embodiment, the task scheduling module reads the database and obtains a device list corresponding to each type of analysis task from the database. After obtaining the device list corresponding to each type of analysis task, the task scheduling module generates a pre-selection queue corresponding to each type of analysis task based on the device list, wherein the pre-selection queue includes one or more devices to be analyzed.

[0040] In this embodiment of the application, the step of generating the pre-selection queue corresponding to each type of analysis task based on the device list corresponding to each type of analysis task can be achieved in the following way:

[0041] A-1) Obtain the concurrent processing capability of each type of analysis server cluster in the multi-type analysis server cluster;

[0042] A-2) Generate a pre-selected queue for each type of analysis task based on the device list corresponding to each type of analysis task and the concurrent processing capability of the analysis server cluster for each type of analysis task.

[0043] In the above scheme, the generation of the pre-selected queue corresponding to each type of analysis task based on the device list corresponding to each type of analysis task and the concurrent processing capability of each type of analysis server cluster can be achieved in the following way:

[0044] A-2.1) Calculate the R value corresponding to each type of analysis task based on the number of devices in the device list corresponding to each type of analysis task and the concurrent processing capability of each type of analysis server cluster;

[0045] A-2.2) Sort each type of analysis task according to the R value corresponding to each type of analysis task in ascending order;

[0046] A-2.3) In accordance with the order of each type of analysis task, add the devices in the device list corresponding to the analysis task to the pre-selection queue corresponding to that analysis task.

[0047] In some alternative implementations, if a device is configured with multiple analysis tasks, and all of these analysis tasks are either polling analysis tasks or continuous analysis tasks, then the device is included in only one pre-selection queue.

[0048] In some alternative implementations, if a device is configured with multiple analysis tasks, and some of the analysis tasks are polling analysis tasks and others are continuous analysis tasks, then the device is included in the pre-selection queue corresponding to the polling analysis tasks and the pre-selection queue corresponding to the continuous analysis tasks.

[0049] Step 202: Obtain the concurrent processing capability of each type of analysis server cluster in the multi-type analysis server cluster, and generate a queue to be analyzed based on the concurrent processing capability of each type of analysis server cluster and the pre-selected queue corresponding to each type of analysis task.

[0050] In this embodiment of the application, the generation of the queue to be analyzed based on the concurrent processing capability of each type of analysis server cluster and the pre-selected queue corresponding to each type of analysis task can be achieved in the following way:

[0051] B-1) Calculate the R value corresponding to each type of analysis task based on the number of devices in the device list corresponding to each type of analysis task and the concurrent processing capability of each type of analysis server cluster;

[0052] B-2) Sort each type of analysis task in ascending order of the R value corresponding to each type of analysis task;

[0053] B-3) In accordance with the order of each type of analysis task, a specific number of devices are taken from the pre-selected queue corresponding to the analysis task and added to the queue to be analyzed.

[0054] Here, the step of retrieving a specific number of devices from the pre-selected queue corresponding to the analysis task and adding them to the queue to be analyzed can be achieved in the following way:

[0055] B-3.1) Determine the concurrent processing capacity of the analysis server cluster corresponding to the analysis task;

[0056] B-3.2) Determine the number of devices to be retrieved from the pre-selected queue corresponding to the analysis task based on the concurrent processing capacity of the analysis server cluster corresponding to the analysis task and the number of devices already included in the queue to be analyzed and configured with the analysis task;

[0057] B3.3) and based on the number of devices to be retrieved, a specific number of devices are retrieved from the pre-selected queue corresponding to the analysis task and added to the queue to be analyzed.

[0058] Step 203: Perform video surveillance analysis on each device to be analyzed in the queue to be analyzed.

[0059] In this embodiment of the application, the video surveillance analysis of each device to be analyzed in the queue to be analyzed can be achieved in the following ways:

[0060] C-1) The video frame extraction module receives the frame extraction message sent by the task scheduling module, performs frame extraction operation on the video of each device to be analyzed in the queue according to the frame extraction message, and encapsulates the extracted image into a topic message.

[0061] (C-2) The intelligent analysis module obtains the extracted images based on the topic message and performs video surveillance analysis on the extracted images. Furthermore, the intelligent analysis module encapsulates the analysis results into a result reporting message.

[0062] To facilitate understanding of the technical solutions in the embodiments of this application, the following uses specific application examples to illustrate the technical solutions in the embodiments of this application.

[0063] Figure 3 This is a flowchart illustrating the intelligent analysis method based on video surveillance provided in the embodiments of this application. Figure 2 ,like Figure 3 As shown, the intelligent analysis method based on video surveillance includes:

[0064] Step 301: When the task scheduling module starts, it retrieves the list of devices to be analyzed from the database and generates a pre-selected queue corresponding to each type of analysis task based on the analysis tasks configured for each device in the list of devices to be analyzed.

[0065] Specifically, pre-selected queues for various analysis tasks can be generated in the following ways:

[0066] 1) The task scheduling module calls the resource management module to obtain the concurrent processing capabilities of various types of analysis server clusters. For example, it obtains the concurrent processing capabilities of n types of analysis server clusters, namely P1…Pn.

[0067] 2) The task scheduling module calculates the R value corresponding to each type of analysis task. For example, the R values ​​corresponding to n types of analysis tasks are R1…Rn.

[0068] Where Ri = number of devices corresponding to the i-th type of analysis task / concurrent processing capacity of the analysis server cluster corresponding to the i-th type of analysis task, and i is a positive integer greater than or equal to 1 and less than or equal to n.

[0069] 3) Sort each type of analysis task according to the R value from smallest to largest, and record it in the cache.

[0070] 4) In accordance with the order of each type of analysis task, add the devices corresponding to the analysis task to the pre-selection queue corresponding to that analysis task.

[0071] In some alternative implementations, if a device is configured with multiple analysis tasks, and these tasks are either polling or continuous, the device will only be included in a pre-selection queue. Specifically, the device will be included in the pre-selection queue corresponding to the analysis task with the smallest R value.

[0072] As an example: There are two analysis tasks: video quality detection and fire detection. There are 30,000 devices configured for video quality detection and 3,000 devices configured for fire detection. Of these, 1,000 devices are configured for both. The concurrent processing capacity of the analysis server cluster for video quality detection is 200 channels, and the concurrent processing capacity of the analysis server cluster for fire detection is 200 channels. The R-value for video quality detection is 30,000 / 200 = 150, and the R-value for fire detection is 3,000 / 200 = 15. When establishing the pre-selection queues, the pre-selection queues corresponding to fire detection are established first, in ascending order of R value. The 2,000 devices configured with fire detection and the 1,000 devices configured with both fire detection and video quality detection are included in the pre-selection queue corresponding to fire detection, resulting in 3,000 devices in the pre-selection queue corresponding to fire detection. Next, the pre-selection queue corresponding to video quality detection is established. Since each device is included in only one pre-selection queue, the pre-selection queue corresponding to video quality detection contains only 29,000 devices.

[0073] In some alternative implementations, if a device is configured with multiple analysis tasks, some of which are polling analysis tasks and others are continuous analysis tasks, the analysis tasks are first divided into two main categories: polling analysis tasks Cp and continuous analysis tasks Cs. For each of the polling analysis tasks Cp and continuous analysis tasks Cs, the device is included in the pre-selection queue corresponding to the polling analysis task and the pre-selection queue corresponding to the continuous analysis task.

[0074] It should be noted that for devices configured with continuous analysis tasks, once the device is placed in the analysis queue, it will be removed from the corresponding pre-selected queue, and the continuous analysis task will continuously occupy analysis resources. Therefore, the polling analysis task Cp and the continuous analysis task Cs should be handled separately.

[0075] As an example: If a device is configured with a polling analysis task and a continuous analysis task, such as polling for fire detection and continuously performing quality inspection, then the device should be placed in two pre-selection queues: one for polling fire detection and the other for continuous quality inspection.

[0076] In the above scheme, considering the principle of resource conservation, the device is included in the corresponding pre-selection queue based on the principle of minimizing the R value, so as to save the hardware resources of the video frame extraction module.

[0077] Step 302: The task scheduling module, at specific time intervals, selects a specific number of devices from each pre-selected queue and adds them to the queue to be analyzed, according to the order of R values ​​from smallest to largest.

[0078] Specifically, a specific number of devices can be removed from each pre-selected queue and added to the queue to be analyzed in the following way:

[0079] 1) The task scheduling module calls the resource management module to obtain the concurrent processing capacity (i.e., concurrent processing count) of various types of analysis server clusters. For example, if the concurrent processing counts of n types of analysis server clusters are obtained as P1…Pn, then for each type of analysis server cluster, Nx is initialized to the concurrent processing count of that type of analysis server cluster.

[0080] 2) Initialize i to 0.

[0081] 3) Select devices from the pre-selected queue Qi corresponding to R value Ri, whose concurrent processing capacity is equal to that of the corresponding analysis server cluster. For each device, the corresponding analysis task in the pre-selected queue Qi is Ai. Check the detection times Ti+1…Tn corresponding to other types of analysis tasks Ai+1…An for that device. If the current time is greater than Tx, include the analysis task of type x in the analysis task list and update Tx. The update method for Tx is: calculate "(total number of devices with other types of analysis tasks * single frame extraction duration) / concurrent processing capacity of other types of analysis server clusters * inspection interval" to obtain the polling interval for each device corresponding to other types of analysis tasks. Add this polling interval to the current time to set the next detection time. Decrement Nx by 1 and record the current access position Qil of the pre-selected queue Qi.

[0082] 4) Increment i by 1 and repeat step 3) until all preselected queues have been processed.

[0083] In the above scheme, for devices configured with continuous analysis tasks, once a device is added to the analysis queue, it is removed from the corresponding pre-selected queue to avoid duplicate analysis. Simultaneously, the task scheduling module calls the resource management module to modify the concurrent processing capacity of the analysis server cluster corresponding to the current analysis task.

[0084] As an example: There are two analysis tasks: video quality detection and fire detection. There are 30,000 devices configured for video quality detection and 3,000 devices configured for fire detection. Of these, 1,000 devices are configured for both. The concurrent processing capacity of the analysis server cluster for video quality detection is 200 channels, and the concurrent processing capacity of the analysis server cluster for fire detection is 200 channels. The R-value for video quality detection is 30,000 / 200 = 150, and the R-value for fire detection is 3,000 / 200 = 15. The pre-selection queue for fire detection contains 3,000 devices, while the pre-selection queue for video quality detection contains only 29,000 devices. When adding devices from the two pre-selected queues to the analysis queue, in ascending order of R value, first select 200 devices (i.e., the concurrency of the analysis server cluster corresponding to fire detection) from the pre-selected queue corresponding to fire detection and add them to the analysis queue. If n of these 200 devices are configured with both fire detection and video quality detection, then 200-n devices are selected from the pre-selected queue corresponding to video quality detection and added to the analysis queue.

[0085] In some optional implementations, the task scheduling module provides an external interface for user-side calls. When it receives a user's call to add a specific analysis task for a specific device, it calls the resource management module to obtain the concurrent processing capabilities P1…Pn of various analysis server clusters, updates the R values ​​corresponding to various analysis tasks, for example, the R value corresponding to an analysis task = "(number of devices corresponding to the current analysis task + number of changes) / concurrent processing capability of the analysis server cluster corresponding to the analysis task", and rearranges the analysis tasks in ascending order of R values. This sorting is compared with the order in the cache. If the order has not changed, the newly added device is added to the pre-selection queue according to step 301, where the device is inserted after the current processing position Qil in the pre-selection queue to facilitate a quick response to the user's new requests. If the order has changed, each pre-selection queue needs to be cleared, and each pre-selection queue needs to be regenerated according to step 301. At the same time, the newly added device is inserted at the front of the pre-selection queue to facilitate a quick response to the user's new requests. It should be noted that during normal system operation, the add operation is performed in batches.

[0086] In the above-described scheme of this application embodiment, the resource management module has the following functions: I) Querying the database at startup to obtain the concurrent processing capacity of all analysis server clusters. II) Providing a concurrent processing capacity update interface, which is called when the task scheduling module initiates a request for a continuous analysis task for a certain device to decrement the concurrent processing capacity of the analysis server cluster by 1. III) Providing a concurrent processing capacity query interface for the task scheduling module to call.

[0087] Step 303: The task scheduling module retrieves the live stream of the device based on the queue to be analyzed at specific intervals, constructs a frame extraction message and sends it to Kafka. The frame extraction message carries the live stream address of each device and the type of the corresponding analysis task.

[0088] Step 304: The video frame extraction module receives the frame extraction message sent by the task scheduling module, performs video frame extraction operation according to the frame extraction message, and encapsulates the extracted images into a topic message.

[0089] In some optional implementations, for the video stream being polled for analysis, the video frame extraction module receives and parses the frame extraction message sent by the task scheduling module, performs video frame extraction based on the frame extraction message, saves the extracted images as .mat files to the file system (such as FastFS), and obtains the file storage path (such as a URL) of the images. The video frame extraction module then encapsulates the file storage path of the extracted images into different types of intelligent analysis messages (i.e., topic messages) based on the analysis task list in the frame extraction message and sends them.

[0090] In some optional implementations, for continuously analyzed live streams, the video frame extraction module receives and parses the frame extraction message sent by the task scheduling module, performs video frame extraction operations according to the frame extraction message, and saves the image after each frame extraction as a .mat file to the file system (such as FastFS) based on the single frame extraction duration in the frame extraction message, and obtains the file storage path of the image. The video frame extraction module encapsulates the file storage path of the extracted images into different types of intelligent analysis messages (i.e., topic messages) according to the analysis task list in the frame extraction message and sends them. When the video frame extraction module receives a stop frame extraction message, it parses the stop frame extraction message and stops the frame extraction operation.

[0091] In particular, for live streams that fail to acquire frames or fail to extract frames consecutively, the video frame extraction module needs to report the error information.

[0092] Step 305: The intelligent analysis module receives the topic message sent by the video frame extraction module, obtains the extracted images according to the topic message, and performs video monitoring analysis on the extracted images.

[0093] In this embodiment of the application, the intelligent analysis module receives and parses the intelligent parsing message (i.e., topic message) sent by the video frame extraction module, performs intelligent analysis on the extracted images, and encapsulates the analysis results into the result reporting message.

[0094] The technical solution of this application embodiment has at least the following beneficial effects:

[0095] On the one hand, multiple pre-selected queues are used to reduce database operations and improve polling efficiency. Specifically, during initialization, the database is read, and devices are placed into pre-selected queues for different analysis tasks according to the time interval between the processing of each device by the corresponding analysis server cluster. Subsequently, devices are directly retrieved from the pre-selected queues and placed into the queue to be analyzed, reducing database operations during the polling process and improving polling efficiency. When a new intelligent analysis request is received from a user, the pre-selected queues are adjusted in real time according to the polling interval for each type of intelligent analysis after the new request, ensuring that the new analysis request can be processed as soon as possible.

[0096] On the other hand, by sharing the video frame extraction module, the access pressure on the existing monitoring system interface is reduced, thus lowering the consumption of network and computing resources. Specifically, for devices that need to perform multiple intelligent analyses simultaneously (i.e., devices configured with multiple analysis tasks), they are placed into a pre-selection queue of only the same intelligent analysis type according to the analysis type. This avoids repeatedly calling the monitoring system interface to obtain live streams, reducing the consumption of network and frame extraction server computing resources due to multi-stream concurrency.

[0097] On the other hand, establishing a unified queue for all intelligent analysis tasks facilitates unified task management and system deployment and maintenance. Specifically, devices from multiple pre-selected queues are placed into the same queue at specific time intervals according to a scheduling strategy. All subsequent operations, such as obtaining live streams from devices via the monitoring system interface and sending frame extraction messages, are performed on the same queue, which facilitates unified management of a large number of intelligent analysis tasks and also facilitates system deployment and maintenance.

[0098] On the other hand, a dynamically adjustable scheduling strategy is used to ensure fairness. Specifically, based on the time interval between each device being processed by the corresponding analysis server cluster, and combined with the current concurrent processing capacity of each type of analysis server cluster, a specific number of devices are selected from the pre-selected queue and placed into the queue to be analyzed. If a device in the current queue needs to perform other types of intelligent analysis at the same time, it is enqueued according to its original polling interval to ensure the fairness of scheduling.

[0099] Figure 4 This is a schematic diagram of the structural composition of the intelligent analysis system based on video surveillance provided in the embodiments of this application, as shown below. Figure 4 As shown, the intelligent analysis system based on video surveillance includes: a task scheduling module 401, a resource management module 402, a video frame extraction module 403, and an intelligent analysis module 404; wherein,

[0100] The task scheduling module 401 is configured to obtain a device list corresponding to each type of analysis task in a multi-type analysis task, and generate a pre-selection queue corresponding to each type of analysis task based on the device list, wherein the pre-selection queue includes one or more devices to be analyzed; and to call the resource management module 402 to obtain the concurrent processing capability of each type of analysis server cluster in the multi-type analysis server cluster; and generate a queue to be analyzed based on the concurrent processing capability of each type of analysis server cluster and the pre-selection queue corresponding to each type of analysis task.

[0101] The resource management module 402 is used to accept the call from the task scheduling module and obtain the concurrent processing capacity of each type of analysis server cluster in the multi-type analysis server cluster.

[0102] The video frame extraction module 403 is used to perform frame extraction operation on the video of each device to be analyzed in the queue to be analyzed.

[0103] The intelligent analysis module 404 is used to perform video monitoring analysis on the frame-extracted images corresponding to each device in the analysis queue.

[0104] In some optional embodiments, the task scheduling module 401 is specifically used for:

[0105] A pre-selected queue is generated based on the device list corresponding to each type of analysis task and the concurrent processing capability of the analysis server cluster for each type of analysis task.

[0106] In some optional embodiments, the task scheduling module 401 is specifically used for:

[0107] The R value for each type of analysis task is calculated based on the number of devices in the device list corresponding to each type of analysis task and the concurrent processing capability of the analysis server cluster for each type of analysis task.

[0108] Sort the analysis tasks according to the R value corresponding to each type of analysis task in ascending order;

[0109] According to the order of each type of analysis task, the devices in the device list corresponding to the analysis task are included in the pre-selection queue corresponding to that analysis task.

[0110] In some alternative implementations, if a device is configured with multiple analysis tasks, and all of the multiple analysis tasks are either polling analysis tasks or continuous analysis tasks, then the device is included in only one pre-selection queue; or, if a device is configured with multiple analysis tasks, and some of the multiple analysis tasks are polling analysis tasks and others are continuous analysis tasks, then the device is included in both the pre-selection queue corresponding to the polling analysis tasks and the pre-selection queue corresponding to the continuous analysis tasks.

[0111] In some optional embodiments, the task scheduling module 401 is specifically used for:

[0112] The R value for each type of analysis task is calculated based on the number of devices in the device list corresponding to each type of analysis task and the concurrent processing capability of the analysis server cluster for each type of analysis task.

[0113] Sort the analysis tasks according to the R value corresponding to each type of analysis task in ascending order;

[0114] According to the order of each type of analysis task, a specific number of devices are taken from the pre-selected queue corresponding to the analysis task and added to the queue to be analyzed.

[0115] In some optional embodiments, the task scheduling module 401 is specifically used for:

[0116] Determine the concurrent processing capacity of the analysis server cluster corresponding to the analysis task;

[0117] The number of devices to be retrieved from the pre-selected queue corresponding to the analysis task is determined based on the concurrent processing capacity of the analysis server cluster corresponding to the analysis task and the number of devices already included in the queue to be analyzed and configured with the analysis task.

[0118] Based on the number of devices to be retrieved, a specific number of devices are retrieved from the pre-selected queue corresponding to the analysis task and added to the queue to be analyzed.

[0119] In some optional embodiments, the video frame extraction module 403 is used to receive a frame extraction message, perform frame extraction operation on the video of each device to be analyzed in the queue to be analyzed according to the frame extraction message, and encapsulate the extracted image into a topic message.

[0120] The intelligent analysis module 404 is used to obtain the extracted images based on the topic message and to perform video monitoring analysis on the extracted images.

[0121] In some optional embodiments, the video frame extraction module 403 is specifically used for:

[0122] For the polling analysis video stream, a frame extraction operation is performed according to the frame extraction message, the extracted images are uploaded to the file system, and the file storage path of the images is obtained; the file storage path of the images is encapsulated into a topic message according to the frame extraction message and sent.

[0123] For continuously analyzed live streams, frame extraction is performed according to the frame extraction message. The image after each frame extraction is uploaded to the file system, and the file storage path of the image is obtained, using the single frame extraction duration in the frame extraction message as the unit. The file storage path of the image is encapsulated into a topic message and sent according to the frame extraction message. After receiving a stop frame extraction message, the frame extraction operation is stopped according to the stop frame extraction message.

[0124] In some optional implementations, the intelligent analysis module 404 is specifically used to: obtain the image after frame extraction based on the topic message, perform intelligent video surveillance analysis on the image after frame extraction, and encapsulate the analysis results into a result reporting message.

[0125] Those skilled in the art should understand that Figure 4 The implementation functions of each module in the intelligent analysis system based on video surveillance shown can be understood by referring to the relevant descriptions of the aforementioned methods. Figure 4 The functions of each module in the video surveillance-based intelligent analysis system shown can be implemented through programs running on a processor or through specific logic circuits.

[0126] Figure 5 This is a schematic structural diagram of an electronic device provided in an embodiment of this application. The electronic device is used to implement the intelligent analysis system based on video surveillance in the above-described scheme. Figure 5 The illustrated electronic device 500 includes a processor 510, which can call and run computer programs from memory to implement the methods in the embodiments of this application.

[0127] Optionally, such as Figure 5 As shown, the electronic device 500 may further include a memory 520. The processor 510 can retrieve and run computer programs from the memory 520 to implement the methods described in the embodiments of this application.

[0128] The memory 520 can be a separate device independent of the processor 510, or it can be integrated into the processor 510.

[0129] Optionally, such as Figure 5As shown, the electronic device 500 may also include a transceiver 530, which the processor 510 can control to communicate with other devices. Specifically, it can send information or data to other devices or receive information or data sent by other devices.

[0130] The transceiver 530 may include a transmitter and a receiver. The transceiver 530 may further include antennas, and the number of antennas may be one or more.

[0131] The electronic device 500 can implement the corresponding processes of the various methods implemented in the embodiments of this application, which will not be described in detail here for the sake of brevity.

[0132] Figure 6 This is a schematic structural diagram of the chip according to an embodiment of this application. Figure 6 The chip 600 shown includes a processor 610, which can call and run computer programs from memory to implement the methods in the embodiments of this application.

[0133] Optionally, such as Figure 6 As shown, chip 600 may further include memory 620. Processor 610 can retrieve and run computer programs from memory 620 to implement the methods described in this embodiment.

[0134] The memory 620 can be a separate device independent of the processor 610, or it can be integrated into the processor 610.

[0135] Optionally, the chip 600 may also include an input interface 630. The processor 610 can control the input interface 630 to communicate with other devices or chips; specifically, it can acquire information or data sent by other devices or chips.

[0136] Optionally, the chip 600 may also include an output interface 640. The processor 610 can control the output interface 640 to communicate with other devices or chips, specifically, to output information or data to other devices or chips.

[0137] The chip can implement the corresponding processes of the various methods in the embodiments of this application, which will not be described in detail here for the sake of brevity.

[0138] It should be understood that the chip mentioned in the embodiments of this application may also be referred to as a system-on-a-chip, system chip, chip system, or system-on-a-chip, etc.

[0139] It should be understood that the processor in the embodiments of this application may be an integrated circuit chip with signal processing capabilities. In implementation, the steps of the above method embodiments can be completed by integrated logic circuits in the processor's hardware or by instructions in software form. The processor described above can be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this application can be directly embodied in the execution of a hardware decoding processor, or executed by a combination of hardware and software modules in the decoding processor. The software modules can be located in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. The storage medium is located in memory, and the processor reads information from the memory and, in conjunction with its hardware, completes the steps of the above method.

[0140] It is understood that the memory in the embodiments of this application can be volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as Static Random Access Memory (SRAM), Dynamic Random Access Memory (DRAM), Synchronous DRAM (SDRAM), Double Data Rate SDRAM (DDR SDRAM), Enhanced Synchronous DRAM (ESDRAM), Synchlink DRAM (SLDRAM), and Direct Rambus RAM (DR RAM). It should be noted that the memory used in the systems and methods described herein is intended to include, but is not limited to, these and any other suitable types of memory.

[0141] It should be understood that the above-described memory is exemplary and not a limiting description. For example, the memory in the embodiments of this application may also be static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM), and direct memory bus RAM (DR RAM), etc. That is to say, the memory in the embodiments of this application is intended to include, but is not limited to, these and any other suitable types of memory.

[0142] This application also provides a computer-readable storage medium for storing a computer program. This computer-readable storage medium can implement the corresponding processes of the various methods implemented in the embodiments of this application; for the sake of brevity, these will not be elaborated upon here.

[0143] This application also provides a computer program product, including computer program instructions. This computer program product can implement the corresponding processes of the various methods implemented in the embodiments of this application; for the sake of brevity, these will not be elaborated upon here.

[0144] This application also provides a computer program. This computer program can implement the corresponding processes of the various methods implemented in the embodiments of this application, which will not be described in detail here for the sake of brevity.

[0145] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0146] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0147] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0148] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0149] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0150] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0151] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. An intelligent analysis method based on video surveillance, characterized in that, The method includes: Obtain a list of devices corresponding to each type of analysis task in the multi-type analysis task, and generate a pre-selection queue corresponding to each type of analysis task based on the list of devices corresponding to each type of analysis task, wherein the pre-selection queue includes one or more devices to be analyzed; Obtain the concurrent processing capability of each type of analysis server cluster in the multi-type analysis server cluster, and generate a queue to be analyzed based on the concurrent processing capability of each type of analysis server cluster and the pre-selected queue corresponding to each type of analysis task. Perform video surveillance analysis on each device to be analyzed in the queue of devices to be analyzed; The step of generating a queue to be analyzed based on the concurrent processing capacity of each type of analysis server cluster and the pre-selected queue corresponding to each type of analysis task includes: The R value for each type of analysis task is calculated based on the number of devices in the device list corresponding to each type of analysis task and the concurrent processing capacity of the analysis server cluster for each type of analysis task; wherein, the R value for each type of analysis task = the number of devices corresponding to each type of analysis task / the concurrent processing capacity of the analysis server cluster for each type of analysis task. Sort the analysis tasks according to the R value corresponding to each type of analysis task in ascending order; According to the order of each type of analysis task, a specific number of devices are taken from the pre-selected queue corresponding to the analysis task and added to the queue to be analyzed.

2. The method according to claim 1, characterized in that, The step of generating a pre-selected queue for each type of analysis task based on the device list corresponding to each type of analysis task includes: Obtain the concurrent processing capacity of each type of analysis server cluster in multiple analysis server clusters; A pre-selected queue is generated based on the device list corresponding to each type of analysis task and the concurrent processing capability of the analysis server cluster for each type of analysis task.

3. The method according to claim 2, characterized in that, The process of generating a pre-selected queue for each type of analysis task based on the device list corresponding to each type of analysis task and the concurrent processing capability of each type of analysis server cluster includes: The R value for each type of analysis task is calculated based on the number of devices in the device list corresponding to each type of analysis task and the concurrent processing capability of the analysis server cluster for each type of analysis task. Sort the analysis tasks according to the R value corresponding to each type of analysis task in ascending order; According to the order of each type of analysis task, the devices in the device list corresponding to the analysis task are included in the pre-selection queue corresponding to that analysis task.

4. The method according to claim 3, characterized in that, If a device is configured with multiple analysis tasks, and all of these tasks are either polling analysis tasks or continuous analysis tasks, then the device will only be included in one pre-selection queue; or, If a device is configured with multiple analysis tasks, and some of the analysis tasks are polling analysis tasks while others are continuous analysis tasks, then the device is included in the pre-selection queue corresponding to the polling analysis tasks and the pre-selection queue corresponding to the continuous analysis tasks.

5. The method according to claim 1, characterized in that, The step of retrieving a specific number of devices from the pre-selected queue corresponding to the analysis task and adding them to the queue to be analyzed includes: Determine the concurrent processing capacity of the analysis server cluster corresponding to the analysis task; The number of devices to be retrieved from the pre-selected queue corresponding to the analysis task is determined based on the concurrent processing capacity of the analysis server cluster corresponding to the analysis task and the number of devices already included in the queue to be analyzed and configured with the analysis task. Based on the number of devices to be retrieved, a specific number of devices are retrieved from the pre-selected queue corresponding to the analysis task and added to the queue to be analyzed.

6. The method according to any one of claims 1 to 4, characterized in that, The step of performing video surveillance analysis on each device to be analyzed in the queue includes: Receive frame extraction messages, perform frame extraction operations on the video of each device to be analyzed in the queue of devices to be analyzed according to the frame extraction messages, and encapsulate the extracted images into a topic message; The extracted images are obtained based on the topic message, and the extracted images are then subjected to video monitoring analysis.

7. The method according to claim 6, characterized in that, The step of encapsulating the extracted image into a topic message based on the extracted frame message includes: For the polling analysis video stream, a frame extraction operation is performed according to the frame extraction message, the extracted images are uploaded to the file system, and the file storage path of the images is obtained; the file storage path of the images is encapsulated into a topic message according to the frame extraction message and sent. For continuously analyzed live streams, frame extraction is performed according to the frame extraction message. The image after each frame extraction is uploaded to the file system, and the file storage path of the image is obtained, using the single frame extraction duration in the frame extraction message as the unit. The file storage path of the image is encapsulated into a topic message and sent according to the frame extraction message. After receiving a stop frame extraction message, the frame extraction operation is stopped according to the stop frame extraction message.

8. The method according to claim 6, characterized in that, The video monitoring and analysis of the topic messages includes: Based on the topic message, the extracted images are obtained, the extracted images are subjected to intelligent video surveillance analysis, and the analysis results are encapsulated into a result reporting message.

9. An intelligent analysis system based on video surveillance, characterized in that, The system includes: a task scheduling module, a resource management module, a video frame extraction module, and an intelligent analysis module; wherein, The task scheduling module is configured to obtain a device list corresponding to each type of analysis task in a multi-type analysis task, and generate a pre-selected queue corresponding to each type of analysis task based on the device list, wherein the pre-selected queue includes one or more devices to be analyzed; and to call the resource management module to obtain the concurrent processing capacity of each type of analysis server cluster in the multi-type analysis server cluster; and generate a queue to be analyzed based on the concurrent processing capacity of each type of analysis server cluster and the pre-selected queue corresponding to each type of analysis task. The resource management module is used to accept the call from the task scheduling module and obtain the concurrent processing capacity of each type of analysis server cluster in the multi-type analysis server cluster. The video frame extraction module is used to perform frame extraction operations on the video of each device to be analyzed in the queue to be analyzed. The intelligent analysis module is used to perform video monitoring analysis on the frame-extracted images corresponding to each device in the analysis queue. The task scheduling module is specifically used for: The R value for each type of analysis task is calculated based on the number of devices in the device list corresponding to each type of analysis task and the concurrent processing capacity of the analysis server cluster for each type of analysis task; wherein, the R value for each type of analysis task = the number of devices corresponding to each type of analysis task / the concurrent processing capacity of the analysis server cluster for each type of analysis task. Sort the analysis tasks according to the R value corresponding to each type of analysis task in ascending order; According to the order of each type of analysis task, a specific number of devices are taken from the pre-selected queue corresponding to the analysis task and added to the queue to be analyzed.

10. An electronic device, characterized in that, include: A processor and a memory for storing a computer program, the processor for calling and running the computer program stored in the memory to perform the method as described in any one of claims 1 to 8.

11. A chip, characterized in that, include: A processor for retrieving and running a computer program from memory, causing a device on which the chip is mounted to perform the method as described in any one of claims 1 to 8.

12. A computer-readable storage medium, characterized in that, Used to store a computer program that causes a computer to perform the method as described in any one of claims 1 to 8.

Images