A method and system for real-time target detection of multi-channel videos in a substation

Abstract

The present invention discloses a method and system for real-time target detection of multi-channel video in a substation, comprising: obtaining multi-channel video stream data of a substation to be detected; calling a hardware environment parameter interface, obtaining current hardware environment parameters, and judging the current hardware environment; based on the current hardware environment, calling an interface function under a related hardware environment, decoding the obtained multi-channel video stream data; merging the decoded multi-channel video streams, and then inputting them into a trained target detection model, and outputting the target detection results of each channel. The present invention can be adapted to different hardware environments or hardware facilities at the same time, and can realize real-time target detection of multi-channel videos in both a GPU environment and a CPU environment without changing the algorithm model.

Description

Technical Field

[0001] The present invention relates to the technical field of target detection, and in particular to a method and system for real-time target detection of multi-channel videos in a transformer substation. Background Art

[0002] The statements in this section merely provide background information related to the present invention and do not necessarily constitute prior art.

[0003] Real-time video detection technology based on target detection mainly detects and classifies specific objects of interest from images or videos. This technology can be applied to the remote visual monitoring of substations, and real-time video detection technology can be used to detect and alarm targets such as those not wearing work clothes, not wearing safety helmets, or hanging suspended objects.

[0004] Current target detection models mostly rely on neural network models, which require a lot of computing power. Existing technical solutions mostly deploy models under NVIDIA's GPU or Intel's CPU, increasing the demand for real-time target detection in multiple videos by increasing the algorithm. However, due to the inconsistency between the two in terms of architectural parameters and processing algorithms, the isolation between the routes is relatively large and cannot be deployed across hardware.

[0005] However, unlike academic research, various demand scenarios will appear in practical applications. Current video-based real-time object detection technologies are mostly implemented based on specific hardware deployments or developed based on specific model frameworks, and cannot flexibly respond to different algorithm change requirements and different hardware environments. Summary of the invention

[0006] In order to solve the above problems, the present invention proposes a real-time target detection method and system for multi-channel videos in a substation, which can be flexibly deployed in various application scenarios or different hardware environments without changing the algorithm model, so as to maximize the satisfaction of the needs of different users.

[0007] In some embodiments, the following technical solutions are adopted:

[0008] A method for real-time target detection of multi-channel videos in a substation, comprising:

[0009] Obtain multi-channel video stream data of the substation to be inspected;

[0010] Call the hardware environment parameter interface to obtain the current hardware environment parameters and determine the current hardware environment;

[0011] Based on the current hardware environment, call the interface function under the relevant hardware environment to decode the acquired multi-channel video stream data;

[0012] The decoded multi-channel video streams are merged and then input into the trained object detection model to output the object detection results of each channel.

[0013] As a further solution, current hardware environment parameters are obtained to determine the current hardware environment, where the current hardware environment includes a CPU environment or a GPU environment.

[0014] As a further solution, after determining the current hardware environment, the method further includes: configuring the detection target of each video stream based on a parameter configuration interface under the current hardware environment.

[0015] As a further solution, based on the detection targets configured in each channel, the detection results of each channel of the target detection model are screened, and only the detection results of the detection targets configured in each channel are retained and output.

[0016] As a further solution, the target detection model selects a deep neural network model, and compresses the target detection model based on a model compression method under the current hardware environment.

[0017] As a further solution, the process of compressing the target detection model includes: inter-layer fusion or tensor fusion, wherein the inter-layer fusion includes horizontal merging and vertical merging.

[0018] As a further solution, the decoding and merging of multi-channel video stream data, the processing of the target detection model and the output of the detection results of each video stream are all implemented based on the gstreamer framework.

[0019] In other embodiments, the following technical solutions are adopted:

[0020] A multi-channel video real-time target detection system for a substation, comprising:

[0021] A data acquisition module, used to acquire multi-channel video stream data of the substation to be detected;

[0022] The hardware environment judgment module is used to call the hardware environment parameter interface, obtain the current hardware environment parameters, and judge the current hardware environment;

[0023] The data decoding module is used to call the interface function under the relevant hardware environment based on the current hardware environment to decode the acquired multi-channel video stream data;

[0024] The target detection module is used to merge the decoded multi-channel video streams, and then input them into the trained target detection model to output the target detection results of each channel.

[0025] In other embodiments, the following technical solutions are adopted:

[0026] A terminal device comprises a processor and a memory, wherein the processor is used to implement various instructions; and the memory is used to store a plurality of instructions, wherein the instructions are suitable for being loaded by the processor and executing the above-mentioned substation multi-channel video real-time target detection method.

[0027] In other embodiments, the following technical solutions are adopted:

[0028] A computer-readable storage medium stores a plurality of instructions, wherein the instructions are suitable for being loaded by a processor of a terminal device and executing the above-mentioned substation multi-channel video real-time target detection method.

[0029] Compared with the prior art, the present invention has the following beneficial effects:

[0030] (1) The present invention proposes a substation multi-channel video target detection method that can adapt to different hardware environments at the same time. It can adapt to different hardware environments or hardware facilities at the same time, and can realize multi-channel video real-time target detection in a GPU environment or a CPU environment without changing the algorithm model; based on different hardware environments, a target detection method adapted to the corresponding target environment is adopted, including: video decoding, video stream merging, model compression method, target detection and output process, etc. At the same time, different target detection results can be output for each video stream respectively, which can meet the independence between the detection results of multiple channels of video; at the same time, it can greatly reduce the requirements of the target detection algorithm for hardware memory and performance, and improve the target detection efficiency.

[0031] (2) The present invention reduces the consumption of computing resources by compressing the target detection model, making the entire target detection model smaller in structure, less computationally intensive, and more efficient in processing speed.

[0032] Other features and advantages of additional aspects of the present invention will be given in part in the following description, and in part will become obvious from the following description, or will be learned through the practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 The present invention is a flowchart of a method for real-time target detection of multiple video channels in a substation according to an embodiment of the present invention. DETAILED DESCRIPTION

[0034] It should be noted that the following detailed descriptions are illustrative and are intended to provide further explanation of the present application. Unless otherwise specified, all technical and scientific terms used in the present invention have the same meanings as those commonly understood by those skilled in the art to which the present application belongs.

[0035] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit the exemplary embodiments according to the present application. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "comprise" and / or "include" are used in this specification, it indicates the presence of features, steps, operations, devices, components and / or combinations thereof.

[0036] Embodiment 1

[0037] In one or more embodiments, a method for real-time target detection of multiple video channels in a substation is disclosed, referring to Figure 1 , specifically including the following process:

[0038] (1) Obtain multiple video stream data of the substation to be inspected; specifically, when performing remote video keying on the substation, obtain multiple substation remote monitoring data. The multiple videos refer to video data from different inspection points, which come from different cameras.

[0039] (2) Call the hardware environment parameter interface to obtain the current hardware environment parameters and determine the current hardware environment;

[0040] Specifically, the current hardware environment can be a GPU environment or a CPU environment. If it is a GPU environment, the function interface of the GPU is called and a video data processing method adapted to the GPU is adopted. Similarly, if it is a CPU environment, the function interface of the CPU is called and a video data processing method adapted to the CPU is adopted.

[0041] The video data processing method includes the following processes: video stream data decoding, multi-channel video stream data merging, target detection model processing and output.

[0042] (3) Based on the current hardware environment, call the interface function under the relevant hardware environment to decode the acquired multi-channel video stream data;

[0043] The process of decoding the multi-channel video stream data of the substation is the same as the prior art and will not be described in detail.

[0044] (4) The decoded multi-channel substation video streams are merged and then input into the trained target detection model to output the target detection results of each channel.

[0045] In this embodiment, after obtaining the current hardware environment, the detection target of each video stream can be configured based on the parameter configuration interface under the current hardware environment, that is, the target detection item of each stream can be set.

[0046] This embodiment is based on the gstreamer framework, and the decoding of video streams, merging of multiple video streams, and concurrent detection and output of multiple videos are all performed under this framework. The framework supports concurrent detection of multiple videos, and a configuration interface can be built through a standard configuration interface function. In the configuration interface, the target detection project of each video channel can be selected, for example: Channel 1 can detect whether work clothes are not worn in the substation, and Channel 2 can detect whether safety helmets are not worn in the substation, etc.; it can also be set whether the adapter of the channel needs to output real-time detection video.

[0047] For the merging of data streams, you can create a series of components and connect them together so that the data stream can be transmitted between the connected components. Each component has a special function interface. For some components, the function interfaces are used to read file data and decode file data. The function interfaces of some components only output the corresponding data to specific devices (for example, sound card devices). Several components can be connected together to create a pipeline to complete the task of merging video streams. Gstreamer integrates N videos together through pipelines, and then performs target detection. It also separates multiple videos in an orderly manner through demutex to achieve synchronous detection and different output targets.

[0048] Based on the detection targets configured for each channel, after the target detection model outputs the target detection results for each channel, the target detection results for each channel can be filtered, and only the detection results of the detection targets configured for each channel are retained and output, while the remaining target detection results are filtered out, so that each channel can output the target detection results configured by itself; at the same time, a function interface for outputting alarm information is given, and the alarm text information and alarm pictures of each video can be obtained through the interface.

[0049] In this embodiment, the target detection model uses a deep neural network model, and specifically a model based on darknet yolov5 can be used. The collected image sample data is annotated using an image annotation tool, and the target detection model is trained according to the target detection type requirements.

[0050] In order to reduce the amount of data calculation of the model without affecting the calculation accuracy, this embodiment compresses the model file and compresses the model space volume through quantization, pruning, distillation and other technologies to achieve less consumption of computing resources.

[0051] Model compression mainly includes: inter-layer fusion or tensor fusion; there are many layers in the deep neural network structure. If the model reasoning is deployed on GPU-based hardware, the calculation operations of each layer are completed by the GPU, but in fact the GPU completes the calculation by starting different CUDA cores. The CUDA core calculates tensors very quickly, but often a lot of time is wasted on starting the CUDA core and reading and writing the input and output tensors of each layer, which causes a bottleneck in memory bandwidth and a waste of GPU resources.

[0052] For example, the inter-layer fusion process includes horizontal or vertical merging of layers, which greatly reduces the number of layers. Horizontal merging can merge convolution, bias, and activation layers into a CBR structure that only occupies one CUDA core. Vertical merging can merge layers with the same structure but different weights into a wider layer that also only occupies one CUDA core. The computational graph after merging has fewer layers and occupies fewer CUDA cores, so the entire model structure will be smaller, faster, and more efficient.

[0053] Data precision calibration: Most deep learning frameworks use 32-bit floating point precision for tensors in the network when training neural networks. Once the network training is completed, the data precision can be appropriately reduced during the deployment and reasoning process because back propagation is not required, such as reducing it to FP16 or INT8 precision. Lower data precision will reduce memory usage and latency, and make the model smaller, but the accuracy loss generally does not exceed 5%.

[0054] Through the above processing of the model file, the consumption of computing resources can be reduced, making the entire target detection model structure smaller, the amount of calculation smaller, and the processing speed more efficient.

[0055] Embodiment 2

[0056] In one or more embodiments, a substation multi-channel video real-time target detection system is disclosed, comprising:

[0057] A data acquisition module, used to acquire multi-channel video stream data of the substation to be detected;

[0058] The hardware environment judgment module is used to call the hardware environment parameter interface, obtain the current hardware environment parameters, and judge the current hardware environment;

[0059] The data decoding module is used to call the interface function under the relevant hardware environment based on the current hardware environment to decode the acquired multi-channel video stream data;

[0060] The target detection module is used to merge the decoded multi-channel video streams, and then input them into the trained target detection model to output the target detection results of each channel.

[0061] It should be noted that the specific implementation process of each of the above modules is implemented using the method disclosed in Example 1 and will not be described in detail.

[0062] Embodiment 3

[0063] In one or more embodiments, a terminal device is disclosed, including a server, the server including a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor implements the multi-channel video real-time target detection method of the substation in Example 1 when executing the program. For the sake of brevity, it will not be described here.

[0064] It should be understood that in this embodiment, the processor may be a central processing unit CPU, and the processor may also be other general-purpose processors, digital signal processors DSP, application-specific integrated circuits ASIC, off-the-shelf programmable gate arrays FPGA or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0065] The memory may include a read-only memory and a random access memory, and provide instructions and data to the processor. A portion of the memory may also include a non-volatile random access memory. For example, the memory may also store information about the device type.

[0066] In the implementation process, each step of the above method can be completed by an integrated logic circuit of hardware in a processor or an instruction in the form of software.

[0067] Embodiment 4

[0068] In one or more embodiments, a computer-readable storage medium is disclosed, in which a plurality of instructions are stored, wherein the instructions are suitable for being loaded by a processor of a terminal device and executing the method for real-time target detection in multi-channel videos of a substation described in Embodiment 1.

[0069] Although the above describes the specific implementation mode of the present invention in conjunction with the accompanying drawings, it is not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art on the basis of the technical solution of the present invention without creative work are still within the scope of protection of the present invention.

Claims

1. A method for real-time target detection of multi-channel videos in a substation, characterized in that: include: Obtain multi-channel video stream data of the substation to be inspected; Call the hardware environment parameter interface to obtain the current hardware environment parameters and determine the current hardware environment; The current hardware environment includes a CPU environment or a GPU environment; After determining the current hardware environment, the method further includes: configuring the detection target of each video stream based on the parameter configuration interface under the current hardware environment; Based on the current hardware environment, call the interface function under the relevant hardware environment to decode the acquired multi-channel video stream data; The decoded multi-channel video streams are merged and then input into the trained object detection model to output the object detection results of each channel.

2. A method for real-time target detection of multiple video channels in a substation according to claim 1, characterized in that: Based on the detection targets configured in each channel, the detection results of each channel of the target detection model are screened, and only the detection results of the detection targets configured in each channel are retained and output.

3. A method for real-time target detection of multiple-channel videos in a substation as claimed in claim 1, characterized in that: The target detection model selects a deep neural network model and compresses the target detection model based on a model compression method under the current hardware environment.

4. A method for real-time target detection of multiple video channels in a substation as claimed in claim 3, characterized in that: The process of compressing the target detection model includes: inter-layer fusion or tensor fusion, wherein the inter-layer fusion includes horizontal merging and vertical merging.

5. A method for real-time target detection of multiple video channels in a substation as claimed in claim 1, characterized in that: The decoding and merging of multi-channel video stream data, the processing of the target detection model, and the output of the detection results of each video stream are all implemented based on the gstreamer framework.

6. A multi-channel video real-time target detection system for a substation, characterized in that: include: A data acquisition module, used to acquire multi-channel video stream data of the substation to be detected; The hardware environment judgment module is used to call the hardware environment parameter interface, obtain the current hardware environment parameters, and judge the current hardware environment; The current hardware environment includes a CPU environment or a GPU environment; After determining the current hardware environment, the method further includes: configuring the detection target of each video stream based on the parameter configuration interface under the current hardware environment; The data decoding module is used to call the interface function under the relevant hardware environment based on the current hardware environment to decode the acquired multi-channel video stream data; The target detection module is used to merge the decoded multi-channel video streams, and then input them into the trained target detection model to output the target detection results of each channel.

7. A terminal device, comprising a processor and a memory, wherein the processor is used to implement each instruction; and the memory is used to store multiple instructions, characterized in that: The instructions are suitable for being loaded by a processor and executing the method for real-time target detection of multiple-channel videos in a substation according to any one of claims 1 to 5.

8. A computer-readable storage medium storing a plurality of instructions, characterized in that: The instructions are suitable for being loaded by a processor of a terminal device and executing the method for real-time target detection of multiple-channel videos of a substation according to any one of claims 1 to 5.

Images