ONVIF standard-based edge end detection method and system

Abstract

The invention discloses an edge end detection method and system based on an ONVIF standard. The edge end detection system based on the ONVIF standard comprises an ONVIF interface module, a real-time video transmission module, an image processing module and an AI model reasoning module. The real-time video transmission module is used for packaging a video stream in an H.264 format obtained from a camera through an RTP (Real-time Transport Protocol), and controlling real-time transmission of the video stream and attached information thereof through an RTCP (Real-time Transmission Control Protocol) and an RTSP (Real-time Transport Protocol); and packaging a video stream and related control information by using the ONVIF standard to realize manufacturer equipment docking and equipment control of multiple ONVIF standards. The edge end detection method and the edge end detection system based on the ONVIF standard disclosed by the invention have the beneficial effects that by adopting a camera video stream transmission method based on the ONVIF standard, efficient management and video stream acquisition of multiple cameras by edge end equipment can be realized, and the equipment compatibility and the real-time performance of data transmission are improved.

Description

Technical Field

[0001] The present invention belongs to the fields of computer vision, deep learning, and distributed computing technology, and specifically relates to an edge detection method and system based on the ONVIF standard. Background Art

[0002] The publication number is CN101266689A, the subject name is a moving target detection method and device, and its IPC classification number is G06T7 / 20, G06K9 / 00. Its technical solution discloses:

[0003] 1. Probability density value calculation: Calculate the probability density value of the pixel points of the current input image using the pixel values ​​of the previous several frames of the current input image.

[0004] 2. Threshold comparison: Compare the calculated probability density value with the pre-set threshold to determine the moving target image on the current input image.

[0005] 3. Differential detection: including time differential detection and / or background differential detection to improve the accuracy of moving target detection.

[0006] Taking the above invention patent application as an example, with the development of deep learning technology, computer vision has been widely used in various fields. In the field of regional biological health monitoring and detection, the full-time monitoring and management of the health status of pigs can realize the intelligentization of pig breeding and reduce the economic cost and human management cost of breeding.

[0007] At present, the traditional method has the defects of long time consumption and excessive consumption of computing resources due to the limited computing power of the equipment, which causes problems such as freezing of the monitoring video stream. In addition, the traditional method also has problems such as slow model inference speed and low pig detection accuracy. Summary of the invention

[0008] In view of the prior art, the present invention overcomes the above defects and provides an edge detection method and system based on the ONVIF standard.

[0009] The edge detection method and system based on the ONVIF standard disclosed in the patent application of the present invention mainly aims to solve the problems of unsmooth video stream output of pig health status detection, slow model inference speed and low detection accuracy.

[0010] Another purpose of the edge detection method and system based on the ONVIF standard disclosed in the patent application of the present invention is to use an edge computing device based on the RK3588 chip, equipped with an NPU, to meet the needs of stable video stream transmission, AI model reasoning acceleration, etc.

[0011] Another purpose of the edge detection method and system based on the ONVIF standard disclosed in the patent application of the present invention is to design a video stream reading and image processing method to improve the resolution and frame rate of the video stream output.

[0012] Another purpose of the edge detection method and system based on the ONVIF standard disclosed in the patent application of the present invention is to design corresponding target detection algorithms according to the health status of pigs and deploy them on edge computing devices for reasoning.

[0013] The present invention adopts the following technical scheme, an edge detection system based on the ONVIF standard, including an ONVIF interface module, a real-time video transmission module, an image processing module and an AI model reasoning module, wherein:

[0014] ONVIF interface module uses gSOAP tool to parse the WSDL document of ONVIF standard to generate code base and service interface framework. ONVIF interface module includes device discovery interface, device management interface and media service interface. ONVIF interface module performs device IP search, device information management and video stream address configuration and acquisition in the same network segment;

[0015] The real-time video transmission module encapsulates the H.264 video stream obtained from the camera through the RTP protocol, controls the real-time transmission of the video stream and its accompanying information through the RTCP protocol and the RTSP protocol; and uses the ONVIF standard to encapsulate the video stream and related control information;

[0016] The image processing module uses the FFmpeg tool to obtain the video stream according to the corresponding camera RTSP address; initializes the rmpp parameters, sets the video stream buffer and initializes the decoder parameters, and decodes the obtained H.264 original video stream; the decoding is completed and the callback is called to allocate space for the decoded data and output the decoded data saved in YUV format; the image is cropped through OpenCV;

[0017] The AI ​​model inference module pre-processes the collected pig data; the above pig data are made into data sets respectively, and the data sets are loaded into each Yolov5 network for training respectively; the trained pt model is converted into an onnx model, and then converted into an rknn model through the RKNN-Toolkit2 tool to facilitate subsequent reasoning; the rknn model is deployed on the edge computing device for reasoning: the rknn library is used to initialize the reasoning context and the rknn model; the rknn model is called for reasoning, the image frame is superimposed, and the memory is released after the reasoning is completed; in the system scheduling design, a multi-threaded reasoning method is adopted.

[0018] As a preferred technical solution of the above technical solutions, the pig data includes facial and body shape data for reflecting the daily activity status of the pigs, body temperature data for judging whether the pigs are healthy, and weight data for quantifying the expected market release date of the pigs.

[0019] As the preferred technical solution of the above technical solutions, "adopting multi-threaded reasoning method in system scheduling design" is specifically implemented as the following steps: start multiple threads to call NPU for model initialization, and start multiple threads to obtain camera video stream; realize efficient transmission of video stream data and reasoning results in various sub-modules of NPU through message queues, so that the video stream can maintain high-accuracy reasoning and output at 1080P resolution and 60FPS.

[0020] The present invention adopts the following technical scheme, and the edge detection method based on the ONVIF standard includes the following steps:

[0021] Step S1: Use the gSOAP tool to parse the WSDL document of the ONVIF standard to generate a code base and a service interface framework. The ONVIF interface module includes a device discovery interface, a device management interface, and a media service interface. The ONVIF interface module performs device IP search, device information management, and video stream address configuration and acquisition within the same network segment;

[0022] Step S2: Encapsulate the H.264 video stream obtained from the camera through the RTP protocol, control the real-time transmission of the video stream and its accompanying information through the RTCP protocol and the RTSP protocol; use the ONVIF standard to encapsulate the video stream and related control information;

[0023] Step S3: Use FFmpeg tool to obtain video stream according to the corresponding camera RTSP address; initialize rmpp parameters, set video stream buffer and initialize decoder parameters, decode the acquired H.264 original video stream; callback after decoding, allocate space for decoded data and output the decoded data saved in YUV format; crop the image through OpenCV;

[0024] Step S4: pre-process the collected pig data; make the above pig data into data sets respectively, and load the data sets into each Yolov5 network for training; convert the trained pt model into an onnx model, and then convert it into a rknn model through the RKNN-Toolkit2 tool to facilitate subsequent reasoning; deploy the rknn model on the edge computing device for reasoning: use the rknn library to initialize the reasoning context and rknn model; call the rknn model for reasoning, overlay the image frame, and release the memory after the reasoning is completed; in the system scheduling design, adopt a multi-threaded reasoning method.

[0025] As a preferred technical solution of the above technical solutions, in step S4, the pig data includes facial and body shape data for reflecting the daily activity status of the pigs, body temperature data for judging whether the pigs are healthy, and weight data for quantifying the expected market release date of the pigs.

[0026] As the preferred technical solution of the above technical solutions, in step S4, "in system scheduling design, adopting multi-threaded reasoning method" is specifically implemented as the following steps: start multiple threads to call NPU for model initialization, and start multiple threads to obtain camera video stream; through the message queue, realize efficient transmission of video stream data and reasoning results in each sub-module of NPU, so that the video stream can maintain high-accuracy reasoning and output at 1080P resolution and 60FPS.

[0027] The edge detection method and system based on ONVIF standard disclosed in the present invention have the following beneficial effects:

[0028] 1. The camera video stream transmission method based on the ONVIF standard can realize efficient management of multiple cameras and video stream acquisition by edge devices, improving device compatibility and real-time data transmission.

[0029] 2. The image processing method of the edge computing device uses FFmpeg and Opencv to process the acquired original video stream, which can improve the speed of video stream encoding and decoding and the subsequent reasoning speed of the video stream.

[0030] 3. By integrating data such as pigs' daily activity data, health status, and expected market date to form a standardized pig management system, intelligent breeding can be achieved and breeding and management costs can be reduced.

[0031] 4. On the edge computing device, a multi-threaded approach is adopted to transmit video data and inference results on various sub-modules of the NPU through message queues, which can achieve efficient data transmission and synchronization, and enable multiple video streams to maintain high-accuracy inference and output. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 It is a flow chart of the present invention.

[0033] Figure 2 It is a schematic diagram of calling NPU inference acceleration of the present invention. DETAILED DESCRIPTION

[0034] The present invention discloses an edge detection method and system based on ONVIF standard. Figure 1-2 , the specific implementation modes of the present invention are further described.

[0035] See attached figure Figure 1 to Figure 2 , Figure 1 The main steps of the edge detection method based on the ONVIF standard are shown. Figure 2 It shows the working principle of calling NPU and inference acceleration.

[0036] Example 1.

[0037] Preferably, the edge detection system based on the ONVIF standard includes an ONVIF interface module, a real-time video transmission module, an image processing module and an AI model reasoning module, wherein:

[0038] The ONVIF interface module uses the gSOAP tool to parse the WSDL document of the ONVIF standard to generate a code base and a service interface framework. The ONVIF interface module includes a device discovery interface, a device management interface, and a media service interface. The ONVIF interface module performs device IP search (used to search for cameras and distributed computing devices in the LAN), device information management (used to manage all camera monitoring devices in the LAN), and video stream address configuration and acquisition (to facilitate the acquisition of video streams) within the same network segment to meet compatibility requirements;

[0039] The real-time video transmission module uses the RTP protocol to encapsulate the H.264 video stream obtained from the camera, and controls the real-time transmission of the video stream and its accompanying information (including camera ptz control data) through the RTCP protocol and RTSP protocol; uses the ONVIF standard to encapsulate the video stream and related control information to achieve multi-ONVIF standard manufacturer equipment docking and equipment control;

[0040] The image processing module uses the FFmpeg tool to obtain the video stream according to the corresponding camera RTSP address; initializes the rmpp parameters, sets the video stream buffer and initializes the decoder parameters, and decodes the obtained H.264 original video stream; the decoding is completed and the callback is called to allocate space for the decoded data and output the decoded data saved in YUV format; the image is cropped through OpenCV;

[0041] The AI ​​model inference module pre-processes the collected pig data; the above pig data are made into data sets respectively, and the data sets are loaded into each Yolov5 network for training respectively; the trained pt model is converted into an onnx model, and then converted into an rknn model through the RKNN-Toolkit2 tool to facilitate subsequent reasoning; the rknn model is deployed on the edge computing device for reasoning: the rknn library is used to initialize the reasoning context and the rknn model; the rknn model is called for reasoning, the image frame is superimposed, and the memory is released after the reasoning is completed; in the system scheduling design, a multi-threaded reasoning method is adopted.

[0042] Among them, pig data include facial and body data used to reflect the daily activity status of pigs, body temperature data used to determine whether the pigs are healthy, and weight data used to quantify the expected date of slaughter of pigs.

[0043] Among them, "using multi-threaded reasoning method in system scheduling design" is specifically implemented as the following steps: start multiple threads to call NPU for model initialization, and start multiple threads to obtain camera video streams; through message queues, realize efficient transmission of video stream data and reasoning results in various sub-modules of NPU, so that the video stream can maintain high-accuracy reasoning and output at 1080P resolution and 60FPS.

[0044] Preferably, the edge detection method based on the ONVIF standard comprises the following steps:

[0045] Step S1: Use the gSOAP tool to parse the WSDL document of the ONVIF standard to generate a code base and a service interface framework. The ONVIF interface module includes implementing a device discovery interface, a device management interface, and a media service interface. The ONVIF interface module performs device IP search (for searching cameras and distributed computing devices in a local area network), device information management (for managing all camera monitoring devices in a local area network), and video stream address configuration and acquisition (for easy acquisition of video streams) in the same network segment to meet compatibility requirements;

[0046] Step S2: Encapsulate the H.264 video stream obtained from the camera through the RTP protocol, and control the real-time transmission of the video stream and its accompanying information (including parameters such as camera ptz control) through the RTCP protocol and the RTSP protocol; use the ONVIF standard to encapsulate the video stream and related control information to achieve multi-ONVIF standard manufacturer equipment docking and equipment control;

[0047] Step S3: Use FFmpeg tool to obtain video stream according to the corresponding camera RTSP address; initialize rmpp parameters, set video stream buffer and initialize decoder parameters, decode the acquired H.264 original video stream; callback after decoding, allocate space for decoded data and output the decoded data saved in YUV format; crop the image through OpenCV;

[0048] Step S4: pre-process the collected pig data; make the above pig data into data sets respectively, and load the data sets into each Yolov5 network for training; convert the trained pt model into an onnx model, and then convert it into a rknn model through the RKNN-Toolkit2 tool to facilitate subsequent reasoning; deploy the rknn model on the edge computing device for reasoning: use the rknn library to initialize the reasoning context and rknn model; call the rknn model for reasoning, overlay the image frame, and release the memory after the reasoning is completed; in the system scheduling design, adopt a multi-threaded reasoning method.

[0049] Among them, in step S4, the pig data includes facial and body data for reflecting the daily activity status of the pigs, body temperature data for judging whether the pigs are healthy, and weight data for quantifying the expected release date of the pigs.

[0050] Among them, in step S4, "in system scheduling design, adopt multi-threaded reasoning method" is specifically implemented as the following steps: start multiple threads to call NPU for model initialization, and start multiple threads to obtain camera video stream; through the message queue, realize efficient transmission of video stream data and reasoning results in each sub-module of NPU, so that the video stream can maintain high-accuracy reasoning and output at 1080P resolution and 60FPS.

[0051] It is worth mentioning that the technical features such as FFmpeg tools involved in the patent application of this invention should be regarded as prior art. The specific structure, working principle and possible control method and spatial layout method of these technical features can be selected according to the conventional selection in the field, and should not be regarded as the inventive point of the patent of this invention. The patent of this invention will not be further elaborated.

[0052] For those skilled in the art, it is still possible to modify the technical solutions described in the aforementioned embodiments, or to make equivalent substitutions for some of the technical features therein. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. An edge detection system based on ONVIF standard, characterized in that: It includes ONVIF interface module, real-time video transmission module, image processing module and AI model inference module, among which: ONVIF interface module uses gSOAP tool to parse the WSDL document of ONVIF standard to generate code base and service interface framework. ONVIF interface module includes device discovery interface, device management interface and media service interface. ONVIF interface module performs device IP search, device information management and video stream address configuration and acquisition in the same network segment; The real-time video transmission module encapsulates the H.264 video stream obtained from the camera through the RTP protocol, controls the real-time transmission of the video stream and its accompanying information through the RTCP protocol and the RTSP protocol; and uses the ONVIF standard to encapsulate the video stream and related control information; The image processing module uses the FFmpeg tool to obtain the video stream according to the corresponding camera RTSP address; initializes the rmpp parameters, sets the video stream buffer and initializes the decoder parameters, and decodes the obtained H.264 original video stream; the decoding is completed and the callback is called to allocate space for the decoded data and output the decoded data saved in YUV format; the image is cropped through OpenCV; The AI ​​model inference module pre-processes the collected pig data; the above pig data are made into data sets respectively, and the data sets are loaded into each Yolov5 network for training respectively; the trained pt model is converted into an onnx model, and then converted into an rknn model through the RKNN-Toolkit2 tool to facilitate subsequent reasoning; the rknn model is deployed on the edge computing device for reasoning: the rknn library is used to initialize the reasoning context and the rknn model; the rknn model is called for reasoning, the image frame is superimposed, and the memory is released after the reasoning is completed; in the system scheduling design, a multi-threaded reasoning method is adopted.

2. The edge detection system based on the ONVIF standard according to claim 1, characterized in that: Pig data includes facial and body data to reflect the daily activity status of pigs, body temperature data to determine whether the pigs are healthy, and weight data to quantify the expected market date of the pigs.

3. The edge detection system based on the ONVIF standard according to claim 1, characterized in that: "In system scheduling design, multi-threaded reasoning is adopted" is specifically implemented as follows: start multiple threads to call NPU for model initialization, and start multiple threads to obtain camera video streams; through message queues, efficient transmission of video stream data and reasoning results is achieved in various sub-modules of NPU, so that the video stream can maintain high-accuracy reasoning and output at 1080P resolution and 60FPS.

4. An edge detection method based on ONVIF standard, characterized in that: The following steps are involved: Step S1: Use the gSOAP tool to parse the WSDL document of the ONVIF standard to generate a code base and a service interface framework. The ONVIF interface module includes a device discovery interface, a device management interface, and a media service interface. The ONVIF interface module performs device IP search, device information management, and video stream address configuration and acquisition within the same network segment; Step S2: Encapsulate the H.264 video stream obtained from the camera through the RTP protocol, control the real-time transmission of the video stream and its accompanying information through the RTCP protocol and the RTSP protocol; use the ONVIF standard to encapsulate the video stream and related control information; Step S3: Use the FFmpeg tool to obtain the video stream according to the corresponding camera RTSP address; initialize the rmpp parameters, set the video stream buffer and initialize the decoder parameters, and decode the obtained H.264 original video stream; The decoding is completed and the callback is called to allocate space for the decoded data and output the decoded data saved in YUV format; the image is cropped through OpenCV; Step S4: preprocessing the collected pig data; The above pig data are made into data sets, and the data sets are loaded into various Yolov5 networks for training; the trained pt model is converted into an onnx model, and then converted into an rknn model through the RKNN-Toolkit2 tool to facilitate subsequent reasoning; the rknn model is deployed on the edge computing device for reasoning: the rknn library is used to initialize the reasoning context and rknn model; the rknn model is called for reasoning, the image frame is superimposed, and the memory is released after the reasoning is completed; In the system scheduling design, multi-threaded reasoning method is adopted.

5. The edge detection method based on the ONVIF standard according to claim 4, characterized in that: In step S4, the pig data includes facial and body data for reflecting the daily activity status of the pigs, body temperature data for judging whether the pigs are healthy, and weight data for quantifying the expected market release date of the pigs.

6. The edge detection method based on the ONVIF standard according to claim 4, characterized in that: In step S4, "in system scheduling design, adopt multi-threaded reasoning method" is specifically implemented as the following steps: start multiple threads to call NPU for model initialization, and start multiple threads to obtain camera video streams; through message queues, realize efficient transmission of video stream data and reasoning results in various sub-modules of NPU, so that the video stream can maintain high-accuracy reasoning and output at 1080P resolution and 60FPS.

Images