An edge detection method and system based on the ONVIF standard

Abstract

The application discloses an edge terminal detection method and system based on an ONVIF standard. The edge terminal detection system based on the ONVIF standard comprises an ONVIF interface module, a real-time video transmission module, an image processing module and an AI model reasoning module. The real-time video transmission module encapsulates a video stream in an H.264 format obtained from a camera through an RTP protocol, and controls real-time transmission of the video stream and its accompanying information through an RTCP protocol and an RTSP protocol; the video stream and related control information are encapsulated using the ONVIF standard, so that multi-ONVIF-standard manufacturer equipment docking and device control are realized. The edge terminal detection method and system based on the ONVIF standard have the beneficial effects that the camera video stream transmission method based on the ONVIF standard can realize efficient management of multiple cameras by an edge terminal device and video stream acquisition, and improves device compatibility and real-time data transmission.

Description

TECHNICAL FIELD

[0001] The application belongs to the field of computer vision, deep learning and distributed computing technology, and particularly relates to an edge detection method and system based on an ONVIF standard. BACKGROUND

[0002] A motion target detection method and device are disclosed in patent application CN101266689A, the IPC classification numbers of which are G06T7 / 20 and G06K9 / 00, and the technical solution of which discloses that:

[0003] 1. Probability density value calculation: the pixel values of the pixel points in the previous several frames of images of the current input image are used to calculate the probability density values of the pixel points of the current input image.

[0004] 2. Threshold comparison: the calculated probability density values are compared with a pre-set threshold to determine the motion target image on the current input image.

[0005] 3. Difference detection: including time difference detection and / or background difference detection to improve the accuracy of motion target detection.

[0006] Taking the technical solution disclosed in the above patent application as an example, with the development of deep learning technology, computer vision has been widely applied in various fields. In the field of regional biological health monitoring and detection, full-time domain monitoring and management of pig health status can realize intelligent pig breeding, reduce breeding economic cost and labor management cost.

[0007] At present, the traditional method has defects such as long time consumption and excessive consumption of computing resources due to limited device computing power, causing problems such as lag of monitoring video stream. In addition, the traditional method also has problems such as slow model inference speed and low pig detection accuracy. SUMMARY

[0008] The present application overcomes the above defects and provides an edge detection method and system based on an ONVIF standard.

[0009] The edge detection method and system based on the ONVIF standard disclosed in the present patent application mainly aims to solve the problems of unsmooth pig health status detection video stream output, slow model inference speed and low detection accuracy.

[0010] Another purpose of the edge detection method and system based on the ONVIF standard disclosed in the present patent application is to use an edge computing device based on an RK3588 chip, which is equipped with an NPU, to meet the needs of stable video stream transmission and AI model inference acceleration.

[0011] The application discloses an edge detection method and system based on an ONVIF standard, and another object is to design a video stream reading and image processing method to improve the resolution and frame rate of video stream output.

[0012] The application discloses an edge detection method and system based on an ONVIF standard, and another object is to design a corresponding target detection algorithm for the health state of pigs and deploy the algorithm on an edge computing device for inference.

[0013] The application adopts the following technical scheme, an edge detection system based on an ONVIF standard, comprising an ONVIF interface module, a real-time video transmission module, an image processing module and an AI model inference module, wherein:

[0014] The ONVIF interface module parses a WSDL document of the ONVIF standard by using a gSOAP tool to generate a code library and a service interface framework, the ONVIF interface module comprises a device discovery interface, a device management interface and a media service interface, and the ONVIF interface module performs device IP search, device information management and video stream address configuration and acquisition in the same network segment;

[0015] The real-time video transmission module encapsulates a video stream in an H.264 format acquired from a camera by using an RTP protocol, controls real-time transmission of the video stream and the accompanying information by using an RTCP protocol and an RTSP protocol, and encapsulates the video stream and related control information by using the ONVIF standard;

[0016] The image processing module acquires the video stream according to a corresponding camera RTSP address by using an FFmpeg tool, initializes rmpp parameters, sets a video stream buffer and initializes decoder parameters, decodes the acquired H.264 raw video stream, performs callback after decoding, allocates space to output and save decoded data in a YUV format, and trims the image by using OpenCV.

[0017] The AI model inference module pre-processes collected pig data, makes the pig data into data sets respectively, loads the data sets into respective Yolov5 networks for training respectively, converts a pt model obtained by training into an onnx model, and then converts the onnx model into an rknn model by using an RKNN-Toolkit2 tool, so as to facilitate subsequent inference; the rknn model is deployed on an edge computing device for inference: an rknn library is used to initialize inference context and the rknn model, the rknn model is called for inference, an image frame is superimposed, and memory is released after inference is completed; in system scheduling design, a multi-thread inference mode is adopted.

[0018] As a preferred technical solution of the above technical solution, the pig data includes face and body data for reflecting daily activity state of the pig, body temperature data for judging whether the pig is healthy, and weight data for quantifying the pig's pre-slaughter date.

[0019] As a preferred technical solution of the above technical solution, on the system scheduling design, a multi-thread reasoning mode is adopted, which is specifically implemented as follows: a plurality of threads are started to call the NPU for model initialization, and a plurality of threads are started to obtain the camera video stream; the video stream data and the reasoning result are efficiently transmitted among the sub-modules of the NPU through a message queue, so that the video stream can maintain high-accuracy reasoning and output under 1080P resolution and 60FPS.

[0020] The application adopts the following technical solution: an edge detection method based on the ONVIF standard, comprising the following steps:

[0021] Step S1: the WSDL document of the ONVIF standard is parsed by using the gSOAP tool to generate a code library and a service interface framework, the ONVIF interface module includes a device discovery interface, a device management interface and a media service interface, and the ONVIF interface module performs device IP search, device information management and video stream address configuration and acquisition in the same network segment;

[0022] Step S2: the H.264 format video stream obtained from the camera is encapsulated through the RTP protocol, the real-time transmission of the video stream and the accompanying information is controlled through the RTCP protocol and the RTSP protocol, and the video stream and the related control information are encapsulated using the ONVIF standard;

[0023] Step S3: the video stream is obtained according to the corresponding camera RTSP address by using the FFmpeg tool; the rmpp parameters are initialized, the video stream buffer is set and the decoder parameters are initialized, the obtained H.264 raw video stream is decoded; the decoding completion callback is performed, the space for the decoded data is allocated, and the decoded data is saved in YUV format; the image is cropped through OpenCV;

[0024] Step S4: the collected pig data is preprocessed; the pig data is made into a data set respectively, and the data set is loaded into each Yolov5 network for training; the pt model obtained by training is converted into an onnx model, and then converted into an rknn model through the RKNN-Toolkit2 tool, so as to facilitate subsequent reasoning; the rknn model is deployed on the edge computing device for reasoning: the rknn library is used to initialize the reasoning context and the rknn model; the rknn model is called for reasoning, the image frame is superimposed, and the memory is released after the reasoning is completed; on the system scheduling design, a multi-thread reasoning mode is adopted.

[0025] As a preferred technical solution of the above technical solution, in step S4, the pig data includes face and body data for reflecting the daily activity state of the pig, body temperature data for judging whether the pig is healthy, and weight data for quantifying the pig's expected market date.

[0026] As a preferred technical solution of the above technical solution, in step S4, the "multi-thread reasoning mode is adopted in system scheduling design" is specifically implemented as the following steps: multiple threads are started to call NPU for model initialization, and multiple threads are started to obtain camera video stream; the efficient transmission of video stream data and inference results is realized through message queues in each sub-module of NPU, so that the video stream maintains high accuracy of inference and output under 1080P resolution and 60FPS.

[0027] The edge detection method and system based on the ONVIF standard disclosed by the application have the following beneficial effects:

[0028] 1. The camera video stream transmission method based on the ONVIF standard can realize efficient management and video stream acquisition of multiple cameras by the edge device, and improves the device compatibility and real-time data transmission.

[0029] 2. The image processing method of the edge computing device processes the obtained original video stream by using FFmpeg and Opencv, which can improve the video stream encoding and decoding speed and the subsequent inference speed of the video stream.

[0030] 3. The daily activity data, health status and expected market date of the pig are fused to form a standardized pig management system, which can realize intelligent breeding and reduce breeding and management costs.

[0031] 4. On the edge computing device, a multi-thread mode is adopted, and the transmission of video data and inference results is realized through message queues in each sub-module of NPU, which can realize efficient data transmission and synchronization, and multiple video streams maintain high accuracy of inference and output. BRIEF DESCRIPTION OF DRAWINGS

[0032] Fig. 1 is a flowchart of the application.

[0033] Fig. 2 is a schematic diagram of calling NPU inference acceleration of the application. DETAILED DESCRIPTION

[0034] The application discloses an edge detection method and system based on an ONVIF standard, and the following is a preferred embodiment (Embodiment 1) combined with the drawings Figs. 1-2 , the specific implementation of the application is further described.

[0035] Referring to the drawingsFigs. 1-2 , Fig. 1 The main steps of the edge detection method based on the ONVIF standard are shown, Fig. 2 The working principle of calling the NPU and inference acceleration is shown.

[0036] Embodiment 1.

[0037] Preferably, the edge detection system based on the ONVIF standard comprises an ONVIF interface module, a real-time video transmission module, an image processing module, and an AI model inference module, wherein:

[0038] The ONVIF interface module parses the WSDL document of the ONVIF standard using the gSOAP tool to generate a code library and a service interface framework. The ONVIF interface module includes a device discovery interface, a device management interface, and a media service interface. The ONVIF interface module performs device IP search (for searching for cameras and distributed computing devices within a local area network), device information management (for managing all camera monitoring devices within a local area network), and video stream address configuration and acquisition (for facilitating video stream acquisition) within the same network segment, thereby meeting compatibility requirements.

[0039] The real-time video transmission module encapsulates the H.264 format video stream obtained from the camera through the RTP protocol, controls the real-time transmission of the video stream and its accompanying information (including camera ptz control data) through the RTCP protocol and the RTSP protocol, encapsulates the video stream and related control information using the ONVIF standard, and realizes the connection and device control of multiple ONVIF standard manufacturer devices.

[0040] The image processing module uses the FFmpeg tool to obtain the video stream according to the corresponding camera RTSP address, initializes the rmpp parameters, sets the video stream buffer and initializes the decoder parameters, decodes the obtained H.264 raw video stream, calls back after decoding, allocates space for the decoded data, outputs and saves the decoded data in YUV format, and crops the image through OpenCV.

[0041] The AI model inference module preprocesses the collected pig data, creates data sets from the pig data, loads the data sets into each Yolov5 network for training, converts the trained pt model into an onnx model, and then converts it into an rknn model through the RKNN-Toolkit2 tool for subsequent inference. The rknn model is deployed on the edge computing device for inference: the rknn library is used to initialize the inference context and the rknn model, the rknn model is called for inference, the image frame is superimposed, and the memory is released after inference. In the system scheduling design, a multi-threaded inference method is adopted.

[0042] Among them, the pig data includes face and body data for reflecting the daily activity state of the pig, body temperature data for judging whether the pig is healthy, and weight data for quantifying the pig pre-finish date.

[0043] Among them, "on the system scheduling design, using multi-thread reasoning mode" is implemented as follows: starting multiple threads to call NPU for model initialization, and starting multiple threads to obtain camera video stream; through the message queue, the efficient transmission of video stream data and inference results is realized in each sub-module of NPU, so that the video stream can maintain high accuracy of reasoning and output under 1080P resolution and 60FPS.

[0044] Preferably, the edge detection method based on the ONVIF standard includes the following steps:

[0045] Step S1: parse the WSDL document of the ONVIF standard using the gSOAP tool to generate a code library and a service interface framework, the ONVIF interface module includes an implementation device discovery interface, a device management interface and a media service interface, the ONVIF interface module searches for device IP in the same network segment (for searching for cameras and distributed computing devices in the local area network), manages device information (for managing all camera monitoring devices in the local area network), and configures and obtains video stream address (for obtaining video stream), thereby meeting the compatibility requirements;

[0046] Step S2: encapsulate the H.264 format video stream obtained from the camera through the RTP protocol, control the real-time transmission of the video stream and its accompanying information (including camera ptz control parameters) through the RTCP protocol and the RTSP protocol; encapsulate the video stream and related control information using the ONVIF standard to realize the connection and control of devices of multiple ONVIF standard manufacturers;

[0047] Step S3: use the FFmpeg tool to obtain the video stream according to the corresponding camera RTSP address; initialize the rmpp parameter, set the video stream buffer and initialize the decoder parameter, and decode the obtained H.264 raw video stream; decode callback, allocate space for the decoded data, and output and save the decoded data in YUV format; crop the image through OpenCV;

[0048] Step S4: preprocessing the collected pig data; making the pig data into data sets respectively, and loading the data sets into each Yolov5 network for training respectively; converting the pt model obtained by training into an onnx model, and then converting the onnx model into an rknn model through an RKNN-Toolkit2 tool, so as to facilitate subsequent inference; deploying the rknn model on an edge computing device for inference: using an rknn library to initialize inference context and the rknn model; calling the rknn model for inference, superimposing an image frame, releasing memory after inference is completed; and using a multi-thread inference mode in system scheduling design.

[0049] In step S4, the pig data includes face and body data for reflecting the daily activity state of the pig, body temperature data for judging whether the pig is healthy, and body weight data for quantifying the pig pre-slaughter date.

[0050] In step S4, the multi-thread inference mode in system scheduling design is implemented as follows: multiple threads are started to call the NPU for model initialization, and multiple threads are started to obtain the camera video stream; the efficient transmission of the video stream data and the inference result is realized through a message queue in each sub-module of the NPU, so that the video stream can be kept at a high accuracy of inference and output under the condition of 1080P resolution and 60FPS.

[0051] It is worth mentioning that the technical features such as the FFmpeg tool involved in the present patent application should be regarded as prior art, and the specific structure, working principle and possible control mode and spatial arrangement mode of these technical features can be selected by using the conventional selection in the field, and should not be regarded as the invention point of the present patent, and the present patent will not be further expanded and detailed.

[0052] For those skilled in the art, the technical solutions recorded in the foregoing embodiments can still be modified, or some technical features can be replaced equivalently, and any modification, equivalent replacement, improvement, etc. within the spirit and principles of the present application should be included in the protection scope of the present application.

Claims

1. An edge detection system based on the ONVIF standard, characterized in that, It includes an ONVIF interface module, a real-time video transmission module, an image processing module, and an AI model inference module, among which: The ONVIF interface module uses the gSOAP tool to parse the ONVIF standard WSDL document to generate a code library and service interface framework. The ONVIF interface module includes a device discovery interface, a device management interface, and a media service interface. The ONVIF interface module performs device IP search, device information management, and video stream address configuration and acquisition within the same network segment. The real-time video transmission module encapsulates the H.264 format video stream acquired from the camera using the RTP protocol, and controls the real-time transmission of the video stream and its accompanying information using the RTCP and RTSP protocols; it also uses the ONVIF standard to encapsulate the video stream and related control information. The image processing module uses FFmpeg to obtain the video stream based on the corresponding camera RTSP address; initializes rmpp parameters, sets the video stream buffer and initializes decoder parameters, and decodes the obtained H.264 raw video stream; a decoding completion callback is executed, which allocates space for the decoded data and outputs it as decoded data in YUV format; and the image is cropped using OpenCV. The AI ​​model inference module preprocesses the collected pig data; it creates datasets from the pig data and loads each dataset into a Yolov5 network for training; it converts the trained PT model into an ONNX model, and then into a RKNN model using the RKNN-Toolkit2 tool for easier inference; it deploys the RKNN model on edge computing devices for inference: using the RKNN library, it initializes the inference context and the RKNN model; it calls the RKNN model for inference, overlays image bounding boxes, and releases memory after inference is complete; in the system scheduling design, a multi-threaded inference approach is adopted. "In the system scheduling design, a multi-threaded inference approach is adopted" is specifically implemented in the following steps: multiple threads are started to call the NPU to initialize the model, and multiple threads are started to acquire camera video streams; efficient transmission of video stream data and inference results is achieved in various sub-modules of the NPU through message queues.

2. The edge detection system based on the ONVIF standard according to claim 1, characterized in that, Pig data includes facial and body posture data to reflect the daily activity status of pigs, body temperature data to determine the health status of pigs, and weight data to quantify the expected slaughter date of pigs.

3. An edge detection method based on the ONVIF standard, characterized in that, Includes the following steps: Step S1: Use the gSOAP tool to parse the WSDL document of the ONVIF standard to generate a code library and service interface framework. The ONVIF interface module includes a device discovery interface, a device management interface, and a media service interface. The ONVIF interface module performs device IP search, device information management, and video stream address configuration and acquisition within the same network segment. Step S2: Encapsulate the H.264 format video stream obtained from the camera using the RTP protocol; control the real-time transmission of the video stream and its accompanying information using the RTCP and RTSP protocols; encapsulate the video stream and related control information using the ONVIF standard. Step S3: Use FFmpeg to obtain the video stream based on the corresponding camera RTSP address; initialize rmpp parameters, set the video stream buffer and initialize decoder parameters, and decode the obtained H.264 raw video stream; Upon completion of decoding, a callback is triggered to allocate space for the decoded data and output it as YUV format. The image is then cropped using OpenCV. Step S4: Preprocess the collected pig data; The pig data was compiled into datasets and loaded into various Yolov5 networks for training. The trained PT models were converted into ONNX models, and then into RKNN models using the RKNN-Toolkit2 tool for easier inference. The RKNN models were deployed on edge computing devices for inference: the RKNN library was used to initialize the inference context and the RKNN model; the RKNN model was called for inference, image boxes were overlaid, and memory was released after inference was completed; a multi-threaded inference approach was adopted in the system scheduling design; the "multi-threaded inference approach" was implemented in the following steps: multiple threads were started to call the NPU for model initialization, and multiple threads were started to acquire camera video streams; efficient transmission of video stream data and inference results was achieved in various sub-modules of the NPU through message queues.

4. The edge detection method based on the ONVIF standard according to claim 3, characterized in that, In step S4, the pig data includes facial and body posture data to reflect the daily activity status of pigs, body temperature data to determine whether pigs are healthy, and weight data to quantify the expected slaughter date of pigs.

Images