Intelligent mobile monitoring device and system based on target detection

Abstract

The invention discloses an intelligent mobile monitoring device and system based on target detection. The monitoring device comprises a camera, a target detection module, a motion control module and an intelligent trolley. The camera is mounted on the intelligent trolley and is used for acquiring a video of a monitored area; the target detection module is used for carrying out target detection on a video of a monitoring area and acquiring the position of a target in a video image; and the motion control module is used for controlling a motor driving signal and a steering engine steering signal of the intelligent trolley according to the position of the target in the video image. According to the device, autonomous movement and target following can be realized, and the flexibility and the intelligent level of video monitoring are improved.

Description

Technical Field

[0001] The invention relates to an intelligent mobile monitoring device and system based on target detection, belonging to the technical field of video monitoring. Background Art

[0002] At present, video surveillance is usually achieved by installing cameras. The cameras are installed on a fixed position of the pan / tilt, and the monitoring angle is obviously limited. This limitation directly leads to a large number of monitoring blind spots in the monitoring area, making these areas unable to be effectively monitored. In order to achieve comprehensive coverage, traditional monitoring systems often need to install a large number of cameras at multiple angles and locations, and have high requirements for the installation environment, making the deployment process of the entire system extremely complicated, involving numerous installation links and a large amount of manpower investment. When it is necessary to continuously monitor a specific moving target, it is necessary to fuse and analyze the videos collected by multiple cameras within the range of motion, which is a large workload. Summary of the invention

[0003] Purpose of the invention: The technical problem to be solved by the present invention is to provide an intelligent mobile monitoring device based on target detection in view of the shortcomings of the prior art, which can realize autonomous movement and target following to improve the flexibility and intelligence level of video monitoring.

[0004] In order to solve the above technical problems, the present invention discloses an intelligent mobile monitoring device based on target detection, comprising: a camera 100, a target detection module 200, a motion control module 300, and an intelligent car 400; the camera 100 is installed on the intelligent car 400 to obtain the video of the monitoring area; the target detection module 200 is used to perform target detection on the video of the monitoring area and obtain the position of the target in the video image; the motion control module 300 is used to control the motor drive signal and the steering gear steering signal of the intelligent car 400 according to the position of the target in the video image.

[0005] Furthermore, the target detection module 200 uses the YOLOv2 model based on the Hi3516 platform to perform target detection on the video of the monitoring area to obtain a rectangular frame of the target in the video image.

[0006] Furthermore, the YOLOv2 model is trained using a pedestrian detection data set, and the prototxt model structure file and the weights model weight file are output after the training is completed; the trained YOLOv2 model is quantized using the RuyiStudio tool to obtain a wk model file, and the wk model file runs on the embedded processor end of the Hi3516 platform.

[0007] Furthermore, the intelligent mobile monitoring device also includes a video stream push module 230, which compresses and encodes the video stream collected by the camera or the video frame after target detection, and pushes the compressed and encoded video to the cloud server or PC using the RTMP protocol.

[0008] Furthermore, the video stream pushing module 230 uses H.264 to compress and encode the video frames.

[0009] Furthermore, the motion control module 300 controls the motion of the smart car, including:

[0010] Determine the relative distance and relative orientation between the target and the smart car according to the position of the target in the video image;

[0011] According to the relative distance and relative orientation between the target and the smart car, the PID algorithm is used to control the smart car motor drive PWM signal and the servo steering signal to make the target located in the center of the camera's field of view.

[0012] Furthermore, the intelligent mobile monitoring device further includes a distance measuring module 310, which is arranged on the intelligent car and is used to obtain a distance d between the front of the intelligent car and an obstacle;

[0013] When the distance d between the smart car and the obstacle in front of it is less than or equal to the obstacle avoidance threshold, the obstacle avoidance mode is triggered; the motion control module controls the movement of the smart car according to the following steps:

[0014] S1, the smart car stops moving forward, turns left and right by angle θ, and measures and obtains the distances d1 and d2 between the front of the smart car and the obstacle;

[0015] S2. If |d1-d2|<Δd, the motion control module is also used to control the smart car to turn right 90°; Δd is the preset obstacle position judgment threshold;

[0016] If |d1-d2|≥Δd and d1>d2, the motion control module is also used to control the smart car to turn left, and the left turning angle is θ L , the calculation steps are:

[0017] In the triangle formed by d, θ, and d1, the length L1 of the side opposite to θ is calculated according to the cosine theorem:

[0018]

[0019] The angle α opposite to d1 L for:

[0020] The left turning angle is θ L is:L = 180 - α L ;

[0021] If |d1 - d2| ≥ Δd and d1 < d2, the motion control module is further configured to control the intelligent vehicle to turn right, and the right turn angle is θ R , and the calculation steps are as follows:

[0022] In the triangle formed by d, θ, and d2, the length L2 of the opposite side of θ is calculated according to the cosine theorem:

[0023]

[0024] The opposite angle α of d2 R is:

[0025] The right turn angle is θ R is: θ R = 180 - α R ;

[0026] S3. Measure again to obtain the distance d between the front of the intelligent vehicle and the obstacle. If d is greater than the obstacle avoidance threshold, the obstacle avoidance mode ends, and the motion control module controls the intelligent vehicle to continue moving forward;

[0027] If d is less than or equal to the obstacle avoidance threshold, jump to S1 for steering obstacle avoidance.

[0028] Furthermore, the ranging module 310 uses ultrasonic ranging or laser ranging.

[0029] Furthermore, the intelligent mobile monitoring device further includes a status publishing and subscribing module 320. The status publishing and subscribing module publishes the real-time status information of the intelligent vehicle to the cloud server through the MQTT protocol and subscribes to the remote instructions issued by the cloud server.

[0030] On the other hand, the present invention also discloses an intelligent mobile monitoring system, including a cloud server and at least one intelligent mobile monitoring device, and the intelligent mobile monitoring device is the above-mentioned intelligent mobile monitoring device.

[0031] Advantageous effects: Compared with the prior art, the intelligent mobile monitoring device and system disclosed in the present invention have the following advantageous effects:

[0032] 1. Detect and follow the target in real time, and can dynamically adjust the shooting angle to achieve blind area-free monitoring;

[0033] 2. Connect to the cloud server using the MQTT protocol and push the video stream using the RTMP protocol, thereby supporting real-time data synchronization and remote control to achieve intelligent management;

[0034] 3. One mobile smart car can cover multiple areas, has a wide range of applications and flexible deployment, and can reduce the number of monitoring devices;

[0035] 4. The system has the ability to identify targets and automatically follow them, and also supports cloud alarm push, enabling active response and timely measures. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] The present invention will be further described in detail below in conjunction with the accompanying drawings and specific embodiments, and the above and / or other advantages of the present invention will become more clear.

[0037] Figure 1 It is a schematic diagram of the composition of the intelligent mobile monitoring device disclosed in the present invention;

[0038] Figure 2 It is a schematic diagram of the composition of the target detection module constructed based on the Hi3516 platform disclosed in the embodiment;

[0039] Figure 3 It is a schematic diagram of the composition of the intelligent mobile monitoring device based on the Hi3516 platform and the Hi3861 platform disclosed in the embodiment;

[0040] Figure 4 This is a schematic diagram of the calculation of the steering angle in the obstacle avoidance mode;

[0041] Figure 5 The figure is a schematic diagram of the composition of the intelligent mobile monitoring system disclosed in the present invention. DETAILED DESCRIPTION

[0042] The present invention discloses an intelligent mobile monitoring device, which can detect and follow a target, dynamically adjust the shooting angle, and realize monitoring without blind spots. Figure 1 As shown, the intelligent mobile monitoring device includes: a camera 100, a target detection module 200, a motion control module 300, and an intelligent car 400; the camera 100 is installed on the intelligent car 400 to obtain the video of the monitoring area; the target detection module 200 is used to perform target detection on the video of the monitoring area, and obtain the position and confidence of the target in the video image; the motion control module 300 is used to control the motor drive signal and the steering signal of the intelligent car 400 according to the position of the target in the video image, thereby controlling the movement of the intelligent car 400 so that the target is located in the center of the camera's field of view.

[0043] In this embodiment, the target detection module 200 uses the YOLOv2 model based on the Hi3516 platform to perform target detection on the video of the monitoring area, and obtains the rectangular frame and confidence of the target in the video image. Figure 2As shown, the Hi3516 platform includes a video frame preprocessing module 210 and a YOLOv2 model prediction module 220. The video frame preprocessing module 210 preprocesses the real-time video stream collected by the camera 100, including YUV to RGB conversion, image denoising and other operations; the preprocessed image is input into the YOLOv2 model prediction module 220 to obtain the rectangular outer frame and confidence of the target in the input image. The prediction result, that is, the target rectangular outer frame is transmitted to the motion control module. In order to obtain a stable target position, the detection result data of three consecutive frames of images are smoothed and then passed to the motion control module.

[0044] This embodiment targets pedestrians and tracks and monitors the target person. The YOLOv2 model in the YOLOv2 model prediction module 220 has been trained using a pedestrian detection data set. The training can be performed on a PC, and the model effect is tested on the PC using a test image. After the training is completed, the prototxt model structure file and the weights model weight file are output; wherein the prototxt file is a configuration file used to describe the neural network structure and each layer of parameters, and the weights file is a model weight file obtained by training. After that, the RuyiStudio tool is used to quantize the trained YOLOv2 model. When quantizing, the corresponding chip type is selected. In this embodiment, Hi3516DV300 is used. After quantization, the wk model file is obtained. This wk model file can be run on the embedded processor end of the Hi3516 platform. The YOLOv2 model is deployed on the embedded processor end of the Hi3516 platform. The specific deployment process includes copying the wk model file, completing video preprocessing, creating an AI reasoning thread, calling the wk model, etc.

[0045] The Hi3516 platform also includes a video stream push module 230, which is used to compress and encode the video stream collected by the camera or the video frame after target detection, and use the real-time information transmission RTMP protocol to push the compressed and encoded video to the cloud server or PC for users to view; users play the received video through decoding software and view the monitoring screen in real time. In this embodiment, the video stream push module 230 uses the FFmpeg6.0 library to perform H.264 compression encoding on the video frame to reduce the transmission bandwidth occupation; users can view the video using VLC player on the cloud or PC.

[0046] The motion control module 300 controls the motion of the smart car according to the received target rectangular frame coordinates. In this embodiment, the motion control module is implemented based on the Hi3861 platform. Figure 3As shown. The target detection module 200 transmits the target rectangular frame to the motion control module 300 through the UART serial port, that is, the Hi3516 platform and the Hi3861 platform transmit data through the UART serial port, and the transmission protocol is specifically configured as 115200 baud rate and 8-bit data position. After the motion control module receives the coordinates of the four vertices of the rectangular frame of the target in the video image, it generates the motion instructions of the smart car through the PID control algorithm to control the motion trajectory and speed of the car, so that the target remains in the center of the video image and the distance between it and the car remains relatively stable. Specifically:

[0047] Determine the relative distance and relative orientation between the target and the smart car according to the position of the target in the video image;

[0048] According to the relative distance and relative orientation between the target and the smart car, the PID algorithm is used to control the smart car motor drive PWM signal and the servo steering signal to make the target located in the center of the camera's field of view.

[0049] The PID control algorithm is an algorithm that controls according to the P (proportional), I (integral), and D (differential) of the input deviation. In this embodiment, the coefficients of the PID in the algorithm are adjusted according to the actual response speed and steering sensitivity of the car through field debugging. The motion state of the smart car is obtained by the gyroscope.

[0050] The smart car may encounter obstacles during its movement. In this embodiment, a distance measurement module 310 is provided on the smart car to obtain the distance d between the obstacle in front of the smart car; when the distance d between the smart car and the obstacle in front of the car is less than or equal to the obstacle avoidance threshold, the obstacle avoidance mode is triggered; in this embodiment, the obstacle avoidance threshold is 20 cm; the motion control module controls the movement of the smart car according to the following steps:

[0051] S1, the smart car stops moving forward, turns left and right by an angle θ, respectively, and measures and obtains the distances d1 and d2 between the front of the smart car and the obstacle; in this embodiment, θ=30°;

[0052] S2. If |d1-d2|<Δd, that is, d1 and d2 are approximately equal, it means that the obstacle is located in front of the smart car. Figure 4 As shown in (a); at this time, the motion control module controls the smart car to turn right 90°; Δd is the preset obstacle position judgment threshold, which is 2 cm in this embodiment;

[0053] If |d1-d2|≥Δd and d1>d2, the space on the left side of the smart car is relatively open, such as Figure 4 As shown in (b), the motion control module controls the smart car to turn left, and the left turning angle is θ L ;

[0054] In the triangle formed by d, θ, and d1, the length L1 of the side opposite θ is calculated according to the cosine theorem as follows:

[0055]

[0056] The angle α opposite d1 L is:

[0057] The left turning angle is θ L is: θ L = 180 - α L ;

[0058] If |d1 - d2| ≥ Δd and d1 < d2, the space on the right side of the intelligent vehicle is relatively more open at this time. As shown in (c) of Figure 4 , the motion control module controls the intelligent vehicle to turn right, and the right turning angle is θ R ;

[0059] In the triangle formed by d, θ, and d2, the length L2 of the side opposite θ is calculated according to the cosine theorem as follows:

[0060]

[0061] The angle α opposite d2 R is:

[0062] The right turning angle is θ R is: θ R = 180 - α R ;

[0063] S3. Measure the distance d between the intelligent vehicle and the obstacle directly in front again. If d is greater than the obstacle avoidance threshold, the obstacle avoidance mode ends, and the motion control module controls the intelligent vehicle to continue moving forward;

[0064] If d is less than or equal to the obstacle avoidance threshold, jump to S1 for turning obstacle avoidance.

[0065] The ranging module 310 can use ultrasonic ranging or laser ranging. In this embodiment, an ultrasonic ranging sensor with the model HC-SR04 is used.

[0066] The intelligent mobile monitoring device also includes a status publishing and subscription module 320, which publishes the real-time status information of the intelligent car to the cloud server through the MQTT protocol, and subscribes to the remote commands issued by the cloud server, so that the remote user can obtain the movement status of the intelligent car in real time, such as speed, position, battery status, working mode, etc.; if special circumstances occur, such as target tracking loss, car movement is blocked, etc., an alarm is promptly sent to the cloud server to ensure that the user understands the movement of the car in real time and handles abnormalities in time, which is convenient for cloud monitoring and recording.

[0067] In addition, users can issue remote commands through the cloud interface. The command content can include: mode switching, speed and balance following distance adjustment, alarm response, etc. The publish-subscribe module 320 on the Hi3861 side receives and parses the command. The motion control module adjusts the car's working mode, reference speed, target tracking balance position and other parameters according to the parsed command, or directly performs the corresponding motion operation. After the command is executed, the car sends feedback data to the cloud and updates the monitoring status to ensure that the user can see the effect after execution.

[0068] This embodiment also provides an intelligent mobile monitoring system, including a cloud server and at least one intelligent mobile monitoring device as described above. Figure 5 As shown, each smart mobile monitoring device can monitor different targets, and multiple smart mobile monitoring devices can monitor multiple targets. In this embodiment, a cloud server is established based on the Huawei Cloud IoT platform LoTDA. The user logs in to the Huawei Cloud IAM account through the application and enters the smart mobile monitoring system ID. After that, the application will access Huawei Cloud to obtain a token, realize the connection between the smart mobile monitoring system and the LoTDA platform, and open the monitoring / command sending interface of the corresponding device.

[0069] Each intelligent mobile monitoring device is turned on, and each peripheral is accessed through the GPIO port and I2C bus to complete initialization. The publish-subscribe module sends a connection request to the cloud server through the MQTT protocol. After that, the intelligent mobile monitoring system enters a standby state, waiting for the triggering of the user or the scheduled task.

[0070] The present invention provides a concept and method of an intelligent mobile monitoring device and system. There are many methods and ways to implement the technical solution. The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention. These improvements and modifications should also be regarded as the protection scope of the present invention. All components not specified in this embodiment can be implemented by existing technologies.

Claims

1. An intelligent mobile monitoring device based on target detection, characterized in that: include: A camera (100), a target detection module (200), a motion control module (300), and an intelligent car (400); the camera (100) is installed on the intelligent car (400) and is used to obtain a video of a monitored area; The target detection module (200) is used to perform target detection on the video of the monitoring area to obtain the position of the target in the video image; the motion control module (300) is used to control the motor drive signal and the steering gear steering signal of the intelligent car (400) according to the position of the target in the video image.

2. The intelligent mobile monitoring device according to claim 1, characterized in that: The target detection module (200) performs target detection on the video of the monitoring area based on the Hi3516 platform using the YOLOv2 model to obtain a rectangular outer frame of the target in the video image.

3. The intelligent mobile monitoring device according to claim 2, characterized in that: The YOLOv2 model is trained using a pedestrian detection dataset, and after training, a prototxt model structure file and a weights model weight file are output; The trained YOLOv2 model is quantized using the RuyiStudio tool to obtain a wk model file, which runs on the embedded processor end of the Hi3516 platform.

4. The intelligent mobile monitoring device according to claim 1, characterized in that: The intelligent mobile monitoring device also includes a video stream pushing module (230), which compresses and encodes the video stream collected by the camera or the video frame after target detection, and pushes the compressed and encoded video to a cloud server or a PC using the RTMP protocol.

5. The intelligent mobile monitoring device according to claim 4, characterized in that: The video stream pushing module (230) uses H.264 to compress and encode the video frames.

6. The intelligent mobile monitoring device according to claim 1, characterized in that: The motion control module (300) controls the motion of the smart car, including: Determine the relative distance and relative orientation between the target and the smart car according to the position of the target in the video image; According to the relative distance and relative orientation between the target and the smart car, the PID algorithm is used to control the smart car motor drive PWM signal and the servo steering signal to make the target located in the center of the camera's field of view.

7. The intelligent mobile monitoring device according to claim 1, characterized in that: The intelligent mobile monitoring device also includes a distance measurement module (310), which is arranged on the intelligent car and is used to obtain the distance d between the front of the intelligent car and the obstacle; When the distance d between the smart car and the obstacle in front of it is less than or equal to the obstacle avoidance threshold, the obstacle avoidance mode is triggered; the motion control module controls the movement of the smart car according to the following steps: S1, the smart car stops moving forward, turns left and right by angle θ, and measures and obtains the distances d1 and d2 between the front of the smart car and the obstacle; S2. If |d1-d2|<Δd, the motion control module is also used to control the smart car to turn right 90°; Δd is the preset obstacle position judgment threshold; If |d1-d2|≥Δd and d1>d2, the motion control module is also used to control the smart car to turn left, and the left turning angle is θ L , the calculation steps are: In the triangle formed by d, θ, and d1, the length L1 of the side opposite to θ is calculated according to the cosine theorem: The angle α opposite to d1 L for: The left turning angle is θ L is: L =180-α L ; If |d1 - d2| ≥ Δd and d1 < d2, the motion control module is further configured to control the intelligent vehicle to turn right by an angle of θ R , and the calculation steps are as follows: In the triangle formed by d, θ, and d2, the length of the side opposite to θ, L2, is calculated according to the law of cosines: The angle α opposite to d2 R for: The right turn angle is θ R is: R =180-α R ; S3, measure again to obtain the distance d between the front of the smart car and the obstacle. If d is greater than the obstacle avoidance threshold, the obstacle avoidance mode ends, and the motion control module controls the smart car to continue moving forward; If d is less than or equal to the obstacle avoidance threshold, jump to S1 and turn to avoid the obstacle.

8. The intelligent mobile monitoring device according to claim 5, characterized in that: The distance measurement module (310) adopts ultrasonic distance measurement or laser distance measurement.

9. The intelligent mobile monitoring device according to claim 1, characterized in that: The intelligent mobile monitoring device also includes a status publishing and subscription module (320), which publishes the real-time status information of the intelligent car to the cloud server through the MQTT protocol and subscribes to the remote instructions issued by the cloud server.

10. An intelligent mobile monitoring system, characterized in that: It comprises a cloud server and at least one intelligent mobile monitoring device, wherein the intelligent mobile monitoring device is the intelligent mobile monitoring device according to any one of claims 1 to 9.

Images