An intelligent mobile monitoring device and system based on target detection

Abstract

The application discloses an intelligent mobile monitoring device and system based on target detection, wherein the monitoring device comprises a camera, a target detection module, a motion control module, and an intelligent trolley; the camera is installed on the intelligent trolley and is used for acquiring a video of a monitoring area; the target detection module is used for performing target detection on the video of the monitoring area and acquiring the position of a target in the video image; and the motion control module is used for controlling the motor driving signal and the steering signal of the steering gear of the intelligent trolley according to the position of the target in the video image. The device can realize autonomous movement and target following, and improves the flexibility and intelligent level of video monitoring.

Description

TECHNICAL FIELD

[0001] The application relates to an intelligent mobile monitoring device and system based on target detection and belongs to the technical field of video monitoring. BACKGROUND

[0002] At present, video monitoring is usually achieved by installing a camera, and the camera is installed on a fixed-position holder, so that the monitoring angle is obviously limited. The limitation directly leads to a large number of monitoring blind areas in the monitoring area, so that the areas cannot be effectively monitored. In order to achieve comprehensive coverage, a large number of cameras need to be installed at multiple angles and positions in a traditional monitoring system, and the installation environment has a high requirement, so that the deployment process of the whole system is extremely complex, involves numerous installation links and a large amount of manpower investment. When a specific moving target needs to be continuously monitored, videos collected by multiple cameras in the moving range need to be fused and analyzed, and the workload is large. SUMMARY

[0003] The application aims to solve the technical problems of the prior art and provide an intelligent mobile monitoring device based on target detection, which can realize autonomous movement and target following, so as to improve the flexibility and intelligent level of video monitoring.

[0004] In order to solve the above technical problems, the application discloses an intelligent mobile monitoring device based on target detection, which comprises a camera 100, a target detection module 200, a motion control module 300 and an intelligent trolley 400. The camera 100 is installed on the intelligent trolley 400 and is used for acquiring videos of a monitoring area. The target detection module 200 is used for detecting targets in the videos of the monitoring area and acquiring positions of the targets in video images. The motion control module 300 is used for controlling motor driving signals and steering signals of the intelligent trolley 400 according to the positions of the targets in the video images.

[0005] Further, the target detection module 200 detects targets in the videos of the monitoring area based on a Hi3516 platform and adopts a YOLOv2 model to acquire rectangular frames of the targets in the video images.

[0006] Further, the YOLOv2 model is trained by using a pedestrian detection data set, and a prototxt model structure file and a weights model weight file are output after the training. A RuyiStudio tool is used to quantize the trained YOLOv2 model to obtain a wk model file, and the wk model file runs on an embedded processor end of the Hi3516 platform.

[0007] Further, the intelligent mobile monitoring device further comprises a video stream pushing module 230, which compresses and encodes the video stream collected by the camera or the video frame after target detection, and pushes the compressed and encoded video to the cloud server or PC end using the RTMP protocol.

[0008] Further, the video stream pushing module 230 compresses and encodes the video frame using H.264.

[0009] Further, the motion control module 300 controls the motion of the intelligent car, including:

[0010] determining the relative distance and relative direction of the target and the intelligent car according to the position of the target in the video image;

[0011] controlling the motor drive PWM signal and the steering signal of the steering engine of the intelligent car using the PID algorithm according to the relative distance and relative direction of the target and the intelligent car, so that the target is located at the center of the field of view of the camera.

[0012] Further, the intelligent mobile monitoring device further comprises a distance measuring module 310, which is arranged on the intelligent car and is used to obtain the distance d between the front of the intelligent car and the obstacle;

[0013] When the distance d between the intelligent car and the obstacle in front of the car is less than or equal to the obstacle avoidance threshold, the obstacle avoidance mode is triggered; the motion control module controls the motion of the intelligent car according to the following steps:

[0014] S1, the intelligent car stops moving forward, and turns left and right by an angle θ, respectively, and measures and obtains the distances d1 and d2 between the front of the intelligent car and the obstacle;

[0015] S2, if |d1-d2|<Δd, the motion control module is further used to control the intelligent car to turn right by 90°; Δd is a preset obstacle direction judgment threshold;

[0016] If |d1-d2|≥Δd and d1>d2, the motion control module is further used to control the intelligent car to turn left, and the left turning angle is θ L , and the calculation steps are:

[0017] In the triangle formed by d, θ, and d1, the length L1 of the opposite side of θ is calculated according to the cosine law:

[0018]

[0019] the opposite angle α of d1 L is:

[0020] the left turning angle is θ L is: θL = 180 - a L ;

[0021] If |d1-d2|≥Ad and d1 R , the calculation steps are as follows:

[0022] In the triangle formed by d, theta, and d2, the length L2 of the opposite side of theta is calculated according to the cosine theorem:

[0023]

[0024] The opposite angle a of d2 is: R

[0025] The right turning angle is theta R R = 180 - a R ;

[0026] S3, the distance d between the front of the intelligent car and the obstacle is measured again, and if d is greater than the obstacle avoidance threshold, the obstacle avoidance mode ends, and the motion control module controls the intelligent car to continue moving forward;

[0027] If d is less than or equal to the obstacle avoidance threshold, jump to S1 to turn and avoid obstacles.

[0028] Further, the distance measuring module 310 adopts ultrasonic ranging or laser ranging.

[0029] Further, the intelligent mobile monitoring device further comprises a state publishing and subscribing module 320, which publishes the real-time state information of the intelligent car to the cloud server through the MQTT protocol, and subscribes to the remote instructions issued by the cloud server.

[0030] On the other hand, the present application also discloses an intelligent mobile monitoring system, comprising a cloud server and at least one intelligent mobile monitoring device, wherein the intelligent mobile monitoring device is the above-mentioned intelligent mobile monitoring device.

[0031] Advantages: Compared with the prior art, the intelligent mobile monitoring device and system disclosed by the present application have the following advantages:

[0032] 1. Real-time detection and following of the target, dynamic adjustment of the shooting angle, and realization of blind area-free monitoring;

[0033] 2. The MQTT protocol is used to connect with the cloud server, and the RTMP protocol is used to push the video stream, so as to support real-time data synchronization and remote control, and realize intelligent management;

[0034] ​​3. One mobile intelligent trolley can cover multiple areas, has wide application range, flexible deployment, and can reduce the number of monitoring devices;

[0035] 4. The system has target recognition and automatic following capability, supports cloud alarm push, realizes active response and timely measures. BRIEF DESCRIPTION OF DRAWINGS

[0036] The above and / or other aspects of the present application will become apparent by referring to the following description taking in conjunction with the accompanying drawings, wherein:

[0037] Figure 1 A composition diagram of the intelligent mobile monitoring device disclosed in the present application is shown in the figure;

[0038] Figure 2 A composition diagram of the target detection module based on the Hi3516 platform disclosed in the embodiment is shown in the figure;

[0039] Figure 3 A composition diagram of the intelligent mobile monitoring device based on the Hi3516 platform and the Hi3861 platform disclosed in the embodiment is shown in the figure;

[0040] Figure 4 A calculation diagram of the steering angle in the obstacle avoidance mode is shown in the figure;

[0041] Figure 5 A composition diagram of the intelligent mobile monitoring system disclosed in the present application is shown in the figure. DETAILED DESCRIPTION

[0042] The present application discloses an intelligent mobile monitoring device, which can detect and follow a target, dynamically adjust the shooting angle, and realize blind area-free monitoring. Figure 1 As shown in the figure, the intelligent mobile monitoring device comprises a camera 100, a target detection module 200, a motion control module 300, and an intelligent trolley 400. The camera 100 is installed on the intelligent trolley 400 and is used to acquire the video of the monitoring area. The target detection module 200 is used to perform target detection on the video of the monitoring area and acquire the position and confidence of the target in the video image. The motion control module 300 is used to control the motor drive signal and the steering signal of the steering engine of the intelligent trolley 400 according to the position of the target in the video image, so as to control the motion of the intelligent trolley 400 and make the target located at the center of the camera field of view.

[0043] In this embodiment, the target detection module 200 performs target detection on the video of the monitoring area based on the Hi3516 platform and uses the YOLOv2 model to acquire the rectangular outer frame and the confidence of the target in the video image. Figure 2As shown, the Hi3516 platform includes a video frame preprocessing module 210 and a YOLOv2 model prediction module 220. The video frame preprocessing module 210 pre-processes the real-time video stream collected by the camera 100, including YUV to RGB conversion, image denoising, and the like; the pre-processed image is input to the YOLOv2 model prediction module 220 to obtain the rectangular frame of the target in the input image and the confidence. The prediction result, i.e., the rectangular frame of the target, is transmitted to the motion control module. In order to obtain a stable target position, the detection result data of three consecutive images are smoothed and then transmitted to the motion control module.

[0044] The embodiment takes pedestrians as targets and tracks and monitors the target person. The YOLOv2 model in the YOLOv2 model prediction module 220 has been trained using a pedestrian detection dataset. The training can be performed on a PC, and the model effect test can be performed on the PC using test pictures. The training outputs a prototxt model structure file and a weights model weight file; the prototxt file is a configuration file used to describe the neural network structure and the parameters of each layer, and the weights file is a model weight file obtained by training. Then, the RuyiStudio tool is used to quantize the trained YOLOv2 model. When quantizing, the corresponding chip type is selected, and the Hi3516DV300 is used in the embodiment. After quantization, a wk model file is obtained, which can run on the embedded processor of the Hi3516 platform. The YOLOv2 model is deployed on the embedded processor of the Hi3516 platform. The specific deployment process includes copying the wk model file, completing video preprocessing, creating an AI inference thread, and calling the wk model.

[0045] The Hi3516 platform further includes a video stream pushing module 230 for compressing and encoding the video stream collected by the camera or the video frame after target detection, pushing the compressed and encoded video to a cloud server or a PC end using the Real-Time Information Transfer (RTMP) protocol, and allowing a user to view the video; the user plays the received video through decoding software and views the monitoring picture in real time. In the embodiment, the video stream pushing module 230 uses the FFmpeg 6.0 library to compress and encode the video frame using H.264 to reduce the transmission bandwidth occupation; the user can view the video using the VLC player on the cloud or PC end.

[0046] The motion control module 300 controls the motion of the intelligent car according to the received target rectangular frame coordinates. In the embodiment, the motion control module is implemented based on the Hi3861 platform. The intelligent mobile monitoring device constructed in this way is as shown in FIG. 6. Figure 3The target detection module 200 transmits the rectangular outer frame of the target to the motion control module 300 through a UART serial port, that is, data is transmitted between the Hi3516 platform and the Hi3861 platform through the UART serial port, and the transmission protocol is specifically configured as 115200 baud rate and 8-bit data bit. After receiving the coordinates of the four vertices of the rectangular outer frame of the target in the video image, the motion control module generates a motion instruction of the intelligent car through a PID control algorithm, so as to control the motion trajectory and motion speed of the car, so that the target is kept in the center of the video image and the distance between the target and the car is kept relatively stable. Specifically:

[0047] According to the position of the target in the video image, the relative distance and relative direction of the target and the intelligent car are determined;

[0048] According to the relative distance and relative direction of the target and the intelligent car, a PID algorithm is used to control the motor driving PWM signal and the steering signal of the steering engine of the intelligent car, so that the target is located in the center of the field of view of the camera.

[0049] The PID control algorithm is an algorithm for controlling P (proportion), I (integration) and D (derivative) according to the input deviation. In this embodiment, the coefficients of PID in the algorithm are adjusted according to the actual response speed and steering sensitivity of the car through on-site debugging. The motion state of the intelligent car is obtained by a gyroscope.

[0050] The intelligent car may encounter obstacles during movement. In this embodiment, a distance measuring module 310 is arranged on the intelligent car to obtain the distance d between the front of the intelligent car and the obstacle; when the distance d between the intelligent car and the obstacle in front of the car is less than or equal to the obstacle avoidance threshold, the obstacle avoidance mode is triggered; in this embodiment, the obstacle avoidance threshold is 20 cm; the motion control module controls the movement of the intelligent car according to the following steps:

[0051] S1, the intelligent car stops moving forward, and rotates left and right by an angle θ, respectively, to measure and obtain the distances d1 and d2 between the front of the intelligent car and the obstacle; in this embodiment, θ = 30°;

[0052] S2, if |d1-d2|<Δd, that is, d1 and d2 are approximately equal, indicating that the obstacle is located in front of the intelligent car, as shown in (a) of Figure 4 ; at this time, the motion control module controls the intelligent car to turn right by 90°; Δd is a preset obstacle direction judgment threshold, which is 2 cm in this embodiment;

[0053] If |d1-d2|≥Δd and d1>d2, the space on the left side of the intelligent car is relatively more open, as shown in (b) of Figure 4 ; the motion control module controls the intelligent car to turn left, and the left turning angle is θ L ;

[0054] In the triangle formed by d, θ, and d1, the length L1 of the opposite side of θ is calculated according to the cosine theorem as follows:

[0055]

[0056] The opposite angle α of d1 L is:

[0057] The left turning angle is θ L is: θ L = 180 - α L ;

[0058] If |d1 - d2| ≥ Δd and d1 < d2, at this time, the space on the right side of the intelligent vehicle is relatively more open, as shown in (c) of Figure 4 , the motion control module controls the intelligent vehicle to turn right, and the right turning angle is θ R ;

[0059] In the triangle formed by d, θ, and d2, the length L2 of the opposite side of θ is calculated according to the cosine theorem as follows:

[0060]

[0061] The opposite angle α of d2 R is:

[0062] The right turning angle is θ R is: θ R = 180 - α R ;

[0063] S3. Measure again to obtain the distance d between the intelligent vehicle and the obstacle directly in front. If d is greater than the obstacle avoidance threshold, the obstacle avoidance mode ends, and the motion control module controls the intelligent vehicle to continue moving forward;

[0064] If d is less than or equal to the obstacle avoidance threshold, jump to S1 for turning obstacle avoidance.

[0065] The ranging module 31******* can use ultrasonic ranging or laser ranging. In this embodiment, an ultrasonic ranging sensor with the model HC - SR04 is used.

[0066] The intelligent mobile monitoring device further comprises a state publishing and subscribing module 320, which publishes real-time state information of the intelligent car to the cloud server through the MQTT protocol, and subscribes to remote instructions issued by the cloud server, so that a remote user can obtain the motion state of the intelligent car in real time, such as speed, position, battery state, working mode, etc.; if a special situation occurs, such as target tracking loss, car motion obstruction, etc., the cloud server is alarmed in time, so that the user can master the car motion in real time, handle the exception in time, and facilitate cloud monitoring and recording.

[0067] In addition, the user can issue a remote instruction through the cloud interface. The instruction content can include: mode switching, speed and balance following distance adjustment, alarm response, etc. The publishing and subscribing module 320 on the Hi3861 side receives the instruction and parses it, and the motion control module adjusts the working mode, reference speed, target tracking balance position, etc. of the car according to the parsed instruction, or directly executes the corresponding motion operation. After the instruction is executed, the car sends feedback data to the cloud to update the monitoring state, so that the user can see the effect after execution.

[0068] The embodiment also provides an intelligent mobile monitoring system, which comprises a cloud server and at least one intelligent mobile monitoring device as described above. Figure 5 As shown in the figure, each intelligent mobile monitoring device can monitor different targets, and multiple intelligent mobile monitoring devices can realize multi-target monitoring. In the embodiment, the cloud server is established based on the Huawei cloud Internet of Things platform LoTDA, the user logs in to the Huawei cloud IAM account through an application program, and inputs the intelligent mobile monitoring system ID. Thereafter, the application program accesses the Huawei cloud to obtain a token, realizes connection between the intelligent mobile monitoring system and the LoTDA platform, and opens the monitoring / instruction sending interface of the corresponding device.

[0069] Each intelligent mobile monitoring device is started, initializes through the GPIO port and the I2C bus to access each external device, and the publishing and subscribing module sends a connection request to the cloud server through the MQTT protocol. Thereafter, the intelligent mobile monitoring system enters a standby state and waits for triggering of the user or a scheduled task.

[0070] The application provides an idea and method of an intelligent mobile monitoring device and system, and there are many methods and approaches to realize the technical solution, and the above description is only a preferred embodiment of the application. It should be noted that, for ordinary skilled persons in the technical field, some improvements and refinements can be made without departing from the principle of the application, and these improvements and refinements should also be regarded as the protection scope of the application. The components not explicitly described in the embodiment can be realized by using the prior art.

Claims

1. An intelligent mobile monitoring device based on target detection, characterized in that, include: Camera (100), target detection module (200), motion control module (300), and smart car (400); the camera (100) is installed on the smart car (400) and is used to acquire video of the monitored area; The target detection module (200) is used to detect targets in the video of the monitored area and obtain the position of the target in the video image; the motion control module (300) is used to control the motor drive signal and servo steering signal of the intelligent vehicle (400) according to the position of the target in the video image; The intelligent mobile monitoring device also includes a ranging module (310), which is installed on the intelligent vehicle and is used to obtain the distance d between the front of the intelligent vehicle and the obstacle. When the distance d between the intelligent vehicle and an obstacle directly in front of it is less than or equal to the obstacle avoidance threshold, the obstacle avoidance mode is triggered; the motion control module controls the movement of the intelligent vehicle according to the following steps: S1. The intelligent car stops moving forward and rotates to the left and right by angles θ respectively. Measure and obtain the distances d1 and d2 between the front of the intelligent car and the obstacle. S2, if |d1-d2| < The motion control module is also used to control the smart car to turn 90° to the right; The preset threshold for determining the location of obstacles; If |d1-d2|≥ Furthermore, d1 > d2, and the motion control module is also used to control the intelligent vehicle to turn left, with a left turn angle of θ. L The calculation steps are as follows: In the triangle formed by d, θ, and d1, the length of the side opposite θ is... Calculated using the Law of Cosines: ； diagonal of d1 for: ; The left turn angle is θ L for: ; If |d1 - d2| ≥ and d1 < d2, the motion control module is further configured to control the intelligent vehicle to turn right, and the right turn angle is θ R , and the calculation steps are as follows: In the triangle formed by d, θ, and d², the length of the side opposite θ is... Calculated using the Law of Cosines: ； diagonal of d2 for: ; Right turn angle θ R for: ; S3. Measure the distance d between the front of the smart car and the obstacle again. If d is greater than the obstacle avoidance threshold, the obstacle avoidance mode ends, and the motion control module controls the smart car to continue moving forward. If d is less than or equal to the obstacle avoidance threshold, jump to S1 and perform turning obstacle avoidance.

2. The intelligent mobile monitoring device according to claim 1, characterized in that, The target detection module (200) uses the YOLOv2 model on the Hi3516 platform to detect targets in the video of the monitored area and obtain the rectangular outline of the target in the video image.

3. The intelligent mobile monitoring device according to claim 2, characterized in that, The YOLOv2 model is trained using a pedestrian detection dataset, and the output after training is a prototxt model structure file and a weights model weight file. The trained YOLOv2 model was quantized using the RuyiStudio tool to obtain a wk model file, which was then run on the embedded processor of the Hi3516 platform.

4. The intelligent mobile monitoring device according to claim 1, characterized in that, The intelligent mobile monitoring device also includes a video stream push module (230), which compresses and encodes the video stream captured by the camera or the video frames after target detection, and pushes the compressed and encoded video to the cloud server or PC using the RTMP protocol.

5. The intelligent mobile monitoring device according to claim 4, characterized in that, The video stream push module (230) uses H.264 to compress and encode video frames.

6. The intelligent mobile monitoring device according to claim 1, characterized in that, The motion control module (300) controls the movement of the intelligent vehicle, including: The relative distance and relative orientation between the target and the intelligent vehicle are determined based on the target's position in the video image; Based on the relative distance and orientation between the target and the intelligent vehicle, a PID algorithm is used to control the intelligent vehicle's motor drive PWM signal and servo steering signal to keep the target in the center of the camera's field of view.

7. The intelligent mobile monitoring device according to claim 5, characterized in that, The ranging module (310) uses ultrasonic ranging or laser ranging.

8. The intelligent mobile monitoring device according to claim 1, characterized in that, The intelligent mobile monitoring device also includes a status publish-subscribe module (320), which publishes the real-time status information of the intelligent vehicle to the cloud server through the MQTT protocol and subscribes to remote commands issued by the cloud server.

9. An intelligent mobile monitoring system, characterized in that, It includes a cloud server and at least one intelligent mobile monitoring device, wherein the intelligent mobile monitoring device is the intelligent mobile monitoring device according to any one of claims 1-8.

Images