Multi-robot collaborative visual monitoring method and system

Abstract

This invention relates to the field of automated visual surveillance technology and discloses a collaborative robotic visual surveillance method and system. By deeply integrating visual perception with multi-robot collaborative control, this method enables autonomous perception and collaborative surveillance of suspicious targets within a surveillance area. By utilizing multiple mobile surveillance robots, this method not only expands the perception range and enhances the flexibility of the surveillance system, but also effectively prevents and deters criminal activity, achieving automated autonomous visual surveillance and reducing the involvement of security personnel.

Description

Technical Field

[0001] The present invention relates to the technical field of automated visual monitoring, and in particular to a multi-robot collaborative visual monitoring method and system. Background Art

[0002] Video surveillance systems are now ubiquitous in various living and working environments, such as shopping malls, warehouses, and private residences. However, traditional video surveillance solutions typically deploy one or more cameras in fixed locations. These solutions can only provide a limited deterrent effect on criminals or passively assist police in discovering evidence after the fact, but cannot effectively prevent criminal activity. To proactively prevent or stop criminal activity, such systems typically require at least one professional security officer, which not only increases system operation and maintenance costs but also poses a risk of compromising public privacy. Consequently, automated visual surveillance systems have gradually attracted widespread attention. Their primary goal is to reduce reliance on security personnel and enhance the automation and intelligence of video surveillance systems through a series of intelligent sensing and recognition technologies.

[0003] The relevant technologies involved in the current automated visual monitoring system are mainly divided into scene understanding technology involved in the field of computer vision and multi-robot control technology involved in the field of robotics, according to different technical focuses.

[0004] In the field of computer vision, automated understanding of surveillance scenes is achieved mainly through a series of data processing on the real-time video streams obtained by visual cameras. The main tasks of scene understanding include: (1) detecting and tracking various moving targets from image sequences; (2) analyzing and predicting the behavior of moving targets; (3) discovering suspicious targets and issuing alarms for potential criminal behavior or dangerous events, etc., ultimately improving the automation and intelligence of the surveillance system. With the development of deep learning technology, various learning-based methods have made great progress in target detection, target tracking, target re-identification, human pose estimation, pedestrian trajectory prediction, etc. The target detection and tracking tasks mainly focus on detecting and continuously tracking moving targets from continuous video frames; target re-identification hopes to match the same target from multiple non-overlapping images from multiple camera systems to achieve cross-camera tracking of the target; human pose estimation is used to estimate the posture of human joints in three-dimensional space and combine it with trajectory prediction for subsequent analysis of its behavior.

[0005] Furthermore, since target occlusion and non-rigid deformation pose numerous challenges to the accuracy of target detection and behavior understanding, scene understanding performance is highly susceptible to the location and angle of the surveillance camera. While it is possible to increase coverage of the surveillance area by deploying multiple cameras in different locations, thereby reducing the probability of target occlusion, as the number of cameras increases, low-cost camera calibration methods and the optimization of multi-sensor deployment locations will become challenges in further optimizing and improving the system. Furthermore, camera synchronization, finding common targets across multiple cameras, and efficient data transmission methods are also issues that must be considered in research. In summary, implementing automated visual surveillance tasks using fixed-position cameras in large-scale surveillance scenarios still faces numerous challenges.

[0006] In robotics research, to overcome the shortcomings of using fixed-position cameras, numerous studies have attempted to employ multiple mobile robots to collaborate and perform various types of surveillance tasks. For example, by designing control strategies, multiple ground mobile robots or drones can be used to achieve collaborative control tasks such as formation, convergence, coverage, encirclement, and tracking. However, these robotics studies focus primarily on robot motion control, particularly controller design, without considering the relevant perception issues encountered during surveillance. Furthermore, their controller implementation relies on precise real-time positioning of each robot and the monitored target, ignoring the actual sensor inputs that robots can obtain in real-world scenarios. In complex, dynamic scenarios, this assumedly accurate position information is often not accurately perceived in real time. Consequently, these control systems still face numerous challenges before they can be put into practical use. Summary of the Invention

[0007] To solve the above technical problems, the present invention provides a robot collaborative visual monitoring method and system, which integrates the scene understanding technology in the field of computer vision with the multi-robot collaborative control technology in the field of robotics, fully taps the perception and understanding capabilities of computer vision technology for real scenes, and combines the collaborative and maneuverable capabilities of multi-robot systems to achieve continuous dynamic monitoring of suspicious targets based on multi-robot collaboration in real scenes.

[0008] In order to solve the above technical problems, the present invention adopts the following technical solutions:

[0009] A multi-robot collaborative visual monitoring method, which uses multiple mobile surveillance robots equipped with visual cameras to achieve collaborative visual perception and collaborative tracking monitoring of suspicious targets, includes the following steps:

[0010] Step 1: Use the suspicious target detection model based on the YOLO target detection algorithm to detect the mobile surveillance robot R i Real-time monitoring image I iPerform suspicious target detection and obtain the suspicious target area I i r ;

[0011] Step 2: When the suspicious target is not lost, the target tracking algorithm is used to continuously track the suspicious target; when the suspicious target is lost, the suspicious target is relocated according to step 1;

[0012] Step 3: All suspicious target areas {I i r Extract feature points to obtain a set of suspicious target feature points; construct a feature description of the target feature points to obtain a feature descriptor;

[0013] Step 4: By calculating the similarity between two feature descriptors, it is determined whether the suspicious targets corresponding to the two feature descriptors are the same suspicious target. A unique mapping is constructed for all suspicious targets detected by each mobile surveillance robot to the suspicious targets detected by other mobile surveillance robots.

[0014] Step 5: Control multiple mobile surveillance robots through tracking control algorithms to collaboratively track and monitor the same suspicious target.

[0015] Specifically, in step 2, while continuously tracking the suspicious target using the target tracking algorithm, the position of the suspicious target is corrected through step 1 at fixed time intervals.

[0016] Specifically, the target tracking algorithm is a MeanShift target tracking algorithm.

[0017] Specifically, in step 3, the suspicious target area {I i r}Extract feature points; use the BRIFE descriptor in the ORB algorithm to construct a feature description of the target feature points.

[0018] Specifically, step four includes:

[0019] Step 41: Measuring the similarity between different feature descriptors by calculating the Hamming distance between different feature descriptors;

[0020] Step 42: Use the k-nearest neighbor method to match feature points. For each feature point, use the similarity metric between feature descriptors to select the k most similar points; where k = 2;

[0021] Step 43: By calculating target similarity, a unique mapping is constructed for all suspicious targets detected in each mobile surveillance robot to the suspicious targets of other mobile surveillance robots; the target similarity is set as the ratio of the number of matching points between suspicious targets to the total number of feature points. If the ratio is greater than the similarity threshold, a mapping relationship between the two suspicious targets is established.

[0022] Specifically, in step 2, a target tracking algorithm is used to track the suspicious target and obtain dynamic target tracking results. In step 5, when multiple mobile surveillance robots are controlled by the tracking control algorithm, each mobile surveillance robot constructs an angle error and a visual error based on the dynamic target tracking results. The angle error and the visual error are collectively referred to as the visual tracking error e(t). Two PID controllers are used to reduce the distance error and the angle error respectively, thereby achieving linear speed control and angular speed control of the mobile surveillance robot. The form of the PID controller is as follows:

[0023] u(t)=K p e(t)+K i ∫e(t)+K d de(t);

[0024] Where u(t) is the control input, e(t) is the visual tracking error, K p , K i , K d It is an adjustable parameter.

[0025] A multi-robot collaborative visual monitoring system, comprising:

[0026] The suspicious target detection module detects suspicious targets on the mobile surveillance robot R through a suspicious target detection model based on the YOLO target detection algorithm. i Real-time monitoring image I i Perform suspicious target detection and obtain the suspicious target area I i r ;

[0027] The moving target tracking module uses the target tracking algorithm to continuously track the suspicious target when the suspicious target is not lost; when the suspicious target is lost, the suspicious target is relocated according to step 1;

[0028] The feature extraction description module uses the feature extraction algorithm to extract all suspicious target areas {I i r Extract feature points to obtain a set of suspicious target feature points; construct a feature description of the target feature points to obtain a feature descriptor;

[0029] The cross-camera matching module calculates the similarity between two feature descriptors to determine whether the suspicious objects corresponding to the two feature descriptors are the same suspicious object. It also creates a unique mapping from all suspicious objects detected by each mobile surveillance robot to the suspicious objects detected by other mobile surveillance robots.

[0030] The collaborative tracking module controls multiple mobile surveillance robots to collaboratively track and monitor the same suspicious target through a tracking control algorithm.

[0031] The monitoring system in the present invention corresponds to the monitoring method, and the preferred solutions proposed for the monitoring method are also applicable to the monitoring system.

[0032] Compared with the prior art, the beneficial technical effects of the present invention are:

[0033] 1. Unlike traditional visual surveillance systems that rely on fixed cameras and security personnel, this invention proposes a novel automated visual surveillance solution. By deeply integrating visual perception with multi-robot collaborative control, it achieves autonomous perception and collaborative surveillance of suspicious targets within the monitoring area. By utilizing multiple mobile surveillance robots, this not only expands the perception range and enhances the flexibility of the surveillance system, but also effectively prevents and deters criminal activity, achieving automated, autonomous visual surveillance and reducing the need for security personnel.

[0034] 2. In the visual perception part, unlike mainstream deep learning solutions, by combining deep learning object detection with traditional feature extraction and description, a complete scene perception and understanding process including detection, tracking, extraction, and matching is constructed. This greatly improves the efficiency of data processing and transmission between multiple robots, ensures the real-time performance of the system, and enhances the practicality of the algorithm.

[0035] 3. In the tracking control part, by making full use of the target system obtained in the perception stage and using a tracking control method based on visual features, the traditional control method’s dependence on high-precision positioning information is reduced, and the adaptability of the multi-robot system in surveillance and roundup tasks is improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 Schematic diagram of the software and hardware modules of the multi-robot collaborative visual monitoring system of the present invention;

[0037] Figure 2 Schematic diagram of a suspicious target detection model of the present invention;

[0038] Figure 3 This is a flow chart of the real-time dynamic tracking algorithm for moving targets of the present invention;

[0039] Figure 4 Schematic diagram of suspicious target detection results of multiple robots in the present invention;

[0040] Figure 5 This is a schematic diagram of the mobile surveillance robot of the present invention tracking a suspicious moving target in real time;

[0041] Figure 6 A schematic diagram of cross-camera matching of suspicious targets in the multi-robot collaborative perception system of the present invention;

[0042] Figure 7 This is a diagram of the experimental process of an embodiment of the present invention. DETAILED DESCRIPTION

[0043] A preferred embodiment of the present invention will be described in detail below with reference to the accompanying drawings.

[0044] This invention integrates scene understanding technology in the field of computer vision with multi-robot collaborative control technology in the field of robotics, fully tapping into the perception and understanding capabilities of computer vision technology for real scenes, and combining it with the collaborative and maneuverable capabilities of multi-robot systems to achieve continuous dynamic monitoring of suspicious targets based on multi-robot collaboration in real scenes.

[0045] The basic concept of this invention is to deploy a group of mobile surveillance robots equipped with visual cameras in the surveillance area, and use the multi-robot collaborative visual surveillance method of this invention to conduct autonomous patrols, thereby overcoming the shortcomings of using fixed cameras that can only monitor passively, maximizing the surveillance range and increasing the flexibility of the surveillance system. At the same time, relying on computer vision technology to perceive surrounding dynamic targets in real time, it reduces the reliance of traditional control methods on positioning information and increases the applicability of multi-robot systems in real-world scenarios. Specifically, the multi-robot collaborative visual monitoring method and monitoring system proposed in this invention utilizes a multi-robot system to divide the entire monitoring process into two major parts: visual perception and tracking control.

[0046] In the collaborative visual perception section, a complete scene understanding data processing pipeline combining learning-based and feature-based methods is proposed. This includes target detection and tracking, target feature extraction and description, and cross-camera target matching. This allows for real-time acquisition of feature information of suspicious targets to be captured and cross-camera tagging of them.

[0047] The collaborative tracking control part, based on the above-mentioned perception information and combined with the tracking control algorithm, uses multiple mobile surveillance robots to achieve collaborative tracking and monitoring of suspicious targets.

[0048] The hardware and software modules of the multi-robot collaborative visual monitoring system are as follows: Figure 1 As shown in the figure, the hardware part includes multiple mobile surveillance robots equipped with visual cameras, and the software part includes two parts: collaborative visual perception algorithm and collaborative tracking control algorithm.

[0049] First, when no suspicious targets are present, the mobile surveillance robots will independently perform patrol missions. The collaborative visual perception algorithm, using a suspicious target detection model, monitors the mission scene in real time for suspicious targets. Once a suspicious target is detected, each robot will utilize a series of precise and real-time visual perception technologies, including real-time dynamic tracking of the suspicious target using a motion target tracking module, constructing a key feature description of the suspicious target using feature extraction and description, and then performing multi-target cross-camera matching with neighboring robots based on this key feature description, providing a foundation for subsequent collaborative tracking and surveillance. Finally, a vision-based collaborative tracking control algorithm is used to achieve tracking and surveillance of the suspicious target.

[0050] 4.1 Suspicious Target Detection Model

[0051] The multi-robot collaborative visual surveillance task that the present invention is concerned with first requires that a single robot be able to independently detect and identify suspicious targets, which is the basis for the entire surveillance system to perform subsequent collaborative tracking tasks. Suspicious target detection and identification means that the robot can determine the position of the suspicious target in the image and identify its category from the real-time image obtained by the visual camera it carries. Generally speaking, the classification of suspicious behaviors needs to be studied in a targeted manner according to the specific application scenarios, such as the detection of abnormal behaviors such as theft and climbing over. In order to focus on the research of general algorithms, the present invention assumes that the task background is patrol surveillance of private territories such as unmanned factories and private residences. Therefore, pedestrian targets that appear at abnormal times are set as suspicious intruders.

[0052] In order to realize real-time detection of suspicious targets based on embedded devices on mobile surveillance robots, this paper follows the design ideas of the YOLO series target detection algorithms and uses a lightweight target detection network as a suspicious target detection model. The specific network model structure diagram of the suspicious target detection network is shown in the figure. Figure 2 shown. Figure 2 In it, Conv represents convolution, Maxpool represents maximum pooling, Upsample represents upsampling, Input represents input, and Output represents output.

[0053] Based on the above suspicious target detection model, each mobile surveillance robot can build a complete suspicious target detection node by i Real-time monitoring image I i The target detection network can be used to obtain the position and size of the target within the robot's field of view, and further obtain the region of interest I of the suspicious target based on the position and size output by the network. i r , used for processing by subsequent modules.

[0054] 4.2 Moving Target Tracking Module

[0055] Suspicious targets may flee, and their speed and direction of movement may change dramatically in a short period of time. Therefore, it is difficult to track suspicious targets only through target detection to meet the task requirements of real-time dynamic monitoring and capture. Therefore, based on suspicious target detection, the present invention proposes a real-time dynamic tracking algorithm for moving targets based on the combination of target detection and target tracking to achieve stable real-time tracking of dynamic targets. The flow chart of the real-time dynamic tracking algorithm for moving targets is shown in the figure. Figure 3 shown.

[0056] Specifically, when the suspicious target detection node detects a suspicious target, it will publish the location of the corresponding area of the suspicious target to the motion target tracking module in real time. The motion target tracking module uses the MeanShift target tracking algorithm to achieve small-scale real-time tracking of the moving target. In the subsequent tracking process, in order to deal with the loss of the tracking target due to the large-scale rapid movement of the suspicious target, the suspicious target detection model will be promptly called to relocate the position of the suspicious target. Not only that, during the dynamic tracking of the suspicious target, the suspicious target detection model will be called at fixed time intervals to correct the position of the suspicious target, ensuring that the use of the real-time dynamic tracking algorithm for moving targets improves the tracking speed while also achieving both tracking accuracy and stability.

[0057] 4.3 Multi-object Cross-camera Matching

[0058] In actual scenarios, due to the limited camera field of view of a single robot, as well as object occlusion, lighting and field of view changes, it is difficult to achieve continuous monitoring of suspicious targets in large-scale surveillance scenarios by relying solely on a single robot through a suspicious target detection model and a motion target tracking algorithm. Moreover, considering the limited maneuverability of a single mobile surveillance robot, it is also difficult to track and monitor suspicious targets when they escape. Therefore, the present invention uses a multi-robot system, based on the target detection and tracking results of multiple mobile surveillance robots in the above-mentioned suspicious target detection model and the real-time dynamic tracking algorithm for motion targets, to collaboratively monitor and perceive suspicious targets from multiple angles and positions, greatly reducing the probability of missing suspicious targets. The core of this part is how to use the suspicious target perception results (sometimes multiple suspicious targets) obtained by a single robot to share and fuse them with the perception results of its neighboring robots, and finally obtain a unified and complete description of all suspicious targets. The present invention models the collaborative perception between multiple robots as a problem of matching multiple suspicious targets across cameras.

[0059] Multiple target cross-camera matching refers to matching each robot R i The suspicious target area set {I i r}One by one with the neighboring robots R j The obtained suspicious target area set {I j r}, ultimately obtaining a mapping relationship for each robot between the suspicious targets detected by that robot and those detected by its neighboring robots. This mapping relationship eliminates the limitation that different robots cannot distinguish between different suspicious targets due to their different positions and fields of view. This allows the subsequent tracking control module to utilize the perception results of the same suspicious target by different robots to achieve collaborative tracking and monitoring of the suspicious target.

[0060] The implementation process of multi-target cross-camera matching includes target feature point extraction, target feature point description, and feature-based multi-target cross-camera matching.

[0061] 4.3.1 Target feature point extraction: To match the same object from two different images, the most direct way is to directly perform subtraction on the two images, and roughly determine whether the two are the same by analyzing the size of the error. However, this method is almost impossible to use in practice, because even in actual scenes, even if it is two images of the same object, the difference in pixel values may still be large due to changes in lighting, angle, and size. Therefore, in order to accurately and efficiently match the same object in two different perspectives, it is necessary to be able to extract features that do not change due to camera movement and rotation from different images of the same object, and then match the objects based on these features. The ORB feature extraction algorithm takes into account the scale and rotation invariance of SIFT features while ensuring the extraction speed. Therefore, in this section, the ORB feature extraction algorithm is used to extract all suspicious target areas {I i r}Perform feature point extraction.

[0062] 4.3.2 Target Feature Point Description: Next, we need to extract feature point information from the pixels in the suspicious target area and construct its feature description, that is, the feature descriptor, to facilitate the subsequent efficient matching of feature points and targets. Because the BRIEF descriptor randomly selects point pairs and uses binary strings to encode the descriptor, it facilitates the storage and matching of feature points and has extremely high computational and transmission efficiency. It is particularly suitable for task scenarios where multiple robots are matching features of multiple targets. Therefore, in this section, we still use the BRIFE descriptor in the ORB algorithm to construct the feature description of the target feature points.

[0063] 4.3.3 Cross-camera matching of multiple target feature points: After completing the target feature point extraction and target feature point description, although the same target has different pixel values under different robot camera perspectives, it is likely to have similar feature points, and similar feature points will also have similar feature descriptors. Therefore, using the above steps to construct target feature points and target feature point descriptions, by measuring the similarity of feature descriptors between targets, it is possible to easily achieve cross-camera matching of the same suspicious target. The cross-camera matching process of suspicious targets is mainly divided into three steps: similarity measurement, feature point matching, and target matching:

[0064] First, the similarity between different descriptors can be measured by calculating the distance. For example, the binary BRIEF descriptor can be measured using the Hamming distance.

[0065] Secondly, based on the similarity measure between feature descriptors, it is necessary to find the most similar feature points in the target feature point set to achieve feature point matching. In the specific implementation, feature point matching is achieved by using the k-nearest neighbor method. For each feature point, the similarity measure between descriptors is used to select the k most similar points (usually k=2).

[0066] Finally, a target similarity metric is used to construct a unique mapping of all suspicious targets detected in each robot to the suspicious targets of its neighboring robots. The metric is set as the ratio of the number of matching points between targets to their total number of feature points. At the same time, a similarity threshold is set to prevent the situation where the same target does not exist.

[0067] Finally, based on the above steps, feature-based multi-target cross-camera matching is achieved in a multi-robot system, ensuring both matching accuracy and real-time matching.

[0068] 4.4 Multi-robot collaborative tracking control algorithm based on visual information

[0069] To further achieve collaborative tracking and surveillance of suspicious targets, each mobile surveillance robot needs to convert its visual perception results into corresponding control instructions. Specifically, it needs to move in the desired direction and maintain a certain distance from the suspicious target to prevent the collaborative visual perception algorithm from failing due to close proximity, resulting in loss of continuous surveillance of the suspicious target. To reduce the mobile surveillance robot's reliance on positioning information in real-world environments, the collaborative tracking control algorithm will rely on visual information to achieve real-time tracking and control of the mobile surveillance robot. Specifically, each robot will construct a visual tracking error e(t) regarding angle and distance based on the dynamic target tracking results. Two PID controllers of the following forms are used: PIDv and PIDw are used to reduce the distance error and angle error, respectively, and to control the linear velocity and angular velocity of the mobile surveillance robot, respectively:

[0070] u(t)=K p e(t)+K i ∫e(t)+K d de(t);

[0071] Where u(t) is the control input, e(t) is the visual tracking error, and K p , K i , K d It is an adjustable parameter.

[0072] Example

[0073] The present invention conducts a multi-robot collaborative monitoring experiment in a real environment to demonstrate the feasibility and robustness of the multi-robot collaborative visual monitoring method proposed in the present invention in a real environment. In the actual experiment, three Pioneer3DX mobile robots equipped with visual cameras were used as mobile monitoring robots, and real dynamic pedestrian targets in the monitoring environment were used as suspicious targets. The three mobile monitoring robots were equipped with three types of cameras, namely ZED stereo camera, Kinect v2 camera and TX2 onboard camera, to test the adaptability of the monitoring method to images with different resolutions. In addition, two types of computing devices, including two laptops and NVIDIA TX2, were used to verify the real-time performance of the algorithm on different computing platforms. The multiple robots use the ROS operating system in conjunction with the local area network to achieve information interaction between individuals, and a 32-bit microprocessor (44.2368MHz) is used to control the motion of each robot.

[0074] First, using the suspicious target detection model, multiple mobile monitoring robots monitor suspicious targets at different locations based on different perspectives. Figure 4 shown.

[0075] Based on the monitoring results of the suspicious target detection model, the real-time tracking results of the suspicious pedestrian target using the moving target tracking algorithm are as follows: Figure 5 shown.

[0076] Then the three mobile surveillance robots perform cross-camera matching on the suspicious targets they continuously track. Figure 6 shown.

[0077] Finally, the entire experimental process in the real scene is as follows Figure 7 In the initial frame, the three mobile surveillance robots are unable to cover all surveillance areas due to the obstruction of buildings, and only the third robot finds the suspected intruder ( Figure 7 As the suspicious object moves, the second robot and the first robot discover the suspicious object in turn ( Figure 7As the robots move, they share and match features among themselves, gradually acquiring a more complete picture of the suspicious target. They then utilize a collaborative tracking control algorithm to continuously adjust the tracking angle and speed, ultimately achieving coordinated surveillance and tracking of the target.

[0078] Figure 4 、 Figure 5 、 Figure 6 、 Figure 7 Apply the diagram only to the scene, even if there is no Figure 4 、 Figure 5 、 Figure 6 、 Figure 7 , those skilled in the art can also understand the technical solutions in the description of the present invention.

[0079] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above and that the invention can be embodied in other specific forms without departing from the spirit or essential characteristics of the invention. Therefore, the embodiments should be considered in all respects as illustrative and non-limiting, and the scope of the invention is defined by the appended claims rather than the foregoing description. It is intended that all variations within the meaning and range of equivalents of the claims be embraced herein, and any reference signs in the claims should not be construed as limiting the claims to which they relate.

[0080] In addition, it should be understood that although this specification is described in terms of implementation methods, not every implementation method contains only one independent technical solution. This narrative method of the specification is only for the sake of clarity. Those skilled in the art should regard the specification as a whole. The technical solutions in each embodiment can also be appropriately combined to form other implementation methods that can be understood by those skilled in the art.

Claims

1. A multi-robot collaborative visual surveillance method, which uses multiple mobile surveillance robots equipped with visual cameras to achieve collaborative visual perception and collaborative tracking surveillance of suspicious targets, includes the following steps: Step 1: Use the suspicious target detection model based on the YOLO target detection algorithm to detect the mobile surveillance robot R i Real-time monitoring image I i Perform suspicious target detection and obtain the suspicious target area I i r ; Step 2: When the suspicious target is not lost, the target tracking algorithm is used to continuously track the suspicious target; when the suspicious target is lost, the suspicious target is relocated according to step 1; Step 3: All suspicious target areas {I i r Extract feature points to obtain a set of suspicious target feature points; construct a feature description of the target feature points to obtain a feature descriptor; Step 4: By calculating the similarity between two feature descriptors, it is determined whether the suspicious targets corresponding to the two feature descriptors are the same suspicious target. A unique mapping is constructed for all suspicious targets detected by each mobile surveillance robot to the suspicious targets detected by other mobile surveillance robots. Step 5: Control multiple mobile surveillance robots through tracking control algorithms to collaboratively track and monitor the same suspicious target.

2. The multi-robot collaborative visual monitoring method according to claim 1, characterized in that: In step 2, while continuously tracking the suspicious target using the target tracking algorithm, the position of the suspicious target is corrected through step 1 at fixed time intervals.

3. The multi-robot system visual monitoring method according to claim 1 or 2, characterized in that: The target tracking algorithm is the MeanShift target tracking algorithm.

4. The multi-robot collaborative visual monitoring method according to claim 1, characterized in that: In step 3, the suspicious target area {I i r }Extract feature points; use the BRIFE descriptor in the ORB algorithm to construct a feature description of the target feature points.

5. The multi-robot collaborative visual monitoring method according to claim 4, characterized in that: Step 4 specifically includes: Step 41: Measuring the similarity between different feature descriptors by calculating the Hamming distance between different feature descriptors; Step 42: Use the k-nearest neighbor method to match feature points. For each feature point, use the similarity measure between feature descriptors to select the k most similar points. Step 43: By calculating target similarity, a unique mapping is constructed for all suspicious targets detected by each mobile surveillance robot to the suspicious targets detected by other mobile surveillance robots; the target similarity is set as the ratio of the number of matching points between suspicious targets to the total number of feature points. If the ratio is greater than the similarity threshold, a mapping relationship between the two suspicious targets is established.

6. The multi-robot collaborative visual monitoring method according to claim 1, characterized in that: In step 2, a target tracking algorithm is used to track the suspicious target and obtain dynamic target tracking results. In step 5, when multiple mobile surveillance robots are controlled by the tracking control algorithm, each mobile surveillance robot constructs an angle error and a visual error based on the dynamic target tracking results. The angle error and the visual error are collectively referred to as the visual tracking error e(t). Two PID controllers are used to reduce the distance error and the angle error respectively, thus achieving linear speed control and angular speed control of the mobile surveillance robot. The form of the PID controller is as follows: u(t)=K p e(t)+K i ∫e(t)+K d de(t); Where u(t) is the control input, e(t) is the visual tracking error, K p , K i , K d It is an adjustable parameter.

7. A multi-robot collaborative visual monitoring system, characterized in that: include: The suspicious target detection module detects suspicious targets on the mobile surveillance robot R through a suspicious target detection model based on the YOLO target detection algorithm. i Real-time monitoring image I i Perform suspicious target detection and obtain the suspicious target area I i r ; The moving target tracking module uses the target tracking algorithm to continuously track the suspicious target when the suspicious target is not lost; when the suspicious target is lost, the suspicious target is relocated using the suspicious target detection model; The feature extraction description module uses the feature extraction algorithm to extract all suspicious target areas {I i r Extract feature points to obtain a set of suspicious target feature points; construct a feature description of the target feature points to obtain a feature descriptor; The cross-camera matching module calculates the similarity between two feature descriptors to determine whether the suspicious objects corresponding to the two feature descriptors are the same suspicious object. It also creates a unique mapping from all suspicious objects detected by each mobile surveillance robot to the suspicious objects detected by other mobile surveillance robots. The collaborative tracking module controls multiple mobile surveillance robots to collaboratively track and monitor the same suspicious target through a tracking control algorithm.

Images