Rail-mounted patrol robot and data processing method of fixed monitoring camera

Abstract

The invention provides a data processing method for a rail-mounted patrol robot and a fixed monitoring camera. The method comprises the following steps: carrying out real-time target detection and identification on a video stream captured by the fixed monitoring camera by using a deep learning model; calculating homogeneous coordinates of the target in the coordinate system of the fixed monitoring camera by using the bounding box coordinates of the target in the image and the internal reference matrix of the fixed monitoring camera; converting the homogeneous coordinate of the target from a fixed monitoring camera coordinate system to a mobile camera coordinate system; performing target tracking by using a Kalman filter according to the position of the target in the mobile camera coordinate system, and predicting the position of the target at the next moment; according to the predicted position of the target, the moving distance of the rail-mounted patrol robot and the rotating angle of the moving camera are adjusted, so that the target is kept in the optimal observation view field; and updating the position of the target by using an image shot by the mobile camera, and feeding back to a coordinate system of the fixed monitoring camera to realize continuous tracking of the target.

Description

Technical Field

[0001] The invention relates to the field of security monitoring, and in particular to a data processing method for a rail-mounted patrol robot and a fixed monitoring camera. Background Art

[0002] With the continuous progress of society and the development of science and technology, the demand for modern security systems is increasing. Traditional security systems mainly rely on "human defense", "technical defense" and "physical defense". "Human defense" means regular patrols and inspections by security personnel, "technical defense" mainly relies on fixed surveillance cameras with intelligent algorithms for remote video monitoring, and "physical defense" is achieved through physical security measures such as exterior walls, wire mesh, and fences. However, these traditional security methods have some shortcomings. For example, due to the limitations of the installation location and fixed viewing angle of fixed surveillance cameras, it is easy to form monitoring blind spots, and it is impossible to achieve full coverage of the monitoring area, which in turn affects the security effect.

[0003] In recent years, with the rapid development of cutting-edge technologies such as artificial intelligence, big data, cloud computing and the Internet of Things, these technologies have been widely used in various scenarios and are capable of more decision-making tasks. In order to meet the needs of modern intelligent security, the "robot + security" solution came into being. This solution uses a rail-mounted patrol robot to make up for the blind spots of the fixed surveillance camera's viewing angle, and provides a sensing method that the fixed surveillance camera does not have, so that the perimeter intrusion detection system can monitor intrusion incidents along the entire line. However, how to effectively combine the data between the rail-mounted patrol robot and the fixed surveillance camera to achieve inspection and personnel tracking and positioning is still a technical difficulty. Summary of the invention

[0004] The technical problem to be solved by the present invention is to provide a data processing method for a rail-mounted patrol robot and a fixed surveillance camera, so as to solve the problems that traditional security means easily form monitoring blind spots, cannot achieve full coverage of the monitoring area, and cannot effectively combine different data.

[0005] In order to solve the above technical problems, the technical solution of the present invention is: a data processing method of a rail-mounted patrol robot and a fixed surveillance camera provided by the present invention comprises at least one fixed surveillance camera and one rail-mounted patrol robot and a mobile camera arranged on the rail-mounted patrol robot, and is characterized in that it comprises the following steps:

[0006] S1. Real-time object detection and recognition of video streams captured by fixed surveillance cameras using deep learning models;

[0007] S2. When the target is detected, the target's bounding box coordinates in the image and the internal parameter matrix K of the fixed surveillance camera are used to calculate the homogeneous coordinates X of the target in the fixed surveillance camera coordinate system. fix ;

[0008] S3, using the known transformation matrix T from the fixed monitoring camera coordinate system to the mobile camera coordinate system, transforming the homogeneous coordinates of the target from the fixed monitoring camera coordinate system to the mobile camera coordinate system;

[0009] S4, according to the position of the target in the mobile camera coordinate system, use the Kalman filter to track the target and predict the next moment position of the target;

[0010] S5. According to the predicted position of the target, the moving distance of the rail-mounted patrol robot and the rotation angle of the mobile camera are adjusted to keep the target within the optimal observation field of view;

[0011] S6. Use the image captured by the mobile camera to update the position of the target and feed it back to the fixed monitoring camera coordinate system to achieve continuous tracking of the target.

[0012] In the preferred embodiment, steps S1 to S6 also include the following steps:

[0013] S101, using deep learning models to perform real-time target detection and recognition on video streams captured by fixed surveillance cameras;

[0014] Among them, the target recognition probability P = f DL (I), where I is the input image, f DL It is a deep learning model;

[0015] S102: When a target is detected, the target's bounding box coordinates in the image and the internal parameter matrix K of the fixed surveillance camera are used to calculate the homogeneous coordinates X of the target in the fixed surveillance camera coordinate system. fix ;

[0016] Among them, X fix = K·[x, y, w, h] T , where [x, y, w, h] are the normalized bounding box coordinates;

[0017] S103, using the known transformation matrix T from the fixed monitoring camera coordinate system to the mobile camera coordinate system, transform the homogeneous coordinates X fix Convert from the fixed surveillance camera coordinate system to the mobile camera coordinate system;

[0018] Among them, X pan =T·X fix , where X pan is the homogeneous coordinate of the target in the moving camera coordinate system, T is the change matrix;

[0019] S104, tracking the target using a Kalman filter according to the position of the target in the mobile camera coordinate system, and predicting the position of the target at the next moment, i.e., the predicted position;

[0020] in,

[0021] is the predicted current state, F is the state transfer matrix, B is the control matrix, u k is the control input;

[0022] S105, adjusting the moving distance of the rail-mounted patrol robot and the rotation angle of the mobile camera according to the predicted position of the target, so as to keep the target within the optimal observation field of view;

[0023] Where Δd = Δx·f x , Δθ=Δy·f y , Δd and Δθ are the adjusted moving distance and rotation angle respectively, Δx and Δy are the target position deviations, and f x and f y are the focal length and resolution parameters of the camera;

[0024] S106, using the image captured by the mobile camera to update the position of the target, and feeding it back to the fixed monitoring camera coordinate system to achieve continuous tracking of the target;

[0025] Among them, X fix，new =T -1 ·X pan , X fix , new is the updated homogeneous coordinate of the target in the fixed surveillance camera coordinate system.

[0026] In a preferred embodiment, a calibration process is used to determine a transformation matrix, which consists of two parts: a rotation matrix R and a translation vector t;

[0027] Calibration is achieved by placing calibration objects of known positions in the fields of view of the two cameras.

[0028] In the preferred embodiment, steps S1 to S6 also include the following steps:

[0029] S201: Camera internal parameter calibration: perform internal parameter calibration on the fixed surveillance camera and the mobile camera respectively to obtain the internal parameter matrix K of the fixed surveillance camera. fixed And the intrinsic parameter matrix K of the mobile camera pan ;

[0030] S202, placing a calibration object with a known position in the field of view of the two cameras, and recording its coordinates (Xo, Yo, Zo) in the world coordinate system;

[0031] S203, detecting corner points of the calibration object in each image captured by the camera, and calculating positions of these corner points in the image coordinate system;

[0032] S204, calculating homogeneous coordinates;

[0033] S205, constructing a transformation matrix;

[0034] S206, constructing a linear equation system using the homogeneous coordinates of the calibration object in the world coordinate system and the homogeneous coordinates of the calibration object in the two camera coordinate systems;

[0035] S207, solving the rotation matrix R and the translation vector t, and determining the transformation matrix T.

[0036] Another data processing method of a rail-mounted patrol robot and a fixed surveillance camera of the present invention comprises at least one fixed surveillance camera, one rail-mounted patrol robot, and a mobile camera and a radar system arranged on the rail-mounted patrol robot, and is characterized in that it comprises the following steps:

[0037] S01. Use deep learning models to perform real-time object detection and recognition on video streams captured by fixed surveillance cameras;

[0038] S02. When the target is detected, the radar coordinates are converted into homogeneous coordinates X in the fixed surveillance camera coordinate system. radar ; Using the bounding box coordinates of the target in the image and the internal parameter matrix K of the fixed surveillance camera, calculate the homogeneous coordinates X of the target in the fixed surveillance camera coordinate system fix ;

[0039] S03, combining radar and visual data, using weighted average method to fuse the two sets of data, and obtain the estimated position of the target in the fixed monitoring camera coordinate system;

[0040] S04, in the fixed monitoring camera coordinate system, using the prediction model to predict the next position of the target in the fixed monitoring camera coordinate system according to the fusion data;

[0041] S05, converting the first predicted position into the mobile camera coordinate system;

[0042] S06. In the mobile camera coordinate system, the prediction model is used again to predict the next position of the target;

[0043] S07, adjusting the moving distance of the rail-mounted patrol robot and the rotation angle of the mobile camera according to the re-predicted position of the target, so as to keep the target within the optimal observation field of view;

[0044] S08. Update the position of the target using the image captured by the mobile camera and feed it back to the fixed monitoring camera coordinate system to achieve continuous tracking of the target.

[0045] In the preferred embodiment, steps S01 to S08 also include the following steps:

[0046] S001. Use deep learning models to perform real-time target detection and recognition on video streams captured by fixed surveillance cameras;

[0047] Among them, the target recognition probability P = f DL (I), I is the input image, f DL It is a deep learning model;

[0048] S002, the radar detects a moving target and provides its speed and direction relative to the fixed surveillance camera coordinate system;

[0049] Radar data includes: target distance r, speed v, and angle θ r ;The fixed surveillance camera detects the target through the deep learning model and provides the bounding box coordinates of the target;

[0050] Convert the radar coordinates to homogeneous coordinates X in the fixed surveillance camera coordinate system radar ;

[0051]

[0052] S003. After the target is detected, the target's bounding box coordinates in the image and the internal parameter matrix K of the fixed surveillance camera are used to calculate the homogeneous coordinates X of the target in the fixed surveillance camera coordinate system. fix ;

[0053] Among them, X fix = K·[x, y, w, h] T ;

[0054] where [x, y, w, h] are the normalized bounding box coordinates;

[0055] S004. Combine radar and visual data and use a weighted average method to fuse the data of the two sensors to obtain the estimated position of the target in the coordinate system of the fixed monitoring camera;

[0056] Among them, X fusion =αX fix +(1-α)X radar ;

[0057] α is the weight coefficient, which ranges from 0 to 1 and is dynamically adjusted according to the reliability of the sensor. fusion represents the fused data, X fixrepresents the data from a fixed surveillance camera, X radar Represents data from radar;

[0058] S005. In the fixed surveillance camera coordinate system, use a Kalman filter or other prediction model to predict the next position of the target in the fixed surveillance camera coordinate system based on the fusion data;

[0059] Among them, the fused position X fusion As the measurement input of the Kalman filter, the Kalman filter is used to update the target state

[0060] is the updated state estimate, K k is the Kalman gain, Z k is the measured value, here is X fusion ; H is the observation matrix;

[0061] S006, converting the first predicted position in step S005 into the mobile camera coordinate system;

[0062] X pan =T·X fix ;

[0063] S007. In the moving camera coordinate system, the Kalman filter is used again to predict the next position of the target;

[0064] Current predictions need to take into account the dynamic characteristics of the mobile camera, including rotation speed and acceleration;

[0065] S008. According to the predicted position of the target in step S007, the moving distance of the rail-mounted patrol robot and the rotation angle of the mobile camera are adjusted to keep the target within the optimal observation field of view;

[0066] S009. Update the position of the target using the image captured by the mobile camera and feed it back to the fixed monitoring camera coordinate system to achieve continuous tracking of the target.

[0067] In a preferred embodiment, the prediction model includes one of a Kalman filter, a particle filter, a mean shift algorithm or an extended Kalman filter.

[0068] In a preferred embodiment, the position is predicted in the mobile camera coordinate system before being fed back to the fixed monitoring camera coordinate system, and the predicted position is fed back to the fixed monitoring camera coordinate system to achieve continuous tracking of the target.

[0069] In a preferred embodiment, a calibration process is used to determine a transformation matrix, which consists of two parts: a rotation matrix R and a translation vector t;

[0070] Calibration is achieved by placing calibration objects of known positions in the fields of view of the two cameras.

[0071] In the preferred embodiment, steps S01 to S08 also include the following steps:

[0072] S301, camera internal parameter calibration;

[0073] First, each camera needs to be calibrated internally to obtain their internal parameter matrix K; optionally, the fixed surveillance camera and the mobile camera are calibrated internally to obtain the internal parameter matrix K of the fixed surveillance camera. fixed And the intrinsic parameter matrix K of the mobile camera pan ;

[0074]

[0075] Among them, f x That is, f xfixed and f xpan and f y That is, f yfixed and f ypan Represents the focal length along the x-axis and y-axis of the image, respectively; used to describe the magnification of the camera lens;

[0076] c x and c y That is c xfixed ,c yfixed ,c xpan ,c ypan represents the principal point, that is, the position of the center of the image in the image coordinate system;

[0077] S302, coordinate measurement of the calibration object;

[0078] Place a calibration object with a known position in the field of view of the two cameras and record its coordinates (Xo, Yo, Zo) in the world coordinate system;

[0079] S303, extracting the image coordinates of the calibration object;

[0080] In each image captured by the camera, detect the corner points of the calibration object and calculate the positions of these corner points in the image coordinate system;

[0081] S304, calculating homogeneous coordinates;

[0082] For fixed surveillance cameras, the image coordinates (u, v) are converted to homogeneous coordinates X fix The formula is:

[0083] For a moving camera, the image coordinates (u′, v′) are converted to homogeneous coordinates X pan The formula is:

[0084] The formula for calculating the homogeneous coordinates of the calibration object in the world coordinate system is:

[0085]

[0086] S305, constructing a transformation matrix;

[0087] When converting between two camera coordinate systems, it is necessary to determine the transformation matrix T, which includes the rotation matrix R and the translation vector t;

[0088] Use the homogeneous coordinates of the calibration object in the world coordinate system and the homogeneous coordinates of the calibration object in the two camera coordinate systems to construct a linear equation system:

[0089] X fix =RX world +t

[0090] X pan =RX world +t;

[0091] S306, solving the rotation matrix and translation vector;

[0092] Solve for R and t, using the coordinates of multiple calibration objects;

[0093] If there are n calibration objects, 2n equations are constructed and solved by the least squares method or singular value decomposition;

[0094] Combine all equations into a matrix form AX=B, where A is composed of X fix and X pan The matrix is ​​composed of X, which contains R and t, and B is the matrix composed of X world The matrix formed; use SVD to solve:

[0095] A=UΣVT

[0096] X=VΣ-1UTB

[0097] Extract the rotation matrix R and translation vector t from the solved X matrix;

[0098] S307, transformation matrix verification;

[0099] In the actual monitoring scenario, the transformation matrix T is used to transform the target position in the fixed monitoring camera coordinate system to the mobile camera coordinate system, and a tracking test is performed to verify its accuracy.

[0100] The invention provides a data processing method for a rail-mounted patrol robot and a fixed monitoring camera, comprising at least one fixed monitoring camera, one rail-mounted patrol robot and a mobile camera arranged on the rail-mounted patrol robot, and the method comprises the following steps: using a deep learning model to perform real-time target detection and recognition on a video stream captured by the fixed monitoring camera; when a target is detected, using the bounding box coordinates of the target in the image and the internal reference matrix K of the fixed monitoring camera to calculate the homogeneous coordinates Xfix of the target in the fixed monitoring camera coordinate system; using the known transformation matrix T from the fixed monitoring camera coordinate system to the mobile camera coordinate system, converting the homogeneous coordinates of the target from the fixed monitoring camera coordinate system to the mobile camera coordinate system; according to the position of the target in the mobile camera coordinate system, using a Kalman filter to track the target and predict the position of the target at the next moment; according to the predicted position of the target, adjusting the moving distance of the rail-mounted patrol robot and the rotation angle of the mobile camera to keep the target within the optimal observation field of view; using an image taken by the mobile camera to update the position of the target, and feeding back to the fixed monitoring camera coordinate system to achieve continuous tracking of the target. Another data processing method for a rail-mounted patrol robot and a fixed surveillance camera provided by the present invention combines radar and visual data, uses a weighted average method to fuse the two sets of data, improves the accuracy of target position estimation, adjusts the moving distance of the rail-mounted patrol robot and the rotation angle of the mobile camera according to the predicted position, and ensures that the target is always within the optimal observation field of view. By feeding back and updating the target position, continuous tracking and monitoring of the target is achieved. BRIEF DESCRIPTION OF THE DRAWINGS

[0101] The present invention will be further described below in conjunction with the accompanying drawings and embodiments:

[0102] Figure 1 A flow chart of a data processing method for a rail-mounted patrol robot and a fixed surveillance camera of the present invention;

[0103] Figure 2 This is a flow chart of step S2 of embodiment 1 of the present invention;

[0104] Figure 3 It is a flow chart of another data processing method of the rail-mounted patrol robot and the fixed surveillance camera of the present invention;

[0105] Figure 4 This is a flow chart of Example 4 of the present invention. DETAILED DESCRIPTION

[0106] Embodiment 1:

[0107] like Figure 1A data processing method for a rail-mounted patrol robot and a fixed surveillance camera comprises at least one fixed surveillance camera, one rail-mounted patrol robot and a mobile camera arranged on the rail-mounted patrol robot. The method comprises the following steps:

[0108] S1. Real-time object detection and recognition of video streams captured by fixed surveillance cameras using deep learning models;

[0109] S2. When the target is detected, the target's bounding box coordinates in the image and the internal parameter matrix K of the fixed surveillance camera are used to calculate the homogeneous coordinates X of the target in the fixed surveillance camera coordinate system. fix ;

[0110] S3, using the known transformation matrix T from the fixed monitoring camera coordinate system to the mobile camera coordinate system, transforming the homogeneous coordinates of the target from the fixed monitoring camera coordinate system to the mobile camera coordinate system;

[0111] S4, according to the position of the target in the mobile camera coordinate system, use the Kalman filter to track the target and predict the next moment position of the target;

[0112] S5. According to the predicted position of the target, the moving distance of the rail-mounted patrol robot and the rotation angle of the mobile camera are adjusted to keep the target within the optimal observation field of view;

[0113] S6. Use the image captured by the mobile camera to update the position of the target and feed it back to the fixed monitoring camera coordinate system to achieve continuous tracking of the target.

[0114] In the preferred embodiment, steps S1 to S6 also include the following steps:

[0115] S101, using deep learning models to perform real-time target detection and recognition on video streams captured by fixed surveillance cameras;

[0116] Among them, the target recognition probability P = f DL (I), where I is the input image, f DL It is a deep learning model;

[0117] S102: When a target is detected, the target's bounding box coordinates in the image and the internal parameter matrix K of the fixed surveillance camera are used to calculate the homogeneous coordinates X of the target in the fixed surveillance camera coordinate system. fix ;

[0118] Among them, X fix =K·[x,y,ω,h] T , where [x, y, w, h] are the normalized bounding box coordinates;

[0119] S103, using the known transformation matrix T from the fixed monitoring camera coordinate system to the mobile camera coordinate system, transform the homogeneous coordinates X fix Convert from the fixed surveillance camera coordinate system to the mobile camera coordinate system;

[0120] Among them, X pan =T·X fix , where X pan is the homogeneous coordinate of the target in the moving camera coordinate system, T is the change matrix;

[0121] S104, tracking the target using a Kalman filter according to the position of the target in the mobile camera coordinate system, and predicting the position of the target at the next moment, i.e., the predicted position;

[0122] in,

[0123] is the predicted current state, F is the state transfer matrix, B is the control matrix, u k is the control input;

[0124] S105, adjusting the moving distance of the rail-mounted patrol robot and the rotation angle of the mobile camera according to the predicted position of the target, so as to keep the target within the optimal observation field of view;

[0125] Where Δd = Δx·f x , Δθ=Δy·f y , Δd and Δθ are the adjusted moving distance and rotation angle respectively, Δx and Δy are the target position deviations, and f x and f y are the focal length and resolution parameters of the camera;

[0126] S106, using the image captured by the mobile camera to update the position of the target, and feeding it back to the fixed monitoring camera coordinate system to achieve continuous tracking of the target;

[0127] Among them, X fix，new =T-1·X pan , X fix , new is the updated homogeneous coordinate of the target in the fixed surveillance camera coordinate system.

[0128] In a preferred embodiment, a calibration process is used to determine a transformation matrix, which consists of two parts: a rotation matrix R and a translation vector t;

[0129] Calibration is achieved by placing calibration objects of known positions in the field of view of the two cameras. This solution, through a precise calibration process, can ensure the accuracy of the transformation matrix T between the two cameras, allowing the system to respond to the movement of the target more quickly, improve the accuracy of target positioning, enable the system to better adapt to different monitoring scenarios and environmental changes, and enhance the robustness of the system.

[0130] In the preferred embodiment, steps S1 to S6 also include the following steps:

[0131] S201: Camera internal parameter calibration: perform internal parameter calibration on the fixed surveillance camera and the mobile camera respectively to obtain the internal parameter matrix K of the fixed surveillance camera. fixed And the intrinsic parameter matrix K of the mobile camera pan ;

[0132] S202, placing a calibration object with a known position in the field of view of the two cameras, and recording its coordinates (Xo, Yo, Zo) in the world coordinate system;

[0133] S203, detecting corner points of the calibration object in each image captured by the camera, and calculating positions of these corner points in the image coordinate system;

[0134] S204, calculating homogeneous coordinates;

[0135] S205, constructing a transformation matrix;

[0136] S206, constructing a linear equation system using the homogeneous coordinates of the calibration object in the world coordinate system and the homogeneous coordinates of the calibration object in the two camera coordinate systems;

[0137] S207, solving the rotation matrix R and the translation vector t, and determining the transformation matrix T.

[0138] Embodiment 2:

[0139] like Figure 2 As shown, another data processing method for a rail-mounted patrol robot and a fixed surveillance camera comprises at least one fixed surveillance camera, one rail-mounted patrol robot, and a mobile camera and a radar system arranged on the rail-mounted patrol robot, and is characterized by comprising the following steps:

[0140] S01. Use deep learning models to perform real-time object detection and recognition on video streams captured by fixed surveillance cameras;

[0141] S02. When the target is detected, the radar coordinates are converted into homogeneous coordinates X in the fixed surveillance camera coordinate system. radar ; Use the bounding box coordinates of the target in the image and the internal parameter matrix K of the fixed surveillance camera to calculate the homogeneous coordinates X of the target in the fixed surveillance camera coordinate system fix ;

[0142] S03, combining radar and visual data, using weighted average method to fuse the two sets of data, and obtain the estimated position of the target in the fixed monitoring camera coordinate system;

[0143] S04, in the fixed monitoring camera coordinate system, using the prediction model to predict the next position of the target in the fixed monitoring camera coordinate system according to the fusion data;

[0144] S05, converting the first predicted position into the mobile camera coordinate system;

[0145] S06. In the mobile camera coordinate system, the prediction model is used again to predict the next position of the target;

[0146] S07, adjusting the moving distance of the rail-mounted patrol robot and the rotation angle of the mobile camera according to the re-predicted position of the target, so as to keep the target within the optimal observation field of view;

[0147] S08. Update the position of the target using the image captured by the mobile camera and feed it back to the fixed monitoring camera coordinate system to achieve continuous tracking of the target.

[0148] In the preferred embodiment, steps S01 to S08 also include the following steps:

[0149] S001. Use deep learning models to perform real-time target detection and recognition on video streams captured by fixed surveillance cameras;

[0150] Among them, the target recognition probability P = f DL (I), I is the input image, f DL It is a deep learning model;

[0151] S002, the radar detects a moving target and provides its speed and direction relative to the fixed surveillance camera coordinate system;

[0152] Radar data includes: target distance r, speed v, and angle θ r ;The fixed surveillance camera detects the target through the deep learning model and provides the bounding box coordinates of the target;

[0153] Convert the radar coordinates to homogeneous coordinates X in the fixed surveillance camera coordinate system radar ;

[0154]

[0155] S003. After the target is detected, the target's bounding box coordinates in the image and the internal parameter matrix K of the fixed surveillance camera are used to calculate the homogeneous coordinates X of the target in the fixed surveillance camera coordinate system. fix ;

[0156] Among them, X fix = K·[x, y, w, h] T ;

[0157] where [x, y, w, h] are the normalized bounding box coordinates;

[0158] S004. Combine radar and visual data and use a weighted average method to fuse the data of the two sensors to obtain the estimated position of the target in the coordinate system of the fixed monitoring camera;

[0159] Among them, X fusion =αX fix +(1-α)X radar ;

[0160] α is the weight coefficient, which ranges from 0 to 1 and is dynamically adjusted according to the reliability of the sensor. fusion represents the fused data, X fix represents the data from a fixed surveillance camera, X radar Represents data from radar;

[0161] S005. In the fixed surveillance camera coordinate system, use a Kalman filter or other prediction model to predict the next position of the target in the fixed surveillance camera coordinate system based on the fusion data;

[0162] Among them, the fused position X fusion As the measurement input of the Kalman filter, the Kalman filter is used to update the target state

[0163] is the updated state estimate, K k is the Kalman gain, Z k is the measured value, here is X fusion ; H is the observation matrix;

[0164] S006, converting the first predicted position in step S005 into the mobile camera coordinate system;

[0165] X pan =TX fix ;

[0166] S007. In the moving camera coordinate system, the Kalman filter is used again to predict the next position of the target;

[0167] Current predictions need to take into account the dynamic characteristics of the mobile camera, including rotation speed and acceleration;

[0168] S008. According to the predicted position of the target in step S007, the moving distance of the rail-mounted patrol robot and the rotation angle of the mobile camera are adjusted to keep the target within the optimal observation field of view;

[0169] S009. Update the position of the target using the image captured by the mobile camera and feed it back to the fixed monitoring camera coordinate system to achieve continuous tracking of the target.

[0170] Embodiment 3:

[0171] As further described in conjunction with Example 2, in a preferred embodiment, the prediction model includes one of a Kalman filter, a particle filter, a mean shift algorithm or an extended Kalman filter.

[0172] According to this scheme, preferably, the Kalman filter adopts an extended Kalman filter, which approximates the nonlinear state equation and observation equation to linear equations by performing first-order Taylor expansion near the current estimation point. First, for the nonlinear state equation and observation equation, they are approximated to linear equations by Taylor series expansion, and generally only the first-order Taylor expansion terms are retained, thereby converting the nonlinear problem into a linear problem, so that the Kalman filter algorithm can be applied to nonlinear systems; secondly, the state equation and observation equation are Taylor expanded at the current estimation point and the first-order approximation is taken to obtain the linearized state equation and observation equation. The linearized state equation is used to predict the prior estimate of the next state and calculate its covariance matrix. The state estimate is updated using the observed data, including calculating the observation residual, the covariance matrix of the observation residual and the Kalman gain, and then the state estimate is updated and the covariance matrix is ​​updated.

[0173] Preferably, the particle filter represents the distribution of the target state by randomly sampling a large number of particles, each particle represents a possible state, and is assigned a weight to measure its matching degree with the observed data. Through continuous iterative prediction, weight update and resampling, the particle set gradually converges to the target's true state distribution, and finally obtains the estimated value of the target state by weighted average of the particles. At the initial moment, a group of particles is randomly generated, and an initial weight is assigned to each particle. The particles are evenly distributed in the state space, or initialized according to prior knowledge; the state of each particle is updated according to the dynamic model to simulate the evolution of the target in time; when new observation data is obtained, the weight of each particle is calculated according to the observation model, and the weight reflects the consistency between the particle and the observation data, and is calculated using the likelihood function; the larger the value of the likelihood function, the more likely the state represented by the particle is to generate the current observation data; the particle set is resampled according to the particle weight, the particles with larger weights are retained, and the particles with smaller weights are eliminated, so that the particle set is concentrated in the high probability area; the estimated value of the target state is obtained by weighted average of the resampled particles.

[0174] Preferably, the mean shift algorithm is used to find the density peak in the data distribution through density estimation, so as to identify the center of the data cluster. First, randomly select a data point in the data set as the initial center point; with the current center point as the center, determine a neighborhood range according to the set bandwidth parameter, and find all data points in the neighborhood; calculate the vectors from all data points in the neighborhood to the center point, and average them to obtain the offset vector, and weight the data points in the neighborhood according to the kernel function so that the data points closer to the center point contribute more to the offset vector; move the center point a certain distance along the direction of the offset vector to obtain a new center point; determine whether the convergence condition is met, that is, whether the moving distance of the center point is less than the preset threshold; if the convergence condition is not met, return to continue iterating; if the convergence condition is met, the current center point is a density peak point, representing the center of a data cluster; repeat the above steps until all data points are marked as visited, that is, all data cluster centers are found. Finally, the data points are classified according to the cluster center to which each data point belongs, so as to achieve the prediction of the target position.

[0175] In a preferred embodiment, the position is predicted in the mobile camera coordinate system before being fed back to the fixed monitoring camera coordinate system, and the predicted position is fed back to the fixed monitoring camera coordinate system to achieve continuous tracking of the target.

[0176] In a preferred embodiment, a calibration process is used to determine a transformation matrix, which consists of two parts: a rotation matrix R and a translation vector t; the calibration is achieved by placing calibration objects with known positions in the fields of view of the two cameras.

[0177] Embodiment 4:

[0178] Further described in conjunction with Examples 2 and 3, in the preferred solution, steps S01 to S08 also include the following steps:

[0179] S301, camera internal parameter calibration;

[0180] First, each camera needs to be calibrated internally to obtain their internal parameter matrix K; optionally, the fixed surveillance camera and the mobile camera are calibrated internally to obtain the internal parameter matrix K of the fixed surveillance camera. fixed And the intrinsic parameter matrix K of the mobile camera pan ;

[0181]

[0182] Among them, f x That is, f xfixed and f xpan and f y That is, f yfixed and f ypanRepresents the focal length along the x-axis and y-axis of the image, respectively; used to describe the magnification of the camera lens;

[0183] c x and c y That is c xfixed ,c yfixed ,c xpan ,c ypan represents the principal point, that is, the position of the center of the image in the image coordinate system;

[0184] S302, coordinate measurement of the calibration object;

[0185] Place a calibration object with a known position in the field of view of the two cameras and record its coordinates (Xo, Yo, Zo) in the world coordinate system;

[0186] S303, extracting the image coordinates of the calibration object;

[0187] In each image captured by the camera, detect the corner points of the calibration object and calculate the positions of these corner points in the image coordinate system;

[0188] S304, calculating homogeneous coordinates;

[0189] For fixed surveillance cameras, the image coordinates (u, v) are converted to homogeneous coordinates X fix The formula is:

[0190] For a moving camera, the image coordinates (u′, v′) are converted to homogeneous coordinates X pan The formula is:

[0191] The formula for calculating the homogeneous coordinates of the calibration object in the world coordinate system is:

[0192]

[0193] S305, constructing a transformation matrix;

[0194] When converting between two camera coordinate systems, it is necessary to determine the transformation matrix T, which includes the rotation matrix R and the translation vector t;

[0195] Use the homogeneous coordinates of the calibration object in the world coordinate system and the homogeneous coordinates of the calibration object in the two camera coordinate systems to construct a linear equation system:

[0196] X fix =RX world +t

[0197] X pan =RX world +t;

[0198] S306, solving the rotation matrix and translation vector;

[0199] Solve for R and t, using the coordinates of multiple calibration objects;

[0200] If there are n calibration objects, 2n equations are constructed and solved by the least squares method or singular value decomposition;

[0201] Combine all equations into a matrix form AX=B, where A is composed of X fix and X pan The matrix is ​​composed of X, which contains R and t, and B is the matrix composed of X world The matrix formed; use SVD to solve:

[0202] A=UΣVT

[0203] X=VΣ-1UTB

[0204] Extract the rotation matrix R and translation vector t from the solved X matrix;

[0205] S307, transformation matrix verification;

[0206] In the actual monitoring scenario, the transformation matrix T is used to transform the target position in the fixed monitoring camera coordinate system to the mobile camera coordinate system, and a tracking test is performed to verify its accuracy.

[0207] The above-mentioned embodiments are only preferred technical solutions of the present invention and should not be regarded as limitations of the present invention. The embodiments and features in the embodiments of the present application can be arbitrarily combined with each other without conflict. The protection scope of the present invention shall be the technical solutions recorded in the claims, including the equivalent replacement solutions of the technical features in the technical solutions recorded in the claims. That is, equivalent replacement improvements within this scope are also within the protection scope of the present invention.

Claims

1. A data processing method for a rail-mounted patrol robot and a fixed surveillance camera, comprising at least one fixed surveillance camera, one rail-mounted patrol robot and a mobile camera arranged on the rail-mounted patrol robot, characterized in that The following steps are involved: S1. Real-time object detection and recognition of video streams captured by fixed surveillance cameras using deep learning models; S2. When the target is detected, the target's bounding box coordinates in the image and the internal parameter matrix K of the fixed surveillance camera are used to calculate the homogeneous coordinates X of the target in the fixed surveillance camera coordinate system. fix ; S3, using the known transformation matrix T from the fixed monitoring camera coordinate system to the mobile camera coordinate system, transforming the homogeneous coordinates of the target from the fixed monitoring camera coordinate system to the mobile camera coordinate system; S4, according to the position of the target in the mobile camera coordinate system, use the Kalman filter to track the target and predict the next moment position of the target; S5. According to the predicted position of the target, the moving distance of the rail-mounted patrol robot and the rotation angle of the mobile camera are adjusted to keep the target within the optimal observation field of view; S6. Use the image captured by the mobile camera to update the position of the target and feed it back to the fixed monitoring camera coordinate system to achieve continuous tracking of the target.

2. The data processing method of a rail-mounted patrol robot and a fixed surveillance camera according to claim 1, characterized in that Steps S1 to S6 also include the following steps: S101, using deep learning models to perform real-time target detection and recognition on video streams captured by fixed surveillance cameras; Among them, the target recognition probability P = f DL (I), where I is the input image, f DL It is a deep learning model; S102: When a target is detected, the target's bounding box coordinates in the image and the internal parameter matrix K of the fixed surveillance camera are used to calculate the homogeneous coordinates X of the target in the fixed surveillance camera coordinate system. fix ; Among them, X fix = K·[x, y, w, h] T , where [x, y, w, h] are the normalized bounding box coordinates; S103, using the known transformation matrix T from the fixed monitoring camera coordinate system to the mobile camera coordinate system, transform the homogeneous coordinates X fix Convert from the fixed surveillance camera coordinate system to the mobile camera coordinate system; Among them, X pan =T·X fix , where X pan is the homogeneous coordinate of the target in the moving camera coordinate system, T is the change matrix; S104, tracking the target using a Kalman filter according to the position of the target in the mobile camera coordinate system, and predicting the position of the target at the next moment, i.e., the predicted position; in, is the predicted current state, F is the state transfer matrix, B is the control matrix, u k is the control input; S105, adjusting the moving distance of the rail-mounted patrol robot and the rotation angle of the mobile camera according to the predicted position of the target, so as to keep the target within the optimal observation field of view; Where Δd = Δx·f x , Δθ=Δy·f y , Δd and Δθ are the adjusted moving distance and rotation angle respectively, Δx and Δy are the target position deviations, and f x and f y are the focal length and resolution parameters of the camera; S106, using the image captured by the mobile camera to update the position of the target, and feeding it back to the fixed monitoring camera coordinate system to achieve continuous tracking of the target; Among them, X fix，new =T-1·X pan , X fix , new is the updated homogeneous coordinate of the target in the fixed surveillance camera coordinate system.

3. The data processing method of a rail-mounted patrol robot and a fixed surveillance camera according to claim 1 is characterized in that: A calibration process is used to determine the transformation matrix, which consists of two parts: the rotation matrix R and the translation vector t; Calibration is achieved by placing calibration objects of known positions in the fields of view of the two cameras.

4. The data processing method of a rail-mounted patrol robot and a fixed surveillance camera according to claim 1, characterized in that Steps S1 to S6 also include the following steps: S201: Camera internal parameter calibration: perform internal parameter calibration on the fixed surveillance camera and the mobile camera respectively to obtain the internal parameter matrix K of the fixed surveillance camera. fixed And the intrinsic parameter matrix K of the mobile camera pan ; S202, placing a calibration object with a known position in the field of view of the two cameras, and recording its coordinates (Xo, Yo, Zo) in the world coordinate system; S203, detecting corner points of the calibration object in each image captured by the camera, and calculating positions of these corner points in the image coordinate system; S204, calculating homogeneous coordinates; S205, constructing a transformation matrix; S206, constructing a linear equation system using the homogeneous coordinates of the calibration object in the world coordinate system and the homogeneous coordinates of the calibration object in the two camera coordinate systems; S207, solving the rotation matrix R and the translation vector t, and determining the transformation matrix T.

5. A data processing method for a rail-mounted patrol robot and a fixed surveillance camera, comprising at least one fixed surveillance camera, one rail-mounted patrol robot, and a mobile camera and a radar system arranged on the rail-mounted patrol robot, characterized in that The following steps are involved: S01. Use deep learning models to perform real-time object detection and recognition on video streams captured by fixed surveillance cameras; S02. When the target is detected, the radar coordinates are converted into homogeneous coordinates X in the fixed surveillance camera coordinate system. radar ; Using the bounding box coordinates of the target in the image and the intrinsic parameter matrix K of the fixed surveillance camera, calculate the homogeneous coordinates X of the target in the fixed surveillance camera coordinate system fix ; S03, combining radar and visual data, using weighted average method to fuse the two sets of data, and obtain the estimated position of the target in the fixed monitoring camera coordinate system; S04, in the fixed monitoring camera coordinate system, using the prediction model to predict the next position of the target in the fixed monitoring camera coordinate system according to the fusion data; S05, converting the first predicted position into the mobile camera coordinate system; S06. In the mobile camera coordinate system, the prediction model is used again to predict the next position of the target; S07, adjusting the moving distance of the rail-mounted patrol robot and the rotation angle of the mobile camera according to the re-predicted position of the target, so as to keep the target within the optimal observation field of view; S08. Update the position of the target using the image captured by the mobile camera and feed it back to the fixed monitoring camera coordinate system to achieve continuous tracking of the target.

6. The data processing method of a rail-mounted patrol robot and a fixed surveillance camera according to claim 5, characterized in that the steps S01 to S08 also include the following steps: S001. Use deep learning models to perform real-time target detection and recognition on video streams captured by fixed surveillance cameras; Among them, the target recognition probability P = f DL (I), I is the input image, f DL It is a deep learning model; S002, the radar detects a moving target and provides its speed and direction relative to the fixed surveillance camera coordinate system; Radar data includes: target distance r, speed v, and angle θ r ;The fixed surveillance camera detects the target through the deep learning model and provides the bounding box coordinates of the target; Convert the radar coordinates to homogeneous coordinates X in the fixed surveillance camera coordinate system radar ; S003. After the target is detected, the target's bounding box coordinates in the image and the internal parameter matrix K of the fixed surveillance camera are used to calculate the homogeneous coordinates X of the target in the fixed surveillance camera coordinate system. fix ; Among them, X fix = K·[x, y, w, h] T ; where [x, y, w, h] are the normalized bounding box coordinates; S004. Combine radar and visual data and use a weighted average method to fuse the data of the two sensors to obtain the estimated position of the target in the coordinate system of the fixed monitoring camera; Among them, X fusion =αX fix +(1-α)X radar ; α is the weight coefficient, which ranges from 0 to 1 and is dynamically adjusted according to the reliability of the sensor. fusion represents the fused data, X fix represents the data from a fixed surveillance camera, X radar Represents data from radar; S005. In the fixed surveillance camera coordinate system, use a Kalman filter or other prediction model to predict the next position of the target in the fixed surveillance camera coordinate system based on the fusion data; Among them, the fused position X fusion As the measurement input of the Kalman filter, the Kalman filter is used to update the target state is the updated state estimate, K k is the Kalman gain, Z k is the measured value, here is X fusion ; H is the observation matrix; S006, converting the first predicted position in step S005 into the mobile camera coordinate system; X pan =T·X fix ; S007. In the moving camera coordinate system, the Kalman filter is used again to predict the next position of the target; Current predictions need to take into account the dynamic characteristics of the mobile camera, including rotation speed and acceleration; S008. According to the predicted position of the target in step S007, the moving distance of the rail-mounted patrol robot and the rotation angle of the mobile camera are adjusted to keep the target within the optimal observation field of view; S009. Update the position of the target using the image captured by the mobile camera and feed it back to the fixed monitoring camera coordinate system to achieve continuous tracking of the target.

7. The data processing method of a rail-mounted patrol robot and a fixed surveillance camera according to claim 5 is characterized in that: The prediction model includes one of a Kalman filter, a particle filter, a mean shift algorithm or an extended Kalman filter.

8. The data processing method of a rail-mounted patrol robot and a fixed surveillance camera according to claim 5 is characterized by: Before feeding back to the fixed monitoring camera coordinate system, the position is predicted in the mobile camera coordinate system, and the predicted position is fed back to the fixed monitoring camera coordinate system to achieve continuous tracking of the target.

9. The data processing method of a rail-mounted patrol robot and a fixed surveillance camera according to claim 5, characterized in that: A calibration process is used to determine the transformation matrix, which consists of two parts: the rotation matrix R and the translation vector t; Calibration is achieved by placing calibration objects of known positions in the fields of view of the two cameras.

10. The data processing method of a rail-mounted patrol robot and a fixed surveillance camera according to claim 9, characterized in that the steps S01 to S08 also include the following steps: S301, camera internal parameter calibration; First, each camera needs to be calibrated internally to obtain their internal parameter matrix K; optionally, the fixed surveillance camera and the mobile camera are calibrated internally to obtain the internal parameter matrix K of the fixed surveillance camera. fixed And the intrinsic parameter matrix K of the mobile camera pan ; Among them, f x That is, f xfixed and f xpan and f y That is, f yfixed and f ypan Represents the focal length along the x-axis and y-axis of the image, respectively; used to describe the magnification of the camera lens; c x and c y That is c xfixed ,c yfixed ,c xpan ,c ypan represents the principal point, that is, the position of the center of the image in the image coordinate system; S302, coordinate measurement of the calibration object; Place a calibration object with a known position in the field of view of the two cameras and record its coordinates (Xo, Yo, Zo) in the world coordinate system; S303, extracting the image coordinates of the calibration object; In each image captured by the camera, detect the corner points of the calibration object and calculate the positions of these corner points in the image coordinate system; S304, calculating homogeneous coordinates; For fixed surveillance cameras, the image coordinates (u, v) are converted to homogeneous coordinates X fix The formula is: For a mobile camera, the image coordinates (u′, v′) are converted to homogeneous coordinates Xpa n The formula is: The formula for calculating the homogeneous coordinates of the calibration object in the world coordinate system is: S305, constructing a transformation matrix; When converting between two camera coordinate systems, it is necessary to determine the transformation matrix T, which includes the rotation matrix R and the translation vector t; Use the homogeneous coordinates of the calibration object in the world coordinate system and the homogeneous coordinates of the calibration object in the two camera coordinate systems to construct a linear equation system: X fix =RX world +t X pan =RX world +t; S306, solving the rotation matrix and translation vector; Solve for R and t, using the coordinates of multiple calibration objects; If there are n calibration objects, 2n equations are constructed and solved by the least squares method or singular value decomposition; Combine all equations into a matrix form AX=B, where A is composed of X fix and X pan The matrix is ​​composed of X, which contains R and t, and B is the matrix composed of X world The matrix formed; use SVD to solve: A=UΣVT X=VΣ-1UTB Extract the rotation matrix R and translation vector t from the solved X matrix; S307, transformation matrix verification; In the actual monitoring scenario, the transformation matrix T is used to transform the target position in the fixed monitoring camera coordinate system to the mobile camera coordinate system, and a tracking test is performed to verify its accuracy.

Images