Target tracking method and device, underwater accompanying robot and storage medium

Abstract

The invention discloses a target tracking method and device, an underwater accompanying robot and a storage medium, and belongs to the technical field of underwater accompanying robots. The target tracking method comprises the following steps: acquiring an underwater video stream through a binocular stereoscopic vision camera, and carrying out preprocessing such as defogging, sharpening and denoising on an image; generating a depth-of-field map by using a binocular matching algorithm to determine the three-dimensional space position of the target; detecting divers by adopting a YOLO target detection algorithm, and performing unique identity confirmation in combination with active mark point detection; through information fusion, a depth-of-field map, a target detection result and a modulation optical signal detection result are integrated, the spatial position and the movement speed of a target are accurately calculated, and the information is transmitted to an underwater robot control system, so that high-precision target tracking is realized. According to the method, challenges caused by target fuzziness and dressing similarity in the underwater environment are effectively solved, and the accuracy and reliability of target tracking are improved.

Description

Technical Field

[0001] The present invention relates to the technical field of underwater companion robots, and more specifically, to a target tracking method and device, an underwater companion robot and a storage medium. Background Art

[0002] In underwater environments, target tracking technology faces many challenges. The particularities of the underwater environment, such as the attenuation of light and the interference of impurities in the water, result in poor image quality, color distortion, blur, and low contrast. This makes it difficult for traditional target detection and tracking methods to effectively identify and track underwater targets. In addition, targets in underwater environments usually have similar appearances, such as the similarity of divers' clothing, which further increases the complexity of target identification and tracking. Therefore, the target tracking capabilities of existing technologies in such environments are obviously insufficient.

[0003] At present, although there are a variety of target detection algorithms such as YOLO (You Only Look Once) and SSD (Single Shot MultiBox Detector) that can provide good target recognition capabilities under normal lighting conditions, their effectiveness is greatly reduced in underwater environments. These algorithms rely on clear image features, but underwater, images are usually limited by light propagation and impurities in the water, which affects the clarity of the target and the accuracy of feature extraction. At the same time, the target tracking algorithm needs to maintain the tracking of the target in a continuous image sequence, but underwater, due to image quality limitations, traditional tracking algorithms may be difficult to work effectively, especially when the target is blurred in the image or blocked by other objects.

[0004] To solve these problems, current technical solutions usually rely on advanced image processing and deep learning technologies, but they still cannot fully solve all the challenges in underwater target tracking. The existing technologies for target tracking in underwater environments still have problems such as insufficient accuracy, high algorithm complexity, and large computing resource requirements. Therefore, there is an urgent need for an improved technical solution to improve the accuracy and robustness of target tracking in underwater environments. Summary of the invention

[0005] The present invention provides a target tracking method and device, an underwater companion robot and a storage medium to solve the technical problem of accurately tracking a fuzzy target in an underwater environment.

[0006] In order to solve the above technical problems, the technical solution of the present invention is as follows:

[0007] A target tracking method, wherein a light emitting device capable of emitting a modulated light signal is placed on a tracked target, and the method comprises the following steps:

[0008] A binocular camera is used to acquire a video sequence containing continuous frames, and image preprocessing is performed on the video sequence, wherein the image preprocessing includes sharpening processing, defogging processing and denoising processing to obtain a preprocessed image sequence;

[0009] Three processing threads are started simultaneously on the preprocessed image sequence. The first processing thread uses a binocular matching algorithm to generate a depth map; the second processing thread uses a target detection algorithm and a target tracking algorithm to identify and track the target, and generates bounding box information containing the ID of the tracked target; the third processing thread uses a modulated light detection algorithm to obtain the center position of the light-emitting device;

[0010] The depth map generated by the three processing threads, the bounding box information containing the tracked target ID, and the center position of the light-emitting device are fused, the center position of the light-emitting device and the bounding box position of the target are marked on the depth map, and the spatial position and speed of the tracked target in the fused information are calculated to achieve target tracking.

[0011] Further, the step of obtaining a video sequence containing continuous frames includes:

[0012] When the system leaves the factory, the binocular cameras are calibrated to make the viewing angles of the two cameras consistent, and the intrinsic and extrinsic parameters of the cameras are corrected;

[0013] The intrinsic parameters of the camera include focal length, principal point and distortion coefficient, and the extrinsic parameters of the camera include rotation matrix and translation vector; the rotation matrix is ​​a 3x3 matrix used to describe the rotation of the camera coordinate system relative to the world coordinate system; the translation vector is a 3x1 vector used to describe the translation of the camera coordinate system relative to the world coordinate system;

[0014] Use a calibrated stereo camera to capture a video sequence containing continuous frames, where each pair of frames is taken by the left camera and the right camera simultaneously for subsequent stereo matching processing.

[0015] Further, the step of performing image preprocessing on the video sequence includes:

[0016] Each frame in the acquired video sequence is preprocessed and a sharpening filter is applied to enhance the edge details in the image to improve the clarity of the target; the formula for enhancing the edge of the image is:

[0017]

[0018] Among them, I sharp (x, y) is the sharpened image, I raw (x,y) is the original image, α is the sharpening factor, is the Laplace operator; x represents the horizontal coordinate in the image, and y represents the vertical coordinate in the image;

[0019] The haze effect in the image is removed by estimating the dark channel and atmospheric light of the image to improve the contrast and visibility of the image. The specific formula is:

[0020]

[0021] Among them, I d (x, y) is the image after dehazing, I raw (x, y) is the original image, A is the atmospheric light estimation value, t(x, y) is the transmittance; x represents the horizontal coordinate in the image, and y represents the vertical coordinate in the image;

[0022] Gaussian filter is used to smooth the image and reduce the noise in the image to reduce the interference to subsequent processing.

[0023] The specific formula is:

[0024]

[0025] Among them, I denoised (x, y) is the pixel value of the denoised image at position (x, y); I raw (x′, y′) is the pixel value of the original image at position (x′, y′); σ is the standard deviation of the Gaussian filter, which is used to control the smoothness of the filter; x and y are the pixel coordinates of the denoised image; x′ and y′ are the pixel coordinates of the original image.

[0026] Furthermore, the specific steps of the first processing thread generating a depth map by using a binocular matching algorithm include:

[0027] Capture images from a pair of calibrated stereo cameras;

[0028] The image is subjected to stereo correction to obtain a corrected left image I L (x,y) and right image I R (x,y);

[0029] Among them, x represents the horizontal coordinate in the image, and y represents the vertical coordinate in the image;

[0030] Calculate the disparity map D(x,y) using the formula:

[0031] D(x, y) = x L -x R

[0032] Among them, x L and x R Represent the horizontal coordinates of the same point in the left image and the right image respectively;

[0033] The depth map Z(x, y) is calculated based on the disparity map D(x, y). The formula is:

[0034]

[0035] Where f is the focal length of the camera, and B is the camera baseline distance;

[0036] Output the generated depth map Z(x, y);

[0037] The specific steps of the second processing thread using the target detection algorithm and the target tracking algorithm to identify and track the target and generate bounding box information containing the ID of the tracked target include:

[0038] Use convolutional neural network for target detection, the formula is:

[0039] P(x, y, c) = softmax(f CNN (I(x,y)))

[0040] Where P(x, y, c) is the confidence of the target category c at position (x, y), and f CNN It is a convolutional neural network model;

[0041] Generate the initial bounding box information of the target based on the target detection result [x min ,y min , x max ,y max ];

[0042] Among them, x min is the horizontal coordinate of the upper left corner of the bounding box; y min is the vertical coordinate of the upper left corner of the bounding box; x max is the horizontal coordinate of the lower right corner of the bounding box; y max is the vertical coordinate of the lower right corner of the bounding box;

[0043] Use the Kalman filter algorithm to track the target and update the bounding box position of the target. The formula is:

[0044]

[0045]

[0046] in, is the predicted state of the target, F is the state transfer matrix, B is the control matrix, u k is the control vector, w k is the process noise, H is the observation matrix, v k is the observation noise;

[0047] Output the bounding box information containing the tracked target ID;

[0048] The specific steps of the third processing thread using the modulated light detection algorithm to obtain the center position of the light emitting device include:

[0049] Acquire an image I(x, y) including the light emitting device;

[0050] Apply the modulated light detection algorithm and perform frequency domain analysis to extract the modulation frequency of the light emitting device. The formula is:

[0051]

[0052] in, represents the Fourier transform, is the frequency domain representation of the image I(x, y);

[0053] In the frequency domain image, find the peak position corresponding to the frequency of the light-emitting device and calculate its corresponding spatial domain position. The formula is:

[0054]

[0055] According to the peak position converted into the spatial domain, the center position of the light emitting device (x c ,y c );

[0056] Output the center position of the light emitting device (x c ,y c ).

[0057] Furthermore, the information fusion step includes:

[0058] Extracting depth information from the depth map Z(x, y) generated by the first processing thread;

[0059] The target bounding box information [x min ,y min , x max ,y max ] to extract the bounding box position of the target;

[0060] The center position (x) of the light emitting device obtained from the third processing thread c ,y c ) to extract the coordinates of the light emitting device;

[0061] Mark the center position of the light fixture on the depth map and use color markers or symbols to mark the coordinates of the light fixture (x c ,y c );

[0062] Mark the bounding box position of the target on the depth map to form a rectangular frame with the vertices of the frame (x min ,y min) and (x max ,y max );

[0063] The steps to calculate the spatial position of the tracked target are:

[0064] The center position of the target (x t ,y t ), the formula is:

[0065]

[0066]

[0067] According to the position and depth value of the bounding box of the target in the depth map, the spatial coordinates (X t , Y t , Z t ), the formula is:

[0068]

[0069]

[0070] Z t = Z(x t ,y t )

[0071] Among them, (x 0 ,y 0 ) is the coordinate of the optical center of the image, f is the focal length of the camera, Z(x t ,y t ) is the depth of field value of the target position;

[0072] The steps to calculate the velocity of the tracked target are:

[0073] Record the target center position in consecutive frames and spatial coordinates Where k represents the kth frame;

[0074] Calculate the target's velocity V between consecutive frames t :

[0075]

[0076] Here, Δt is the time interval between two frames.

[0077] A target tracking device, comprising:

[0078] A video sequence acquisition module, used for acquiring a video sequence containing continuous frames;

[0079] An image preprocessing module is used to perform image preprocessing on the video sequence, wherein the image preprocessing includes sharpening processing, defogging processing and denoising processing to obtain a preprocessed image sequence;

[0080] A binocular matching module is used to execute a first processing thread and generate a depth map using a binocular matching algorithm based on the preprocessed image sequence;

[0081] The target detection and tracking module is used to execute the second processing thread, identify and track the target using the target detection algorithm and the target tracking algorithm based on the preprocessed image sequence, and generate bounding box information containing the ID of the tracked target;

[0082] A modulated light detection module, used to execute a third processing thread, and obtain the center position of the light emitting device using a modulated light detection algorithm based on the preprocessed image sequence;

[0083] An information fusion module is used to fuse the depth map generated by the three processing threads, the bounding box information containing the ID of the tracked target, and the center position of the light-emitting device, and mark the center position of the light-emitting device and the bounding box position of the target on the depth map;

[0084] The calculation module is used to calculate the spatial position and velocity of the tracked target in the fusion information to achieve target tracking.

[0085] Furthermore, the video sequence acquisition module adopts a binocular stereo vision camera, and the binocular stereo vision camera is used to capture a video sequence containing continuous frames at a fixed frame rate.

[0086] Furthermore, the image preprocessing module includes a sharpening processing submodule, a defogging processing submodule and a denoising processing submodule; the sharpening processing submodule is used to perform sharpening processing on the image to enhance image details; the defogging processing submodule is used to remove the haze effect in the image to improve image clarity; the denoising processing submodule is used to reduce noise in the image to make the image smoother;

[0087] The binocular matching module includes a disparity calculation submodule and a depth map generation submodule; the disparity calculation submodule is used to calculate the disparity map according to the preprocessed image sequence; the depth map generation submodule is used to generate the depth map from the disparity map;

[0088] The target detection and tracking module includes a target detection submodule and a target tracking submodule; the target detection submodule is used to identify the target in the video using a target detection algorithm; the target tracking submodule is used to track the motion trajectory of the target, generate the bounding box information of the target, and mark the target ID;

[0089] The modulated light detection module includes a light spot detection submodule and a light spot center positioning submodule; the light spot detection submodule is used to detect the modulated light signal emitted by the light emitting device and determine the position of the light spot; the light spot center positioning submodule is used to determine the center position of the light spot;

[0090] The information fusion module includes an information synthesis submodule and a marking submodule; the information synthesis submodule is used to synthesize the depth map, the bounding box information and the center position of the light spot; the marking submodule is used to mark the bounding box and the center position of the light spot of the target on the depth map;

[0091] The calculation module includes a spatial position calculation submodule and a speed calculation submodule; the spatial position calculation submodule is used to calculate the position of the target in three-dimensional space according to the fusion information; the speed calculation submodule is used to calculate the speed of the target at different time points.

[0092] An underwater companion robot comprises a memory, a processor and a target tracking program stored in the memory and executable on the processor. When the target tracking program is executed by the processor, the various steps of the target tracking method described above are implemented.

[0093] A storage medium stores a target tracking program, and when the target tracking program is executed by a processor, each step of the target tracking method as described above is implemented.

[0094] Compared with the prior art, the technical solution of the present invention has the following beneficial effects:

[0095] The present invention provides a target tracking method and device, an underwater companion robot and a storage medium. Through the combination of binocular stereo vision and modulated light signals, the detection and tracking accuracy of targets in underwater environments are significantly improved, and the problem of traditional methods identifying blurred targets in underwater environments is solved; through preprocessing technology and active marker point detection, the robustness of the system in complex underwater environments is improved, and the interference of light attenuation and impurities in water is effectively dealt with; through optimization of algorithm flow and information fusion technology, the computing overhead is reduced, and the real-time performance and processing efficiency of the system are improved; the modulated light signal device assists in identification, solves the problem of target identification caused by the similarity of divers' clothing, and ensures the uniqueness and accuracy of target tracking; the method can be widely used in various underwater operation scenarios, and improves the effectiveness and safety of underwater robots in practical applications. BRIEF DESCRIPTION OF THE DRAWINGS

[0096] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0097] Figure 1 is a flowchart of a target tracking method provided in one embodiment of the present application;

[0098] Figure 2 is a schematic diagram of a target tracking method provided in one embodiment of the present application;

[0099] Figure 3 It is a system block diagram of a target tracking device provided in one embodiment of the present application. DETAILED DESCRIPTION

[0100] In order to better understand the purpose, structure and function of the present invention, the technical solution of the present invention is further described in detail below in conjunction with the accompanying drawings and specific preferred embodiments.

[0101] In the description of the present invention, it should be understood that the terms "left side", "right side", "upper part", "lower part", etc. indicate the orientation or positional relationship based on the orientation or positional relationship shown in the drawings, which is only for the convenience of describing the present invention and simplifying the description, and does not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation. "First", "second", etc. do not indicate the importance of the parts, and therefore cannot be understood as a limitation on the present invention. The specific dimensions used in the embodiments are only for illustrating the technical solution and do not limit the scope of protection of the present invention. It is understandable to those skilled in the art that some well-known structures and their descriptions in the drawings may be omitted.

[0102] Unless otherwise clearly specified and limited, the terms "install", "set", "connect", "fix" and the like should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection or an indirect connection through an intermediate medium, it can be the internal connection of two elements or the interaction relationship between two elements. For ordinary technicians in this field, the specific meanings of the above terms in this application can be understood according to specific circumstances.

[0103] Embodiment 1:

[0104] like Figure 1-2 As shown, the present invention provides a technical solution:

[0105] A target tracking method, wherein a light emitting device capable of emitting a modulated light signal is placed on a tracked target, and the method comprises the following steps:

[0106] A binocular camera is used to acquire a video sequence containing continuous frames, and image preprocessing is performed on the video sequence, wherein the image preprocessing includes sharpening processing, defogging processing and denoising processing to obtain a preprocessed image sequence;

[0107] Three processing threads are started simultaneously on the preprocessed image sequence. The first processing thread uses a binocular matching algorithm to generate a depth map; the second processing thread uses a target detection algorithm and a target tracking algorithm to identify and track the target, and generates bounding box information containing the ID of the tracked target; the third processing thread uses a modulated light detection algorithm to obtain the center position of the light-emitting device;

[0108] The depth map generated by the three processing threads, the bounding box information containing the tracked target ID, and the center position of the light-emitting device are fused, the center position of the light-emitting device and the bounding box position of the target are marked on the depth map, and the spatial position and speed of the tracked target in the fused information are calculated to achieve target tracking.

[0109] This embodiment provides a target tracking method, which places a light-emitting device that can emit a modulated light signal on the tracked target; by combining the depth map and the position information of the light-emitting device, the three-dimensional spatial position and motion trajectory of the target can be accurately calculated, thereby improving the tracking accuracy; by processing three threads in parallel, the processing efficiency and real-time performance of the system are improved, so that target tracking can maintain high efficiency in a rapidly changing environment; image preprocessing steps such as sharpening, defogging and denoising can significantly improve image quality, reduce interference, and improve the stability and reliability of target detection and tracking; by fusing multiple information sources (depth map, bounding box information and light-emitting device position), more comprehensive target information is provided, which helps to achieve efficient tracking in complex scenes; this method is applicable to target tracking in a variety of scenes, such as moving targets in dynamic environments, especially in low light or poor visual conditions, the combination of the light-emitting device and depth information improves the robustness of tracking.

[0110] Embodiment 2:

[0111] Based on Embodiment 1, the step of acquiring a video sequence containing continuous frames includes:

[0112] When the system leaves the factory, the binocular cameras are calibrated to make the viewing angles of the two cameras consistent, and the intrinsic and extrinsic parameters of the cameras are corrected;

[0113] The intrinsic parameters of the camera include focal length, principal point and distortion coefficient, and the extrinsic parameters of the camera include rotation matrix and translation vector; the rotation matrix is ​​a 3x3 matrix used to describe the rotation of the camera coordinate system relative to the world coordinate system; the translation vector is a 3x1 vector used to describe the translation of the camera coordinate system relative to the world coordinate system;

[0114] Use a calibrated stereo camera to capture a video sequence containing continuous frames, where each pair of frames is taken by the left camera and the right camera simultaneously for subsequent stereo matching processing.

[0115] Further, the step of performing image preprocessing on the video sequence includes:

[0116] Each frame in the acquired video sequence is preprocessed and a sharpening filter is applied to enhance the edge details in the image to improve the clarity of the target; the formula for enhancing the edge of the image is:

[0117]

[0118] Among them, I sharp (x, y) is the sharpened image, I raw (x,y) is the original image, α is the sharpening factor, is the Laplace operator; x represents the horizontal coordinate in the image, and y represents the vertical coordinate in the image;

[0119] The haze effect in the image is removed by estimating the dark channel and atmospheric light of the image to improve the contrast and visibility of the image. The specific formula is:

[0120]

[0121] Among them, I d (x, y) is the image after dehazing, I raw (x, y) is the original image, A is the atmospheric light estimation value, t(x, y) is the transmittance; x represents the horizontal coordinate in the image, and y represents the vertical coordinate in the image;

[0122] Gaussian filter is used to smooth the image and reduce the noise in the image to reduce the interference to subsequent processing.

[0123] The specific formula is:

[0124]

[0125] Among them, I denoised (x, y) is the pixel value of the denoised image at position (x, y); I raw(x′, y′) is the pixel value of the original image at position (x′, y′); σ is the standard deviation of the Gaussian filter, which is used to control the smoothness of the filter; x and y are the pixel coordinates of the denoised image; x′ and y′ are the pixel coordinates of the original image.

[0126] Furthermore, the specific steps of the first processing thread generating a depth map by using a binocular matching algorithm include:

[0127] Capture images from a pair of calibrated stereo cameras;

[0128] The image is subjected to stereo correction to obtain a corrected left image I L (x,y) and right image I R (x,y);

[0129] Among them, x represents the horizontal coordinate in the image, and y represents the vertical coordinate in the image;

[0130] Calculate the disparity map D(x,y) using the formula:

[0131] D(x,y)=x L -x R

[0132] Among them, x L and x R Represent the horizontal coordinates of the same point in the left image and the right image respectively;

[0133] The depth map Z(x, y) is calculated based on the disparity map D(x, y). The formula is:

[0134]

[0135] Where f is the focal length of the camera, and B is the camera baseline distance;

[0136] Output the generated depth map Z(x, y);

[0137] The specific steps of the second processing thread using the target detection algorithm and the target tracking algorithm to identify and track the target and generate bounding box information containing the ID of the tracked target include:

[0138] Use convolutional neural network for target detection, the formula is:

[0139] P(x, y, c) = softmax(f CNN (I(x,y)))c

[0140] Where P(x, y, c) is the confidence of the target category c at position (x, y), and f CNN It is a convolutional neural network model;

[0141] Generate the initial bounding box information of the target based on the target detection result [x min ,y min , x max ,y max ];

[0142] Among them, x min is the horizontal coordinate of the upper left corner of the bounding box; y min is the vertical coordinate of the upper left corner of the bounding box; x max is the horizontal coordinate of the lower right corner of the bounding box; y max is the vertical coordinate of the lower right corner of the bounding box;

[0143] Use the Kalman filter algorithm to track the target and update the bounding box position of the target. The formula is:

[0144]

[0145]

[0146] in, is the predicted state of the target, F is the state transfer matrix, B is the control matrix, u k is the control vector, w k is the process noise, H is the observation matrix, v k is the observation noise;

[0147] Output the bounding box information containing the tracked target ID;

[0148] The specific steps of the third processing thread using the modulated light detection algorithm to obtain the center position of the light emitting device include:

[0149] Acquire an image I(x, y) including the light emitting device;

[0150] Apply the modulated light detection algorithm and perform frequency domain analysis to extract the modulation frequency of the light emitting device. The formula is:

[0151]

[0152] in, represents the Fourier transform, is the frequency domain representation of the image I(x, y);

[0153] In the frequency domain image, find the peak position corresponding to the frequency of the light-emitting device and calculate its corresponding spatial domain position. The formula is:

[0154]

[0155] According to the peak position converted into the spatial domain, the center position of the light emitting device (x c ,y c);

[0156] Output the center position of the light emitting device (x c ,y c ).

[0157] Optionally, the target detection algorithm can also use the existing YOLO or SSD algorithm to combine the preprocessing method, target tracking algorithm, active landmark detection method, etc. to achieve diver detection and unique identity confirmation. Compared with the classification network, the target detection network can not only provide the category information of the target, but also provide the circumscribed rectangle (also known as the bounding box) of the detected target to obtain the target position information. YOLO and SSD are currently the more mainstream target detection networks. Accurate target detection is achieved by training the network using a large number of training samples with bounding box annotation information.

[0158] The target detection algorithm can detect the target to be identified in a static image, but it cannot provide the semantic information of the target. For example, in a video containing multiple people, the target detection algorithm can detect how many people are contained in each frame, but it cannot tell which people between the previous and next frames are the same person as in the previous frame, and therefore cannot perform unique identity confirmation.

[0159] Similarly, in our application environment, there may be other divers intruding into the picture. In this application environment, it is necessary to combine the target detection algorithm with the target tracking algorithm to confirm the unique identity of the detected target.

[0160] The target tracking algorithm includes the traditional tracking method based on local information and the target tracking method based on deep learning. The traditional method is to calculate some statistical features such as the color value distribution of the image block in a local area, and find the image block that meets the feature in the subsequent image sequence to achieve the target tracking process. The target tracking based on the deep learning method is to train a discrimination network, the input is the image blocks of the previous and next frames, and then the network outputs whether the two image blocks are a target, so as to achieve target tracking.

[0161] In underwater environment applications, since the tracked target is a diver whose movement is relatively slow, the position can be estimated based on the spatial information in the previous and next frames to assist the tracking algorithm in eliminating false targets.

[0162] Furthermore, the information fusion step includes:

[0163] Extracting depth information from the depth map Z(x, y) generated by the first processing thread;

[0164] The target bounding box information [x min ,y min, x max ,y max ] to extract the bounding box position of the target;

[0165] The center position (x) of the light emitting device obtained from the third processing thread c ,y c ) to extract the coordinates of the light emitting device;

[0166] Mark the center position of the light fixture on the depth map and use color markers or symbols to mark the coordinates of the light fixture (x c ,y c );

[0167] Mark the bounding box position of the target on the depth map to form a rectangular frame with the vertices of the frame (x min ,y min ) and (x max ,y max );

[0168] The steps to calculate the spatial position of the tracked target are:

[0169] The center position of the target (x t ,y t ), the formula is:

[0170]

[0171]

[0172] According to the position and depth value of the bounding box of the target in the depth map, the spatial coordinates (X t , Y t , Z t ), the formula is:

[0173]

[0174]

[0175] Z t = Z(x t ,y t )

[0176] Among them, (x 0 ,y 0 ) is the coordinate of the optical center of the image, f is the focal length of the camera, Z(x t ,y t ) is the depth of field value of the target position;

[0177] The steps to calculate the velocity of the tracked target are:

[0178] Record the target center position in consecutive frames and spatial coordinates Where k represents the kth frame;

[0179] Calculate the target's velocity V between consecutive frames t :

[0180]

[0181] Here, Δt is the time interval between two frames.

[0182] Furthermore, underwater, the attenuation of blue-green light at 480±30nm is the smallest, so a blue-green light generating device is equipped as a light-emitting device. At the same time, due to the camera acquisition frequency and in order not to affect the recorded picture, the blue-green light generating device is designed to be a breathing light mode. The light intensity of this frequency in the acquired image is filtered and the light intensity curve is calculated to detect the modulated signal.

[0183] In this embodiment, through precise camera calibration and binocular matching algorithm, the accuracy of the depth map is further improved, making the target position estimation more accurate; the refined image preprocessing steps (sharpening, defogging, and denoising) effectively improve the image quality, which is helpful for subsequent target detection and tracking; the use of advanced target detection and tracking algorithms improves the accuracy and stability of target tracking; the application of modulated light detection algorithm makes the detection of light-emitting devices more accurate, further enhancing the system's ability to track targets; the comprehensive use of multiple data sources (depth map, bounding box information, light-emitting device position) realizes comprehensive analysis and tracking of the target.

[0184] Embodiment 3:

[0185] like Figure 3 As shown, the present invention provides a technical solution:

[0186] A target tracking device, comprising:

[0187] A video sequence acquisition module, used for acquiring a video sequence containing continuous frames;

[0188] An image preprocessing module is used to perform image preprocessing on the video sequence, wherein the image preprocessing includes sharpening processing, defogging processing and denoising processing to obtain a preprocessed image sequence;

[0189] A binocular matching module is used to execute a first processing thread and generate a depth map using a binocular matching algorithm based on the preprocessed image sequence;

[0190] The target detection and tracking module is used to execute the second processing thread, identify and track the target using the target detection algorithm and the target tracking algorithm based on the preprocessed image sequence, and generate bounding box information containing the ID of the tracked target;

[0191] A modulated light detection module, used to execute a third processing thread, and obtain the center position of the light emitting device using a modulated light detection algorithm based on the preprocessed image sequence;

[0192] An information fusion module is used to fuse the depth map generated by the three processing threads, the bounding box information containing the ID of the tracked target, and the center position of the light-emitting device, and mark the center position of the light-emitting device and the bounding box position of the target on the depth map;

[0193] The calculation module is used to calculate the spatial position and velocity of the tracked target in the fusion information to achieve target tracking.

[0194] This embodiment proposes a target tracking device, which can effectively process and analyze video sequences to achieve accurate tracking of targets. By integrating multiple functional modules such as video sequence acquisition, image preprocessing, binocular matching, target detection and tracking, modulated light detection, and information fusion into one system, the solution achieves the following beneficial effects:

[0195] The sharpening, defogging and denoising processes of the image preprocessing module enhance the clarity and details of the image, making subsequent target detection and tracking more accurate; the depth map generated by the binocular matching module makes the spatial information of the target clearer and improves the system's ability to locate the target in three-dimensional space; the target detection and tracking module can simultaneously identify and track the target, and generate bounding box information containing the target ID, further improving the system's target recognition and tracking capabilities; the modulated light detection module can accurately locate the center position of the light-emitting device, which is particularly important for certain application scenarios (such as underwater tracking); the information fusion module comprehensively processes the depth map, target bounding box information and the center position of the light spot, and marks them on the depth map, providing intuitive target tracking results; the calculation module can calculate the spatial position and speed of the target based on the fused information, helping to achieve more accurate target tracking and prediction.

[0196] Embodiment 4:

[0197] On the basis of Example 3, the video sequence acquisition module adopts a binocular stereo vision camera, which is used to collect video sequences containing continuous frames at a fixed frame rate. This device is mainly used in underwater environments. Due to the large amount of impurities in the water and the certain absorption rate of light propagating in the water, the collected images usually have the characteristics of color distortion, blur, low contrast, etc.; at the same time, since this project requires the use of binocular stereo vision to measure the position of the detected target in three-dimensional space, it is necessary to perform pre-processing processes including defogging, sharpening, and denoising on the image to facilitate subsequent processing.

[0198] Furthermore, the image preprocessing module includes a sharpening processing submodule, a defogging processing submodule and a denoising processing submodule; the sharpening processing submodule is used to perform sharpening processing on the image to enhance image details; the defogging processing submodule is used to remove the haze effect in the image to improve image clarity; the denoising processing submodule is used to reduce noise in the image to make the image smoother;

[0199] The binocular matching module includes a disparity calculation submodule and a depth map generation submodule; the disparity calculation submodule is used to calculate the disparity map according to the preprocessed image sequence; the depth map generation submodule is used to generate the depth map from the disparity map;

[0200] The target detection and tracking module includes a target detection submodule and a target tracking submodule; the target detection submodule is used to identify the target in the video using a target detection algorithm; the target tracking submodule is used to track the motion trajectory of the target, generate the bounding box information of the target, and mark the target ID;

[0201] The modulated light detection module includes a light spot detection submodule and a light spot center positioning submodule; the light spot detection submodule is used to detect the modulated light signal emitted by the light emitting device and determine the position of the light spot; the light spot center positioning submodule is used to determine the center position of the light spot;

[0202] The information fusion module includes an information synthesis submodule and a marking submodule; the information synthesis submodule is used to synthesize the depth map, the bounding box information and the center position of the light spot; the marking submodule is used to mark the bounding box and the center position of the light spot of the target on the depth map;

[0203] The calculation module includes a spatial position calculation submodule and a speed calculation submodule; the spatial position calculation submodule is used to calculate the position of the target in three-dimensional space according to the fusion information; the speed calculation submodule is used to calculate the speed of the target at different time points.

[0204] Embodiment 5:

[0205] The present invention provides a technical solution:

[0206] An underwater companion robot comprises a memory, a processor and a target tracking program stored in the memory and executable on the processor. When the target tracking program is executed by the processor, the various steps of the target tracking method described above are implemented.

[0207] The working principle of the underwater companion robot is:

[0208] After obtaining the video sequence, preprocessing is performed first, including image enhancement, defogging, and denoising;

[0209] Then, three threads are opened at the same time. First, a depth map is generated based on the binocular matching algorithm.

[0210] The second processing thread uses the target detection algorithm and the target tracking algorithm to generate the target detection result with ID information and give the bounding box position with ID;

[0211] The last thread is to detect the modulated light method and give the center position of the light emitting device;

[0212] Then, the three obtained results are fused to determine the center position of the light-emitting device and the position of the bounding box, and marked on the depth map;

[0213] Calculate the real spatial position and speed of the detection and tracking target, and send this information to the robot motion control part to achieve accurate target tracking.

[0214] By integrating the target tracking program into the underwater companion robot, the robot can track the target in the underwater environment; this implementation method enables the target tracking technology to be applied to complex underwater environments and meet practical application needs.

[0215] Embodiment 6:

[0216] The present invention provides a technical solution:

[0217] A storage medium stores a target tracking program, and when the target tracking program is executed by a processor, each step of the target tracking method as described above is implemented.

[0218] This embodiment provides a medium for storing a target tracking program, so that the target tracking program can be applied on different devices; in this way, the target tracking technology can be conveniently deployed on various processors, thereby expanding its application scope.

[0219] Obviously, the above embodiments of the present invention are merely examples for clearly illustrating the present invention, and are not intended to limit the embodiments of the present invention. For those skilled in the art, other different forms of changes or modifications can be made based on the above description. It is not necessary and impossible to list all the embodiments here. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the claims of the present invention.

Claims

1. A target tracking method, characterized in that: A light emitting device capable of emitting a modulated light signal is placed on the tracked target, and the method comprises the following steps: A binocular camera is used to acquire a video sequence containing continuous frames, and image preprocessing is performed on the video sequence, wherein the image preprocessing includes sharpening processing, defogging processing and denoising processing to obtain a preprocessed image sequence; Three processing threads are started simultaneously on the preprocessed image sequence. The first processing thread uses a binocular matching algorithm to generate a depth map; the second processing thread uses a target detection algorithm and a target tracking algorithm to identify and track the target, and generates bounding box information containing the ID of the tracked target; the third processing thread uses a modulated light detection algorithm to obtain the center position of the light-emitting device; The depth map generated by the three processing threads, the bounding box information containing the tracked target ID, and the center position of the light-emitting device are fused, the center position of the light-emitting device and the bounding box position of the target are marked on the depth map, and the spatial position and speed of the tracked target in the fused information are calculated to achieve target tracking.

2. The target tracking method according to claim 1, characterized in that: The steps of obtaining a video sequence containing consecutive frames include: When the system leaves the factory, the binocular cameras are calibrated to make the viewing angles of the two cameras consistent, and the intrinsic and extrinsic parameters of the cameras are corrected; The intrinsic parameters of the camera include focal length, principal point and distortion coefficient, and the extrinsic parameters of the camera include rotation matrix and translation vector; the rotation matrix is ​​a 3x3 matrix used to describe the rotation of the camera coordinate system relative to the world coordinate system; the translation vector is a 3x1 vector used to describe the translation of the camera coordinate system relative to the world coordinate system; Use a calibrated stereo camera to capture a video sequence containing continuous frames, where each pair of frames is taken by the left camera and the right camera simultaneously for subsequent stereo matching processing.

3. The target tracking method according to claim 1, characterized in that: The steps of image preprocessing for video sequences include: Each frame in the acquired video sequence is preprocessed and a sharpening filter is applied to enhance the edge details in the image to improve the clarity of the target; the formula for enhancing the edge of the image is: Among them, I sharp (x, y) is the sharpened image, I raw (x, y) is the original image, α is the sharpening factor, is the Laplace operator; x represents the horizontal coordinate in the image, and y represents the vertical coordinate in the image; The haze effect in the image is removed by estimating the dark channel and atmospheric light of the image to improve the contrast and visibility of the image. The specific formula is: Among them, I d (x, y) is the image after dehazing, I raw (x, y) is the original image, A is the atmospheric light estimation value, t(x, y) is the transmittance; x represents the horizontal coordinate in the image, and y represents the vertical coordinate in the image; Gaussian filter is used to smooth the image and reduce the noise in the image to reduce the interference to subsequent processing.

4. The target tracking method according to claim 1, characterized in that: The specific steps of the first processing thread generating a depth map by using a binocular matching algorithm include: Capture images from a pair of calibrated stereo cameras; The image is subjected to stereo correction to obtain a corrected left image I L (x, y) and the right image I R (x, y); Among them, x represents the horizontal coordinate in the image, and y represents the vertical coordinate in the image; Calculate the disparity map D(x, y), the formula is: D(x,y)=x L -x R Among them, x L and x R Represent the horizontal coordinates of the same point in the left image and the right image respectively; The depth map Z(x, y) is calculated based on the disparity map D(x, y). The formula is: Where f is the focal length of the camera, and B is the camera baseline distance; Output the generated depth map Z(x, y); The specific steps of the second processing thread using the target detection algorithm and the target tracking algorithm to identify and track the target and generate bounding box information containing the ID of the tracked target include: Use convolutional neural network for target detection, the formula is: P(x,y,c)=softmax(f CNN (I(x,y)) Where P(x, y, c) is the confidence of the target category c at position (x, y), and f CNN It is a convolutional neural network model; Generate the initial bounding box information of the target based on the target detection result [x min ,y min , x max ,y max ]; Among them, X min is the horizontal coordinate of the upper left corner of the bounding box; y min is the vertical coordinate of the upper left corner of the bounding box; x max is the horizontal coordinate of the lower right corner of the bounding box; y max is the vertical coordinate of the lower right corner of the bounding box; Use the Kalman filter algorithm to track the target and update the bounding box position of the target. The formula is: in, is the predicted state of the target, F is the state transfer matrix, B is the control matrix, uk is the control vector, W k is the process noise, H is the observation matrix, vk is the observation noise; Output the bounding box information containing the tracked target ID; The specific steps of the third processing thread using the modulated light detection algorithm to obtain the center position of the light emitting device include: Acquire an image I(x, y) including the light emitting device; Apply the modulated light detection algorithm and perform frequency domain analysis to extract the modulation frequency of the light emitting device. The formula is: in, represents the Fourier transform, is the frequency domain representation of the image I(x, y); In the frequency domain image, find the peak position corresponding to the frequency of the light-emitting device and calculate its corresponding spatial domain position. The formula is: According to the peak position converted into the spatial domain, the center position of the light emitting device (x c ,y c ); Output the center position of the light emitting device (x c ,y c ).

5. The target tracking method according to claim 1, characterized in that: The information fusion step includes: Extracting depth information from the depth map Z(x, y) generated by the first processing thread; The target bounding box information [x min ,y min , x max ,y max ] to extract the bounding box position of the target; The center position (x) of the light emitting device obtained from the third processing thread c ,y c ) to extract the coordinates of the light emitting device; Mark the center position of the light fixture on the depth map and use color markers or symbols to mark the coordinates of the light fixture (x c ,y c ); Mark the bounding box position of the target on the depth map to form a rectangular frame with the vertices of the frame (x min ,y min ) and (x max ,y max ); The steps to calculate the spatial position of the tracked target are: The center position of the target (x t ,y t ), the formula is: According to the position and depth value of the bounding box of the target in the depth map, the spatial coordinates (X t , Y t , Z t ), the formula is: Z t =Z(x t ,y t ) Among them, (x0, y0) is the coordinate of the optical center of the image, f is the focal length of the camera, and Z(x t ,y t ) is the depth of field value of the target position; The steps to calculate the velocity of the tracked target are: Record the target center position in consecutive frames and spatial coordinates Where k represents the kth frame; Calculate the target's velocity V between consecutive frames t : Here, Δt is the time interval between two frames.

6. A target tracking device, characterized in that: include: A video sequence acquisition module, used for acquiring a video sequence containing continuous frames; An image preprocessing module is used to perform image preprocessing on the video sequence, wherein the image preprocessing includes sharpening processing, defogging processing and denoising processing to obtain a preprocessed image sequence; A binocular matching module is used to execute a first processing thread and generate a depth map using a binocular matching algorithm based on the preprocessed image sequence; The target detection and tracking module is used to execute the second processing thread, identify and track the target using the target detection algorithm and the target tracking algorithm based on the preprocessed image sequence, and generate bounding box information containing the ID of the tracked target; A modulated light detection module, used to execute a third processing thread, and obtain the center position of the light emitting device using a modulated light detection algorithm based on the preprocessed image sequence; An information fusion module is used to fuse the depth map generated by the three processing threads, the bounding box information containing the ID of the tracked target, and the center position of the light-emitting device, and mark the center position of the light-emitting device and the bounding box position of the target on the depth map; The calculation module is used to calculate the spatial position and velocity of the tracked target in the fusion information to achieve target tracking.

7. The target tracking device according to claim 6, characterized in that: The video sequence acquisition module adopts a binocular stereo vision camera, and the binocular stereo vision camera is used to capture a video sequence containing continuous frames at a fixed frame rate.

8. The target tracking device according to claim 6, characterized in that: The image preprocessing module includes a sharpening processing submodule, a defogging processing submodule and a denoising processing submodule; the sharpening processing submodule is used to perform sharpening processing on the image to enhance image details; the defogging processing submodule is used to remove the haze effect in the image to improve image clarity; the denoising processing submodule is used to reduce the noise in the image to make the image smoother; The binocular matching module includes a disparity calculation submodule and a depth map generation submodule; the disparity calculation submodule is used to calculate the disparity map according to the preprocessed image sequence; the depth map generation submodule is used to generate the depth map from the disparity map; The target detection and tracking module includes a target detection submodule and a target tracking submodule; The target detection submodule is used to identify targets in the video using a target detection algorithm; The target tracking submodule is used to track the target's motion trajectory, generate the target's bounding box information, and mark the target ID; The modulated light detection module includes a light spot detection submodule and a light spot center positioning submodule; the light spot detection submodule is used to detect the modulated light signal emitted by the light emitting device and determine the position of the light spot; the light spot center positioning submodule is used to determine the center position of the light spot; The information fusion module includes an information synthesis submodule and a marking submodule; the information synthesis submodule is used to synthesize the depth map, the bounding box information and the center position of the light spot; the marking submodule is used to mark the bounding box and the center position of the light spot of the target on the depth map; The calculation module includes a spatial position calculation submodule and a speed calculation submodule; the spatial position calculation submodule is used to calculate the position of the target in the three-dimensional space according to the fusion information; The speed calculation submodule is used to calculate the speed of the target at different time points.

9. An underwater companion robot, characterized in that: The underwater companion robot includes a memory, a processor, and a target tracking program stored in the memory and executable on the processor. When the target tracking program is executed by the processor, the various steps of the target tracking method as described in any one of claims 1 to 5 are implemented.

10. A storage medium, characterized in that: The storage medium stores a target tracking program, and when the target tracking program is executed by the processor, each step of the target tracking method according to any one of claims 1 to 5 is implemented.

Images