A real-time dynamic scene focal plane tracking method and tracking system

Abstract

The present invention provides a real-time dynamic scene focal plane tracking method and a tracking system, belonging to the field of image processing technology. The present invention obtains a depth image, an RGB image and an ambient light parameter of a current scene, performs light compensation on the RGB image, pre-processes the depth image, and fuses the features of the two to obtain a comprehensive feature; then, based on the fused comprehensive feature, a preset target detection model is applied to perform target recognition to obtain at least one tracking target; then, an optical flow method is used to track the position of the tracking target, and the posture (including rotation angle and tilt degree) of the tracking target is estimated in combination with the depth image; finally, the focal plane of the RGB camera is dynamically adjusted according to the relative displacement between the tracking target and the RGB camera, so that the focal plane is overlapped with the calculated optimal focal plane, thereby realizing real-time and accurate tracking and focal plane adjustment of the target object in the dynamic scene.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to a real-time dynamic scene focal plane tracking method and tracking system. Background Art

[0002] In the fields of digital image processing, computer vision, and robot visual navigation, real-time dynamic scene focal plane tracking methods have always been an important research topic. Especially in complex and changing dynamic environments, how to capture and track target objects in real time and accurately while ensuring that the focal plane of the imaging device (such as RGB camera) and the target object maintain the best match is of great significance for improving image quality and enhancing the accuracy and reliability of target detection.

[0003] Traditional target tracking methods mainly rely on a single source of visual information (such as RGB images), which performs well in scenes with stable lighting conditions and relatively simple backgrounds. However, in dynamically changing scenes, due to factors such as unstable lighting conditions, the movement of background objects, and changes in the posture of the target object, traditional target tracking methods often fail to achieve satisfactory results.

[0004] Therefore, it is necessary to provide a real-time dynamic scene focal plane tracking method and tracking system to solve the above technical problems. Summary of the invention

[0005] To solve the above technical problems, the present invention provides a real-time dynamic scene focal plane tracking method and tracking system, which improves the accuracy and robustness of target tracking by combining the features of depth images and RGB images, and realizes real-time and accurate tracking and imaging of target objects in dynamic scenes by dynamically adjusting the focal plane of the RGB camera.

[0006] The present invention provides a real-time dynamic scene focal plane tracking method, the tracking method comprising the following steps:

[0007] S1: Get the depth image, RGB image and ambient lighting parameters of the current scene;

[0008] S2: performing illumination compensation on the RGB image based on the ambient illumination parameter, and preprocessing the depth image, fusing the visual features of the compensated RGB image and the geometric features of the preprocessed depth image to obtain comprehensive features;

[0009] S3: Based on the fused comprehensive features, a preset target detection model is applied to perform target recognition in the current scene to obtain at least one tracking target;

[0010] S4: Tracking the position of the at least one tracking target by using an optical flow method, and estimating the posture of the at least one tracking target in combination with the depth image, wherein the posture includes a rotation angle and a tilt degree;

[0011] S5: Calculate the best focal plane for each tracking target by combining the depth image and the position, rotation angle and inclination of at least one tracking target that has been detected, and dynamically adjust the focal plane of the RGB camera according to the relative displacement between each tracking target and the RGB camera, so that the focal plane of the RGB camera coincides with the calculated best focal plane.

[0012] Preferably, in step S1, the depth image, RGB image and ambient lighting parameters are respectively acquired by a depth camera, an RGB camera and a lighting sensor deployed in the current scene.

[0013] Preferably, in step S2, the illumination compensation of the RGB image includes:

[0014] Determining the illumination intensity and illumination direction of the current scene according to the acquired ambient illumination parameters;

[0015] Based on the illumination intensity and illumination direction of the current scene, a histogram equalization illumination compensation algorithm is applied to adjust the brightness and contrast of the RGB image.

[0016] Preferably, in step S2, preprocessing of the depth image includes noise suppression, depth smoothing and edge preservation.

[0017] Preferably, in step S2, the fusion process includes:

[0018] S201: extracting visual features from the illumination-compensated RGB image, wherein the visual features include color and texture; and extracting geometric features from the preprocessed depth image, wherein the geometric features include distance and depth gradient;

[0019] S202: Mapping the geometric features of the depth image into the coordinate system of the visual features of the RGB image;

[0020] S203: Using a feature fusion algorithm to fuse the visual features and the mapped geometric features to obtain comprehensive features.

[0021] Preferably, step S3 specifically includes:

[0022] S301: Input the comprehensive feature into a preset target detection model for forward propagation calculation, and the target detection model outputs at least one tracking target and confidence in the current scene, wherein the target detection model is a convolutional neural model trained based on a comprehensive feature training set;

[0023] S302: Classifying the at least one tracking target based on a preset confidence threshold to obtain a category of each tracking target;

[0024] S303: Allocate a unique identification ID to each tracking target that passes the confidence screening.

[0025] Preferably, step S4 comprises:

[0026] S401: Based on the RGB images between adjacent frames, the motion vectors of the pixels in the current scene are calculated using an optical flow algorithm to construct an optical flow field;

[0027] S402: using the information of the optical flow field, updating the position of the tracking target detected in the previous frame in the current frame;

[0028] S403: Estimate the posture of the tracked target in the three-dimensional space using a three-dimensional coordinate reconstruction method based on the depth gradient of the depth image, wherein the posture includes a rotation angle and a tilt degree.

[0029] Preferably, step S5 comprises:

[0030] S501: Calculating a target depth of the tracking target in the depth image based on the position and posture of the tracking target, and calculating an optimal focal plane distance of the tracking target based on the target depth;

[0031] S502: Calculate a relative position vector between the center of the tracking target and the center of the RGB camera in combination with the depth image and the posture of the tracking target, and correct the calculated optimal focal plane distance based on the relative position vector;

[0032] S503: adjusting the focal plane of the RGB camera according to the corrected optimal focal plane distance to ensure that the tracked target is on the optimal focal plane.

[0033] The present invention also provides a real-time dynamic scene focal plane tracking system, which is applied to a real-time dynamic scene focal plane tracking method. The tracking system includes:

[0034] The acquisition module is used to obtain the depth image, RGB image and ambient lighting parameters of the current scene;

[0035] An image processing module, configured to perform illumination compensation on the RGB image based on the ambient illumination parameters, preprocess the depth image, and fuse the visual features of the compensated RGB image with the geometric features of the preprocessed depth image to obtain comprehensive features;

[0036] A target recognition module, used to apply a preset target detection model to perform target recognition in the current scene based on the fused comprehensive features to obtain at least one tracking target;

[0037] a posture estimation module, configured to track the position of the at least one tracking target using an optical flow method, and estimate the posture of the at least one tracking target in combination with the depth image, wherein the posture includes a rotation angle and a tilt degree;

[0038] A focal plane adjustment module is used to calculate the best focal plane for each tracking target by combining the depth image and the position, rotation angle and inclination degree of at least one tracking target that has been detected, and dynamically adjust the focal plane of the RGB camera according to the relative displacement between each tracking target and the RGB camera, so that the focal plane of the RGB camera coincides with the calculated best focal plane.

[0039] Compared with the related art, the real-time dynamic scene focal plane tracking method and tracking system provided by the present invention have the following beneficial effects:

[0040] The present invention obtains a depth image, an RGB image and ambient lighting parameters of the current scene, performs lighting compensation on the RGB image, pre-processes the depth image, and fuses the features of the two to obtain a comprehensive feature; then, based on the fused comprehensive feature, a preset target detection model is applied to perform target recognition to obtain at least one tracking target; then, the position of the tracking target is tracked using an optical flow method, and the posture of the tracking target (including the rotation angle and the degree of inclination) is estimated in combination with the depth image; finally, the focal plane of the RGB camera is dynamically adjusted according to the relative displacement between the tracking target and the RGB camera to make it coincide with the calculated optimal focal plane, thereby realizing real-time and accurate tracking and focal plane adjustment of the target object in the dynamic scene. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 A flow chart of a real-time dynamic scene focal plane tracking method provided by the present invention;

[0042] Figure 2 A module structure diagram of a real-time dynamic scene focal plane tracking system provided by the present invention. DETAILED DESCRIPTION

[0043] The present invention will be further described in detail below in conjunction with the accompanying drawings and embodiments. It is to be understood that the specific embodiments described herein are only used to explain the present invention, rather than to limit the present invention. It should also be noted that, for ease of description, only the parts related to the present invention, rather than all structures, are shown in the accompanying drawings. In addition, the embodiments of the present invention and the features in the embodiments may be combined with each other without conflict.

[0044] It should also be noted that, for ease of description, only the part relevant to the present invention but not all the content is shown in the accompanying drawings. It should be mentioned before discussing the exemplary embodiments in more detail that some exemplary embodiments are described as the processing or method described as a flow chart. Although the flow chart describes each operation (or step) as being processed sequentially, many operations therein can be implemented in parallel, concurrently or simultaneously. In addition, the order of each operation can be rearranged. When its operation is completed, the processing can be terminated, but it can also have additional steps not included in the accompanying drawings. The processing can correspond to a method, a function, a procedure, a subroutine, a subprogram, etc.

[0045] Embodiment 1

[0046] The present invention provides a real-time dynamic scene focal plane tracking method, referring to Figure 1 As shown, the tracking method includes the following steps:

[0047] S1: Get the depth image, RGB image and ambient lighting parameters of the current scene.

[0048] In this embodiment, a depth image containing depth information and an RGB image containing visual information of the current scene are obtained by a depth camera and an RGB camera respectively. At the same time, an ambient light sensor is used to monitor and record the lighting parameters of the current scene. In addition, the depth camera and the RGB camera need to perform internal and external parameter calibration to ensure accurate registration and time synchronization between images, so as to ensure that the acquired depth image and RGB image have high precision and consistency.

[0049] S2: Perform illumination compensation on the RGB image based on the ambient illumination parameters, preprocess the depth image, and fuse the visual features of the compensated RGB image with the geometric features of the preprocessed depth image to obtain comprehensive features.

[0050] In this embodiment, according to the ambient lighting parameters, a histogram equalization illumination compensation method is used to adjust the brightness and contrast of the RGB image to reduce the impact of illumination changes on target detection; the depth image preprocessing includes using methods including but not limited to median filtering and Gaussian filtering for noise suppression, applying depth smoothing technology to eliminate the discontinuity of depth information, and using an edge detection algorithm to retain key structural features of the depth image to improve the visual quality of the RGB image, enhance the geometric features of the depth image, and provide clearer and more accurate input data for subsequent feature fusion and target detection.

[0051] Visual features including color and texture are extracted from the RGB image after illumination compensation, and geometric features including distance and depth gradient are extracted from the preprocessed depth image. The visual features and geometric features are fused to obtain a comprehensive feature. The fusion of visual features and geometric features significantly improves the accuracy and robustness of target detection, and can effectively identify and locate targets even under complex lighting conditions.

[0052] S3: Based on the fused comprehensive features, a preset target detection model is applied to perform target recognition in the current scene to obtain at least one tracking target.

[0053] In this embodiment, the comprehensive features are input into the preset target detection model. Inside the model, the comprehensive features are first extracted and abstracted through a series of convolutional layers to capture the local and global features of the target. Subsequently, candidate target regions are generated through the region proposal network, which are called RoIs (Region of Interest); RoIs are further processed through the RoIPooling layer to ensure that they are of the same size to facilitate subsequent classification and regression tasks; next, the fully connected layer in the model classifies and regresses the RoIs, identifies the target category and refines the bounding box position; finally, the model outputs a result containing the target category, bounding box coordinates and confidence score, which is used to indicate the location and category of the target in the image. By fusing the visual features of the RGB image and the geometric features of the depth image, the target detection model can understand the scene more comprehensively and improve the accuracy and robustness of target recognition, especially in scenes with complex lighting conditions or cluttered backgrounds.

[0054] S4: Tracking the position of the at least one tracking target using an optical flow method, and estimating the posture of the at least one tracking target in combination with the depth image, wherein the posture includes a rotation angle and a tilt degree.

[0055] In this embodiment, based on the RGB images of adjacent frames, the Lucas-Kanade optical flow algorithm is used to calculate the motion vector of the pixel point, update the position of the target, and estimate the three-dimensional posture of the target, including the rotation angle and tilt degree, in combination with the depth information in the depth image. This realizes real-time tracking and posture estimation of the target in the dynamic scene, provides dynamic position and posture information for focal plane adjustment, and improves the continuity and accuracy of tracking.

[0056] S5: Calculate the best focal plane for each tracking target by combining the depth image and the position, rotation angle and inclination of at least one tracking target that has been detected, and dynamically adjust the focal plane of the RGB camera according to the relative displacement between each tracking target and the RGB camera, so that the focal plane of the RGB camera coincides with the calculated best focal plane.

[0057] In this embodiment, based on the target position, posture and depth information, the shortest distance from the target to the camera, that is, the optimal focal plane distance, is calculated using the principle of triangular geometry. The relative displacement of the target and the camera is taken into account, and the focal plane of the RGB camera is dynamically adjusted by changing the focal length of the camera lens or adjusting the physical distance between the camera and the target to ensure that the target is clearly focused. Ultimately, dynamic focal plane tracking is achieved, and the target can be kept on the optimal focal plane even when the target moves quickly or the camera itself moves, thereby improving the visual quality and tracking stability.

[0058] Specifically, in step S2, the illumination compensation of the RGB image includes:

[0059] The illumination intensity and illumination direction of the current scene are determined according to the acquired ambient illumination parameters.

[0060] Based on the illumination intensity and illumination direction of the current scene, a histogram equalization illumination compensation algorithm is applied to adjust the brightness and contrast of the RGB image.

[0061] In this embodiment, the brightness and contrast of the RGB image are adjusted according to the ambient lighting parameters, especially the lighting intensity and direction. Histogram equalization is a commonly used image enhancement technology that increases the contrast of the image by redistributing the grayscale of the image, thereby improving the overall visual effect of the image. In scenes with uneven lighting, histogram equalization can effectively alleviate the problem of local over-brightness or over-darkness, making the details of the entire image richer.

[0062] The implementation steps are as follows:

[0063] a) Calculate the illumination influence coefficient: Based on the acquired illumination intensity and direction, calculate the degree to which each pixel is affected by illumination. This coefficient reflects the influence of illumination on the brightness distribution of the image.

[0064] b) Construct the original grayscale histogram: Count the occurrence frequency of each grayscale level in the RGB image to form the original grayscale histogram.

[0065] c) Calculate the cumulative distribution function (CDF): Based on the original grayscale histogram, calculate the cumulative distribution function. The CDF reflects the proportion of the number of pixels less than or equal to a certain grayscale level to the total number of pixels.

[0066] d) Mapping grayscale: CDF is used to map the original grayscale to a new grayscale, with the aim of making the grayscale distribution of the image more uniform, thereby enhancing the contrast.

[0067] e) Apply the lighting influence coefficient: Combine the lighting influence coefficient with the newly mapped grayscale to adjust the brightness value of each pixel to adapt to the current lighting conditions.

[0068] f) Outputting the compensated RGB image: After completing the above steps, an RGB image with optimized brightness and contrast is output to reduce the impact of illumination changes on subsequent target detection.

[0069] Specifically, in step S2, the fusion process includes:

[0070] S201: extracting visual features from the illumination-compensated RGB image, wherein the visual features include color and texture; and extracting geometric features from the preprocessed depth image, wherein the geometric features include distance and depth gradient.

[0071] In this embodiment, a SIFT feature detector is used on the illumination-compensated RGB image to extract and describe key points in the image and texture patterns around them.

[0072] For depth images, geometric feature extraction focuses on distance and depth gradient information. Through the difference operation of the depth image, the depth gradient of each pixel can be calculated, which reflects the spatial change rate of the depth information and helps to identify the object boundary and surface structure. At the same time, directly reading the distance information of each pixel from the depth image can understand the relative position relationship of the objects in the scene.

[0073] S202: Mapping the geometric features of the depth image into the coordinate system of the visual features of the RGB image.

[0074] In this embodiment, in order to enable the fusion of geometric features and visual features in the same coordinate system, it is necessary to transform the coordinates of the geometric features of the depth image, specifically: calibrate the external parameters between the depth camera and the RGB camera, that is, determine the relative rotation and translation matrix between the two, so as to map the point cloud data in the depth image to the pixel coordinates of the RGB image. By applying these transformations, the position information of each pixel point in the depth image is associated with the visual features of the corresponding pixel in the RGB image, achieving consistency in the feature space.

[0075] S203: Using a feature fusion algorithm to fuse the visual features and the mapped geometric features to obtain comprehensive features.

[0076] In this embodiment, feature fusion uses feature concatenation to merge the visual feature vector and the geometric feature vector into a longer vector. The fused comprehensive features contain the complementary advantages of visual and geometric information and can better describe the appearance and shape attributes of the target.

[0077] Specifically, step S3 specifically includes:

[0078] S301: Input the comprehensive features into a preset target detection model for forward propagation calculation, and the target detection model outputs at least one tracking target and confidence in the current scene, wherein the target detection model is a convolutional neural model trained based on a comprehensive feature training set.

[0079] In this embodiment, the comprehensive features are taken as input and passed to a preset target detection model, which is a convolutional neural network model trained based on a large number of comprehensive feature training sets. After training, the model learns to identify targets of different categories from the comprehensive features. The forward propagation process of the model includes multi-layer convolution, pooling, and full connection operations to gradually extract features and make predictions. The output results include the bounding box position, category prediction, and corresponding confidence score of at least one potential tracking target. The confidence score reflects the degree of certainty of the model's judgment on the target category.

[0080] S302: Classify the at least one tracking target based on a preset confidence threshold to obtain a category of each tracking target.

[0081] In this embodiment, all potential tracking targets output are screened according to a preset confidence threshold. Only targets with confidence levels higher than the threshold are considered valid detection results. This threshold can be adjusted according to application scenarios and requirements to balance detection accuracy and recall rate. For each target that passes the confidence screening, the model will output its category, such as a person, vehicle, or other object category.

[0082] S303: Allocate a unique identification ID to each tracking target that passes the confidence screening.

[0083] In this embodiment, for each tracking target that passes the confidence screening, a unique identification ID is assigned to it. This ID is used to uniquely identify the target in the subsequent tracking process. Even if the tracking target temporarily leaves the field of view and reappears, it can be associated with the previous data through the ID to achieve continuous and consistent tracking.

[0084] Specifically, step S4 includes:

[0085] S401: Based on the RGB images between adjacent frames, the motion vectors of the pixels in the current scene are calculated using an optical flow algorithm to construct an optical flow field.

[0086] In this embodiment, in order to track the movement of the target between consecutive frames, the Lucas-Kanade optical flow algorithm is used to analyze the RGB images between adjacent frames. The optical flow algorithm can calculate the direction and distance of each pixel moving between two frames of images to form the so-called "optical flow field". This step is based on pixel-level motion estimation, which can capture the slight displacement changes of the target and provide a basis for subsequent position updates. The optical flow field calculated by the optical flow algorithm can provide the motion trajectory of the target on the image plane. Even when the target moves quickly or the ambient light changes, the displacement of the target can be accurately tracked, thereby improving the continuity and accuracy of tracking.

[0087] S402: Using the information of the optical flow field, update the position of the tracking target detected in the previous frame in the current frame.

[0088] In this embodiment, based on the obtained optical flow field, the position information of the tracked target detected in the previous frame is combined with the corresponding optical flow vector in the current frame to update the predicted position of the target in the current frame. Specifically, the optical flow vector is mapped to the center point of the target bounding box to infer the new position of the target.

[0089] S403: Estimate the posture of the tracked target in the three-dimensional space using a three-dimensional coordinate reconstruction method based on the depth gradient of the depth image, wherein the posture includes a rotation angle and a tilt degree.

[0090] In this embodiment, in order to estimate the three-dimensional posture of the target, the depth gradient information in the depth image is used in combination with the two-dimensional motion information provided by the optical flow field, and a three-dimensional coordinate reconstruction algorithm is adopted, including but not limited to the PnP (Perspective-n-Point) algorithm or the ICP (Iterative Closest Point) algorithm, to estimate the rotation and translation parameters of the target relative to the camera. The three-dimensional coordinate reconstruction algorithm can calculate the rotation angle and tilt degree of the target in the three-dimensional space based on the key points of the target in the depth image and its depth information. Through the combination of the depth image and the optical flow field, accurate posture information of the target in the three-dimensional space can be provided, which is crucial for subsequent focal plane adjustment.

[0091] Specifically, step S5 includes:

[0092] S501: Calculating a target depth of a tracking target in the depth image based on the position and posture of the tracking target, and calculating an optimal focal plane distance of the tracking target based on the target depth.

[0093] In this embodiment, the obtained position information (such as coordinates) and posture information (such as rotation angle and tilt degree) of the tracked target are combined with the depth image to calculate the specific depth value of the target in the depth image. This depth value represents the distance of the target relative to the camera. Then, by analyzing the size, shape and depth distribution of the target and applying the triangular geometry principle, the shortest distance from the target to the camera lens, that is, the optimal focal plane distance, is calculated. This distance is a key parameter for dynamically adjusting the focal plane of the camera, and can accurately determine the actual distance of the target relative to the camera, providing precise guidance for subsequent focal plane adjustments, ensuring that the camera can capture the clearest image of the target at any time, even when the target moves quickly or has large posture changes.

[0094] S502: Calculate a relative position vector between the center of the tracking target and the center of the RGB camera in combination with the depth image and the posture of the tracking target, and correct the calculated optimal focal plane distance based on the relative position vector.

[0095] In this embodiment, considering that the change of the target's posture may affect the calculation of the optimal focal plane distance, further correction is required. By analyzing the posture of the target in the depth image, the relative position vector of the target center relative to the camera center can be obtained. This vector contains the offset information of the target in three-dimensional space, such as the displacement in the front, back, left, right, and up and down directions. Combined with this vector, the optimal focal plane distance can be adjusted more accurately to adapt to the change of the target posture.

[0096] S503: adjusting the focal plane of the RGB camera according to the corrected optimal focal plane distance to ensure that the tracked target is on the optimal focal plane.

[0097] In this embodiment, the focal plane of the camera is dynamically adjusted by controlling the autofocus mechanism of the RGB camera or manually adjusting the focal length of the lens according to the corrected optimal focal plane distance. If the distance between the target and the camera changes, or the position or posture of the target itself changes, the focal plane of the camera will be adjusted accordingly to keep the target always on the optimal focal plane, ensuring the clarity and sharpness of the target image, and realizing real-time adjustment of the focal plane in dynamic scenes, greatly improving the quality of the target image, especially when the target moves quickly or the position of the camera itself changes, it can respond quickly to keep the target image clear and stable.

[0098] The working principle of a real-time dynamic scene focal plane tracking method provided by the present invention is as follows: the present invention obtains a depth image, an RGB image and ambient lighting parameters of the current scene, performs lighting compensation on the RGB image, pre-processes the depth image, and fuses the features of the two to obtain comprehensive features; then, based on the fused comprehensive features, a preset target detection model is applied to perform target recognition to obtain at least one tracking target; then, the position of the tracking target is tracked using the optical flow method, and the posture of the tracking target (including rotation angle and tilt degree) is estimated in combination with the depth image; finally, the focal plane of the RGB camera is dynamically adjusted according to the relative displacement between the tracking target and the RGB camera, so that it coincides with the calculated optimal focal plane, thereby realizing real-time and accurate tracking and focal plane adjustment of the target object in the dynamic scene.

[0099] Embodiment 2

[0100] The present invention also provides a real-time dynamic scene focal plane tracking system, which is applied to a real-time dynamic scene focal plane tracking method. Figure 2 As shown, the tracking system includes:

[0101] The acquisition module 100 is used to acquire the depth image, RGB image and ambient lighting parameters of the current scene.

[0102] The image processing module 200 is used to perform illumination compensation on the RGB image based on the ambient illumination parameters, and to preprocess the depth image, and to fuse the visual features of the compensated RGB image and the geometric features of the preprocessed depth image to obtain comprehensive features.

[0103] The target recognition module 300 is used to apply a preset target detection model to perform target recognition in the current scene based on the fused comprehensive features to obtain at least one tracking target.

[0104] The posture estimation module 400 is used to track the position of the at least one tracking target by using the optical flow method, and estimate the posture of the at least one tracking target in combination with the depth image, wherein the posture includes a rotation angle and a tilt degree.

[0105] The focal plane adjustment module 500 is used to calculate the best focal plane for each tracking target by combining the depth image and the position, rotation angle and inclination degree of at least one tracking target that has been detected, and dynamically adjust the focal plane of the RGB camera according to the relative displacement between each tracking target and the RGB camera, so that the focal plane of the RGB camera coincides with the calculated best focal plane.

[0106] The present application is described with reference to flowcharts and / or block diagrams of methods, devices (systems) and computer program products according to embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0107] A person skilled in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by instructing related hardware through a program, and the program can be stored in a computer-readable storage medium, and the storage medium includes a read-only memory (ROM), a random access memory (RAM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), a one-time programmable read-only memory (OTPROM), an electronically erasable rewritable read-only memory (EEPROM), a compact disc (CD-ROM) or other optical disc storage, magnetic disk storage, magnetic tape storage, or any other computer-readable medium that can be used to carry or store data.

[0108] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

Claims

1. A real-time dynamic scene focal plane tracking method, characterized in that: The tracking method includes the following steps: S1: Get the depth image, RGB image and ambient lighting parameters of the current scene; S2: performing illumination compensation on the RGB image based on the ambient illumination parameter, and preprocessing the depth image, fusing the visual features of the compensated RGB image and the geometric features of the preprocessed depth image to obtain comprehensive features; S3: Based on the fused comprehensive features, a preset target detection model is applied to perform target recognition in the current scene to obtain at least one tracking target; S4: Tracking the position of the at least one tracking target by using an optical flow method, and estimating the posture of the at least one tracking target in combination with the depth image, wherein the posture includes a rotation angle and a tilt degree; S5: Calculate the best focal plane for each tracking target by combining the depth image and the position, rotation angle and inclination of at least one tracking target that has been detected, and dynamically adjust the focal plane of the RGB camera according to the relative displacement between each tracking target and the RGB camera, so that the focal plane of the RGB camera coincides with the calculated best focal plane; Step S5 includes: S501: calculating the target depth of the tracking target in the depth image based on the position and posture of the tracking target, and calculating the optimal focal plane distance of the tracking target based on the target depth; S502: calculating the relative position vector between the center of the tracking target and the center of the RGB camera in combination with the depth image and the posture of the tracking target, and correcting the calculated optimal focal plane distance based on the relative position vector; S503: adjusting the focal plane of the RGB camera according to the corrected optimal focal plane distance to ensure that the tracking target is on the optimal focal plane.

2. A real-time dynamic scene focal plane tracking method according to claim 1, characterized in that: In step S1, the depth image, RGB image and ambient lighting parameters are respectively acquired by a depth camera, an RGB camera and a lighting sensor deployed in the current scene.

3. A real-time dynamic scene focal plane tracking method according to claim 2, characterized in that: In step S2, the illumination compensation of the RGB image includes: Determining the illumination intensity and illumination direction of the current scene according to the acquired ambient illumination parameters; Based on the illumination intensity and illumination direction of the current scene, a histogram equalization illumination compensation algorithm is applied to adjust the brightness and contrast of the RGB image.

4. A real-time dynamic scene focal plane tracking method according to claim 3, characterized in that: In step S2, the preprocessing of the depth image includes noise suppression, depth smoothing and edge preservation.

5. A real-time dynamic scene focal plane tracking method according to claim 4, characterized in that: In step S2, the fusion process includes: S201: extracting visual features from the illumination-compensated RGB image, wherein the visual features include color and texture; and extracting geometric features from the preprocessed depth image, wherein the geometric features include distance and depth gradient; S202: Mapping the geometric features of the depth image into the coordinate system of the visual features of the RGB image; S203: Using a feature fusion algorithm to fuse the visual features and the mapped geometric features to obtain comprehensive features.

6. A real-time dynamic scene focal plane tracking method according to claim 5, characterized in that: Step S3 specifically includes: S301: Input the comprehensive feature into a preset target detection model for forward propagation calculation, and the target detection model outputs at least one tracking target and confidence in the current scene, wherein the target detection model is a convolutional neural model trained based on a comprehensive feature training set; S302: Classifying the at least one tracking target based on a preset confidence threshold to obtain a category of each tracking target; S303: Allocate a unique identification ID to each tracking target that passes the confidence screening.

7. A real-time dynamic scene focal plane tracking method according to claim 6, characterized in that: Step S4 includes: S401: Based on the RGB images between adjacent frames, the motion vectors of the pixels in the current scene are calculated using an optical flow algorithm to construct an optical flow field; S402: using the information of the optical flow field, updating the position of the tracking target detected in the previous frame in the current frame; S403: Estimate the posture of the tracked target in the three-dimensional space using a three-dimensional coordinate reconstruction method based on the depth gradient of the depth image, wherein the posture includes a rotation angle and a tilt degree.

8. A real-time dynamic scene focal plane tracking system, applied to a real-time dynamic scene focal plane tracking method as claimed in any one of claims 1 to 7, characterized in that: The tracking system includes: The acquisition module is used to obtain the depth image, RGB image and ambient lighting parameters of the current scene; An image processing module, configured to perform illumination compensation on the RGB image based on the ambient illumination parameters, preprocess the depth image, and fuse the visual features of the compensated RGB image with the geometric features of the preprocessed depth image to obtain comprehensive features; A target recognition module, used to apply a preset target detection model to perform target recognition in the current scene based on the fused comprehensive features to obtain at least one tracking target; a posture estimation module, configured to track the position of the at least one tracking target using an optical flow method, and estimate the posture of the at least one tracking target in combination with the depth image, wherein the posture includes a rotation angle and a tilt degree; A focal plane adjustment module is used to calculate the best focal plane for each tracking target by combining the depth image and the position, rotation angle and inclination degree of at least one tracking target that has been detected, and dynamically adjust the focal plane of the RGB camera according to the relative displacement between each tracking target and the RGB camera, so that the focal plane of the RGB camera coincides with the calculated best focal plane.

Images