Gimbal tracking shooting automatic light supplement method and device, gimbal and storage medium

Abstract

The application relates to the technical field of gimbals, and provides a gimbal tracking and shooting automatic light supplement method and device, a gimbal and a storage medium. The method comprises the following steps: performing time sequence alignment and space calibration fusion on original image data streams collected in real time to obtain a color point cloud sequence; performing target detection and light re-sampling on current frame point cloud data in the color point cloud sequence to obtain target ambient light field data; performing difference comparison and light compensation reverse solving on preset light target distribution data and the target ambient light field data to generate a light supplement control parameter set; and performing light supplement according to the light supplement control parameter set. According to the application, the target region is subjected to spherical re-sampling and anchor point matching by constructing a color point cloud sequence to form target ambient light field data, and then the preset light target distribution is combined to perform reverse solving to generate light supplement control parameters, so that accurate automatic light supplement in the gimbal tracking and shooting process is finally realized.

Description

Technical Field

[0001] This application relates to the field of gimbal technology, and in particular to a gimbal tracking and shooting automatic supplementary lighting method, device, gimbal, and storage medium. Background Technology

[0002] When using a gimbal for scenarios such as portrait tracking, live streaming, short video recording, meeting minutes, or mobile surveillance, the subject is constantly moving, and ambient lighting may fluctuate due to changes in indoor / outdoor conditions, backlighting from windows, light switching, or obstructions. In these situations, the gimbal not only needs to stably track the target's position but also simultaneously maintain image brightness, subject visibility, and the natural consistency of skin tone and overall color tone. Otherwise, issues such as a dark subject, localized overexposure, prominent facial shadows, or abrupt brightness changes during tracking can easily occur. Therefore, automatic fill light during tracking shooting is a strong practical application requirement.

[0003] Existing gimbals typically make supplementary lighting decisions during tracking and shooting based on only two-dimensional image brightness, local area grayscale, or fixed thresholds. This makes it difficult to accurately reflect the true three-dimensional lighting distribution around the target. Therefore, the supplementary lighting intensity and direction are usually not precise enough, which can easily lead to problems such as excessive or insufficient supplementary lighting or inconsistent colors. Summary of the Invention

[0004] In view of this, this application provides a method, device, gimbal, and storage medium for automatic fill light during gimbal tracking shooting, in order to solve the problems of insufficient fill light intensity and direction during gimbal tracking shooting, which can easily lead to excessively strong or weak fill light or inconsistent colors.

[0005] The first aspect of this application provides a method for automatic fill light during gimbal tracking shooting, the method comprising:

[0006] Temporal alignment and spatial calibration fusion are performed on the raw image data stream acquired in real time to obtain a color point cloud sequence; The method involves performing target detection and illumination resampling on the current frame point cloud data in the color point cloud sequence to obtain target environment light field data. This includes: performing target detection on the current frame point cloud data in the color point cloud sequence according to a preset detection target to obtain a subset of target region point clouds; calculating the target centroid coordinates based on the target region point cloud subset; performing illumination resampling based on a preset distance threshold and the target centroid coordinates to obtain original light field data; and performing direction matching and nearest-point retention processing on the original light field data based on a preset anchor point feature set to obtain target environment light field data. The preset target illumination distribution data and the target environment light field data are compared for differences and the illumination compensation is solved in reverse to generate a set of supplementary lighting control parameters. Supplemental lighting is performed according to the set of supplemental lighting control parameters.

[0007] Optionally, the original image data stream includes depth image frames and color image frames, and the step of performing temporal alignment and spatial calibration fusion on the real-time acquired original image data stream to obtain a color point cloud sequence includes: Based on the acquisition timestamp and the preset time difference threshold, the real-time acquired depth image frames and color image frames are matched for time nearest neighbor to form synchronously acquired depth and color data pairs. Based on the preset intrinsic parameter matrix of the depth sensor, the depth pixel coordinates and corresponding depth values ​​in the depth and color data pair are back-projected to generate a first three-dimensional spatial point in the depth camera coordinate system. The first three-dimensional spatial point is then converted into a second three-dimensional spatial point in the color camera coordinate system according to the preset spatial transformation parameters. Based on the preset intrinsic parameter matrix of the color sensor, the second three-dimensional spatial points are projected forward to obtain the sub-pixel coordinates corresponding to each second three-dimensional spatial point. Based on the sub-pixel coordinates, bilinear interpolation is performed on the color values ​​in the depth and color data pairs to generate point cloud data for each data pair. The point cloud data is sorted according to time sequence to obtain a color point cloud sequence.

[0008] Optionally, the step of calculating the target centroid coordinates based on the target region point cloud subset, and performing illumination resampling based on a preset distance threshold and the target centroid coordinates to obtain the original light field data includes: The average values ​​of the three-dimensional spatial coordinates in the target region point cloud subset are aggregated to obtain the target centroid coordinates. Based on a preset distance threshold and the target centroid coordinates, a set of candidate coordinates is selected from the current frame point cloud data; Calculate the spatial vector between each coordinate point in the candidate coordinate set and the target centroid coordinates. Using the target centroid coordinates as the origin, perform normalized projection on each coordinate point in the candidate coordinate set according to the spatial vector to obtain the light field projection point corresponding to each coordinate point. The color values ​​and spatial vectors corresponding to each coordinate point in the candidate coordinate set are associated and stored with the light field projection points to obtain the original light field data.

[0009] Optionally, the step of performing direction matching and nearest-point retention processing on the original light field data based on a preset anchor point feature set to obtain the target environment light field data includes: Based on the spatial vector and the preset anchor point feature set, candidate anchor point sets corresponding to each anchor point feature are selected from the light field projection points of the original light field data. Calculate the Euclidean distance between each projection point in the candidate anchor point set and the target centroid coordinates, and take the light field projection point corresponding to the smallest Euclidean distance as the target anchor point corresponding to the anchor point feature; The color value and spatial vector associated with the light field projection point are associated with and stored with the target anchor point to obtain the target environment light field data.

[0010] Optionally, the step of performing difference comparison and inverse solution of illumination compensation on the preset illumination target distribution data and the target environment light field data to generate a set of supplementary lighting control parameters includes: The target environment light field data is subjected to spatial context aggregation information completion processing to obtain the current illumination distribution data; Based on the preset illumination channel set, the current illumination distribution data and the preset illumination target distribution data are compared and calculated channel by channel to obtain the supplementary lighting difference distribution data; Based on the preset light distribution parameters, the supplementary light difference distribution data is solved by least squares optimization in reverse constraint to obtain the light source intensity weight solution. The intensity weighting solution of the light source is corrected according to the preset chromaticity channel ratio to obtain multi-channel intensity values; The multi-channel intensity values ​​are linearly transformed and the signals are modulated to generate a set of supplementary lighting control parameters.

[0011] Optionally, the step of performing a channel-by-channel difference comparison calculation on the current illumination distribution data and the preset illumination target distribution data according to a preset illumination channel set to obtain supplementary lighting difference distribution data includes: Read the spherical harmonic coefficients of the current illumination distribution data and the preset illumination target distribution data under the same illumination channel, and then perform successive subtraction of the coefficients to obtain the supplementary light difference distribution data.

[0012] Optionally, the method further includes: After obtaining the supplementary lighting difference distribution data, the supplementary lighting difference distribution data is subjected to inverse constraint solution of least squares optimization according to the preset light distribution parameters of the supplementary lighting module to obtain the light source intensity weight solution. The output light intensity of the supplementary lighting module is controlled according to the light source intensity weight solution to fill the gap between the current ambient light and the ideal light.

[0013] A second aspect of this application provides a gimbal-tracking automatic fill light device for shooting, the device comprising: The alignment and fusion module is used to perform temporal alignment and spatial calibration fusion on the raw image data stream acquired in real time to obtain a color point cloud sequence; The target sampling module is used to perform target detection and illumination resampling on the current frame point cloud data in the color point cloud sequence to obtain target environment light field data. This includes: performing target detection on the current frame point cloud data in the color point cloud sequence according to a preset detection target to obtain a target region point cloud subset; calculating the target centroid coordinates based on the target region point cloud subset; performing illumination resampling based on a preset distance threshold and the target centroid coordinates to obtain original light field data; and performing direction matching and nearest-point retention processing on the original light field data based on a preset anchor point feature set to obtain target environment light field data. The difference compensation module is used to compare the difference between the preset illumination target distribution data and the target environment light field data, and to solve the illumination compensation in reverse to generate a set of supplementary lighting control parameters. A supplementary lighting driver module is used to provide supplementary lighting based on the supplementary lighting control parameter set.

[0014] A third aspect of this application provides a gimbal, the gimbal including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the gimbal tracking and shooting automatic supplementary lighting method described above.

[0015] The fourth aspect of this application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the gimbal tracking and automatic supplemental lighting method described above.

[0016] In summary, this application includes at least the following beneficial technical effects: 1. By performing temporal nearest neighbor matching, spatial calibration, and point cloud fusion on depth image frames and color image frames, a spatiotemporally synchronized color point cloud sequence can be obtained, thereby improving the integrity and positioning accuracy of target environment data and reducing errors caused by sensor asynchrony.

[0017] 2. By resampling the unit sphere with the target centroid as the center and combining the anchor point feature set with the nearest point retention process, complex scenes can be compressed into structured target environment light field data, highlighting key lighting information around the target, reducing the interference of distant background and occlusion factors, and improving the robustness of lighting modeling.

[0018] 3. By comparing the target ambient light field data with the preset illumination target distribution and performing reverse calculation to obtain multi-channel supplementary lighting control parameters, it is possible to achieve quantitative adjustment of supplementary lighting intensity, color and direction, so that the supplementary lighting process changes from experience-based brightening to precise control, thereby improving the uniformity, visibility and stability of the image during tracking shooting. Attached Figure Description

[0019] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1 This is a flowchart of an automatic fill light method for gimbal tracking and shooting provided in an embodiment of this application; Figure 2 This is a functional module diagram of an automatic fill light device for gimbal tracking and shooting provided in an embodiment of this application; Figure 3 This is a schematic diagram of the gimbal structure provided in the embodiments of this application. Detailed Implementation

[0021] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0022] The gimbal tracking and automatic lighting method of this application operates within a gimbal. The gimbal integrates a system-on-a-chip (SoC) with a multi-core CPU, GPU, and neural network processing unit. It also houses a depth sensor, a color image sensor, and a lighting module. The SoC is connected to the depth and color sensors via a high-speed bus and contains the program instructions for performing all calculations required for the gimbal tracking and automatic lighting method. The color image sensor and depth sensor are physically adjacent and have a fixed rigid body offset relationship. The depth sensor acquires depth image frames, and the color image sensor captures color image frames. The lighting module employs a multi-channel independently controllable lighting array. Each lighting array has pre-calibrated light distribution parameters and chromaticity channel ratios. Each lighting module integrates red, green, and blue LEDs and driving circuitry. The lighting arrays are connected to the SoC via a PWM bus, receiving lighting control parameters and performing response adjustments to achieve precise control of light intensity and color temperature. The gimbal is also equipped with a high-precision hardware clock, which can be used to add a timestamp to each frame of data output by the depth sensor and color sensor.

[0023] like Figure 1 The diagram shown is a flowchart of the automatic lighting method for gimbal tracking and shooting provided in this application embodiment. The automatic lighting method for gimbal tracking and shooting provided in this application embodiment includes the following steps.

[0024] Step S1: Perform temporal alignment and spatial calibration fusion on the real-time acquired raw image data stream to obtain a color point cloud sequence.

[0025] When a gimbal performs a tracking and shooting task, a depth sensor (such as a time-of-flight sensor or a LiDAR sensor) perceives the straight-line distance from each point on the surface of an object in the scene to the sensor's optical center, thus outputting a depth image frame. In the depth image frame, the value of each pixel is not color, but a depth value, which precisely records the physical distance from the corresponding object point to the camera's optical center. A color image sensor (such as an RGB camera) perceives the ambient light reflected from the surface of objects in the scene, thus outputting a color image frame. Each pixel in the color image frame contains intensity values ​​for the red, green, and blue color channels, which together depict the visual texture of the scene.

[0026] Depth image frames and color image frames are two consecutive independent data streams. By fusing the two into the original image data stream, the spatial location and optical properties of the target can be perceived during the illumination analysis process.

[0027] Since the depth sensor and color sensor are two independent hardware modules, each with its own independent clock source and exposure trigger mechanism, depth image frames and color image frames may be out of sync at the same physical moment. Directly performing subsequent coordinate fusion on temporally misaligned depth and color frames will lead to a mismatch between geometric position and color texture, especially when the gimbal or target is moving; this mismatch will cause severe 3D data distortion. In this embodiment, the acquisition timestamp recorded at the hardware level is extracted for each depth image frame and each color image frame. For each newly arrived color image frame, the buffer of the depth image frames is searched to find the depth image frame with the smallest absolute difference between its acquisition timestamp and the current color image frame. Then, the minimum time difference is compared with a preset time difference threshold. When the minimum time difference is less than the threshold, it indicates that the color image frame and the depth image frame are matched in time, and they are combined into a synchronously acquired depth and color data pair. If the minimum time difference exceeds the threshold, it indicates that there are not enough synchronous depth frames in the current time period, and the color frame will be discarded and not participate in subsequent processing. The time difference threshold in this application is set based on half the sampling period of the sensor with the lower frame rate between the depth sensor and the color sensor. For example, if the lower frame rate is the depth sensor, operating at 30 frames per second, its single-frame sampling period is approximately 33.3 ms, and the preset time difference threshold is 16.67 ms. This time nearest neighbor matching operation ensures that each pair of data entering the subsequent fusion step reflects the 3D geometry and 2D texture at the same instant, thereby guaranteeing the temporal consistency and data reliability of the color point cloud sequence. For instance, when a gimbal rotates rapidly to track a cyclist, the positions of trees and buildings in the environment change rapidly in the image. If the depth frame and color frame differ by even 30 milliseconds, the outline color of the trees in the fused point cloud may show a visible shift from their depth shape.

[0028] After completing temporal alignment and obtaining paired depth and color data pairs, backprojection processing is performed on the depth pixel coordinates and corresponding depth values ​​in the depth and color data pairs according to the preset intrinsic parameter matrix of the depth sensor. The direct data output by the depth sensor is the coordinates of the pixels on the image plane (i.e., depth pixel coordinates) and their corresponding depth values, which is essentially a two-dimensional matrix representation and does not directly contain X and Y coordinate information in three-dimensional space. Through backprojection processing, the two-dimensional pixel coordinates and depth values ​​are restored to the true three-dimensional spatial points. Specifically, for a pixel located in row u and column v in a depth image frame, with a known depth value z, the preset intrinsic parameter matrix of the depth sensor contains the focal length of the depth camera and the pixel offset of the optical center on the image. By multiplying the inverse matrix of this intrinsic parameter matrix with the homogeneous coordinates of the pixel, and then multiplying the result by the depth value z, the position of the pixel in the three-dimensional coordinate system of the depth camera itself can be calculated. This position is called the first three-dimensional spatial point. The first three-dimensional spatial point is defined in the coordinate system of the depth camera and accurately describes the spatial position of the object point relative to the optical center of the depth camera.

[0029] Because the depth camera and color camera have different physical mounting positions on the gimbal, meaning their optical centers do not coincide, there is a fixed rigid body transformation relationship between their three-dimensional coordinate systems. Therefore, after obtaining the first three-dimensional spatial point defined in the depth camera coordinate system, it must be transformed to the color camera coordinate system to establish a geometric relationship with the color image. Specifically, preset spatial transformation parameters are obtained, including a rotation matrix and a translation vector. By multiplying the rotation matrix by the coordinate vector of the first three-dimensional spatial point and adding the translation vector, the new coordinates of the point in the color camera coordinate system of the color sensor are obtained; these new coordinates are called the second three-dimensional spatial point. Unifying the geometric information from different coordinate systems to the same reference frame lays the foundation for accurately projecting the three-dimensional point back into the two-dimensional color image. For example, on a gimbal, the depth sensor might be mounted three centimeters to the left of the color camera. For a target directly in front, the depth sensor sees it slightly to the right of its own center, while the color camera sees the same target at its own center. Through preset rotation and translation transformations, the point that was "off to the right" in the depth point cloud was corrected to the "center" position in the color camera coordinate system, thus making the geometric data and color texture data fit together perfectly in space.

[0030] Subsequently, using the preset intrinsic parameter matrix of the color sensor, the second 3D spatial point, already transformed into the color camera coordinate system, is subjected to forward projection processing. Forward projection maps a point in 3D space onto a 2D color image plane. Specifically, by multiplying the color camera intrinsic parameter matrix by the coordinate vector of the second 3D spatial point and performing homogeneous coordinate normalization on the result, a floating-point pixel coordinate is obtained. The row and column values ​​of the pixel coordinate are usually not integers, but are accurate to several decimal places. This pixel coordinate is called the sub-pixel coordinate corresponding to the second 3D spatial point. This sub-pixel coordinate represents the theoretically most accurate imaging position of the 3D point. However, a color image is a discrete pixel array indexed by integer rows and columns, and does not have color values ​​directly located at the sub-pixel coordinates. Therefore, to obtain the most accurate color information of the 3D point, bilinear interpolation processing is needed on the color image frames in the depth-color data pair based on this sub-pixel coordinate. Bilinear interpolation locates the four nearest integer pixels around the subpixel coordinates. Based on the horizontal and vertical distances from the subpixel to the center of these four pixels, a weighted average of the red, green, and blue color values ​​of these four pixels is calculated, resulting in a smooth and accurate interpolated color value. This interpolated color value is then bound to the 3D coordinates of the corresponding first or second 3D spatial point, resulting in a complete point cloud data unit. This interpolation sampling operation avoids quantization errors introduced by rounding the projected coordinates, thus ensuring the accuracy of the color representation of the fused point cloud. Especially when tracking a person wearing striped or gradient clothing, even slight movement can cause jagged color transitions at the edges of the clothing due to direct rounding. Bilinear interpolation, however, provides a smooth transition, realistically reproducing the fine textures at the subpixel level.

[0031] Finally, the point cloud data is sorted according to time sequence. Real-time data acquisition by the gimbal is a continuous process, with depth and color data pairs generated frame by frame. To construct a color point cloud sequence that describes dynamic environmental changes, the generated point cloud data must be arranged chronologically according to the acquisition timestamps carried by the depth or color frames in each data pair. The resulting data stream is the final color point cloud sequence. Each frame in the color point cloud sequence is a dense point cloud composed of numerous colored 3D points, accurately describing the 3D geometry of the scene at a given instant while preserving the visible light color texture information of the scene. The purpose of obtaining this color point cloud sequence is to provide a continuous, high-fidelity 3D environmental perception foundation for the entire automatic illumination method. In the subsequent target detection and illumination resampling steps, the system extracts the precise 3D position of the tracked target from the current frame point cloud data of this sequence, uses this as the origin to construct a light field description, and then calculates the amount of illumination lacking the target at the current moment.

[0032] Step S2: Perform target detection and illumination resampling on the current frame point cloud data in the color point cloud sequence to obtain target environment light field data.

[0033] The current frame point cloud data refers to the frame of point cloud data that is currently at the input end of the processing flow in the color point cloud sequence output in real time in step S1. During the continuous shooting process of the gimbal, step S1 continuously generates color point cloud sequences in chronological order and sends each newly generated frame of point cloud data into a first-in-first-out processing queue. Step S2 retrieves data frame by frame from the head of the queue for processing, and the frame of point cloud data retrieved at this time is the current frame point cloud data. The current frame point cloud data completely describes the three-dimensional geometry and optical texture of the surfaces of all visible objects within the field of view of the gimbal at the instant corresponding to its acquisition timestamp. It is composed of a massive number of color three-dimensional points, each of which contains a three-dimensional spatial coordinate in the color camera coordinate system and a set of red, green, and blue color channel values. In this application, step S2 accurately separates the spatial region of the tracked target itself from the three-dimensional snapshot, and uses this as the origin to reconstruct the illumination contribution of the surrounding environment to it into a compact representation form that is easy to analyze later, namely, the target environment light field data.

[0034] First, target detection is performed on the current frame point cloud data according to the preset detection target. The detection target is the visual feature template extracted by the user manually selecting the target through the viewfinder or automatically locking it by the system when the gimbal initiates the tracking task. This feature template can be the outline of a person's body, the distribution of facial feature points, or the appearance model of a specific object such as a race car or a bird. The target detection operation is not performed directly on the 3D point cloud when processing the current frame point cloud data. Instead, it utilizes the pixel-level mapping relationship known in step S1 to extract the corresponding color image frame of the current frame point cloud data and runs the visual tracking algorithm on that color image frame. This algorithm searches for the image region that best matches the preset detection target feature template and outputs a 2D bounding box surrounding that region. Since step S1 already associated each 3D point with its precise pixel coordinates on the color image frame when generating the current frame point cloud data, this mapping relationship is used for reverse lookup: all points whose corresponding projection positions in 3D space fall within the 2D bounding box are filtered out, and these points together constitute a subset of the target region point cloud. The aforementioned target detection operation can accurately segment a local point cloud belonging only to the tracked target from a complex point cloud containing foreground, background, ground, sky, and other elements. This eliminates interference from background objects for accurate subsequent calculation of the target's spatial position. For example, when a gimbal automatically tracks a skier, regardless of whether there is a dense forest or other skiers behind the skier, after target detection based on their feature templates, the resulting subset of the target area point cloud only contains the outline points of the skier's body, while points from tree trunks or other people in the background are completely removed.

[0035] After obtaining a clean subset of the target region point cloud, the three-dimensional spatial coordinates within this subset are aggregated using average values ​​to obtain the target centroid coordinates. In this operation, the X-coordinate value of each point within the target region point cloud subset is summed and divided by the total number of points to obtain the centroid's X-coordinate value; the Y and Z coordinates are calculated in the same way. The calculated target centroid coordinates represent a single three-dimensional spatial point, geometrically representing the center of the tracked target's three-dimensional volume at the current moment. The average aggregation operation used in this application is computationally simple, has a rapid response, and effectively smooths out minor fluctuations on the point cloud surface caused by sensor noise or local occlusion, providing a stable and representative target spatial position reference. The target centroid coordinates serve as the spatial origin for all subsequent lighting geometry calculations. For example, for a race car being tracked and filmed by a gimbal, its target region point cloud subset contains thousands of points from the car body and driver. The centroid coordinates obtained through averaging will fall near the geometric center of the race car, and subsequent analysis will focus on the distribution of ambient light illuminating the entire race car around this center point.

[0036] After establishing the target's centroid coordinates, the lighting environment surrounding the target is structurally sampled. Specifically, a candidate coordinate set is first selected from the current frame's point cloud data based on a preset distance threshold and the target's centroid coordinates. The preset distance threshold is determined by adding the maximum effective range of the supplementary lighting module to the typical scale of the tracked target. During the selection process, all 3D points in the current frame's point cloud data are traversed, and the 3D straight-line distance from each point to the target's centroid coordinates is calculated. When the calculated distance is less than or equal to the preset distance threshold, the 3D coordinates of that point are retained and added to the candidate coordinate set. Distance filtering filters out a large number of distant points that contribute little to lighting analysis but significantly increase computational burden, compressing the amount of data processed later into a reasonable and meaningful spatial range for the target's lighting. For example, in an indoor shooting scene, if the preset distance threshold is set to five meters, the white wall two meters away from the athlete and the brown floor four meters away will be retained, while the faint light at the end of the corridor ten meters away will be reasonably ignored.

[0037] After selecting the candidate coordinate set, a geometric relationship is established for each coordinate point pointing towards the target centroid. The spatial vector between each coordinate point in the candidate coordinate set and the target centroid coordinates is calculated. This is done by subtracting the X, Y, and Z components of the target centroid coordinates from the three-dimensional coordinates of the candidate point, resulting in a three-dimensional vector originating from the target centroid and pointing towards the candidate point. This three-dimensional vector not only contains the candidate point's orientation relative to the target, but its length directly represents the distance between the candidate point and the target. Subsequently, using the target centroid coordinates as the origin, a normalized projection is performed on each coordinate point in the candidate coordinate set based on the calculated spatial vector. The specific operation of the normalized projection is to scale the length of the spatial vector to a value of 1 while keeping its direction unchanged. The endpoint of the original spatial vector then falls on a unit sphere with the target centroid as its center and a radius of 1. The point falling on this unit sphere is the light field projection point corresponding to that candidate coordinate point.

[0038] The color values ​​and spatial vectors corresponding to each coordinate point in the candidate coordinate set are associated and stored with the light field projection points. Each 3D point in the candidate coordinate set is itself a colored point with RGB color information output in step S1. The red, green, and blue color values ​​of this point, as well as its spatial vector to the target centroid, are combined and bound to its corresponding light field projection point. After performing this association storage on all candidate points, a series of data combinations are obtained, each of which describes the light field information in a certain direction as seen from the target: the light field projection point indicates the precise solid angle direction, the length of the spatial vector gives the distance to the light source, and the color value describes the brightness and color of the light source. This series of data combinations is called the raw light field data. The raw light field data is the original material library for subsequently constructing standardized target environment light field data.

[0039] After obtaining the original light field data, it is standardized and sampled to become a final form with fixed dimensions that can be directly used for illumination calculations—that is, the target environment light field data. Specifically, based on the spatial vector and a preset anchor point feature set, candidate anchor point sets corresponding to each anchor point feature are selected from the light field projection points of the original light field data. The preset anchor point feature set is a fixed number of uniformly distributed spatial direction points generated on a unit sphere using the Fibonacci spiral algorithm during system initialization; each anchor point feature represents a discrete sampling direction. The selection operation is as follows: for each anchor point in the anchor point feature set, all light field projection points in the original light field data are traversed, and the angle between the vector of the unit direction of the light field projection point and the vector of the anchor point direction is calculated. Those light field projection points with the smallest angle or the closest directions are grouped into a set; this set is the candidate anchor point set centered on that anchor point feature. By using pre-laid, uniform anchor points as a sampling grid, the continuously distributed, variable number of light field projection points on the sphere are assigned to these finite, fixed sampling directions, thereby achieving spatial regularization of light field information.

[0040] After assigning the continuous light field to a discrete anchor point grid, the candidate anchor point set corresponding to an anchor point feature may contain multiple light field projection points from different distances. Therefore, it is necessary to calculate the Euclidean distance between each projection point in the candidate anchor point set and the target centroid coordinates. The distance value used for calculation here is already implied in the length of the spatial vector associated with the light field projection points. Among all associated projection points, the light field projection point with the smallest Euclidean distance is found and used as the target anchor point corresponding to that anchor point feature. By retaining the nearest point, the illumination contribution of background objects occluded by the foreground in the same direction can be discarded. For example, a bright window is located four meters to the left front of the target, while there is a dark bookshelf two meters between the window and the target. For anchor points in this direction, the projection point distance from the bookshelf is two, and the projection point distance from the window is four. After the nearest point retention filter, the light field projection point of the dark bookshelf will be retained as the target anchor point of that anchor point, while the illumination from the distant window will be blocked. This is consistent with the physical situation where the target is actually occluded by the bookshelf and mainly receives the influence of the faint light from the bookshelf surface.

[0041] Finally, the color values ​​and spatial vectors associated with the light field projection points are stored in association with the target anchor points. That is, the RGB color value of the light field projection point representing the final illumination contribution in the direction of the anchor point, and its spatial vector to the target centroid, are written into the data structure of the anchor point feature. For anchor points that fail to match any light field projection points in the original light field data, their data structures remain uninitialized to represent near-field surfaces not directly observed by the sensor in that direction. After all anchor point features have undergone this assignment or are left empty, the fixed-size structured dataset composed of these anchor points containing color and geometric information is the final target environment light field data. The target environment light field data accurately depicts the local illumination environment of the tracked target in the current shooting frame in an extremely compressed yet visually complete manner. It provides a direct, quantified, and uniformly formatted input basis for subsequent difference comparison with preset illumination target distribution data and inverse solution of illumination compensation.

[0042] Step S3: Compare the differences between the preset target illumination distribution data and the target environment light field data, and perform reverse solution for illumination compensation to generate a set of supplementary lighting control parameters. Perform supplementary lighting according to the set of supplementary lighting control parameters.

[0043] The target environment light field data is a structured local light field snapshot with the three-dimensional centroid of the tracked target as the origin and composed of a fixed number of anchor point features. In the dataset, each anchor point corresponds to a specific spatial direction on a unit sphere. Some anchor points have been successfully assigned color values ​​and spatial vectors from the original light field data, while others may be in an uninitialized state due to the field of view limitations of the gimbal camera, indicating a lack of direct sensor observation in that direction. The preset target illumination distribution data is a pre-designed data model describing the omnidirectional incident light distribution that the target position should receive under ideal shooting conditions. The specific content of this data model is a set of ideal coefficients defined on the spherical harmonic basis functions, describing an illumination distribution with omnidirectional uniformity, softness, and neutral color temperature characteristics, such as setting the illumination chromaticity to 5,500 Kelvin natural white light in accordance with photographic standards. The preset target illumination distribution data was formulated during the system design phase based on photographic aesthetics and imaging sensor characteristics, with the goal of providing a constant and ideal reference benchmark for subsequent difference comparisons.

[0044] First, information completion is performed on the target ambient light field data. It should be understood that while the target ambient light field data is precise, it is inherently incomplete. At any given moment, the gimbal camera can only capture the surface of objects within its field of view, leaving anchor points behind the target and outside the edge of the field of view in an uninitialized state. Directly using this incomplete data for illumination difference analysis will lead to biased and erroneous judgments that underestimate the total amount of ambient light due to missing information. Information completion processing is designed to address this incompleteness problem. This processing is achieved by analyzing the spatial context relationships between initialized anchor points: for each uninitialized anchor point, all initialized anchor points are traversed, calculating the angle between the spatial direction of the initialized anchor point and the direction of the current empty anchor point, as well as the color intensity and distance value of the initialized anchor points. Based on the spatial direction, color intensity, and distance information provided by these initialized anchor points, an adaptive spatial context aggregation function is used to infer the brightness and color of the ambient light that may exist in the blank direction, and the calculation results are assigned to the empty anchor point. In real physical environments, illumination typically exhibits spatial continuity and regularity. For instance, ambient light reflected from a white wall does not change abruptly in brightness and color within a nearby solid angle range. For example, when a pan-tilt-zoom (PTZ) camera tracks a speaker standing by a window, the camera can only see the window directly in front of the speaker; the interior wall behind the speaker is out of view. Through spatial context aggregation, the system can reasonably infer, based on the observed incident light from the window and the reflection characteristics of other visible surfaces in the room, that the illumination from the direction of the wall behind the speaker is roughly the diffuse reflection level of warm-colored indoor lighting, thus completing the missing data. Once all empty anchor points have been completed, a complete spherical dataset is obtained, with each anchor point assigned a valid color and distance value. Performing a spherical harmonic expansion on the complete spherical dataset—that is, calculating its inner product with a set of preset spherical harmonic basis functions—results in a set of coefficients. This set of coefficients accurately and completely describes the omnidirectional ambient incident illumination received by the tracked target at its current location and is referred to as the current illumination distribution data. The above operations transform the discrete, local, and incomplete observation snapshots output in step 2 into a continuous, complete, and differentiable mathematical function, enabling quantitative comparison with preset illumination target distribution data within the same mathematical space.

[0045] Based on a preset lighting channel set, a channel-by-channel difference comparison calculation is performed on the current lighting distribution data and the preset lighting target distribution data to obtain supplementary lighting difference distribution data. The lighting channel set represents the response characteristics of the human visual system and the physical output channels of the supplementary lighting LEDs, typically including red, green, and blue channels, and sometimes also including independent brightness channels. The specific operation of the channel-by-channel difference comparison involves reading the spherical harmonic coefficients (SHCs) of the current lighting distribution data and the preset lighting target distribution data under the same lighting channel, and then performing a successive subtraction of the coefficients. That is, for the red channel, the difference between the current lighting red channel SHC and the target lighting red channel SHC is calculated; the same operation is performed for the green and blue channels. Through subtraction, a new set of SHCs is obtained, describing a supplementary lighting function that is unevenly distributed in three-dimensional space and composed of specific colors. The supplementary lighting function indicates how much light the target still lacks, and from which directions and in what colors this lacking light needs to be supplemented. This embodiment decouples complex, comprehensive lighting decisions into pure mathematical difference calculations on three or more independent color channels. This allows subsequent inverse solving to be performed independently on each channel, simplifying the problem while ensuring accurate overall tonal balance of the target after lighting. For example, in a scene with mixed lighting of warm sunlight and blue sky at dusk, channel-by-channel difference comparison might reveal that the blue channel lighting on the subject matches the target on the side facing the sky, but is less effective on the side facing the warm-colored wall, while the red channel is the opposite. The lighting difference distribution data quantifies and records these two distinct spatial-spectral compensation requirements.

[0046] After obtaining the supplementary lighting difference distribution data, the data is solved using a least-squares optimization inverse constraint method based on the preset lighting module light distribution parameters to obtain the light source intensity weighted solution. The preset lighting module light distribution parameters are a dataset obtained through precise photometry in the laboratory before the gimbal leaves the factory, describing the light intensity distribution of each supplementary lighting module in three-dimensional space. Specifically, for each independent supplementary lighting module in the supplementary lighting array, its light distribution parameters describe the relative luminous intensity distribution in each direction on a unit sphere with the target centroid as the origin when the module is turned on. This parameter can also be represented by spherical harmonic coefficients. The goal of the inverse constraint solution is to find a set of non-negative light source intensity weight values ​​and assign them to each supplementary lighting module so that when these supplementary lighting modules emit light according to their respective weights, the combined illumination distribution they produce at the target position mathematically approximates the supplementary lighting difference distribution data as closely as possible. This is accomplished using a least-squares optimization solver with non-negative constraints. A target function is constructed to calculate the spherical harmonic coefficients (SHCs) of each supplementary lighting module. These SHCs are multiplied by their respective intensity weights and summed term by term to obtain the composite SHC. Then, the sum of squares of the differences between this composite SHC and the SHCs of the supplementary lighting difference distribution data is calculated. Under the constraint that all weight values ​​must be greater than or equal to zero, the algorithm iteratively adjusts each intensity weight until the value of the target function is minimized. The weight values ​​obtained at this point constitute the light source intensity weight solution. This reverse solution operation, which derives the required combination of light source attributes from the desired illumination effect, requires the intervention of numerical optimization techniques. Through this solution, the system accurately determines how much light each supplementary lighting module needs to emit to bridge the gap between the current ambient light and the ideal light.

[0047] After obtaining the light source intensity weighting solutions for each supplementary lighting module, to ensure the final mixed supplementary light has strictly accurate colors, the light source intensity weighting solutions need to be corrected according to the preset chroma channel ratios to obtain the multi-channel intensity values ​​of the supplementary lighting modules. The preset chroma channel ratio is a pre-calibrated set of correction coefficients that records the precise emission ratios required between the red, green, and blue LEDs or cool and warm color temperature LEDs in each supplementary lighting module to synthesize standard white light. When the supplementary lighting module supports independent multi-channel control, multiplying the light source intensity weighting solution by the channel factors in the corresponding supplementary lighting module's chroma channel ratio yields the independent intensity value for each color channel within that module. These values ​​are collectively referred to as multi-channel intensity values. For example, if a supplemental lighting module is determined to require an intensity of 0.8, and the module's chromaticity channel ratio specifies a baseline ratio of 1:1.2 for red and green LEDs, then after correction, the red channel intensity value will be 0.8, and the green channel intensity value will be 0.96, thus ensuring the synthesized light color temperature is the expected 5500 Kelvin white light. This correction operation fundamentally guarantees that the system compensates for light intensity without introducing undesirable color deviations.

[0048] Finally, the process of converting the multi-channel intensity values ​​into instructions executable by the physical pan-tilt unit (PTZ) is performed. This involves linear transformation and signal modulation of the multi-channel intensity values ​​to generate a set of supplementary lighting control parameters. The linear transformation maps floating-point numbers representing relative intensity to the hardware control range that drives the supplementary lighting LEDs. If the supplementary lighting driver chip uses eight-bit resolution pulse-width modulation (PWM), the multi-channel intensity values ​​are multiplied by 255 and rounded to the nearest integer to obtain the duty cycle values. Signal modulation involves packaging these digital values ​​into data frames conforming to the supplementary lighting module's communication protocol. These data frames include a frame header, duty cycle data for each channel, and a checksum, and are transmitted via the PTN's wired bus or low-power wireless link. This entire set of values ​​and data frames constitutes the supplementary lighting control parameter set. Upon receiving the supplementary lighting control parameter set, the supplementary lighting module's microcontroller parses it and updates its timer's compare register, thereby changing the duty cycle of the PWM signal driving the LED circuit. This allows for precise control of the actual brightness and color of the supplementary lighting within milliseconds.

[0049] By performing linear transformation and signal modulation on the multi-channel intensity values, a set of supplementary lighting control parameters is generated. This ensures that every beam of light emitted by the supplementary lighting module is precisely matched with the actual lighting gap of the target in terms of brightness, direction, and color. As a result, images with uniform brightness and accurate color reproduction can still be captured even in backlight or low-light environments.

[0050] Step S4: Perform supplementary lighting according to the supplementary lighting control parameter set.

[0051] The preset supplementary lighting module is controlled to perform supplementary lighting operation according to the supplementary lighting control parameter set.

[0052] This application obtains a color point cloud sequence by performing temporal alignment and spatial calibration fusion on the real-time acquired raw image data stream. Target detection and illumination resampling are then performed on the current frame point cloud data in the color point cloud sequence to obtain target environment light field data. Difference comparison and inverse solution of the preset illumination target distribution data and target environment light field data are then performed to generate a set of supplementary lighting control parameters, which drive a preset supplementary lighting module. This application constructs a color point cloud sequence, performs spherical resampling and anchor point matching on the target area to form target environment light field data, and then combines this with the preset illumination target distribution to generate supplementary lighting control parameters, ultimately achieving precise automatic supplementary lighting during gimbal tracking and shooting.

[0053] like Figure 2 The diagram shown is a functional block diagram of an automatic fill light device for gimbal tracking and shooting provided in an embodiment of this application.

[0054] In some embodiments, the gimbal tracking and automatic lighting device 2 may include multiple functional modules composed of computer program segments. The computer programs for each program segment in the gimbal tracking and automatic lighting device 2 may be stored in the server's memory and executed by at least one processor to perform (see details). Figure 1 (Description) The function of the gimbal tracking shooting automatic fill light method.

[0055] In this embodiment, the gimbal tracking and automatic fill light device 2 can be divided into multiple functional modules according to its function. These functional modules may include: an alignment and fusion module 21, a target sampling module 22, a difference compensation module 23, and a fill light driving module 24. The module referred to in this invention is a series of computer program segments that can be executed by at least one processor and perform a fixed function, stored in memory. In this embodiment, the functions of each module will be detailed in subsequent embodiments.

[0056] Alignment and fusion module 21 is used to perform temporal alignment and spatial calibration fusion on the real-time acquired raw image data stream to obtain a color point cloud sequence; Target sampling module 22 is used to perform target detection and illumination resampling on the current frame point cloud data in the color point cloud sequence to obtain target environment light field data; The difference compensation module 23 is used to compare the difference between the preset illumination target distribution data and the target environment light field data, and to solve the illumination compensation in reverse to generate a set of supplementary lighting control parameters. The supplementary lighting driving module 24 is used to provide supplementary lighting according to the supplementary lighting control parameter set.

[0057] It should be understood that the various variations and specific embodiments of the methods provided in the above embodiments are also applicable to the gimbal tracking and shooting automatic fill light device of this embodiment. Through the foregoing detailed description of the gimbal tracking and shooting automatic fill light method, those skilled in the art can clearly understand the implementation method of the gimbal tracking and shooting automatic fill light device in this embodiment. For the sake of brevity, it will not be described in detail here.

[0058] like Figure 3 The diagram shown is a structural schematic of the gimbal provided in an embodiment of this application.

[0059] In a preferred embodiment of the present invention, the gimbal 3 may include, but is not limited to: a memory 31, at least one processor 32, and at least one communication bus 33.

[0060] Those skilled in the art should understand that Figure 3The structure of the gimbal 3 shown does not constitute a limitation of the embodiments of the present invention. The gimbal 3 may also include more or fewer other hardware or software than shown, or different component arrangements.

[0061] In some embodiments, the gimbal 3 is a gimbal that can automatically perform numerical calculations and / or information processing according to pre-set or stored instructions, and its hardware includes, but is not limited to, microprocessors, application-specific integrated circuits, programmable gate arrays, digital processors, and embedded gimbals.

[0062] It should be noted that the gimbal 3 described is merely an example. Other existing or future electronic products that are applicable to this application should also be included within the scope of protection of this application and are incorporated herein by reference.

[0063] In some embodiments, the memory 31 stores a computer program that, when executed by the at least one processor 32, implements all or part of the steps in the gimbal tracking and automatic supplemental lighting method described above. The memory 31 includes a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), a one-time programmable read-only memory (OTPROM), an electronically erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM) or other optical disc storage, disk storage, magnetic tape storage, or any other computer-readable medium capable of carrying or storing data. Further, the computer-readable storage medium may primarily include a program storage area and a data storage area, wherein the program storage area may store an operating system, at least one application program required for a function, etc.

[0064] In some embodiments, the at least one processor 32 is the control unit of the gimbal 3, connecting various components of the gimbal 3 via various interfaces and lines. It executes programs or modules stored in the memory 31 and calls data stored in the memory 31 to perform various functions of the gimbal 3 and process data. For example, when the at least one processor 32 executes the computer program stored in the memory 31, it implements all or part of the steps of the gimbal tracking and shooting automatic supplementary lighting method described in this embodiment; or it implements all or part of the functions of the gimbal tracking and shooting automatic supplementary lighting device. The at least one processor 32 can be composed of integrated circuits, such as a single-packaged integrated circuit or multiple integrated circuits with the same or different functions, including combinations of one or more central processing units (CPUs), microprocessors, digital processing chips, graphics processors, and various control chips.

[0065] In some embodiments, the at least one communication bus 33 is configured to enable communication between the memory 31 and the at least one processor 32, etc. Although not shown, the gimbal 3 may also include a power supply (e.g., a battery) to power the various components. Preferably, the power supply can be logically connected to the at least one processor 32 via a power management device, thereby enabling functions such as charging, discharging, and power consumption management. The power supply may also include one or more DC or AC power supplies, recharging devices, power fault detection circuits, power converters or inverters, power status indicators, and other arbitrary components. The gimbal 3 may also include various sensors, Bluetooth modules, Wi-Fi modules, etc., which will not be elaborated further here.

[0066] The integrated unit implemented as a software functional module described above can be stored in a computer-readable storage medium. This software functional module, stored in a storage medium, includes several instructions to cause a gimbal (which may be a personal computer, a gimbal, or a network gimbal, etc.) or a processor to execute portions of the methods described in the various embodiments of this application.

[0067] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules is only a logical functional division, and other division methods may be used in actual implementation.

[0068] The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical units; they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0069] The above are all preferred embodiments of this application, and are not intended to limit the scope of protection of this application. Therefore, all equivalent changes made in accordance with the structure, shape and principle of this application should be covered within the scope of protection of this application.

Claims

1. A method for automatic fill light during gimbal tracking and shooting, characterized in that, The method includes: The process involves temporal alignment and spatial calibration fusion of real-time acquired raw image data streams to obtain a color point cloud sequence. This includes: performing temporal nearest neighbor matching on real-time acquired depth image frames and color image frames based on the acquisition timestamp and a preset time difference threshold to form synchronously acquired depth and color data pairs; back-projecting the depth pixel coordinates and corresponding depth values ​​in the depth and color data pairs according to a preset intrinsic parameter matrix of the depth sensor to generate a first 3D spatial point in the depth camera coordinate system; converting the first 3D spatial point into a second 3D spatial point in the color camera coordinate system according to preset spatial transformation parameters; forward-projecting the second 3D spatial point according to a preset intrinsic parameter matrix of the color sensor to obtain sub-pixel coordinates corresponding to each second 3D spatial point; performing bilinear interpolation on the color values ​​in the depth and color data pairs based on the sub-pixel coordinates to generate point cloud data for each data pair; and sorting the point cloud data according to temporal order to obtain a color point cloud sequence. The method involves performing target detection and illumination resampling on the current frame point cloud data in the color point cloud sequence to obtain target environment light field data. This includes: performing target detection on the current frame point cloud data in the color point cloud sequence according to a preset detection target to obtain a subset of target region point clouds; calculating the target centroid coordinates based on the target region point cloud subset; performing illumination resampling based on a preset distance threshold and the target centroid coordinates to obtain original light field data; and performing direction matching and nearest-point retention processing on the original light field data based on a preset anchor point feature set to obtain target environment light field data. The method involves performing a difference comparison and inverse solution for illumination compensation on preset target illumination distribution data and target ambient light field data to generate a set of supplementary lighting control parameters. This includes: performing spatial context aggregation to complete the target ambient light field data and obtain current illumination distribution data; calculating a channel-by-channel difference comparison between the current illumination distribution data and preset target illumination distribution data based on a preset illumination channel set to obtain supplementary lighting difference distribution data; performing inverse least-squares optimization on the supplementary lighting difference distribution data based on preset light distribution parameters to obtain a light source intensity weight solution; correcting the light source intensity weight solution based on a preset chromaticity channel ratio to obtain multi-channel intensity values; and performing linear transformation and signal modulation on the multi-channel intensity values ​​to generate a set of supplementary lighting control parameters. Supplemental lighting is performed according to the set of supplemental lighting control parameters.

2. The automatic fill light method for gimbal tracking and shooting according to claim 1, characterized in that, The step of calculating the target centroid coordinates based on the target region point cloud subset, and resampling the illumination according to a preset distance threshold and the target centroid coordinates to obtain the original light field data includes: The average values ​​of the three-dimensional spatial coordinates in the target region point cloud subset are aggregated to obtain the target centroid coordinates. Based on a preset distance threshold and the target centroid coordinates, a set of candidate coordinates is selected from the current frame point cloud data; Calculate the spatial vector between each coordinate point in the candidate coordinate set and the target centroid coordinates. Using the target centroid coordinates as the origin, perform normalized projection on each coordinate point in the candidate coordinate set according to the spatial vector to obtain the light field projection point corresponding to each coordinate point. The color values ​​and spatial vectors corresponding to each coordinate point in the candidate coordinate set are associated and stored with the light field projection points to obtain the original light field data.

3. The automatic fill light method for gimbal tracking and shooting according to claim 2, characterized in that, The step of performing direction matching and nearest-point retention processing on the original light field data based on a preset anchor point feature set to obtain the target environment light field data includes: Based on the spatial vector and the preset anchor point feature set, candidate anchor point sets corresponding to each anchor point feature are selected from the light field projection points of the original light field data. Calculate the Euclidean distance between each projection point in the candidate anchor point set and the target centroid coordinates, and take the light field projection point corresponding to the smallest Euclidean distance as the target anchor point corresponding to the anchor point feature; The color values ​​and spatial vectors associated with the light field projection points are associated with and stored with the target anchor points to obtain the target environment light field data.

4. The automatic fill light method for gimbal tracking and shooting according to claim 1, characterized in that, The step of performing a channel-by-channel difference comparison calculation on the current illumination distribution data and the preset illumination target distribution data according to a preset illumination channel set to obtain supplementary lighting difference distribution data includes: Read the spherical harmonic coefficients of the current illumination distribution data and the preset illumination target distribution data under the same illumination channel, and then perform successive subtraction of the coefficients to obtain the supplementary light difference distribution data.

5. The automatic fill light method for gimbal tracking and shooting according to claim 4, characterized in that, The method further includes: After obtaining the supplementary lighting difference distribution data, the supplementary lighting difference distribution data is solved by least squares optimization in reverse constraint solution according to the preset light distribution parameters of the supplementary lighting module to obtain the light source intensity weight solution. The output light intensity of the supplementary lighting module is controlled according to the light source intensity weight solution to fill the gap between the current ambient light and the ideal light.

6. A gimbal-guided tracking and automatic fill light device, characterized in that, The device includes: The alignment and fusion module is used to perform temporal alignment and spatial calibration fusion on the real-time acquired raw image data stream to obtain a color point cloud sequence. This includes: performing temporal nearest neighbor matching on real-time acquired depth image frames and color image frames based on the acquisition timestamp and a preset time difference threshold to form synchronously acquired depth and color data pairs; performing back projection on the depth pixel coordinates and corresponding depth values ​​in the depth and color data pairs according to a preset intrinsic parameter matrix of the depth sensor to generate a first three-dimensional spatial point in the depth camera coordinate system; converting the first three-dimensional spatial point into a second three-dimensional spatial point in the color camera coordinate system according to preset spatial transformation parameters; performing forward projection on the second three-dimensional spatial point according to a preset intrinsic parameter matrix of the color sensor to obtain the sub-pixel coordinates corresponding to each second three-dimensional spatial point; performing bilinear interpolation on the color values ​​in the depth and color data pairs based on the sub-pixel coordinates to generate point cloud data for each data pair; and sorting the point cloud data according to temporal order to obtain a color point cloud sequence. The target sampling module is used to perform target detection and illumination resampling on the current frame point cloud data in the color point cloud sequence to obtain target environment light field data. This includes: performing target detection on the current frame point cloud data in the color point cloud sequence according to a preset detection target to obtain a target region point cloud subset; calculating the target centroid coordinates based on the target region point cloud subset; performing illumination resampling based on a preset distance threshold and the target centroid coordinates to obtain original light field data; and performing direction matching and nearest-point retention processing on the original light field data based on a preset anchor point feature set to obtain target environment light field data. The difference compensation module is used to perform difference comparison and inverse solution of illumination compensation on preset illumination target distribution data and target environment light field data to generate a set of supplementary lighting control parameters. This includes: performing spatial context aggregation information completion processing on the target environment light field data to obtain current illumination distribution data; performing channel-by-channel difference comparison calculation on the current illumination distribution data and preset illumination target distribution data according to a preset illumination channel set to obtain supplementary lighting difference distribution data; performing inverse constraint solution of least squares optimization on the supplementary lighting difference distribution data according to preset light distribution parameters to obtain a light source intensity weight solution; correcting the light source intensity weight solution according to a preset chromaticity channel ratio to obtain multi-channel intensity values; and performing linear transformation and signal modulation on the multi-channel intensity values ​​to generate a set of supplementary lighting control parameters. A supplementary lighting driver module is used to provide supplementary lighting based on the supplementary lighting control parameter set.

7. A gimbal, characterized in that, The gimbal includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the steps of the gimbal tracking and shooting automatic supplementary lighting method according to any one of claims 1 to 5.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the steps of the gimbal tracking and automatic fill light method according to any one of claims 1 to 5.

Images