Gimbal vision loss recovery method and recovery system

Abstract

This application provides a method and system for recovering visual tracking loss caused by a gimbal, applied in the field of gimbal technology. The method includes: when a target is lost during the process of the gimbal tracking body tracking and capturing a target, determining a first motion direction trend of the gimbal body based on the motion trend of the inertial measurement unit; controlling the gimbal body to perform target operations; performing visual detection during the target operations, and controlling the gimbal body to track and capture the target when the gimbal body visually locks onto the target; the target operations include: controlling the gimbal body to move according to the first motion direction trend; and controlling the camera of the gimbal body to zoom based on the first motion trend confidence level when the confidence level of the first motion trend is greater than a preset confidence threshold. This method enables active recovery of tracking after target loss.

Description

Technical Field

[0001] This application relates to the field of gimbal technology, and in particular to a method and system for recovering visual loss from a gimbal. Background Technology

[0002] In intelligent gimbal tracking systems, visual algorithms continuously lock onto targets and drive the gimbal to follow them through image recognition. Related technologies, after a target is lost, predict future trajectories based on historical motion trajectories within the image using a Markov / Bayes model and adjust the gimbal accordingly, performing full-image re-identification within a fixed window (≥2 s).

[0003] The related technologies have the following limitations: Motion inference after loss of target relies entirely on visual historical trajectories. This means that once the target is obscured or moves out of the frame, the lack of effective observation data causes the inference to quickly fail, leading to the gimbal stopping or spinning erratically. In this situation, the gimbal's tracking and shooting cannot be actively recovered; it can only be passively recovered through human intervention. Therefore, existing gimbal tracking and shooting technologies suffer from the problem of being unable to actively recover after the target is obscured or moves out of the frame. Summary of the Invention

[0004] This application provides a method and system for recovering lost vision from a gimbal, which can solve the problem that existing gimbal tracking and shooting technologies cannot actively recover vision after the target is obscured or moves out of the frame.

[0005] In a first aspect, embodiments of this application provide a gimbal vision loss recovery method, applied to a gimbal vision loss recovery system. The gimbal vision loss recovery system includes a gimbal main body and an extension component. The gimbal main body is used to track and capture images of a target, and the extension component is worn on the target. The extension component includes an inertial measurement unit. The method includes: In the process of the gimbal body tracking and shooting the tracked target, if the target is lost due to visual tracking, the first motion direction trend of the gimbal body is determined based on the motion trend of the inertial measurement unit. Control the gimbal body to perform the target operation; During the process of the gimbal body performing the target operation, visual detection is performed. If the gimbal body visually locks onto the tracking target, the gimbal body is controlled to track and photograph the tracking target. The target operation includes: Control the gimbal body to move in accordance with the first direction of motion; Furthermore, if the confidence level of the first motion trend is greater than a preset confidence threshold, the camera of the gimbal body is controlled to zoom based on the confidence level of the first motion trend. The confidence level of the first motion trend is determined based on the projection of the first motion direction trend onto the optical axis of the camera of the gimbal body. The confidence level of the first motion trend is used to indicate the degree to which the tracked target is approaching or moving away from the camera.

[0006] Optionally, determining the first motion direction trend of the gimbal body based on the motion trend of the inertial measurement unit includes: The second motion direction trend measured by the inertial measurement unit is obtained, and the second motion direction trend is the motion direction trend in the first coordinate system corresponding to the measurement by the inertial measurement unit. Based on the mapping relationship between the first coordinate system and the second coordinate system corresponding to the gimbal body, the second motion direction trend is mapped to the second coordinate system to obtain the first motion direction trend.

[0007] Optionally, before mapping the second motion direction trend to the second coordinate system based on the mapping relationship between the first coordinate system and the second coordinate system corresponding to the gimbal body to obtain the first motion direction trend, the method further includes: During the process of the gimbal body tracking and capturing the target, the motion directions of N first visual tracking frames of the gimbal body and the N first velocity directions of the inertial measurement unit are obtained in the first time period. The motion directions of the center of the N first visual tracking frames and the N first velocity directions correspond one-to-one according to the time frame corresponding to the acquisition. Based on the motion directions of the centers of the N first visual tracking boxes and the N first velocity directions, a covariance matrix is ​​constructed; The covariance matrix is ​​orthogonally decomposed to obtain the target rotation matrix; The motion directions of the center of M second visual tracking frames of the gimbal body and the M second velocity directions of the inertial measurement unit are obtained in the second time period. The second time period is the period after the first time period. The motion directions of the center of M second visual tracking frames and the M second velocity directions correspond one-to-one with the time frames corresponding to the acquisition time. Based on the center motion directions of the M second visual tracking boxes and the M second velocity directions, the target rotation matrix is ​​verified. If the verification is successful, the target rotation matrix is ​​determined as the mapping relationship.

[0008] Optionally, obtaining the second motion direction trend measured by the inertial measurement unit includes: The pose, angular velocity, motion trend, and velocity of the inertial measurement unit are obtained. Based on the pose and the angular velocity, determine the pose integral; Based on the motion trend, the velocity, and the pose integral, a motion trend vector is determined, wherein the second motion direction trend includes the motion trend vector.

[0009] Optionally, the camera controlling the gimbal body to zoom based on the first motion trend confidence includes: Based on the first motion trend confidence level, the camera on the gimbal body is controlled to perform a one-step zoom; or... Based on the first motion trend confidence level, the camera of the gimbal body is controlled to perform a one-step zoom, and if the duration of the one-step zoom performed by the camera of the gimbal body based on the first motion trend confidence level exceeds a preset time threshold, continuous zoom is performed based on the first motion trend confidence level.

[0010] Optionally, before controlling the gimbal body to perform the target operation, the method further includes: The inertial measurement unit (IMU) measures the velocity of the last frame, and a preset maximum waiting time and a preset minimum waiting time are obtained. The velocity of the last frame is the velocity measured by the IMU in the last frame before the visual tracking of the gimbal body loses the target. The preset time threshold is determined based on the last frame velocity, and the preset time threshold is between the preset maximum waiting time and the preset minimum waiting time.

[0011] Optionally, before controlling the gimbal body to perform the target operation, the method further includes: The confidence level of the second motion trend is determined based on the projection of the first motion direction trend onto the optical axis of the camera on the gimbal body. Obtain the received signal strength index (RSSI) rate of change of the signal received by the gimbal body from the extended component; The confidence level of the second motion trend is adjusted based on the rate of change of RSSI to obtain the confidence level of the first motion trend.

[0012] Optionally, the camera controlling the gimbal body zooms based on the first motion trend confidence level, including: When the first motion trend confidence level indicates that the tracked target is moving away from the camera, the camera of the gimbal body is controlled to increase the focal length, wherein the step size for increasing the focal length is a first preset step size.

[0013] Optionally, the camera controlling the gimbal body zooms based on the first motion trend confidence level, including: When the first motion trend confidence level indicates that the tracked target is approaching the camera, the camera of the gimbal body is controlled to reduce the focal length, wherein the step size for reducing the focal length is a second preset step size.

[0014] Secondly, embodiments of this application also provide a gimbal vision loss recovery system, the gimbal vision loss recovery system including a gimbal body, an extension component and a processor, the gimbal body being used to track and capture a target, the extension component being worn on the target, the extension component including an inertial measurement unit, and the processor being configured to execute the gimbal vision loss recovery method described in the first aspect.

[0015] In this embodiment, an extension component including an inertial measurement unit is worn on the tracked target. In the event of visual tracking loss of the target, the motion trend of the tracked target can be determined through the motion trend of the inertial measurement unit, thereby determining the first motion direction trend of the gimbal body. This drives the gimbal body to actively move in the direction of the first motion direction trend, thus achieving active recovery tracking after target loss. Furthermore, the projection of the motion direction trend onto the optical axis generates a first motion trend confidence level. This means that it is possible not only to determine which direction the tracked target is moving, but also whether the target is "approaching" or "moving away" from the lens. Based on this, the focal length is dynamically adjusted to ensure that when the tracked target reappears, it falls into the detection screen window with an appropriate imaging size, further promoting active recovery tracking after target loss. Attached Figure Description

[0016] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the description of the embodiments of this application will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0017] Figure 1 This is a flowchart of the gimbal vision loss recovery method provided in the embodiments of this application; Figure 2 This is a flowchart of the three-mode switching of the gimbal vision loss recovery method provided in the embodiments of this application; Figure 3 This is a flowchart of the IMU-to-gimbal coordinate system mapping and pre-pointing of the gimbal vision loss recovery method provided in the embodiments of this application; Figure 4 This is a flowchart of the optical axis projection master criterion, link confidence correction, and zoom triggering of the gimbal vision loss recovery method provided in this application embodiment; Figure 5 This is a structural diagram of a gimbal vision loss recovery system provided in an embodiment of this application. Detailed Implementation

[0018] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0019] This application provides a method for recovering visual loss from a gimbal, applied to a gimbal visual loss recovery system. The system includes a gimbal main body and an extension component. The gimbal main body is used to track and capture images of a target, and the extension component is worn on the target. The extension component includes an inertial measurement unit (IMU). The extension component also includes a wireless communication module for communication between the extension component and the gimbal main body. The gimbal main body includes a camera, a motor drive, and a visual processing unit. The extension component can be worn on the target (armband, waist pack, etc.) to acquire the target's motion direction and acceleration; after visual failure, it performs short-time integral estimation of the heading / velocity direction to drive pre-pointing, without outputting reliable displacement or directly sensing the radial distance relative to the camera.

[0020] See Figure 1 , Figure 1 This is a flowchart of the gimbal vision loss recovery method provided in the embodiments of this application, such as... Figure 1 As shown, it includes the following steps: Step 101: In the process of the gimbal body tracking and shooting the target, if the target is lost due to visual tracking, the first motion direction trend of the gimbal body is determined based on the motion trend of the inertial measurement unit.

[0021] Step 102: Control the gimbal body to perform the target operation.

[0022] Step 103: During the operation of the target by the gimbal body, visual detection is performed. If the gimbal body visually locks onto the tracking target, the gimbal body is controlled to track and photograph the tracking target.

[0023] The target operation includes: Control the gimbal body to move in accordance with the first direction of motion; Furthermore, if the confidence level of the first motion trend is greater than a preset confidence threshold, the camera of the gimbal body is controlled to zoom based on the confidence level of the first motion trend. The confidence level of the first motion trend is determined based on the projection of the first motion direction trend onto the optical axis of the camera of the gimbal body. The confidence level of the first motion trend is used to indicate the degree to which the tracked target is approaching or moving away from the camera.

[0024] Among them, gimbal-based subject tracking shooting refers to the camera's image recognition algorithm detecting the tracking target in the frame in real time. During the process of gimbal-based subject tracking shooting of the tracking target, the working mode of the gimbal-based subject can be called Mode A (visual tracking mode). In Mode A, the gimbal-based subject continuously detects and tracks the tracking target, and determines the bounding box of the tracking target and the tracking confidence level. The tracking confidence level is used to indicate the confidence level that the tracking target is completely within the shooting frame of the gimbal-based subject.

[0025] Target loss in visual tracking of the gimbal body refers to the inability to determine the target's position in the frame during tracking and shooting, resulting in tracking interruption. This includes situations such as the target completely exiting the frame or being partially obscured. For example, when tracking a runner (wearing an extension device), if the runner is obscured while passing through bushes, their position is lost in the frame; this is a case of target loss in visual tracking. In an optional implementation, whether the gimbal body exhibits target loss in visual tracking is determined by the tracking confidence level. If the tracking confidence level is below the threshold for L consecutive frames, it can be determined that the gimbal body has lost visual tracking of the target, where L is an integer greater than 1.

[0026] For example, when tracking a runner (wearing an extension device) through bushes, if the tracking confidence score is less than 0.3 (threshold) for 5 consecutive frames, it is determined that the gimbal has lost visual tracking target.

[0027] The inertial measurement unit is worn on the tracking target. The motion trend of the inertial measurement unit is the same as the motion trend of the tracking target. The inertial measurement unit can send the motion trend information of the tracking target to the gimbal body through a wireless communication module (such as Bluetooth). The gimbal body can determine the first motion direction trend based on the motion trend information. The first motion direction trend is in the same direction as the motion trend of the tracking target. If the gimbal body moves according to the first motion direction trend, the probability of capturing the tracking target can be increased.

[0028] In step 102, the target operation performed by controlling the gimbal body includes driving the gimbal to pre-point compensation according to the first motion direction trend, and maintaining the lens facing the motion direction inferred from the first motion direction trend. It should be noted that the first motion direction trend includes the heading trend and the velocity direction, rather than the absolute position or displacement.

[0029] The target operation also includes: The camera controlling the gimbal body zooms based on the first motion trend confidence level. For example, if the tracked target walks towards the lens and the distance is too close, part of the tracked target image will be lost in the picture. This will trigger the tracking confidence level to be below the threshold for L consecutive frames. In this case, simply controlling the gimbal body to move according to the first motion direction trend is not enough to capture the entire tracked target, and the focal length needs to be reduced. Whether to zoom depends on whether the first motion trend confidence level is greater than the preset confidence threshold. It can be understood that if the tracked target has walked out of the frame captured by the gimbal body, but only slightly approaches or moves away from the lens, then the first motion trend confidence level will also be relatively small. At this time, there is no need to zoom (reduce unnecessary actions), and it is only necessary to move according to the first motion direction trend. However, if the first motion trend confidence level is greater than the preset confidence threshold, it means that moving according to the first motion direction trend is not enough to capture the entire tracked target, and then zooming is required.

[0030] Step 103: During the execution of the target operation, visual detection is performed in parallel within the Region of Interest (ROI) centered on the IMU pre-pointing direction (the direction pointed to by the first motion direction trend); when the detection is successful, visual tracking is resumed directly without waiting for timeout or entering the active recapture stage.

[0031] In this embodiment, an extension component including an inertial measurement unit is worn on the tracked target. This transfers the sensing data source from the camera's view to a motion sensor on the target itself. In the event of visual tracking loss, the motion trend of the tracked target can be determined through the motion trend of the inertial measurement unit, thereby determining the first motion direction trend of the gimbal. This drives the gimbal to actively move in the direction of the first motion direction trend, thus achieving active recovery tracking after target loss. Furthermore, the projection of the motion direction trend onto the optical axis generates a first motion trend confidence level. This means that it can not only determine which direction the tracked target is moving, but also whether the target is "approaching" or "moving away" from the lens. Based on this, the focal length is dynamically adjusted to ensure that when the tracked target reappears, it falls into the detection window with an appropriate imaging size, further promoting active recovery tracking after target loss.

[0032] Optionally, determining the first motion direction trend of the gimbal body based on the motion trend of the inertial measurement unit includes: The second motion direction trend measured by the inertial measurement unit is obtained, and the second motion direction trend is the motion direction trend in the first coordinate system corresponding to the measurement by the inertial measurement unit. Based on the mapping relationship between the first coordinate system and the second coordinate system corresponding to the gimbal body, the second motion direction trend is mapped to the second coordinate system to obtain the first motion direction trend.

[0033] Understandably, the first coordinate system of the inertial measurement unit (IMU) is different from the second coordinate system of the gimbal. The second motion direction trend measured by the IMU is the motion direction trend in the first coordinate system. If this trend is directly applied to the gimbal, it will lead to a large error, affecting the efficiency of tracking recovery and even causing tracking recovery failure. However, by mapping the second motion direction trend to the first motion direction trend according to the mapping relationship between the first and second coordinate systems, the above problem can be solved, improving the success rate and efficiency of tracking recovery.

[0034] Optionally, before mapping the second motion direction trend to the second coordinate system based on the mapping relationship between the first coordinate system and the second coordinate system corresponding to the gimbal body to obtain the first motion direction trend, the method further includes: During the process of the gimbal body tracking and capturing the target, the motion directions of N first visual tracking frames of the gimbal body and the N first velocity directions of the inertial measurement unit are obtained in the first time period. The motion directions of the center of the N first visual tracking frames and the N first velocity directions correspond one-to-one according to the time frame corresponding to the acquisition. Based on the motion directions of the centers of the N first visual tracking boxes and the N first velocity directions, a covariance matrix is ​​constructed; The covariance matrix is ​​orthogonally decomposed to obtain the target rotation matrix; The motion directions of the center of M second visual tracking frames of the gimbal body and the M second velocity directions of the inertial measurement unit are obtained in the second time period. The second time period is the period after the first time period. The motion directions of the center of M second visual tracking frames and the M second velocity directions correspond one-to-one with the time frames corresponding to the acquisition time. Based on the center motion directions of the M second visual tracking boxes and the M second velocity directions, the target rotation matrix is ​​verified. If the verification is successful, the target rotation matrix is ​​determined as the mapping relationship.

[0035] In this embodiment, during the first time period when the gimbal body is in mode A, the motion directions of multiple first visual tracking frames of the gimbal body are simultaneously acquired. and the multiple first velocity directions of the inertial measurement unit After removing the data from the static frames and low-confidence frames, N first visual tracking box center motion directions and N first velocity directions are obtained.

[0036] Based on the motion directions of the centers of the N first visual tracking boxes and the N first velocity directions, the following matrix can be constructed: ; By performing orthogonal decomposition on the above matrix, the target rotation matrix can be obtained. .

[0037] After obtaining the target rotation matrix, the motion directions of the centers of the M second visual tracking boxes acquired in the second time period are used as a basis. and the M second velocity directions This is used to verify the target rotation matrix.

[0038] Alignment correlation coefficient consecutive M frames higher When the matrix is ​​stable (verification passed) and a preset value (e.g., 0.85) is used, the target rotation matrix is ​​locked and used as the mapping relationship. If the samples in the first and second time periods are insufficient, the previous valid solution is used. The same mapping relationship can also be established or obtained through factory calibration, initial user pairing calibration, etc.

[0039] By employing the method described in this embodiment, the accuracy of the mapping relationship can be improved, thereby increasing the accuracy of the first motion direction trend and thus improving the success rate and efficiency of tracking recovery.

[0040] Optionally, obtaining the second motion direction trend measured by the inertial measurement unit includes: The pose, angular velocity, motion trend, and velocity of the inertial measurement unit are obtained. Based on the pose and the angular velocity, determine the pose integral; Based on the motion trend, the velocity, and the pose integral, a motion trend vector is determined, wherein the second motion direction trend includes the motion trend vector.

[0041] Combination Figure 3 This embodiment will be described.

[0042] Based on the pose and angular velocity of the inertial measurement unit, the pose integral can be determined: , in, For the quaternion determined based on the pose of the inertial measurement unit, ω is the angular velocity.

[0043] Based on the motion trend, the velocity, and the pose integral, determine the motion trend vector: ; in, The motion trend is denoted as and the speed is denoted as . The net acceleration after gravity compensation is determined based on the pose integral. The time step of the IMU acquisition cycle is used to acquire the pose, angular velocity, motion trend, and velocity of the inertial measurement unit. This data is the latest data acquired by the IMU when the gimbal loses visual tracking of the target.

[0044] like Figure 3 As shown, after determining the second motion direction trend through this embodiment, the second motion direction trend is mapped to the first motion direction trend in the coordinate system of the gimbal body.

[0045] Based on the trend of the first direction of motion, the motion trends of yaw and pitch can be determined, that is, the motion trends of the horizontal and vertical rotation angles that control the gimbal rotation.

[0046] , ; The velocity direction is mapped from the coordinate system to the coordinate system of the gimbal body.

[0047] Optionally, the camera controlling the gimbal body to zoom based on the first motion trend confidence includes: Based on the first motion trend confidence level, the camera on the gimbal body is controlled to perform a one-step zoom; or... Based on the first motion trend confidence level, the camera of the gimbal body is controlled to perform a one-step zoom, and if the duration of the one-step zoom performed by the camera of the gimbal body based on the first motion trend confidence level exceeds a preset time threshold, continuous zoom is performed based on the first motion trend confidence level.

[0048] like Figure 2 As shown, in mode A, the gimbal continuously tracks and captures the target. If the tracking confidence level is below a threshold for L consecutive frames, it can be determined that the gimbal has lost visual tracking of the target, and at this time, it enters mode B. In mode B, while moving based on the first motion direction trend, it also performs a one-step zoom based on the first motion trend confidence level. A one-step zoom means zooming once according to a preset step size.

[0049] In Mode B, duration In the following case, mode C is entered. In mode C, while moving based on the first motion direction trend, continuous zoom is also performed based on the confidence level of the first motion trend. The step size of each zoom in continuous zoom is also the preset step size. The difference between mode B and mode C is that mode B performs one-step zoom, while mode C performs continuous zoom.

[0050] In this embodiment, the target operation zooming includes two modes. First, a one-step zoom mode is performed. Compared to direct, large-amplitude or continuous zooming, one-step zooming is a discrete, small-amplitude adjustment. This avoids sudden, drastic image scaling (Zoom Shock) due to confidence fluctuations in the early stages of visual loss, ensuring image stability and a smooth transition. If visual tracking is not restored after a preset time threshold, continuous zooming is switched to. This ensures that the focal length continuously tracks distance changes over long periods as the target moves away or closer, preventing the target from becoming too small or too large again during zooming. Through this embodiment, the zoom action achieves smoothness and adaptability, avoiding drastic image changes and effectively handling continuous changes in target distance, thus improving image quality stability during the visual recovery phase.

[0051] Optionally, before controlling the gimbal body to perform the target operation, the method further includes: The inertial measurement unit (IMU) measures the velocity of the last frame, and a preset maximum waiting time and a preset minimum waiting time are obtained. The velocity of the last frame is the velocity measured by the IMU in the last frame before the visual tracking of the gimbal body loses the target. The preset time threshold is determined based on the last frame velocity, and the preset time threshold is between the preset maximum waiting time and the preset minimum waiting time.

[0052] The preset time threshold can be determined through this embodiment, as follows: ; The speed of the last frame is defined as follows: the faster the speed, the more time thresholds are preset. The smaller the size, the earlier the active recapture will be entered (this can be understood as follows: if the target is tracked faster, the waiting time will be longer, and the target will be very far away or very close to the gimbal body, which will take longer to zoom in, reducing the efficiency of visual tracking recovery. Entering recapture faster can improve this efficiency; if the target is tracked slower, visual tracking recovery can be achieved in mode B, so there is no need to remember to enter recapture (mode C)). , These are all pre-set parameters. For example, =7m, =3m / s, =0.1, =1s, =8s, calculate s.

[0053] By determining the preset time threshold using the method described in this embodiment, and controlling the time to transition from mode B to mode C based on different last frame velocities, the efficiency of visual tracking recovery can be improved.

[0054] Optionally, before controlling the gimbal body to perform the target operation, the method further includes: The confidence level of the second motion trend is determined based on the projection of the first motion direction trend onto the optical axis of the camera on the gimbal body. Obtain the received signal strength index (RSSI) rate of change of the signal received by the gimbal body from the extended component; The confidence level of the second motion trend is adjusted based on the rate of change of RSSI to obtain the confidence level of the first motion trend.

[0055] In this embodiment, the motion trend vector of the IMU (i.e., the second motion direction trend mentioned above) is... After mapping the coordinates of the gimbal body to the coordinate system, the first motion direction trend is obtained. In the unit vector of the camera's optical axis The projection is: ; A value less than 0 indicates that the motion component is moving away from the optical axis / away from the lens, generating a second confidence level for indicating the motion trend away from the lens (which can be called the basic confidence level for the moving trend away from the lens). ; A value greater than 0 indicates that the motion component is oriented towards the optical axis / closer to the lens, generating a second motion trend confidence level (which can be called the proximity trend base confidence level) to indicate the proximity to the lens. ; when If the value is less than the preset confidence threshold (e.g., the target is moving laterally and the RSSI remains unchanged), then only pre-pointing is performed and zooming is not triggered.

[0056] After determining the confidence level of the second trend, adjustments can be made based on the rate of change of the RSSI. For example... Figure 4 As shown, the RSSI change rate is determined by the wireless link measurement module and the change rate calculation module, and the adjustment is based on the RSSI change rate as follows: , , ; in, Indicates the rate of change of RSSI. Indicates comparison and , <0 (indicating distance) and If the RSSI signal is less than 1 (indicating a weakening signal), then the judgment that "the tracked target is moving away from the camera" is more credible. Greater than 1 can improve The value of the value, that is, to increase the confidence level; conversely, if the value is... and The conclusions regarding tracking whether the target moves away from or near the camera are contradictory. Therefore... If the value is less than 1, the confidence level is lowered. This is an anti-interference design; if the RSSI signal fluctuates significantly, the weight w will decrease. This means the system will automatically reduce its reliance on RSSI, preventing the gimbal from zooming erratically due to signal interference. For example: Scenario 1: Good signal environment (high w) <0, meaning the IMU says "moving away", and the RSSI signal is also steadily weakening.

[0057] At this point, w is high, and the system considers the RSSI data to be reliable. Then the function... This will significantly increase confidence; the system is very certain that the target is moving away and will decisively execute Zoom In.

[0058] Scenario 2: Poor signal environment (w is very low, such as severe multipath effect) The IMU says it's "moving away," and although the RSSI signal is generally weak, it keeps fluctuating wildly (for example, because someone walked by and blocked it). At this point, w is low, and the system considers the RSSI data to be "too noisy and unreliable." Therefore, the function... The correction effect of RSSI will be ignored. The system can degenerate into "IMU-only mode," only performing pre-pointing, avoiding easy zooming to prevent being misled by false signal changes, or not performing corrections at all, setting the second motion trend confidence level as the first motion trend confidence level. Specifically, through... and To make a comparison, in In such cases, link correction is suspended.

[0059] Optionally, the camera controlling the gimbal body zooms based on the first motion trend confidence level, including: When the first motion trend confidence level indicates that the tracked target is moving away from the camera, the camera of the gimbal body is controlled to increase the focal length, wherein the step size for increasing the focal length is a first preset step size.

[0060] Optionally, the camera controlling the gimbal body zooms based on the first motion trend confidence level, including: When the first motion trend confidence level indicates that the tracked target is approaching the camera, the camera of the gimbal body is controlled to reduce the focal length, wherein the step size for reducing the focal length is a second preset step size.

[0061] In this embodiment, When zoom in is triggered, When this happens, ZoomOut is triggered. Among other things, The preset confidence threshold is set.

[0062] The working modes of the gimbal body are summarized as follows:

[0063] The following is a summary of mode B:

[0064] The following is a summary of mode C:

[0065] The specific application of the method of this application will be illustrated through the following embodiments: Example 1: Outdoor filming The user tracks a runner (wearing an extension device). The runner moves through bushes; if the confidence level is less than 0.3 for five consecutive frames, mode B is switched, and the coordinate system mapping is locked. The IMU displays a motion of approximately 3 m / s away from the camera, projected along the optical axis. <0, generate RSSI decreases consistently, triggering a Zoom In step; no hits within the ROI. 2.3s ( After entering mode C, continuously zoom in to 2×, detect human body contours in full frames, and return to mode A after passing through the bushes.

[0066] Example 2: Indoor Multipath and Early Recovery The indoor RSSI was fluctuating violently. Exceeding the threshold, The zoom link was paused. The gimbal's pure IMU pre-pointing smoothly followed the target. After about 1.2 seconds, the target turned back to face the camera, and the target was recaptured within the ROI, recovering quickly from B to A.

[0067] Example 3: Target Approach The target runs towards the camera; occlusion causes loss, switching to mode B. IMU motion trend mapping followed by optical axis projection. >0, Exceeding the threshold triggers a Zoom Out. Upon timeout, enters mode C for continuous Zoom Out; once the target is exposed again, reverts to mode A.

[0068] Example 4: Lateral movement (without focusing) The target moves laterally along the right side of the lens; the IMU shows a large motion trend, but the optical axis projection is large. Below the threshold, RSSI remains essentially unchanged. The system only performs IMU pre-pointing and does not trigger zoom; the ROI recovers quickly after the target re-enters the frame.

[0069] The method described in this embodiment can achieve the following beneficial effects: The target IMU still provides short-term motion direction trends after visual failure, overcoming the failure of purely visual historical trajectories.

[0070] Indirectly inferring distance / nearness through optical axis projection, logical closed loop, and preventing false triggering due to indefocusing during lateral movement.

[0071] Pre-pointing is driven solely by velocity direction to avoid dual-integral drift.

[0072] IMU projection master criterion + link confidence correction, automatic degradation in multipath scenarios.

[0073] Dynamically preset time thresholds, early ROI recovery, and coordinate system mapping can be calibrated and automatically solved.

[0074] See Figure 5 , Figure 5 This is a structural diagram of a gimbal vision loss recovery system 500 provided in one embodiment of this application. Figure 5 As shown, the PTZ vision loss recovery system 500 includes: The gimbal vision loss recovery system 500 includes a gimbal body 501, an expansion component 502, and a processor 503. The gimbal body 501 is used to track and capture a target. The expansion component 502 is worn on the target and includes an inertial measurement unit. The processor 503 is configured to execute the above-described gimbal vision loss recovery method.

[0075] Since the processor 503 is configured to execute the above-described gimbal vision loss recovery method, the gimbal vision loss recovery system 500 can achieve all the beneficial effects of the above-described gimbal vision loss recovery method. To avoid repetition, it will not be described again.

[0076] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0077] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0078] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this application without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of this application.

Claims

1. A method for recovering visual loss caused by a gimbal, characterized in that, An application is made in a gimbal vision loss recovery system, the gimbal vision loss recovery system includes a gimbal body and an expansion component, the gimbal body is used to track and capture a target, and the expansion component is used to be worn on the target, the expansion component includes an inertial measurement unit; The method for restoring lost vision from the gimbal includes: In the process of the gimbal body tracking and shooting the tracked target, if the target is lost due to visual tracking, the first motion direction trend of the gimbal body is determined based on the motion trend of the inertial measurement unit. Control the gimbal body to perform the target operation; During the process of the gimbal body performing the target operation, visual detection is performed. If the gimbal body visually locks onto the tracking target, the gimbal body is controlled to track and photograph the tracking target. The target operation includes: Control the gimbal body to move in accordance with the first direction of motion; Furthermore, if the confidence level of the first motion trend is greater than a preset confidence threshold, the camera of the gimbal body is controlled to zoom based on the confidence level of the first motion trend. The confidence level of the first motion trend is determined based on the projection of the first motion direction trend onto the optical axis of the camera of the gimbal body. The confidence level of the first motion trend is used to indicate the degree to which the tracked target is approaching or moving away from the camera.

2. The method according to claim 1, characterized in that, The determination of the first motion direction trend of the gimbal body based on the motion trend of the inertial measurement unit includes: The second motion direction trend measured by the inertial measurement unit is obtained, and the second motion direction trend is the motion direction trend in the first coordinate system corresponding to the measurement by the inertial measurement unit. Based on the mapping relationship between the first coordinate system and the second coordinate system corresponding to the gimbal body, the second motion direction trend is mapped to the second coordinate system to obtain the first motion direction trend.

3. The method according to claim 2, characterized in that, Before mapping the second motion direction trend to the second coordinate system based on the mapping relationship between the first coordinate system and the second coordinate system corresponding to the gimbal body to obtain the first motion direction trend, the method further includes: During the process of the gimbal body tracking and capturing the target, the motion directions of N first visual tracking frames of the gimbal body and the N first velocity directions of the inertial measurement unit are obtained in the first time period. The motion directions of the center of the N first visual tracking frames and the N first velocity directions correspond one-to-one according to the time frame corresponding to the acquisition. Based on the motion directions of the centers of the N first visual tracking boxes and the N first velocity directions, a covariance matrix is ​​constructed; The covariance matrix is ​​orthogonally decomposed to obtain the target rotation matrix; The motion directions of the center of M second visual tracking frames of the gimbal body and the M second velocity directions of the inertial measurement unit are obtained in the second time period. The second time period is the period after the first time period. The motion directions of the center of M second visual tracking frames and the M second velocity directions correspond one-to-one with the time frames corresponding to the acquisition time. Based on the center motion directions of the M second visual tracking boxes and the M second velocity directions, the target rotation matrix is ​​verified. If the verification is successful, the target rotation matrix is ​​determined as the mapping relationship.

4. The method according to claim 2, characterized in that, The step of obtaining the second motion direction trend measured by the inertial measurement unit includes: The pose, angular velocity, motion trend, and velocity of the inertial measurement unit are obtained. Based on the pose and the angular velocity, determine the pose integral; Based on the motion trend, the velocity, and the pose integral, a motion trend vector is determined, wherein the second motion direction trend includes the motion trend vector.

5. The method according to claim 1, characterized in that, The camera controlling the gimbal body to zoom based on the first motion trend confidence includes: Based on the first motion trend confidence level, the camera on the gimbal body is controlled to perform a one-step zoom; or... Based on the first motion trend confidence level, the camera of the gimbal body is controlled to perform a one-step zoom, and if the duration of the one-step zoom performed by the camera of the gimbal body based on the first motion trend confidence level exceeds a preset time threshold, continuous zoom is performed based on the first motion trend confidence level.

6. The method according to claim 5, characterized in that, Before controlling the gimbal body to perform the target operation, the method further includes: The inertial measurement unit (IMU) measures the velocity of the last frame, and a preset maximum waiting time and a preset minimum waiting time are obtained. The velocity of the last frame is the velocity measured by the IMU in the last frame before the visual tracking of the gimbal body loses the target. The preset time threshold is determined based on the last frame velocity, and the preset time threshold is between the preset maximum waiting time and the preset minimum waiting time.

7. The method according to any one of claims 1-6, characterized in that, Before controlling the gimbal body to perform the target operation, the method further includes: The confidence level of the second motion trend is determined based on the projection of the first motion direction trend onto the optical axis of the camera on the gimbal body. Obtain the received signal strength index (RSSI) rate of change of the signal received by the gimbal body from the extended component; The confidence level of the second motion trend is adjusted based on the rate of change of RSSI to obtain the confidence level of the first motion trend.

8. The method according to any one of claims 1-6, characterized in that, The camera controlling the gimbal body zooms based on the first motion trend confidence level, including: When the first motion trend confidence level indicates that the tracked target is moving away from the camera, the camera of the gimbal body is controlled to increase the focal length, wherein the step size for increasing the focal length is a first preset step size.

9. The method according to any one of claims 1-6, characterized in that, The camera controlling the gimbal body zooms based on the first motion trend confidence level, including: When the first motion trend confidence level indicates that the tracked target is approaching the camera, the camera of the gimbal body is controlled to reduce the focal length, wherein the step size for reducing the focal length is a second preset step size.

10. A gimbal vision loss recovery system, characterized in that, The gimbal vision loss recovery system includes a gimbal body, an expansion component, and a processor. The gimbal body is used to track and capture images of a target. The expansion component is worn on the target and includes an inertial measurement unit. The processor is configured to perform the gimbal vision loss recovery method according to any one of claims 1-9.

Images