An XR camera motion control system and method based on multi-objective visual tracking and predictive control

Abstract

The application discloses an XR camera motion control system and method based on multi-target visual tracking and predictive control, and belongs to the technical fields of computer vision, XR virtual filming and camera motion control. The method collects video frames, device states, lens states and rendering time stamps, performs target detection, human pose estimation, multi-target identity maintenance and short-time trajectory prediction, generates a composition cost and an expected camera position according to a principal point deviation, a head space, a line of sight blank, a motion front blank, a target picture proportion and an occlusion rate, and outputs a gimbal, a track, a mechanical arm, a zoom and a focus control amount through a constrained predictive control, while synchronizing real camera parameters to a virtual camera. In a low confidence or abnormal state, the system performs maintenance, deceleration, switching memory points, freezing zoom or manual takeover and archiving. The application can improve the consistency of subject composition, mechanical motion safety and virtual-real lens synchronization in XR filming.

Description

Technical Field

[0001] This invention belongs to the fields of computer vision, XR virtual filming, camera motion control and intelligent broadcasting technology, and specifically relates to an XR camera motion control system and method based on multi-target visual tracking and predictive control. Background Technology

[0002] XR virtual filming requires placing real camera movement, lens parameters, virtual camera pose, and 3D scene rendering on the same timeline. News broadcasts and anchors in virtual studios may move between multiple memory points, while film and television special effects shots may execute complex paths using tracks, digital pan-tilt units, or robotic arms. Camera control must maintain the main composition while also ensuring mechanical safety and the continuity of the live broadcast.

[0003] Existing camera motion control systems can record camera position memory points, control the optical center, focus, and zoom of live broadcast cameras, and interface with rendering systems in real time. Motion control equipment for film special effects is suitable for repeatedly executing preset trajectories, while motion control equipment for live television broadcasts is suitable for rapid camera position recall in news programs. These devices provide the hardware foundation for automated control, but target positions and directorial composition intentions still primarily rely on manual input.

[0004] When relying solely on fixed memory points, the subject will deviate from the preset position if the anchor deviates, multiple people temporarily exchange positions, or props on site obstruct the view. Tracking based solely on the center of the target detection box cannot distinguish the person's facing direction, gaze direction, and movement trend, easily pulling the camera back to the geometric center, resulting in insufficient overhead space, insufficient white space in front of the subject, or camera wobbling back and forth.

[0005] Traditional Kalman filters can estimate smooth trajectories, but they have limitations in predicting sudden turns, multiple people colliding, and scene interactions due to their single-peak prediction. Ordinary target trackers may also experience identity switching after short-term occlusion. If an incorrect identity directly drives the robotic arm or gimbal, the control error will be amplified by the device's movement. Therefore, visual confidence must be included in the decision-making process along with the safety control boundary.

[0006] Camera motion control also involves focusing, zooming, and FOV synchronization of the virtual camera. Controlling only the gimbal without updating the focal length and focus when the target distance changes will cause instability in subject scale and sharpness; when real lens parameters are not synchronized to the rendering end, the perspective of the virtual scene will be inconsistent with the actual foreground. Therefore, a visual closed-loop control scheme is needed for XR shooting, which ensures mutual constraints between target detection, human pose, identity preservation, motion prediction, composition evaluation, mechanical control, lens control, and virtual camera synchronization, and provides traceable conservative actions when visual results are unreliable.

[0007] In this invention, XR stands for Extended Reality; RT-DETR stands for Real-Time Detection Transformer; RTMPose stands for Real-Time Multi-Person Pose Estimation; MPC stands for Model Predictive Control; PTZ stands for Pan-Tilt-Zoom; FOV stands for Field of View; GPU stands for Graphics Processing Unit; LED stands for Light Emitting Diode; IoU stands for Intersection over Union; GIoU stands for Generalized Intersection over Union; DFL stands for Distribution Focal Loss; and KL stands for Kullback-Leibler. Summary of the Invention

[0008] The purpose of this invention is to provide an XR camera motion control system and method based on multi-target visual tracking and predictive control, which converts the identity, posture and motion trend of the subject into executable gimbal, track, robotic arm, focus and zoom control quantities, thereby reducing subject out-of-frame, sudden mechanical motion changes and asynchronous virtual and real lens parameters.

[0009] To achieve the above objectives, the present invention adopts the following technical solution: The system first acquires camera video, gimbal or robotic arm encoder data, track position data, lens focal length and focus readings, memory points, and rendering timestamps. All inputs are aligned with frame numbers and device clocks; data with time differences exceeding a threshold are not directly used for closed-loop control, but instead enter buffer alignment or low-confidence processing.

[0010] The visual target perception unit uses a target detection network to output the category, bounding box, and confidence score of people, vehicles, products, and props. For human targets, the system further runs a pose estimation network within the detection area to determine the person's orientation, standing position, and range of motion based on key points such as the head center, shoulder line, hip, and limbs. As a preferred implementation, RT-DETR is used for target detection and RTMPose is used for human key points. However, this invention is not limited to a specific open-source network; equivalent models with the same detection, pose estimation, and confidence score output capabilities can be implemented.

[0011] The multi-target identity preservation unit preferably uses ByteTrack. The system first associates high-confidence detection boxes, and then uses low-confidence detection boxes that still have motion continuity to fill in the short-term occluded targets. In addition to the intersection-union ratio and motion distance, the association cost also includes human keypoint skeleton similarity, clothing appearance vector and director-specified target identifiers to reduce identity switching when multiple people meet.

[0012] The target short-time motion prediction unit does not simply extrapolate the velocity of the previous frame, but instead reads a fixed length of historical trajectory, attitude changes, target category, stage boundaries, and relationships between adjacent targets. The optimized Trajectron++ outputs the future trajectory distribution with a conditional variational structure, which can retain multiple possible branches for turns, pauses, and multi-person interactions. The system generates candidate camera positions using the prediction mean and expands the safety margin using the prediction variance.

[0013] The composition quality evaluation unit translates the director's understandable rules of composition into numerical values. For news shots of people, the evaluation focuses on the position of the eyes or head, overhead space, negative space in the direction of the gaze, and the scale of the upper body; for vehicle or product shots, the evaluation focuses on the proportion of the target, the orientation of the product, the continuity of the surrounding angle, and background occlusion; for shots of multiple people, the evaluation also includes the spacing between people and the overall frame.

[0014] The desired camera position generation unit calculates the camera state based on the future position of the target and the composition target. The gimbal system outputs the desired yaw and pitch angles, the track system outputs the desired track position, the robotic arm system outputs the end effector pose, and the lens control system outputs the desired focal length and focus distance. All of the above desired values ​​form a continuous transition with the current equipment state, and abrupt changes without transition are not allowed.

[0015] The constrained predictive control unit evaluates multiple candidate control sequences within a finite prediction time domain. The objective function includes optical center and mapping errors, target scale errors, device motion energy, velocity changes, acceleration changes, jerk, and virtual camera synchronization errors; the constraints are derived from gimbal angle, robotic arm joints, track endpoints, device maximum speed, and on-site safe distance.

[0016] To prevent learning model errors from directly driving the device, the system sets a safety shield after the MPC output. The safety shield checks visual confidence, identity stability, prediction variance, control boundaries, and collision distance; if the conditions are not met, the control quantity is projected to the allowed set, or the previous frame is held, the speed is reduced, the memory point is returned, the zoom is frozen, and manual intervention is prompted.

[0017] The lens parameter control unit maps the focal length ring or lens encoder readings to the actual focal length and FOV, and calculates the focus amount based on the target distance. Zoom control not only follows the target's proportion in the frame, but is also constrained by zoom speed and image breathing effect, avoiding frequent focus adjustments when the subject moves slightly.

[0018] The virtual camera synchronization unit writes the real camera's extrinsic parameters, focal length, field of view, focus distance, and frame timestamps to the rendering end. Camera motion control and virtual background updates use the same control cycle; if the rendering end confirms that the delay exceeds a threshold, the system can delay the real output, freeze virtual parameters, or reduce the camera speed to maintain consistency between the virtual and real worlds.

[0019] After adopting the above scheme, the live broadcast equipment for television stations can retain the ability to recall memory points while making small-scale continuous corrections based on the actual position of the anchor; the special effects equipment for movies can use the visual target state as feedback for trajectory execution, verifying whether the target is still within the predetermined composition area in repeated shots and surround shots. This invention does not limit the camera motion mechanism model, nor does it limit the independent claims to a specific open-source network. Equivalent modules with target detection, attitude estimation, identity association, probabilistic trajectory prediction, and constrained control functions can all be implemented. Attached Figure Description

[0020] Figure 1 This is a schematic diagram of the overall structure of the XR camera motion control system of the present invention.

[0021] Figure 2 This is the overall flowchart of the XR camera motion control method of the present invention.

[0022] Figure 3 This is a flowchart of the target detection, pose estimation, and screen false target suppression process of the present invention.

[0023] Figure 4 This is a flowchart of the multi-target identity preservation and short-term trajectory prediction of the present invention.

[0024] Figure 5 This is a flowchart of the composition quality evaluation and expected camera position generation process of the present invention.

[0025] Figure 6 This is the closed-loop diagram of the constrained control and safety shielding of the MPC in this invention.

[0026] Figure 7 This is a flowchart of the lens parameter control and virtual camera synchronization process of the present invention.

[0027] Figure 8 This is a flowchart of the low-confidence rollback, manual takeover, and archiving review process of this invention.

[0028] Figure 9 This is a schematic diagram illustrating the product launch or XR filming application scenarios of this invention. Detailed Implementation

[0029] The following is combined Figures 1 to 9 The present invention will be further described below. Figure 1This represents the complete link from video and device status input to mechanical control, lens control, and virtual camera synchronization; Figure 2 The overall steps of the representation method; Figures 3 to 8 The following aspects are further refined: target perception, identity association and trajectory prediction, composition evaluation, MPC security control, lens parameter synchronization, and low confidence backoff. Figure 9 This indicates the equipment connection relationships at a product launch or XR filming location.

[0030] Before system deployment, device description files are created for cameras, pan-tilt units, tracks, or robotic arms. These files record the device coordinate system, zero position, travel distance, speed, acceleration, jerk, communication cycle, emergency stop status, lens focal length mapping, and focus mapping. For existing Othka-MocoTV or Othka-MocoBS devices, memory points and encoder data can be read via the control interface without altering the existing underlying servo closed loop.

[0031] The video acquisition and control status use a unified timestamp. The arrival time of the camera image, the sampling time of the device status, and the rendering confirmation time are stored separately. The system aligns the device status to the image frame through nearest neighbor or linear interpolation. If the image delay suddenly increases, the current frame is only used for display and archiving, and is not used to calculate new mechanical control quantities.

[0032] Training samples are organized into five categories: news articles about people, interviews with multiple people, vehicle demonstrations, product surround shots, and occlusion recovery. Each sample contains at least the following labels: target category, target identity, bounding box, human keypoints, visibility, director-selected composition point, device status, and whether automatic control is allowed. Data from different TV stations or film sets are isolated by project to prevent test clips from entering the training set.

[0033] The training loss of an object detection network can be expressed as follows:

[0034] (1)

[0035] In the formula, This represents the total loss in target detection. Indicates category loss. This represents the generalized intersection-regression loss. Indicates the distribution focus loss. , and The corresponding weights are used. This formula is used to train the network for detecting people, vehicles, and products. News anchors, vehicles, and small products have significant scale differences; therefore, the training batches simultaneously retain close-up, panoramic, and fast zoom-in / zoom-out clips. Images of people and advertisements on the LED screen are marked as non-control targets to prevent the system from identifying the screen content as the main subject of the scene.

[0036] RTMPose operates only within the person detection area. The system uses keypoints at the tip of the nose, both eyes, both ears, and shoulders to estimate the head center and shoulder line direction, and uses hip and ankle keypoints to determine whether the person is starting or stopping. When the speaker is obscured by the podium, the head and shoulder keypoints can still maintain compositional reference; full limb visibility is not required. The keypoint loss can be expressed as:

[0037] (2)

[0038] In the formula, Indicates key point losses. For the number of effective key points, For the first Visibility weight of key points To label the coordinates, For predicting coordinates, the square of the norm of subscript 2 represents the coordinate error.

[0039] Combination Figure 3 The system first uses RT-DETR to detect people, vehicles, and product targets in the live video. Then, RTMPose extracts key points for the head, shoulders, torso, and limbs within the detected area. At each decision point, the system compares the detection confidence, key point visibility, and screen area suppression results. If the threshold is met, it outputs the detection box, pose reference point, and confidence score. If the threshold is not met, it retains the previously confirmed target, requests subsequent frames, and generates a low-confidence record to avoid directly altering the controlled object due to missed detections in a single frame.

[0040] ByteTrack's trajectory state includes the target bounding box center, width and height, velocity, category, appearance vector, and keypoint summary. After the director selects the primary target via the interface, the primary target's identifier is written into the trajectory state; other characters, even if they have higher confidence levels in the short term, will not seize camera control if the switching conditions are not met. The identity association cost can be expressed as:

[0041] (3)

[0042] In the formula, For trajectory With detection The associated costs, For the intersection, union, and comparison of the two, , and These are respectively the visual distance, the skeletal distance, and the movement distance. , , and The weights are used to indicate the identity association; the lower the cost, the more reliable the identity association. The trajectory state can be written as:

[0043] (4)

[0044] In the formula, For a moment The trajectory state, and Centered on the target box and Width and height, and Represents the velocity of the image plane. For gesture and identity summary vectors.

[0045] Combination Figure 4 ByteTrack receives the current detection results and historical trajectories, and sequentially completes the association of high-confidence detection boxes, the supplementary association of low-confidence detection boxes, and the update of the target's lifecycle. When the identity is stable, Trajectron++ predicts the future trajectory distribution based on the target's historical position, velocity, pose, and stage semantics, and outputs the identity trajectory and prediction covariance; when the identity is unstable, the system calls appearance and skeleton features for verification and sends the corresponding segments to the identity verification queue.

[0046] Trajectron++ takes as input several consecutive frames the target center, velocity, pose state, and scene semantics. Stage edges, podiums, tables and chairs, track boundaries, etc., are encoded as unreachable or low-probability regions; multi-person interactions are represented by a graph structure, and the model outputs a multimodal probability distribution of future trajectories, rather than a single path.

[0047] (5)

[0048] In the formula, For historical target trajectory, For scene semantics and neighboring target relationships, For the future trajectory, As a latent variable, Let represent the conditional probability distribution. The trajectory prediction loss can be expressed as:

[0049] (6)

[0050] In the formula, For trajectory prediction loss, For expectation operator, and These are the actual and predicted trajectories, respectively. For Kullback-Leibler divergence, For the posterior distribution, As a prior distribution, This represents the divergence weight.

[0051] The reason for choosing Trajectron++ is that the live target may suddenly stop, turn around, or move towards another memory point, and a single velocity extrapolation cannot describe these branches. Instead of following every branch with a low probability, the controller calculates the expected camera position using the highest probability trajectory and uses the distribution covariance to increase the safe distance or reduce the movement speed.

[0052] Composition templates are chosen by the photographer before the assignment begins. A single-person close-up template places the center of the eyes near the upper third line; a half-body news template simultaneously constrains the overhead space, shoulder width, and the edge of the podium; a multi-person template frames the target group within a safe frame; a product surround template constrains the product's center, orientation, and background white space. The cost of composition can be expressed as:

[0053] (7)

[0054] In the formula, For the total cost of composition, , , , and These represent the optical center, overhead space, blank space in front of the moving object, target size, and occlusion error, respectively. to The scene template weights are used. The optical center deviation can be further expressed as:

[0055] (8)

[0056] In the formula, For optical center deviation, As the target reference point, As the target point of the composition template, This is a weight matrix in the horizontal and vertical directions, with... The norm of the subscript represents the weighted distance; different lenses can set different weights for horizontal and vertical errors.

[0057] Combination Figure 5 The composition evaluation unit reads the target's identity, future position, and the composition template selected by the cameraman, and calculates the optical center deviation, subject scale, overhead space, line-of-sight margin, and forward margin. When the composition quality meets the requirements, the system inversely solves the desired camera position, focal length, and focus state and outputs the desired equipment state; when it does not meet the requirements, it adjusts the composition template or safety boundary, generates a new composition target, and then re-evaluates.

[0058] The desired camera position calculation first transforms the target's three-dimensional position or the relative position estimated from the image scale into the device coordinate system, and then solves for the desired optical axis based on the composition target. For a digital pan-tilt head, the control variables are yaw, pitch, and zoom; for a track camera, the track position is added; for a robotic arm, the end effector position and attitude are added, but ultimately they are still converted into joint commands by the device's own inverse kinematics.

[0059] MPC operates in a rolling time-domain manner. Each control cycle solves for the control sequence of several future steps based on the current target state and predicted trajectory. Only the first step is issued, and the new visual results and device encoder are read in the next frame to resolve the sequence. This method can absorb model prediction errors and facilitates explicit constraints for device limits and safety distances. When the actual zoom state is synchronized to the rendering end, the field of view can be calculated using the following formula:

[0060] (9)

[0061] In the formula, For the virtual camera's field of view, The effective width of the sensor. This is the actual focal length.

[0062] The equipment model is identified using low-speed teaching data. Within the safe zone, the system applies small yaw, pitch, orbit, and zoom commands, recording encoder response, delay, and hysteresis to form a discrete state matrix.

[0063] (10)

[0064] In the formula, For the first The device status of each control step. To control the input, and This is the discrete state matrix identified by the gimbal, track, or robotic arm. The MPC objective function can be expressed as:

[0065] (11)

[0066] In the formula, Let MPC be the objective function, and the summation symbol denotes summation over the prediction time domain. To account for composition and camera position errors, To control the quantity, To control the increment, , and The error, energy, and smoothness weight matrix is ​​shown in the superscript. This indicates transpose.

[0067] The control candidate quantity should satisfy the following boundary:

[0068] (12)

[0069] In the formula, and These are the lower and upper limits of the control quantity, respectively. and These represent the upper and lower limits of the equipment's operating state, respectively; these boundaries are jointly determined by the gimbal angle, track travel, robotic arm joints, and lens rate of change. The jerk can be expressed as:

[0070] (13)

[0071] In the formula, For the first The accelerometer of each control step, and For adjacent control step acceleration, To control the cycle; for The limiting mechanism is used to reduce the impact on the camera lens of the track and robotic arm.

[0072] Combination Figure 6 The MPC safety control unit performs rolling time-domain optimization based on the desired machine position and the encoder status of the equipment, and verifies the gimbal or robotic arm limits, collision distance, visual confidence, and control feasibility at the judgment node. If the verification passes, only the first control quantity of the current cycle is issued, and then the encoder acknowledgment is read and the control status is synchronized; if the verification fails, the amplitude is limited, the memory point is returned, or a manual takeover prompt is given, while the control log and the reason for the rollback are recorded.

[0073] The safety shield is located between the learning model and the device driver. It reads emergency stops, joint limits, track endpoints, collision zones, cable areas, and manually restricted areas; any candidate control variable that violates even one hard constraint will not be issued to the device. Insufficient visual confidence is considered a soft risk; the system prioritizes deceleration, holding, or returning to the memory point, while retaining manual confirmation of entry. The safety confidence score can be expressed as:

[0074] (14)

[0075] In the formula, For safety confidence scoring, , , and These are detection, identity association, trajectory prediction, and device status confidence. , , and Let the weights be denoted as . The actual control quantity can be expressed as:

[0076] (15)

[0077] In the formula, For MPC candidate control variables, To meet the safe and feasible range for mechanical limits and collision distances, This represents the projection operation. This represents the actual amount of control measures issued.

[0078] Combination Figure 7 Zoom control is generated by looking up a table based on the target's proportion in the frame and the actual focal length. Focus control is generated based on distance measurement, the target's relative distance, or the lens calibration curve; if distance measurement is invalid, the current focus is maintained and the cameraman is prompted for confirmation, avoiding significant changes in focus position based solely on the detection frame size. The focus curve is calibrated using multiple known distances, and the zoom curve is calibrated using focal length ring readings and the actual field of view (FOV). The system saves the zoom up and zoom down direction curves separately to compensate for mechanical hysteresis; if lens temperature or load changes cause increased errors, the cameraman can recalibrate the lens mapping without retraining the visual model.

[0079] The real and virtual cameras share frame numbers, poses, and camera states. After the rendering end confirms successful writing, the archiving unit saves the detection target, trajectory number, composition cost, MPC candidate actions, safety masking results, and final control variables for that frame; when a misalignment occurs, it can distinguish between anomalies caused by visual delay, device response, lens mapping, or rendering confirmation. The synchronous mapping from real device parameters to rendering engine camera parameters can be represented as:

[0080] (16)

[0081] In the formula, Indicates time, and This refers to the switching between virtual and real cameras. Focal length Focusing distance This is a synchronous mapping from real device parameters to camera parameters in the rendering engine.

[0082] Combination Figure 8 The system supports three modes: automatic, assisted, and manual. Automatic mode allows the device to be driven by visual results; assisted mode only provides composition suggestions and target memory points, without issuing motion commands; manual mode is operated by the cameraman, while the system continues to record the target trajectory and device status, providing samples for subsequent training or review. Control inputs are transitioned via a ramp function when switching modes.

[0083] Emergency stop and manual joystick input have the highest priority. After an emergency stop is triggered, the algorithm can only record the state and cannot automatically resume movement; if the manual joystick input exceeds the dead zone, the system smoothly exits automatic mode and displays visual suggestions to the cameraman. Re-entering automatic mode requires the device to be stationary, the target's identity to be confirmed, and safety conditions to be re-verified.

[0084] Data is archived by shot segments rather than indefinitely. Each shot segment records the start and end frames, target identity, model version, composition template, equipment model, memory points, control status, and anomaly index; normal frames can be sampled and saved, while frames near identity switching, low confidence, security masking, and manual takeover are saved in their entirety.

[0085] Combination Figure 9 In product launches or XR filming, a mechanical motion mechanism can be equipped with a real camera to perform camera movements in the presence of vehicles, products, lighting, display backgrounds, and on-site personnel. The mechanical mechanism outputs the joint or track status, the vision unit obtains the product position, orientation, and screen proportion from the monitoring screen, the control unit generates correction values ​​under the constraints of equipment limits and personnel safety zones, and the rendering end updates the virtual background based on the synchronized camera parameters, thus forming an engineering link for on-site shooting, motion control, and virtual scene synchronization.

[0086] For mobile phone products, RT-DETR detects the main body of the product and the display stand, with local appearance features used to maintain product identity; for vehicles, the system simultaneously detects the vehicle's body frame and frontal orientation. Short-term target prediction primarily compensates for relative positional changes caused by the movement of the turntable, vehicle, or robotic arm, and does not consider background advertising images as control targets.

[0087] Soft and hard restricted areas can be set up on site. Hard restricted areas are for mechanical collision and personnel safety, and automatic entry is prohibited in any mode. Soft restricted areas are for director's shot requirements or cable slack; automatic mode defaults to avoid these areas, but cameramen can confirm entry in manual mode. Restricted area versions and site layout. Figure 1 Same as archive.

[0088] Model updates are not directly overwritten during live streaming. The new model is first played back in the recorded video and on the device simulator, and then compared with target identity, composition score, and control suggestions in an auxiliary mode; it is only activated in the next task after the cameraman confirms it. Each update records a model summary, training data range, threshold, and rollback version.

[0089] After the task is completed, a debriefing summary is automatically generated, including the duration of the main target, the number of identity switching, the low confidence interval, the reason for security masking, memory point recall, manual takeover, and the synchronization status of the virtual camera. The summary only summarizes observable records and does not directly express the composition score as an artistic quality conclusion. The technical effect of this invention is evaluated through a traceable closed-loop system, without pre-setting a uniform recognition accuracy rate or composition score; during implementation, it must at least be proven that the visual target can form a continuous identity, the control candidate is mechanically constrained, the security branch can block non-executable actions, the lens parameters can be synchronized with the virtual camera, and the process nodes shown in the attached figures can be verified in the records.

Claims

1. A motion control method for XR cameras based on multi-target visual tracking and predictive control, characterized in that, Includes the following steps: S1. Acquire video frames, target selection information, camera external parameters, pan-tilt status, track status, robotic arm status, lens focal length reading, focus reading, preset memory points, and rendering end timestamps captured by the on-site camera, and time-align the video frames, motion mechanism status, lens status, and rendering end timestamps according to the video frame number and device clock. S2. Perform target detection on the video frame to obtain the category, bounding box, and detection confidence of the target (person, vehicle, product, or prop); when the detected target is a person, extract key points of the head, shoulders, torso, and limbs in the corresponding detection area, and determine the target reference point, target orientation, action state, and posture confidence based on the visibility of the key points, head center, shoulder line direction, and limb changes. S3. Based on the current detection results, historical trajectories, target selection information, crossover and union ratio of detection boxes, appearance features, key point structural similarity, and target motion direction, multiple candidate targets are associated with identities to form a continuous identity trajectory that includes the target box center, target scale, velocity, category, pose summary, identity identifier, and lifecycle state. S4. Based on the continuous identity trajectory, target historical position, target speed, posture change, scene boundary, stage or shooting area semantics and adjacent target relationship, predict the target's position distribution at multiple future time nodes, and output the future trajectory mean, prediction covariance and trajectory prediction confidence. S5. Based on the composition template selected by the cameraman or director, evaluate the composition quality of the target reference point and future position distribution, calculate the composition cost including at least optical center deviation, overhead space, white space in the direction of sight, white space in front of movement, target screen ratio and occlusion rate, and generate the desired camera state based on the composition cost, current camera extrinsic parameters, target predicted position, focal length and target screen ratio. S6. Based on the desired camera state and the current device state, construct a rolling time-domain predictive control optimization problem. Under the constraints of camera motion range, gimbal angle, track boundary, robotic arm joint limit, speed, acceleration, jerk, collision distance, cable safety area, and lens rate of change, solve the candidate control sequence and select the first control quantity in the candidate control sequence as the candidate control quantity. S7. Before issuing the candidate control quantity, perform a safety shield. Determine whether the candidate control quantity meets the safety execution conditions based on the detection confidence, identity association confidence, trajectory prediction confidence, prediction covariance, device feedback status, mechanical limit, and collision distance. If it meets the conditions, issue the camera motion control quantity, zoom control quantity, and focus control quantity. If it does not meet the conditions, perform at least one of the following processing on the candidate control quantity: projection, limiting, replacement, maintaining the previous control quantity, reducing motion speed, switching preset memory points, freezing zoom, or switching to manual control. S8. Synchronously write the external parameters, focal length, field of view, focusing distance, final control quantity and frame timestamp of the real camera into the virtual camera, and archive the detected target, target identity, predicted trajectory, composition cost, candidate control sequence, security masking result, final control quantity and abnormal triggering reason.

2. The XR camera motion control method according to claim 1, characterized in that, In step S2, target detection uses a real-time target detection network, preferably RT-DETR, to identify people, vehicles, products and props; key point extraction of people uses a real-time multi-person pose estimation network, preferably RTMPose, and uses the head center, shoulder line direction, hip position and key points of the limbs as the basis for judging the person's orientation, standing state, starting state or stopping state.

3. The XR camera motion control method according to claim 1, characterized in that, In step S3, the multi-target identity association adopts a phased association method. First, the high-confidence detection box is associated with the confirmed trajectory, and then the low-confidence detection box with motion continuity is associated with the unmatched trajectory. The identity association cost includes at least the detection box intersection and union cost, appearance distance, key point skeleton distance, and motion distance, and the weights of each cost are adjusted according to different target types.

4. The XR camera motion control method according to claim 1, characterized in that, In step S4, the future location distribution of the target is generated by a conditional probability trajectory prediction network, preferably Trajectron++; the conditional probability trajectory prediction network uses the target's historical trajectory as a basis. Scene semantics and neighboring target relationships Given the condition, generate the future trajectory. conditional probability distribution The predicted covariance will be used to expand the safety margin or reduce the control speed.

5. The XR camera motion control method according to claim 1, characterized in that, In step S5, the composition cost Calculated based on the following relationship: ； in, This indicates the optical center deviation between the target reference point and the target point of the composition template. Indicates overhead spatial error. This indicates the error in the amount of white space left in front of the line of sight or movement. This indicates the error in the proportion of the target image. Indicates occlusion error. , , , , Weights are assigned to the lens type or composition template; when the composition cost exceeds a preset threshold, the desired camera state is regenerated after adjusting the desired frame size, target scale, safety boundary, or candidate memory points.

6. The XR camera motion control method according to claim 1, characterized in that, In step S5, the desired camera state is calculated based on the target's future position, target reference point, current camera extrinsic parameters, lens focal length, target image occupancy, and composition template. For a digital pan-tilt unit, the desired camera state includes the desired yaw angle, desired pitch angle, and desired zoom state. For a track camera, the desired camera state also includes the desired track position. For a robotic arm camera, the desired camera state also includes the robotic arm end-effector pose, which is converted into joint control commands by the device inverse kinematics.

7. The XR camera motion control method according to claim 1, characterized in that, In step S6, the state update relationship of the rolling time-domain predictive control optimization problem is as follows: ； The predictive control objective function is: ； in, For the first The device status of each control step. For the first The control input for each control step and This refers to the discrete state matrix identified by the gimbal, track, or robotic arm. This could be due to compositional error or camera position error. To control the increment, , and These are the error, energy, and smoothness weight matrices, respectively; the rolling time-domain predictive control optimization problem also imposes constraints on velocity, acceleration, and jerk, where the jerk... according to calculate, and For adjacent control step acceleration, To control the cycle.

8. The XR camera motion control method according to claim 1, characterized in that, In step S7, the security shield is based on the security confidence score. The security confidence score determines whether a candidate control variable is executable. Calculated based on the following relationship: ； in, Indicates the confidence level of the detection. Indicates the confidence level of identity association. This represents the confidence level of trajectory prediction. Indicates the confidence level of the device status. , , and For the corresponding weights; when When the value is below the safety threshold or the candidate control quantity violates mechanical limits, collision distances, track boundaries, joint limits, or camera change rate constraints, the actual executed control quantity is obtained according to the following relationship: ； in, To predict the candidate control quantity for the control output. To meet the safe and feasible range for mechanical limits and collision distances, This represents the projection operation onto the safe and feasible region. This represents the actual amount of control measures issued.

9. The XR camera motion control method according to claim 1, characterized in that, In step S8, the lens focal length is obtained by mapping the readings from the focal length ring or lens encoder, and the field of view is... Based on the effective width of the sensor and true focal length The focus distance is calculated based on the ranging value, the relative distance to the target, or the lens calibration curve. The synchronous mapping from real camera parameters to virtual camera parameters includes at least real camera transformation, virtual camera transformation, focal length, focus distance, and frame timestamp, so that the real camera movement, lens parameter changes, and virtual scene rendering remain consistent in time.

10. A motion control system for an XR camera based on multi-target visual tracking and predictive control, characterized in that, include: The video and device status acquisition module is used to acquire video frames, target selection information, camera external parameters, gimbal status, track status, robotic arm status, lens focal length reading, focus reading, preset memory points and rendering end timestamps, and perform time alignment. The visual target perception module is used to detect people, vehicles, products or props, extract key points of the human body in the human detection area, and output the target reference point, target orientation, action state and detection confidence. The multi-target identity preservation module is used to maintain the continuous identity trajectory of multiple targets based on the cross-union ratio of detection boxes, appearance features, key point structural similarity, target motion direction and target selection information. The target short-term motion prediction module is used to generate future position distribution, prediction covariance and trajectory prediction confidence based on historical trajectory, attitude change, scene boundary and adjacent target relationship; The composition quality evaluation and desired camera position generation module is used to calculate the composition cost based on the composition template, optical center deviation, overhead space, blank space in the direction of the line of sight, blank space in front of the movement, target screen ratio and occlusion rate, and generate the desired camera state. The constrained predictive control module is used to solve for candidate control sequences under constraints such as camera motion range, gimbal angle, track boundary, robotic arm joint limit, speed, acceleration, jerk, collision distance, and lens rate of change. The lens parameter control module is used to generate zoom control values, focus control values, true focal length, field of view, and focus distance. The virtual camera synchronization module is used to synchronously write the external parameters, focal length, field of view, focus distance, and frame timestamps of the real camera into the virtual camera. The low-confidence re-inspection and archiving module is used to perform at least one of the following processes when the detection confidence, identity association confidence, trajectory prediction confidence, mapping quality, or equipment status does not meet the threshold conditions: maintain the previous control value, reduce the motion speed, switch the preset memory point, freeze the zoom, or switch to manual control, and save the trigger reason and control record.

Images