Motion control method, device, equipment and medium of camera

Abstract

A motion control method, device, equipment and medium of a camera are disclosed. The method comprises: acquiring perception information of a target camera, and generating corresponding scene semantic information according to the perception information; determining a corresponding target camera operation strategy according to shooting subject information and picture space layout indicated by the scene semantic information; predicting a camera control sequence in a future time sequence based on the target camera operation strategy, to drive the target camera to rotate according to the camera control sequence. The application determines scene semantic information by understanding the perception information, determines a matching target camera operation strategy according to the scene semantic information, and then converts the target camera operation strategy into a camera control instruction, to realize automatic camera operation with scene semantic understanding capability and humanized smoothness, avoid mechanical tracking relying on target coordinates only, and realize intelligent camera operation with director thinking.

Description

Technical Field

[0001] This invention relates to the field of camera control technology, and in particular to a method, apparatus, device, and medium for controlling the motion of a camera. Background Technology

[0002] In existing technologies, mainstream handheld gimbals and automatic tracking devices generally employ closed-loop control schemes based on traditional computer vision. A typical process involves: using Kernel Correlation Filtering (KCF) or a detection-tracking algorithm to obtain the rectangular region of interest (ROI) of the target; calculating the pixel deviation between the center of this region and the center of the frame; and inputting this deviation into a PID controller to drive the gimbal motor, keeping the target centered in the frame. While this method can achieve basic target tracking, it has significant limitations in professional film and television production scenarios.

[0003] First, existing camera movement logic is simplistic, forcibly locking the subject to the absolute center of the frame and failing to dynamically adjust compositional offset based on subject-related information. Furthermore, it lacks awareness of framing intentions, hindering the creation of emotionally expressive camera movements and resulting in stiff, uninspired images with weak narrative. Second, its control mechanism relies on real-time error feedback, lacking foresight and smooth motion. For example, the PID controller passively responds to changes in the target's state, easily producing overshoot, lag, or jerking when the target stops abruptly, turns back, or changes speed. Moreover, the acceleration and deceleration processes are linearly abrupt, lacking the breathiness and S-shaped speed curve of a human photographer's shots, resulting in noticeable mechanical and electronic shakiness in the camera movement, failing to meet the demands of high-quality image creation for natural, smooth, and intentional camera movement. Summary of the Invention

[0004] This invention provides a motion control method, device, equipment, and medium for a camera to solve the problem of rigid camera movements and a strong mechanical feel caused by the lack of scene semantic understanding capabilities in existing automatic tracking cameras.

[0005] According to one aspect of the present invention, a motion control method for a camera is provided, comprising: Acquire the perception information of the target camera and generate corresponding scene semantic information based on the perception information; Based on the subject information and spatial layout of the scene indicated by the semantic information, determine the corresponding target camera movement strategy; Based on the target camera movement strategy, the camera control sequence in the future time series is predicted, so as to drive the target camera to rotate according to the camera control sequence.

[0006] According to another aspect of the present invention, a motion control device for a camera is provided, comprising: The scene understanding module is used to acquire the perception information of the target camera and generate corresponding scene semantic information based on the perception information; The camera movement strategy determination module is used to determine the corresponding target camera movement strategy based on the shooting subject information and the spatial layout of the frame indicated by the scene semantic information. The control sequence prediction module is used to predict the camera control sequence in the future time series based on the target camera movement strategy, so as to drive the target camera to rotate according to the camera control sequence.

[0007] According to another aspect of the present invention, an electronic device is provided, the electronic device comprising: At least one processor; and A memory that is communicatively connected to at least one processor; wherein, The memory stores a computer program that can be executed by at least one processor, such that the at least one processor is able to perform the motion control method of the camera according to any embodiment of the present invention.

[0008] According to another aspect of the present invention, a computer-readable storage medium is provided, which stores computer instructions for causing a processor to execute and implement a motion control method for a camera according to any embodiment of the present invention.

[0009] According to another aspect of this application, a computer program product is provided, which includes a computer program that, when executed by a processor, implements the motion control method of a camera according to any embodiment of this application.

[0010] The technical solution of this invention determines scene semantic information by understanding perceived information, determines a matching target camera movement strategy based on the scene semantic information, and then converts the target camera movement strategy into camera control commands. This achieves automatic camera movement with scene semantic understanding capabilities and anthropomorphic smoothness, avoiding mechanical tracking that relies solely on target coordinates. It realizes intelligent camera movement with directorial thinking, significantly improving the viewing experience, narrative quality, and professionalism of video content.

[0011] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of the present invention, nor is it intended to limit the scope of the invention. Other features of the invention will become readily apparent from the following description. Attached Figure Description

[0012] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0013] Figure 1This is a flowchart of a motion control method for a camera according to an embodiment of the present invention; Figure 2 This is a flowchart of another motion control method for a camera provided according to an embodiment of the present invention; Figure 3 This is a flowchart of another motion control method for a camera provided according to an embodiment of the present invention; Figure 4 This is a schematic diagram of the structure of a motion control device for a camera according to an embodiment of the present invention; Figure 5 This is a schematic diagram of the structure of an electronic device that implements the motion control method for a camera according to an embodiment of the present invention. Detailed Implementation

[0014] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0015] It should be noted that the terms "candidate," "target," etc., used in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0016] Figure 1 This invention provides a flowchart of a camera motion control method, applicable to the implementation of intelligent camera pan-tilt-zoom automatic camera movement. The method can be executed by a camera motion control device, which can be implemented in hardware and / or software. This motion control device can be configured within the camera or communicate with a server capable of controlling the camera. Figure 1 As shown, the method includes: S110. Acquire the perception information of the target camera and generate corresponding scene semantic information based on the perception information.

[0017] The target camera refers to the camera device that needs to be controlled or scheduled. Its motion control is actively adjusted based on scene semantic information and camera movement strategies to achieve a specific shooting intention. Examples of target cameras include handheld gimbals, robotic arms, or drone pods. The target camera's perception information refers to the raw or pre-processed data collected by the target camera itself or its cooperating sensors during operation. This perception information reflects the current shooting environment status of the target camera. For example, perception information includes the current video stream data collected by the target camera; further, perception information includes the current video stream data and historical continuous frame sequences.

[0018] Based on the recognition of perceived information, scene semantic information corresponding to the current shooting environment is generated. Scene semantic information refers to a structured description with clear semantic meaning obtained after high-level abstraction of the original perceived data, used to represent information about the scene content in the current scene and the relationships between different scene contents. For example, a deep learning-based visual-language large model can be used to process the perceived information to generate corresponding scene semantic information, or an object detection and tracking network can be used to process the perceived information to generate corresponding scene semantic information. Scene semantic information includes at least the shooting subject information and the spatial layout of the scene. The shooting subject information refers to the category, identity, location, motion state, and behavioral state of key objects in the scene. The spatial layout of the scene refers to information such as the relative position information between the subject and the background, between subjects, available motion space, occlusion relationships, and scene type.

[0019] In one feasible embodiment, the sensing information includes multimodal sensing information, which includes at least image sensing information, audio sensing information, device posture information, and lens parameter information.

[0020] To improve the accuracy and richness of scene semantic information recognition, the perceptual information includes multimodal perceptual information, which are complementary perceptual signals collected from different physical dimensions or information types. Through collaborative analysis, the state and intent of the current shooting scene can be understood more comprehensively and robustly.

[0021] Specifically, image perception information includes visual data captured by the image sensor of the target camera. For example, image perception information includes real-time video streams, preprocessed visual features (such as optical flow maps, saliency maps, object detection boxes, segmentation masks, etc.), and cached historical continuous frame sequences for temporal modeling and motion prediction. It is used to identify subjects (such as people and objects) in a scene and analyze their position, posture, behavior, and spatial relationship with the environment. Audio perception information refers to sound signals collected by microphones or microphone arrays and their derived semantic features, used to assist in determining acoustic events or interactive intentions. For example, audio perception information includes sound source direction and speech activity detection results. By locating the sound source direction using a microphone array, when someone speaks off-screen, the pan-tilt unit automatically turns towards the sound source to capture the image, compensating for the shortcomings of purely visual methods in occluded or silent conditions. Device attitude information refers to sensor data reflecting the spatial orientation and motion state of the target camera. It is typically provided by a built-in or external inertial measurement unit (IMU). For example, device attitude information includes three-axis angular velocity, three-axis acceleration, and the fused attitude angles. This information is used to compensate for camera shake, establish a motion model in the world coordinate system, and provide closed-loop feedback for gimbal control. Lens parameter information refers to metadata describing the camera's optical imaging characteristics, directly affecting image composition and geometric mapping. This includes, but is not limited to, exposure parameters such as current focal length, zoom magnification, field of view, focus distance, aperture value, and shutter speed. Lens parameter information can be used not only for image geometric correction but also as constraints on camera movement strategies, such as limiting rapid panning in telephoto mode to avoid image shake.

[0022] Optionally, multimodal sensing information also includes depth information, such as depth maps or point cloud data acquired by LiDAR, which can be used to more accurately handle complex spatial occlusion relationships, such as shooting through crowds.

[0023] This embodiment overcomes the limitations of a single modality by fusing multimodal perception information, achieving highly robust semantic understanding of complex dynamic scenes, thus providing a reliable basis for subsequent camera movement strategy selection and control sequence generation.

[0024] Optionally, multimodal perception information from the target camera is acquired, the multimodal perception information is preprocessed, and recognition is performed based on the preprocessed multimodal perception information to generate corresponding scene semantic information.

[0025] Specifically, preprocessing includes, but is not limited to: denoising, normalizing, and resizing video frames; aligning and synchronizing other modal perception information such as IMU, audio, and depth based on timestamps of image perception information; organizing the current frame and historical frames based on a sliding window; and extracting intermediate features such as candidate subject regions, human key points, head pose, facial orientation, and motion vectors.

[0026] S120. Determine the corresponding target camera movement strategy based on the shooting subject information and the spatial layout of the frame indicated by the scene semantic information.

[0027] Based on the subject information and spatial layout indicated by the scene semantic information, a target camera movement strategy matching the current scene is selected from multiple preset candidate camera movement strategies. Alternatively, the subject information and spatial layout indicated by the scene semantic information are input into a large visual language model, which then outputs the corresponding target camera movement strategy.

[0028] For example, candidate camera movement strategies include, but are not limited to, tracking, cinematic panning, zooming in / out, circling shots, over-the-shoulder shots, and automatic composition correction. Based on a rule engine, classification model (such as a lightweight neural network), or reinforcement learning strategy, the matching degree of each candidate camera movement strategy is scored by comprehensively considering factors such as the subject information and the subject priority, movement trend, and composition quality score in the spatial layout of the frame. The target camera movement strategy is determined from the candidate camera movement strategies based on the score results.

[0029] S130. Based on the target camera movement strategy, predict the camera control sequence in the future time series, so as to drive the target camera to rotate according to the camera control sequence.

[0030] The camera control sequence refers to a set of instructions arranged chronologically to precisely control the camera's motion state, covering the acquisition of video frames for the next N frames. For example, the camera control sequence includes the angular velocity sequence of each degree of freedom of the gimbal in each of the next N frames; the coordinate path point information of each degree of freedom of the gimbal; and the gimbal attitude curve parameters, such as spline curve parameters and Bézier curve control points.

[0031] In one feasible embodiment, the camera control sequence includes at least the point trajectory information, angular velocity information, and timestamp information for each frame in the future time sequence.

[0032] Specifically, point trajectory information refers to the target camera's path in three-dimensional space as its orientation angle changes over time. This includes yaw (horizontal rotation angle, controlling left and right turns); pitch (vertical tilt angle, controlling vertical tilt); and roll (rotation angle around the optical axis, in devices supporting a three-axis gimbal). Point trajectory information specifies the angle value to be reached in each frame of the future time sequence, forming a continuous angle trajectory. Angular velocity information refers to the expected rate of change of angular velocity for each frame, used to guide the target camera to reach the target angle at an appropriate rate. Angular velocity information is used to smooth acceleration / deceleration, avoid mechanical shocks caused by sudden changes, and improve tracking accuracy and dynamic response. Timestamp information is used to indicate that each control command is accompanied by a precise timestamp, ensuring the timing consistency of multi-axis collaboration with external systems (such as image acquisition and audio synchronization), and guaranteeing that the control sequence is executed at the correct time. Optionally, the camera control sequence may also include zoom control instructions for each frame in the future time sequence, such as target focal length, zoom magnification, or digital scaling ratio, to coordinate with camera movement strategies to achieve changes in shot size, such as synchronous zooming during zoom-in.

[0033] For example, each candidate camera movement strategy predefines a corresponding standard kinematic template, such as circular trajectory parameters and acceleration curves. The corresponding target camera movement strategy template is instantiated based on parameters such as the current subject position and distance, generating a specific control sequence. Alternatively, the target camera movement strategy can be transformed into an optimization objective, such as minimizing the distance the subject deviates from the center of the frame. The optimal control sequence is then calculated through optimization and used as the camera control sequence for future frames. Alternatively, a trained deep neural network can be used, taking the target camera movement strategy as input and outputting the camera control sequence for the next N frames. The network input can also include scene semantic information.

[0034] The generated camera control sequence is sent to the gimbal servo control system of the target camera to drive the gimbal motors. For example, the controller parses the angle and angular velocity commands for each frame; generates PWM signals or current commands through a PID control algorithm; and drives the brushless motors or stepper motors of the yaw / pitch / roll axes to ensure the gimbal's actual posture smoothly and accurately tracks the predicted trajectory. Simultaneously, closed-loop correction can be performed using IMU feedback to compensate for disturbances or model errors. Alternatively, the adjustment angle can be determined based on the camera control sequence, and the target camera can be manually adjusted according to this angle, or the target camera's angle can be adjusted using other mechanical devices. The specific implementation type of the mechanical device is not limited here.

[0035] This embodiment transforms high-level camera movement strategies into structured control sequences containing angles, angular velocities, and timestamps, such as tracking the subject, advancing the lens, and circling shots. These are then converted into low-level control signals that can be directly used by the actuators, i.e., camera control sequences within the future timeline. This achieves high-precision, low-shake, and cinematic-quality automatic camera movement.

[0036] Optionally, the process of determining the camera control sequence is repeated according to a preset control period, such as every 20ms or 50ms. For example, in response to the arrival command of the preset control period, the perception information of the target camera corresponding to the current control period is obtained, and scene semantic information corresponding to the current control period is generated based on the perception information; the target camera movement strategy corresponding to the current control period is determined based on the shooting subject information and the spatial layout of the screen indicated by the scene semantic information; based on the target camera movement strategy, the camera control sequence in the future time sequence corresponding to the current control period is predicted, so as to drive the gimbal motor to rotate according to the camera control sequence.

[0037] This embodiment avoids control delays caused by scene changes through periodic prediction, thereby improving the timeliness of camera motion control.

[0038] In one feasible embodiment, S130 includes: Determine the initial camera control sequence for future timeframes based on the target camera movement strategy; The initial camera control sequence is optimized based on kinematic and stability constraints to obtain the final camera control sequence in future time series.

[0039] The initial camera control sequence primarily reflects the camera movement intentions at higher levels, but due to model training biases, sudden scene changes, or idealized assumptions in the strategy, it may not fully meet the physical limitations or shooting stability requirements of the actual gimbal hardware. For example, it may output angular velocities exceeding the maximum motor speed or produce high-frequency jitter in acceleration changes. Therefore, to ensure that the control sequence can be executed and the image appears natural, two types of core constraints are applied to the initial camera control sequence directly generated based on the target camera movement strategy.

[0040] Kinematic constraints are determined based on the physical capability boundaries of the gimbal hardware to ensure that commands remain within the acceptable range of the target camera. Examples of kinematic constraints include: upper limit constraints on angles (ensuring each axis angle does not exceed mechanical limits); upper limit constraints on angular velocity (ensuring the angular velocity at any given moment does not exceed the maximum output speed of the motor); and upper limit constraints on angular acceleration (ensuring the rate of change of angular velocity is limited by motor torque and inertia, preventing overload during start-up / stop). Stability constraints are determined based on the visual quality of the video image to suppress unnecessary or interfering motion. Examples of stability constraints include: image jitter suppression constraints (minimizing high-frequency, small angular fluctuations through smoothing filtering or optimizing the objective function); compositional continuity constraints (preventing sudden changes in the subject's position within the image, requiring the subject's trajectory to maintain low-order continuity in the image plane); and motion intention constraints. Figure 1 Consistency constraints are used to ensure that the optimized trajectory remains consistent with the semantic goals of the original camera movement strategy.

[0041] For example, the initial camera control sequence is used as the initial value to construct a constrained optimization problem, such as quadratic programming (QP) or nonlinear programming (NLP). The objective function is constructed based on kinematic and stability constraints. The optimal trajectory that satisfies all constraints is solved in each control cycle to achieve dynamic correction. Finally, the gimbal motor is driven to rotate according to the constrained and corrected final camera control sequence.

[0042] This embodiment optimizes the camera control sequence by applying constraints, ensuring that the final camera control sequence retains the semantic intent of the original target camera movement strategy while also meeting the kinematic feasibility and shooting stability requirements of the gimbal, thereby improving the accuracy of camera motion control.

[0043] The technical solution of this embodiment determines scene semantic information by understanding perceived information, determines a matching target camera movement strategy based on the scene semantic information, and then converts the target camera movement strategy into camera control commands. This achieves automatic camera movement with scene semantic understanding capabilities and anthropomorphic smoothness, avoiding mechanical tracking that relies solely on target coordinates. It realizes intelligent camera movement with directorial thinking, significantly improving the viewing experience, narrative quality, and professionalism of video content. Furthermore, the camera control sequence corresponds to a future time sequence, making the lens movement more continuous and smooth, and optimizing the problems of sudden stops, turns, and image shake.

[0044] Figure 2 This is a flowchart of another motion control method for a camera provided in an embodiment of the present invention. This embodiment further refines the process of determining the camera movement strategy in the above embodiments. Figure 2 As shown, the method includes: S210. Acquire the perception information of the target camera and generate corresponding scene semantic information based on the perception information.

[0045] S220. Determine the subject intent prediction result based on the subject information and spatial layout indicated by the scene semantic information.

[0046] Among these, subject information refers to high-level semantic features extracted from perceptual data regarding one or more key objects, including but not limited to: subject category, identity identifier, location information, motion state, behavioral attributes, and facial orientation or gaze direction. Image spatial layout refers to the structural relationship between the subject and its surrounding environment in visual space, including but not limited to: the subject's relative position in the frame, the geometric relationship between the subject and the background or other subjects, available space for movement, scene context, and aesthetically pleasing compositional areas. Subject information and image spatial layout together constitute the contextual understanding foundation of scene image information.

[0047] Subject intention prediction results refer to the inference of a subject's future short-term behavior or interaction goals. They are observable and predictable behavioral or state trends, and can take the form of discrete categories, continuous trajectories, or probability distributions. For example, subject intention prediction results include behavioral intention categories, future trajectory predictions, and visual attention focus. Among them, behavioral intention categories include speaking, preparing to leave the frame, and interacting with others. Future trajectory prediction refers to predicting the subject's position information within a preset time period in the future based on historical movement trends. Visual attention focus refers to predicting the area where the subject is most likely to be noticed by the audience, as well as the predicted object of the subject's attention in the frame.

[0048] For example, rule-based inference engines determine the subject's intent prediction result based on the subject information and the spatial layout of the frame; or based on machine learning models, such as LSTM, the subject information and the spatial layout of the frame are input into the model, and the intent label or trajectory information is output.

[0049] The system predicts the subject's intentions, infers the subject's behavioral goals, interaction tendencies, or visual focus in the current or short-term future. This prediction result will serve as an important basis for the dynamic adjustment of subsequent camera movement strategies, enabling camera control to upgrade from passively responding to the current state to actively predicting future behavior, significantly improving the intelligence, smoothness, and narrative expressiveness of camera movements.

[0050] S230. Based on the subject's intention, predict the result, and determine the corresponding target camera movement strategy.

[0051] Among them, shooting intent refers to the shooting goal or aesthetic orientation defined by the user, preset rules or context. For example, shooting intent can be determined by user instructions, or by the correspondence between scene recognition results and shooting intent mapping table, or by the shooting intent can be determined based on high-level task objectives.

[0052] The subject intent prediction result reflects the dynamic intent of the subject, while the shooting intent reflects the subjective expressive needs of the image. By integrating the subject intent prediction result and the shooting intent, joint reasoning and strategy mapping are performed to match and determine the target shooting strategy that best suits the current overall context from the preset candidate shooting strategies. The candidate shooting strategies include at least: tracking, cinematic panning, zooming in / out, circling shooting, automatic composition correction, and multi-subject switching.

[0053] For example, a multi-dimensional decision table can be constructed. For instance, if the subject's intention is to speak and the shooting intention is to take meeting notes, then a smooth transition to a close-up is selected; if the subject's intention is to interact with multiple people and the shooting intention is to express a narrative, then an over-the-shoulder perspective switch or a two-person medium shot composition is selected; or, based on a pre-trained lightweight classifier, the input is the feature vector of the subject's intention prediction result and the shooting intention and encoding result, and the output is the target camera movement strategy.

[0054] In one feasible embodiment, S230 includes: Determine the subject's movement trend and line of sight based on the predicted subject's intention; Determine scene state inference information based on the subject's movement trend and gaze direction; Based on the scene state and shooting intention, determine the corresponding target camera movement strategy.

[0055] The subject's intent prediction result is an inference of the subject's future short-term behavior based on scene semantic information. Its output can be a probability distribution, category label, or continuous trajectory. The subject's motion trend refers to the spatial displacement direction and velocity change trend of the subject over a future period, such as moving at a constant speed to the left of the frame, accelerating towards the camera, about to stop and turn around, or lingering slightly in place. For example, the subject's motion trend is determined through trajectory prediction models, such as LSTM (Long Short-Term Memory), Kalman filtering, or optical flow analysis, and is typically represented as a velocity vector, acceleration, or future position sequence.

[0056] The direction of a subject's gaze refers to the spatial orientation of their head or the focus of their visual attention. Examples include staring at the center of the camera, looking at someone in the upper right corner of the frame, looking down at an object in their hand, or having a wandering gaze without a clear focus. For instance, the direction of a subject's gaze can be inferred from the subject's intention prediction results through facial landmark detection, 3D head pose estimation, or eye-tracking models, and is usually represented as a unit vector or azimuth angle.

[0057] By fusing and analyzing the subject's movement trend and gaze direction, higher-order scene state inference information is generated to describe the dynamic semantic state of the current scene. For example, scene state inference information includes, but is not limited to: interaction pattern recognition; for instance, if subject A's gaze is directed at subject B, and B is moving towards A, it is inferred that a conversation is about to begin; if multiple people's gazes converge on the same area, it is inferred that they are all paying attention to an event or object. It also includes behavioral phase judgment; for example, if a subject slowly walks towards the podium and scans the audience, it is inferred that they are preparing to begin a speech; if a subject quickly retreats and avoids eye contact, it is inferred that the interaction is ending and they are preparing to leave. It can also include attention focus area localization; for example, by combining gaze direction and spatial geometry, it calculates the three-dimensional points of interest that the subject is most likely to focus on, such as an exhibit, an audience member, or a control panel. Furthermore, it can assess the scene's dynamic level, determining whether the scene is in a static presentation, a moderately interactive, or a highly dynamic pursuit state based on movement speed, frequency of direction changes, and gaze switching rate.

[0058] For example, a rule engine is used to determine scene state reasoning information based on the subject's movement trend and gaze direction by modeling the relationships between subjects through a graph neural network or a temporal state machine. The output scene state reasoning information is a structured semantic label or a state vector.

[0059] The scene state inference information and shooting intent are then jointly used to make a decision to determine the most suitable target camera movement strategy. For example, based on a pre-trained lightweight classifier, the input is the feature vector of scene state inference information, the shooting intent, and the encoding result, and the output is the target camera movement strategy.

[0060] For example, the scene state inference information is that the character's body center of gravity is leaning forward, indicating an imminent start to run to the left; the character is speaking, so a medium close-up is suitable and large panning shots should be limited; the pet is running fast and there is insufficient white space in front of it, so space should be reserved in the direction of movement; the scene contains two subjects in conversation, so the composition should be dynamically balanced between the two, etc.; the target camera movement strategy is Strategy A (left-shifting camera movement): when the subject is facing left and begins to move, predict the visual white space area in the next T seconds, and then output the left-shifting camera movement trajectory; Strategy B (progressing camera movement): when the subject is facing the camera, and it is determined from the scene and intention that a progression and close-up are needed, then output the progressive camera movement trajectory.

[0061] This embodiment achieves stronger contextual understanding through a four-level reasoning chain from intent to behavioral features, then to scene state, and finally to camera movement strategy. This results in higher camera movement intelligence, avoids mechanical tracking, and enables director-like proactive composition. Furthermore, it can still make reasonable camera movement decisions in complex and ever-changing scenes, demonstrating good robustness and adaptability.

[0062] Optionally, the subject's movement trend, gaze direction, and emotion recognition result are determined based on the subject's intention prediction result; scene state inference information is determined based on the subject's movement trend, gaze direction, and emotion recognition result; and the corresponding target camera movement strategy is determined based on the scene state inference information and shooting intention.

[0063] This embodiment dynamically adjusts the camera movement speed based on the subject's facial expression (happy, nervous, sad) (e.g., fast camera movement when happy, slow camera movement when sad), further improving the accuracy of the camera movement strategy.

[0064] S240. Based on the target camera movement strategy, predict the camera control sequence in the future time series, so as to drive the target camera to rotate according to the camera control sequence.

[0065] The technical solution of this embodiment avoids the limitations of single-dimensional decision-making by simultaneously considering the objective behavioral trends of the subject and the subjective expressive goals of the shooting. The integration of the two makes the camera movement both in line with real dynamics and in line with creative intent, thereby generating a natural, professional and narratively logical automatic camera movement effect.

[0066] Figure 3 This is a flowchart of another camera motion control method provided by an embodiment of the present invention. This embodiment further refines the optimization and adjustment process of the camera control sequence corresponding to the continuous control cycle in the above embodiments. For example... Figure 3 As shown, the method includes: S310. Acquire the perception information of the target camera and generate corresponding scene semantic information based on the perception information.

[0067] S320. Based on the subject information and spatial layout of the scene indicated by the semantic information, determine the corresponding target camera movement strategy.

[0068] S330. Based on the target camera movement strategy corresponding to the current control cycle, predict the second camera control sequence within the second future time series corresponding to the current control cycle.

[0069] In this embodiment of the invention, the entire process of determining the target camera movement strategy is divided into continuous control cycles with fixed time intervals, such as executing a complete camera control sequence determination closed loop every 20 milliseconds, 50 milliseconds, or 100 milliseconds. Each control cycle independently completes the entire process from environmental perception to control command generation, thereby achieving real-time response to dynamic scenes.

[0070] The current control cycle refers to the control execution window at the current time t, which is the control iteration unit being processed relative to the previous cycle (t-1) and the next cycle (t+1). At the beginning of the current control cycle, the latest perception information is collected to predict the camera control sequence, and the resulting control sequence is used to guide the pan-tilt dynamics for a period of time in the future.

[0071] The second future time sequence is a time interval lasting N frames, starting from the first frame after the current moment, for example, covering [t+1, t+N]. The window length N can be dynamically adjusted according to the system latency tolerance, gimbal response speed, and scene dynamics. The control sequence corresponding to this second future time sequence is entirely driven by the target camera movement strategy determined within the current control cycle.

[0072] S340. Obtain the first camera control sequence predicted in the previous control cycle. The first camera control sequence corresponds to the first future time sequence.

[0073] The previous control cycle refers to the control iteration unit that has been completed immediately before the current control cycle. For example, if the current control cycle is t, then the previous control cycle is marked as t-1 in time. The execution time window corresponding to the previous control cycle is [t-1, t), which is a complete control loop that just ended before the current time t. The camera movement strategy selection in this cycle is based on the perception information available at time t-1. Furthermore, the first camera control sequence predicted in the previous control cycle is also used to guide the gimbal movement for a future period starting from time t. For example, the first future time sequence is a time interval of N frames starting from the first frame after the previous control cycle, such as covering [t, t+N-1].

[0074] S350. Adjust the control sequence of the second camera corresponding to the second future time sequence predicted in the current control cycle according to the first camera control sequence.

[0075] Since the previous control cycle predicts the first camera control sequence covering the first future time series, and the current control cycle predicts the second camera control sequence covering the second future time series, there is an overlap in the time regions of the first and second camera control sequences. Therefore, in order to avoid decision-making errors caused by different camera control actions predicted by different control cycles at the same time point, it is necessary to adjust the second camera control sequence according to the first camera control sequence to avoid control command jumps or PTZ jitter due to repeated planning.

[0076] For example, based on the first camera control sequence and the second camera control sequence, the overlapping interval of the first future time sequence and the second future time sequence in time is determined, and the first action subsequence and the second action subsequence within the overlapping interval of the first camera control sequence and the second camera control sequence are extracted; the first action subsequence and the second action subsequence are weighted and fused to obtain the fused action subsequence; based on the fused action subsequence, the part of the corresponding overlapping interval in the second camera control sequence is updated to generate the final second camera control sequence.

[0077] The fusion weights of the first and second action subsequences can be determined based on the degree of difference between them. For example, when the difference between the first and second action subsequences is small, the first action subsequence is given a higher weight to maintain the smoothness and continuity of the motion; when the difference is large, the weight of the second action subsequence is increased to ensure a rapid response to scene changes or new camera movements. The degree of difference is judged by comparing the gimbal angles of the two subsequences within the overlapping time period. If the angle change trends are consistent and the numerical deviations are small, it is considered a low difference; if the angle directions are opposite and the change amplitudes are significantly different, it is considered a high difference. For example, multiple difference level thresholds can be set. When the difference is below the first threshold, the fusion weight of the second action subsequence is set to a lower value, such as 20%–30%, mainly following the historical trajectory; when the difference is between the first and second thresholds, the weight increases linearly or stepwise; when the difference exceeds the second threshold, the weight is increased to a higher level, such as 80%–95%.

[0078] By using an adaptive weighting mechanism based on the degree of difference, it is possible to flexibly respond to dynamic changes in the environment while maintaining image stability. This avoids jitter caused by blindly trusting new predictions and also prevents the loss of the ability to capture sudden behaviors due to over-reliance on historical sequences.

[0079] In one feasible embodiment, S350 includes: Based on the first camera control sequence and the second camera control sequence, determine the time overlap interval between the first future timing sequence and the second future timing sequence, and extract the first action subsequence and the second action subsequence within the overlap interval of the first camera control sequence and the second camera control sequence. Apply kinematic consistency constraints to the first and second action subsequences to generate optimized action subsequences that satisfy the constraints. Based on the optimized action subsequence, update the corresponding overlapping interval in the second camera control sequence.

[0080] The first camera control sequence is predicted in the previous control cycle, and the time range it covers is called the first future time series. For example, the first future time series starts from the current time t and lasts for N frames, covering [t, t+N-1]. The second camera control sequence is predicted in the current control cycle, and the time range it covers is called the second future time series. For example, the second future time series starts from the next time t+1 and covers [t+1, t+N]. Since both time series are oriented towards the near future, there is a common time period on the time axis, that is, from t+1 to t+N-1. This interval is the time overlap interval.

[0081] Specifically, by comparing the timestamp information of the first camera control sequence and the second camera control sequence, the overlapping interval is automatically identified, and the corresponding control instruction segments are extracted from them respectively. For example, the portion covering [t+1, t+N-1] is extracted from the first camera control sequence to form the first action subsequence; the portion of the same time range is extracted from the second camera control sequence to form the second action subsequence. These two subsequences describe two different control expectations for the same future time period.

[0082] To ensure that the gimbal's movements are smooth, natural, and in accordance with physical laws, kinematic consistency constraints are applied to the two sub-sequences of movements mentioned above. The core objective of kinematic consistency constraints is to ensure that the final output trajectory remains continuous in key kinematic quantities such as angle, angular velocity, and angular acceleration, and to avoid jerk (angular acceleration) jumps that are sensitive to the human eye.

[0083] Specifically, an optimization problem is constructed based on kinematic consistency constraints. Using the second action subsequence as the initial reference and incorporating the first action subsequence as historical guidance, the optimal trajectory is solved under the following constraints: minimizing the positional difference between the first and second action subsequences; minimizing the velocity and acceleration differences between the first and second action subsequences; and satisfying the gimbal hardware angle range and motor capability limitations. Solving this constrained optimization problem generates an optimized action subsequence that satisfies all kinematic and stability requirements. This sequence respects the latest decision intent of the current cycle while inheriting the motion continuity of the previous cycle. That is, the optimized action subsequence satisfies: continuous angle changes between adjacent time steps; continuous angular velocity changes; continuous angular acceleration changes; and no jerk jumps exceeding a threshold.

[0084] Finally, the generated optimized action subsequence replaces the original instructions within the corresponding overlapping intervals of the original second camera control sequence. For example, the non-overlapping portion of the second camera control sequence (i.e., the newly added frame at time t+N in the second sequence) remains unchanged, preserving its forward-looking nature; the overlapping portion is replaced by the optimized action subsequence. Optionally, the smoothing optimization of the camera control sequence can be achieved through spline interpolation, Bézier curve fitting, optimal control, MPC, least squares smoothing, Kalman post-processing, etc.

[0085] This embodiment obtains an updated second camera control sequence through kinematic consistency constraints, which not only has the ability to respond to the current scene in real time, but also ensures seamless connection with historical actions, thereby achieving a highly stable and intelligent automatic camera movement effect.

[0086] The technical solution of this embodiment effectively solves the contradiction between periodic repetitive prediction and motion continuity by adjusting the camera control sequence of adjacent control cycles, ensuring the global consistency of the camera movement process in long sequences, and significantly improving the stability and professionalism of video shooting.

[0087] Figure 4 This is a schematic diagram of the structure of a motion control device for a camera provided in an embodiment of the present invention. Figure 4 As shown, the device includes: The scene understanding module 410 is used to acquire the perception information of the target camera and generate corresponding scene semantic information based on the perception information; The camera movement strategy determination module 420 is used to determine the corresponding target camera movement strategy based on the shooting subject information and the spatial layout of the frame indicated by the scene semantic information. The control sequence prediction module 430 is used to predict the camera control sequence in the future time sequence based on the target camera movement strategy, so as to drive the target camera to rotate according to the camera control sequence.

[0088] The technical solution of this embodiment determines scene semantic information by understanding the perceived information, determines the matching target camera movement strategy based on the scene semantic information, and then converts the target camera movement strategy into camera control commands. This achieves automatic camera movement with scene semantic understanding capabilities and anthropomorphic smoothness, avoiding mechanical tracking that relies solely on target coordinates. It realizes intelligent camera movement with directorial thinking, significantly improving the viewing experience, narrative quality, and professionalism of video content.

[0089] Optionally, the camera movement strategy determination module 420 includes: The subject intent prediction result determination unit is used to determine the subject intent prediction result based on the shooting subject information and the spatial layout of the screen indicated by the scene semantic information; The target camera movement strategy determination unit is used to determine the corresponding target camera movement strategy based on the subject's intention prediction result and the shooting intention.

[0090] Optional, the target camera movement strategy determination unit is specifically used for: Determine the subject's movement trend and gaze direction based on the subject's intention prediction results; Determine scene state reasoning information based on the subject's movement trend and the subject's line of sight; Based on the scene state reasoning information and the shooting intention, the corresponding target camera movement strategy is determined.

[0091] Optionally, the sequence prediction module 430 is controlled, specifically for: The initial camera control sequence for future timeframes is determined based on the target camera movement strategy. The initial camera control sequence is optimized based on kinematic and stability constraints to obtain the final camera control sequence in the future time series.

[0092] Optionally, the device further includes a continuous periodic sequence adjustment module, comprising: The previous cycle sequence acquisition unit is used to acquire the first camera control sequence predicted in the previous control cycle, and the first camera control sequence corresponds to the first future time series. The current period sequence adjustment unit is used to control and adjust the second camera control sequence corresponding to the second future time sequence predicted in the current control period according to the first camera control sequence.

[0093] Optionally, the current period sequence adjustment unit is specifically used for: Based on the first camera control sequence and the second camera control sequence, determine the time overlap interval between the first future timing sequence and the second future timing sequence, and extract the first action subsequence and the second action subsequence within the overlap interval. Apply kinematic consistency constraints to the first action subsequence and the second action subsequence to generate an optimized action subsequence that satisfies the constraints; Based on the optimized action subsequence, update the portion of the second camera control sequence corresponding to the overlapping interval.

[0094] Optionally, the sensing information includes multimodal sensing information, which includes at least image sensing information, audio sensing information, device posture information, and lens parameter information; The camera control sequence includes at least the point trajectory information, angular velocity information, and timestamp information for each frame in the future time sequence.

[0095] The motion control device for a camera provided in the embodiments of the present invention can execute the motion control method for a camera provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0096] The acquisition, storage, use, and processing of data in this application comply with relevant national laws and regulations and do not violate public order and good morals.

[0097] According to embodiments of this disclosure, this disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0098] Figure 5 A schematic diagram of an electronic device 10, which can be used to implement embodiments of the present invention, is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices (e.g., helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the invention described and / or claimed herein.

[0099] like Figure 5 As shown, the electronic device 10 includes at least one processor 11 and a memory, such as a read-only memory (ROM) 12 or a random access memory (RAM) 13, communicatively connected to the at least one processor 11. The memory stores computer programs executable by the at least one processor. The processor 11 can perform various appropriate actions and processes based on the computer program stored in the ROM 12 or loaded from storage unit 18 into the RAM 13. The RAM 13 can also store various programs and data required for the operation of the electronic device 10. The processor 11, ROM 12, and RAM 13 are interconnected via a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0100] Multiple components in electronic device 10 are connected to I / O interface 15, including: input unit 16, such as keyboard, mouse, etc.; output unit 17, such as various types of displays, speakers, etc.; storage unit 18, such as disk, optical disk, etc.; and communication unit 19, such as network card, modem, wireless transceiver, etc. Communication unit 19 allows electronic device 10 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0101] Processor 11 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. Processor 11 performs the various methods described above, such as motion control methods for cameras.

[0102] In some embodiments, the camera motion control method may be implemented as a computer program tangibly contained in a computer-readable storage medium, such as storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or installed on electronic device 10 via ROM 12 and / or communication unit 19. When the computer program is loaded into RAM 13 and executed by processor 11, one or more steps of the camera motion control method described above may be performed. Alternatively, in other embodiments, processor 11 may be configured to perform the camera motion control method by any other suitable means (e.g., by means of firmware).

[0103] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific reference products (ASSPs), systems-on-a-chip (SoCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0104] Computer programs used to implement the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when executed by the processor, the computer programs cause the functions / operations specified in the flowcharts and / or block diagrams to be performed. The computer programs may be executed entirely on a machine, partially on a machine, or as a standalone software package, partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0105] In the context of this invention, a computer-readable storage medium can be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, apparatus, or device. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination thereof. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0106] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0107] The systems and technologies described herein can be implemented in computing systems that include back-end components (e.g., as data servers), or computing systems that include switching components (e.g., application servers), or computing systems that include front-end components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such back-end, switching, or front-end components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), blockchain networks, and the Internet.

[0108] A computing system can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a hosting product within the cloud computing service system to address the shortcomings of traditional physical hosts and VPS services, such as high management difficulty and weak business scalability.

[0109] In particular, according to embodiments of the present invention, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of the present invention include a computer program product comprising a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication unit 19, or installed from storage unit 18, or installed from ROM 12. When the computer program is executed by processor 11, it performs the functions defined in the methods of the embodiments of the present invention.

[0110] This invention also provides a computer program product, including a computer program that, when executed by a processor, implements the motion control method for a camera as provided in any embodiment of this application.

[0111] In implementing the computer program product, computer program code for performing the operations of this invention can be written in one or more programming languages ​​or a combination thereof. Programming languages ​​include object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as C or similar languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0112] It should be understood that the various forms of processes shown above can be used, with steps reordered, added, or deleted. For example, the steps described in this invention can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this invention can be achieved, and this is not limited herein.

[0113] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A motion control method for a camera, characterized in that, The method includes: Acquire the perception information of the target camera, and generate corresponding scene semantic information based on the perception information; Based on the subject information and spatial layout of the scene indicated by the semantic information, the corresponding target camera movement strategy is determined; Based on the target camera movement strategy, the camera control sequence in the future time series is predicted, so as to drive the target camera to rotate according to the camera control sequence.

2. The method according to claim 1, characterized in that, The step of determining the corresponding target camera movement strategy based on the shooting subject information and the spatial layout of the frame indicated by the scene semantic information includes: The subject intent prediction result is determined based on the subject information and spatial layout indicated by the scene semantic information. Based on the predicted subject intent and the shooting intent, the corresponding target camera movement strategy is determined.

3. The method according to claim 2, characterized in that, The step of determining the corresponding target camera movement strategy based on the subject's intention prediction result and the shooting intention includes: Determine the subject's movement trend and gaze direction based on the subject's intention prediction results; Determine scene state reasoning information based on the subject's movement trend and the subject's line of sight; Based on the scene state reasoning information and the shooting intention, the corresponding target camera movement strategy is determined.

4. The method according to claim 1, characterized in that, The prediction of the camera control sequence in the future time series based on the target camera movement strategy includes: The initial camera control sequence for future timeframes is determined based on the target camera movement strategy. The initial camera control sequence is optimized based on kinematic and stability constraints to obtain the final camera control sequence in the future time series.

5. The method according to any one of claims 1-4, characterized in that, The method also includes: Obtain the first camera control sequence predicted in the previous control cycle, where the first camera control sequence corresponds to the first future time sequence; The second camera control sequence, which is predicted to correspond to the second future time sequence in the current control cycle, is adjusted according to the first camera control sequence.

6. The method according to claim 5, characterized in that, The step of adjusting the second camera control sequence, which is predicted to correspond to the second future time sequence in the current control period based on the first camera control sequence, includes: Based on the first camera control sequence and the second camera control sequence, determine the time overlap interval between the first future timing sequence and the second future timing sequence, and extract the first action subsequence and the second action subsequence within the overlap interval. Apply kinematic consistency constraints to the first action subsequence and the second action subsequence to generate an optimized action subsequence that satisfies the constraints; Based on the optimized action subsequence, update the portion of the second camera control sequence corresponding to the overlapping interval.

7. The method according to claim 1, characterized in that, The sensing information includes multimodal sensing information, which includes at least image sensing information, audio sensing information, device posture information, and lens parameter information. The camera control sequence includes at least the point trajectory information, angular velocity information, and timestamp information for each frame in the future time sequence.

8. A motion control device for a camera, characterized in that, The device includes: The scene understanding module is used to acquire the perception information of the target camera and generate corresponding scene semantic information based on the perception information; The camera movement strategy determination module is used to determine the corresponding target camera movement strategy based on the shooting subject information and the spatial layout of the screen indicated by the scene semantic information; The control sequence prediction module is used to predict the camera control sequence in the future time series based on the target camera movement strategy, so as to drive the target camera to rotate according to the camera control sequence.

9. An electronic device, characterized in that, The electronic device includes: At least one processor; and A memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the motion control method of the camera according to any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that, when executed by a processor, implement the motion control method for the camera as described in any one of claims 1-7.

Images