A camera robot trajectory planning method based on voice intention analysis

Abstract

The application discloses a camera robot trajectory planning method based on voice intention analysis, which comprises the following steps: acquiring multi-modal instructions, site layout parameters and device parameters containing lens optical feasible region; the multi-modal instructions are analyzed into a lens operation intention by a multi-modal large model, and are quantified into a picture effect constraint sequence described by image space quantities such as target subject imaging size proportion, picture position, background perspective magnification and depth of field range; a three-dimensional virtual scene is established and a target subject and a background reference are calibrated; an imaging projection model is established to map the end position and the lens optical parameters into the image space quantities; the picture effect constraint sequence is used as a constraint to jointly solve the motion trajectory and the lens optical parameter sequence; the execution result of the lens operation intention can be quantitatively verified on the picture, the imaging effect fluctuation caused by the separation of motion and optics planning is eliminated, and the imaging effect can be maintained when the device capacity is limited.

Description

Technical Field

[0001] This application relates to the field of intelligent camera equipment control technology, specifically to a camera robot trajectory planning method based on voice intent parsing. Background Technology

[0002] In film and television production, studio program recording, live broadcasting, and sports event broadcasting, intelligent camera robots are gradually replacing human cameramen in camera movement and shooting tasks. In related technologies, camera movement for camera robots is typically achieved through teach programming or remote control. Operators need to pre-set the robot's movement path and lens parameters point by point, resulting in long programming cycles, high requirements for operator expertise, and the need for reprogramming when shooting needs change. This makes it difficult to adapt to the frequent adjustments to camera positions and shot sizes required during program recording.

[0003] Some related technologies incorporate speech recognition or natural language understanding modules to categorize user language commands into preset camera movement templates, and then execute the shooting by calling the fixed trajectory corresponding to the template. However, this method can only cover a limited template library, making it difficult to accurately understand the complex camera movement intentions expressed by users through sketches, gestures, or reference videos. Furthermore, the results of intent parsing are usually abstract category labels or intent vectors, lacking a calculable correspondence between them and the amount of motion the robot can perform. The same label can produce significantly different footage under different locations and camera configurations, and the correctness of the parsing results cannot be verified before shooting.

[0004] Furthermore, in related technologies, robot motion trajectories and lens optical parameters are generally planned separately. This means the robot's spatial motion trajectory is planned first, and then the focal length and focus are adjusted based on feedback from the captured image during execution, with the lens optical parameters passively following. Due to the time consumption of image processing and the response delay of the lens motor, the feedback adjustment has a lag of one to several frames. When the robot moves at a high speed, the image size of the subject in the frame fluctuates significantly. When performing camera movement techniques such as sliding zoom, which require high precision in matching robot movement with lens zoom, post-compensation methods cannot guarantee the stability of the image effect, resulting in a high rate of unusable shots. Summary of the Invention

[0005] In view of this, the technical problem to be solved by the present invention is: how to provide a camera robot trajectory planning method based on voice intent parsing, so as to accurately convert the user's camera movement intention into a quantifiable and verifiable imaging effect, and enable the robot movement and lens optical parameters to be planned and executed collaboratively according to the imaging effect.

[0006] To achieve the above objectives, this invention proposes a camera robot trajectory planning method based on voice intent parsing, comprising the following steps: Step S1: Obtaining user-input multimodal commands, site layout parameters, and intelligent camera robot device parameters. The multimodal commands include at least two combinations of voice commands, gesture trajectories, hand-drawn sketches, and reference video frames. The intelligent camera robot device parameters include robot kinematic parameters and lens optical feasible region. The lens optical feasible region includes focal length range, zoom rate upper limit, focus rate upper limit, and aperture adjustment range. Step S2: Inputting the multimodal commands into a multimodal large-scale model for... Cross-modal fusion analysis is performed to obtain the camera movement intention, which is then quantified into a sequence of image effect constraints arranged along a unified time axis. Each constraint element in the sequence describes the imaging effect that the image should present at the corresponding moment using image space quantities. These image space quantities include at least two of the following: target subject imaging size ratio, target subject image position, background perspective magnification, and depth of field range. Step S3: A three-dimensional virtual scene is established based on the site layout parameters. A digital twin model of the intelligent camera robot is set in the three-dimensional virtual scene, and the spatial positions of the target subject and background reference objects are calibrated. Step S4: Establish a... Like a projection model, the imaging projection model maps the pose of the intelligent camera robot end effector and the lens optical parameters together into the image space quantity; using the intelligent camera robot end effector pose sequence and the lens optical parameter sequence as joint optimization variables, and the mapping relationship formed by the image effect constraint sequence through the imaging projection model as constraint conditions, it is jointly solved in the three-dimensional virtual scene to obtain the initial motion trajectory and the lens optical parameter sequence synchronized with the initial motion trajectory in time. The lens optical parameter sequence includes the focal length sequence, the focus distance sequence, and the aperture sequence; Step S5: Perform kinematic feasibility verification on the initial motion trajectory. Collision detection is performed, and the optical feasible region of the lens optical parameter sequence is verified. When any verification fails, based on the complementary mapping relationship between the end pose and the lens optical parameters in the imaging projection model for the same image space quantity, the image effect increment is redistributed between the motion trajectory and the lens optical parameter sequence and the solution is re-solved to obtain the executable motion trajectory and the executable lens optical parameter sequence. Step S6: The executable motion trajectory and the executable lens optical parameter sequence are aligned along a unified time axis and converted into a cooperative control command output to the intelligent camera robot actuator.

[0007] Further, in step S2, quantifying the camera movement intention into a sequence of visual effect constraints includes: Step S21: The multimodal large model parses the camera movement intention into multiple keyframe effect descriptions arranged in the shooting sequence. Each keyframe effect description includes the values ​​of the target subject's image size ratio, the target subject's image position, and the background perspective magnification at the corresponding moment; Step S22: Interpolation is performed between adjacent keyframe effect descriptions according to a preset effect transition curve to generate a sequence of visual effect constraints for each moment; Step S23: When the multimodal instruction includes a reference video frame, the target subject is detected frame by frame in the reference video frame, the target subject's image size ratio sequence is calculated, and the background feature points are tracked to calculate the background scale change sequence. The calculated target subject image size ratio sequence and background scale change sequence are used as constraint templates to correct the interpolated visual effect constraint sequence.

[0008] Further, in step S4, the imaging projection model is established as follows: based on the perspective projection relationship, the proportion of the target subject's imaging size is expressed as a mapping relationship between the lens focal length, the distance from the end pose to the target subject, and the actual size of the target subject; the background perspective magnification is expressed as the relationship between the ratio of the distance from the end pose to the target subject and the distance from the end pose to the background reference object over time; the depth of field range is expressed as the relationship between the foreground depth of field and the background depth of field determined by the focal length, aperture, and focusing distance; during joint solution, the equality constraints or interval constraints corresponding to each image spatial quantity in the image effect constraint sequence are used as hard constraints, and the weighted sum of the trajectory curvature change rate, focal length change rate, and focusing distance change rate is used as the objective function. The direct collocation method is used to discretize the continuous-time joint optimization problem into a nonlinear programming problem for solution, and the end pose sequence and lens optical parameter sequence are obtained simultaneously.

[0009] Further, in step S5, the reallocation of image effect increments between the motion trajectory and the lens optical parameter sequence includes: when the focal length sequence touches the focal length range boundary or zoom rate limit in the lens optical feasible domain at any time period, according to the imaging projection model, the change in the target subject's image size ratio that cannot be achieved by focal length change is converted into a radial displacement increment of the end pose along the target subject's visual axis, and the radial displacement increment is superimposed on the motion trajectory of the corresponding time period before re-performing kinematic feasibility verification and collision detection; when the initial motion trajectory is determined to be infeasible by collision detection or workspace verification at any time period, according to the imaging projection model, the change in the target subject's image size ratio corresponding to the reduced radial displacement is converted into a focal length increment, and the focal length increment is superimposed on the focal length sequence of the corresponding time period before re-performing optical feasible domain verification; when both the radial displacement increment and the focal length increment are infeasible in the same time period, the constraint boundary of the background perspective magnification rate in the image effect constraint sequence of that time period is relaxed within a preset tolerance range, while keeping the target subject's image size ratio constraint unchanged.

[0010] Furthermore, when the camera movement intention includes a zoom slider indicator, the image effect constraint sequence is set as follows: the proportion of the target subject's image size remains constant during the zoom slider period, and the background perspective magnification changes monotonically in a specified direction during the zoom slider period; in the joint solution of step S4, a coupling equation constraint is applied to ensure that the ratio of the lens focal length to the distance from the end pose to the target subject remains constant during the zoom slider period, and the focus distance sequence changes synchronously with the distance from the end pose to the target subject; based on the lens's focus breathing characteristic curve, the field of view offset caused by the focus distance change is calculated, the field of view offset is converted into a focal length compensation amount and superimposed on the focal length sequence, so that the proportion of the target subject's image size does not drift due to focus breathing during the zoom slider period.

[0011] Furthermore, after step S6, the process also includes a step of maintaining the image effect during execution: during the execution of the executable motion trajectory, the captured image is acquired in real time, the actual imaging size ratio and actual image position of the target subject are detected, and the effect deviation between the actual value and the constraint value at the corresponding moment in the image effect constraint sequence is calculated; using the motion response bandwidth of the intelligent camera robot body as the dividing frequency, the effect deviation is separated into high-frequency deviation components and low-frequency deviation components; according to the imaging projection model, the high-frequency deviation components are converted into lens optical parameter correction amounts and sent to the lens drive mechanism, and the low-frequency deviation components are converted into end-effector pose correction amounts and sent to the robot motion controller, so that the actual imaging effect during the execution process converges to the image effect constraint sequence.

[0012] Compared with related technologies, the camera robot trajectory planning method based on voice intent parsing proposed in this invention has the following advantages: the camera movement intent is quantified into a sequence of image effect constraints described by image space quantities, and the execution result of the intent can be directly measured and verified on the screen; the robot end pose and lens optical parameters are jointly solved forward by the imaging projection model, and the imaging effect is locked in the planning stage without relying on the execution period feedback compensation, thus eliminating the imaging size fluctuation and adjustment lag caused by separate planning; when the verification fails, the image effect increment is redistributed between the motion channel and the optical channel, and the imaging effect is maintained at the lowest cost even when the equipment capacity is limited; during the sliding zoom period, a coupling constraint with a constant ratio of focal length to object distance is applied and the field of view shift caused by focus breathing is compensated, and the subject imaging size drift is suppressed; during the execution period, the effect deviation is corrected by frequency division according to the robot motion response bandwidth, the high frequency deviation is borne by the faster optical channel, and the low frequency deviation is borne by the motion channel, and the actual imaging effect stably converges to the image effect constraint sequence. Attached Figure Description

[0013] Figure 1 A flowchart illustrating the overall process of a camera robot trajectory planning method based on voice intent parsing provided in this application; Figure 2 A flowchart illustrating the process of generating the visual effect constraint sequence provided in this application; Figure 3 A schematic diagram illustrating the mapping relationship of the imaging projection model provided in this application; Figure 4 A flowchart of the incremental redistribution process of the visual effects provided in this application; Figure 5 A schematic diagram of the sliding zoom period coupling constraint and focus breathing compensation provided in this application. Detailed Implementation

[0014] The present application will now be described in further detail with reference to embodiments. It is to be understood that the specific embodiments described herein are merely illustrative of the present application and are not intended to limit the scope of the application.

[0015] The technical solutions in the embodiments of this application will be clearly and completely described below. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments in this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0016] This embodiment provides a camera robot trajectory planning method based on voice intent parsing, applied to camera movement and shooting scenarios of a multi-degree-of-freedom camera arm in a studio. The camera arm is a six-degree-of-freedom robotic arm mounted on a linear guide rail, with a cinema lens and a full-frame camera mounted at its end. The zoom ring, focus ring, and aperture ring of the lens are each driven by an independent servo motor. (Refer to...) Figure 1 The method includes steps S1 to S6, which are sequentially connected to achieve the transformation from multimodal camera movement intentions to collaborative control commands.

[0017] In step S1, multimodal commands input by the user are acquired through the interactive terminal. Voice commands are captured by a microphone array, and the original audio is preserved. For example, the user might say, "The camera slowly zooms in from a wide shot to the subject's upper body, then circles around to the subject's side and front, and finally performs a sliding zoom." Gesture trajectories are captured by a depth camera and recorded as a three-dimensional temporal sequence of hand joints, used to express the direction and magnitude of the camera movement. Hand-drawn sketches are captured by the touchscreen, representing the expected path from a top-down perspective. Reference video frames are video clips uploaded by the user. Multimodal commands are required to include at least two of the above forms because: a single voice command is insufficient to describe the geometric shape of a spatial path, and a single sketch is insufficient to express the rhythm and framing changes of the camera movement. With two or more modalities complementing each other, the camera movement intention has sufficient constraints in both the spatial and temporal dimensions, thus reducing the quantification error in step S2. This eliminates the need to repeatedly ask the user for supplementary information, reducing the number of interaction rounds.

[0018] Site layout parameters include the three-dimensional dimensions of the shooting area, the position and envelope of fixed obstacles, the area occupied by lighting rigs and teleprompters, and the activity area of ​​the target subject. Intelligent camera robot device parameters include robot kinematic parameters and lens optical feasible domain. Robot kinematic parameters include joint travel, maximum angular velocity, maximum angular acceleration, maximum end effector linear velocity, and maximum linear acceleration; lens optical feasible domain includes focal length range, zoom rate limit, focus rate limit, and aperture adjustment range. In one example, the focal length range is 24mm to 200mm, the zoom rate limit is 30mm / s, the focus distance range is 0.8m to infinity, the focus rate limit is calculated based on the focus ring rotation speed, and the aperture adjustment range is F2.8 to F22. Obtaining and incorporating the lens's optical feasible region into the planning input during the input phase is crucial because both the joint solution in step S4 and the incremental redistribution in step S5 use the optical feasible region as a boundary condition. Without this boundary condition, the resulting sequence of lens optical parameters might exceed the lens's physical capabilities, leading to phenomena such as zoom pauses and repetitive focus searches, resulting in uncontrolled imaging. Replanning would then require interrupting the shooting process. By pre-setting the optical feasible region as a planning constraint, all output optical parameter sequences fall within the lens's capabilities, preventing out-of-bounds commands during execution and ensuring the continuity of the shooting process.

[0019] In step S2, the multimodal instructions are input into the multimodal large model for cross-modal fusion parsing. The multimodal large model encodes the speech, sketches, gestures, and reference video frames separately, then fuses them within a unified semantic space to obtain the camera movement intention. The camera movement intention includes the target subject, the context of shot changes, the direction of movement, rhythm descriptions, and identifiers of special camera movement techniques. After obtaining the camera movement intention, this embodiment does not stop at the intention representation but further quantifies the camera movement intention into a sequence of visual effect constraints arranged along a unified timeline, referring to… Figure 2 The generation of this sequence includes steps S21 to S23.

[0020] In step S21, the multimodal large model parses the camera movement intention into multiple keyframe effect descriptions arranged in the shooting sequence. Each keyframe effect description includes the target subject's image size percentage, the target subject's position in the frame, and the background perspective magnification at the corresponding moment. Taking the aforementioned instruction as an example, the model outputs four keyframes: at 0 seconds, the percentage is 0.25, the subject is in the center of the frame, and the background magnification is 1.0; at 3 seconds, the percentage is 0.45, the subject is in the left third of the frame; at 6 seconds, the percentage is 0.65, the subject moves to the side and front; from 6 seconds to 10 seconds is the sliding zoom period, the percentage remains at 0.65, and the background magnification monotonically increases from 1.0 to 1.6. The reason for using keyframe granularity instead of having the model directly output numerical sequences moment by moment is that: when the large model directly generates long numerical sequences, the values ​​at adjacent moments are prone to jitter, and the jitter will be transmitted through constraints as meaningless acceleration and deceleration on the trajectory; the keyframe granularity is consistent with the granularity of human descriptions of camera movement, the model output is stable, and the numerical quality is controllable.

[0021] In step S22, interpolation is performed between the effect descriptions of adjacent keyframes according to a preset effect transition curve to generate a time-by-time sequence of visual effect constraints. The effect transition curves include three types: linear, ease-in / ease-out, and S-shaped. The selection is based on rhythmic descriptive words from the camera movement intention; for example, an ease-in / ease-out curve is used for "slowly zooming in," while a steep S-shaped curve is used for "suddenly zooming in." The interpolation process ensures that the constraint sequence is smooth and continuous in time, preventing the inflection points of the constraints themselves from being transmitted as abrupt changes in trajectory.

[0022] In step S23, when the multimodal instruction includes a reference video frame, the target subject is detected frame by frame in the reference video frame. The sequence of the target subject's imaging size proportion in the reference segment over time is calculated, and the background scale change sequence is calculated by tracking background feature points. The two calculated measured sequences are used as constraint templates. After time axis alignment, the constraint sequence of the interpolated image effect is corrected. The correction method can be to directly replace the constraint values ​​of the corresponding time period or to use weighted fusion. The reason for introducing measured template correction is that the rhythmic details expressed by the user in the reference film, such as the breathing feeling during the zoom-in process and the specific inflection point of the speed change, are difficult to fully describe in words. The transition curve obtained by pure language path parsing can only be approximated. The effect time curve obtained from the measured reference segment retains all the rhythmic information of the original film. Solving the constraint trajectory with the measured curve results in a style reproduction accuracy that is significantly higher than that of the language description path. Users do not need to have any professional terminology to obtain the same camera movement rhythm as the reference film.

[0023] After obtaining the sequence of constraints for the visual effect, it is necessary to explain the significance of image spatial quantities as the constraint carrier. In related technologies, the result of intent parsing is a category label or abstract intent vector. There is no calculable mapping between the label and the robot's executable motion quantities, resulting in large differences in the visuals executed by the same label in different locations and with different lenses, and it is impossible to verify the correctness of the parsing before shooting. This embodiment advances the parsing endpoint from intent representation to visual effect constraints, and all constraint elements are image spatial quantities: the target subject imaging size ratio is defined as the ratio of the subject imaging height to the frame height; the target subject image position is defined as the horizontal and vertical coordinates of the subject's bounding box center in the normalized image coordinate system; the background perspective magnification is defined as the ratio of the background reference object imaging scale to the imaging scale at the initial moment; the depth of field range is defined as the range of object distances in which the image is clearly imaged. There is a stable correspondence between image spatial quantities and shooting language. For example, a panoramic view corresponds to a subject ratio of 0.20 to 0.35, a half-body shot corresponds to a subject ratio of 0.55 to 0.70, and increased background compression corresponds to a monotonically increasing background perspective magnification.

[0024] The benefits of using image space quantities as constraint carriers are reflected in three aspects: First, constraint values ​​can be directly measured from virtual rendered images or real-world footage, and whether the camera movement intention is correctly executed can be quantitatively verified, establishing a verifiable standard between the user's expectations of the shooting results and the system output; Second, there is a clear optical mapping relationship between image space quantities and the robot's end-effector pose and lens optical parameters, which can be used as constraints in the optimization problem to directly participate in the solution of step S4, and the intention information is not lost in the planning stage; Third, the constraints are arranged in sequence along a unified time axis, and the rhythm information of the camera movement (slow at first and then fast, uniform speed around, etc.) is completely preserved at the trajectory level, avoiding the rigid rhythm caused by template-based execution.

[0025] In step S3, a three-dimensional virtual scene is created based on the site layout parameters, referring to... Figure 3 In a 3D virtual scene, a digital twin model of the intelligent camera robot is set up based on the robot's kinematic parameters. The digital twin model has the same link dimensions, joint strokes, and end effector relationships as the real robot, and the spatial positions of the target subject and background reference objects are calibrated. The position of the target subject is obtained by a positioning system or visual detection within the venue; the background reference objects are static objects located behind the subject and visible in the frame, such as a studio backdrop or set pillars. The reason for explicitly calibrating the background reference objects is that the calculation of background perspective magnification depends on the distance from the end effector pose to the background reference object. If there are no calibrated background reference objects in the scene, background effect constraints cannot participate in the solution of step S4, and camera movements that depend on changes in background scale, such as zooming, will degenerate into ordinary push-pull shots, and the spatial compression aspect of the camera movement intention cannot be realized. After calibration, the distance between the subject and the background in any end effector pose can be directly calculated in the virtual scene, providing geometric input for the imaging projection model.

[0026] In step S4, the imaging projection model is first established, referring to... Figure 3 Based on perspective projection, the image height of the target subject is directly proportional to the lens focal length and inversely proportional to the distance from the end point to the target subject. Let the actual height of the subject be H, the focal length be f, the distance from the end point to the subject be d, and the frame height be S. Then, the image size ratio R of the target subject satisfies the condition that R equals f multiplied by H divided by d divided by S. From the above formula, it can be seen that the same ratio change can be achieved by either a change in focal length or a radial displacement along the visual axis. The focal length and object distance form a complementary mapping on the image size ratio. This complementary mapping relationship is the physical basis for the incremental redistribution of image effects in step S5. The background perspective magnification is expressed as the ratio of the distance from the end point to the subject to the distance from the end point to the background reference object over time. When the subject ratio remains constant while the end point moves along the visual axis, the background image scale changes according to the ratio of focal length to background distance, and the sense of spatial compression changes accordingly. The depth of field range is determined by the allowable circle of confusion diameter c, aperture value N, focal length f, and focusing distance L. The foreground depth of field is approximately equal to N x c x L squared divided by (f squared plus N x c x L), and the background depth of field is approximately equal to N x c x L squared divided by (f squared minus N x c x L). The aperture sequence is obtained by inverse solving of the depth of field constraints.

[0027] The input to the imaging projection model is the end-effector pose and lens optical parameters, and the output is image spatial quantities: the subject proportion channel is established according to the perspective projection relationship; the subject position channel is obtained by projecting the subject's azimuth angle in the camera coordinate system through a pinhole model; the background perspective magnification channel is obtained by the ratio of the focal length to the background distance relative to the initial moment; the depth of field channel is obtained by calculating the relationship between the foreground depth of field and the background depth of field. The correspondence between the image size, the nominal focal length scale of the lens, and the actual focal length in the model is obtained through offline calibration, and the calibration data is embedded in the model in the form of a lookup table to ensure mapping accuracy.

[0028] A joint solution is then performed. The end-effector pose sequence and lens optical parameter sequence are used as joint optimization variables, with the mapping relationship formed by the image effect constraint sequence through the imaging projection model as the constraint condition. Simultaneously, robot kinematic constraints, lens optical feasible region constraints, and collision-free initial scene constraints are superimposed. The objective function is the weighted minimization of three terms: trajectory curvature change rate, focal length change rate, and focus distance change rate. The trajectory curvature change rate term ensures smooth robot movement and avoids image shake; the focal length change rate term suppresses meaningless reciprocating zooming; and the focus distance change rate term suppresses focus breathing jitter. These three terms work together to select the smoothest visual solution while satisfying the image effect requirements.

[0029] The joint solution employs the direct collocation method, discretizing the continuous-time joint optimization problem into several collocations along the time axis. For example, a 10-second camera movement is discretized into 200 collocations. The optimization variables at each collocation include the end position, end pose, focal length, focusing distance, and aperture. Adjacent collocations are connected by a cubic polynomial and a continuity constraint is applied. In the sequence of image effect constraints, the proportion of the target subject's image size uses interval constraints instead of strict equations, with a tolerance of ±0.02. The tolerance for the subject's position is 3% of the frame width. The coupling relationship during the zoom-sliding period uses equation constraints. The reason for using interval constraints for proportion is that strict equations make the feasible region too narrow, and the solution is prone to failure when there are many collocations. On the other hand, when the proportion fluctuation is less than 0.02, it is difficult for the viewer to perceive. The tolerance is set according to the difference that the human eye can just perceive, thus balancing the success rate of the solution and the visual appeal of the image. The initial values ​​are generated by inverse geometry. That is, the dynamic and optical speed limitations are ignored first, and the coarse pose and coarse focal length that satisfy the first and last effects are obtained by inverse geometry according to the effect description of each key frame. Then, the coarse solution is used as the initial value to call the sequence quadratic programming solver to solve the complete problem. When the number of points is 200, a single solution can be completed in a few seconds, which meets the time requirements of the pre-shooting planning.

[0030] In a numerical example, the frame height is 24mm, the subject height is 1.75m, and the constraint requires the subject's proportion to smoothly increase from 0.25 to 0.65 within 8 seconds. If a fixed-position pure zoom solution is used, the focal length needs to increase from 35mm to 91mm; if a fixed-focal-length pure movement solution is used, the end effector needs to move forward from 10.2m to 3.9m, which is a large forward movement and is prone to conflict with foreground props. The solution given by the joint solution is a coordinated combination of moving the end effector forward by 0.7m and increasing the focal length from 35mm to 85mm, where both the robot's motion and zoom are in their respective low-load ranges.

[0031] The above comparison illustrates that the reason for incorporating the end pose and lens optical parameters into the same optimization problem is that: once the trajectory determined in the separate planning touches the boundary of the workspace, the focal length can only be passively compensated or even exceeds the boundary, and there is no negotiation mechanism between the two stages; joint solution weighs the motion cost and optical cost within the same problem, resulting in a larger feasible solution space and a higher constraint satisfaction rate.

[0032] In some shooting tasks, the image effect constraint sequence also includes depth-of-field constraints. For example, in an interview scene, the interviewee needs to be sharp throughout while the background remains blurred. The corresponding depth-of-field constraint is that the sharp object distance range covers the area in front of and behind the subject within 0.3m and does not cover background reference objects. During the solution process, the depth-of-field constraint is transformed into a range limit on the aperture value through the calculation relationship between foreground and background depth of field, and participates in the optimization along with focal length and focus distance. The reason for including depth of field in the joint solution instead of manually adjusting the aperture on-site is that: the focal length and focus distance change continuously during camera movement, and the depth-of-field range corresponding to the same aperture value varies greatly at different times. Relying solely on a fixed aperture cannot maintain a blurred relationship throughout the entire camera movement. The aperture sequence output by the joint solution changes synchronously with the focal length and focus distance, and the degree of background blur remains consistent throughout the change of framing, resulting in a unified image style.

[0033] After the solution is completed, the aperture sequence is also checked for exposure linkage. The aperture change is synchronously converted into ISO or shutter speed compensation to ensure that the image brightness does not jump during the camera movement.

[0034] In step S5, kinematic feasibility verification and collision detection are performed on the initial motion trajectory, and optical feasible region verification is performed on the lens optical parameter sequence. Kinematic feasibility verification involves solving the inverse kinematics of the end-effector pose at each moment of the trajectory and checking joint travel and velocity / acceleration limits; collision detection checks the minimum distance between the robot body, end-effector, and scene objects in the 3D virtual scene; optical feasible region verification checks whether the focal length, zoom rate, focusing distance, focusing rate, and aperture fall within the feasible region moment by moment. (Refer to...) Figure 4If any verification fails, based on the complementary mapping relationship between the end pose and lens optical parameters in the imaging projection model for the same image spatial quantity, while keeping the image effect constraint sequence unchanged, the image effect increment is redistributed between the motion trajectory and the lens optical parameter sequence and the solution is resolved. The redistribution includes three branches executed sequentially.

[0035] Branch 1 handles focal length boundary issues. When the focal length sequence reaches the focal length range boundary or zoom rate limit at any time, the change in subject proportion that cannot be achieved by focal length changes is converted into a radial displacement increment along the target subject's visual axis, based on the imaging projection model. This increment is then superimposed onto the motion trajectory for the corresponding time period and re-performed for kinematic feasibility verification and collision detection. In one example, the constraint requires the subject proportion to increase from 0.30 to 0.70 within 2 seconds. Based on the current camera position, this requires the focal length to increase from 60mm to 140mm, resulting in a zoom rate of 40mm / s, exceeding the 30mm / s limit. The excess proportion increment is converted into a 0.65m forward movement of the end effector using complementary mapping, which is then handled by the robot's motion. The focal length only needs to increase to 118mm, and the zoom rate returns to within the limit. This conversion utilizes the complementarity of focal length and object distance in terms of proportion, ensuring that the image effect constraint sequence remains unchanged throughout, and the viewer's perception of the change in field of view is completely consistent with the original intent.

[0036] Branch Two handles infeasible trajectories. When the initial trajectory is deemed infeasible by collision detection or workspace verification at any time period, the change in the subject's proportion corresponding to the reduced radial displacement is converted into a focal length increment. This increment is then superimposed onto the focal length sequence for the corresponding time period and the optical feasibility domain is re-verified. In one example, the planned forward path interferes with the on-site track vehicle, reducing the forward movement by 0.8m. The proportion gap, after conversion, corresponds to a focal length increment of 12mm. After superposition, the focal length sequence remains within the feasible domain, and the image quality is unaffected by the obstacle. Branch Two ensures that obstacle avoidance no longer comes at the expense of framing; the robot makes room while the lens compensates for the image quality, and obstacle avoidance and image quality are no longer mutually exclusive.

[0037] Branch 3 handles situations where both sides are not feasible. When both radial displacement increment and focal length increment are not feasible in the same time period, the constraint boundary of background perspective magnification in the image effect constraint sequence for that time period is relaxed within a preset tolerance range. The tolerance range is preset to 10% of the constraint value, while keeping the target subject's image size proportion constraint unchanged. The order of trade-offs is set according to perceptual sensitivity: the subject's proportion directly determines the shot size and narrative focus; deviations in proportion will change the semantics of the shot, which is keenly perceived by the viewer; background perspective magnification is an effect at the atmospheric level, and small deviations do not change the semantics of the shot. In situations where only limited concessions can be made, priority is given to preserving the semantic level effect, while concessions are made at the atmospheric level, resulting in the least overall loss of image effect.

[0038] After executing the three branches sequentially, the solution is recalculated until an executable motion trajectory and an executable lens optical parameter sequence are obtained. The image effect constraint sequence is used as an invariant in the redistribution process because: related technologies directly smooth the trajectory around obstacles after verification failure, and the image effect passively deteriorates with the trajectory change and the degree of degradation is uncontrollable; this embodiment regards the image effect as the primary object to be maintained in the planning. When the equipment capacity is insufficient, the load is first transferred between two physical paths that can achieve the same image effect. Only when it is really impossible to maintain the image effect will a limited concession be made in the less sensitive effect dimension, thus significantly improving the controllability of the imaging effect.

[0039] In this embodiment, when the camera movement intention includes a zoom slider indicator, the image effect constraint sequence is set as follows: the proportion of the target subject's image size remains constant during the zoom slider period, and the background perspective magnification changes monotonically in a specified direction during the zoom slider period. (Refer to...) Figure 5 According to the imaging projection model, the constant proportion of the subject is equivalent to the constant ratio of focal length to distance from the end to the subject. Based on the above equivalence relationship, the coupled equation constraint of the constant ratio of focal length to object distance is applied in the joint solution of step S4, and the focusing distance sequence is made to change synchronously with the distance from the end to the subject to ensure that the subject is in focus throughout the entire process.

[0040] In a numerical example, the end point moves back from 3m to 6m along the visual axis from the subject, and the coupling constraint drives the focal length to increase synchronously from 35mm to 70mm, while the ratio remains constant and the proportion of the subject remains unchanged; the background plate is 10m away from the subject, and the background distance changes from 13m to 16m. The ratio of focal length to background distance increases from 2.69 to 4.38, and the background is magnified by about 1.63 times, with the spatial compression effect presented as intended.

[0041] In actual lenses, the field of view shifts when the focusing distance changes, and the equivalent focal length deviates from the nominal value. This phenomenon is known as focus breathing. Without intervention, although the ratio of the focal length scale to the object distance remains nominally constant under coupling constraints, the actual image proportion will still drift, disrupting the core visual impression of an unchanged subject size.

[0042] To eliminate drift, the lens's focus breathing characteristic curve is pre-calibrated offline: the lens is placed on a calibration frame, and the calibration board is photographed at different focus distances, with the actual field of view measured. A curve corresponding to the focus distance and equivalent focal length is fitted. During the planning phase, the field of view offset caused by changes in focus distance is calculated based on the breathing characteristic curve. This offset is converted into a focal length compensation amount and superimposed on the focal length sequence. In one example, when the focus distance of an 85mm lens is moved from 5m to 2.5m, the equivalent focal length increases to 87.6mm, the field of view decreases by approximately 3%, and the compensation amount is -2.5mm. After compensation, the fluctuation in the subject's proportion during the sliding zoom period decreases from approximately 3% without compensation to less than 0.5%. Related technologies achieve sliding zoom through image feedback, continuously measuring the subject's size and adjusting the focal length during execution, resulting in a delay of one to several frames. Fluctuations in the subject's size are visible during rapid zooming. This embodiment pre-superimposes the breathing compensation amount into the focal length sequence during the planning phase, which is a feedforward compensation, unaffected by feedback delay, and remains stable even under rapid zooming.

[0043] In step S6, the executable motion trajectory and the executable lens optical parameter sequence are aligned along a unified time axis, interpolated according to a control period (e.g., 4ms), and converted into robot joint space commands and zoom, focus, and aperture motor commands. After attaching a timestamp of the same source, these commands are sent to the robot motion controller and lens drive mechanism, respectively. The reason for strictly aligning the time axis is that motion commands and optical commands belong to two separate controllers. If the clocks are not synchronized, the focal length change will lead or lag behind the end-effector displacement, causing instantaneous fluctuations in the subject's imaging proportion, especially noticeable during the zoom-sliding period. By using timestamps of the same source and a unified interpolation period, the motion channel and optical channel remain synchronized at the millisecond level, ensuring the imaging effect matches the planned result.

[0044] Following step S6, there is also a step to maintain the image effect during execution. During the execution of the executable motion trajectory, the camera captures images in real time at a frame rate of 50Hz. The actual image size ratio and actual image position of the target subject are obtained through target detection and tracking. These are compared with the constraint values ​​at the corresponding moments in the image effect constraint sequence to obtain the effect deviation. The sources of effect deviation include the on-site movement of the target subject, robot trajectory tracking errors, and site vibrations.

[0045] The effect deviation is divided into frequencies using the motion response bandwidth of the intelligent camera robot as the dividing frequency. The motion response bandwidth is obtained through offline identification; for example, the position loop bandwidth is measured to be 1.5Hz after applying a frequency sweep command to the end effector. During execution, the effect deviation is processed by a first-order low-pass filter with a cutoff frequency of 1.5Hz. The filtered output is the low-frequency deviation component, and the high-frequency deviation component is obtained by subtracting the low-frequency component from the original deviation. Based on the imaging projection model, the high-frequency deviation component is converted into lens optical parameter correction, the proportion of high-frequency deviation is converted into focal length fine-tuning, and the high-frequency deviation of the image position is converted into the offset of the optical image stabilization unit, which is then sent to the lens drive mechanism for execution. The low-frequency deviation component is converted into end effector pose correction, which is sent to the robot motion controller and superimposed on the original trajectory for execution.

[0046] The rationale for frequency division and channel-based correction based on motion response bandwidth is as follows: The robot itself has a large inertia; forcibly tracking deviation components exceeding its own bandwidth not only fails to converge but also induces mechanical vibration, which is then transmitted to the image. Lens motors have small travel, low inertia, and a response bandwidth typically above 20Hz, making them suitable for absorbing small-amplitude, high-frequency deviations. However, their limited travel prevents them from absorbing large low-frequency deviations caused by the subject's wide-range movement. After frequency division, each channel operates within its capability range. High-frequency jitter is instantly smoothed by the optical channel, while trend-based deviations are calmly corrected by the motion channel, ensuring both deviation convergence speed and steady-state accuracy. In a comparative example, the target subject moves slowly at a rhythm of 0.5Hz with a small sway of approximately 3Hz. When corrected using only the robot pose single-channel correction, the residual fluctuation of the subject's proportion is about 1.8%. After frequency division and dual-channel correction, the residual fluctuation is reduced to 0.4%, and the actual imaging effect stably converges to the image effect constraint sequence.

[0047] The two correction channels also incorporate limiting and anti-collision mechanisms during execution: the focal length fine-tuning of the optical channel is limited to 2% of the planned focal length value, and the optical image stabilization offset is limited to 80% of the stabilization unit's travel; any amount exceeding these limits is transferred to the low-frequency channel for cumulative processing. The pose correction amount of the motion channel is filtered with jerk limiting before being superimposed, ensuring that the correction action does not produce visible stutters in the image. When the focal length fine-tuning of the high-frequency channel and the coupling constraint of the zoom period coexist, the fine-tuning amount is applied to the final instruction after the focal length compensation amount, and the actual applied fine-tuning amount is written back into the calculation stage of the imaging projection model, preventing the two focal length adjustments from canceling each other out. These mechanisms ensure stable operation of the dual-channel correction during long-term execution, preventing back-and-forth pulling between channels.

[0048] It should be noted that in the above embodiments, the multimodal large model can be any pre-trained model with cross-modal understanding capabilities; the nonlinear programming solver can be replaced by the sequential quadratic programming method with the interior point method; the first-order low-pass filter can be replaced by the second-order Butterworth filter; and background feature point tracking can use ORB features or optical flow methods. All of the above substitutions do not depart from the concept of this invention. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this invention should be included within the protection scope of this invention.

Claims

1. A camera robot trajectory planning method based on voice intent parsing, characterized in that, Includes the following steps: Step S1: Obtain the multimodal instructions, site layout parameters, and intelligent camera robot device parameters input by the user. The multimodal instructions include a combination of at least two of the following: voice instructions, gesture trajectories, hand-drawn sketches, and reference video frames. The intelligent camera robot device parameters include robot kinematic parameters and lens optical feasible region. The lens optical feasible region includes focal length range, zoom rate limit, focus rate limit, and aperture adjustment range. Step S2: Input the multimodal instruction into the multimodal large model for cross-modal fusion analysis to obtain the camera movement intention, and quantify the camera movement intention into a sequence of image effect constraints arranged along a unified time axis. Each constraint element in the image effect constraint sequence describes the imaging effect that the image should present at the corresponding moment using image space quantities. The image space quantities include at least two of the following: target subject imaging size ratio, target subject image position, background perspective magnification, and depth of field range. Step S3: Establish a three-dimensional virtual scene based on the site layout parameters, set up a digital twin model of the intelligent camera robot in the three-dimensional virtual scene, and mark the spatial positions of the target subject and background reference objects; Step S4: Establish an imaging projection model, which maps the pose of the intelligent camera robot end effector and the lens optical parameters into the image space quantity; using the intelligent camera robot end effector pose sequence and the lens optical parameter sequence as joint optimization variables, and the mapping relationship formed by the image effect constraint sequence through the imaging projection model as constraint conditions, jointly solve in the three-dimensional virtual scene to obtain the initial motion trajectory and the lens optical parameter sequence synchronized with the initial motion trajectory in time. The lens optical parameter sequence includes the focal length sequence, the focus distance sequence, and the aperture sequence. Step S5: Perform kinematic feasibility verification and collision detection on the initial motion trajectory, and perform optical feasibility domain verification on the lens optical parameter sequence; when any verification fails, based on the complementary mapping relationship between the end pose and the lens optical parameters on the same image space quantity in the imaging projection model, while keeping the image effect constraint sequence unchanged, reallocate the image effect increment between the motion trajectory and the lens optical parameter sequence and solve it again to obtain the executable motion trajectory and the executable lens optical parameter sequence; Step S6: Align the executable motion trajectory and the executable lens optical parameter sequence along a unified time axis, and convert them into collaborative control commands to be output to the intelligent camera robot actuator.

2. The camera robot trajectory planning method based on voice intent parsing according to claim 1, characterized in that, In step S2, quantifying the camera movement intention into a sequence of visual effect constraints includes: Step S21: The multimodal large model parses the camera movement intention into multiple keyframe effect descriptions arranged in the shooting sequence. Each keyframe effect description includes the values ​​of the target subject imaging size ratio, the target subject image position, and the background perspective magnification at the corresponding moment. Step S22: Interpolate between adjacent keyframe effect descriptions according to a preset effect transition curve to generate a time-by-time image effect constraint sequence; Step S23: When the multimodal instruction includes a reference video frame, the target subject is detected frame by frame in the reference video frame, the target subject imaging size ratio sequence is calculated, and the background feature points are tracked to calculate the background scale change sequence. The calculated target subject imaging size ratio sequence and background scale change sequence are used as constraint templates to correct the interpolated image effect constraint sequence.

3. The camera robot trajectory planning method based on voice intent parsing according to claim 1, characterized in that, In step S4, the imaging projection model is established as follows: based on the perspective projection relationship, the proportion of the target body's imaging size is expressed as a mapping relationship between the lens focal length, the distance from the end pose to the target body, and the actual size of the target body. Background perspective magnification is expressed as the ratio of the distance from the end pose to the target subject to the distance from the end pose to the background reference object over time; depth of field is expressed as the relationship between foreground depth of field and background depth of field, which is determined by focal length, aperture, and focus distance. During the joint solution, the equality constraints or interval constraints corresponding to each image spatial quantity in the image effect constraint sequence are used as hard constraints. The objective function is to minimize the weighted sum of the trajectory curvature change rate, focal length change rate, and focus distance change rate. The direct collocation method is used to discretize the continuous-time joint optimization problem into a nonlinear programming problem for solution, and the end pose sequence and lens optical parameter sequence are obtained simultaneously.

4. The camera robot trajectory planning method based on voice intent parsing according to claim 1, characterized in that, In step S5, the reallocation of image effect increments between the motion trajectory and the lens optical parameter sequence includes: When the focal length sequence touches the focal length range boundary or zoom rate limit in the lens optical feasible domain at any time period, according to the imaging projection model, the change in the proportion of the target subject's imaging size that cannot be achieved by focal length change is converted into the radial displacement increment of the end pose along the visual axis of the target subject. The radial displacement increment is then superimposed on the motion trajectory of the corresponding time period and kinematic feasibility verification and collision detection are performed again. When the initial motion trajectory is determined to be infeasible by collision detection or workspace verification at any time period, the change in the proportion of the target body imaging size corresponding to the reduced radial displacement is converted into focal length increment according to the imaging projection model. The focal length increment is then superimposed on the focal length sequence of the corresponding time period and the optical feasibility domain verification is performed again. When both radial displacement increment and focal length increment are not feasible in the same time period, the constraint boundary of background perspective magnification in the image effect constraint sequence of that time period is relaxed within the preset tolerance range, while keeping the target subject imaging size ratio constraint unchanged.

5. The camera robot trajectory planning method based on voice intent parsing according to claim 3, characterized in that, When the camera movement intention includes a zoom slider indicator, the image effect constraint sequence is set as follows: the proportion of the target subject's image size remains constant during the zoom slider period, and the background perspective magnification changes monotonically in a specified direction during the zoom slider period. In the joint solution of step S4, a coupling equation constraint is applied to keep the ratio of the lens focal length to the distance from the end pose to the target body constant during the sliding zoom period, and the focus distance sequence is made to change synchronously with the distance from the end pose to the target body. Based on the focus breathing characteristic curve of the lens, the field of view shift caused by the change in focus distance is calculated. The field of view shift is converted into a focal length compensation amount and superimposed on the focal length sequence so that the proportion of the target subject's image size does not drift due to focus breathing during the zoom period.

6. The camera robot trajectory planning method based on voice intent parsing according to claim 1, characterized in that, Following step S6, a step for maintaining the visual effects during execution is also included: During the execution of the executable motion trajectory, the captured images are acquired in real time, the actual imaging size ratio and actual image position of the target subject are detected, and the effect deviation between the actual value and the corresponding time constraint value in the image effect constraint sequence is calculated. Using the motion response bandwidth of the intelligent camera robot as the dividing frequency, the effect deviation is separated into high-frequency deviation components and low-frequency deviation components. Based on the imaging projection model, the high-frequency deviation components are converted into lens optical parameter correction values ​​and sent to the lens driving mechanism, and the low-frequency deviation components are converted into end-effector pose correction values ​​and sent to the robot motion controller, so that the actual imaging effect during the execution process converges to the image effect constraint sequence.

Images