Photographing method and system for ground omnidirectional movement with autonomous control of multi-axis angle

Abstract

The application discloses a kind of ground omnidirectional movement photography methods and systems of self-control multi-axis angle, which comprises: based on the history motion skeleton sequence, audio data and preset visual composition style label obtained, generate virtual motion trajectory;Based on virtual motion trajectory, attitude is solved, and the motion coordinates of omnidirectional movement device, three-axis mechanical arm and three-axis gimbal camera are obtained;Based on the motion coordinates of omnidirectional movement device, three-axis mechanical arm and three-axis gimbal camera, the current motion image of target object is collected;Based on current motion image and audio data, virtual motion trajectory is corrected, and final motion trajectory is obtained;Through final motion trajectory control omnidirectional movement device, three-axis mechanical arm and three-axis gimbal camera, photography collection is carried out to target object.The above scheme can greatly reduce the threshold of high-quality video creation, realize high-precision and intelligent copying of excellent visual works.

Description

Technical Field

[0001] This application relates to the field of intelligent robots and automated film and television shooting technology, specifically to a photography method and system for autonomously controlling multi-axis angle omnidirectional ground movement. Background Technology

[0002] With the rise of short video platforms and the live streaming industry, users' demand for high-quality, cinematic (camera movement) shooting is increasing. Especially in scenarios such as dance filming and influencer store visits, complex camera movements (such as panning, zooming, and tracking) often require professional cinematographers to operate complex tracks, cranes, or handheld gimbals. Currently, existing automated shooting solutions mainly fall into the following categories: Automatic follow gimbal: These products can usually only achieve horizontal or vertical rotation in place, and cannot achieve spatial displacement, resulting in limited composition.

[0003] Industrial-grade photography robots: Although they can achieve extremely high-precision trajectory movements, they are bulky, expensive, and have an extremely complex programming process, making it difficult for ordinary video creators to master their operating logic.

[0004] Consumer-grade photography trolleys: Existing photography trolleys typically only have the ability to move the chassis and lack the flexible coordination of multi-axis robotic arms, making it impossible to achieve drastic changes in elevation or pitch.

[0005] Therefore, how to lower the creative threshold of complex camera movements and achieve high-precision and intelligent replication of the camera movement trajectories of excellent visual works is a technical problem that urgently needs to be solved in this field. Summary of the Invention

[0006] This application aims to provide a photographic method and system for autonomously controlling multi-axis angle omnidirectional ground movement, which can significantly lower the threshold for high-quality video creation and achieve high-precision and intelligent replication of the camera movement trajectory of excellent visual works.

[0007] The technical solution of this application is implemented as follows: In a first aspect, embodiments of this application provide a method for autonomously controlling multi-axis angle omnidirectional ground movement during photography, the method comprising: Obtain the historical motion skeleton sequence, audio data, and preset visual composition style tags of the target object; and generate a virtual motion trajectory based on the historical motion skeleton sequence, the audio data, and the preset visual composition style tags; Based on the virtual motion trajectory, attitude calculation is performed to obtain the motion coordinates of the omnidirectional mobile device, the three-axis robotic arm, and the three-axis gimbal camera. Based on the motion coordinates of the omnidirectional mobile device, the three-axis robotic arm, and the three-axis gimbal camera, the current motion image of the target object is acquired. Based on the current motion image and the audio data, the virtual motion trajectory is corrected to obtain the final motion trajectory; The omnidirectional moving device, three-axis robotic arm, and three-axis gimbal camera are controlled by the final motion trajectory to capture photographic images of the target object.

[0008] In the above scheme, generating a virtual motion trajectory based on the historical motion skeleton sequence, the audio data, and the preset visual composition style tags includes: Motion features are extracted from the historical motion skeleton sequence to obtain motion features; Audio features are extracted from the audio data to obtain beat features; The virtual motion trajectory is generated using the motion features, the beat features, and the preset visual composition style tags.

[0009] In the above scheme, the step of performing attitude calculation based on the virtual motion trajectory to obtain the motion coordinates of the omnidirectional mobile device, the three-axis robotic arm, and the three-axis gimbal camera includes: The virtual motion trajectory is calculated using a preset spatial kinematics model and a solver to obtain the initial motion coordinates of the omnidirectional mobile device, the three-axis robotic arm, and the three-axis gimbal camera; wherein the preset spatial kinematics model is constructed based on the omnidirectional mobile device, the three-axis robotic arm, and the three-axis gimbal camera; The initial motion coordinates of the omnidirectional moving device, the three-axis robotic arm, and the three-axis gimbal camera are geometrically smoothed to obtain the smoothed initial motion coordinates of the omnidirectional moving device, the three-axis robotic arm, and the three-axis gimbal camera. The smoothed motion coordinates are subjected to time-series filtering to obtain the motion coordinates of the omnidirectional moving device, the three-axis robotic arm, and the three-axis gimbal camera.

[0010] In the above scheme, the step of correcting the virtual motion trajectory based on the current motion image and the audio data to obtain the final motion trajectory includes: Based on the current motion image and the virtual motion trajectory, temporal prediction and trajectory compensation are performed to obtain the corrected motion trajectory; Based on the audio data, the corrected motion trajectory is further corrected to obtain the final motion trajectory.

[0011] In the above scheme, the step of performing temporal prediction and trajectory compensation based on the current motion image and the virtual motion trajectory to obtain the corrected motion trajectory includes: Target recognition is performed on the current motion image to obtain the full-body key points of the target object in the current motion image, as well as the coordinates of its geometric center; Based on the coordinates of the key points and geometric center of the whole body, the target object is temporally predicted to determine the spatial motion trend and displacement vector; Based on the spatial motion trend and displacement vector, the virtual motion trajectory is subjected to deviation calculation and high-frequency interpolation to obtain the corrected motion trajectory.

[0012] In the above scheme, the step of calculating the deviation and performing high-frequency interpolation on the virtual motion trajectory based on the spatial motion trend and displacement vector to obtain the corrected motion trajectory includes: Based on the spatial motion trend and displacement vector, determine the real-time coordinates of the target object; Based on the real-time coordinates of the target object and the predetermined center coordinates, a composition deviation vector is calculated; wherein, the center coordinates represent the exact center of the image or the golden ratio point. Acquire the real-time depth information of the target object and the camera's intrinsic parameter matrix; and determine the three-dimensional physical space deviation vector based on the composition deviation vector, real-time depth information, and intrinsic parameter matrix; The six-dimensional spatial incremental correction vector is determined by using the preset dynamic safety frame window threshold area and the three-dimensional physical space deviation vector. Based on the six-dimensional spatial incremental correction vector, the virtual motion trajectory is corrected to obtain the corrected motion trajectory.

[0013] In the above scheme, the step of performing a second correction on the modified motion trajectory based on the audio data to obtain the final motion trajectory includes: Based on the audio data, the weak beat phase and the strong beat phase are determined; When the audio data is in a weak beat phase, the first motion trajectory corresponding to the weak beat phase is accelerated in the corrected motion trajectory to determine the target motion trajectory corresponding to the weak beat phase. When the audio data is in a strong beat phase, a second motion trajectory corresponding to the strong beat phase is determined from the corrected motion trajectory. The final motion trajectory is determined based on the target motion trajectory corresponding to the weak beat phase and the second motion trajectory corresponding to the strong beat phase.

[0014] Secondly, embodiments of this application provide a photographic system for autonomously controlling multi-axis angle omnidirectional ground movement. This system includes: an acquisition module, an attitude calculation module, a data acquisition module, and a correction module; wherein... The acquisition module is used to acquire the historical motion skeleton sequence, audio data, and preset visual composition style tags of the target object; and to generate a virtual motion trajectory based on the historical motion skeleton sequence, the audio data, and the preset visual composition style tags. The attitude calculation module is used to perform attitude calculation based on the virtual motion trajectory to obtain the motion coordinates of the omnidirectional mobile device, the three-axis robotic arm and the three-axis gimbal camera. The acquisition module is used to acquire the current motion image of the target object based on the motion coordinates of the omnidirectional mobile device, the three-axis robotic arm and the three-axis gimbal camera. The correction module is used to correct the virtual motion trajectory based on the current motion image and the audio data to obtain the final motion trajectory; The acquisition module is also used to control the omnidirectional moving device, the three-axis robotic arm, and the three-axis gimbal camera through the final motion trajectory to capture photographs of the target object.

[0015] Thirdly, embodiments of this application provide a photographic device capable of autonomously controlling multi-axis angle omnidirectional ground movement, comprising: a processor and a memory; wherein, The memory is used to store computer programs; The processor is configured to call and run the computer program from the memory to perform the method as described in the first aspect.

[0016] Fourthly, embodiments of this application provide a computer-readable storage medium storing executable instructions for causing a processor to perform the method described in the first aspect.

[0017] This application provides a method and system for autonomously controlling multi-axis omnidirectional ground movement for photography. The method includes: acquiring a historical motion skeleton sequence, audio data, and preset visual composition style tags for a target object; generating a virtual motion trajectory based on the historical motion skeleton sequence, the audio data, and the preset visual composition style tags; performing attitude calculation based on the virtual motion trajectory to obtain the motion coordinates of an omnidirectional mobile device, a three-axis robotic arm, and a three-axis gimbal camera; acquiring a current motion image of the target object based on the motion coordinates of the omnidirectional mobile device, the three-axis robotic arm, and the three-axis gimbal camera; correcting the virtual motion trajectory based on the current motion image and the audio data to obtain a final motion trajectory; and controlling the omnidirectional mobile device, the three-axis robotic arm, and the three-axis gimbal camera through the final motion trajectory to capture photographic images of the target object. The aforementioned solution integrates historical motion skeleton sequences, audio data, and preset visual composition style tags in a multimodal manner to generate virtual motion trajectories that are cinematic and rhythmically consistent with the music. Users do not need to master complex robot trajectory programming or professional camera movement parameter expressions; the device can automatically understand music and dance, generating and replicating the camera movement logic of blockbuster films, significantly improving the production efficiency of short videos and film production pipelines. Based on the virtual motion trajectory, posture calculation and correction are performed to obtain the final motion trajectory. Even when faced with complex impromptu steps and positioning errors of the target object, the composition remains rock-solid, greatly increasing the success rate of shooting dance scenes and lowering the threshold for high-quality video creation, achieving high-precision and intelligent replication of the camera movement trajectories of excellent visual works. Attached Figure Description

[0018] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the specification, serve to explain the technical solutions of this application. Obviously, the drawings described below are merely some embodiments of this application, and those skilled in the art can obtain other drawings based on these drawings without any inventive effort.

[0019] The flowcharts shown in the accompanying drawings are merely illustrative and do not necessarily include all content and operations / steps, nor do they necessarily have to be performed in the described order. For example, some operations / steps can be broken down, while others can be combined or partially combined; therefore, the actual execution order may change depending on the specific circumstances.

[0020] Figure 1 A schematic flowchart of an optional ground-based omnidirectional photography method for autonomously controlling multi-axis angles, provided in an embodiment of this application; Figure 2A schematic diagram of a ground-based omnidirectional photographic system that autonomously controls multi-axis angles, as provided in this application embodiment. Figure 1 ; Figure 3 A schematic diagram of a ground-based omnidirectional photographic system that autonomously controls multi-axis angles, as provided in this application embodiment. Figure 2 ; Figure 4 A schematic diagram of the structure of a ground-based omnidirectional photography system that autonomously controls multi-axis angles, as provided in an embodiment of this application; Figure 5 This is a schematic diagram of the structure of a photographic device that can autonomously control multi-axis angles and move omnidirectionally on the ground, as provided in an embodiment of this application. Detailed Implementation

[0021] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the specific technical solutions of this application will be further described in detail below with reference to the accompanying drawings of the embodiments of this application. The following embodiments are used to illustrate this application, but are not intended to limit the scope of this application.

[0022] Unless otherwise defined, all technical and scientific terms used in this application have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains. The terminology used in this application is for the purpose of describing embodiments of this application only and is not intended to be limiting of this application.

[0023] In the following description, references to "some embodiments," "this embodiment," "this application embodiment," and examples, etc., describe a subset of all possible embodiments. However, it is understood that "some embodiments" may be the same subset or different subset of all possible embodiments and may be combined with each other without conflict.

[0024] If the application documents contain similar descriptions such as "first / second", the following explanation shall be added: In the following description, the terms "first / second / third" are used only to distinguish similar objects and do not represent a specific order of objects. It is understood that "first / second / third" may be interchanged in a specific order or sequence where permitted, so that the embodiments of this application described herein can be implemented in an order other than that illustrated or described herein.

[0025] This application provides a method for autonomously controlling multi-axis angle omnidirectional ground movement during photography. Figure 1 This is an optional flowchart illustrating an autonomous, multi-axis, omnidirectional ground-based photographic method provided in an embodiment of this application, which will be combined with... Figure 1 The steps shown are explained.

[0026] S101. Obtain the historical motion skeleton sequence, audio data, and preset visual composition style tags of the target object; and generate a virtual motion trajectory based on the historical motion skeleton sequence, audio data, and preset visual composition style tags.

[0027] In some embodiments of this application, preset visual composition style tags include surround push-pull and low-angle pitch follow, etc.

[0028] In some embodiments of this application, a photography method for autonomously controlling multi-axis angle omnidirectional ground movement is adapted to short video shooting scenarios.

[0029] In some embodiments of this application, a photographic method for autonomously controlling multi-axis angle omnidirectional ground movement is adapted to a photographic system for autonomously controlling multi-axis angle omnidirectional ground movement.

[0030] like Figure 2 and Figure 3 As shown, the autonomous multi-axis angle-controlled omnidirectional ground-based photography system includes an omnidirectional motion device 100, a three-axis robotic arm 200, a three-axis gimbal camera 300, and a control system 400. The omnidirectional motion device 100 includes Mecanum wheels 101, and a depth camera 401 is mounted on the control system 400. At the hardware level, the autonomous multi-axis angle-controlled omnidirectional ground-based photography system is constructed by connecting the omnidirectional motion device 100, the three-axis robotic arm 200, and the three-axis gimbal camera 300 in series to create a spatial motion execution terminal with six degrees of freedom redundancy. The control system 400, as the core control hub, integrates a main controller, a wireless communication module, and the depth camera 401, which provides temporal environmental feature capture. At the software algorithm level, the control system 400 internally constructs a dual-track parallel algorithm architecture consisting of a camera trajectory autonomous generation module and a dynamic image stabilization tracking control module. The deep integration logic of software and hardware is as follows: the control system 400 uses the depth camera 401 as the core perception source to capture multimodal features of the environment in real time. Its internal dual-track algorithm module solves the optimal visual composition trajectory in real time based on the feature data. Subsequently, the control system 400 accurately distributes the high-dimensional control command to the four Mecanum wheels 101 of the omnidirectional mobile device 100, the three active joints of the three-axis robotic arm 200, and the servo driver of the three-axis gimbal camera 300 through a unified spatial kinematics and dynamics inverse solution model. This forms a high-frequency closed-loop integrated control of environmental perception, intelligent decision-making, dynamics calculation, and multi-axis execution terminal response.

[0031] In some embodiments of this application, the historical motion skeleton sequence, audio data, and preset visual composition style tags of the target object are obtained; motion features are extracted from the historical motion skeleton sequence to obtain motion features; audio features are extracted from the audio data to obtain beat features; and a virtual motion trajectory is generated using the motion features, beat features, and preset visual composition style tags.

[0032] For example, taking a dancer as the target object, the control system 400 deploys a multimodal cinematography transformer (MCT). The dancer's historical motion skeleton sequence is acquired via a depth camera 401, or input from an external video source (Dancer PoseTokens). Audio data of the dancer is collected via microphones deployed on a robotic arm. Motion features are extracted from the historical motion skeleton sequence, and time-frequency transformation is performed on the audio data to obtain beat features. The motion features, beat features, and preset visual composition style labels are input into the multimodal cinematography transformer. Internally, the model uses a cross-attention mechanism to calculate the correlation matrix between the dancer's motion trajectory, the timing of musical downbeats, and the compositional intention, automatically predicting the virtual motion trajectory that best matches visual aesthetics and musical rhythm. .

[0033] Understandably, it breaks through the limitations of traditional equipment that manually marks and plans trajectories, and for the first time integrates dancer postures, music rhythms and compositional aesthetics in a multimodal way to generate a spatial 3D motion curve (i.e., virtual motion trajectory) that has a cinematic feel and matches the rhythm of the music.

[0034] S102. Based on the virtual motion trajectory, perform attitude calculation to obtain the motion coordinates of the omnidirectional mobile device, the three-axis robotic arm, and the three-axis gimbal camera.

[0035] In some embodiments of this application, S102 can be implemented by S1021-S1023, as follows: S1021. The virtual motion trajectory is calculated using a preset spatial kinematics model and a solver to obtain the initial motion coordinates of the omnidirectional mobile device, the three-axis robotic arm, and the three-axis gimbal camera. The preset spatial kinematics model is constructed based on the omnidirectional mobile device, the three-axis robotic arm, and the three-axis gimbal camera.

[0036] S1022. Perform geometric smoothing on the initial motion coordinates of the omnidirectional moving device, the three-axis robotic arm, and the three-axis gimbal camera to obtain the smoothed initial motion coordinates of the omnidirectional moving device, the three-axis robotic arm, and the three-axis gimbal camera.

[0037] S1023. Perform time-series filtering on the smoothed motion coordinates to obtain the motion coordinates of the omnidirectional moving device, the three-axis robotic arm, and the three-axis gimbal camera.

[0038] For example, with the geometric center of the omnidirectional mobile device 100 as the origin of the world coordinate system, and combining the DH (Denavit-Hartenberg) parameters of the three-axis robotic arm 200 and the rotation matrix of the three-axis gimbal camera 300, a preset spatial kinematic model of six degrees of freedom (chassis 2 degrees of freedom, robotic arm 1 degree of freedom and gimbal 3 degrees of freedom) is constructed.

[0039] After obtaining the virtual motion trajectory, based on a pre-defined spatial kinematics model and solver, the planar displacement (X, Y) of the omnidirectional mobile device, the vertical displacement (Z) of the robotic arm, and the spatial attitude coordinates (XYZ spatial position and RPY roll-pitch-yaw angle) of the three-axis gimbal camera are calculated using either the weighted pseudo-inverse matrix method or a quadratic programming (QP) algorithm. This ensures that the robotic arm avoids singularities and joint limits during movement. The planar displacement (X, Y) of the omnidirectional mobile device, the vertical displacement (Z) of the robotic arm, and the spatial attitude coordinates of the three-axis gimbal camera represent the initial motion coordinates of the omnidirectional mobile device, the three-axis robotic arm, and the three-axis gimbal camera, respectively. The virtual motion trajectory is discretized into the initial motion coordinates of the corresponding motion points of the omnidirectional mobile device, the three-axis robotic arm, and the three-axis gimbal camera.

[0040] Quintic B-spline curves or cubic Hermite interpolation (PCHIP) are used to continuously fit the initial motion coordinates of the discrete omnidirectional moving device, the three-axis robotic arm, and the three-axis gimbal camera, ensuring the continuity of the trajectory in terms of position, velocity, and acceleration.

[0041] By introducing an extended Kalman filter (EKF) or a Savitzky-Golay filter, high-frequency noise in the inverse kinematics solution is filtered out, and a precise and smooth trajectory is finally planned for direct drive by the hardware actuator, namely the motion coordinates of the omnidirectional mobile device, the three-axis robotic arm and the three-axis gimbal camera.

[0042] Understandably, a unified spatial dynamics model was constructed, combining an omnidirectional mobile device, a three-axis robotic arm, and a three-axis gimbal camera, totaling six degrees of freedom. Through redundant calculations, fifth-order B-spline interpolation, and Kalman filtering, the potential for abrupt changes in mechanical joints and trajectory jitter caused by the neural network output was completely eliminated, ensuring high precision and smoothness of multi-axis linkage.

[0043] S103. Based on the motion coordinates of the omnidirectional mobile device, the three-axis robotic arm, and the three-axis gimbal camera, acquire the current motion image of the target object.

[0044] In some embodiments of this application, when the virtual motion trajectory is actually executed based on the motion coordinates of the omnidirectional mobile device, the three-axis robotic arm, and the three-axis gimbal camera, the current motion image of the target object is acquired.

[0045] S104. Based on the current motion image and audio data, the virtual motion trajectory is corrected to obtain the final motion trajectory.

[0046] In some embodiments of this application, S104 can be implemented by S1041-S1042, as follows: S1041. Based on the current motion image and virtual motion trajectory, perform temporal prediction and trajectory compensation to obtain the corrected motion trajectory.

[0047] In some embodiments of this application, target recognition is performed on the current motion image to obtain the full-body key points of the target object in the current motion image, as well as the geometric center coordinates; based on the full-body key points and geometric center coordinates, temporal prediction is performed on the target object to determine the spatial motion trend and displacement vector; based on the spatial motion trend and displacement vector, deviation calculation and high-frequency interpolation are performed on the virtual motion trajectory to obtain the corrected motion trajectory.

[0048] In some embodiments of this application, the real-time coordinates of the target object are determined based on the spatial motion trend and displacement vector; a composition deviation vector is calculated based on the real-time coordinates of the target object and a pre-determined center coordinate; wherein the center coordinate represents the center of the image or the golden section point; real-time depth information of the target object and the camera's internal parameter matrix are obtained; and a three-dimensional physical space deviation vector is determined based on the composition deviation vector, real-time depth information, and internal parameter matrix; a six-dimensional spatial incremental correction vector is determined through a preset dynamic safety frame window threshold area and the three-dimensional physical space deviation vector; and the virtual motion trajectory is corrected based on the six-dimensional spatial incremental correction vector to obtain the corrected motion trajectory.

[0049] For example, depth camera 401 captures images of the dancer in real time at a high frame rate (e.g., 60fps), i.e., the current motion image. The dancer's full-body key points and geometric center coordinates are extracted using a target recognition algorithm (e.g., ViTPose or a modified AlphaPose).

[0050] The extracted full-body keypoints and geometric center coordinates from the current motion image, as well as those from historical motion images, are fed into a Spatio-Temporal Graph Convolutional Network (ST-GCN) or a Long Short-Term Memory (LSTM) network to predict the dancer's future movements. Spatial motion trends and displacement vectors within a time period.

[0051] After receiving the predicted future spatial movement trend of the dancer from step one, the control system 400 does not directly change the original virtual movement trajectory of the robotic arm 200 and the gimbal 300. Instead, it achieves active decoupling compensation for the composition deviation through the following discrete, step-by-step refined sub-steps: Sub-step (1): Calculation of multi-dimensional composition deviation based on perspective projection and spatial coordinate transformation.

[0052] Extraction of 2D composition deviation vector in pixel coordinate system: After acquiring the spatial motion trend and displacement vector, the depth camera 401 acquires image frames containing the dancer in real time. Through target detection and human pose recognition algorithms, the real-time coordinates of the physical geometric center of the dancer's entire skeleton in the pixel coordinate system are extracted in the current frame (i.e., the image under the spatial motion trend). The control system 400 compares it with the ideal composition center coordinates of the image plane (such as the center of the screen or a preset golden ratio point). By comparing the images, the compositional deviation vector at the two-dimensional pixel level is calculated. : ; in, This is the lateral deviation; This is the longitudinal deviation; The x-coordinate of the physical geometric center of the dancer's entire skeleton in the current frame in the pixel coordinate system; The vertical coordinate of the physical geometric center of the dancer's entire skeleton in the current frame in the pixel coordinate system.

[0053] Projection mapping and decoupling component calculation of 3D physical space deviation: In order to convert the 2D deviation in pixel space into a physical space control quantity that can be executed by the omnidirectional chassis, the control system 400 uses the real-time depth information of the target measured by the depth camera 401. (i.e., the distance from the target to the camera's optical center), combined with the camera's internal parameter matrix. We construct a mapping relationship between pixel deviation and camera spatial physical deviation.

[0054] Furthermore, the control system 400 reads the rotation matrix constructed from the feedback of the sensors on each axis of the omnidirectional moving device 100, the three-axis robotic arm 200, and the three-axis gimbal camera 300 at the current moment. (i.e., from the camera coordinate system to the world coordinate system) The rotation transformation matrix is ​​derived from the homogeneous transformation matrix. (Extracted).

[0055] The control system 400 uses the pixel mapping deviation vector calculated above. Multiply by real-time depth information Through the camera intrinsic inverse matrix With spatial rotation matrix The cascaded transformation directly solves for the three-dimensional physical space deviation vector in the world coordinate system (physical space). : ; in, This is the inverse matrix of the camera's intrinsic parameters, used to convert pixel-level dynamic compositional deviations into physical displacement increments in the camera coordinate system. Through the cascaded multiplication of the above matrices, the control system 400 accurately calculates and extracts the dancer's real-time lateral physical deviation relative to the original shooting trajectory in the physical world within a unified spatiotemporal dimension. and longitudinal physical deviation This serves as the input benchmark for subsequent chassis motion compensation.

[0056] Sub-step (2): Adaptive discrimination and compensation activation based on dynamic safety frame.

[0057] Safety dead zone detection: To prevent frequent chassis vibration caused by high-frequency minute vibrations of the camera or visual noise, the control system 400 internally sets a dynamic safety frame window threshold area. .

[0058] In specific implementation, as an optional approach, the horizontal threshold δ of the dynamic security frame window threshold region Ω in this embodiment is... x and longitudinal threshold δ y The units of measurement, and the real-time depth information Z acquired by depth camera 401. c The units used must be kept strictly consistent.

[0059] Specifically, taking the International System of Units (SI) as an example, if the physical displacement dimensions in each coordinate system of the control system 400 all adopt the meter (m) as the base unit, then the lateral threshold δ x An exemplary value can be set to 0.05m (i.e., 50mm), with a longitudinal threshold δ. y An exemplary value can be set to 0.05m (i.e., 50mm). To balance the control stability of the omnidirectional motion device 100 with the tolerance of the visual composition of the screen, in actual engineering deployment, the lateral threshold δ... x With longitudinal threshold δ y The value range is preferably adaptively set between 0.02m and 0.10m (i.e., 20mm to 100mm).

[0060] Adaptive compensation activation: When the calculated physical deviation satisfies and When the system determines that the drawing error is within the allowable range, chassis compensation is not triggered; when the condition is met... or At that time, the adaptive error compensator is immediately activated, generating a six-dimensional spatial incremental correction vector containing only the planar translation correction: ; in, This is the corrected lateral physical deviation; This is the corrected longitudinal physical deviation.

[0061] By setting the threshold of the above interval, when the dancer's impromptu spatial movement causes a lateral physical deviation ΔX of 0.03m, since it does not exceed the threshold limit of 0.05m, the system determines that the current visual composition is still within the safe dead zone, and the control system 400 does not send speed compensation increments to the omnidirectional motion device 100, thereby completely isolating the frequent micro-adjustment shaking of the chassis caused by high-frequency noise or slight shaking of the camera; while when the dancer's rapid sliding causes a lateral physical deviation ΔX of 0.08m, since it breaks through the 0.05m safety frame boundary, the adaptive error compensator is immediately activated, thereby precisely driving the four Mecanum wheels 101 to perform lateral compensation translation.

[0062] Sub-step (3): High-frequency time-series trajectory interpolation and smooth reconstruction based on boundary constraints.

[0063] Controlling timing alignment and frequency decoupling: Because the virtual motion trajectory generated by the multimodal large model belongs to low-frequency signals (e.g. This system cannot directly drive a high-response motor. It uses a real-time interpolation algorithm to interpolate the original low-frequency virtual motion trajectory. Based on this, the physical space increment correction vector calculated in the aforementioned sub-step (2) is... Perform high-frequency timing fusion.

[0064] Cubic Hermitian Spline (PCHIP) High-Frequency Interpolation: To ensure the continuity of robot motion, within each sliding control cycle window, the algorithm introduces cubic Hermitian spline interpolation with boundary velocity constraints. This high-frequency interpolation reconstructs the virtual motion trajectory and the correction values, generating the corrected motion trajectory. : ; in, For virtual motion trajectories; This is the trajectory function for cubic Hermitian spline interpolation.

[0065] Multi-axis degree of freedom decoupled execution: In this high-frequency reconstructed control trajectory, the incremental correction is strictly decoupled and applied to the X-axis and Y-axis translational degrees of freedom of the omnidirectional moving device 100, driving the four Mecanum wheels 101 to perform lateral / longitudinal compensation; while the Z-axis trajectory of the three-axis robotic arm 200 responsible for vertical motion, and the RPY attitude trajectory of the three-axis gimbal camera 300 responsible for roll / pitch / yaw, are still strictly locked and executed within this sequence, thus achieving absolute physical compositional stabilization without destroying the original intention of film-level camera movement.

[0066] It is understandable that during the execution of the virtual motion trajectory, the system does not passively respond to compositional deviations, but rather uses a graph convolutional network to predict the future skeletal movements of the subject and performs high-frequency interpolation correction on the virtual motion trajectory in real time, thus achieving active and zero-latency compositional stabilization.

[0067] S1042. Based on the audio data, the corrected motion trajectory is further corrected to obtain the final motion trajectory.

[0068] In some embodiments of this application, a weak beat phase and a strong beat phase are determined based on audio data; when the audio data is in a weak beat phase, the first motion trajectory corresponding to the weak beat phase is accelerated and corrected in the corrected motion trajectory to determine the target motion trajectory corresponding to the weak beat phase; when the audio data is in a strong beat phase, the second motion trajectory corresponding to the strong beat phase is determined in the corrected motion trajectory; and the final motion trajectory is determined based on the target motion trajectory corresponding to the weak beat phase and the second motion trajectory corresponding to the strong beat phase.

[0069] For example, in order to prevent abrupt changes in camera movement caused by visual compensation from damaging the audiovisual effect, the music-track multimodal collaborative constraint module introduces a music rhythm dynamics constraint mechanism: dynamically scaling the trajectory correction amplitude based on the energy envelope (Onset Envelope) and beat point of the currently playing music (i.e., audio data).

[0070] During the "weak beat" or flat section of the music (i.e., the weak beat phase), the robot's corrective acceleration is limited to ensure smooth and stable image quality. That is, the acceleration of the first motion trajectory corresponding to the weak beat phase is corrected in the corrected motion trajectory to determine the target motion trajectory corresponding to the weak beat phase.

[0071] During the "strong beat" or drop point of the music (i.e., the strong beat phase), the chassis movement flexibility is unleashed, allowing for a wide range of chassis adjustments, so that the camera movement corrections perfectly match the musical rhythm. When the audio data is in the strong beat phase, the second motion trajectory corresponding to the strong beat phase is determined within the corrected motion trajectory; based on the target motion trajectory corresponding to the weak beat phase and the second motion trajectory corresponding to the strong beat phase, the final motion trajectory is determined.

[0072] Understandably, the innovative introduction of an audiovisual multimodal constraint module perfectly aligns the image stabilization compensation amplitude with the music's repetitive beats. Simultaneously, in conjunction with underlying model predictive control, a high-frequency, highly robust closed-loop motion control response is achieved under strong constraints that meet the physical limits of multi-axis hardware (torque and speed).

[0073] S105. By controlling the omnidirectional moving device, three-axis robotic arm, and three-axis gimbal camera through the final motion trajectory, photographic data is collected from the target object.

[0074] Understandably, by multimodally fusing historical motion skeleton sequences, audio data, and preset visual composition style tags, a virtual motion trajectory with a cinematic feel and in sync with the music's rhythm can be generated. Users do not need to master complex robot trajectory programming or professional camera movement parameter expressions; the device can automatically understand music and dance, generate and replicate the camera movement logic of blockbuster films, significantly improving the production efficiency of short videos and film production pipelines. Based on the virtual motion trajectory, posture calculation and correction are performed to obtain the final motion trajectory. Even when faced with complex impromptu steps and positioning errors of the target object, the composition remains rock-solid, greatly improving the success rate of shooting dance scenes and lowering the threshold for high-quality video creation, achieving high-precision and intelligent replication of the camera movement trajectories of excellent visual works.

[0075] In some embodiments of this application, a low-level model predictive control (MPC) based on full kinematic constraints inputs the final motion trajectory to the system's low-level dynamic control core. The MPC controller of the control system 400 employs a model predictive control (MPC) or whole-body control (WBC) architecture. This model integrates the dynamics of the Mecanum wheel chassis of the omnidirectional mobile device 100, the joint torque limits of the three-axis robotic arm 200, and the nonlinear disturbances of the three-axis gimbal camera 300 into the state-space equations. By solving a constrained real-time optimization problem within the finite time domain of rolling, complex translation and image stabilization commands are precisely transformed into wheel speed commands for the four Mecanum wheels 101 (speed allocation using the inverse Mecanum wheel formula) and torque and angle encoder control signals for each joint servo motor, achieving absolute high-frequency image stabilization control, thereby enabling photographic acquisition of the target object.

[0076] For example, let's take a low-angle, surround-sound, stabilized image shot that is synchronized with a deep bass beat as an example: Virtual motion trajectory generation: The multimodal large model MCT recognizes that the current music has a strong rhythm of 120 BPM and the dancer is performing ground street dance moves, and automatically generates a virtual motion trajectory that loops around and has a robotic arm rising and falling.

[0077] Motion execution: The differential rotation of the Mecanum wheel 101 drives the omnidirectional moving device 100 to perform tangential translation, the three-axis robotic arm 200 performs downward diving and outward extension movements, and the three-axis gimbal camera 300 corrects the pitch angle in real time to maintain alignment.

[0078] Prediction and interpolation: When a dancer slides to the left due to improvisation, the control system predicts the displacement in advance through ST-GCN, and the visual servo module quickly calculates the deviation.

[0079] Rhythm Constraints and Underlying Motion Control: Just as the music enters its downbeat, the multimodal constraint module allows for maximum compensation, and the bottom MPC controller instantly calculates the lateral translation speed increment of the four Mecanum wheels 101. Without altering the original lifting trajectory of the robotic arm, the chassis utilizes its omnidirectional movement characteristics to laterally "slide" and fill in the gaps, perfectly locking the dancer at the golden ratio point of the frame, achieving film-level smooth and stable imaging.

[0080] This application has the following beneficial effects: First, it greatly lowers the threshold for professional-grade audiovisual creation, realizing "AI-powered intelligent directorial camera movement": With the help of the multimodal Transformer large model, users do not need to master complex robot trajectory programming or professional camera movement parameter expression. The device can automatically understand music and dance, generate and replicate the camera movement logic of blockbuster films, and greatly improve the production efficiency of short videos and film and television production.

[0081] Secondly, it overcomes the industry pain point of low success rate caused by traditional camera robots "running dead trajectories": by introducing temporal pose prediction (ST-GCN) and high-frequency trajectory interpolation algorithms, the camera robot has powerful intelligent anti-disturbance capabilities while replicating classic camera movements. Even when faced with the complex improvisational steps and positioning errors of dancers, it can ensure that the composition is rock-solid, greatly improving the success rate of shooting dance scenes.

[0082] Furthermore, it achieves deep synergy between camera movement aesthetics, image stabilization compensation, and musical rhythm (audiovisual integration): through music-trajectory multimodal collaborative constraints, the robot's mechanical motion logic and visual compensation amplitude are perfectly integrated into the strong and weak rhythms of the music. The movements are delicate and smooth on weak beats, and quick and agile in filling in the gaps on strong beats, avoiding the stiff and abrupt mechanical feel produced by traditional visual servo systems when stabilizing images.

[0083] Finally, it possesses extremely high engineering feasibility and dynamic control performance: the underlying control adopts an MPC model predictive control architecture, which can calculate the optimal coordinated movement of the chassis Mecanum wheel and the multi-axis robotic arm within milliseconds. This fully utilizes the high-frequency translational degrees of freedom of the Mecanum wheel chassis while greatly reducing the dynamic load on the joints of the robotic arm, ensuring extremely high stability and hardware and software safety of the entire machine in indoor / narrow and complex dance scenarios.

[0084] Based on the above embodiments, this application also provides an autonomously controlled multi-axis angle omnidirectional ground-based photography method. Figure 4 As shown, Figure 4 This is a schematic diagram of the structure of a ground-based omnidirectional photographic system that autonomously controls multi-axis angles for omnidirectional ground movement, as provided in an embodiment of this application. The system 4 includes: an acquisition module 41, an attitude calculation module 42, a data acquisition module 43, and a correction module 44; wherein... The acquisition module 41 is used to acquire the historical motion skeleton sequence, audio data and preset visual composition style tags of the target object; and generate a virtual motion trajectory based on the historical motion skeleton sequence, the audio data and the preset visual composition style tags. The attitude calculation module 42 is used to perform attitude calculation based on the virtual motion trajectory to obtain the motion coordinates of the omnidirectional mobile device, the three-axis robotic arm and the three-axis gimbal camera. The acquisition module 43 is used to acquire the current motion image of the target object based on the motion coordinates of the omnidirectional mobile device, the three-axis robotic arm and the three-axis gimbal camera. The correction module 44 is used to correct the virtual motion trajectory based on the current motion image and the audio data to obtain the final motion trajectory; The acquisition module 43 is also used to control the omnidirectional moving device, the three-axis robotic arm and the three-axis gimbal camera through the final motion trajectory to capture photographs of the target object.

[0085] Based on the above embodiments, this application also provides a photographic device for autonomously controlling multi-axis angle omnidirectional ground movement, such as... Figure 5 As shown, Figure 5This is a schematic diagram of the structure of a photographic device capable of autonomously controlling multi-axis angle omnidirectional ground movement, provided in an embodiment of this application. The photographic device 5 includes a processor 501 and a memory 502. The memory 502 stores computer programs; the processor 501 retrieves and runs the computer programs from the memory to execute a photographic method for autonomously controlling multi-axis angle omnidirectional ground movement as described in the above embodiment.

[0086] In the embodiments of this application, the processor 501 described above can be at least one of the following: Application-Specific Integrated Circuit (ASIC), Digital Signal Processor (DSP), Digital Signal Processing Device (DSPD), Programmable Logic Device (PLD), Field Programmable Gate Array (FPGA), Central Processing Unit (CPU), Controller, Microcontroller, and Microprocessor. It is understood that for different devices, the electronic device used to implement the above processor function can also be other types, and the embodiments of this application do not specifically limit it.

[0087] This application provides a computer-readable storage medium storing a computer program for implementing, when executed by a processor, a photographic method for autonomously controlling multi-axis angle omnidirectional ground movement as described in any of the above embodiments.

[0088] For example, the program instructions corresponding to the autonomous multi-axis angle omnidirectional ground movement photography method in this embodiment can be stored on storage media such as optical discs, hard disks, and USB flash drives. When the program instructions corresponding to the autonomous multi-axis angle omnidirectional ground movement photography method in the storage medium are read or executed by an electronic device, the autonomous multi-axis angle omnidirectional ground movement photography method described in any of the above embodiments can be realized.

[0089] Furthermore, the functional modules in the embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional module.

[0090] If the integrated unit is implemented as a software functional module and is not sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this embodiment, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute all or part of the steps of the method of this embodiment. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0091] It should be understood that the phrases "one embodiment," "an embodiment," or "some embodiments" mentioned throughout the specification mean that a specific feature, structure, or characteristic related to the embodiment is included in at least one embodiment of this application. Therefore, "in one embodiment," "in one embodiment," or "in some embodiments" appearing throughout the specification do not necessarily refer to the same embodiment. Furthermore, these specific features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. It should be understood that in the various embodiments of this application, the sequence numbers of the above-described processes do not imply a sequential order of execution; the execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application. The sequence numbers of the embodiments in this application are merely for descriptive purposes and do not represent the superiority or inferiority of the embodiments. The descriptions of the various embodiments above tend to emphasize the differences between the various embodiments; their similarities or commonalities can be referred to mutually, and for the sake of brevity, these will not be repeated here.

[0092] The modules described above as separate components may or may not be physically separate. The components shown as modules may or may not be physical modules. They may be located in one place or distributed across multiple network units. Some or all of the modules may be selected to achieve the purpose of this embodiment according to actual needs.

[0093] In addition, each functional module in the various embodiments of this application can be integrated into one processing unit, or each module can be a separate unit, or two or more modules can be integrated into one unit; the integrated modules can be implemented in hardware or in the form of hardware plus software functional units.

[0094] Those skilled in the art will understand that all or part of the steps of the above method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it performs the steps of the above method embodiments. The aforementioned storage medium includes various media that can store program code, such as mobile storage devices, read-only memory (ROM), magnetic disks, or optical disks.

[0095] The methods disclosed in the several method embodiments provided in this application can be arbitrarily combined without conflict to obtain new method embodiments.

[0096] The features disclosed in the several product embodiments provided in this application can be arbitrarily combined without conflict to obtain new product embodiments.

[0097] The features disclosed in the several method or device embodiments provided in this application can be arbitrarily combined without conflict to obtain new method or device embodiments.

[0098] The above description is merely an embodiment of this application, but the protection scope of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the protection scope of this application. Therefore, the protection scope of this application should be determined by the protection scope of the claims.

Claims

1. A method for autonomously controlling multi-axis angle omnidirectional ground movement in photography, characterized in that, include: Obtain the historical motion skeleton sequence, audio data, and preset visual composition style tags of the target object; And based on the historical motion skeleton sequence, the audio data, and the preset visual composition style tags, a virtual motion trajectory is generated; Based on the virtual motion trajectory, attitude calculation is performed to obtain the motion coordinates of the omnidirectional mobile device, the three-axis robotic arm, and the three-axis gimbal camera. Based on the motion coordinates of the omnidirectional mobile device, the three-axis robotic arm, and the three-axis gimbal camera, the current motion image of the target object is acquired. Based on the current motion image and the audio data, the virtual motion trajectory is corrected to obtain the final motion trajectory; The omnidirectional moving device, three-axis robotic arm, and three-axis gimbal camera are controlled by the final motion trajectory to capture photographic images of the target object.

2. The method according to claim 1, characterized in that, The step of generating a virtual motion trajectory based on the historical motion skeleton sequence, the audio data, and the preset visual composition style tags includes: Motion features are extracted from the historical motion skeleton sequence to obtain motion features; Audio features are extracted from the audio data to obtain beat features; The virtual motion trajectory is generated using the motion features, the beat features, and the preset visual composition style tags.

3. The method according to claim 1, characterized in that, The attitude calculation based on the virtual motion trajectory yields the motion coordinates of the omnidirectional mobile device, the three-axis robotic arm, and the three-axis gimbal camera, including: The virtual motion trajectory is calculated using a preset spatial kinematics model and solver to obtain the initial motion coordinates of the omnidirectional mobile device, the three-axis robotic arm, and the three-axis gimbal camera; wherein the preset spatial kinematics model is constructed based on the omnidirectional mobile device, the three-axis robotic arm, and the three-axis gimbal camera. The initial motion coordinates of the omnidirectional moving device, the three-axis robotic arm, and the three-axis gimbal camera are geometrically smoothed to obtain the smoothed initial motion coordinates of the omnidirectional moving device, the three-axis robotic arm, and the three-axis gimbal camera. The smoothed motion coordinates are subjected to time-series filtering to obtain the motion coordinates of the omnidirectional moving device, the three-axis robotic arm, and the three-axis gimbal camera.

4. The method according to claim 1, characterized in that, The step of correcting the virtual motion trajectory based on the current motion image and the audio data to obtain the final motion trajectory includes: Based on the current motion image and the virtual motion trajectory, temporal prediction and trajectory compensation are performed to obtain the corrected motion trajectory; Based on the audio data, the corrected motion trajectory is further corrected to obtain the final motion trajectory.

5. The method according to claim 4, characterized in that, The step of performing temporal prediction and trajectory compensation based on the current motion image and the virtual motion trajectory to obtain the corrected motion trajectory includes: Target recognition is performed on the current motion image to obtain the full-body key points of the target object in the current motion image, as well as the coordinates of its geometric center; Based on the coordinates of the key points and geometric center of the whole body, the target object is temporally predicted to determine the spatial motion trend and displacement vector; Based on the spatial motion trend and displacement vector, the virtual motion trajectory is subjected to deviation calculation and high-frequency interpolation to obtain the corrected motion trajectory.

6. The method according to claim 5, characterized in that, The step of calculating the deviation and performing high-frequency interpolation on the virtual motion trajectory based on the spatial motion trend and displacement vector to obtain the corrected motion trajectory includes: Based on the spatial motion trend and displacement vector, determine the real-time coordinates of the target object; Based on the real-time coordinates of the target object and the predetermined center coordinates, a composition deviation vector is calculated; wherein, the center coordinates represent the exact center of the image or the golden ratio point. Acquire the real-time depth information of the target object and the camera's intrinsic parameter matrix; and determine the three-dimensional physical space deviation vector based on the composition deviation vector, real-time depth information, and intrinsic parameter matrix; The six-dimensional spatial incremental correction vector is determined by using the preset dynamic safety frame window threshold area and the three-dimensional physical space deviation vector. Based on the six-dimensional spatial incremental correction vector, the virtual motion trajectory is corrected to obtain the corrected motion trajectory.

7. The method according to claim 4, characterized in that, The step of performing a second correction on the modified motion trajectory based on the audio data to obtain the final motion trajectory includes: Based on the audio data, the weak beat phase and the strong beat phase are determined; When the audio data is in a weak beat phase, the first motion trajectory corresponding to the weak beat phase is accelerated in the corrected motion trajectory to determine the target motion trajectory corresponding to the weak beat phase. When the audio data is in a strong beat phase, a second motion trajectory corresponding to the strong beat phase is determined from the corrected motion trajectory. The final motion trajectory is determined based on the target motion trajectory corresponding to the weak beat phase and the second motion trajectory corresponding to the strong beat phase.

8. A ground-based omnidirectional photographic system with autonomous control of multi-axis angles, characterized in that, include: The module comprises an acquisition module, an attitude calculation module, a data acquisition module, and a correction module; among which, The acquisition module is used to acquire the historical motion skeleton sequence, audio data, and preset visual composition style tags of the target object; and to generate a virtual motion trajectory based on the historical motion skeleton sequence, the audio data, and the preset visual composition style tags. The attitude calculation module is used to perform attitude calculation based on the virtual motion trajectory to obtain the motion coordinates of the omnidirectional mobile device, the three-axis robotic arm and the three-axis gimbal camera. The acquisition module is used to acquire the current motion image of the target object based on the motion coordinates of the omnidirectional mobile device, the three-axis robotic arm and the three-axis gimbal camera. The correction module is used to correct the virtual motion trajectory based on the current motion image and the audio data to obtain the final motion trajectory; The acquisition module is also used to control the omnidirectional moving device, the three-axis robotic arm, and the three-axis gimbal camera through the final motion trajectory to capture photographs of the target object.

9. A photographic device capable of autonomously controlling multi-axis angle omnidirectional ground movement, characterized in that, include: Processor and memory, of which, The memory is used to store computer programs; The processor is configured to call and run the computer program from the memory to perform the method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, It stores executable instructions for causing a processor to execute, thereby implementing the method of any one of claims 1 to 7.

Images