Photographing method, apparatus, device, storage medium and product

Abstract

The application belongs to the technical field of shooting, and discloses a shooting method, device, equipment, storage medium and product. The application determines a target machine position vector of a shooting device according to pose information of a person being shot, the shooting device comprising a mechanical arm and a camera mounted at the end of the mechanical arm, then determines a joint motion trajectory of the mechanical arm and an optical zoom motor position of the camera according to the target machine position vector, and finally shoots the person being shot based on the joint motion trajectory and the optical zoom motor position. The application first determines the target machine position vector of the shooting device according to the pose information of the person being shot, and then shoots the person being shot based on the joint motion trajectory of the mechanical arm and the optical zoom motor position of the camera. Compared with the prior art in which the person being shot needs to manually adjust the camera frequently, the above-mentioned method of the application can shoot the person being shot by the camera mounted on the mechanical arm in a single-person shooting scene, thereby improving shooting efficiency and the quality of shooting content.

Description

Technical Field

[0001] This application relates to the field of photography technology, and more particularly to a photography method, apparatus, device, storage medium, and product. Background Technology

[0002] With the booming development of the short video and live streaming industries, more and more individual creators are devoting themselves to content creation such as knowledge explanation, product review, and tutorial recording. However, in solo shooting scenarios, creators need to balance content expression and image composition, frequently manually adjusting camera position, focal length, and shooting angle, resulting in low shooting efficiency, limited image expressiveness, and severely restricting the upper limit of content quality. Summary of the Invention

[0003] The main purpose of this application is to provide a shooting method, apparatus, device, storage medium and product, which aims to solve the technical problem that in single-person shooting scenarios, the subject needs to frequently manually adjust the camera, resulting in low shooting efficiency and poor content quality.

[0004] To achieve the above objectives, this application provides a shooting method, which includes the following steps: The target camera position vector of the shooting device is determined based on the posture information of the subject. The shooting device includes a robotic arm and a camera mounted at the end of the robotic arm. The joint motion trajectory of the robotic arm and the position of the optical zoom motor of the camera are determined based on the target position vector. The subject is photographed based on the joint movement trajectory and the position of the optical zoom motor.

[0005] Optionally, determining the target camera position vector of the shooting device based on the subject's posture information includes: Acquire images of the shooting area where the subject is located, and determine the subject's posture information based on the captured images; Determine the attitude state vector based on the attitude information; Acquire the voice commands issued by the subject and determine the voice intent vector based on the voice commands; The target camera position vector of the shooting device is determined based on the posture state vector and the voice intent vector.

[0006] Optionally, acquiring a captured image of the area where the subject is located, and determining the subject's posture information based on the captured image, includes: Acquire images of the area where the subject is located, including depth images and color images; Two-dimensional coordinates of joint points are extracted from the color image using a key point detection model. Based on the pixel depth values ​​corresponding to the depth image, the two-dimensional coordinates of the joints are converted into three-dimensional coordinates. The pose information of the subject is determined based on the three-dimensional coordinates of the joints.

[0007] Optionally, determining the subject's posture information based on the three-dimensional coordinates of the joint points includes: Select the core joint coordinates of the torso from the three-dimensional coordinates of the joints, and determine the three-dimensional centroid coordinates of the subject based on the core joint coordinates of the torso. Calculate the ratio of the vertical distance between the head and the centroid based on the three-dimensional centroid coordinates and the three-dimensional head coordinates of the subject; A trunk direction vector is constructed based on the three-dimensional coordinates of the joint points, and the trunk tilt angle is determined based on the angle between the trunk direction vector and the vertical reference vector. The body orientation angle is determined based on the shoulder's three-dimensional coordinates in the joint's three-dimensional coordinates. The subject's posture information is determined based on the vertical distance ratio, the torso tilt angle, and the body orientation angle.

[0008] Optionally, determining the attitude state vector based on the attitude information includes: Acquire IMU data collected by the IMU sensor worn by the subject, the IMU data including acceleration and angular velocity; The attitude information is updated based on the acceleration and the angular velocity to obtain the updated attitude information; The attitude state vector is determined based on the updated attitude information.

[0009] Optionally, determining the target camera position vector of the shooting device based on the posture state vector and the voice intent vector includes: The speech intent vector is used as the key, the pose state vector is used as the query, and the fused feature vector is calculated through an attention mechanism. The fused feature vector is input into the decision network to obtain the target camera position vector, which includes the joint angle value of the robotic arm and the optical zoom ratio of the camera mounted at the end of the robotic arm.

[0010] Optionally, determining the joint motion trajectory of the robotic arm and the position of the optical zoom motor of the camera based on the target camera position vector includes: The target joint position of the robotic arm is determined based on the target position vector, and the current joint position of the robotic arm is determined. The joint motion trajectory of the robotic arm is determined based on the current joint position and the target joint position; During the movement of the robotic arm along the joint trajectory, the position of the optical zoom motor is determined based on the change in distance between the subject and the camera.

[0011] Optionally, the step of photographing the subject based on the joint motion trajectory and the position of the optical zoom motor includes: During the process of photographing the subject based on the joint motion trajectory and the position of the optical zoom motor, the corresponding behavior sequence of the subject is obtained; The behavior sequence is analyzed using a behavior sequence classification model to obtain the subject's initial shooting pattern and corresponding confidence level. Based on the confidence level and the initial shooting mode, a new target camera position vector is determined, and the steps of determining the joint motion trajectory of the robotic arm and the position of the optical zoom motor of the camera based on the target camera position vector are returned. The subject is then photographed based on the new joint motion trajectory and the new optical zoom motor position.

[0012] Optionally, during the process of photographing the subject based on the joint motion trajectory and the position of the optical zoom motor, obtaining the corresponding behavioral sequence of the subject includes: During the process of photographing the subject based on the joint movement trajectory and the position of the optical zoom motor, the corresponding shooting image of the subject is acquired; The positional offset of the subject is determined based on the captured image, and the positional offset includes horizontal offset and vertical offset; The position of the robotic arm is fine-tuned based on the horizontal and vertical offsets, and the behavioral sequence of the subject is obtained based on the fine-tuned position.

[0013] Optionally, determining the positional offset of the subject based on the captured image includes: The actual position of the subject is obtained by detecting the captured image using a target detection model. The target position of the subject is determined according to the composition rules corresponding to the subject. The actual position is compared with the target position, and the positional offset of the subject is determined based on the comparison result.

[0014] Optionally, the step of fine-tuning the position of the robotic arm based on the horizontal and vertical offsets, and obtaining the behavioral sequence corresponding to the subject based on the fine-tuned position, includes: If the horizontal offset is greater than a preset horizontal threshold or the vertical offset is greater than a preset vertical threshold, the offset direction is determined, and the offset direction includes the horizontal offset direction and the vertical offset direction. The compensation direction is determined based on the offset direction; The position of the robotic arm is fine-tuned based on the compensation direction and the preset fine-tuning step size, and the behavior sequence corresponding to the subject is obtained based on the fine-tuned position.

[0015] Optionally, determining the new target camera position vector based on the confidence level and the initial shooting mode includes: If the confidence level is less than or equal to a preset threshold, return to the step of determining the target camera position vector of the shooting device based on the subject's posture information, and determine a new target camera position vector; If the confidence level is greater than a preset threshold, the initial shooting mode is taken as the target shooting mode; A new target camera position vector is determined based on the preset camera position parameters corresponding to the target shooting mode.

[0016] Furthermore, to achieve the above objectives, this application also provides a shooting device, the shooting device comprising: A vector determination module is used to determine the target camera position vector of the shooting device based on the posture information of the subject. The shooting device includes a robotic arm and a camera mounted at the end of the robotic arm. The trajectory determination module is used to determine the joint motion trajectory of the robotic arm and the position of the optical zoom motor of the camera based on the target camera position vector. The shooting module is used to take pictures of the subject based on the joint movement trajectory and the position of the optical zoom motor.

[0017] Optionally, the vector determination module is further configured to acquire a captured image of the shooting area where the subject is located, and determine the subject's posture information based on the captured image; determine a posture state vector based on the posture information; acquire a voice command issued by the subject, and determine a voice intent vector based on the voice command; and determine the target camera position vector of the shooting device based on the posture state vector and the voice intent vector.

[0018] Optionally, the vector determination module is further configured to acquire a captured image of the shooting area where the subject is located, the captured image including a depth image and a color image; extract two-dimensional coordinates of joints from the color image using a key point detection model; convert the two-dimensional coordinates of joints into three-dimensional coordinates of joints based on the pixel depth values ​​corresponding to the depth image; and determine the pose information of the subject based on the three-dimensional coordinates of joints.

[0019] Optionally, the trajectory determination module is further configured to determine the target joint position of the robotic arm based on the target camera position vector, and determine the current joint position of the robotic arm; determine the joint motion trajectory of the robotic arm based on the current joint position and the target joint position; and determine the position of the optical zoom motor based on the change in distance between the subject and the camera during the movement of the robotic arm according to the joint motion trajectory.

[0020] Optionally, the shooting module is further configured to: acquire a behavior sequence corresponding to the subject during the process of shooting the subject based on the joint motion trajectory and the position of the optical zoom motor; analyze the behavior sequence using a behavior sequence classification model to obtain the initial shooting mode and corresponding confidence level of the subject; determine a new target camera position vector based on the confidence level and the initial shooting mode; return to the step of determining the joint motion trajectory of the robotic arm and the position of the optical zoom motor of the camera based on the target camera position vector; and shoot the subject based on the new joint motion trajectory and the new optical zoom motor position.

[0021] In addition, to achieve the above objectives, this application also proposes a shooting device, which includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the shooting method described above.

[0022] In addition, to achieve the above objectives, this application also proposes a storage medium, which is a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, it implements the steps of the shooting method described above.

[0023] In addition, to achieve the above objectives, this application also provides a computer program product, which includes a computer program that, when executed by a processor, implements the steps of the shooting method described above.

[0024] This application determines the target camera position vector of the shooting device based on the subject's posture information. The shooting device includes a robotic arm and a camera mounted on the end of the robotic arm. Then, based on the target camera position vector, it determines the joint motion trajectory of the robotic arm and the position of the camera's optical zoom motor. Finally, it captures the subject based on the joint motion trajectory and the position of the optical zoom motor. This application first determines the target camera position vector of the shooting device based on the subject's posture information, automatically determining the target camera position vector that the shooting device needs to achieve. Then, it captures the subject based on the joint motion trajectory of the robotic arm and the position of the camera's optical zoom motor. Compared to existing methods that require the subject to frequently manually adjust the camera, the above method of this application can capture the subject in a single-person shooting scenario using a camera mounted on a robotic arm, thereby improving shooting efficiency and the quality of the captured content. Attached Figure Description

[0025] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0026] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0027] Figure 1 This is a flowchart illustrating the first embodiment of the shooting method of this application; Figure 2 This is a flowchart illustrating the second embodiment of the shooting method of this application; Figure 3 This is a flowchart illustrating the third embodiment of the shooting method of this application; Figure 4 This is a schematic diagram of the overall process of one embodiment of the shooting method of this application; Figure 5 This is a structural block diagram of the first embodiment of the imaging device of this application; Figure 6 This is a schematic diagram of the structure of the shooting device in the hardware operating environment involved in the embodiments of this application.

[0028] The realization of the purpose, functional features and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0029] It should be understood that the specific embodiments described herein are merely illustrative of the technical solutions of this application and are not intended to limit this application.

[0030] To better understand the technical solution of this application, a detailed description will be provided below in conjunction with the accompanying drawings and specific implementation methods.

[0031] It should be noted that the executing entity of this application can be a computing service device with data processing, network communication and program execution functions, such as a computer, which can be used to control the shooting equipment.

[0032] Based on this, the embodiments of this application provide a shooting method, referring to... Figure 1 , Figure 1 This is a flowchart illustrating the first embodiment of the shooting method of this application.

[0033] In this embodiment, the shooting method includes the following steps: Step S10: Determine the target camera position vector of the shooting device based on the subject's posture information. The shooting device includes a robotic arm and a camera mounted on the end of the robotic arm.

[0034] Understandably, the subject can be someone who needs to be filmed with a camera, or a creator of short videos or live streams. The subject's posture information can be determined, which can include four basic postures: standing, sitting, bending, and turning.

[0035] It should be understood that the target camera position vector of the shooting device can be determined based on the posture information. The shooting device may include a robotic arm and a camera mounted on the end of the robotic arm. During the movement of the robotic arm, the subject can be photographed by the camera mounted on the end of the robotic arm. The target camera position vector may include the target camera position vector that the robotic arm and the camera want to achieve, such as the target angle values ​​of the six joints of the robotic arm, the optical zoom ratio of the camera, etc.

[0036] Step S20: Determine the joint motion trajectory of the robotic arm and the position of the optical zoom motor of the camera based on the target position vector.

[0037] It should be understood that the target camera position vector is equivalent to the endpoint position. The joint motion trajectory of the robotic arm and the position of the optical zoom motor of the camera can be determined based on the current position and the endpoint position. The joint motion trajectory can be the motion trajectory of the robotic arm from the current position to the endpoint position, and the position of the optical zoom motor can be the position of the optical zoom motor of the camera when it takes a picture of the subject at the endpoint position.

[0038] Furthermore, in order to effectively obtain the joint motion trajectory of the robotic arm and the position of the optical zoom motor of the camera, in this embodiment, step S20 includes: determining the target joint position of the robotic arm based on the target camera position vector, and determining the current joint position of the robotic arm; determining the joint motion trajectory of the robotic arm based on the current joint position and the target joint position; and determining the position of the optical zoom motor based on the change in distance between the subject and the camera during the movement of the robotic arm according to the joint motion trajectory.

[0039] Understandably, after obtaining the target camera position vector, the target pose of the end effector can be decomposed into the angle values ​​that each of the six joints needs to achieve using an inverse kinematics solver. These angle values ​​are then used as the target joint positions. Inverse kinematics refers to the mathematical process of calculating the angle values ​​that each joint needs to achieve based on the target pose of the robotic arm's end effector (the end with the camera). It serves as a bridge connecting the target camera position vector and the actual joint motion commands. Simultaneously, the current joint positions of the robotic arm can be determined, i.e., the angle values ​​of the six joints at the current moment.

[0040] It should be understood that the joint motion trajectory of the robotic arm can be generated using fifth-order polynomial interpolation, with the boundary condition that the velocity and acceleration of the current joint position and the target joint position are both zero, ensuring the smoothness and continuity of the motion. This is achieved using the fifth-order polynomial function q(t) = a0 + a1t + a2t² + a3t³ + a4t. 4 +a5t 5 The method of smoothing the joint motion trajectory through interpolation ensures that the velocity and acceleration are zero at the start and end of the motion, avoiding impact and vibration during the robotic arm's movement. The coefficients of the fifth-order polynomial are determined by the target angle difference: a3 = 10Δθ, a4 = -15Δθ, a5 = 6Δθ, where Δθ represents the change in joint angle, i.e., the difference between the target angle and the current angle. The joint motion trajectory is discretized into 100 equally divided points. Considering the safety requirements of the robotic arm's movement, the system combines dynamic constraints: a maximum angular velocity of 1.5 rad / s and a maximum acceleration of 3.0 rad / s², to constrain and verify the trajectory. Each discrete point undergoes dynamic constraint verification, and values ​​exceeding the limits are limited.

[0041] In practical implementation, as the robotic arm moves along the joint trajectory, the position of the optical zoom motor of the camera lens can be synchronously adjusted by a PID closed-loop controller. PID closed-loop control refers to a proportional-integral-derivative control algorithm used to synchronously adjust the position of the optical zoom motor of the camera lens. By continuously comparing the deviation between the actual zoom position and the target zoom position and outputting the adjustment amount, the coordination between physical displacement and optical zoom is achieved. The PID parameters are: proportional coefficient Kp=0.8, integral coefficient Ki=0.1, and derivative coefficient Kd=0.05. The controller calculates the required zoom adjustment amount based on the change in distance between the target camera position and the subject, and performs 10 iterations of closed-loop correction to ensure that the subject is always located at the preset golden ratio point or center area during camera position switching. Golden ratio composition refers to placing the subject in a key position in the frame according to the golden ratio (approximately 0.618) rather than the exact center, resulting in a more natural and aesthetically pleasing visual balance in the image.

[0042] Step S30: Take a picture of the subject based on the joint motion trajectory and the position of the optical zoom motor.

[0043] Understandably, as the robotic arm moves based on the joint motion trajectory, it can drive the camera to move, allowing the camera to take pictures of the subject based on the position of the optical zoom motor.

[0044] This embodiment determines the target camera position vector of the shooting device based on the subject's posture information. The shooting device includes a robotic arm and a camera mounted on the end of the robotic arm. Then, based on the target camera position vector, it determines the joint motion trajectory of the robotic arm and the position of the camera's optical zoom motor. Finally, it captures the subject based on the joint motion trajectory and the position of the optical zoom motor. This embodiment first determines the target camera position vector of the shooting device based on the subject's posture information, automatically determining the target camera position vector that the shooting device needs to achieve. Then, it captures the subject based on the joint motion trajectory of the robotic arm and the position of the camera's optical zoom motor. Compared to existing methods that require the subject to frequently manually adjust the camera, this embodiment can capture the subject in a single-person shooting scenario using a camera mounted on a robotic arm, thereby improving shooting efficiency and the quality of the captured content.

[0045] refer to Figure 2 , Figure 2 This is a flowchart illustrating the second embodiment of the shooting method of this application.

[0046] Based on the first embodiment described above, in this embodiment, step S10 includes: Step S101: Obtain a captured image of the area where the subject is located, and determine the subject's posture information based on the captured image.

[0047] Understandably, a camera can capture images of the area where the subject is located, and the area can include the subject's complete three-dimensional state, and the subject's posture information can be determined based on the captured images.

[0048] Further, in this embodiment, step S101 includes: acquiring a captured image of the shooting area where the subject is located, the captured image including a depth image and a color image; extracting two-dimensional coordinates of joints from the color image using a key point detection model; converting the two-dimensional coordinates of joints into three-dimensional coordinates of joints based on the pixel depth values ​​corresponding to the depth image; and determining the pose information of the subject based on the three-dimensional coordinates of joints.

[0049] It should be understood that images of the subject's location can be continuously captured using an RGB-D depth camera at a frame rate of ≥30fps. These images include both depth and color images. A keypoint detection model is then used to extract the two-dimensional coordinates of joints from the color image. This keypoint detection model can be a lightweight OpenPose variant or other human pose estimation model, capable of extracting the spatial coordinates of more than 18 joints, such as shoulders, elbows, wrists, hips, knees, and ankles, from the RGB color image. These two-dimensional joint coordinates are then combined with the pixel depth values ​​from the depth image to convert them into three-dimensional joint coordinates, and the subject's pose information is determined based on these three-dimensional joint coordinates.

[0050] Furthermore, in order to effectively determine the subject's posture information, in this embodiment, determining the subject's posture information based on the three-dimensional coordinates of the joint points includes: selecting the core joint coordinates of the torso from the three-dimensional joint coordinates, and determining the three-dimensional centroid coordinates of the subject based on the core joint coordinates of the torso; calculating the vertical distance ratio between the head and the centroid based on the three-dimensional centroid coordinates and the three-dimensional head coordinates of the subject; constructing a torso direction vector based on the three-dimensional joint coordinates, and determining the torso tilt angle based on the angle between the torso direction vector and the vertical reference vector; determining the body orientation angle based on the shoulder three-dimensional coordinates in the three-dimensional joint coordinates; and determining the subject's posture information based on the vertical distance ratio, the torso tilt angle, and the body orientation angle.

[0051] Understandably, the coordinates of the core joints of the torso can be selected from the three-dimensional coordinates of the joints. The coordinates of the core joints of the torso can include the coordinates of the left shoulder, right shoulder, left hip, right hip, etc. The coordinates of each core joint of the torso are weighted and summed to obtain the three-dimensional centroid coordinates of the subject.

[0052] It should be understood that the vertical distance can be calculated based on the three-dimensional centroid coordinates and the three-dimensional head coordinates. The vertical distance ratio is the vertical distance normalized to the torso length and can be determined based on the vertical distance, shoulder center coordinates, and hip center coordinates. The torso direction vector is the vector pointing from the center of the hip to the center of the shoulder. The vertical reference vector is the direction of gravity, i.e., the positive Z-axis direction of the world coordinate system. The angle between the torso direction vector and the vertical reference vector is determined and used as the torso tilt angle. The yaw angle of the body can be determined using the direction of the line connecting the left and right shoulders, i.e., the body's orientation in the horizontal plane (XY plane), as the orientation angle.

[0053] In practice, the subject's posture information can be determined by combining the vertical distance ratio, torso tilt angle, and body facing angle. For example, if the rate of change of facing angle is >40° / s and the torso tilt angle is <15°, the posture information is turning around; if the torso tilt angle is >30° and the torso tilt angle is <75°, the posture information is bending over; if the vertical distance ratio is <0.55 and the hip Z-coordinate is <knee Z-coordinate -0.05m, the posture information is sitting down; and in other cases, the posture information is standing.

[0054] Step S102: Determine the attitude state vector based on the attitude information.

[0055] Further, in this embodiment, step S102 includes: acquiring IMU data collected by the IMU sensor worn by the subject, the IMU data including acceleration and angular velocity; updating the attitude information according to the acceleration and the angular velocity to obtain updated attitude information; and determining an attitude state vector based on the updated attitude information.

[0056] Understandably, to address the issues of occlusion and measurement lag that occur when depth cameras are used to photograph subjects moving rapidly, IMU data collected by an IMU sensor worn by the subject can be acquired. This IMU data may include acceleration and angular velocity.

[0057] It should be understood that Extended Kalman Filtering (EKF) can be used for sensor-level data fusion: IMU data drives the prediction step, updating velocity and attitude states; visual observations drive the update step, correcting position and orientation. The EKF state vector contains nine dimensions (3D position, 3D velocity, yaw angle, pitch angle, and roll angle), with process noise and measurement noise parameters set to 0.01 and 0.1 respectively, resulting in updated attitude information. Extended Kalman Filtering is a nonlinear state estimation algorithm that linearizes the nonlinear system using a first-order Taylor expansion, optimally fusing visual observations from the depth camera with inertial measurements from the IMU, eliminating measurement biases from a single sensor during occlusion or rapid movement. Finally, the attitude state vector is determined based on the updated attitude information.

[0058] Step S103: Obtain the voice command issued by the subject and determine the voice intent vector based on the voice command.

[0059] Understandably, the audio stream captured by the microphone array, i.e., the voice commands issued by the subject, can be converted into text in real time by the lightweight speech recognition model Whisper Tiny deployed on the edge computing unit. Subsequently, the semantic parsing module performs structured analysis on the text, extracting two types of key information: first, the camera operation intention, including 10 basic intention types such as zoom in (ZOOM_IN), zoom out (ZOOM_OUT), high-angle shot (HIGH_ANGLE), side shot (SIDE_SHOT), and follow (FOLLOW); second, the intensity parameter, which is mapped to three levels of intensity adverbs such as point, large amplitude, and fast speed: 0.3 (slight), 0.6 (moderate), and 1.0 (large amplitude), thus forming a speech intention vector. The speech is 16-dimensional, containing one-hot encoding of the intention and the intensity parameter.

[0060] Step S104: Determine the target camera position vector of the shooting device based on the posture state vector and the voice intent vector.

[0061] It should be understood that the speech intent vector and the posture state vector can be fused through a cross-attention mechanism to obtain the target camera position vector of the shooting device.

[0062] Further, in this embodiment, step S104 includes: using the voice intent vector as a key value, using the posture state vector as a query, and calculating a fused feature vector through an attention mechanism; inputting the fused feature vector into a decision network to obtain a target camera position vector, wherein the target camera position vector includes the joint angle value of the robotic arm and the optical zoom ratio of the camera mounted at the end of the robotic arm.

[0063] Understandably, the speech intent vector can be used as the key and value, and the posture state vector as the query. A fused feature vector is calculated after attention weights are applied, achieving dynamic alignment and weighted fusion of the speech intent vector and posture state vector. This fused feature vector is input into a pre-trained camera position decision Transformer. After inference through a multi-head self-attention layer and a feedforward network, it outputs a target camera position vector with seven dimensions. The first six dimensions represent the target angle values ​​of the six joints of the robotic arm, and the seventh dimension represents the optical zoom ratio. This process enables precise control of the physical device using natural language commands such as "Push it closer."

[0064] This embodiment acquires images of the shooting area where the subject is located, determines the subject's posture information based on the images, then determines a posture state vector based on the posture information, acquires the subject's voice commands, determines a voice intent vector based on the voice commands, and finally determines the target camera position vector of the shooting device based on the posture state vector and the voice intent vector. This embodiment determines the posture state vector based on the subject's posture information, determines the voice intent vector based on the voice commands, and then fuses the posture state vector and the voice intent vector to improve the accuracy of determining the target camera position vector of the shooting device.

[0065] refer to Figure 3 , Figure 3 This is a flowchart illustrating the third embodiment of the shooting method of this application.

[0066] Based on the above embodiments, in this embodiment, step S30 includes: Step S301: During the process of photographing the subject based on the joint motion trajectory and the position of the optical zoom motor, obtain the behavior sequence corresponding to the subject.

[0067] Understandably, during the process of photographing the subject based on the joint movement trajectory and the position of the optical zoom motor, the subject's corresponding behavioral sequence can also be monitored in real time, i.e., behavioral observation record.

[0068] Further, in this embodiment, step S301 includes: during the process of photographing the subject based on the joint motion trajectory and the position of the optical zoom motor, acquiring the corresponding shooting image of the subject; determining the position offset of the subject based on the shooting image, the position offset including horizontal offset and vertical offset; fine-tuning the position of the robotic arm based on the horizontal offset and the vertical offset, and acquiring the corresponding behavior sequence of the subject based on the fine-tuned position.

[0069] It should be understood that during the process of photographing the subject based on joint motion trajectories and the position of the optical zoom motor, the corresponding captured image can be continuously monitored through a real-time image analysis module. The positional offset of the subject can be determined based on the captured image. The positional offset can include horizontal offset and vertical offset. Horizontal offset refers to the offset of the subject from the target position in the horizontal direction, and vertical offset refers to the offset of the subject from the target position in the vertical direction.

[0070] In practice, the position of the robotic arm can be fine-tuned based on horizontal and vertical offsets, and the position of the camera can also be fine-tuned, so as to obtain the behavior sequence of the subject at the fine-tuned position.

[0071] Furthermore, in this embodiment, determining the positional offset of the subject based on the captured image includes: detecting the captured image using a target detection model to obtain the actual position of the subject; determining the target position of the subject according to the composition rules corresponding to the subject; comparing the actual position with the target position, and determining the positional offset of the subject based on the comparison result.

[0072] Understandably, the captured image can be detected using an object detection model, such as the YOLOv8n model, to locate the center coordinates and area proportion of the bounding box of the subject. Alternatively, a facial landmark detection model can be used to determine the facial position, thereby obtaining the actual position of the subject in the captured image (actual center X, center Y).

[0073] It should be understood that the target position of the subject can be determined according to the composition rules corresponding to the subject. The composition template supports four rules, including the golden ratio (target position 0.382, 0.382), center (0.5, 0.5), rule of thirds (0.333, 0.333), and white space guidance (0.4, 0.5). The golden ratio composition is used by default.

[0074] In practice, the actual horizontal position is compared with the target horizontal position to obtain the horizontal offset of the subject. Similarly, the actual vertical position is compared with the target vertical position to obtain the vertical offset of the subject.

[0075] Furthermore, in this embodiment, the step of fine-tuning the position of the robotic arm based on the horizontal offset and the vertical offset, and obtaining the behavioral sequence corresponding to the subject based on the fine-tuned position, includes: determining the offset direction when the horizontal offset is greater than a preset horizontal threshold or the vertical offset is greater than a preset vertical threshold, wherein the offset direction includes a horizontal offset direction and a vertical offset direction; determining a compensation direction based on the offset direction; fine-tuning the position of the robotic arm based on the compensation direction and a preset fine-tuning step size, and obtaining the behavioral sequence corresponding to the subject based on the fine-tuned position.

[0076] Understandably, if the horizontal offset exceeds a preset horizontal threshold or the vertical offset exceeds a preset vertical threshold, it indicates that the composition needs to be corrected. The preset horizontal and vertical thresholds can be manually set beforehand. If the composition needs correction, the offset direction is determined. The offset direction can include both horizontal and vertical offset directions. The horizontal offset direction refers to the offset direction of the actual position relative to the target position in the horizontal direction, and the vertical offset direction refers to the offset direction of the actual position relative to the target position in the vertical direction.

[0077] It should be understood that the compensation direction can be determined based on the offset direction. The compensation direction is opposite to the offset direction; for example, when the horizontal offset direction is to the left, the horizontal compensation direction is to the right; when the vertical offset direction is upward, the vertical compensation direction is downward. The position of the robotic arm is then fine-tuned based on the compensation direction and a preset fine-tuning step size, which can be 0.5 compensation units. The corresponding behavioral sequence of the subject is then obtained based on the fine-tuned position. The perceived data for each frame can include posture type, body orientation, and centroid coordinates. This sub-pixel-level fine-tuning ensures the accuracy of the image composition, maintaining the preset composition effect in real time even when the subject is moving.

[0078] Step S302: Analyze the behavior sequence using a behavior sequence classification model to obtain the initial shooting pattern and corresponding confidence level of the subject.

[0079] Understandably, the behavior sequence can be a record of the subject's behavior observations, including the contextual history of each frame of perceptual data and voice commands. The behavior sequence can be maintained in a sliding window buffer of length 30. After the buffer accumulates at least 10 frames of observation data, the entire behavior sequence is analyzed by an LSTM behavior sequence classification model to obtain the subject's initial shooting mode and corresponding confidence level. The initial shooting mode can include frontal half-body, side full-body, overhead shot of a table, and following movement. The confidence level represents the accuracy of the initial shooting mode output by the behavior sequence classification model.

[0080] Step S303: Determine a new target camera position vector based on the confidence level and the initial shooting mode, return to the step of determining the joint motion trajectory of the robotic arm and the position of the optical zoom motor of the camera based on the target camera position vector, and take a picture of the subject based on the new joint motion trajectory and the new optical zoom motor position.

[0081] It should be understood that the confidence level and the initial shooting mode can be used to determine whether the target camera position vector needs to be adjusted. If so, a new target camera position vector is determined, and the steps of determining the joint motion trajectory of the robotic arm and the position of the optical zoom motor of the camera based on the new target camera position vector are returned to obtain the new joint motion trajectory and the new optical zoom motor position. The subject is then photographed based on the new joint motion trajectory and the new optical zoom motor position to achieve seamless shooting mode switching.

[0082] Further, in this embodiment, determining a new target camera position vector based on the confidence level and the initial shooting mode includes: if the confidence level is less than or equal to a preset threshold, returning to the step of determining the target camera position vector of the shooting device based on the subject's posture information, and determining a new target camera position vector; if the confidence level is greater than the preset threshold, using the initial shooting mode as the target shooting mode; and determining a new target camera position vector based on the preset camera position parameters corresponding to the target shooting mode.

[0083] Understandably, if the confidence level is less than or equal to the preset threshold, it indicates that the current behavior pattern of the subject is unclear or has not changed. In this case, the process returns to the steps described above for determining the target camera position vector of the shooting device based on the subject's posture information, and a new target camera position vector is determined.

[0084] It should be understood that when the confidence level is greater than the preset threshold, it indicates that the subject's behavior pattern has changed significantly, such as changing from standing to sitting. In this case, the initial shooting mode can be used as the target shooting mode. The corresponding preset camera position parameters are loaded according to the target shooting mode. The shooting modes include: front half-body mode (joint angle [0, -0.3,0, 0, 0, 0], zoom 2.0x), side full-body mode ([0.8, 0, 0, 0, 0, 0], zoom 1.0x), top-down table view mode ([0, -1.2, 0, 0, 0, 0], zoom 1.5x), and following walking mode ([0, -0.1, 0, 0, 0, 0], zoom 1.2x). The preset camera position parameters are then used as the new target camera position vector.

[0085] In the specific implementation, refer to Figure 4 , Figure 4 This is a schematic diagram of the overall process of one embodiment of the shooting method of this application, as shown below. Figure 4 As shown, the shooting method proposed in this embodiment includes the following steps: S1: Real-time perception of multimodal photographer status; S2: Voice command reception and semantic parsing; S3: Multi-axis collaborative trajectory planning of robotic arm and optical zoom; S4: Adaptive composition feedback closed-loop detection; If the composition deviation exceeds the threshold, then proceed to S5: Millimeter-level compensation motion execution; S6: Behavioral sequence context analysis; If the mode switching confidence exceeds the threshold, then proceed to S7: Automatic switching of shooting mode and loading of preset camera position.

[0086] This embodiment, during the process of photographing a subject based on joint motion trajectories and the position of the optical zoom motor, acquires the subject's corresponding behavioral sequence. Then, it analyzes the behavioral sequence using a behavioral sequence classification model to obtain the subject's initial shooting mode and corresponding confidence level. Based on the confidence level and the initial shooting mode, it determines a new target camera position vector. It then returns to the step of determining the robotic arm's joint motion trajectory and the camera's optical zoom motor position based on the target camera position vector, and finally photographs the subject based on the new joint motion trajectory and the new optical zoom motor position. This embodiment determines the initial shooting mode and corresponding confidence level based on the subject's behavioral sequence, and then determines a new target camera position vector based on the confidence level and the initial shooting mode. It can automatically predict shooting mode changes and preload corresponding preset camera position parameters, and then re-determine the new joint motion trajectory and the new optical zoom motor position based on the new target camera position vector. Therefore, even without voice commands, it can actively adjust the camera position like a professional photographer, simulating a photographer's predictive ability.

[0087] Reference Figure 5 , Figure 5 This is a structural block diagram of the first embodiment of the imaging device of this application.

[0088] like Figure 5 As shown, the imaging device proposed in this application includes: Vector determination module 10 is used to determine the target camera position vector of the shooting device based on the posture information of the subject. The shooting device includes a robotic arm and a camera mounted at the end of the robotic arm. The trajectory determination module 20 is used to determine the joint motion trajectory of the robotic arm and the position of the optical zoom motor of the camera based on the target position vector. The shooting module 30 is used to take pictures of the subject based on the joint movement trajectory and the position of the optical zoom motor.

[0089] This embodiment determines the target camera position vector of the shooting device based on the subject's posture information. The shooting device includes a robotic arm and a camera mounted on the end of the robotic arm. Then, based on the target camera position vector, it determines the joint motion trajectory of the robotic arm and the position of the camera's optical zoom motor. Finally, it captures the subject based on the joint motion trajectory and the position of the optical zoom motor. This embodiment first determines the target camera position vector of the shooting device based on the subject's posture information, automatically determining the target camera position vector that the shooting device needs to achieve. Then, it captures the subject based on the joint motion trajectory of the robotic arm and the position of the camera's optical zoom motor. Compared to existing methods that require the subject to frequently manually adjust the camera, this embodiment can capture the subject in a single-person shooting scenario using a camera mounted on a robotic arm, thereby improving shooting efficiency and the quality of the captured content.

[0090] It should be noted that the workflow described above is merely illustrative and does not limit the scope of protection of this application. In practical applications, those skilled in the art can select some or all of it to achieve the purpose of this embodiment according to actual needs, and no restrictions are imposed here.

[0091] In addition, for technical details not described in detail in this embodiment, please refer to the shooting method provided in any embodiment of this application, which will not be repeated here.

[0092] Based on the first embodiment of the shooting device described in this application, a second embodiment of the shooting device of this application is proposed.

[0093] In this embodiment, the vector determination module 10 is further configured to acquire a captured image of the shooting area where the subject is located, and determine the subject's posture information based on the captured image; determine a posture state vector based on the posture information; acquire a voice command issued by the subject, and determine a voice intent vector based on the voice command; and determine the target camera position vector of the shooting device based on the posture state vector and the voice intent vector.

[0094] Furthermore, the vector determination module 10 is also used to acquire a captured image of the shooting area where the subject is located, the captured image including a depth image and a color image; extract two-dimensional coordinates of joints from the color image using a key point detection model; convert the two-dimensional coordinates of joints into three-dimensional coordinates of joints according to the pixel depth values ​​corresponding to the depth image; and determine the pose information of the subject based on the three-dimensional coordinates of joints.

[0095] Furthermore, the vector determination module 10 is also used to select the core joint coordinates of the torso from the three-dimensional coordinates of the joints, and determine the three-dimensional centroid coordinates of the subject based on the core joint coordinates of the torso; calculate the vertical distance ratio between the head and the centroid based on the three-dimensional centroid coordinates and the three-dimensional head coordinates of the subject; construct a torso direction vector based on the three-dimensional coordinates of the joints, and determine the torso tilt angle based on the angle between the torso direction vector and the vertical reference vector; determine the body orientation angle based on the three-dimensional coordinates of the shoulders in the three-dimensional coordinates of the joints; and determine the posture information of the subject based on the vertical distance ratio, the torso tilt angle, and the body orientation angle.

[0096] Furthermore, the vector determination module 10 is also used to acquire IMU data collected by the IMU sensor worn by the subject, the IMU data including acceleration and angular velocity; update the attitude information according to the acceleration and the angular velocity to obtain updated attitude information; and determine the attitude state vector based on the updated attitude information.

[0097] Furthermore, the vector determination module 10 is also used to use the voice intent vector as the key value, the posture state vector as the query value, and to calculate the fused feature vector through an attention mechanism; the fused feature vector is input into the decision network to obtain the target camera position vector, the target camera position vector including the joint angle value of the robotic arm and the optical zoom ratio of the camera mounted at the end of the robotic arm.

[0098] Furthermore, the trajectory determination module 20 is also used to determine the target joint position of the robotic arm based on the target camera position vector, and to determine the current joint position of the robotic arm; to determine the joint motion trajectory of the robotic arm based on the current joint position and the target joint position; and to determine the position of the optical zoom motor based on the change in distance between the subject and the camera during the movement of the robotic arm according to the joint motion trajectory.

[0099] Furthermore, the shooting module 30 is also used to acquire the behavior sequence corresponding to the subject during the process of shooting the subject based on the joint motion trajectory and the position of the optical zoom motor; analyze the behavior sequence through a behavior sequence classification model to obtain the initial shooting mode and corresponding confidence level of the subject; determine a new target camera position vector based on the confidence level and the initial shooting mode; return to the step of determining the joint motion trajectory of the robotic arm and the position of the optical zoom motor of the camera based on the target camera position vector; and shoot the subject based on the new joint motion trajectory and the new optical zoom motor position.

[0100] Furthermore, the shooting module 30 is also used to acquire a shooting image corresponding to the subject during the process of shooting the subject based on the joint movement trajectory and the position of the optical zoom motor; determine the position offset of the subject based on the shooting image, the position offset including horizontal offset and vertical offset; fine-tune the position of the robotic arm based on the horizontal offset and the vertical offset, and acquire the behavior sequence corresponding to the subject based on the fine-tuned position.

[0101] Furthermore, the shooting module 30 is also used to detect the shooting scene through a target detection model to obtain the actual position of the subject; determine the target position of the subject according to the composition rules corresponding to the subject; compare the actual position with the target position, and determine the position offset of the subject according to the comparison result.

[0102] Furthermore, the shooting module 30 is also used to determine the offset direction when the horizontal offset is greater than a preset horizontal threshold or the vertical offset is greater than a preset vertical threshold, the offset direction including the horizontal offset direction and the vertical offset direction; determine the compensation direction according to the offset direction; fine-tune the position of the robotic arm based on the compensation direction and the preset fine-tuning step size, and obtain the behavior sequence corresponding to the subject based on the fine-tuned position.

[0103] Furthermore, the shooting module 30 is also configured to, when the confidence level is less than or equal to a preset threshold, return to the step of determining the target camera position vector of the shooting device based on the subject's posture information, and determine a new target camera position vector; when the confidence level is greater than the preset threshold, use the initial shooting mode as the target shooting mode; and determine a new target camera position vector based on the preset camera position parameters corresponding to the target shooting mode.

[0104] Other embodiments or specific implementations of the imaging device of this application can be found in the above-described method embodiments, and will not be repeated here.

[0105] This application provides a shooting device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the shooting method in the above embodiment 1.

[0106] The following is for reference. Figure 6 The diagram illustrates a structural schematic of a shooting device suitable for implementing the embodiments of this application. The shooting device in the embodiments of this application may include, but is not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Descriptions), PMPs (Portable Media Players), in-vehicle terminals (e.g., in-vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers. Figure 6 The shooting device shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of this application.

[0107] like Figure 6As shown, the imaging device may include a processing unit 1001 (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1002 or a program loaded from a storage device 1003 into a random access memory (RAM) 1004. The RAM 1004 also stores various programs and data required for the operation of the imaging device. The processing unit 1001, ROM 1002, and RAM 1004 are interconnected via a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Typically, the following systems can be connected to the I / O interface 1006: input devices 1007 including, for example, a touchscreen, touchpad, keyboard, mouse, image sensor, microphone, accelerometer, gyroscope, etc.; output devices 1008 including, for example, a liquid crystal display (LCD), speaker, vibrator, etc.; storage devices 1003 including, for example, magnetic tape, hard disk, etc.; and communication devices 1009. Communication device 1009 allows the capturing device to communicate wirelessly or wiredly with other devices to exchange data. Although capturing devices with various systems are shown in the figures, it should be understood that implementing or possessing all of the systems shown is not required. More or fewer systems may be implemented alternatively.

[0108] Specifically, according to the embodiments disclosed in this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments disclosed in this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device, or installed from storage device 1003, or installed from ROM 1002. When the computer program is executed by processing device 1001, it performs the functions defined in the methods of the embodiments disclosed in this application.

[0109] The shooting device provided in this application, employing the shooting method described in the above embodiments, can solve the technical problem that in single-person shooting scenarios, the subject needs to frequently manually adjust the camera, resulting in low shooting efficiency and poor content quality. Compared with the prior art, the beneficial effects of the shooting device provided in this application are the same as those of the shooting method provided in the above embodiments, and other technical features of the shooting device are the same as those disclosed in the previous embodiment method, and will not be repeated here.

[0110] It should be understood that the various parts disclosed in this application can be implemented using hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics can be combined in any suitable manner in one or more embodiments or examples.

[0111] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0112] This application provides a computer-readable storage medium having computer-readable program instructions (i.e., a computer program) stored thereon, the computer-readable program instructions being used to execute the shooting method in the above embodiments.

[0113] The computer-readable storage medium provided in this application may be, for example, a USB flash drive, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, system, or device. The program code contained on the computer-readable storage medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination thereof.

[0114] The aforementioned computer-readable storage medium may be included in the shooting device; or it may exist independently and not be assembled into the shooting device.

[0115] The aforementioned computer-readable storage medium carries one or more programs that, when executed by the imaging device, cause the imaging device to: determine a target camera position vector based on the subject's posture information, the imaging device including a robotic arm and a camera mounted at the end of the robotic arm; determine the joint motion trajectory of the robotic arm and the position of the optical zoom motor of the camera based on the target camera position vector; and capture an image of the subject based on the joint motion trajectory and the position of the optical zoom motor.

[0116] Computer program code for performing the operations of this application can be written in one or more programming languages ​​or a combination thereof. These programming languages ​​include object-oriented programming languages—such as Python, Java, Smalltalk, and C++—and conventional procedural programming languages—such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a Local Area Network (LAN) or a Wide Area Network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0117] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0118] The modules described in the embodiments of this application can be implemented in software or hardware. The names of the modules do not necessarily limit the functionality of the unit itself.

[0119] The readable storage medium provided in this application is a computer-readable storage medium that stores computer-readable program instructions (i.e., a computer program) for executing the above-described shooting method. This solves the technical problem that in single-person shooting scenarios, the subject needs to frequently manually adjust the camera, resulting in low shooting efficiency and poor content quality. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in this application are the same as those of the shooting method provided in the above embodiments, and will not be repeated here.

[0120] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the shooting method described above.

[0121] The computer program product provided in this application can solve the technical problem that in single-person shooting scenarios, the subject needs to frequently manually adjust the camera, resulting in low shooting efficiency and poor content quality. Compared with the prior art, the beneficial effects of the computer program product provided in this application are the same as the beneficial effects of the shooting method provided in the above embodiments, and will not be repeated here.

[0122] All user-related data involved in this application (e.g., captured images, voice commands, etc.) was obtained with the user's permission or consent. In other words, when this application is applied to specific products or technologies, user permission is required to acquire and process the relevant data, and the processing of the data must comply with the relevant laws, regulations, and regulatory standards of the relevant countries and regions. For example, when it is necessary to obtain the user's current geographical location, a location acquisition prompt can be displayed on the user's terminal. After receiving confirmation from the user regarding the location acquisition prompt, the terminal can obtain the user's current geographical location.

[0123] The above description is only a part of the embodiments of this application and does not limit the scope of protection of this application. All equivalent structural transformations made under the technical concept of this application and using the content of this application specification and drawings, or direct / indirect applications in other related technical fields, are included in the scope of protection of this application.

[0124] This application provides A1. A shooting method, the shooting method comprising the following steps: The target camera position vector of the shooting device is determined based on the posture information of the subject. The shooting device includes a robotic arm and a camera mounted at the end of the robotic arm. The joint motion trajectory of the robotic arm and the position of the optical zoom motor of the camera are determined based on the target position vector. The subject is photographed based on the joint movement trajectory and the position of the optical zoom motor.

[0125] A2. The shooting method as described in A1, wherein determining the target camera position vector of the shooting device based on the subject's posture information includes: Acquire images of the shooting area where the subject is located, and determine the subject's posture information based on the captured images; Determine the attitude state vector based on the attitude information; Acquire the voice commands issued by the subject and determine the voice intent vector based on the voice commands; The target camera position vector of the shooting device is determined based on the posture state vector and the voice intent vector.

[0126] A3. The shooting method as described in A2, wherein acquiring a shooting image of the shooting area where the subject is located, and determining the subject's posture information based on the shooting image, includes: Acquire images of the area where the subject is located, including depth images and color images; Two-dimensional coordinates of joint points are extracted from the color image using a key point detection model. Based on the pixel depth values ​​corresponding to the depth image, the two-dimensional coordinates of the joints are converted into three-dimensional coordinates. The pose information of the subject is determined based on the three-dimensional coordinates of the joints.

[0127] A4. The shooting method as described in A3, wherein determining the posture information of the subject based on the three-dimensional coordinates of the joint points includes: Select the core joint coordinates of the torso from the three-dimensional coordinates of the joints, and determine the three-dimensional centroid coordinates of the subject based on the core joint coordinates of the torso. Calculate the ratio of the vertical distance between the head and the centroid based on the three-dimensional centroid coordinates and the three-dimensional head coordinates of the subject; A trunk direction vector is constructed based on the three-dimensional coordinates of the joint points, and the trunk tilt angle is determined based on the angle between the trunk direction vector and the vertical reference vector. The body orientation angle is determined based on the shoulder's three-dimensional coordinates in the joint's three-dimensional coordinates. The subject's posture information is determined based on the vertical distance ratio, the torso tilt angle, and the body orientation angle.

[0128] A5. The shooting method as described in A2, wherein determining the attitude state vector based on the attitude information includes: Acquire IMU data collected by the IMU sensor worn by the subject, the IMU data including acceleration and angular velocity; The attitude information is updated based on the acceleration and the angular velocity to obtain the updated attitude information; The attitude state vector is determined based on the updated attitude information.

[0129] A6. The shooting method as described in A2, wherein determining the target camera position vector of the shooting device based on the posture state vector and the voice intent vector includes: The speech intent vector is used as the key, the pose state vector is used as the query, and the fused feature vector is calculated through an attention mechanism. The fused feature vector is input into the decision network to obtain the target camera position vector, which includes the joint angle value of the robotic arm and the optical zoom ratio of the camera mounted at the end of the robotic arm.

[0130] A7. The shooting method as described in any one of A1 to A6, wherein determining the joint motion trajectory of the robotic arm and the position of the optical zoom motor of the camera based on the target camera position vector includes: The target joint position of the robotic arm is determined based on the target position vector, and the current joint position of the robotic arm is determined. The joint motion trajectory of the robotic arm is determined based on the current joint position and the target joint position; During the movement of the robotic arm along the joint trajectory, the position of the optical zoom motor is determined based on the change in distance between the subject and the camera.

[0131] A8. The shooting method as described in any one of A1 to A6, wherein shooting the subject based on the joint movement trajectory and the position of the optical zoom motor includes: During the process of photographing the subject based on the joint motion trajectory and the position of the optical zoom motor, the corresponding behavior sequence of the subject is obtained; The behavior sequence is analyzed using a behavior sequence classification model to obtain the subject's initial shooting pattern and corresponding confidence level. Based on the confidence level and the initial shooting mode, a new target camera position vector is determined, and the steps of determining the joint motion trajectory of the robotic arm and the position of the optical zoom motor of the camera based on the target camera position vector are returned. The subject is then photographed based on the new joint motion trajectory and the new optical zoom motor position.

[0132] A9. The shooting method as described in A8, wherein during the process of shooting the subject based on the joint motion trajectory and the position of the optical zoom motor, obtaining the behavioral sequence corresponding to the subject includes: During the process of photographing the subject based on the joint movement trajectory and the position of the optical zoom motor, the corresponding shooting image of the subject is acquired; The positional offset of the subject is determined based on the captured image, and the positional offset includes horizontal offset and vertical offset; The position of the robotic arm is fine-tuned based on the horizontal and vertical offsets, and the behavioral sequence of the subject is obtained based on the fine-tuned position.

[0133] A10. The shooting method as described in A9, wherein determining the positional offset of the subject based on the captured image includes: The actual position of the subject is obtained by detecting the captured image using a target detection model. The target position of the subject is determined according to the composition rules corresponding to the subject. The actual position is compared with the target position, and the positional offset of the subject is determined based on the comparison result.

[0134] A11. The shooting method as described in A9, wherein the step of fine-tuning the position of the robotic arm based on the horizontal offset and the vertical offset, and obtaining the behavioral sequence corresponding to the subject based on the fine-tuned position, includes: If the horizontal offset is greater than a preset horizontal threshold or the vertical offset is greater than a preset vertical threshold, the offset direction is determined, and the offset direction includes the horizontal offset direction and the vertical offset direction. The compensation direction is determined based on the offset direction; The position of the robotic arm is fine-tuned based on the compensation direction and the preset fine-tuning step size, and the behavior sequence corresponding to the subject is obtained based on the fine-tuned position.

[0135] A12. The shooting method as described in A8, wherein determining the new target camera position vector based on the confidence level and the initial shooting mode includes: If the confidence level is less than or equal to a preset threshold, return to the step of determining the target camera position vector of the shooting device based on the subject's posture information, and determine a new target camera position vector; If the confidence level is greater than a preset threshold, the initial shooting mode is taken as the target shooting mode; A new target camera position vector is determined based on the preset camera position parameters corresponding to the target shooting mode.

[0136] This application also provides B13. A shooting device, the shooting device comprising: A vector determination module is used to determine the target camera position vector of the shooting device based on the posture information of the subject. The shooting device includes a robotic arm and a camera mounted at the end of the robotic arm. The trajectory determination module is used to determine the joint motion trajectory of the robotic arm and the position of the optical zoom motor of the camera based on the target camera position vector. The shooting module is used to take pictures of the subject based on the joint movement trajectory and the position of the optical zoom motor.

[0137] B14. The shooting device as described in B13, wherein the vector determination module is further configured to acquire a shooting image of the shooting area where the subject is located, and determine the posture information of the subject based on the shooting image; determine a posture state vector based on the posture information; acquire a voice command issued by the subject, and determine a voice intent vector based on the voice command; and determine a target camera position vector of the shooting device based on the posture state vector and the voice intent vector.

[0138] B15. The shooting device as described in B14, wherein the vector determination module is further configured to acquire a shooting image of the shooting area where the subject is located, the shooting image including a depth image and a color image; extract two-dimensional coordinates of joint points from the color image using a key point detection model; convert the two-dimensional coordinates of joint points into three-dimensional coordinates of joint points according to the pixel depth values ​​corresponding to the depth image; and determine the posture information of the subject based on the three-dimensional coordinates of joint points.

[0139] B16. The shooting apparatus as described in any one of B13 to B15, wherein the trajectory determination module is further configured to determine the target joint position of the robotic arm based on the target camera position vector, and determine the current joint position of the robotic arm; determine the joint motion trajectory of the robotic arm based on the current joint position and the target joint position; and determine the position of the optical zoom motor based on the change in distance between the subject and the camera during the movement of the robotic arm according to the joint motion trajectory.

[0140] B17. The shooting device as described in any one of B13 to B15, wherein the shooting module is further configured to: acquire a behavior sequence corresponding to the subject during the process of shooting the subject based on the joint motion trajectory and the position of the optical zoom motor; analyze the behavior sequence through a behavior sequence classification model to obtain the initial shooting mode and corresponding confidence level of the subject; determine a new target camera position vector based on the confidence level and the initial shooting mode; return to the step of determining the joint motion trajectory of the robotic arm and the position of the optical zoom motor of the camera based on the target camera position vector; and shoot the subject based on the new joint motion trajectory and the new optical zoom motor position.

[0141] This application also provides C18. An imaging device, the device comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the imaging method as described in any one of A1 to A12.

[0142] This application also provides D19. A storage medium, which is a computer-readable storage medium, on which a computer program is stored, which, when executed by a processor, implements the steps of the shooting method as described in any one of A1 to A12.

[0143] This application also provides E20. A computer program product comprising a computer program that, when executed by a processor, implements the steps of the shooting method as described in any one of A1 to A12.

Claims

1. A shooting method, characterized in that, The shooting method includes the following steps: The target camera position vector of the shooting device is determined based on the posture information of the subject. The shooting device includes a robotic arm and a camera mounted at the end of the robotic arm. The joint motion trajectory of the robotic arm and the position of the optical zoom motor of the camera are determined based on the target position vector. The subject is photographed based on the joint movement trajectory and the position of the optical zoom motor.

2. The shooting method as described in claim 1, characterized in that, Determining the target camera position vector of the shooting device based on the subject's posture information includes: Acquire images of the shooting area where the subject is located, and determine the subject's posture information based on the captured images; Determine the attitude state vector based on the attitude information; Acquire the voice commands issued by the subject and determine the voice intent vector based on the voice commands; The target camera position vector of the shooting device is determined based on the posture state vector and the voice intent vector.

3. The shooting method as described in claim 2, characterized in that, The step of acquiring a captured image of the area where the subject is located, and determining the subject's posture information based on the captured image, includes: Acquire images of the area where the subject is located, including depth images and color images; Two-dimensional coordinates of joint points are extracted from the color image using a key point detection model. Based on the pixel depth values ​​corresponding to the depth image, the two-dimensional coordinates of the joints are converted into three-dimensional coordinates. The pose information of the subject is determined based on the three-dimensional coordinates of the joints.

4. The shooting method according to any one of claims 1 to 3, characterized in that, Determining the joint motion trajectory of the robotic arm and the position of the optical zoom motor of the camera based on the target camera position vector includes: The target joint position of the robotic arm is determined based on the target position vector, and the current joint position of the robotic arm is determined. The joint motion trajectory of the robotic arm is determined based on the current joint position and the target joint position; During the movement of the robotic arm along the joint trajectory, the position of the optical zoom motor is determined based on the change in distance between the subject and the camera.

5. The shooting method according to any one of claims 1 to 3, characterized in that, The process of photographing the subject based on the joint motion trajectory and the position of the optical zoom motor includes: During the process of photographing the subject based on the joint motion trajectory and the position of the optical zoom motor, the corresponding behavior sequence of the subject is obtained; The behavior sequence is analyzed using a behavior sequence classification model to obtain the subject's initial shooting pattern and corresponding confidence level. Based on the confidence level and the initial shooting mode, a new target camera position vector is determined, and the steps of determining the joint motion trajectory of the robotic arm and the position of the optical zoom motor of the camera based on the target camera position vector are returned. The subject is then photographed based on the new joint motion trajectory and the new optical zoom motor position.

6. The shooting method as described in claim 5, characterized in that, The process of capturing the subject based on the joint motion trajectory and the position of the optical zoom motor, including obtaining the corresponding behavioral sequence of the subject, includes: During the process of photographing the subject based on the joint movement trajectory and the position of the optical zoom motor, the corresponding shooting image of the subject is acquired; The positional offset of the subject is determined based on the captured image, and the positional offset includes horizontal offset and vertical offset; The position of the robotic arm is fine-tuned based on the horizontal and vertical offsets, and the behavioral sequence of the subject is obtained based on the fine-tuned position.

7. A shooting device, characterized in that, The imaging device includes: A vector determination module is used to determine the target camera position vector of the shooting device based on the posture information of the subject. The shooting device includes a robotic arm and a camera mounted at the end of the robotic arm. The trajectory determination module is used to determine the joint motion trajectory of the robotic arm and the position of the optical zoom motor of the camera based on the target camera position vector. The shooting module is used to take pictures of the subject based on the joint movement trajectory and the position of the optical zoom motor.

8. A shooting device, characterized in that, The device includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the imaging method as described in any one of claims 1 to 6.

9. A storage medium, characterized in that, The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, it implements the steps of the shooting method as described in any one of claims 1 to 6.

10. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, implements the steps of the shooting method as described in any one of claims 1 to 6.

Images