Shooting auxiliary robot

Abstract

The invention relates to the technical field of shooting robots, in particular to a shooting assisting robot which comprises a moving chassis module, a lifting adjusting module, an angle adjusting module, a shooting module, a storage module, a multi-mode interaction module and a cooperative control module. The storage module is arranged on the movement chassis module, the storage module comprises a storage box, a storage control mechanism, a door opening and closing mechanism and a placement box, the storage box is arranged on the movement chassis module, an opening is formed in the storage box, and the storage control mechanism is used for controlling the placement box to be stored in the storage box or stretch out of the opening; the door opening and closing mechanism is used for sealing the opening after the placement box is stored in the storage box; by arranging the storage module and the storage control mechanism, the storage box can be controlled to be stored in the storage box, all the shooting assemblies can be effectively protected and prevented from being damaged by the outside, the core problems that traditional equipment is complex in operation, poor in environmental adaptability, insufficient in safety and the like are solved, and the shooting efficiency and the creation freedom degree are remarkably improved.

Description

Technical Field

[0001] The present invention relates to the technical field of shooting robots, and particularly to shooting assistance robots. Background Art

[0002] With the rapid development of the content creation industry, the demand for high-quality video content from creators such as video bloggers (vloggers), travel bloggers, and food bloggers is increasing day by day. However, the traditional shooting methods have the following problems: complex equipment: a large number of devices need to be carried, including cameras, tripods, stabilizers, etc., and the operation is cumbersome; high technical threshold: it is difficult for ordinary users to master professional shooting skills. At present, there are already some shooting assistance devices on the market, but these devices have single functions, low device integration, and do not have the function of integrating shooting control and movement control into one, and the coordination between movement and shooting is poor; the human-computer interaction efficiency is low, and the user operation is still based on traditional button operations, with a small operable range and insufficient control; the adaptability to the environment is insufficient, and it cannot maintain an efficient shooting effect in complex and changeable scenarios.

[0003] At the same time, since the shooting device is a relatively precise and expensive device, there may be a lack of effective protection measures for the shooting components in the prior art during non-use. During transportation or non-use, the shooting components are easily damaged by external factors such as collision, scratching, and dust. Summary of the Invention

[0004] The present invention provides a shooting assistance robot, and the technical problem to be solved is: solving the core problems of complex operation, poor environmental adaptability, and insufficient safety of traditional devices, significantly improving the shooting efficiency and creative freedom, applicable to various scenarios such as film and television production, live broadcast, and outdoor exploration, and by setting a storage module, the storage control mechanism can control the placement box to be stored in the storage box, and the door opening and closing mechanism seals the opening after the placement box is stored, which can effectively protect each shooting component and prevent it from being damaged by the outside world.

[0005] To achieve the above invention object, the technical solution adopted by the present invention is a shooting assistance robot, including a motion chassis module, a lifting adjustment module, an angle adjustment module, a camera module, a storage module, a multi-modal interaction module, and a collaborative control module;

[0006] The motion chassis module is provided with a driving motor and driving wheels, and the motion chassis module is provided with a terrain perception unit and a path planning unit;

[0007] The storage module is arranged on the moving chassis module. The storage module includes a storage box, a storage control mechanism, a door opening and closing mechanism, and a placement box. The storage box is arranged on the moving chassis module. There is an opening on the storage box. The storage control mechanism is used to control the placement box to be stored in the storage box or extended out of the opening, so as to achieve the effect of protecting each shooting component; the door opening and closing mechanism is used to seal the opening after the placement box is stored in the storage box;

[0008] The storage control mechanism includes a storage track, a sliding strip, a support rod, and a pushing cylinder. The storage track is arranged on the inner walls on both sides of the storage box. The storage track is a straight track. The sliding strip is slidably arranged in the storage track. The support rods are hinged on both sides of the sliding strip. The bottom of the placement box is hinged to the support rods. The pushing cylinder is arranged in the storage box. The pushing cylinder is fixedly connected to the sliding strip. There is a stop block in the storage box, below the opening;

[0009] When the pushing cylinder pushes the sliding strip to slide, it simultaneously drives the placement box to slide towards the opening. When the support rod touches the stop block, the support rod rotates, and then pushes the placement box to extend upward from the opening, so as to evenly perform shooting operations.

[0010] The door opening and closing mechanism includes a sealing door, a sealing track, a slider, a rotating gear, and a pushing rack. The sealing track is arranged on the inner walls on both sides of the storage box. The sealing track is a bent track, including two mutually perpendicular and connected horizontal tracks and vertical tracks. The slider is slidably arranged in the sealing track. There are a connecting arm one and a connecting arm two at the bottom of the sealing door. A rotating gear is arranged at the bottom of the connecting arm one. The rotating gear is rotatably arranged on the slider. There is a pulley at the bottom of the connecting arm two. The pulley is slidably arranged in the sealing track. The pushing rack is meshed with the rotating gear;

[0011] When the pushing rack is pushed, the rotating gear rotates, and also pushes the slider to slide, and then the sealing door reverses and moves out from the opening. When the pulley moves from the horizontal track to the vertical track, the sealing door also reverses, and then the sealing door is stored at the side wall of the storage box.

[0012] The pushing rack is connected to the pushing cylinder through a connecting rod. When the pushing cylinder extends, the sealing door is moved out from the opening and vertically stored on the inner wall of the storage box. At the same time, the placement box is pushed to the opening; when the pushing cylinder contracts, the placement box is stored in the storage box. At the same time, the sealing door is flipped and closed to the opening to seal the opening.

[0013] The lifting and adjusting module is arranged in the placement box. The lifting and adjusting module includes a plurality of folding frames that are hinged to each other. The folding frame brackets are controlled to fold and unfold by electric push rods, so as to control the lifting height; a height feedback sensor is arranged on the lifting and adjusting mechanism;

[0014] The angle adjustment module is arranged on the lifting adjustment module, and the angle adjustment module is a three-axis gimbal; an attitude sensor is arranged on the angle adjustment module;

[0015] The camera module is arranged on the angle adjustment module, and is provided with an optical zoom component and an image processing unit;

[0016] The multi-modal interaction module includes:

[0017] A voice recognition unit, configured to receive and parse user voice commands;

[0018] A gesture capture unit, configured to recognize user gesture actions;

[0019] A somatosensory capture unit, the somatosensory capture unit includes a wearable IMU sensor group and a depth camera array, and is configured to capture motion posture data of a user's arm and torso;

[0020] The collaborative control module is configured to: perform spatio-temporal alignment and semantic fusion on voice commands, gesture actions and somatosensory posture data to generate a composite control command;

[0021] Synchronously adjust the robot motion trajectory and camera shooting parameters according to the composite control command.

[0022] Further, the wearable IMU sensor group includes three nine-axis sensors respectively fixed on a user's wrist, elbow and shoulder, and is configured to collect limb joint angular velocity and acceleration data;

[0023] The depth camera array includes two groups of infrared TOF cameras, and is configured to construct three-dimensional coordinates of user bone joint points;

[0024] The somatosensory capture unit includes a data fusion processor, and the data fusion processor performs:

[0025] Receiving IMU raw data and depth image data;

[0026] Fusing IMU and visual data through an extended Kalman filter, and outputting a user's arm pitch angle ψ, torso yaw angle θ and motion speed v.

[0027] Further, the data fusion method of the somatosensory capture unit includes:

[0028] Establishing an upper limb kinematic model, which maps IMU data to joint angles of multiple degrees of freedom:

[0029] q = [q1, q2,..., q n T = f kinematic (a x , a y , a​z , ω x , ω y , ω z )

[0030] Wherein, a is the acceleration and ω is the angular velocity;

[0031] Calculate the target pose deviation through the inverse kinematics algorithm:

[0032] Δp = J(q)Δq

[0033] Wherein, J(q) is the Jacobian matrix, which is used to convert the joint space velocity into the operational space velocity.

[0034] Furthermore, the method for the collaborative control module to execute the somatosensory-gesture-speech instruction fusion includes:

[0035] Hierarchical decision-making architecture:

[0036] First priority: Somatosensory pose instruction, controlling the macroscopic movement of the robot;

[0037] Second priority: Gesture instruction, controlling the fine-tuning of the camera;

[0038] Third priority: Speech instruction, triggering mode switching;

[0039] Spatio-temporal constraint conditions: When the user's somatosensation and speech instructions are detected, activate the high-precision following mode. At this time: The moving speed of the robot is restricted; The zoom ratio of the camera automatically matches the target distance.

[0040] Furthermore, the control logic of the high-precision following mode includes:

[0041] Calculate the relative azimuth angle φ between the user and the target object in real time through the somatosensory data:

[0042]

[0043] Control the robot to move along a circular path with the target object as the center and radius R, and keep the user always in the tangential direction, where R is dynamically adjusted according to the amplitude of the user's outstretched arm.

[0044] The control method of the camera-assisted robot includes the somatosensory control stage:

[0045] Somatosensory data acquisition and preprocessing:

[0046] Obtain the original data of the acceleration and angular velocity of the user's upper limb joints through the IMU sensor group;

[0047] Generate the three-dimensional point cloud data of the user's skeletal joint points through the depth camera array;

[0048] Recognition of the motion intention:

[0049] Input the joint point movement trajectories of consecutive frames into the LSTM neural network to predict the future user movement direction and action type;

[0050] Multi-modal instruction fusion:

[0051] When the somatosensory predicted action is "turn around" and the voice instruction contains an orientation word (such as "shoot to the left"), generate an arc movement path for the robot, and the arc radius r satisfies:

[0052] r = v·t response + k·‖Δθ‖

[0053] where, v is the current speed, t_response = 0.3s is the system response time, Δθ is the turning angle, and k = 0.1m / deg is the calibration coefficient.

[0054] The beneficial effects of the present invention are as follows: By setting up a storage module, the storage control mechanism can control the placement box to be stored in the storage box, and the door opening and closing mechanism seals the opening after the placement box is stored, which can effectively protect each shooting component and prevent it from being damaged by the outside world; Through multi-modal natural interaction, it can be controlled collaboratively in three ways: somatosensory (body movements), gestures (dynamic instructions), and voice (natural language), simplifying the traditional operation process, reducing the multi-modal instruction response delay, and significantly improving the creation efficiency. The whole process is automated. In the high-precision following mode, the robot automatically matches the target distance to adjust the movement trajectory and camera focal length without manual intervention. Precise control and stability optimization, the motion control accuracy, the differential drive system realizes precise positioning, and the pan-tilt angle is accurately adjusted to ensure the stability of the picture composition. The inverse kinematics algorithm calculates the joint space speed in real time and dynamically compensates for the user posture deviation. Adapt to complex terrains, the improved A* algorithm supports path planning under terrains with a 25° slope, and the moving speed is adaptively adjusted. The environmental robustness is enhanced, and multi-sensor fusion is adapted; The dynamic priority is adjusted, and the modal weights are dynamically allocated according to the user distance to reduce the misoperation rate. The user feedback is enhanced, and the vibration motor provides multi-level tactile feedback to avoid the risk of misoperation. The advantages of multi-modal collaboration, hierarchical decision-making architecture, somatosensory instructions control the robot movement, gesture instructions adjust the camera parameters, and voice instructions switch modes to solve the problem of instruction conflicts. Brief Description of the Drawings

[0055] Figure 1 A perspective view of a structure of the auxiliary photography robot provided by an embodiment of the present invention;

[0056] Figure 2 A schematic diagram of the adjustment state of the camera module of a structure of the auxiliary photography robot provided by an embodiment of the present invention;

[0057] Figure 3Schematic diagram of the internal structure of the storage module of the present invention;

[0058] Figure 4 Multi-modal control flow chart of the shooting assistance robot of the present invention;

[0059] List of reference numerals for the drawings:

[0060] 1. Moving chassis module; 2. Lifting adjustment module; 3. Angle adjustment module; 4. Camera module; 5. Driving wheel; 6. Folding frame; 7. Electric push rod; 8. Storage module; 9. Storage box; 10. Storage control mechanism; 11. Door opening and closing mechanism; 12. Placing box; 13. Opening; 14. Storage track; 15. Slide bar; 16. Support rod; 17. Pushing cylinder; 18. Block; 19. Sealing door; 20. Sealing track; 21. Slide block; 22. Rotating gear; 23. Pushing rack; 24. Horizontal track; 25. Vertical track; 26. Connecting arm one; 27. Connecting arm two; 28. Pulley. Detailed implementation manners

[0061] The following further illustrates the detailed implementation manners of the present invention with reference to the drawings. The same components are denoted by the same reference numerals.

[0062] It should be noted that the terms "front", "rear", "left", "right", "up" and "down" used in the following description refer to the directions in the drawings, and the terms "inner" and "outer" respectively refer to the directions towards or away from the geometric center of a specific component.

[0063] In order to make the content of the present invention more clearly understood, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention.

[0064] Embodiment

[0065] The technical solution adopted by the present invention is a shooting assistance robot, including a moving chassis module 1, a lifting adjustment module 2, an angle adjustment module 3, a camera module 4, a storage module 7, a multi-modal interaction module and a cooperative control module;

[0066] The moving chassis module 1 is provided with a driving motor and driving wheels 5, and each driving wheel 5 is connected to a driving wheel 5. The driving wheel 5 can be a round wheel or a caterpillar wheel. The driving chassis module is provided with a differential drive system, and the steering and rotation of the vehicle are achieved by controlling the speed difference between the two driving motors through independent driving motors. When the two driving motors rotate forward at the same speed, the vehicle moves straight forward; if the speed of one driving motor is faster than that of the other, the vehicle will turn towards the side with the slower speed; when the two motors rotate at the same speed but in opposite directions, the vehicle will rotate in place;

[0067] The motion chassis module 1 is provided with a terrain perception unit and a path planning unit;

[0068] The storage module 8 is arranged on the motion chassis module 1. The storage module 8 includes a storage box 9, a storage control mechanism 10, a door opening and closing mechanism 11, and a placement box 12. The storage box 9 is arranged on the motion chassis module 1. An opening 13 is provided on the storage box 9. The storage control mechanism 10 is used to control the placement box 12 to be stored in the storage box 9 or extend out of the opening 13, so as to achieve the effect of protecting each photographed component; The door opening and closing mechanism 11 is used to seal the opening 13 after the placement box 12 is stored in the storage box 9;

[0069] The storage control mechanism 10 includes a storage track 14, a slide bar 15, a support rod 16, and a push cylinder 17. The storage track 14 is arranged on the inner walls on both sides inside the storage box 9. The storage track 14 is a straight track. The slide bar 15 is slidably arranged in the storage track 14. The support rod 16 is hinged on both sides of the slide bar 15. The bottom of the placement box 12 is hingedly connected to the support rod 16. The push cylinder 17 is arranged inside the storage box 9. The push cylinder 17 is fixedly connected to the slide bar 15. A stop block 18 is provided below the opening 13 inside the storage box 9;

[0070] When the push cylinder 17 pushes the slide bar 15 to slide, it simultaneously drives the placement box 12 to slide towards the opening 13. When the support rod 16 contacts the stop block 18, the support rod 16 rotates, and then pushes the placement box 12 to extend upward from the opening 13, so as to uniformly perform the photographing operation.

[0071] The door opening and closing mechanism 11 includes a sealing door 19, a sealing track 20, a slider 21, a rotating gear 22, and a pushing rack 23. The sealing track 20 is arranged on the inner walls on both sides inside the storage box 9. The sealing track 20 is a bent track, including two mutually perpendicular and connected horizontal tracks 24 and vertical tracks 25. The slider 21 is slidably arranged in the sealing track 20. The bottom of the sealing door 19 is provided with a connecting arm one 26 and a connecting arm two 27. A rotating gear 22 is arranged at the bottom of the connecting arm one 26. The rotating gear 22 is rotatably arranged on the slider 21. A pulley 28 is arranged at the bottom of the connecting arm two 27. The pulley 28 is slidably arranged in the sealing track 20. The pushing rack 23 meshes with the rotating gear 22;

[0072] When the pushing rack 23 is pushed, the rotating gear 22 rotates, and by also pushing the slider 21 to slide, the sealing door 19 rotates in reverse and moves out from the opening 13. When the pulley 28 moves from the horizontal track 24 to the vertical track 25, the sealing door 19 also rotates in reverse, and then the sealing door 19 is stored at the side wall inside the storage box 9.

[0073] The driving rack 23 is connected to the driving cylinder through a connecting rod. When the driving cylinder extends, the sealing door 19 is moved out of the opening 13 and vertically stored on the inner wall of the storage box 9. At the same time, the placement box 12 is pushed to the opening 13. When the driving cylinder contracts, the return box is stored in the storage box 9. At the same time, the sealing door 19 is flipped and closed to the opening 13 to seal the opening 13.

[0074] The lifting and adjusting module 2 is arranged in the placement box 11. The lifting and adjusting module 2 includes a plurality of folding frames 6 hinged to each other. The bottommost folding frame 6 is hinged to the moving chassis module 1. The folding of the folding frame 6 is controlled by the electric push rod 7 to control the lifting height. A height feedback sensor is arranged on the lifting and adjusting mechanism.

[0075] The angle adjusting module 3 is arranged on the lifting and adjusting module 2. The angle adjusting module 3 is a three-axis gimbal that can realize the adjustment of angle and position changes. An attitude sensor is arranged on the angle adjusting module 3.

[0076] The camera module 4 is detachably arranged on the angle adjusting module 3 and is provided with an optical zoom component and an image processing unit.

[0077] The multi-modal interaction module includes:

[0078] A voice recognition unit for receiving and parsing user voice commands. The voice data is obtained through a microphone.

[0079] A gesture capture unit for recognizing user gesture actions. The gesture action data is obtained through an infrared device.

[0080] A somatosensory capture unit. The somatosensory capture unit includes a wearable IMU sensor group and a depth camera array for capturing the motion posture data of the user's arms and torso.

[0081] The collaborative control module is configured to: perform spatio-temporal alignment and semantic fusion on the voice commands, gesture actions, and somatosensory posture data to generate a composite control command.

[0082] Synchronously adjust the robot's motion trajectory and the camera shooting parameters according to the composite control command.

[0083] Among them, the wearable IMU sensor group includes three nine-axis sensors respectively fixed on the user's wrist, elbow, and shoulder for collecting limb joint angular velocity and acceleration data.

[0084] The depth camera array includes two groups of infrared TOF cameras for constructing the three-dimensional coordinates of the user's bone joint points.

[0085] The somatosensory capture unit includes a data fusion processor, and the data fusion processor performs:

[0086] Receive the raw IMU data (sampling rate 100Hz) and depth image data (30fps);

[0087] Fuse the IMU and visual data through an extended Kalman filter, and output the pitch angle ψ of the user's arm (accuracy ±0.5°), the yaw angle θ of the torso (accuracy ±1°), and the movement speed v (accuracy ±0.1m / s).

[0088] The data fusion method of the somatosensory capture unit includes:

[0089] Establish an upper limb kinematic model, which maps the IMU data to joint angles of 7 degrees of freedom:

[0090] q = [q1, q2,..., q7] T = f kinematic (a x , a y , a z , ω x , ω y , ω z )

[0091] where a is the acceleration and ω is the angular velocity;

[0092] Calculate the target pose deviation through the inverse kinematics algorithm:

[0093] Δp = J(q)Δq

[0094] where J(q) is the Jacobian matrix, which is used to convert the joint space velocity to the operational space velocity. By calculating the target pose deviation, the action adjustments that the robot needs to make can be accurately determined to achieve the desired motion state of the user.

[0095] The method for the collaborative control module to execute the somatosensory-gesture-voice command fusion includes:

[0096] Hierarchical decision-making architecture:

[0097] First priority: Somatosensory pose command, controlling the macroscopic movement of the robot, such as moving forward and turning;

[0098] Second priority: Gesture command, controlling the fine adjustment of the camera, such as zooming and composition;

[0099] Third priority: Voice command, triggering mode switching, such as "start panoramic shooting";

[0100] where the execution priority is determined by the environmental risk level

[0101] Conflict arbitration

[0102] The environmental risk level is dynamically divided into four levels according to real-time perception data, as shown in the following table:

[0103]

[0104] Among them, the quantification method is as follows: Obstacle distance (d): Real-time detection by lidar / depth camera, with an accuracy of ±2 cm.

[0105] Slope (θ): Calculated by fusing IMU and terrain perception unit data, with an accuracy of ±0.5°.

[0106] Dynamic weight calculation (formula): In the formula, α = 0.6, β = 0.3, γ = 0.1 are weight coefficients, and v user is the user's movement speed.

[0107] When multi-modal instructions conflict, the conflict arbitration strategy is executed according to the following priority, as shown in the following table:

[0108]

[0109] Example scenario:

[0110] Conflicting instructions: Gesture "forward" vs voice "turn right"

[0111] L0: Execute the gesture "forward" hierarchically

[0112] L2: If there is an obstacle in the right-turn direction, ignore the voice instruction and execute the obstacle avoidance path

[0113] Real-time risk monitoring and update

[0114] Sensor fusion:

[0115] Lidar: Scans the environment with a radius of 5 m at 10 Hz to construct a 2D obstacle map.

[0116] Depth camera: Detects dynamic obstacles (such as pedestrians, moving objects) at 30 fps.

[0117] IMU: Updates the slope and robot attitude data at 100 Hz.

[0118] Update frequency: The risk level is recalculated every 100 ms, and the response delay < 50 ms.

[0119] Exception handling:

[0120] When sensor data is abnormal (such as lidar failure), switch to the standby mode (pure vision SLAM positioning). If the risk level calculation fails, it is defaulted to the L2 level and forced to decelerate to 0.3 m / s.

[0121] Verification and effect

[0122] Test data is as follows in the table:

[0123]

[0124]

[0125] Effect comparison:

[0126] Compared with the traditional fixed priority, the accident rate is reduced by 82%.

[0127] The success rate of instruction execution in complex terrain is increased from 75% to 96%.

[0128] Execution priority adjustment mechanism in rainy and foggy weather

[0129] In rainy and foggy weather (humidity ≥ 80%), the environmental perception ability is limited. The system adjusts the execution priority through multi-sensor fusion compensation and dynamic weight correction. The specific strategy is as follows:

[0130] Sensor mode switching and priority adjustment are as follows in the table

[0131]

[0132]

[0133] Conflict arbitration rules in rainy and foggy weather

[0134] Compulsory safety instruction first:

[0135] If the detected obstacle distance < 2m (normal weather is < 1m), directly trigger the L2 risk level and execute the obstacle avoidance path planning.

[0136] Example: When the gesture instruction "go forward" conflicts with the voice instruction "turn left", if the obstacle distance is 1.5m, the obstacle avoidance is given priority over the user instruction.

[0137] Dynamic response degradation:

[0138] The maximum moving speed of the robot is limited to 0.3m / s (0.5m / s in normal weather).

[0139] The path planning algorithm switches to the conservative mode (the safety distance increases from 0.5m to 1m).

[0140] Sensor redundancy strategy:

[0141] When the depth camera fails (obstructed by rain and fog), it completely relies on the data fusion positioning of lidar + IMU, and the weight formula is adjusted to: R fog = 0.7·R lidar + 0.3·R IMU

[0142] Technical verification data, as shown in the following table

[0143]

[0144]

[0145] Spatio-temporal constraint conditions: When it is detected that the user's body feeling is that the two arms are unfolded in a T shape and the voice command is "lock the target", the high-precision following mode is activated. At this time: The moving speed of the robot is limited to 0.5 m / s; The zoom ratio of the camera automatically matches the target distance. Among them, the proportional coefficient k = 0.2 m -1 。

[0146] The control logic of the high-precision following mode includes:

[0147] Real-time calculation of the relative azimuth angle between the user and the target object through body feeling data

[0148]

[0149] Control the robot to move in a circular motion with the target object as the center and a radius R = 2 m, and keep the user always in the tangential direction. Among them, R is dynamically adjusted according to the amplitude of the user's arm extension. Among them, for every 10 cm increase in the extension amplitude, R increases by 0.5 m.

[0150] The control method of the camera-assisted robot includes a body feeling control stage:

[0151] Body feeling data acquisition and preprocessing:

[0152] Obtain the original data of the acceleration and angular velocity of the user's upper limb joints through the IMU sensor group;

[0153] Generate three-dimensional point cloud data of the user's skeletal joint points through the depth camera array;

[0154] Motion intention recognition:

[0155] Input the joint point motion trajectories of 5 consecutive frames into the LSTM neural network to predict the user's motion direction within the next 0.5 seconds, including forward, backward, left, and right, and the action types, including walking, standing still, and turning;

[0156] Multi-modal instruction fusion:

[0157] When the body feeling predicted action is "turn" and the voice command contains a direction word (such as "shoot to the left"), generate an arc-shaped movement path for the robot, and the arc radius r satisfies:

[0158] r = v·t response + k·‖Δθ‖

[0159] Among them, v is the current speed, t_response = 0.3 s is the system response time, Δθ is the turning angle, and k = 0.1 m / deg is the calibration coefficient.

[0160] The LSTM neural network training method in the motion intention recognition of the above steps includes:

[0161] Construct a dataset containing 200 sets of human motion sequences, and each set of sequences contains:

[0162] Input data: Joint angles of 10 consecutive frames (7 joints × 10 frames = 70 dimensions)

[0163] Output label: Motion category of the next 5 frames (8 basic actions)

[0164] The network structure is a two-layer LSTM (128 units in the hidden layer), and the loss function adopts a combination of cross-entropy and motion smoothness penalty term:

[0165]

[0166] Among them, α = 0.7, and T = 5 is the prediction step.

[0167] The communication between the somatosensory capture unit and the robot adopts a hybrid transmission protocol:

[0168] For IMU sensor data: Use the BLE 5.0 protocol, the transmission interval is 10 ms, and the data packet contains a timestamp, a sensor ID, and 9-axis raw data;

[0169] For depth image data: Use Wi-Fi 6 multi-link transmission to synchronously send the left and right camera data through the 2.4 GHz and 5 GHz frequency bands respectively;

[0170] At the collaborative control module side, hardware-level time synchronization is adopted to ensure that the time deviation between the IMU and the depth data is less than 5 ms.

[0171] The somatosensory control stage also includes a feedback enhancement mechanism:

[0172] When the robot executes a somatosensory instruction, it feeds back the execution status to the user through a vibration motor:

[0173] Single short vibration (100 ms): The instruction has been received;

[0174] Double long vibration (300 ms × 2): The instruction has been completed;

[0175] Continuous vibration: The system detects an abnormal state and requires manual intervention.

[0176] And through visual feedback, by setting the LED light ring and setting the extension change, green, red, and blue are respectively used to correspond to single short vibration, double long vibration, and continuous vibration feedback.

[0177] The above are only the preferred embodiments of the present invention patent and are not intended to limit the present invention patent. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principle of the present invention patent shall be included within the protection scope of the present invention patent.

Claims

1. A shooting assistance robot, characterized in that, It includes a mobile chassis module (1), a lifting adjustment module, an angle adjustment module, a camera module, a storage module (8), a multi-modal interaction module, and a cooperative control module; The mobile chassis module (1) is provided with a driving motor and driving wheels, and the mobile chassis module (1) is provided with a terrain perception unit and a path planning unit; The lifting adjustment module is arranged on the mobile chassis module (1). The lifting adjustment module includes a plurality of folding frames hinged to each other. The folding frame brackets are controlled to fold and unfold by electric push rods, thereby controlling the lifting height. A height feedback sensor is arranged on the lifting adjustment mechanism; The angle adjustment module is arranged on the lifting adjustment module. The angle adjustment module is a three-axis gimbal. An attitude sensor is arranged on the angle adjustment module; The camera module is arranged on the angle adjustment module and is provided with an optical zoom component and an image processing unit; The multi-modal interaction module includes: A voice recognition unit for receiving and parsing user voice commands; A gesture capture unit for recognizing user gesture actions; A somatosensory capture unit. The somatosensory capture unit includes a wearable IMU sensor group and a depth camera array for capturing the motion posture data of the user's arms and torso; The cooperative control module is configured to: perform spatio-temporal alignment and semantic fusion on the voice commands, gesture actions, and somatosensory posture data to generate a composite control command; Synchronously adjust the robot's motion trajectory and camera shooting parameters according to the composite control command.

2. The shooting assistance robot according to claim 1, wherein The storage module (8) includes a storage box (9), a storage control mechanism (10), a door opening and closing mechanism (11), and a placement box (12). The lifting adjustment module is arranged in the placement box (12). The storage box (9) is arranged on the mobile chassis module (1). An opening (13) is provided on the storage box (9). The storage control mechanism (10) is used to control the placement box (12) to be stored in the storage box (9) or extend out of the opening (13); The door opening and closing mechanism (11) is used to seal the opening (13) after the placement box (12) is stored in the storage box (9); The storage control mechanism (10) includes a storage track (14), a sliding strip (15), a support rod (16), and a pushing cylinder (17). The storage track (14) is arranged on the inner walls on both sides of the storage box (9). The storage track (14) is a linear track. The sliding strip (15) is slidably arranged in the storage track (14). The support rod (16) is hinged on both sides of the sliding strip (15). The bottom of the placement box (12) is hinged to the support rod (16). The pushing cylinder (17) is arranged in the storage box (9). The pushing cylinder (17) is fixedly connected to the sliding strip (15). A stop block (18) is arranged in the storage box (9) and below the opening (13); The door opening and closing mechanism (11) includes a sealing door (19), a sealing track (20), a slider (21), a rotating gear (22), and a pushing rack (23). The sealing track (20) is arranged on the inner walls of both sides inside the storage box (9). The sealing track (20) is a bent track, including two mutually perpendicular and connected horizontal tracks (24) and vertical tracks (25). The slider (21) is slidably arranged in the sealing track (20). A connecting arm one (26) and a connecting arm two (27) are arranged at the bottom of the sealing door (19). A rotating gear (22) is arranged at the bottom of the connecting arm one (26). The rotating gear (22) is rotatably arranged on the slider (21). A pulley (28) is arranged at the bottom of the connecting arm two (27). The pulley (28) is slidably arranged in the sealing track (20). The pushing rack (23) meshes with the rotating gear (22). When the pushing rack (23) is pushed, the rotating gear (22) rotates, and by also pushing the slider (21) to slide, the sealing door (19) is reversed and moves out from the opening (13). When the pulley (28) moves from the horizontal track (24) to the vertical track (25), the sealing door (19) is also reversed, thereby realizing the storage of the sealing door (19) at the side wall inside the storage box (9). The pushing rack (23) is connected to a pushing cylinder through a connecting rod. When the pushing cylinder extends, the sealing door (19) is moved out from the opening (13) and vertically stored on the inner wall of the storage box (9). At the same time, the placement box (12) is pushed to the opening (13). When the pushing cylinder contracts, the return box is stored in the storage box (9), and at the same time, the sealing door (19) is flipped and closed to the opening (13) to seal the opening (13).

3. The shooting assistance robot according to claim 1, wherein The wearable IMU sensor group includes three nine-axis sensors respectively fixed on the user's wrist, elbow, and shoulder, and is used to collect limb joint angular velocity and acceleration data; The depth camera array includes two groups of infrared TOF cameras, and is used to construct three-dimensional coordinates of the user's skeletal joint points; The somatosensory capture unit includes a data fusion processor, and the data fusion processor executes: Receiving IMU raw data and depth image data; Fusing IMU and visual data through an extended Kalman filter and outputting user body pose data.

4. The shooting assistance robot according to claim 1, wherein, The data fusion method of the somatosensory capture unit includes: Establishing an upper limb kinematic model, which maps IMU data to joint angles of multiple degrees of freedom: Calculating the target pose deviation through an inverse kinematics algorithm.

5. The shooting assistance robot according to claim 1, characterized in that, The method for the collaborative control module to execute somatosensory-gesture-voice command fusion includes: Hierarchical decision-making architecture: First priority: Somatosensory pose command, controlling the macroscopic movement of the robot; Second priority: Gesture command, controlling the fine adjustment of the camera; Third priority: Voice command, triggering mode switching; Spatial and temporal constraint conditions: When detecting the user's somatosensation and voice command, activate the high-precision following mode. At this time: The moving speed of the robot is limited; The camera zoom ratio automatically matches the target distance.

6. The shooting assistance robot according to claim 5, wherein The control logic of the high-precision following mode includes: Real-time calculation of the relative azimuth angle between the user and the target object based on somatosensory data 7. The control method of the shooting assistance robot according to any one of claims 1-6, characterized in that, The control method of the camera-assisted robot includes a somatosensory control stage: Somatosensory data acquisition and preprocessing: Obtain the original acceleration and angular velocity data of the user's upper limb joints through the IMU sensor group; Generate three-dimensional point cloud data of the user's skeletal joint points through the depth camera array; Motion intention recognition: Input the joint point motion trajectories of consecutive frames into the LSTM neural network to predict the future user motion direction and action type; Multimodal instruction fusion.

Images