Mechanical arm shooting control system, control method and medium

Abstract

The invention relates to the technical field of shooting, in particular to a mechanical arm shooting control system and method and a medium. The invention discloses a mechanical arm shooting control system, a control module and a mechanical arm. A corresponding control instruction is generated by presetting a motion track of the mechanical arm and presetting focusing and zooming parameters of the shooting device at a plurality of motion nodes. And the control module receives the instruction, controls the mechanical arm to move according to the track, and dynamically adjusts focusing and zooming parameters based on the current position. The system supports linear and nonlinear algorithms, ensures smooth transition of parameters, and enhances the continuity of shot pictures.

Description

Technical Field

[0001] The present invention relates to the field of shooting technology, and in particular to a robotic arm shooting control system, a control method and a medium. Background Art

[0002] With the growing demand for film and television production and industrial shooting, robotic arms are widely used in shooting scenes to achieve complex camera motion trajectories. However, traditional robotic arm shooting systems have certain limitations in synchronously controlling the movement of the robotic arm and the focus and zoom of the shooting device. Usually, the robotic arm moves according to a preset trajectory, but the focus and zoom parameters of the shooting device need to be adjusted manually, or fixed parameters can only be set before the movement starts. This method cannot meet the needs of real-time adjustment of focus and zoom during shooting, resulting in blurred images, out of focus or poor composition, affecting the shooting effect. Summary of the invention

[0003] The purpose of the present invention is to provide a robotic arm shooting control system, control method and medium to address the deficiencies in the prior art, and to provide a robotic arm shooting control system that can dynamically adjust the focus and zoom parameters of a shooting device in real time during the movement of the robotic arm, so as to ensure the clarity and picture quality of image acquisition and realize the synchronous control of the robotic arm movement and the adjustment of the shooting device parameters.

[0004] The present invention achieves the above-mentioned object through the following technical solutions: A robotic arm shooting control system, comprising a control module and a robotic arm, wherein:

[0005] The control module is used to communicate with the host computer and control the operation of the robotic arm and the shooting device respectively according to the received robotic arm control instructions and the shooting device control instructions; the control module includes a control board circuit, which includes a central processing unit, a communication module, a Bluetooth module, an Ethernet module, a WIFI module, a memory, a power module, etc. An executable program is installed on the control module for receiving host computer instructions and issuing control instructions to the robotic arm and the shooting module.

[0006] The robotic arm is connected to the control module for mounting the camera and driving the camera to move along a preset trajectory according to the robotic arm control command. The robotic arm also has its own controller and control system. The robotic arm control box is connected to external devices through IO, and is connected to the control module through a network cable or WIFI to receive control commands from the control module.

[0007] A robotic arm shooting control system includes a host computer, a control module, a robotic arm and a shooting device, wherein:

[0008] The host computer is used to preset the motion trajectory of the robot arm and generate a control instruction for the robot arm; based on the motion trajectory of the robot arm, preset the focus and zoom parameters of the shooting device at at least two motion nodes of the robot arm; based on the preset focus and zoom parameters, generate a control instruction for the shooting device;

[0009] A control module, which is in communication with the host computer and is used to receive the control instructions of the mechanical arm and the camera device, and control the operation of the mechanical arm and the camera device respectively;

[0010] A mechanical arm, which is in communication with the control module and is used to drive the photographing device to move along a preset motion trajectory according to the control instruction of the mechanical arm;

[0011] The shooting device is installed on the mechanical arm and is in communication connection with the control module. It is used to dynamically adjust the focus and zoom during the movement of the mechanical arm according to the control instructions of the shooting device to collect images.

[0012] As a further solution of the present invention: the control module is also used to receive the position information of the robotic arm in real time, and perform the following operations based on the current position of the robotic arm: when the robotic arm is located on a motion node, the preset focus and zoom parameter values ​​corresponding to the motion node are selected; when the robotic arm is located between adjacent motion nodes, the focus and zoom parameter values ​​corresponding to the current position are calculated between the preset focus and zoom parameter values ​​of the adjacent nodes through an interpolation algorithm, so as to dynamically adjust the focus and zoom of the shooting device during the movement of the robotic arm to ensure the continuity and smoothness of the captured image.

[0013] As a further solution of the present invention: the robotic arm shooting control system further comprises a server in communication connection with the host computer, for storing the motion trajectory of the robotic arm and the focus and zoom parameters set by the host computer;

[0014] As a further solution of the present invention: the host computer of the robotic arm shooting control system can be a smart mobile device or a desktop device. The host computer can be installed with an executable computer program for sending control instructions and data to the robotic arm and the shooting module. When the shooting is completed, the host computer also downloads the shot video content and performs video processing. It can be processed into different video modes according to presets, such as high frame rate slow motion, video fast forward, video reverse playback and other processing schemes.

[0015] As a further solution of the present invention: the shooting device of the robotic arm shooting control system can be a camera of an intelligent mobile device, so the shooting device itself can be used as a host computer device.

[0016] The robotic arm may also be provided with a position feedback device for feeding back current position information to the control module, and a distance detection device for detecting the actual distance between the photographing device and the photographing target; the distance between the photographing device and the photographing target can be measured in a variety of ways. The first is that the position of the photographing target has been determined before shooting, and in a static state, the photographing target distance can be obtained by adding the static distance to the dynamic distance of the photographing device on the robotic arm. In this case, the distance can be measured without a feedback device. The second way is to measure through a distance feedback device, and obtain the dynamic distance of the photographing target through a distance measuring sensor installed on the photographing device. The third way is to obtain the dynamic distance of the photographing target through a visual algorithm solution.

[0017] The host computer includes a user interface, which allows the user to preset the motion trajectory of the robot arm through the user interface, and preset the focus and zoom parameters of the shooting device at at least two motion nodes of the robot arm. The user interface may also include shooting control buttons, a robot arm trajectory selection interface, voice prompts, video processing, work sharing and other interfaces.

[0018] As a further solution of the present invention: the robot arm control instruction is used to control at least one of the start, stop, trajectory planning, selection of preset trajectory and speed adjustment of the robot arm;

[0019] The shooting device control instructions are used for connecting, detecting, starting, stopping, shooting and recording the shooting device. During the movement of the robotic arm, the focus and zoom of the shooting device are dynamically adjusted according to the preset focus and zoom parameter values ​​and / or the focus and zoom parameter values ​​corresponding to the current position of the robotic arm. All other operations of the camera can be controlled. After shooting is completed, the shot works are downloaded;

[0020] When the robot moves linearly between two adjacent motion nodes, the interpolation algorithm adopts a linear interpolation method, by calculating the difference between the preset focus and zoom parameters of the adjacent nodes, the focus and zoom parameters of the shooting device are gradually adjusted at time intervals; when the robot moves nonlinearly between two adjacent motion nodes, the interpolation algorithm adopts a ranging interpolation method, by calculating the distance between the current position of the robot and the shooting target, dividing the interpolation by the distance difference, and dynamically adjusting the focus and zoom parameters of the shooting device. Specifically, the communication method between the control module and the robot arm and the shooting device adopts wireless communication and / or wired communication.

[0021] 1. Position feedback

[0022] Function: Position feedback is used to monitor the position and posture of the robot arm in real time so that the control module can know the current position node of the robot arm or whether it is between two adjacent nodes.

[0023] Implementation method: The trajectory of the robotic arm is pre-planned, including the motion trajectory and speed. Based on the trajectory and speed of the robotic arm, the position of the camera installed at the end of the robotic arm at a certain moment can be calculated.

[0024] Information content: The position feedback device provides the internal position data of the robot arm, such as joint angles, spatial coordinates, etc., which are used to accurately calculate the current position of the end of the robot arm (camera).

[0025] Application scenarios: Position feedback is mainly used to determine the position of the robot arm to achieve preset trajectory and position control. For example, it helps the control module determine whether the robot arm has reached a certain motion node or is between two adjacent nodes, thereby deciding whether to apply preset focus and zoom parameters or interpolation algorithms.

[0026] 2. Distance detection device, using distance sensor

[0027] Function: The distance sensor is used to detect the actual distance between the end of the robotic arm (i.e., the camera) and the target, helping to set or verify the focus and zoom parameters to ensure a clear image of the target.

[0028] Implementation method: The distance measuring sensor can be a laser rangefinder, an ultrasonic sensor, etc., which is installed at the camera or the end of the robotic arm to measure the distance from the end of the robotic arm to the external shooting target.

[0029] Information content: The distance sensor provides the distance information from the camera to the shooting target, which is independent of the internal position of the robot arm. This distance information is used to determine the best focus and zoom parameters of the shooting device.

[0030] Application scenario: The distance sensor is mainly used to set or verify the focus and zoom parameters. It helps the control module to set appropriate focus and zoom parameters according to different distances in the preset stage to ensure that the shooting target is always in a clear focus range.

[0031] As a further solution of the present invention: the system further includes an AI automatic focus and zoom module based on the AI ​​ranging module, which is used to calculate the distance to the shooting target based on the image collected by the shooting device before the robotic arm starts shooting, so as to assist in adjusting the focus and zoom parameters of the shooting device; the AI ​​automatic focus and zoom module includes:

[0032] An image acquisition unit, used to acquire an image of a shooting target before the robotic arm starts shooting;

[0033] An image preprocessing unit, used for denoising and enhancing the collected images;

[0034] A feature extraction unit, used to extract at least one of edge, shape and color features of the photographed object from the preprocessed image;

[0035] A distance calculation unit, configured to calculate the distance between the object and the camera based on the feature, the size of the object, the position information in the image, and the focal length and sensor size of the camera;

[0036] The parameter adjustment unit is used to preset the focus and zoom parameters for the shooting device at each motion node of the robotic arm according to the distance calculated by the distance calculation unit in the preset stage, so as to ensure the best imaging effect at each motion node during the shooting process. Example: In a robotic arm trajectory, we require the shooting target to be 3 meters away from the starting position of the robotic arm. Then the distance of the motion node between the starting position and the end position of the robotic arm is 3 meters minus the straight-line distance traveled by the camera after the movement.

[0037] As a further solution of the present invention: it also includes a QR code, which includes the remote control server address of the robotic arm. When the user uses a smart terminal device to scan the QR code, it automatically jumps to the remote control user interface. Through the remote control user page, the motion trajectory of the robotic arm and / or the shooting mode of the shooting device can be selected, and control instructions are sent to the robotic arm and the shooting device. The robotic arm and the shooting device execute the corresponding instructions and operate and shoot images according to the specified motion trajectory, thereby realizing the ability to remotely control the robotic arm to shoot; the QR code can also be used as a sharing QR code. The user enters an email address or mobile phone number in the remote control user interface. After the robotic arm completes shooting, the work will be automatically sent to the email address or mobile phone number specified by the user, thereby realizing the ability to automatically share.

[0038] As a further solution of the present invention: according to the motion trajectory of the robotic arm, different video processing modes and video playback speeds are preset at different motion nodes or time nodes. For example, fast playback is performed at the first motion node, slow playback is performed at the second motion node, and playback is performed at the third motion node, etc., so that the video shot by the shooting device on the same motion trajectory of the robotic arm can have a variety of display styles, enriching the interest of the work.

[0039] The present invention also provides another technical solution: a robotic arm shooting control method, comprising the following steps:

[0040] Preset step: preset the motion trajectory of the robot arm and the focus and zoom parameters of at least two motion nodes, and store the parameters in the server for the control module to call;

[0041] The first control step: the host computer sends a control instruction generated based on the preset motion trajectory of the robot arm to the control module, so that the control module controls the robot arm to move according to the preset trajectory;

[0042] Dynamic pre-adjustment step: The control module receives the position information of the robot arm in real time, and performs the following operations based on the current position of the robot arm: when the robot arm is located on a motion node, selects the preset focus and zoom parameter values ​​corresponding to the motion node;

[0043] Second control step: the control module generates focus and zoom instructions based on the preset focus and zoom parameter values ​​obtained in the dynamic pre-adjustment step, and sends the instructions to the shooting device;

[0044] Image acquisition steps: The shooting device dynamically adjusts the focus position and zoom ratio of the lens according to the received focus and zoom commands and the parameter values ​​to acquire images, ensuring clear images, smooth adjustments and smooth transitions, and avoiding image incoherence problems caused by sudden changes in parameters.

[0045] As a further solution of the present invention: the dynamic pre-adjustment step is: the control module receives the position information of the robot arm in real time, and performs the following operations based on the current position of the robot arm: when the current position of the robot arm is located on a motion node, the preset focus and zoom parameter values ​​corresponding to the motion node are selected; when the current position of the robot arm is located between adjacent motion nodes, the focus and zoom parameter values ​​corresponding to the current position are calculated between the preset focus and zoom parameter values ​​of the adjacent nodes through an interpolation algorithm;

[0046] The second control step is: the control module generates focus and zoom instructions based on the preset focus and zoom parameter values ​​obtained in the dynamic pre-adjustment step or the focus and zoom parameter values ​​corresponding to the current position, and sends them to the shooting device.

[0047] As a further solution of the present invention: in the dynamic pre-adjustment step, if the robot arm performs linear motion between adjacent motion nodes, the interpolation algorithm adopts a linear interpolation method, which includes using formula 1 to calculate a focus value corresponding to the current position, and using formula 1 to calculate a zoom value corresponding to the current position, and dynamically adjusting the focus and zoom parameter values ​​corresponding to the current position of the camera device;

[0048] Formula 1:

[0049] In formula 1, v represents the focus value at the current moment; a represents the preset focus value of the starting motion node A of the adjacent motion node; b represents the preset focus value of the ending motion node B; t represents the total time required for the robot arm to move from node A to node B; t current Indicates the time elapsed after the robot arm departs from node A, that is, the time difference between the current time and the starting time;

[0050] If the robot arm performs nonlinear motion between two adjacent motion nodes, the interpolation algorithm adopts the ranging interpolation method, including the following steps:

[0051] According to the motion trajectory distance of the robot arm, the path between two adjacent motion nodes is divided into n points, and n is appropriately set according to the actual moving distance between adjacent motion nodes;

[0052] For each segmentation point, calculate the distance x between the current position of the robot arm and the shooting target;

[0053] Based on the distance x and a preset distance-focus relationship function f(x), a required focus parameter v=f(x) is calculated;

[0054] Similarly, based on the distance x and the preset distance-zoom relationship function g(x), the required zoom parameters are calculated;

[0055] Dynamically adjust the focus and zoom parameter values ​​v and g(x) of the camera. Specifically, the control module communicates with the mechanical arm and the camera in a wireless or wired manner, using a standard communication protocol to achieve reliable data transmission.

[0056] As a further solution of the present invention: the control module is provided with a user interface for the user to input preset focus and zoom parameter values, preview the shooting picture in real time, and adjust the parameters during the shooting process;

[0057] The image acquisition step also includes a first real-time feedback fine-tuning step or a second real-time feedback fine-tuning step, wherein:

[0058] First real-time feedback fine-tuning step: the control module receives in real time the distance information between the current position of the robotic arm and the photographed target fed back by the distance detection device installed on the robotic arm; when it is found that there is a difference between the real-time distance data and the preset distance corresponding to the focus and zoom parameters obtained in the dynamic pre-adjustment step, the control module fine-tunes the focus and zoom parameters of the photographing device according to the real-time distance data to ensure that the photographed target is always clearly imaged during the continuous movement of the robotic arm;

[0059] Second real-time feedback fine-tuning step: During the continuous movement of the robotic arm, the AI ​​ranging module regularly collects images from the shooting device, and pre-processes and extracts features from the images; based on the image features and the camera parameters of the shooting device, the real-time distance between the shooting target and the shooting device is calculated; the control module compares the real-time distance with the preset distance corresponding to the focus and zoom parameters obtained in the dynamic pre-adjustment step, and if there is a difference, the focus and zoom parameters of the shooting device are fine-tuned to ensure that the shooting target is always in the best imaging state;

[0060] AI ranging module includes:

[0061] An image acquisition unit, used for periodically acquiring images of a target during the movement of the robotic arm;

[0062] An image preprocessing unit, used for denoising and enhancing the collected images;

[0063] A feature extraction unit, used to extract at least one of edge, shape and color features of the photographed object from the preprocessed image;

[0064] The distance calculation unit is used to calculate the real-time distance between the shooting target and the shooting device based on the characteristics, the size of the shooting target, the position information in the image, and the focal length and sensor size of the camera. Specifically, the control module uses a preset distance and focus and zoom parameter relationship function to fine-tune the focus and zoom parameters of the shooting device according to the difference between the real-time calculated distance and the preset distance.

[0065] The present invention also provides another technical solution: a computer-readable storage medium storing a computer program, wherein when the computer program is executed, any of the aforementioned robotic arm shooting control methods is executed.

[0066] Beneficial effects of the present invention:

[0067] This solution generates corresponding control instructions by presetting the motion trajectory of the robot arm in the host computer and presetting the focus and zoom parameters of the camera device at at least two motion nodes. After receiving these instructions, the control module controls the operation of the robot arm and the camera device respectively. The robot arm moves according to the preset trajectory, and the camera device dynamically adjusts the focus and zoom parameters during the movement according to the camera control instructions.

[0068] Specifically, the beneficial effects of this program are reflected in the following aspects:

[0069] 1. Realize synchronous control and improve shooting quality: By presetting the motion trajectory of the robotic arm and the corresponding focus and zoom parameters in the host computer, the system can dynamically adjust the focus and zoom of the shooting device while the robotic arm moves, realizing synchronous control of the robotic arm movement and the adjustment of the shooting device parameters, ensuring the clarity and stability of the captured image.

[0070] 2. Improve work efficiency and simplify operation process: The use of preset parameters and automatic control reduces the need for manual intervention, avoids the tedious steps of manually adjusting the parameters of the shooting device during the shooting process, and improves the overall work efficiency.

[0071] 3. Enhance the flexibility and adaptability of the system: By presetting the focus and zoom parameters at multiple motion nodes and dynamically adjusting them during motion, the system can adapt to complex shooting scenes and changing shooting needs and meet different shooting effect requirements.

[0072] 4. Avoid image quality degradation and ensure continuity: Dynamically adjust focus and zoom parameters to avoid blurry or discontinuous images caused by inaccurate focus or sudden zoom changes, thereby improving the overall quality of the film. BRIEF DESCRIPTION OF THE DRAWINGS

[0073] Figure 1 It is a schematic diagram of the structural flow of the present invention.

[0074] Figure 2 A schematic diagram of an embodiment of the present invention.

[0075] Figure 3 The figure is a schematic structural diagram of an embodiment of the present invention.

[0076] Figure 4 The figure is a schematic diagram of the structure of a robotic arm in one embodiment.

[0077] Figure 5 It is a structural schematic diagram of another implementation method of the robotic arm.

[0078] Figure 6 This is a schematic diagram of the bottom support plate of the robotic arm.

[0079] Figure 7 Schematic diagram of the retraction of the bottom support plate of the robotic arm.

[0080] Figure 8 Another structural schematic diagram of the bottom of the robotic arm.

[0081] Fig. 9 The figure is a schematic structural diagram of another embodiment of the robotic arm. DETAILED DESCRIPTION

[0082] like Figure 1-Figure 3 As shown, in an embodiment of the present invention, a robotic arm shooting control system is provided, including a control module and a robotic arm, or including a server, a host computer, a control module, a robotic arm and a shooting device.

[0083] in:

[0084] 1. Host computer: The host computer is used to preset the motion trajectory of the robot arm and generate control instructions for the robot arm. Based on the motion trajectory of the robot arm, the focus and zoom parameters of the shooting device are preset at at least two motion nodes of the robot arm. The host computer also generates control instructions for the shooting device. These parameters and instructions can be set through the user interface and stored in the server for the control module to call. In detail, based on the motion trajectory of the robot arm, the focus (Focus) and zoom (Zoom) parameters of the shooting device are preset at at least two motion nodes of the robot arm. These preset parameters and instructions are stored in the server for the control module to call. The host computer includes a user interface, through which the user can preset the motion trajectory of the robot arm and preset the focus and zoom parameters of the shooting device at each motion node. The user interface also allows the user to preview the camera screen in real time, fine-tune the preset parameters, and ensure the accuracy of the parameter settings.

[0085] 2. Control module 3: Control module 3 is connected to the host computer for communication, receives the control instructions of the robot arm and the control instructions of the shooting device, and controls the operation of the robot arm and the shooting device respectively. Control module 3 also receives the position information of the robot arm in real time, and selects appropriate focus and zoom parameter values ​​according to the current position. When the robot arm is located on a motion node, the preset parameters are directly selected; when the robot arm is located between adjacent motion nodes, the parameter value corresponding to the current position is calculated by the interpolation algorithm. In detail, the control module includes a wireless receiving module, a Bluetooth communication module, a wireless WIFI module and an RJ45 interface, and supports Bluetooth communication, WIFI communication, Ethernet communication and serial communication, etc. The control module also includes a voice module, which can perform voice recognition, facilitate the reception of voice commands, and realize voice control operation. The control module comes with a central processing unit and is a programmable system.

[0086] 3. Robotic arm 1: Robotic arm 1 is connected to the control module in communication, and drives the shooting device to move along a preset motion trajectory according to the control instructions of the robotic arm. The robotic arm includes a driving module of the robotic arm. The driving module is connected to the control module via a network cable or WIFI to receive instructions from the control module. A position feedback device is installed on the robotic arm to monitor the position and posture of the robotic arm in real time, and to feed back the position information to the control module. The robotic arm can also be installed with a distance detection device to detect the actual distance between the shooting device and the shooting target. A distance measuring sensor, such as a laser rangefinder, an ultrasonic sensor, etc., can be used, which is installed at the end of the robotic arm on the same plane as the camera.

[0087] 4. Shooting device 2: The shooting device 2 is installed on the robot arm and is connected to the control module. According to the control instructions of the shooting device, the focus and zoom are dynamically adjusted during the movement of the robot arm to collect images. The shooting device is generally an independent camera, a still camera or a mobile phone camera, which has the function of supporting the setting of focus and zoom through control instructions. The shooting device has built-in Bluetooth and WIFI modules for receiving control commands from the host computer. When the shooting device is an independent camera, it receives control instructions from the host computer; when the shooting device is a mobile phone camera, the mobile phone itself can directly control the camera to shoot.

[0088] 5. Server: The server is connected to the host computer and is used to store the motion trajectory and focus and zoom parameters of the robotic arm set by the host computer for the control module to call.

[0089] Furthermore, the system also includes a QR code, which includes the remote control server address of the robotic arm. When the user uses a smart terminal device to scan the QR code, it will automatically jump to the remote control user interface. Through the remote control user page, the motion trajectory of the robotic arm and / or the shooting mode of the shooting device can be selected, and control instructions can be sent to the robotic arm and the shooting device. The robotic arm and the shooting device execute the corresponding instructions and operate and shoot images according to the specified motion trajectory, thereby realizing the ability to remotely control the robotic arm to shoot; the QR code can also be used as a sharing QR code. The user enters an email address or mobile phone number in the remote control user interface. After the robotic arm completes shooting, the work will be automatically sent to the email address or mobile phone number specified by the user, thereby realizing the ability to automatically share.

[0090] Furthermore, according to the motion trajectory of the robotic arm, different video processing modes and video playback speeds are preset at different motion nodes or time nodes. For example, fast playback is performed at the first motion node, slow playback is performed at the second motion node, and playback is performed at the third motion node, etc., so that the video shot by the shooting device on the same motion trajectory of the robotic arm can have a variety of display styles, enriching the interest of the work.

[0091] System functions:

[0092] 1. Robotic arm control

[0093] The control module is electrically or communicatively connected to the robot arm, and sends control instructions to the robot arm to control the robot arm to move along the specified trajectory. The control module is connected to the robot arm via Ethernet or WIFI, and uses the TCP / HTTP protocol to send and receive data. The robot arm control instructions include start, stop, trajectory planning, speed adjustment, etc., as well as a variety of motion control commands, such as joint motion, circular motion, linear motion, spline curve planning, etc.

[0094] 2. Camera control

[0095] The shooting device includes a wireless receiving module, which receives control instructions from the host computer and controls the camera of the shooting device to shoot synchronously when the mechanical arm moves. The camera includes a Bluetooth communication module and a WIFI communication module. The host computer connects to the camera via Bluetooth or WIFI and sends camera control instructions, including shooting device control instructions.

[0096] Camera control commands include setting auto focus or manual focus, zoom in or out, and setting specific zoom and focus values. Camera control commands also include adjusting parameters such as metering, ISO, exposure, aperture, shutter, white balance, clarity, contrast, saturation, brightness, etc., as well as obtaining camera information, shutting down or restarting the system, switching working modes, managing memory cards, and managing files.

[0097] 3. Dynamic adjustment of focus

[0098] According to the motion trajectory and motion time of the robot arm, the distance between the shooting device and the shooting target at multiple motion nodes of the robot arm is obtained, and multiple precise focus parameters are preset at multiple motion nodes of the robot arm. When the robot arm is shooting in motion, the focus of the camera is dynamically adjusted according to the preset focus data to ensure that the shooting picture always remains clear and avoid the accuracy error and delay problems caused by auto focus.

[0099] For example, the robot arm trajectory passes through the starting node A, the intermediate nodes B and C, and the ending node D. At each node, a focus value is preset according to the distance between the node and the shooting target. The camera can capture a clear picture at each node. When the robot arm moves from node A to node B, the camera dynamically adjusts the focus parameter so that the focus parameter smoothly transitions from the preset value of node A to the preset value of node B, and so on, ensuring that the focus of the entire shooting process is always kept in the clearest state.

[0100] 4. Smooth focus and zoom algorithm

[0101] In order to achieve a smooth transition of focus and zoom parameters, interpolation algorithms are used, including linear interpolation and ranging interpolation. Linear interpolation: Applicable to the situation where the robot arm performs linear motion between two adjacent nodes. The calculation method is as follows:

[0102] S1. Determine the preset focus values ​​a and b of the starting node A and the ending node B.

[0103] S2. Calculate the total time t total and the current time t current .

[0104] S3. Calculate the current focus value v using formula 1;

[0105] Formula 1:

[0106] In formula 1, v represents the focus value at the current moment; a represents the preset focus value of the starting motion node A of the adjacent motion node; b represents the preset focus value of the ending motion node B; t represents the total time required for the robot arm to move from node A to node B; t current Indicates the time elapsed after the robot arm departs from node A, that is, the time difference between the current time and the starting time. Example assumptions: Starting focus value a = 10. Ending focus value b = 20. Total time t = 5 seconds. Calculation steps: Calculate the total number of focus adjustment steps c: c = ba = 20-10 = 10. Calculate the time interval y: At time t current = 2 seconds, calculate n and v: Calculate the number of instructions sent: Calculate the current focus value: v = a + n = 10 + 4 = 14. At time t current = 5 seconds (when the robot reaches the end point): calculate the number of instructions sent: Calculate the current focus value: v = 10 + 10 = 20. The focus value has been smoothly adjusted from a = 10 to b = 20.

[0107] S4. The calculation method of zoom parameter is the same as that of focus value.

[0108] Distance measurement interpolation method: Applicable to the case where the robot arm performs nonlinear motion between two adjacent nodes. The calculation method is as follows:

[0109] S1. Split the path between two adjacent nodes into n points.

[0110] S2. For each segmentation point, calculate the distance x between the current position of the robot arm and the shooting target.

[0111] S3. Calculate the focus parameter v=f(x) based on the distance x and a preset distance-focus relationship function f(x).

[0112] S4. The calculation method of zoom parameter is the same as that of focus value.

[0113] 5. Intelligent distance measurement

[0114] A distance detection device, such as an ultrasonic ranging sensor, a laser ranging sensor, an infrared ranging sensor or a millimeter wave radar sensor, is added to the end of the robotic arm and installed on the same plane as the camera to measure the distance between the camera and the target.

[0115] According to the measured distance, the required focus value is calculated by determining the relationship between the focal length and the focus distance. Then, a focus command is sent through the control interface of the camera to keep the shooting picture clear.

[0116] 6. AI distance measurement

[0117] Before the robot arm starts shooting, it captures a frame of the picture in advance, and uses computer vision technology and deep learning algorithms to calculate the distance between the camera and the target. The specific steps are as follows:

[0118] S1. Image acquisition: acquire an image.

[0119] S2. Image preprocessing: denoising, enhancing, etc. the image.

[0120] S3. Feature extraction: Extract key features such as the edge, shape, and color of the target from the image.

[0121] S4. Distance calculation: Calculate the distance between the object and the camera based on the size of the object and the position information in the image, combined with the camera parameters (focal length, sensor size).

[0122] 7. Dynamic zoom

[0123] According to the movement trajectory and time of the robot arm, the zoom parameters are preset at multiple movement nodes. When the robot arm moves and shoots, the camera adjusts the zoom parameters according to the preset zoom data to achieve the zooming in or out of the picture.

[0124] Time node control: Users can set zoom nodes on the timeline of the robot operation, start zooming at the specified time point, and set the corresponding zoom value.

[0125] Smooth zoom: The camera's zoom parameters transition smoothly between two adjacent zoom parameter preset values ​​to ensure smooth image zooming.

[0126] 8. User Interface

[0127] The user interface allows users to control the robot arm to pause at any time, preset the focus and zoom parameters at the current time, and preview the camera image in real time. Users can fine-tune the focus parameters to ensure the clarity of the captured image.

[0128] 9. Other control methods

[0129] Bluetooth and WIFI communication: Both the control module and the camera device support Bluetooth and WIFI communication, and the host computer can send control instructions through these methods.

[0130] Scan code control: Users can establish communication with the robotic arm by scanning the QR code, remotely control the robotic arm and camera to ensure the safety of the operator.

[0131] Voice control system: The control module includes a sound receiving module, which allows the user to control the operation of the robotic arm and the shooting device through voice commands.

[0132] Working principle:

[0133] 1. Preset stage: motion trajectory and parameter preset: The user presets the motion trajectory of the robot arm through the user interface of the host computer, and presets the focus and zoom parameters of the camera at at least two motion nodes of the robot arm. These parameters are stored in the server for the control module to call.

[0134] AI distance measurement assistance (optional): Before the robot arm starts shooting, the AI ​​autofocus and zoom module calculates the distance to the shooting target based on the image captured by the shooting device, and assists in adjusting the focus and zoom parameters of the shooting device. This module includes an image acquisition unit, an image preprocessing unit, a feature extraction unit, a distance calculation unit, and a parameter adjustment unit.

[0135] 2. Shooting stage

[0136] The first control step: the host computer sends the robot arm control instruction generated based on the preset robot arm motion trajectory to the control module, and the control module controls the robot arm to move according to the preset trajectory.

[0137] Dynamic pre-adjustment steps: The control module receives the position information of the robot arm in real time and performs the following operations based on the current position of the robot arm:

[0138] When the current position of the robot arm is located on a motion node, the preset focus and zoom parameter values ​​corresponding to the motion node are selected;

[0139] When the current position of the robot arm is between adjacent motion nodes, the parameter value corresponding to the current position is calculated between the preset focus and zoom parameter values ​​of the adjacent nodes through an interpolation algorithm; the interpolation algorithm can use linear interpolation and / or ranging interpolation.

[0140] Linear interpolation method: When the robot moves linearly between adjacent motion nodes, the linear interpolation method is used to calculate the focus and zoom parameter values ​​at the current position using Formula 1;

[0141] Formula 1:

[0142] In formula 1, v represents the focus value at the current moment; a represents the preset focus value of the starting motion node A of the adjacent motion node; b represents the preset focus value of the ending motion node B; t represents the total time required for the robot arm to move from node A to node B; t current It indicates the time elapsed after the robot arm started from node A, that is, the time difference between the current time and the starting time.

[0143] Distance measurement interpolation method: When the robot arm moves nonlinearly between adjacent motion nodes, the distance measurement interpolation method is used. Based on the distance between the current position of the robot arm and the shooting target, the focus and zoom parameter values ​​of the current position are calculated using the preset distance and focus and zoom relationship functions f(x) and g(x); calculation steps:

[0144] 1. Split the path between two adjacent motion nodes into n points.

[0145] 2. For each segmentation point, calculate the distance x between the current position of the robot arm and the shooting target.

[0146] 3. Use the relationship function v=f(x), z=g(x) to calculate the focus and zoom parameter values.

[0147] Second control step: the control module generates a camera control instruction using the calculated focus and zoom parameter values, and sends the instruction to the camera.

[0148] Image acquisition steps: The shooting device dynamically adjusts the focus position and zoom ratio of the lens according to the received control instructions to capture images to ensure clear pictures and smooth transitions.

[0149] Real-time feedback and fine-tuning (optional)

[0150] Based on the distance detection device: The control module receives the distance information fed back by the distance detection device installed on the robot arm in real time. When it is found that there is a difference between the real-time distance data and the distance corresponding to the preset parameters, the control module fine-tunes the focus and zoom parameters of the shooting device according to the real-time distance to ensure that the shooting target is always clear.

[0151] Based on AI distance measurement: During the movement of the robotic arm, the AI ​​distance measurement module regularly collects images of the shooting target, calculates the real-time distance after image preprocessing and feature extraction. The control module fine-tunes the focus and zoom parameters of the shooting device according to the difference between the real-time distance and the preset parameters.

[0152] Example 1: Linear motion scene

[0153] In an advertisement shooting, a robot arm is required to carry a camera and move in a straight line from position A to position B to shoot a close-up of a product. The user presets the linear motion trajectory of the robot arm on the host computer, and presets the corresponding focus and zoom parameters at positions A and B.

[0154] Dynamic pre-adjustment: The control module uses linear interpolation to calculate the focus and zoom parameters that should be adjusted at the current position according to the formula, ensuring that the parameters of the shooting device transition smoothly and the picture is coherent during the movement of the robotic arm.

[0155] Example 2: Non-linear motion scene

[0156] In a movie shoot, the robot arm needs to move around the actor in an arc to switch between close-up and long-distance shots. Since the motion trajectory is nonlinear, the user presets the focus and zoom parameters of multiple motion nodes on the host computer.

[0157] Dynamic pre-adjustment: The control module adopts the distance measurement interpolation method and uses the distance detection device or AI distance measurement module to calculate the distance between the current position of the robot arm and the actor in real time, and calculates the currently required focus and zoom parameters based on the distance function to ensure that the actor is always clear in the picture and the composition is reasonable.

[0158] Example 3: Real-time feedback and fine-tuning

[0159] In a complex industrial inspection scenario, the position of the target may change slightly.

[0160] Real-time feedback: The control module obtains the distance between the shooting device and the target in real time through the distance detection device. Once a deviation is found, the focus and zoom parameters are immediately fine-tuned to ensure a clear image.

[0161] The robotic arm shooting control system and method of the present invention realizes synchronous control of the robotic arm and the shooting device by presetting the movement trajectory of the robotic arm and the focus and zoom parameters of the shooting device in the upper computer, thereby solving the problem in the prior art that the shooting device parameters cannot be adjusted in real time during the movement of the robotic arm.

[0162] 1. Synchronous control to improve shooting quality

[0163] By coordinating the control of the robot arm and the camera device through the control module, the focus and zoom parameters of the camera device can be adjusted in real time with the movement of the robot arm, ensuring that the shooting target is always in clear focus and appropriate composition, thereby improving image quality.

[0164] 2. Dynamic adjustment, smooth and coherent picture

[0165] An interpolation algorithm is used to smoothly adjust the parameters of the shooting device during the movement of the robotic arm, avoiding the problem of picture incoherence caused by sudden changes in parameters and ensuring the continuity and stability of the image.

[0166] 3.Flexible adaptation, multiple exercise methods

[0167] The system supports linear interpolation and ranging interpolation, can adapt to the linear and nonlinear movement of the robot arm, and is suitable for various shooting scenes such as advertising, movies, industrial inspection, etc., which enhances the applicability of the system.

[0168] 4. Real-time feedback to ensure imaging quality

[0169] Through the position feedback device and the distance detection device, the control module can obtain the position relationship between the robotic arm and the shooting target in real time, and adjust the parameters of the shooting device in time to avoid the degradation of imaging quality due to position deviation.

[0170] 5. Improve work efficiency and simplify operations

[0171] The robot arm trajectory and shooting parameters are set in the preset stage, which reduces the workload of manual adjustment during the shooting process and improves work efficiency. At the same time, the user-friendly interface makes it easy for users to set and adjust parameters.

[0172] 6. AI technology assists, precise focus

[0173] The introduction of the AI ​​ranging module uses image processing and deep learning technology to accurately calculate the distance to the shooting target, assist in adjusting the focus and zoom parameters, and further improve the shooting accuracy and picture quality.

[0174] Specifically, when the robot is located between adjacent motion nodes, the focus and zoom parameter values ​​of the current position are dynamically calculated and generated in real time between the preset parameter values ​​of the adjacent nodes through the interpolation algorithm, rather than directly using the preset parameter values. Preset parameters on motion nodes: In the preset stage, the system will set the corresponding focus and zoom parameters on each motion node, which are static preset values. Smooth transition between nodes: When the robot is between two adjacent motion nodes, the system cannot directly apply the preset parameter value of a certain node, but requires a transition value. Therefore, the interpolation algorithm will dynamically calculate the focus and zoom parameters of the current position based on the preset parameter values ​​of the adjacent nodes, so that the parameter values ​​transition smoothly between adjacent nodes. Real-time interpolation calculation: The interpolation algorithm does not reset or obtain new focus and zoom parameters, but generates the parameters of the current position in real time based on the adjacent preset values. This calculation is completed in real time during the shooting process to adapt to the current position changes of the robot arm and achieve smooth adjustment of the picture. Therefore, the focus and zoom parameter values ​​at the current position are dynamically calculated and generated through an interpolation algorithm, but these dynamically generated values ​​rely on the preset parameters of adjacent nodes rather than re-measuring or re-acquiring new focus and zoom data.

[0175] In detail, when the robotic arm is located between adjacent motion nodes, the focus and zoom parameter values ​​of the current position are generated by real-time dynamic calculation. This process is calculated between the preset focus and zoom parameter values ​​of adjacent nodes according to the interpolation algorithm to obtain the appropriate parameter value for the current position, thereby ensuring a smooth transition of focus and zoom during shooting. Preset stage: Before shooting, the system presets the focus and zoom parameters on each motion node through AI ranging or other methods. These parameters are fixed and stored in the system for direct use during shooting. During shooting: When the robotic arm is located on a preset motion node, the system directly calls the preset focus and zoom parameters of the node. When the robotic arm is between two adjacent motion nodes, the system uses an interpolation algorithm to calculate the focus and zoom parameters corresponding to the current position in real time. This calculation is not preset, but is dynamically generated based on the preset values ​​of adjacent nodes.

[0176] Example: Assume that the focus parameters between moving nodes A and B are a and b respectively. If the robot moves between A and B, and the current position is 50% of the distance from A to B, the interpolation algorithm can calculate the focus parameter at this time as (a+b) / 2. This is a real-time interpolation calculation process to ensure that the parameter change from a to b is smooth. In this way, the focus and zoom parameters calculated in real time by the interpolation algorithm can ensure a smooth transition during the shooting process, avoiding the problem of incoherent images caused by sudden changes in parameters.

[0177] Scan code control: Users can establish communication with the robotic arm by scanning the QR code. The specific plan is that when scanning the QR code, the user initiates a connection request to the server, requesting to connect to the specified robotic arm. The host computer of the robotic arm is connected to the server through TCP, and listens for the scan code request from the user terminal. After listening to the scan code request from the user terminal, the host computer sends an instruction to notify the control board, and the user scans the code to request to operate the robotic arm, thereby establishing a connection between the user end and the control module. The user can initiate a shooting command to the control board to control the robotic arm and the shooting module to shoot; the scan code control method can keep the operator away from the robotic arm to ensure the safety of the operator.

[0178] Voice control system: The control module may include a sound receiving module, which allows the user to control the execution through voice control. When the user sends a voice command to the voice control module through a microphone or other means, the voice control module receives the voice and performs voice recognition. The final content of the voice recognition is judged. If it is a normal control instruction, it is sent to the control module for command processing and execution of the corresponding action; for example, controlling the robotic arm to shoot through sound.

[0179] Robotic arm control: The control module is electrically / communicatively connected to the robotic arm, and the control module sends control instructions to the robotic arm to control the robotic arm to move along a specified trajectory. The control module is connected to the robotic arm via Ethernet or WIFI, and uses the TCP / HTTP protocol to send and receive data. The robotic arm control instructions include: sending the program name (this instruction is sent through port 8082); sending the program content (this instruction is sent through port 8082); joint movement, circular motion, linear motion, single-axis jog; data for executing joint space planning when the servo is in place; data for executing Cartesian space planning when the servo is in place; full circle movement, spline curve planning; spline curve start, spline curve end; start executing the program, stop executing the program, pause executing the program, and continue executing the program; control box digital output, analog output, analog input; terminal digital output, digital input, analog output, analog input; robot mode switching, setting the robot movement speed, setting the robot installation position; setting the collision level, setting the robot positive limit angle, setting the robot negative limit angle; setting the tool calibration point, calculating the tool coordinate system, and setting the tool center point TCP Coordinates; set workpiece calibration points, calculate workpiece coordinate system, set workpiece coordinate system; set tool list, set workpiece list, set end load weight; set end load center of mass coordinates, obtain current joint configuration of robot, enable / disable robot; start arc, end arc, turn on laser, turn off laser, start tracking, stop tracking; start positioning, end positioning, set swing parameters, start swinging, stop swinging; end sensor configuration, end sensor activation, end sensor register writing; force sensor configuration, force sensor activation, force sensor collision protection; set force sensor reference coordinate system, force sensor zeroing, constant force control; return the current actual angle of the joint, return the current actual arc of the joint; return the current actual speed of the joint, return the current actual arc speed of the joint; return the current joint torque, return the weight of the current load, return the center of mass of the current load; return the current tool target posture, return the joint soft limit angle; obtain sensor data in the reference coordinate system, obtain the original data in the sensor coordinate system.

[0180] Camera control instructions: Camera control instructions include shooting device control instructions, which can set auto focus or manual focus; zoom in or zoom out can be set. When setting Zoom in, the picture is pulled in, and when setting Zoom out, the picture is pulled out. The specific Zoom value can be set, and the range is specified according to the camera, for example, the Zoom range value of (0~31). The camera control instructions can set the focus value. When set to manual focus, the camera receives the Focus instruction from the host computer and sets the specified Focus value. The camera control instructions also include the following function settings: metering, iso, exposure, aperture, shutter, angular velocity, white balance, clarity, contrast, saturation, brightness, etc. The camera control instructions also include: obtaining camera information, synchronizing camera time, shutting down or restarting the camera system, setting the camera working mode, switching working modes, such as switching to video recording mode, video playback mode, standby mode, etc. The camera control instructions also include: changing the stream source, built-in stream service, bit rate, stream segmentation, stream frame rate, encoder, bit width, recommended stream, streaming, etc. The camera control instructions also include: memory card management, file management, file download, thumbnail, etc.

[0181] Dynamic adjustment of focus: According to the movement trajectory and movement time of the robot arm, the distance between the camera and the shooting target at multiple movement nodes of the robot arm is obtained, and multiple precise focuses are pre-set for the camera at multiple movement nodes of the robot arm. When the robot arm drives the camera to perform motion shooting, the camera's focus is dynamically adjusted according to the pre-set focus data for image acquisition, avoiding accuracy errors and focus delays caused by auto focus.

[0182] For example, see Figure 2 The robot arm trajectory passes through nodes A, B, C, and D, starting node A, intermediate node B, intermediate node C, and ending node D. At the position of each node, the distance between the node and the shooting target is followed, and a focus value is preset so that the camera can capture a clear picture at this node. This focus value can be continuously fine-tuned on the user interface until the clearest picture appears. Similarly, different focus values ​​are set at points B, C, and D, respectively, so that the camera's picture clarity at points B, C, and D is maximized. The preset focus value (Focus) is stored in a readable storage medium.

[0183] Focusing during motion: preset a precise focus parameter at the starting position of the robot's motion, preset a precise focus parameter at the end position of the robot's motion, and preset multiple precise focus parameters between the initial and end positions of the robot according to the motion changes of the robot's trajectory. The specific number of preset nodes is adjusted according to the motion changes of the robot's trajectory and the distance changes of the shooting target. The final preset parameters enable the robot to dynamically adjust according to the preset values ​​during the motion process, so that the shooting picture always remains clear, and the picture will not be blurred as the distance between the camera and the shooting target changes.

[0184] After the robot trajectory and focus preset value are saved and confirmed, when the user sends a shooting command and uses the trajectory to shoot, the host computer obtains the preset value of each node (such as ABCD). When the machine is initialized, the focus value of node A is read and sent to the camera for use. When the robot moves from node A to node B, the camera dynamically adjusts the focus parameter so that the focus parameter smoothly transitions from the preset value of node A to the preset value of node B. Similarly, when the robot moves from node B to node and then to the final node D, the camera focus value is dynamically adjusted according to the preset value in turn, so that the focus of the entire shooting process is always kept in the clearest state.

[0185] Smooth focus algorithm: Dynamic focus adjustment refers to dynamically adjusting the focal plane to a specified value during camera movement. For two adjacent preset values, the following algorithm can be used to achieve smooth focus settings.

[0186] (1) Linear interpolation method: When the robot moves between two adjacent nodes, for example, from point A to point B, the difference between the preset value of point A and the preset value of point B is first calculated as c, and then the time t required for the robot to move from point A to point B is calculated. The time interval y for sending commands is calculated as follows: y = t / c. Assuming that the preset focus value of point A is a and the preset focus value of point B is b, the focus value sent is: v = a + n (n is the number of times the command is sent after n time intervals); in some embodiments, when the robot moves linearly between two adjacent points, it is more appropriate to use the linear interpolation method for smooth zooming. In detail, a is the preset focus value of the starting point A. b is the preset focus value of the ending point B. c is the total change in focus value, calculated as: c = ba, which represents the total focus value that needs to be adjusted from point A to point B. t is the total time required for the robot to move from point A to point B. y is the time interval for sending focus adjustment commands, calculated as: n is the number of focus adjustment instructions sent (or the number of time intervals that have passed). v is the focus value at the current moment. (2) Distance measurement interpolation method: When the robot moves nonlinearly between two adjacent points, it is more appropriate to use the distance measurement interpolation method. For example, when moving from point A to point B, the distance between point A and point B is divided into n points according to the distance of the robot's movement trajectory (n is appropriately set according to the actual movement distance between two adjacent points). According to the robot's movement trajectory, the distance between the current point and the shooting target is calculated, and the precise focus is set according to the distance: v = f(x), x is the distance between the node and the shooting target, and f(x) is the relationship function between distance and focus (which may be different for each camera), which needs to be obtained through experimental methods.

[0187] Smart distance measurement: If there is no smart distance measurement, the target needs to be fixed at the specified position of the robot arm, and each motion node of the robot arm is calculated based on the initial distance. A smart distance measurement module, such as a distance measurement sensor, can be added to the end of the robot arm to detect the distance between the camera and the target. After obtaining the distance between the camera and the target, the focus parameters can be adjusted in real time according to the dynamic distance, so that the shooting picture is always clear.

[0188] Ultrasonic ranging sensor: At the end of the robotic arm, an ultrasonic ranging sensor is installed on the same plane as the camera lens. The ultrasonic ranging sensor is used to measure the distance by emitting ultrasonic waves and receiving the waves reflected back.

[0189] Laser ranging sensor: A laser ranging sensor is installed at the end of the robotic arm to measure the distance using the transmission time of the laser pulse, that is, to measure the time required for the laser to be emitted, reflected by the target and returned to the receiver.

[0190] Infrared ranging sensor: An infrared ranging sensor is installed at the end of the robotic arm, which uses the different intensities of the infrared signal reflected when it encounters an obstacle to detect the distance of the obstacle.

[0191] Millimeter-wave radar sensor: A millimeter-wave radar ranging sensor is installed at the end of the robotic arm, which uses the reflection of millimeter waves (an electromagnetic wave with a wavelength between centimeters and millimeters) to measure the distance.

[0192] The distance measuring sensor is installed on the same plane as the camera to measure the distance between the camera and the target. The sensor sampling frequency is between tens of milliseconds and one hundred milliseconds.

[0193] Set focus based on distance: For cameras that support focus by sending a specified command (that is, assigning a specified value to the focal plane), then set the focus based on the distance between the camera and the subject, using the following steps:

[0194] 1. Determine the relationship between focal length and focusing distance

[0195] First, based on the relationship between the focal length and focus distance of the camera. The focal length determines the viewing angle and depth of field of the lens, while the focus distance refers to the distance from the lens to the clear imaging plane of the shooting target. Lenses with different focal lengths will have different ranges and characteristics of focus distances. Design a relationship function based on the characteristics.

[0196] 2. Measure distance: Use the distance sensor to dynamically measure the actual distance between the camera and the target. Calculate the focus value based on the relationship function in Article 1. Use the formula to calculate the required focus value based on the focal range and depth of field characteristics of the camera. Refer to the camera's user manual for specific information.

[0197] 3. Send focus command: According to the calculated focal length value, send focus command through the camera control interface to keep the shooting picture clear.

[0198] AI distance measurement: Before the robot arm starts shooting, it captures a frame of the picture in advance, and calculates the distance between the camera and the shooting target through computer vision technology and deep learning algorithms. Adjust the focus parameters in real time according to the dynamic distance. 1. Collect an image and perform denoising and enhancement on the collected image. 2. Extract key features from the preprocessed image: edges, shapes, colors. 3. According to the size of the object and the position information in the image, combined with the camera parameters (focal length, sensor size), calculate the distance between the object and the camera.

[0199] User interface: It includes a user interface that can control the robot arm to pause at any time point, allowing the user to preset the focus parameters at the current time, and preview the clarity of the camera image to ensure the accuracy of parameter settings. In the user interface, the user can fine-tune the focus parameters (Focus) and manually adjust the preset parameters at a specific node position until the shooting image is clearly displayed.

[0200] Dynamic zoom: According to the movement trajectory and time of the robot arm, multiple zoom data (Zoom) are preset for the camera at multiple movement nodes of the robot arm. When the robot arm drives the camera to shoot, the camera adjusts the zoom parameters according to the preset zoom data (Zoom). The filmed works are enlarged or reduced according to the user's preset zoom data, enriching the shooting content.

[0201] During the movement of the robotic arm, the zoom parameters can be preset according to the spatial nodes or time nodes. The spatial node refers to the preset Zoom parameter value at a certain position after the robotic arm moves to that position, and the time node refers to the preset Zoom parameter value at that time point after the robotic arm runs for a specified period of time. When shooting the robotic arm movement, when the robotic arm moves to the specified spatial node or time node, the Zoom command is sent to zoom the camera, so that the photographed work can be enlarged or reduced at the specified position.

[0202] Time node control: The trajectory of the robot is known, so the time required for the robot to run from the starting point to the end point is also known. For example, if the time required from the starting point to the end point is n seconds, the user can set the zoom node within the period of 0 to n seconds. The zoom is specified by the user and starts at a certain time. The zoom value is set by the user. A user interface is needed here. There is a time drawer on the user interface with a time range from 0 to n seconds. The user selects a time point a on the timeline and sets the initial Zoom value za at this time point. Then, the second time point b is selected and the Zoom value zb is set at this time point. Then, when the robot is in motion, when it moves to time point a, it sends a Zoom command to set the Zoom to the value za, and when it reaches time point b, it sets the Zoom to the value zb. Then, in the process of video generation, you will see that the video has achieved the Zoom change from point a to point b.

[0203] Smooth zoom: During the movement of the robotic arm, between two adjacent zoom parameter preset values ​​(for example, two adjacent nodes from point A to point B), the host computer dynamically sends zoom parameter instructions based on the zoom parameter difference between points A and B, so that the zoom parameter of the camera continuously increases / decreases from the preset value of point A and smoothly transitions to the preset value of point B, so that the zoom parameter of the entire shooting process changes smoothly, allowing the shooting picture to be zoomed and displayed smoothly.

[0204] Storage medium: The storage medium is a computer-readable storage medium for storing a computer program, which executes the aforementioned method when it is run. The computer program is stored in a host device and a control module, and the host device is connected to the control module and the camera, and sends instructions via Bluetooth or WIFI to implement the corresponding functions.

[0205] Video processing: After the camera collects data, users can perform video processing according to the video processing methods provided by the system. These processing methods include: video slow motion, video speed change, video forward or reverse playback, and generating video according to a specified speed change mode.

[0206] Robotic arm and its control instructions

[0207] 1. Robotic arm controller I / O

[0208] The robot includes a control box, which contains control I / O. The I / 0 in the control box is used to control various devices, including pneumatic relays, PLCs and stop buttons of tight limit devices. The network interface group in the control box and the user network port can be used to communicate with external devices. The I / O and 485 communication interfaces of the end board can be used to control various devices, including pneumatic relays, PLCs and emergency stop buttons. The I / O connector model is an M12 connector 8-pin female. The robot is connected through the network, and the robot control terminal interface is opened on the PC side. The control interface includes a three-dimensional model of the robot. The I0 setting interface can be entered in the terminal interface. In this interface, the digital output, analog output (0-10v) and end tool digital output, analog output (0-10v) in the robot control box can be manually controlled: DO operation, select the port number, if the DO is low level, the right operation button will display ON, and click the button to set the DO to high level. AO operation, select the port number, enter the value (0-100) in the right input box, the value is a percentage, setting 100 means setting the AO port to 10v. The status display area on the right side of the 3D model will display the current I0 status. For digital input and digital output, if the port level is high, the point will be displayed in green, if it is low, it will be displayed in white; analog input and analog output display values ​​are 0-100, 100 means 10v.

[0209] 2. Robotic arm IO filtering:

[0210] On the control interface, click the "I0 filter" submenu to enter the I0 filter time setting interface. The filter time setting interface includes: control box DI filter time, end board DI filter time, control box AIO filter time, control box Al1 filter time, end board AI0 filter time. Users can set the corresponding parameters according to their needs by clicking the corresponding setting button.

[0211] 3. Robotic arm teaching pendant:

[0212] The IP address of the robot controller is 92.168.57.2. The button box interface defaults to the teach pendant control port, and the IP address is 192.168.58.2. Use a network cable to connect the button box interface to the computer. Set the computer IP address to 192.168.58.10 or the same network segment. Open the browser and enter 192.168.58.2 to access the teach pendant page. The control box accesses the teach pendant page through the network port connected to the button box. Click the "Teaching Simulation" button on the teach pendant, and click its submenu "Program Teaching" to enter the program teaching interface, which mainly realizes the writing and modification of the robot arm teaching program. After clicking the "New" icon button, the user names the file and selects a template as the content of the new file. Click New to successfully create and open the program file.

[0213] The robot user account is divided into three levels: operators can use a few functions, programmers have some function restrictions, and administrators have no function restrictions. Among them, the manual high-speed function is an important safety protection function, which can only be used by programmers and administrators. Operators do not have the manual high-speed function, that is, when the operator controls the robot movement in manual mode, the maximum speed that the robot can reach is 250mm / s, which plays a protective role.

[0214] 4. Coordinate system of the robotic arm:

[0215] In the control terminal interface, you can create various 3D virtual coordinate systems. Take the base coordinate system as an example. Figure 4 and Figure 5 , where the X axis is red, the Y axis is green, and the Z axis is blue. Base coordinate system: In the controller terminal interface, the display is turned on by default in the three-dimensional virtual area of ​​the robot arm, and the fixed mark is at the bottom center of the robot arm base. The three-dimensional virtual base coordinate system can be manually turned off.

[0216] Robot arm tool coordinate system: The tool coordinate system is enabled for display by default. After the control interface is started and the user logs in successfully, the name of the currently applied tool coordinate system and the corresponding parameter data are obtained to initialize the current tool coordinate system. When other tool coordinate systems are applied during use, after the tool coordinate system instruction is successfully applied, the existing tool coordinate system in the robot's 3D virtual area is cleared first, and then the newly applied tool coordinate system parameter data is passed to the 3D coordinate system generation API to generate the tool coordinate system. After generation, the corresponding display is completed in the robot's 3D virtual area. The 3D virtual tool coordinate system can be manually closed for display. Workpiece coordinate system: The workpiece coordinate system is closed by default and can be manually opened for display. The process is consistent with the tool coordinate system. External axis coordinate system: The external axis coordinate system is closed by default and can be manually opened for display. The process is consistent with the tool coordinate system.

[0217] 5. Robot arm soft limit:

[0218] Click "Soft Limit" on the control interface to enter the soft limit interface. There may be other devices within the range of the robot arm. The limit angle can soft limit the robot arm so that the robot arm movement does not exceed a certain coordinate value to prevent the robot arm from colliding. The soft limit triggering robot arm stop is automatically triggered by the robot arm, and there is no stop distance. The administrator can use the default value or enter the angle value. Enter the angle value to limit the positive and negative angles of the robot arm joint respectively. When the input value exceeds the specified soft limit angle value of the robot arm joint, the limit angle will be adjusted to the maximum value that can be set. When the robot arm reports that the command exceeds the limit, it is necessary to enter the drag mode and drag the robot arm joint to within the limit angle.

[0219] 6. Joint motion:

[0220] Under Joint operation, the 6 slider bars in the middle represent the angles of the corresponding axes. Joint movement is divided into single-axis jog and multi-axis linkage. Single-axis jog: Users can control the movement of the robot arm by operating the circular buttons on the left and right sides. In manual mode and joint coordinate system, a certain joint of the robot arm can be rotated. When the robot arm stops beyond the range of motion (soft limit), single-axis jog can be used for manual operation to move the robot arm out of the over-limit position. Single-axis jog is faster and more convenient than other operation modes for rough positioning and large-scale movement. Set the "Long Press Motion Threshold" (the maximum distance the robot arm runs when the button is long pressed, the input value range is 0 to 300) parameter, and long press the circular button to control the movement of the robot arm. If the button is released during the operation of the robot arm, the robot arm will stop moving immediately. If the button is pressed and not released, the robot arm will stop moving after running the value set by the long press motion threshold.

[0221] Multi-axis linkage:

[0222] The user can operate the six sliders in the middle to adjust the corresponding target position of the robot arm. The target position can be determined by observing the 3D virtual robot arm. If the adjusted position does not meet your expectations, click the "Restore" button to return the 3D virtual robot arm to its original position. When the user determines the target position, click the "Apply" button and the physical robot arm will move accordingly.

[0223] Base jog:

[0224] In the base coordinate system, you can operate the circular buttons on the left and right sides to control the robot arm to move linearly on the X, Y, and Z axes or rotate around RX, RY, and RZ. The 6 slider bars in the middle represent the position and motion range on the corresponding coordinate axes. For example, the function of Base jog is similar to the function of single-axis jog in Joint motion.

[0225] Tool Jog

[0226] Select the tool coordinate system, and you can operate the circular buttons on the left and right sides to control the robot arm to move linearly on the X, Y, and Z axes or rotate around RX, RY, and RZ. The 6 slider bars in the middle represent the position and motion range on the corresponding coordinate axes. The function of Tool jogging is similar to the function of single-axis jogging in Joint motion.

[0227] Wobj Jog

[0228] Select workpiece jog, you can operate the circular buttons on the left and right sides to control the robot arm. In the workpiece coordinate system, it moves linearly along the X, Y, and Z axes or rotates around RX, RY, and RZ. The 6 slider bars in the middle represent the position and motion range on the corresponding coordinate axes. The function of Wobj jog is similar to that of single-axis jog in Joint motion.

[0229] Move

[0230] Select Move, you can directly enter the Cartesian coordinate value, click "Calculate joint position", the joint position is displayed as the calculated result, confirm that there is no danger, you can click "Move to this point" to control the robot arm to move to the input Cartesian pose.

[0231] 7. Robot arm teaching pendant programming:

[0232] The operation steps of the teaching programming (TPD) function are as follows:

[0233] (1) Record the initial position: Enter the operation area on the left side of the 3D model and record the current position of the robot arm. Set the name of the point in the edit box and save it;

[0234] (2) Configure trajectory recording parameters: Enter the "TPD" function item to configure trajectory recording parameters, set the trajectory file name, posture type and sampling period, configure DI and DO. During the process of recording the TPD trajectory, you can trigger DI to record the corresponding DO that needs to be output;

[0235] (3) Check the robot mode: Check whether the robot mode is in manual mode. If not, switch to manual mode. In manual mode, there are two ways to switch to the drag-and-teach mode. One is to long press the end button, and the other is to drag the mode switch button on the interface. In TPD recording, it is recommended to switch the robot arm to the drag-and-teach mode from the interface.

[0236] (4) Start recording: Click the "Start recording" button to start trajectory recording, and drag the robot arm to teach the action. In addition, there is a "TPD recording start / stop" function configuration item in the terminal DI configuration. By configuring this function, the user can trigger the "start recording" trajectory function through an external signal. It should be noted that to start recording the trajectory through an external signal, the TPD trajectory information must first be configured on the page.

[0237] (5) Stop recording: After the action teaching is completed, click the "Stop Recording" button to stop the trajectory recording, and then use the drag teaching switch button to make the robot arm exit the drag teaching mode. The teach pendant receives the "Stop trajectory recording success" message, which means that the trajectory recording is successful. Same as step 4, after configuring the "TPD recording start / stop" function, the recording can be stopped by an external signal.

[0238] (6) Teaching Programming: Click New, select Blank Template, click to enter the PTP function programming item, select the initial position point just saved, click the "Add" button, and after the application is completed, a PTP instruction will be displayed in the program file; then click to enter the TPD function programming item, select the trajectory just recorded, set whether it is smooth and the speed scaling ratio, click the "Add" button, and after the application is completed, a MoveTPD instruction will be displayed in the program file; start recording the trajectory through an external signal, and first configure the TPD trajectory information on the page.

[0239] (7) Trajectory reproduction: After the teaching program is edited, switch to the automatic operation mode and click the "Start Running" icon at the top of the interface to start running the program. The robot arm will begin to reproduce the taught movements.

[0240] (8) Track Editing: The TPD track editing area can visualize and edit the track to achieve TPD track pre-analysis and simplification. Select the corresponding track acquisition point, and the track point recorded by the user will be displayed in the three-dimensional space of the robot arm. Then, you can drag the "Start" and "End" scroll bars to simulate and reproduce the start and end points of the track and edit them.

[0241] 8. Robotic arm program instructions

[0242] There are two main ways to add program commands to a file. One way is to open the relevant command and click the Apply button to add the command to the program. The other way is to click the "Add" button first. At this time, the command is not saved in the program file. You need to click "Apply" to save the command to the file. The second way often appears when multiple commands of the same type are issued. We have added the Add button and the function of displaying the content of the added commands for this type of command. Click the Add button to add a command, and the Added commands display all the added commands. Click "Apply" to save the added command to the opened file on the right.

[0243] (1) PTP command: Select the spatial position point to be reached. The smooth transition time setting can achieve continuous movement from this point to the next point. Whether to set the offset can be selected based on the base coordinate system offset or based on the tool coordinate system offset. The x, y, z, rx, ry, and rz offset settings will pop up. The specific PTP path is the optimal path automatically planned by the motion controller.

[0244] (2) Lin command: This command has a similar function to the "PTP" command, but the path to the point reached by this command is a straight line.

[0245] (3) Arc command: The "Arc" command is an arc motion, which includes two points. The first point is the middle transition point of the arc, and the second point is the end point. Both the transition point and the end point can be set to offset or not. You can choose to offset based on the base coordinate system or based on the tool coordinate system, and pop up the x, y, z, rx, ry, and rz offset settings. The end point can be set with a smooth transition radius to achieve a continuous motion effect.

[0246] (3) Circle command: The "Circle" command is a full circle movement, which includes two points. The first point is the transition point 1 in the middle of the full circle, and the second point is the transition point 2 in the middle of the full circle. Transition point 2 can be set to offset or not. The offset is effective for both transition point 1 and transition point 2.

[0247] (4) Spiral command: The "Spiral" command is a spiral motion, which includes three points. The three points form a circle. In the third point setting page, the parameters include the number of spiral turns, attitude correction angle, radius increment and axis direction increment. The number of spiral turns refers to the number of spiral turns. The attitude correction angle corrects the attitude at the end of the spiral and the attitude of the first point of the spiral. The radius increment refers to the increment of the radius per turn. The axis direction increment refers to the increment in the direction of the spiral axis. Set whether to offset. The offset is effective for the entire spiral trajectory.

[0248] (5) N-Spiral command: The "N-Spiral" command is an optimized version of spiral motion. This command only requires one point plus the configuration of various parameters to achieve spiral motion. The robot arm takes the current position as the starting point. The user sets the debugging speed, whether to offset, the number of spiral turns, the spiral inclination, the initial radius, the radius increment, the axis direction increment and the rotation direction. The number of spiral turns is the number of spiral turns, the spiral inclination is the angle between the tool Z axis and the horizontal direction, the posture correction angle corrects the posture at the end of the spiral and the posture of the first point of the spiral, the initial radius is the radius of the first circle, the radius increment is the increment of the radius per circle, the axis direction increment is the increment of the spiral axis direction, and the rotation direction is clockwise and counterclockwise.

[0249] (6) Spline command: This command is divided into three parts: spline group start, spline segment and spline group end. The spline group start is the starting mark of the spline motion. The spline segment includes SPL, SLIN and SCIRC segments.

[0250] (7) N-Spline command: This command is a Spline command algorithm optimization command. It will replace the existing Spline command in the future. This command is divided into three parts: multi-point trajectory start, multi-point trajectory segment and multi-point trajectory end. Multi-point trajectory start is the starting mark of multi-point trajectory movement. Multi-point trajectory segment is to set each trajectory point. Click the icon to enter the point adding interface. Multi-point trajectory end is the end mark of multi-point trajectory movement. Here you can set the control mode and debugging speed. The control mode is divided into given control point and given path point.

[0251] (8) IO command: Click the "IO" icon to enter the IO command editing interface. The "IO" command is divided into two parts: Set IO (SetDO / SPLCSetDO) and Get IO (GetDI / SPLCGetDI). The "SetDO / SPLCSetDo" command can set the specified output DO status, including 16-channel control box digital output and 2-channel tool digital output. The status option "False" is closed and "True" is open. Select "Block" for the blocking option to set the DO status after the movement stops. Select "Non-blocking" to set the DO status during the previous movement. Select "Break" for the smooth trajectory option to set the DO status after the smooth transition radius ends. Select "Serious" to set the DO status during the smooth transition radius movement. When the command is added in the auxiliary thread, select "Yes" to apply the thread. Select "No" to use the command elsewhere. Click "Add" and "Apply".

[0252] In the "GetDI / SPLCGetDI" instruction, select the value of the port number you want to obtain and whether to block. Selecting "blocking" means obtaining the DI status after the movement stops, and selecting the "non-blocking" option means obtaining the DI status during the previous movement. When the instruction is added in the auxiliary thread, select "yes" to apply the thread, and select "no" to use the instruction elsewhere. After selecting, click the "add" and "apply" buttons.

[0253] (9) AI command: This command is divided into two functions: setting analog output (SetAO / SPLCSetAO) and getting analog input (GetAI / SPLCGetAI). "SetAO / SPLCSetAO" selects the analog output to be set, and inputs the value to be set, ranging from 0 to 10. Selecting "Blocking" for the option "Blocking" means setting the AO state after the motion stops, and selecting "Non-blocking" means setting the AO state during the previous motion.

[0254] Select the analog input to be obtained in "GetAI / SPLCGetAl". Select "Blocking" for the blocking option, which means obtaining the AI ​​status after the movement stops. Select "Non-blocking" to obtain the AI ​​status during the previous movement. When the instruction is added in the auxiliary thread, select "Yes" for whether to apply the thread. Select "No" for other places to use the instruction. Click "Add" and "Apply".

[0255] (10) TPD command: In this command, the user first needs to have a recorded trajectory. About trajectory recording: Before preparing to record the trajectory, save the starting point of the trajectory first. When the robot arm is in drag mode, enter the file name, select the period (assuming the value is x, that is, record a point every x milliseconds, and it is recommended to record a point every 4 milliseconds), and click to start recording. The user can drag the robot arm to perform specified movements as needed. After the recording is completed, click Stop Recording to save the previous robot arm's movement trajectory. When a movement cannot be fully recorded, a prompt will be prompted that the number of recorded points exceeds the limit. The user needs to record the movement in several times. When programming the program, first use the PTP instruction to reach the starting point of the corresponding trajectory, then select the trajectory in the TPD trajectory reproduction instruction, select whether to smooth, set the debugging speed, and click "Add" and "Apply" in sequence to insert the program. The trajectory loading instruction is mainly used to pre-read the trajectory file and extract it into trajectory instructions for better application in conveyor belt tracking scenarios.

[0256] (11) Mode command: This command can switch the robot arm to manual mode. It is usually added at the end of a program so that the user can automatically switch the robot arm to manual mode and drag the robot arm after the program ends.

[0257] (12) Pause command: This command is a pause command. When this command is inserted into a program, the robot arm will be in a paused state when the program executes this command. If you want to continue running, click the "Pause / Resume" button in the control area.

[0258] (13) Gripper command: This command is divided into gripper motion control command and gripper activation / reset command. In the gripper control command, the number of the gripper that has been configured and activated is displayed. The user can edit it in the edit box or slide the slider to the required value to complete the setting of the gripper opening and closing, opening and closing speed, and opening and closing torque. The value is a percentage. There is a blocking function option. If blocking is selected, the gripper movement must wait until the previous motion instruction is executed before it can be executed. If non-blocking is selected, the gripper movement is parallel to the previous motion instruction.

[0259] (14) F / T commands: These commands include FT_Guard (collision detection), FT_Contro| (constant force control), FT_Spiral (spiral insertion), FT_Rot ​​(rotational insertion), FT_Lin (linear insertion), FT_FindSurface (surface positioning), and FT_CalCenter (center positioning).

[0260] (15) 3D command: This command is used to generate a 3D vision program instance. Users can refer to the generated program and communicate with other vision devices. It includes two programs: camera calibration and camera capture.

[0261] (16) Thread command: Thread command is an auxiliary thread function. The user can define an auxiliary thread to run simultaneously with the main thread. The auxiliary thread mainly exchanges data with external devices and supports socket communication, robot arm DI status acquisition, robot arm DO status setting, robot arm status information acquisition, and data interaction with the main thread. The data obtained by the auxiliary thread is used to control the judgment of the robot arm movement logic.

[0262] (17) ServoCart command: ServoCart servo control (Cartesian space motion) command. This command can control the motion of the robot arm through absolute posture control or based on the current posture offset. x, y, z, rx, ry, rz (Cartesian position) is the current position of the robot arm. In addition, the user can control the motion of the robot arm by reading trajectory data files, sending trajectory data through socket communication, etc.

[0263] (18) Function command: This command is used to call the function interface function, provide the robot interface function to the customer for selection, and prompt the customer for the parameters required by the function, making it easier for the customer to write script instructions. More functions are being added.

[0264] (19) Xmlrpc command: XML-RPC is a remote procedure call method that uses XML to transfer data between programs through sockets. In this way, the robot controller can call a function (with parameters) in a remote program / service and get the returned structured data. The robot controller is responsible for writing XML-RPC client messages and handling all the details of converting data types to and from XML.

[0265] (20)Acc command: The Acc command is used to realize the function of individually setting the acceleration of the robot arm. By adjusting the acceleration scaling factor of the motion command, the acceleration and deceleration time can be increased or decreased, thereby achieving adjustable rhythm time of the robot arm movement.

[0266] (21) Torque command: This command is a force compliance control command. Through this command, the robot arm can adjust its trajectory to reduce the force when the force is greater than a certain value during force control training; when the force is less than a certain value, the robot arm returns to the original trajectory and continues to move along the original trajectory.

[0267] (22) The Aux-IO command is a command function for the robot to communicate with the PLC to control the external expansion IO. It requires the robot to establish UDP communication with the PLC. On the basis of the original 16 input and output channels, 128 input and output channels can be expanded. The usage of this command is similar to the general IO usage mentioned above.

[0268] 9. Robotic arm calibration:

[0269] In the menu bar of "Robot Body" in "Auxiliary Application", click "Robot Calibration" to enter the robot calibration interface. This function is mainly used to calibrate the zero position of the robot. Click the "Disable" button, drag the axes of the robot, move the robot to the zero position on the robot, and click the "Zero Point Setting" button to set the zero point of the robot.

[0270] Zero point setting: When the robot arm leaves the factory, an initial posture is preset, in which the angles of each joint are 0. When the zero point is set, the robot arm posture corresponds to when each joint of the robot arm moves to a specific position. The zero point is the reference of the robot arm coordinate system. Without the zero point, the robot arm cannot determine its own position. Therefore, in order to obtain the highest possible absolute positioning accuracy, the robot arm needs to be zero-calibrated. Generally, the robot arm needs to be zero-calibrated in the following situations:

[0271] (1) After replacing the mechanical system parts of the robot arm;

[0272] (2) After a violent collision with the workpiece or the environment;

[0273] (3) When operations such as establishing a coordinate system differ greatly from the actual position;

[0274] (4) After the entire system is reinstalled;

[0275] (5) After the encoder battery is replaced;

[0276] (6) After long-distance transportation of the handling robot.

[0277] 10. Matrix movement

[0278] In the menu bar of "Robot Body" in "Auxiliary Application", click "Matrix Movement" to enter the matrix movement configuration function interface. This function controls the regular movement of the robot arm by setting the coordinates of three points and the values ​​of row, column, layer and layer height, which is suitable for common palletizing applications. The first step is to select the robot arm movement mode, "PTP" or "Line", the second step is to set the robot arm movement path, "head to tail walking method" or "bow-shaped walking method", and the third step is to set the stacking method, "stacking" or "unstacking". The fourth step is to teach three points according to the path. The first point is the starting point of the first row. The arm posture during the entire movement process is determined by this point. The second point is the end point of the first row, and the third point is the end point of the last row. The fifth step is to set the number of rows and columns of points, the sixth step is to set the number of layers and the height of each layer, and the last step is to name the matrix motion program file, and a matrix movement program is successfully generated.

[0279] 11. Laser sensor tracking function

[0280] Instruction description: Select the "Laserr" command in the program teaching command interface. It integrates laser related instructions. According to the specific program teaching requirements, add instructions in the corresponding places.

[0281] Laser sensor trajectory reproduction function

[0282] Instruction description: Select the "T-Rec" command in the program teaching command interface. This command is mainly used to obtain the starting point and end point of the laser recognition path and reproduce the trajectory. According to the specific program teaching requirements, add instructions in the corresponding place.

[0283] Attitude Adaptation Configuration Steps

[0284] Step 1: Select the "Track Posture Configuration" button in the user peripheral configuration interface to enter the posture adjustment configuration interface, select the plate type and the actual working direction of the robot arm, adjust the posture of the robot arm, and set the posture point A, posture point B and posture point C respectively. Usually A is the plane posture point, B is the rising edge posture point, and C is the falling edge posture point.

[0285] Step 2: Select the "Adjust" command in the program teaching command interface. Add instructions in the corresponding places according to the specific program teaching requirements.

[0286] 12. Retractable design of the robotic arm base

[0287] When the robot arm is working, it needs to swing in multiple axes, which will cause the base to shake and become unstable. Therefore, the bottom support plate 21 is unfolded (see Figure 6 ), which helps prevent the base from shaking when the robot arm is working. However, when the robot arm is not in use, the bottom support plate 21 can be retracted (see Figure 7 ), saving space and convenient for storage.

[0288] 13. The height of the wheels on the robot arm base is adjustable

[0289] The height of the wheels on the robot base can be adjusted up and down through the screw rod. Figure 8 and Fig. 9 When the device needs to be moved, the bottom wheels can be lowered so that the wheels 22 touch the ground and the bottom support plate 21 does not touch the ground, so that the device can be pushed by the wheels 22 to move, making it easier to move the device to other places. After the device is moved to a designated position, the wheels can be raised by the screw rod 23 so that the wheels do not touch the ground and the bottom support plate 21 touches the ground, and the robot arm can work safely.

Claims

1. A robotic arm shooting control system, characterized in that: It includes a control module and a robotic arm, wherein: The control module is used to communicate with the host computer and control the operation of the robotic arm and the shooting device according to the received robotic arm control instructions and shooting device control instructions respectively; The robotic arm is connected to the control module for installation of the photographing device and drives the photographing device to move along a preset motion trajectory according to the robotic arm control instruction.

2. The robotic arm shooting control system according to claim 1, characterized in that: Also includes: The host computer generates the robot arm control instructions according to the preset robot arm motion trajectory; Based on the motion trajectory of the robotic arm, focus and zoom parameters of the camera device are preset at at least two motion nodes of the robotic arm; Generate camera control instructions based on preset focus and zoom parameters; as well as The camera is installed on the robot arm and is in communication with the control module. According to the camera control command, the camera dynamically adjusts the focus and zoom during the movement of the robot arm to collect images. The control module is also used to obtain the motion trajectory information and motion time of the robot arm, and perform the following operations based on the motion trajectory information and motion time of the robot arm: when the robot arm is located at a specified motion node or a specified time node, select the preset focus and zoom parameter values ​​corresponding to the motion node; When the robot is located between adjacent motion nodes, the focus and zoom parameter values ​​corresponding to the current position are calculated between the preset focus and zoom parameter values ​​of the adjacent nodes through an algorithm.

3. The robotic arm shooting control system according to claim 1, characterized in that: The robotic arm shooting control system also includes a server connected to the host computer for storing the motion trajectory of the robotic arm and the focus and zoom parameters set by the host computer; The host computer includes a user interface, which allows the user to preset the motion trajectory of the robotic arm through the user interface, and preset the focus and zoom parameters of the shooting device at at least two motion nodes of the robotic arm.

4. The robotic arm shooting control system according to claim 2, characterized in that: The robot arm control instruction is used to control at least one of the start, stop, trajectory planning and speed adjustment of the robot arm; The camera control instruction is used to dynamically adjust the focus and zoom of the camera according to preset focus and zoom parameter values ​​and / or focus and zoom parameter values ​​corresponding to the current position of the robot arm during the movement of the robot arm; When the robot arm moves linearly between two adjacent motion nodes, the focus and zoom parameters of the camera device are gradually adjusted at time intervals by calculating the difference between the preset focus and zoom parameters of the adjacent nodes; When the robot arm moves nonlinearly between two adjacent motion nodes, the ranging method is used to calculate the distance between the current position of the robot arm and the shooting target, and the focus and zoom parameters of the shooting device are dynamically adjusted by dividing and interpolating the distance difference.

5. The robotic arm shooting control system according to claim 1, characterized in that: The system also includes an AI autofocus and zoom module based on the AI ​​ranging module, which is used to calculate the distance to the shooting target based on the image collected by the shooting device before the robotic arm starts shooting, so as to assist in adjusting the focus and zoom parameters of the shooting device; the AI ​​autofocus and zoom module includes: An image acquisition unit, used to acquire an image of a shooting target before the robotic arm starts shooting; An image preprocessing unit, used for denoising and enhancing the collected images; A feature extraction unit, used to extract at least one of edge, shape and color features of the photographed object from the preprocessed image; A distance calculation unit, configured to calculate the distance between the object and the camera based on the feature, the size of the object, the position information in the image, and the focal length and sensor size of the camera; The parameter adjustment unit is used to preset focus and zoom parameters for the shooting device at each motion node of the mechanical arm according to the distance calculated by the distance calculation unit in the preset stage.

6. The robotic arm shooting control system according to claim 1, characterized in that: It also includes a QR code, which contains the remote control server address of the robotic arm. When the user uses a smart terminal device to scan the QR code, it will automatically jump to the remote control user interface. Through the remote control user page, the motion trajectory of the robotic arm and / or the shooting mode of the shooting device can be selected, and control instructions can be sent to the robotic arm and the shooting device. The robotic arm and the shooting device execute the corresponding instructions and operate and shoot images according to the specified motion trajectory, thereby realizing the ability to remotely control the robotic arm to shoot; the QR code can also be used as a sharing QR code. The user enters an email address or mobile phone number in the remote control user interface. After the robotic arm completes shooting, the work will be automatically sent to the email address or mobile phone number specified by the user, thereby realizing the ability to automatically share.

7. The robotic arm shooting control system according to claim 1, characterized in that: According to the motion trajectory of the robotic arm, different video processing modes and video playback speeds are preset at different motion nodes or time nodes. For example, fast playback is performed at the first motion node, slow playback is performed at the second motion node, and playback is performed at the third motion node, etc., so that the video shot by the shooting device on the same motion trajectory of the robotic arm can have a variety of display styles, enriching the interest of the work.

8. A robotic arm shooting control method, characterized in that: The steps include: Preset step: preset the motion trajectory of the robot arm and the focus and zoom parameters of at least two motion nodes, and store the parameters in the server for the control module to call; The first control step: the host computer sends a control instruction generated based on the preset motion trajectory of the robot arm to the control module, so that the control module controls the robot arm to move according to the preset trajectory; Dynamic pre-adjustment step: The control module receives the position information of the robot arm in real time, and performs the following operations based on the current position of the robot arm: when the robot arm is located on a motion node, selects the preset focus and zoom parameter values ​​corresponding to the motion node; Second control step: the control module generates focus and zoom instructions based on the preset focus and zoom parameter values ​​obtained in the dynamic pre-adjustment step, and sends the instructions to the shooting device; Image acquisition step: The shooting device dynamically adjusts the focus position and zoom ratio of the lens according to the received focus and zoom instructions and the parameter values ​​to acquire images.

9. The robotic arm shooting control method according to claim 8, characterized in that: The dynamic pre-adjustment steps are as follows: the control module receives the position information of the robot arm in real time, and performs the following operations based on the current position of the robot arm: when the current position of the robot arm is located on a motion node, the preset focus and zoom parameter values ​​corresponding to the motion node are selected; when the current position of the robot arm is located between adjacent motion nodes, the focus and zoom parameter values ​​corresponding to the current position are calculated between the preset focus and zoom parameter values ​​of the adjacent nodes through an interpolation algorithm; The second control step is: the control module generates focus and zoom instructions based on the preset focus and zoom parameter values ​​obtained in the dynamic pre-adjustment step or the focus and zoom parameter values ​​corresponding to the current position, and sends them to the shooting device.

10. The robotic arm shooting control method according to claim 9, characterized in that: In the dynamic pre-adjustment step, if the robot arm performs linear motion between adjacent motion nodes, the interpolation algorithm uses linear interpolation to calculate the focus value corresponding to the current position and the zoom value corresponding to the current position, and dynamically adjusts the focus and zoom parameter values ​​corresponding to the current position of the camera device; If the robot arm performs nonlinear motion between two adjacent motion nodes, the interpolation algorithm adopts the ranging interpolation method, including the following steps: According to the motion trajectory distance of the robot arm, the path between two adjacent motion nodes is divided into n points; For each segmentation point, calculate the distance x between the current position of the robot arm and the shooting target; Based on the distance x and a preset distance-focus relationship function f(x), a required focus parameter v=f(x) is calculated; Similarly, based on the distance x and the preset distance-zoom relationship function g(x), the required zoom parameters are calculated; Dynamically adjust the focus and zoom parameter values ​​v and g(x) of the camera.

11. The robotic arm shooting control method according to claim 8, characterized in that: The control module is provided with a user interface for the user to input preset focus and zoom parameter values, preview the shooting picture in real time, and adjust the parameters during the shooting process; The image acquisition step also includes a first real-time feedback fine-tuning step or a second real-time feedback fine-tuning step, wherein: First real-time feedback fine-tuning step: the control module receives in real time the distance information between the current position of the robotic arm and the shooting target fed back by the distance detection device installed on the robotic arm; when it is found that there is a difference between the real-time distance data and the preset distance corresponding to the focus and zoom parameters obtained in the dynamic pre-adjustment step, the control module fine-tunes the focus and zoom parameters of the shooting device according to the real-time distance data; Second real-time feedback fine-tuning step: During the continuous movement of the robotic arm, the AI ​​ranging module regularly collects images of the shooting device, and pre-processes and extracts features of the images; based on the image features and the camera parameters of the shooting device, the real-time distance between the shooting target and the shooting device is calculated; the control module compares the real-time distance with the preset distance corresponding to the focus and zoom parameters obtained in the dynamic pre-adjustment step, and if there is a difference, the focus and zoom parameters of the shooting device are fine-tuned; AI ranging module includes: An image acquisition unit, used for periodically acquiring images of a target during the movement of the robotic arm; An image preprocessing unit, used for denoising and enhancing the collected images; A feature extraction unit, used to extract at least one of edge, shape and color features of the photographed object from the preprocessed image; The distance calculation unit is used to calculate the real-time distance between the shooting target and the shooting device based on the features, the size of the shooting target, the position information in the image, and the focal length and sensor size of the camera.

12. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed, the robotic arm shooting control method described in any one of claims 8 to 11 is performed.

Images