Semi-automatic control method and system for a photographic robot

Abstract

The present application relates to the technical field of camera operation, and particularly relates to a semi-automatic control method and system for a camera robot, which comprises the following steps: S101, acquiring a first input signal combination; S102, switching or keeping a remote control module to a corresponding function state according to the first input signal combination; S103, when the remote control module is in a first function state, acquiring a second input signal combination associated with a robot module; S104, converting the second input signal combination into a first motion instruction; S105, splitting the first motion instruction into a first instruction and a second instruction, and sending the first instruction and the second instruction to a robot unit and a camera unit in the robot module respectively. The semi-automatic control method provided by the present application provides multiple function states for flexible switching to users, so as to meet different camera operation requirements in an actual film set environment.

Description

[0001] Divisional application

[0002] This application is based on application number 202310964915.3, filed on July 31, 2023.

[0003] This is a divisional application of the Chinese invention patent application entitled "A semi-automatic control method and system for a photography robot". Technical Field

[0004] This invention relates to the field of camera movement technology, and specifically to a semi-automatic control method and system for a photography robot. Background Technology

[0005] With the development of film and television technology, the demand for highly flexible and stable camera movement equipment in the film and television industry has increased significantly. Currently, the commonly used camera movement method is to mount the camera on an industrial robot.

[0006] This is to achieve camera movement in photography. There are generally two types of methods for using industrial robots for camera movement:

[0007] 1) Use a virtual camera to mark points, and then use a camera robot to reproduce the marked point path;

[0008] For example, patent application CN2019106238656 discloses a method, device, and system for generating and controlling robot trajectories. This method uses the pose of a virtual device acquired through a motion capture system to solve for the robot's actual trajectory. Another example is patent application CN202010842227.6, which discloses an offline programming method, system, and electronic device for a camera robot. Similarly, this method discretizes the spatial trajectory of a virtual camera robot and allows real-time viewing of the virtual-real combination effect in a virtual shooting system to help users complete shooting tasks. Yet another example is patent application CN2019101367788, which discloses a method for planning the path of a camera robot and a computer storage medium. This method allows for quick and visual path curve editing of the motion path, enabling users to intuitively view the shooting effect of the virtual camera in real time. However, this path reproduction method based on virtual cameras often has the following problems in actual shooting: First, the data entry is prone to inconsistencies (due to hand tremors, significant operational errors may occur in the data entry during the operation of the virtual camera); second, the operation of this virtual camera is relatively complex, placing higher demands on the professional skills of the photographer (requiring the on-site photographer to ensure artistic quality while being proficient in controlling the various modules of the virtual camera, industrial robot, and real camera; it may even require the participation of multiple people, including the photographer and robot engineer). Therefore, this method is often more suitable for teaching camera shooting trajectories, but it will face the aforementioned problems or limitations in the actual shooting environment of a film set.

[0009] 2) Controlling industrial robots to move the camera via remote control devices (such as teleoperation devices);

[0010] For example, see patent application CN201410357772.0, which discloses a real-time control method for a camera robot based on teleoperation. This method employs communication between the teleoperation device, the PC server, the robot client, and the communication transmission link to control the robot's real-time movement in response to real-time motion command signals. Another example is application CN202121944975.1, which discloses an external triggering device suitable for film and television shooting robots, attempting to improve the real-time performance of signal triggering for camera robots through an external triggering device. However, this real-time control method requires skilled professional operators and manual coordination from the cinematographer to adjust shooting parameters according to the camera movement path, placing extremely high demands on the accurate coordination between professional technicians and cinematographers.

[0011] Furthermore, in existing technologies, to improve shooting quality during camera movement, methods for adjusting or optimizing robot paths have been provided for industrial robots. For example, CN 202010044804.7 discloses a method for calculating the real-time pose of a six-axis robot end effector following a target object. This method can solve for tracking a target object that has no connection to the robot and calculate the pose that the robot end effector should adjust to ensure that it always remains horizontal with the worktable. Another example is CN...

[0012] Application 202110949804.6 discloses a target tracking method based on a camera robot. This method uses a target tracking algorithm to adjust the robot arm and camera in real time during the robot's movement trajectory, ensuring that the camera lens is always aligned with the target object and that the focal length of the camera lens is always at the same distance from the target object. However, this real-time solution and path adjustment method places higher demands on the hardware and software, such as requiring the robot's controller to have higher data processing capabilities, which correspondingly increases the actual filming cost. Summary of the Invention

[0013] The purpose of this invention is to provide a semi-automatic control method for a photography robot, which partially solves or alleviates the above-mentioned shortcomings in the prior art and can improve the stability and accuracy of robot camera movement.

[0014] To solve the aforementioned technical problems, the present invention specifically adopts the following technical solution: A first aspect of the present invention provides a semi-automatic control method for a photography robot, the method comprising:

[0015] S101 acquires a first input signal combination via the remote control module, wherein the first input signal combination includes one or more of the following signals:

[0016] (i) Signals associated with the remote control status of the remote control module;

[0017] (ii) Signals associated with the actual operating status of the remote control module and / or robot module;

[0018] S102 switches or maintains the remote control module in the corresponding functional state according to the first input signal combination;

[0019] S103 When the remote control module is in the first functional state, the second input signal combination associated with the robot module is acquired, the second input signal combination including: a first sub-combination input through the remote control module;

[0020] The first sub-combination includes one or more of the following signals:

[0021] (1) First joystick signal, the first joystick signal including: spatial motion trajectory data of the first joystick; (2) Second joystick signal, the second joystick signal including: Euler angle change data of the second joystick; (3) Length signal, the length signal including: data associated with the task execution length of the robot module;

[0022] S104 converts the second input signal combination into a first motion command through a server connected to the remote control module;

[0023] S105 splits the first motion command into a first command and a second command through the server, and sends the first command and the second command to the robot unit and the camera unit in the robot module, respectively.

[0024] In some embodiments, prior to S104, the following step is also included:

[0025] Obtain first node information for at least two first nodes from the motion trajectory data. The first node information includes: first node coordinates and node labels associated with the first node coordinates.

[0026] The first fitted curve is obtained by using cubic spline interpolation through the node coordinates;

[0027] Determine whether the first fitted curve matches the operating parameters of the robot module. If yes,

[0028] If not, execute S104; otherwise, prompt the user to update the first joystick signal.

[0029] In some embodiments, the operating parameters include: workspace boundaries; correspondingly, the step of determining whether the first fitted curve matches the operating parameters of the robot module includes:

[0030] Obtain the first distance between the first fitted curve and the workspace boundary;

[0031] Determine whether the first spacing falls within a preset spacing range;

[0032] If yes, then execute S104; otherwise, prompt the user to update the first joystick signal.

[0033] In some embodiments, prior to S104, the following step is also included:

[0034] First node information of at least two first nodes is obtained from the motion trajectory data, the first node information including: first node coordinates and a first node label associated with the first node coordinates; and second node information of at least one second node is obtained from the Euler angle variation data, the second node information including: second node attitude angle and a second node label associated with the attitude angle;

[0035] The second fitted curve is calculated using numerical analysis methods based on the information from the first and second nodes.

[0036] The inverse kinematics method is used to calculate at least one axis motion line of the robot unit based on the second fitted curve. The axis motion line is a curve that reflects the relationship between the angle between the joint axes of the robot and time.

[0037] Determine whether the included angle of the joint axis falls within a preset range based on the axis motion line;

[0038] If yes, then execute S104; otherwise, prompt the user to update the first sub-combination.

[0039] And / or, the second input signal combination further includes: a second sub-combination received or acquired from the robot unit, and the second sub-combination includes one or more of the following signals:

[0040] (1) A first feedback signal, the first feedback signal including: the axis speed of at least one axis of the robot unit, and a corresponding first feedback tag, and the second feedback tag including: the time or node number corresponding to the axis speed;

[0041] (2) Second feedback signal, the second feedback signal includes: the world coordinates of the robot end of the robot unit, and the corresponding second feedback label, and the second feedback label includes: the time or node number corresponding to the world coordinates.

[0042] In some embodiments, the first instruction includes: fifth node information of at least two fifth nodes, the fifth node information including: the coordinate position of the fifth node, and the node label corresponding to the coordinate position; accordingly, the method further includes the step of:

[0043] When the server receives at least one corresponding feedback signal within a first set time period I

[0044] Determine whether the feedback signal matches the first instruction. If yes, determine that the actual operating state of the robot module is normal. If no, determine that the actual motion state is the first abnormal state.

[0045] And / or, if the server does not receive the feedback signal within the second set time II, then the actual motion state of the robot module is determined to be a second abnormal state.

[0046] In some embodiments, the method further includes the step of:

[0047] When the server detects that the robot module is in the first abnormal state, it corrects the second instruction according to the feedback signal and generates a first correction signal accordingly.

[0048] The server sends the first correction signal to the camera unit;

[0049] And / or, the method further includes the step of:

[0050] When the server detects that the robot module is in the second abnormal state, the server sends a first communication signal to the robot unit so that the robot unit sends the feedback signal directly to the camera unit;

[0051] The camera unit adaptively adjusts the camera parameter set based on the feedback signal;

[0052] And when the server receives a feedback signal from the robot unit within a third set time III, the server sends a second communication signal to the robot unit so that the robot unit stops sending feedback signals directly to the camera unit.

[0053] In some embodiments, S103 includes the step of:

[0054] S31 detects a user-defined signal, the user-defined signal including at least one or more of the following:

[0055] (1) The signal type of the signal to be added;

[0056] (2) The signal type of the removal signal to be removed;

[0057] (3) Status information of functions to be customized;

[0058] S32 responds to the custom signal by adding the add signal to the corresponding functional state, or by removing the remove signal from the functional state.

[0059] In some embodiments, prior to S32, the following step is also included:

[0060] Obtain at least one of the following priority information:

[0061] (1) The first preset priority corresponding to the added signal or the removed signal;

[0062] (2) The second preset priority corresponding to the functional state;

[0063] (3) The third preset priority of the predefined signal type in the functional state;

[0064] Determine whether the first preset priority matches the second or third preset priority. If yes, execute the corresponding add or remove step. If no, issue the corresponding prompt signal to the user.

[0065] In some embodiments, the spatial motion trajectory data includes: the initial position of the first joystick, and at least one first offset of the first joystick relative to the initial position on at least one axis;

[0066] In some embodiments, the Euler angle variation data includes: the original Euler angle of the second joystick, and at least one second offset generated by the second joystick about at least one axis;

[0067] In some embodiments, the length signal includes: the task that the robot module needs to perform.

[0068] The starting node information;

[0069] In some embodiments, the second instruction includes one or more of the following parameters: aperture, ISO, and focal length.

[0070] A second aspect of the present invention is that a semi-automatic control system for a photography robot is also provided, the system comprising:

[0071] A first input module is configured to acquire a first input signal combination, wherein the first input signal combination includes one or more of the following signals:

[0072] (i) Signals associated with the remote control status of the remote control module;

[0073] (ii) Signals associated with the actual operating status of the remote control module and / or robot module;

[0074] The function switching module is configured to switch or maintain the remote control module in a corresponding functional state according to the first combination of input signals;

[0075] The second input module is configured to acquire a second input signal combination associated with the robot module when the remote control module is in a first functional state. The second input signal combination includes a first sub-combination input through the remote control module.

[0076] The first sub-combination includes one or more of the following signals:

[0077] (1) First joystick signal, the first joystick signal including: spatial motion trajectory data of the first joystick; (2) Second joystick signal, the second joystick signal including: Euler angle change data of the second joystick; (3) Length signal, the length signal including: data associated with the task execution length of the robot module;

[0078] The first instruction conversion module is configured to convert the second input signal combination into a first motion instruction;

[0079] The first instruction transmission module is configured to split the first motion instruction into a first instruction and a second instruction, and send the first instruction and the second instruction to the robot unit and the camera unit in the robot module, respectively.

[0080] Beneficial technical effects:

[0081] This invention provides a semi-automated decision-making model that combines human and artificial intelligence. Specifically,

[0082] The semi-automatic decision-making mode in this invention can switch to a specific functional state corresponding to different application scenarios or stages. For example, for stages such as test shots and actual shots, the semi-automatic decision-making mode can switch to a first functional state and a second functional state, respectively.

[0083] On the one hand, this flexible switching of functional states provides users with greater operational flexibility, meeting their needs for both manual adjustment and intelligent decision-making in different scenarios. On the other hand, setting specific function combinations for specific application scenarios (such as specific operating or control states) also facilitates more accurate collaboration between human and intelligent decision-making.

[0084] From the user's perspective, in a single operation, the user only needs to input a limited number of signal types through the remote control module. Therefore, this semi-automated decision-making mode reduces the learning and operational difficulty for the user to some extent. From the computer's perspective, for a limited number of signal combinations, this invention further ensures accurate and synchronized operation between the robot unit and the camera unit by processing the collected signal combinations as a whole and then breaking them down into smaller parts for operation.

[0085] From another perspective, unlike traditional technical approaches in existing technologies, the main way this invention improves camera movement stability is not by improving the robot's trajectory planning algorithm, but by providing a simple external control method / system that can quickly match and interface with existing equipment (such as existing six-axis robots and cameras). This control method, on the one hand, can quickly find the appropriate tracking data for existing equipment based on limited signal input through a combination of human and intelligent decision-making; on the other hand, it achieves synchronous processing of data from the server, robot unit, and camera unit through data transmission. Furthermore, because this invention avoids significant modifications to the robot's internal trajectory planning algorithm, it can easily interface with various robot products in practical applications, making it applicable to a relatively wide range of products. Attached Figure Description

[0086] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. In all the drawings, similar elements or parts are generally identified by similar reference numerals. The elements or parts in the drawings are not necessarily drawn to scale. Obviously, the drawings described below are some embodiments of the present invention, and those skilled in the art can obtain other drawings based on these drawings without any creative effort.

[0087] Figure 1 This is a flowchart illustrating a robot control method in an exemplary embodiment of the present invention;

[0088] Figure 2 This is a schematic diagram of the modules of a robot control system in an exemplary embodiment of the present invention;

[0089] Figure 3 This is a schematic diagram illustrating the data communication relationship between the remote control module and the server in an exemplary embodiment of the robot control system of the present invention;

[0090] Figure 4 This is a schematic diagram illustrating the process of updating the motion trajectory in a third control state in an exemplary embodiment of the present invention;

[0091] Figure 5 This is a flowchart illustrating a method for synchronizing data between a camera and a robotic unit in an exemplary embodiment of the present invention. Detailed Implementation

[0092] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention.

[0093] In this document, suffixes such as "module," "part," or "unit" used to denote elements are used only for the purpose of illustrative purposes and have no specific meaning in themselves. Therefore, "module,"

[0094] "Component" or "unit" can be used interchangeably.

[0095] In this document, the terms "upper," "lower," "inner," "outer," "front," "rear," "one end," and "the other end," etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are used only for the convenience of describing the present invention and for simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on the present invention. Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance.

[0096] In this document, unless otherwise explicitly specified and limited, the terms "installed," "equipped with," "connected," etc., should be interpreted broadly. For example, "connection" can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection, a direct connection, or an indirect connection through an intermediate medium; it can be a connection within two components. Those skilled in the art can understand the specific meaning of the above terms in this invention based on the specific circumstances.

[0097] In this document, "and / or" includes any and all combinations of one or more of the listed related items.

[0098] In this article, "multiple" means two or more, that is, it includes two, three, four, five, etc.

[0099] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0100] As used in this specification, the term "approximately" typically means + / - 5% of the stated value.

[0101] More typically, the value is + / -4%, more typically, the value is + / -3%, more typically, the value is + / -2%, even more typically, the value is + / -1%, even more typically, the value is + / -4%.

[0102] + / - 0.5%.

[0103] In this specification, some embodiments may be disclosed in a range-bound format. It should be understood that this "range-bound" description is merely for convenience and brevity and should not be construed as a rigid limitation on the disclosed range. Therefore, the description of a range should be considered as specifically disclosing all possible subranges and the individual numerical values ​​within that range. For example, a description of the range 1-6 should be considered as specifically disclosing subranges such as 1 to 3, 1 to 4, 1 to 5, 2 to 4, 2 to 6, 3 to 6, etc., and the individual numbers within this range, such as 1, 2, 3, etc.

[0104] 4, 5, and 6. The above rules apply regardless of the breadth of the scope.

[0105] In this article, "user" usually refers to the actual operator / photographer, or it can also refer to a computer connected to one or more of the remote control module, server, or robot module.

[0106] Example 1

[0107] like Figure 1 As shown, the present invention provides a semi-automatic control method for a photography robot, the method comprising:

[0108] S101 acquires a first input signal combination, wherein the first input signal combination includes one or more of the following signals:

[0109] (i) Signal A associated with the remote control status of the remote control module;

[0110] (ii) Signal B associated with the actual operating status of the remote control module and / or robot module.

[0111] S102 switches or maintains the remote control module in the corresponding functional state according to the first input signal combination.

[0112] In some embodiments, signal A is a switching signal manually input by the user, such as a switching signal manually input by the user (i.e., the operator, such as a photographer) indicating a switch to a first functional state or a second functional state. Alternatively, signal A may also be a preset default signal in the remote control module, and the default signal is associated with a preset functional state, that is, when the remote control module is turned on, it will automatically maintain or switch to the corresponding functional state.

[0113] In some embodiments, signal B is historical operation record information of the remote control module (such as the functional state during the last operation). Alternatively, signal B is the current actual operating state of the robot module (such as stationary or moving state), and when the robot is stationary, the remote control module can preferably be switched to the first functional state.

[0114] S103 When the remote control module is in the first functional state, the second input signal combination associated with the robot module is acquired. The second input signal combination includes: a first sub-combination input through the remote control module.

[0115] S104 converts the second input signal combination into a first motion command.

[0116] S105 splits the first motion command into a first command and a second command, and sends the first command and the second command to the robot unit and the camera unit in the robot module, respectively.

[0117] Preferably, in some embodiments, steps S101-S103 described above can be performed by a remote control module.

[0118] Steps S104 and S105 can be executed by a server (such as a computer) connected to the remote control module.

[0119] In some embodiments, a preset trajectory planning algorithm in the robot unit may be used to calculate the first motion command (such as the position and orientation of the robot end and the camera parameter set) based on the combination of the second input signals.

[0120] In some embodiments, the first instruction includes: fifth node information of at least two fifth nodes.

[0121] The fifth node information includes: the coordinate position of the fifth node (such as spatial position and Euler angles), and the node label corresponding to the coordinates.

[0122] In some embodiments, node labels include: a timestamp (or time) of the node, or a node number (which can be used to reflect the relative position of the node in the robot's motion trajectory). Therefore, node labels can be used to identify and associate corresponding nodes.

[0123] In some embodiments, the second instruction includes: a camera parameter set associated with at least one of the nodes (such as the fifth node), the camera parameter set including: one or more parameters such as aperture, ISO, focal length, etc., and a timestamp or node number corresponding to the parameters (which can be used to reflect the speed or time of camera parameter adjustment).

[0124] In some embodiments, the robot unit is at least one industrial robot, such as a six-axis robot.

[0125] Or three-axis robots, five-axis robots, etc.

[0126] In some embodiments, the camera unit is at least one camera, camcorder, or other device that uses optical principles to image and record images.

[0127] The following describes the preferred operating states (such as...) included in the first functional state of the remote control module in the embodiments of the present invention. Figure 3 The following is an example illustration:

[0128] I. First operation state f1 (also known as dot-mapping state)

[0129] Preferably, in some embodiments, the first operating state is suitable for the user to manually mark the camera movement path before carrying out the actual shooting task.

[0130] To meet users' manual point-tracking needs, specific signal types are collected in the first operating state.

[0131] Determined by the first sub-combination, and the first sub-combination includes one or more of the following signals:

[0132] (1) First joystick signal, the first joystick signal includes: spatial motion trajectory data of the first joystick;

[0133] For example, in some embodiments, the first joystick includes three degrees of freedom of motion, and accordingly, the spatial motion trajectory data further includes the spatial coordinate position M(x, y, z) of at least one point (or node) in the motion trajectory of the first joystick in space (such as the world coordinate system).

[0134] (2) Second joystick signal, the second joystick signal includes: Euler angle change data of the second joystick;

[0135] For example, in some embodiments, the second joystick also includes three degrees of freedom of motion, and accordingly, the Euler angle variation data further includes: the original pose N1 (u1, v1, w1) of the second joystick in space, and the pose N2 (u2, v2, w2) of the second joystick after it has moved under the user's operation.

[0136] (3) Length signal, the length signal including: data associated with the task execution length of the robot module;

[0137] For example, in some embodiments, the task execution length is the task execution duration T, or the execution trajectory length L, or the execution trajectory range (for example, it could be a trajectory range with high execution difficulty selected by the user). That is, before executing the actual shooting task, the user can set the task execution length to pre-rehearse the actual movement trajectory of some points, and then the photographer or the corresponding server can judge whether the pre-rehearsal effect meets expectations.

[0138] In some embodiments, the remote control module is equipped with a joystick, which rotates when operated by the user.

[0139] The rotation angle or number of rotations of the rocker wheel will be converted into the corresponding task execution length according to the preset conversion rules.

[0140] In this embodiment of the invention, by collecting the first signal combination, the needs such as manual dotting and dotting effect preview can be flexibly realized.

[0141] Furthermore, in some embodiments, the step of S104 is further included:

[0142] Obtain first node information for at least two first nodes from the motion trajectory data. The first node information includes: first node coordinates (x, y, z) and node labels associated with the first node coordinates. The node labels include: first timestamp or node number.

[0143] The first fitted curve is obtained by using the node coordinates through cubic spline interpolation.

[0144] Determine whether the first fitted curve matches the working parameters of the robot module (e.g., the performance parameters of an industrial robot). If yes, execute S104; otherwise, prompt the user to update the first joystick signal.

[0145] In some embodiments, the timestamp can be the actual time 1 of the node's movement (such as the actual moment when the first joystick moves to the corresponding node position). Alternatively, the timestamp can be time 2, which is converted from time 1 using a preset conversion rule; where time 2 is the expected time for the camera to move to the target position corresponding to the node in space.

[0146] In some embodiments, the operating parameters include: workspace boundaries; correspondingly, the step of determining whether the first fitted curve matches the operating parameters of the robot module includes:

[0147] Obtain the first distance between the first fitted curve and the workspace boundary;

[0148] Determine whether the first spacing falls within a preset spacing range;

[0149] If yes, then execute S104; otherwise, prompt the user to update the first joystick signal.

[0150] For example, in some embodiments, the distance between the position of each point in the first fitted curve and the boundary coordinates of the industrial robot's workspace (i.e., the workspace boundary) is calculated. If the distance is too small, it is recommended that the user re-enter the first joystick signal or make partial changes to the first joystick signal.

[0151] Therefore, in this embodiment of the invention, the data for marking points can be determined through a combination of user input and intelligent analysis and evaluation, which provides a semi-automatic marking point method that combines manual and intelligent methods.

[0152] In some embodiments, prior to S104, the following step is also included:

[0153] 1) Obtain first node information for at least two first nodes from the motion trajectory data, wherein the first node information includes: first node coordinates and a first label associated with the first node coordinates (such as including a first timestamp or node number); and obtain second node information for at least one second node from the Euler angle change data, wherein the second node information includes: second node attitude angles, such as Euler angles (u, v, w), and a second label associated with the second node attitude (such as including a second timestamp or node number);

[0154] For example, in some embodiments, the first label and the second label can be used to associate the information of the first node and the second node. For example, the node information with the same / similar number or the same / similar time can be integrated to obtain the complete position and pose (x, y, z, u, v, w) of the node.

[0155] 2) The second fitted curve is calculated using numerical analysis methods (e.g., cubic spline interpolation) through the first node information and the second node information;

[0156] 3) Using inverse kinematics methods (e.g., iterative methods, numerical optimization methods, etc.), calculate at least one axis motion line of the robot unit (such as the six-axis motion curve of an industrial robot) based on the second fitted curve, wherein the axis motion line is used to reflect the angle between the joint axes of the robot and time.

[0157] The curve;

[0158] 4) Determine whether the included angle of the joint axis is within a preset range by using the axis motion line;

[0159] If yes, then execute S104; otherwise, prompt the user to update the first sub-combination.

[0160] In this embodiment of the invention, the first or second joystick signal can be quickly analyzed in advance using numerical simulation methods to reduce the possibility of robot unit motion failures caused by proximity to singularities.

[0161] In some embodiments, the judgment results obtained from the above automatic analysis (such as which nodes are close to singular points) can be combined to manually update some trajectory intervals.

[0162] In some embodiments, the fifth node in the first instruction may be a first node and / or a second node obtained through user input. Alternatively, in other embodiments, the fifth node may be a node selected from a first or second fitting curve that meets the requirements according to a preset interval.

[0163] In this embodiment, the first operating state f1 is preferably applicable to scenarios where the movement trajectory of the object to be photographed is relatively clear. From the user's perspective, only the two joysticks on the remote control module need to be manually rotated to complete the automatic control of the camera movement process.

[0164] Furthermore, the semi-automatic decision-making mode combining artificial intelligence and human intervention in this embodiment of the invention provides users with a certain degree of operational freedom (such as using a joystick adjustment method to assist users in achieving simple and quick manual adjustments). On the other hand, this semi-automatic decision-making mode, while meeting the photographer's lens adjustment needs, can assist the photographer in quickly finding a set of point data (such as corresponding node information) that matches the robot's working performance. In other words, this invention reduces the performance requirements of the robot unit to a certain extent, enabling the method to be compatible with more types of robot units (such as different models of robotic arms).

[0165] Achieve good compatibility.

[0166] II. Second operating state f2

[0167] In some embodiments, the second operating state is primarily applied during real-time operation of the robot module.

[0168] Synchronize the tasks between the camera unit and the robot unit of the robot module.

[0169] In some embodiments, to improve the processing speed of task synchronization and ensure the accuracy of synchronization processing, the second input signal combination further includes: a second sub-combination received or acquired from the robot unit, and the second sub-combination preferably includes one or more of the following signals:

[0170] (1) A first feedback signal, the first feedback signal including: the axis speed of at least one axis of the robot unit (i.e. the actual working parameters of the motor in the industrial robot), and a corresponding first feedback tag, and the first feedback tag including: a first feedback time T1 corresponding to the axis speed, and / or a node number corresponding to the axis speed.

[0171] In some embodiments, the first feedback time is the time point corresponding to the shaft speed.

[0172] (2) Second feedback signal, the second feedback signal includes: the world coordinates of the robot end of the robot unit, and the corresponding second feedback label, and the feedback label includes: the second feedback time T2 corresponding to the world coordinates, and / or, the node number corresponding to the world coordinates.

[0173] In some embodiments, world coordinates can be the world coordinates of the robot's end effector at the current moment.

[0174] It can also be the world coordinates of the robot end effector over a future period of time (e.g., 1 second, 2 seconds, etc.) (which can be automatically predicted by the robot module).

[0175] In some embodiments, the second feedback time is the time point corresponding to the world coordinates (i.e., the time it takes for the robot end effector to reach the corresponding position).

[0176] like Figure 5 As shown, in some embodiments, the following steps are also included:

[0177] When the server receives at least one feedback signal within a first set time I, it determines whether the feedback signal matches the first instruction. If yes, it determines that the actual operating state of the robot module (or robot unit) is normal; otherwise, it determines that the actual motion state is the first abnormal state.

[0178] If the feedback signal is not received within the second set time II, the actual motion state is determined to be a second abnormal state.

[0179] For example, in some embodiments, it can be determined whether the coordinates of nodes with the same number (which can be obtained from the first motion command or the first instruction) are the same as or similar to the actual world coordinates.

[0180] If so, the actual operating state of the robot unit is determined to be normal; otherwise, it is the first abnormal state.

[0181] For example, in some embodiments, it can be determined whether the shaft speed falls within a preset threshold range.

[0182] If so, its actual operating status is judged to be normal; otherwise, it is the first abnormal state.

[0183] It is understood that the judgment rules / judgment methods in this invention can be preset by the user.

[0184] In some embodiments, 103 further includes the step of:

[0185] When the server detects that the robot module is in the first abnormal state, it corrects the second instruction according to the feedback signal and generates a first correction signal accordingly (such as data including the corrected camera parameter group); the server sends the first correction signal to the camera unit.

[0186] In some embodiments, a corresponding camera parameter set is calculated based on the world coordinates and the actual / expected coordinates of the target object (such as a person, object, etc.) to be photographed, and the second instruction is updated based on the camera parameter set to send the updated second instruction to the camera unit.

[0187] In some embodiments, the method further includes the step of:

[0188] When the server detects that the robot unit is in a second abnormal state, it sends a first communication signal to the camera unit and / or the robot unit to initiate data communication between the camera unit and the robot unit. Specifically, the robot unit directly sends a first feedback signal or a second feedback signal to the camera unit. At this time, the camera unit switches from single-line communication with the server to dual-line communication with both the server and the robot unit.

[0189] In some embodiments, S103 further includes the step of:

[0190] When the server receives the feedback signal again within the third set time III, it sends a second communication signal to the camera unit and / or the robot unit to stop the data communication between the camera unit and the robot unit. That is, the robot unit directly sends either the first or second feedback signal to the server. At this time, the camera unit switches from two-way communication with both the server and the robot unit back to one-way communication with the server, thus centralizing the task threads with the server.

[0191] Preferably, in some embodiments, the camera unit and the robot unit can transmit data via wired communication to ensure the stability of data transmission in a relatively poor network environment.

[0192] Alternatively, in other embodiments, the camera unit and the robot unit may also transmit data via wireless communication methods such as Bluetooth or a network.

[0193] In this embodiment, after the user completes manual point marking, a finite set of parameters (such as axis velocity or world coordinate data set, etc.) is preferably used in conjunction with a three-way data transmission path (i.e., the data transmission path sequentially from the robot unit, the server, to the camera unit) to enable the tasks of the server, robot unit, and camera unit to be performed synchronously. Furthermore, the amount of data transmission and processing in this process is relatively limited, which to some extent reduces the requirements for network environment and hardware conditions during camera movement.

[0194] Therefore, the operating state in this embodiment is better suited for outdoor shooting environments (such as areas with relatively weak network signals, such as mountains, grasslands, and deserts). Furthermore, using a remote control module to wirelessly control the robot can, to some extent, reduce the limitations imposed on users (such as photographers) by terrain, allowing photographers to move freely and flexibly in the vicinity of the robot while simultaneously controlling its camera movements.

[0195] III. The third operation state f3 (also known as the custom state)

[0196] Preferably, in some embodiments, when the remote control module is in any functional state, it can respond to the user's operation to activate the third operation state. At this time, the user can automatically customize the function combination (i.e., signal acquisition type) of the current functional state according to the needs of the filming location (such as indoor environment, outdoor environment, shooting fast shot, shooting slow shot, etc.).

[0197] In some embodiments, S103 includes the step of:

[0198] A user-defined signal is detected, the user-defined signal including at least one or more of the following:

[0199] (1) The signal type of the signal to be added (the specific signal type can be a signal / data name, such as feedback signal, joystick signal, etc.);

[0200] (2) The signal type of the removal signal to be removed;

[0201] (3) Functional status information to be customized (the functional status information can be the name of the functional status - such as first functional status, second functional status or first operation status, etc.);

[0202] In response to the custom signal, the add signal is added to the corresponding functional state, or the remove signal is removed from the functional state.

[0203] In some embodiments, before adding the add signal to the corresponding functional state in response to the custom signal, the method further includes the following steps:

[0204] Obtain at least one of the following priority information:

[0205] (1) The first preset priority corresponding to the added signal or the removed signal;

[0206] (2) The second preset priority corresponding to the functional state;

[0207] (3) The third preset priority of the predefined signal type in the functional state;

[0208] Determine whether the first preset priority matches the second or third preset priority. If yes, execute the corresponding add or remove step. If no, issue the corresponding prompt signal to the user.

[0209] In some embodiments, signal types with the same priority can be combined in the same functional state.

[0210] In some embodiments, the priority includes a blacklist, which includes information about signal types, functional states, or operational states that conflict with the signal (i.e., are not recommended to be enabled simultaneously).

[0211] For example, in some embodiments, when the signal to be added is signal C, and signal C's blacklist includes signal D, while signal D has already been added by default in the current customizable function state, then the user should be prompted that signal C cannot be added directly.

[0212] In some embodiments, the steps further include:

[0213] Generate corresponding indication signals based on the current functional or operational status;

[0214] The function is specially displayed on the remote control module according to the indication signal (such as indicator lights and text display around the function button / joystick to indicate whether the corresponding function / signal is enabled).

[0215] In some embodiments, the spatial motion trajectory data includes: the initial position of the first joystick, and the position of the first joystick relative to the initial position (x, y, z) along at least one axis (such as the x-axis, y-axis, z-axis).

[0216] At least one first offset generated on the y-axis and z-axis;

[0217] In some embodiments, the Euler angle variation data includes: the original Euler angles of the second joystick (equivalent to the original attitude), and at least one second offset generated by the second joystick about at least one axis.

[0218] The following specific embodiment illustrates steps S104 and S105 of the present invention by way of example:

[0219] The camera position and orientation (which is also equivalent to the position and orientation of the robot's end effector) are read as P0 = (x0, y0, z0, u0, v0, w0);

[0220] Using the robot's world coordinates {W} as reference coordinates, this position is rewritten in homogeneous transformation matrix form:

[0221]

[0222] Where W is the first-world coordinate system referenced by the robot, and C is the robot's end-effector coordinate system.

[0223] This represents the transformation from the robot's end-effector coordinate system to the first-world coordinate system; C P0 represents the displacement of the three degrees of freedom on the xyz axes of the robot's end effector (as read from the first joystick signal).

[0224] In response to attitude changes (such as those read from the second joystick signal), the remote control module's attitude changes from the original attitude A to attitude A'. ′ The corresponding transformation matrix is The relationship between two poses can be expressed as:

[0225] Among them, W ′ Let A represent the second-world coordinate system referenced by the remote control module, and let A represent the coordinate system of the remote control module in its original attitude. ′ Represents the coordinates of the remote control module under the changed attitude.

[0226] Tie;

[0227] Further solve for the relative attitude change matrix with the remote control module itself as the reference frame. This matrix is ​​equivalent to the camera's pose change, thus yielding the camera's relative pose change matrix. Then the attitude change

[0228] The subsequent camera pose is

[0229] The rate of change of position can also be obtained from the joystick signal. The changed position can then be obtained as follows:

[0230]

[0231] in, C P t This represents the position of the robot's end effector after its change (which is also equivalent to the position of the camera after its change).

[0232] k1 is the signal scaling factor, and Δt is the sampling time interval.

[0233] In summary, the homogeneous transformation matrix of the changed camera pose is:

[0234]

[0235] Rewrite the matrix in the form of position-ZYX Euler angles:

[0236] P t =(x t ,y t ,z t ,u t ,v t ,w t )

[0237] From P t At least one fifth node information is collected to generate a corresponding second instruction and send it to the robot unit for execution.

[0238] Of course, in some other embodiments, in step S103, the trajectory planning algorithm pre-stored in the robot unit (that is, the internal algorithm of the industrial robot actually selected) can be directly used for processing.

[0239] In some embodiments, the length signal includes: the length of the task that the robot module needs to perform, such as the starting node information of the trajectory interval to be performed.

[0240] In some embodiments, the second instruction includes one or more of the following parameters: aperture, ISO, and focal length.

[0241] It is understood that, on the one hand, the various functional states in this invention can be customized by the user under corresponding preset rules (such as combining according to priority); on the other hand, the various functional states can also be superimposed on each other or freely switched in real time without conflict.

[0242] It is worth noting that, unlike traditional technical approaches, the core approach of this invention lies in:

[0243] First, while satisfying the photographer's manual camera movement adjustments, the most suitable motion trajectory for the stable movement of the robotic unit is found (rather than finding the most suitable or precise motion trajectory for the subject being photographed). Second, specific functional combinations are set for different application scenarios / states to limit the remote control's operational functions to a certain extent (rather than simply adding more functions). Through the cooperation of these two core approaches, this invention simplifies the operation of the remote control and reduces the requirements for equipment hardware (i.e., to a certain extent, it reduces the data processing burden on the camera and robotic arm, and also reduces the requirements for the network communication environment).

[0244] Furthermore, based on this, the present invention also adopts a novel data interaction method (that is, to allocate and manage the data processing tasks of multiple parties such as remote controllers, servers, cameras and robots through the above-mentioned method steps) to further improve the reliability and stability of data processing during real-time operation.

[0245] Furthermore, the present invention can also solve or alleviate the application limitations that may arise when the signal input type is limited by a flexible function switching method.

[0246] Example 2

[0247] like Figure 2 As shown, the present invention also provides a semi-automatic control system for a photography robot, the system comprising:

[0248] Remote control module 10, the remote control module 10 includes:

[0249] A first input module is configured to acquire a first input signal combination, wherein the first input signal combination includes one or more of the following signals:

[0250] (i) Signals associated with the remote control status of the remote control module 10;

[0251] (ii) Signals associated with the actual operating status of the remote control module 10 and / or the robot module 30;

[0252] The function switching module is configured to switch or maintain the remote control module in a corresponding functional state according to the first combination of input signals;

[0253] The second input module is configured to acquire a second input signal combination associated with the robot module when the remote control module is in a first functional state. The second input signal combination includes a first sub-combination input through the remote control module.

[0254] The first sub-combination includes one or more of the following signals:

[0255] (1) First joystick signal, the first joystick signal including: spatial motion trajectory data of the first joystick; (2) Second joystick signal, the second joystick signal including: Euler angle change data of the second joystick; (3) Length signal, the length signal including: data associated with the task execution length of the robot module;

[0256] and a server 20 connected to the remote control module, the server comprising:

[0257] The first instruction conversion module is configured to convert the second input signal combination into a first motion instruction;

[0258] The first instruction transmission module is configured to split the first motion instruction into a first instruction and a second instruction, and send the first instruction and the second instruction to the robot unit 31 (such as a robotic arm) and the camera unit 32 in the robot module, respectively.

[0259] In some embodiments, the server further includes: a first determination module, and the first determination module includes:

[0260] The first node acquisition unit is configured to acquire first node information of at least two first nodes from the motion trajectory data. The first node information includes: first node coordinates and node labels associated with the first node coordinates.

[0261] The first fitting unit is configured to obtain a first fitting curve using cubic spline interpolation through the node coordinates;

[0262] The first judgment unit is configured to determine whether the first fitted curve matches the working parameters of the robot module. If yes, the first instruction transmission module splits and transmits the corresponding start instruction. If no, the user is prompted to update the first joystick signal.

[0263] In some embodiments, the working parameters include: workspace boundaries; correspondingly, the first determination unit includes:

[0264] The first subunit is configured to obtain a first distance between the first fitted curve and the workspace boundary;

[0265] The second subunit is configured to determine whether the first spacing belongs to a preset spacing range;

[0266] If yes, the first instruction transmission module will split and transmit the corresponding start instruction; otherwise, the user will be prompted to update the first joystick signal.

[0267] In some embodiments, the system further includes:

[0268] The second node acquisition unit is configured to acquire at least two nodes from the motion trajectory data.

[0269] The first node information includes: first node coordinates and a first timestamp or node number associated with the first node coordinates; and second node information obtained from the Euler angle variation data for at least one second node, wherein the second node information includes:

[0270] The second node attitude angle, and the second timestamp or node number associated with the attitude angle;

[0271] The second fitting unit is configured to calculate the second fitting curve using the first node information and the second node information through numerical analysis methods.

[0272] The inverse kinematics unit is configured to calculate at least one axis motion line of the robot unit based on the second fitting curve using the inverse kinematics method, the axis motion line being a curve reflecting the relationship between the angle between the joint axes of the robot and time;

[0273] The second judgment unit is configured to determine whether the included angle of the joint axis is within a preset included angle range by the axis motion line; if yes, the first instruction transmission module starts the splitting and transmission of the corresponding instruction; if no, the user is prompted to update the first sub-combination.

[0274] In some embodiments, the second input signal combination further includes: a second sub-combination received or acquired from the robot unit, and the second sub-combination includes one or more of the following signals:

[0275] (1) A first feedback signal, the first feedback signal including: the axis speed of at least one axis of the robot unit, and a corresponding first feedback tag, the first feedback tag including: the time or node number corresponding to the axis speed;

[0276] (2) Second feedback signal, the second feedback signal includes: the world coordinates of the robot end of the robot unit, and the corresponding second feedback label, the second feedback label includes: the time or node number corresponding to the world coordinates.

[0277] In some embodiments, the first instruction includes: fifth node information of at least two fifth nodes, the fifth node information including: the coordinates of the fifth node and the node label corresponding to the coordinates; correspondingly, the server further includes: a status judgment unit;

[0278] Furthermore, the status judgment unit is configured to determine whether the feedback signal matches the first instruction when the server receives at least one corresponding feedback signal within a first set time I. If yes, the actual operating state of the robot module is determined to be normal; otherwise, the actual motion state is determined to be a first abnormal state.

[0279] And / or, the state determination unit is configured to determine that the actual motion state of the robot module is a second abnormal state when the server does not receive the feedback signal within a second set time II.

[0280] In some embodiments, server 20 further includes:

[0281] A correction unit is configured to, when the server detects that the robot module is in the first abnormal state, correct the second instruction according to the feedback signal and generate a first correction signal accordingly; and send the first correction signal to the camera unit.

[0282] And / or, in some embodiments, server 20 further includes:

[0283] An adjustment unit is configured to, when the server detects that the robot module is in the second abnormal state, send a first communication signal to the robot unit, so that the robot unit directly sends the feedback signal to the camera unit; and the camera unit adaptively adjusts the camera parameter set according to the feedback signal.

[0284] And when the server receives a feedback signal from the robot unit within a third set time III, the server sends a second communication signal to the robot unit so that the robot unit stops sending feedback signals directly to the camera unit.

[0285] In some embodiments, the system further includes a custom module, and the custom module includes:

[0286] A custom signal detection unit is configured to detect user-defined signals, which include at least one or more of the following:

[0287] (1) The signal type of the signal to be added;

[0288] (2) The signal type of the removal signal to be removed;

[0289] (3) Status information of functions to be customized;

[0290] The signal addition and removal unit is configured to add the added signal to the corresponding functional state in response to the custom signal, or to remove the first removal signal from the functional state.

[0291] In some embodiments, the custom module further includes a custom evaluation unit, and the custom evaluation unit is configured to perform the following steps:

[0292] Obtain at least one of the following priority information:

[0293] (1) The first preset priority corresponding to the first add signal or the first remove signal;

[0294] (2) The second preset priority corresponding to the functional state;

[0295] (3) The third preset priority of the predefined signal type in the functional state;

[0296] Determine whether the first preset priority matches the second or third preset priority. If yes, execute the corresponding add or remove step. If no, issue the corresponding prompt signal to the user.

[0297] It is understood that the system in the embodiments of the present invention can implement any of the above methods or steps.

[0298] This will not be elaborated upon here.

[0299] Example 3

[0300] Furthermore, in order to enable real-time control of the robot, such as making local adjustments to the robot's trajectory while the robot unit is moving according to the first instruction; or allowing the user to manually control the robot's movement trajectory in real time, this invention also provides a real-time control method for a photographic robot based on the above embodiments.

[0301] In some embodiments, the real-time control method includes:

[0302] S101 acquires a first input signal combination, wherein the first input signal combination includes one or more of the following signals:

[0303] (i) Signals associated with the remote control status of the remote control module;

[0304] (ii) Signals associated with the actual operating status of the remote control module and / or robot module;

[0305] S102 switches or maintains the remote control module in the corresponding functional state according to the first input signal combination;

[0306] S106 When the remote control module is in the second functional state (equivalent to real-time control state), it acquires the third input signal combination associated with the robot module. The third input signal combination includes: the first input combination input through the remote control module.

[0307] The first input combination includes one or more of the following signals:

[0308] (1) First joystick signal, the first joystick signal includes: spatial motion trajectory data of the first joystick; (2) Second joystick signal, the second joystick signal includes: Euler angle change data of the second joystick; (3) Wheel signal, the wheel signal includes: relative position data of the wheel;

[0309] S107 converts the third input signal combination into a second motion command;

[0310] S108 splits the second motion command into a third command (such as node information associated with the robot's motion trajectory) and a fourth command (such as a camera parameter group), and sends the third command and the fourth command to the robot unit and the camera unit in the robot module, respectively.

[0311] In some embodiments, a preset trajectory planning algorithm in the robot unit can be used to calculate a second motion command (such as the position and orientation of the robot end effector, camera parameter sets, etc.) based on a combination of third input signals. For example, the second motion command can be a fitted curve (or fitted trajectory) calculated according to the trajectory planning algorithm.

[0312] In some embodiments, the second motion command may also be a fitted curve calculated based on the combination of the third input signals using other numerical simulation analysis methods.

[0313] In some embodiments, the second functional state includes one or more of the following control states:

[0314] (i) First control state (i.e. real-time control state), and when the remote control module is in the first control state, the remote control module collects relevant data of the first input combination in real time, such as joystick signal or wheel signal;

[0315] (ii) Second control state (i.e., controlling the posture of the robot's end effector via the motion sensing module - IMU).

[0316] Furthermore, when the remote control module is in the second control state, the remote control module collects relevant data of the second input combination of the third input signal combination in real time, wherein the second input combination includes one or more of the following: the first original posture of the remote control module, the first changed posture of the remote control module, and the second original posture of the robot end effector.

[0317] (iii) Third control state (i.e., real-time correction state), and when the remote control module is in the third control state, the remote control module collects relevant data of the third input combination of the third input signal combination in real time, wherein the third input combination includes one or more of the following:

[0318] The third node information includes: the position of at least one third node to be inserted, and the third time of insertion;

[0319] The fourth node information includes the position of at least one fourth node inserted near the third node.

[0320] The preferred control states included in the second functional state of the remote control module in the embodiments of the present invention will be described by way of example below:

[0321] I. First control state f'1

[0322] Preferably, the first control state f'1 in this embodiment of the invention is suitable for the user to input and adjust the motion trajectory of the industrial robot in real time.

[0323] For example, in some embodiments, when the camera movement is relatively simple, such as when tracking an object moving in a straight line, the first input combination can be a first joystick signal input in real time.

[0324] For example, in some embodiments, the first joystick signal can be quickly analyzed using numerical simulation methods (such as cubic spline interpolation) to determine whether the current real-time input signal meets the operating rules of the robot unit (such as whether it can match the corresponding operating parameters of the motor). Specifically, the determination method or steps can be found in Embodiments 1 and 2 above, and will not be repeated here.

[0325] For example, in some embodiments, when the remote control module is in the first control state f'1, the first input combination includes: a first joystick signal and a second joystick signal input synchronously. Similarly, the rapid analysis and evaluation of the first joystick signal and the second joystick signal can be found in Embodiments 1 and 2 above, and will not be repeated here.

[0326] For example, in some embodiments, the first input combination includes a first joystick signal, a second joystick signal, and a rocker wheel signal that are input synchronously. The rocker wheel signal can be the rotation angle or number of rotations of a rocker wheel mounted on the remote control module, allowing the user to control the movement trajectory range of the robot unit by rotating the rocker wheel.

[0327] II. Second control state f'2

[0328] Preferably, in this embodiment of the invention, the second control state f'2 is suitable for the user to follow the spatial motion trajectory of the robot's end effector and adjust the posture (such as Euler angles) of the robot's end effector in real time.

[0329] Preferably, in this embodiment of the invention, the second control state adjusts the posture of the robot's end effector through a motion sensor mounted on the remote control module. The motion sensor can detect changes in the posture of the remote control module in space. In this state, the user can directly control the posture of the robot's end effector by manually rotating or tilting the remote control module.

[0330] In some embodiments, the motion sensor may be an inertial measurement unit (IMU).

[0331] III. Third control state f'3

[0332] Preferably, the third control state f'3 in this embodiment of the invention is suitable for locally adjusting the robot's motion trajectory during the robot's movement according to the first or third instruction.

[0333] In some embodiments, when the remote control module is in the third control state, S106 includes the following steps:

[0334] S61 determines the simulation constraints based on the third node information, the constraints including: a position threshold range of at least one endpoint, or a derivative threshold range of at least one endpoint;

[0335] S62 uses cubic spline interpolation to simulate and obtain the third fitted curve based on the constraints and the fourth node information.

[0336] For example, in some embodiments, such as Figure 4 As shown, when real-time adjustment of the robot's end effector trajectory L1 is required, the coordinates of the endpoints of the region to be inserted in trajectory L1 (i.e., the coordinates of the two third nodes a and b) can be obtained first. Constraints are then determined based on the position coordinates of nodes a and b (e.g., the coordinate range of the endpoints on both sides of trajectory L2, i.e., the position threshold range). Furthermore, information about the new node to be inserted (i.e., the coordinates of at least one fourth node inserted between nodes a and b) needs to be obtained. Finally, a new local trajectory L2 (equivalent to the third fitting curve) is calculated using a numerical simulation algorithm based on the constraints and the new node coordinates. The complete robot trajectory L3 can then be obtained based on trajectory L2 and trajectory L1' (i.e., trajectory L1 excluding the region between nodes a and b).

[0337] For example, in some embodiments, preset rules can be used to determine the constraints (i.e., derivative threshold range) of the derivative at the corresponding endpoints based on the derivative values ​​at nodes a and b.

[0338] In some embodiments, the method further includes the step of:

[0339] The relationship between the motion velocity and axis velocity of the robot's end effector is established using the Jacobian matrix method.

[0340] The relationship is expressed as follows:

[0341] Where v represents the Cartesian velocity vector, and Θ represents the joint angle vector of the robot (such as a robotic arm). J represents the joint angular velocity vector of the robot. -1 Represents the Jacobian matrix;

[0342] In some embodiments, the robot's axis angular velocity at each moment can be calculated based on the fitted curve (such as the third fitted curve) and the motion velocity v. When the axis angular velocity is greater than a predetermined limit velocity θ... max If the robot enters a strange range where it is prone to shaking, the Cartesian position at that time is recorded and sent to the user, prompting the user to make adaptive adjustments.

[0343] In some embodiments, prior to S61, the following step is also included:

[0344] The insertion interval is determined based on the third node information and the fourth node information. The insertion interval includes: the time difference between the third time and the current time, and / or, the spatial interval between the third node and the current node.

[0345] Determine whether the insertion interval falls within a preset threshold range; if so, proceed to the executed S61.

[0346] If not, a prompt signal will be sent to the user.

[0347] In this embodiment of the invention, when a user performs real-time operations such as adding, deleting, modifying, inserting, or moving, the insertion position or insertion time is first analyzed to avoid damage to the robot motor caused by improper insertion intervals. For example, if the node to be inserted is too close to the actual node position reached by the robot, which is prone to causing undesirable vibrations, it is recommended that the user modify the inserted node.

[0348] In some embodiments, when the remote control module is in the second control state, the second motion command includes: Euler angle data, and the remote control module is also provided with a motion sensing module (such as an attitude sensor) to monitor the spatial attitude of the remote control module (dynamic changes such as flipping and tilting of the remote control module); correspondingly, step S106 includes:

[0349] A first rotation matrix is ​​calculated using a matrix transformation method based on the first original posture (i.e., the original posture of the remote control module) and the first changed posture (the offset of the remote control module on at least one axis). The first rotation matrix represents the relative rotation amount of the first changed posture in the coordinate system of the first original posture.

[0350] The second rotation matrix is ​​calculated using a matrix transformation method based on the first rotation matrix and the second original pose (i.e., the original pose of the robot end effector). The second rotation matrix represents the motion trajectory of the robot end effector around the zyx axis.

[0351] Euler angle data for at least one node are collected from the second rotation matrix.

[0352] In some embodiments, the fourth instruction includes one or more of the following parameters: aperture, ISO, and focal length.

[0353] In some embodiments, the third input signal combination further includes: a second input combination received or acquired from the robot unit, wherein the second input combination includes one or more of the following signals:

[0354] (1) A first feedback signal, the first feedback signal including: the axis speed of at least one axis of the robot unit, and a corresponding first feedback tag, the first feedback tag including: the time or node number corresponding to the axis speed;

[0355] (2) Second feedback signal, the second feedback signal includes: the world coordinates of the robot end of the robot unit, and the corresponding second feedback label, and the second feedback label includes: the time or node number corresponding to the world coordinates.

[0356] In some embodiments, the third instruction includes: sixth node information of at least two sixth nodes, the sixth node information including: the position of the sixth node (such as spatial coordinates and Euler angles), and the node label corresponding to the position.

[0357] For example, in some embodiments, the sixth node information may be at least one node information collected from the third fitting curve. In other embodiments, the sixth node may be corresponding node information directly obtained from the first or second joystick signal. Furthermore, in some embodiments, the sixth node information may also be at least one node information collected at intervals from the first or second fitting curve.

[0358] In some embodiments, the method further includes the step of:

[0359] When the server receives at least one corresponding feedback signal within a first set time period I

[0360] Determine whether the feedback signal matches the third instruction. If yes, determine that the actual operating state of the robot module is normal. If no, determine that the actual motion state is the first abnormal state.

[0361] In some embodiments, if the server does not receive the feedback signal within a second set time II, the actual motion state of the robot module is determined to be a second abnormal state.

[0362] In some embodiments, 103 further includes the step of:

[0363] When the server detects that the robot module is in the first abnormal state, it corrects the fourth instruction according to the feedback signal and generates a second correction signal accordingly.

[0364] The server sends the second correction signal to the camera unit;

[0365] In some embodiments, the method further includes the step of:

[0366] When the server detects that the robot module is in the second abnormal state, the server sends a first communication signal to the robot unit so that the robot unit sends the feedback signal directly to the camera unit;

[0367] The camera unit adaptively adjusts the camera parameter set based on the feedback signal;

[0368] And when the server receives a feedback signal from the robot unit within a third set time III, the server sends a second communication signal to the robot unit so that the robot unit stops sending feedback signals directly to the camera unit.

[0369] It is understood that the real-time control method in this invention can be used in conjunction with the semi-automatic control method in the above embodiments.

[0370] This invention essentially provides an independent control method to help ordinary film and television workers externally control robots. Typically, when using robots to control camera movements, precise timing is crucial; otherwise, issues like stuttering and shaky shots can easily occur. Furthermore, before achieving the desired shooting effect, multiple test shots are often necessary. During these tests, significant adjustments are required to the subject's performance (e.g., changes in movement) and the camera movement (e.g., the camera's rotation path, direction, or speed). The purpose of this invention is to allow users to flexibly adjust the actual camera movement through manual intervention during these dynamic changes.

[0371] During actual filming, the robot's camera movement can be controlled in real time via remote control to complete the shot. Unlike traditional automatic camera movement methods, the robot control method in this invention does not require prior acquisition of route trajectories (such as positioning points) and can acquire information in real time through external signals.

[0372] Furthermore, to help users quickly input accurate control data, this invention also provides a semi-automatic task decision-making mode that combines manual and automatic decision-making. This task decision-making mode can be completed by a single person, reducing the professional skill requirements for the operator; on the other hand, it can reduce the precision requirements of the motor to a certain extent, thereby achieving stable and accurate camera movement at a lower cost.

[0373] In other words, unlike traditional technologies, the technical approach used in this invention to provide camera movement stability does not directly involve selecting higher-precision industrial robots. Instead, this invention preferably uses low-precision industrial robots and employs a semi-automated task decision-making mechanism to improve the stability and accuracy of the low-precision industrial robots during camera movement.

[0374] Example 4

[0375] like Figure 2 As shown, the present invention also provides a real-time control system for a photography robot, the system comprising: a remote control module 10, and the remote control module 10 further comprising:

[0376] A first input module is configured to acquire a first input signal combination, wherein the first input signal combination includes one or more of the following signals:

[0377] (i) Signals associated with the remote control status of the remote control module;

[0378] (ii) Signals associated with the actual operating status of the remote control module and / or robot module;

[0379] The function switching module is configured to switch or maintain the remote control module in a corresponding functional state according to the first combination of input signals;

[0380] The third input module is configured to acquire the third input signal combination associated with the robot module when the remote control module is in the second functional state, the third input signal combination including: a first input combination input through the remote control module;

[0381] The first input combination includes one or more of the following signals:

[0382] (1) First joystick signal, the first joystick signal includes: spatial motion trajectory data of the first joystick; (2) Second joystick signal, the second joystick signal includes: Euler angle change data of the second joystick; (3) Joystick signal, the joystick signal includes: relative position data of the joystick;

[0383] and a server 20 connected to the remote control module 10, wherein the server 20 includes:

[0384] The second instruction conversion module is configured to convert the combination of the third input signals into a second motion instruction;

[0385] The second instruction transmission module is configured to split the second motion instruction into a third instruction and a fourth instruction, and send the third instruction and the fourth instruction to the robot unit 31 and the camera unit 32 in the robot module 30, respectively.

[0386] In some embodiments, the second functional state includes:

[0387] (i) A first control state, and when the remote control module is in the first control state, the remote control module collects relevant data of the first input combination in real time;

[0388] (ii) Second control state, and when the remote control module is in the second control state, the remote control module collects relevant data of the second input combination of the third input signal combination in real time, wherein the second input combination includes one or more of the following: the first original posture of the joystick module, the first changed posture of the joystick module, and the second original posture of the robot end;

[0389] (iii) In the third control state, when the remote control module is in the second control state, the remote control module collects relevant data of the third input combination of the third input signal combination in real time, wherein the third input combination includes one or more of the following:

[0390] The third node information includes: the position of at least one third node to be inserted, and the third time of insertion;

[0391] The fourth node information includes the position of at least one fourth node inserted near the third node.

[0392] In some embodiments, the fourth instruction includes one or more of the following parameters: aperture, ISO, and focal length.

[0393] In some embodiments, the server further includes a simulation module configured to perform the following steps: determining simulation constraints based on the third node information, the constraints including: a position threshold range of at least one endpoint, or a derivative threshold range of at least one endpoint; and simulating a third fitted curve using cubic spline interpolation based on the constraints and the fourth node information. In some embodiments, the simulation module is further configured to perform the following steps: confirming an insertion interval based on the third node information and the fourth node information, the insertion interval including: the time difference between the third time and the current time, and / or, the spatial interval between the third node and the current node;

[0394] Determine whether the insertion interval falls within a preset interval range; if not, issue a prompt signal to the user to prompt the user to modify the third input combination.

[0395] It is understood that the present invention may include any of the functional modules / units in the above embodiments, and may also be used to implement the methods or steps in any of the above embodiments, which will not be repeated here.

[0396] It should be noted that, in addition to the robotic arm disclosed in the above embodiments, the present invention may also use any kind of robot / robotic arm that can be used to achieve the above functions, such as multi-joint robotic arms, Cartesian coordinate system robotic arms, spherical coordinate system robotic arms, polar coordinate system robotic arms, cylindrical coordinate system robotic arms, etc.

[0397] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0398] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a computer terminal (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of the present invention.

[0399] The embodiments of the present invention have been described above with reference to the accompanying drawings. However, the present invention is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of the present invention without departing from the spirit and scope of the claims. All of these forms are within the protection scope of the present invention.

Claims

1. A semi-automatic control method for a photography robot, characterized in that, The method includes: S101 acquires a first input signal combination via the remote control module, wherein the first input signal combination includes one or more of the following signals: (i) Signal A associated with the remote control state of the remote control module; and signal A includes: a switching signal input by the user indicating switching to the first functional state or the second functional state, or a default signal preset in the remote control module; (ii) Signal B, indicating the actual operating status of the remote control module and / or robot module; S102 switches or maintains the remote control module in the corresponding functional state according to the first input signal combination; S103 When the remote control module is in the first functional state, a second input signal combination associated with the robot module is acquired. The second input signal combination includes a first sub-combination input through the remote control module. The first functional state includes a first operation state. The first operation state is used by the user to manually mark the camera movement route before performing the actual shooting task. The first sub-combination includes one or more of the following signals: (1) First joystick signal, the first joystick signal includes: spatial motion trajectory data of the first joystick; (2) Second joystick signal, the second joystick signal includes: Euler angle change data of the second joystick; (3) Length signal, the length signal includes: starting node information of the task to be performed by the robot module; S104 converts the current combination of the second input signals into a first motion command; S105 splits the first motion command into a first command and a second command, and sends the first command and the second command to the robot unit and the camera unit in the robot module, respectively.

2. The semi-automatic control method according to claim 1, characterized in that, Before S104, the following steps are also included: Obtain first node information for at least two first nodes from the spatial motion trajectory data. The first node information includes: first node coordinates and node labels associated with the first node coordinates. The first fitted curve is obtained by using numerical analysis methods based on the coordinates of the first node. Determine whether the first fitted curve matches the working parameters of the robot module. If yes, execute S104; otherwise, prompt the user to update the first joystick signal.

3. The semi-automatic control method according to claim 2, characterized in that, The operating parameters include: workspace boundaries; correspondingly, the step of determining whether the first fitted curve matches the operating parameters of the robot module includes: Obtain the first distance between the first fitted curve and the workspace boundary; Determine whether the first spacing falls within a preset spacing range; If yes, then execute S104; otherwise, prompt the user to update the first joystick signal.

4. The semi-automatic control method according to claim 1, characterized in that, Before S104, the following steps are also included: First node information of at least two first nodes is obtained from the spatial motion trajectory data, the first node information including: first node coordinates and first node label associated with the first node coordinates; and second node information of at least one second node is obtained from the Euler angle variation data, the second node information including: second node attitude angle and second node label associated with the attitude angle. The second fitted curve is calculated using numerical analysis methods based on the information from the first and second nodes. The inverse kinematics method is used to calculate at least one axis motion line of the robot unit based on the second fitted curve. The axis motion line is a curve that reflects the relationship between the angle between the joint axes of the robot and time. Determine whether the included angle of the joint axis falls within a preset range based on the axis motion line; If yes, then execute S104; otherwise, prompt the user to update the first sub-combination. And / or, the second input signal combination further includes: a second sub-combination received or acquired from the robot unit, and the second sub-combination includes one or more of the following signals: (1) A first feedback signal, the first feedback signal including: the axis speed of at least one axis of the robot unit, and a corresponding first feedback tag, and the first feedback tag including: the time or node number corresponding to the axis speed; (2) Second feedback signal, the second feedback signal includes: the world coordinates of the robot end of the robot unit, and the corresponding second feedback label, and the second feedback label includes: the time or node number corresponding to the world coordinates.

5. The semi-automatic control method according to claim 1, characterized in that, The second instruction includes one or more of the following parameters: aperture, ISO, and focal length.

6. The semi-automatic control method according to claim 1, characterized in that, The first joystick has three degrees of freedom of motion, and correspondingly, the spatial motion trajectory data includes the spatial coordinate position M(x, y, z) of at least one node of the motion trajectory of the first joystick in space.

7. The semi-automatic control method according to claim 1, characterized in that, The second joystick includes three degrees of freedom of motion. Accordingly, the Euler angle variation data includes: the original posture N1 (u1, v1, w1) of the second joystick in space, and the posture N2 (u2, v2, w2) of the second joystick after it moves under the user's operation.

8. The semi-automatic control method according to claim 1, characterized in that, The first motion command is calculated based on the combination of the second input signals using a preset trajectory planning algorithm in the robot unit.

9. The semi-automatic control method according to claim 1, characterized in that, The remote control module is equipped with a rocker wheel, and when the user operates the rocker wheel to make it rotate, the rotation angle or number of rotations of the rocker wheel will be converted into the starting node information of the task according to the preset conversion rules.

10. A semi-automatic control system for a photography robot, characterized in that, The system includes: A first input module is configured to acquire a first input signal combination, wherein the first input signal combination includes one or more of the following signals: (i) Signal A associated with the remote control state of the remote control module; and the signal A includes: a switching signal input by the user indicating switching to the first functional state or the second functional state, or a default signal preset in the remote control module; (ii) Signal B, indicating the actual operating status of the remote control module and / or robot module; The function switching module is configured to switch or maintain the remote control module in a corresponding functional state according to the first combination of input signals; The second input module is configured to acquire a second input signal combination associated with the robot module when the remote control module is in a first functional state. The second input signal combination includes a first sub-combination input through the remote control module. The first functional state includes a first operation state. The first operation state is used by the user to manually mark the camera movement route before performing the actual shooting task. The first sub-combination includes one or more of the following signals: (1) First joystick signal, the first joystick signal includes: spatial motion trajectory data of the first joystick; (2) Second joystick signal, the second joystick signal includes: Euler angle change data of the second joystick; (3) Length signal, the length signal includes: starting node information of the task to be performed by the robot module; The first instruction conversion module is configured to convert the second input signal combination into a first motion instruction; The first instruction transmission module is configured to split the first motion instruction into a first instruction and a second instruction, and send the first instruction and the second instruction to the robot unit and the camera unit in the robot module, respectively. The second instruction includes one or more of the following parameters: aperture, sensitivity, and focal length.

Images