Multi-machine teleoperation display control method, device and equipment

Abstract

The embodiment of the application belongs to the technical field of remote operation, and relates to a multi-machine remote operation display control method. The multi-machine comprises a remote-controlled robot and a collaborative robot outside the remote-controlled robot. The method comprises the following steps: acquiring a working picture of the remote-controlled robot; the working picture comprises the collaborative robot and / or a state indication of the collaborative robot; and the working picture is sent to a display for display. The application further provides a remote operation display control device, a computer device and a storage medium. The technical scheme of the application can better assist an operator to complete a multi-machine assisted remote operation task in a multi-machine cooperation process.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of remote operation, in particular to a multi-machine remote operation display control method, device and equipment. BACKGROUND

[0002] With the development of science and technology, the application field of robots is more and more extensive. In some complex and dangerous environments, robots need to be more flexible and have higher human-like working ability. Therefore, remote operation robots emerge as the times require. Remote operation robots are installed with posture sensors, such as inertial sensors (Inertial Measurement Unit, IMU), on the operator. The operator performs target task operation in another real or virtual scene. The motion data of the operator in the operation process is captured by the IMU and sent to the controller. The controller generates corresponding motion control instructions according to the motion data, so as to control the slave robot, thereby achieving the purpose of remote operation.

[0003] However, at present, there is often a lack of a more effective cooperation control or auxiliary control method for the remote operation technology of multiple slave robots cooperating to complete the target task. SUMMARY

[0004] The purpose of the embodiments of the present application is to provide a multi-machine remote operation display control method, device and equipment, so as to better assist the operator to complete the multi-machine assisted remote operation task in the multi-machine cooperation process.

[0005] In a first aspect, the embodiments of the present application provide a multi-machine remote operation display control method, which adopts the technical scheme as follows:

[0006] A multi-machine remote operation display control method, the multi-machine including a remote-controlled robot and a cooperation robot outside the remote-controlled robot, the method comprising:

[0007] Obtaining a working picture of the remote-controlled robot; the working picture including a cooperation robot and / or a state indication of the cooperation robot;

[0008] Sending the working picture to a display for display.

[0009] Further, in an embodiment, the working picture further includes the remote-controlled robot; or

[0010] Before the step of sending the working picture to the display for display, the method further includes the following steps:

[0011] Obtaining state information of the remote-controlled robot;

[0012] Based on the state information, in combination with a coordinate conversion relationship between the image sensor and the remote-controlled robot, a state indication of the remote-controlled robot is generated in the work picture.

[0013] Further, in one embodiment, when the collaborative robot is in motion, before the work picture of the remote-controlled robot is acquired, the following steps are further included:

[0014] An initial work picture of the remote-controlled robot is acquired.

[0015] It is determined whether the collaborative robot exists in the initial work picture.

[0016] If the collaborative robot does not exist in the initial work picture, state information of the collaborative robot is acquired; based on the state information, a state indication of the collaborative robot is generated in the initial work picture to obtain the work picture including the state indication of the collaborative robot.

[0017] If the collaborative robot exists in the initial work picture, the initial work picture is taken as the work picture including the collaborative robot.

[0018] Further, in one embodiment, the determination of whether the collaborative robot exists in the initial work picture includes the following steps:

[0019] The pose of the collaborative robot is obtained.

[0020] The field of view range of the image sensor is acquired.

[0021] It is determined whether the pose of the collaborative robot is located outside the field of view range.

[0022] If it is outside the field of view range, it is considered that the collaborative robot does not exist in the initial work picture; or

[0023] It is determined whether the collaborative robot is recognized in the initial work picture.

[0024] If the collaborative robot is not recognized, it is considered that the collaborative robot does not exist in the initial work picture.

[0025] Further, in one embodiment, the acquisition of the state information of the collaborative robot includes the following steps:

[0026] Joint motion information of the collaborative robot is acquired; based on the joint motion information, the pose of the end of the collaborative robot is obtained; the pose of the end of the collaborative robot is taken as the state information of the collaborative robot; or

[0027] The state information of the collaborative robot collected and sent by the position sensor is acquired.

[0028] Further, in one embodiment, the generating the status indication of the collaborative robot in the work view based on the status information comprises the following steps:

[0029] Obtaining a current position of the collaborative robot in a current frame work view;

[0030] Obtaining a last frame position of the collaborative robot in a last frame work view;

[0031] Constructing an indication mark from the last position to the current position in the current frame work view; or

[0032] Obtaining a current position of the collaborative robot in a current frame work view;

[0033] Extracting a picture boundary closest to the current position in the current frame work view;

[0034] Labeling the picture boundary with an indication mark.

[0035] In a second aspect, embodiments of the present application provide a multi-machine teleoperation display control device, comprising:

[0036] An image acquisition module, configured to acquire a work view of a telecontrolled robot; the work view comprising a collaborative robot and / or a status indication of the collaborative robot;

[0037] An image display module, configured to send the work view to a display for display.

[0038] In a third aspect, embodiments of the present application provide a teleoperation system, comprising an image sensor, a telecontrolled robot, a collaborative robot, a display, and a controller;

[0039] The image sensor is configured to acquire an initial work view of the telecontrolled robot, and send the initial work view to the controller;

[0040] The controller is configured to acquire a work view; the work view comprising a collaborative robot and / or a status indication of the collaborative robot; send the work view to a display for display; wherein the work view is the initial work view or a work view obtained after processing the initial work view.

[0041] In a fourth aspect, embodiments of the present application provide a computer device, comprising a memory and a processor; the memory stores a computer program; the processor executes the computer program to implement the steps of the multi-machine teleoperation display control method according to any one of the above aspects.

[0042] In a fifth aspect, an embodiment of the present application provides a computer readable storage medium, and the computer readable storage medium stores a computer program. The computer program is executed by a processor to implement the steps of the multi-machine teleoperation display control method according to any one of the above aspects.

[0043] Compared with the prior art, the embodiments of the present application have the following beneficial effects:

[0044] The embodiments of the present application display the working condition of the collaborative robot in the working picture, and still generate the state indication about the working condition of the collaborative robot after the robot disappears from the working picture, which facilitates the operator to understand the working condition of the collaborative robot, so that the execution condition of the target task can be more intuitively judged, and the teleoperation task can be better completed. BRIEF DESCRIPTION OF DRAWINGS

[0045] In order to more clearly illustrate the schemes in the present application, the drawings needed in the description of the embodiments of the present application will be briefly introduced. Obviously, the drawings in the following description are some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0046] Figure 1 is an exemplary system architecture diagram in which the present application can be applied;

[0047] Figure 2 is a schematic diagram of an embodiment in which the collaborative robot moves from the field of view of the image sensor to outside the field of view in the present application;

[0048] Figure 3A is a schematic diagram of an embodiment in which the working picture includes the collaborative robot in the present application;

[0049] Figure 3B is a schematic diagram of an embodiment in which the working picture includes the state indication of the collaborative robot in the present application;

[0050] Figure 3C is a schematic diagram of another embodiment in which the working picture includes the state indication of the collaborative robot in the present application;

[0051] Figure 4 is a flowchart of an embodiment of the teleoperation display control method of the present application;

[0052] Figure 5 is a flowchart of another embodiment of the teleoperation display control method of the present application;

[0053] Figure 6 is a structural schematic diagram of an embodiment of the teleoperation display control device of the present application;

[0054] Figure 7 is a structural schematic diagram of one embodiment of the computer device of the present application. DETAILED DESCRIPTION

[0055] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs; the terminology used in the description herein is for describing particular embodiments only and is not intended to be limiting of the application; the description herein and the claims of the application and the above description of the drawings herein, the terms "comprising", "having" and "including" and any variations thereof are intended to cover without limitation; the description herein and the claims of the application or the above description of the drawings herein, the terms "first", "second" and the like are used to distinguish different objects, not to describe a particular order.

[0056] Reference herein to "an embodiment" means that a particular feature, structure, or characteristic described in connection with the embodiment can be included in at least one embodiment of the application. The appearances of the phrase that the phrase in the specification appear in various places in the specification are not necessarily all referring to the same embodiment, or are necessarily referring to different or alternative embodiments. It is explicitly and implicitly understood that the embodiments described herein can be combined.

[0057] In order to make the person skilled in the art better understand the scheme of the present application, the technical solutions in the embodiments of the present application will be described clearly and completely in conjunction with the drawings below.

[0058] As shown in Figure 1 , Fig. 1 is an exemplary system architecture diagram of the teleoperation system of the present application. Figure 1

[0059] The embodiment of the present application provides a teleoperation system 100, which comprises a remote-controlled robot 110, a collaborative robot 120, an image sensor 130, a display 140 and a controller 150.

[0060] The remote-controlled robot 110, the image sensor 130 and the display 140 are respectively in communication connection with the controller 150 in a wired or wireless manner.

[0061] It should be noted that the above-mentioned wireless connection mode can include but is not limited to 3G / 4G / 5G connection, WiFi connection, Bluetooth connection, WiMAX connection, Zigbee connection, UWB (ultra wideband) connection, and other now known or future developed wireless connection modes.

[0062] The image sensor 130 is configured to collect a working picture of the remote-controlled robot 110; the working picture comprises the collaborative robot and / or a state indication of the collaborative robot.​

[0063] It should be noted that, in addition to the work picture including the picture related to the working condition of the remote-controlled robot, the collaborative robot and / or the state indication of the collaborative robot (such as: when the motion state collaborative robot disappears from the work picture), the latter embodiments will be further described in detail.

[0064] In an optional embodiment, when the collaborative robot 120 is fixed, the angle of view of the image sensor 130 can be adjusted in advance so that the work picture collected by the image sensor 130 includes part of the collaborative robot 120. In another embodiment, when the collaborative robot 120 is in a motion state, in some cases, the collaborative robot 120 in motion can disappear from the work picture.

[0065] Specifically, the image sensor can be, but is not limited to, a camera, a video camera, a scanner, or other devices with related functions (mobile phones, computers, etc.), and the like. The work picture can be a two-dimensional image or a multi-dimensional image, and the like.

[0066] The image sensor 130 can be fixed to the remote-controlled robot as needed to follow the motion of the remote-controlled robot, or fixed to a certain preset fixed position outside the remote-controlled robot. For the convenience of understanding, the embodiments of the present application take the image sensor 130 fixed to a certain preset fixed position outside the remote-controlled robot 110 as an example for detailed description, and the angle of view of the image sensor 130 is adjusted in advance as needed.

[0067] The remote-controlled robot 110 is used to move based on the indication of the motion instruction generated by the controller, etc.

[0068] The remote-controlled robot 110 can be, but is not limited to, a humanoid robot, a manipulator, a surgical / medical robot, a service robot, and a driverless robot. For the robot, it can refer to the whole robot, or refer to a part of the robot based on remote operation control, such as: the upper body of a humanoid robot or the paw part of a robot. Taking the manipulator as an example, the posture of the robot described in the following embodiments can refer to the posture of all or part of the joints of the manipulator, such as: the posture of the end of the manipulator, wherein, in an optional embodiment, the end of the manipulator can refer to the center of the flange plate of the output end of the end joint of the manipulator.

[0069] The collaborative robot 120 is used to complete the target task in cooperation with the remote-controlled robot 110.

[0070] Specifically, the collaborative robot can be another remote-controlled robot; in addition, it can also be a non-remote-operated robot based on fixed program or preset model to generate motion instructions, and the like.

[0071] Display 140 is used to show the operator the working screen.

[0072] In an optional embodiment, the image sensor 130 captures the working scene of the remotely controlled robot and sends the working scene to the controller 150 (e.g., a server); the controller 150 sends the working scene directly or after performing some processing on the working scene to the display 140 for display.

[0073] Specifically, the display can be a screen or an AR / VR virtual display device.

[0074] In an optional embodiment, the operator wears an AR virtual display device to display a virtual 3D work screen to the operator.

[0075] In an optional embodiment, the teleoperation system 100 also includes an attitude sensor 160.

[0076] The attitude sensor 160 and the controller 150 communicate with each other via wired or wireless means.

[0077] The attitude sensor 160 is used to collect motion data of key parts of the operator.

[0078] Specifically, attitude sensors can be any existing or future attitude sensors capable of collecting motion-related data, such as IMUs, image sensors, etc. For ease of understanding, as... Figure 1 As shown, the embodiments of this application mainly use an IMU as an example for detailed description.

[0079] Among them, IMU is an inertial measurement unit, which is used to measure motion data related to the target object, including three-dimensional acceleration and three-dimensional rotation angle.

[0080] It should be noted that the aforementioned attitude sensor 160 can be directly fixed to a preset key part of the operator's device (such as...). Figure 1 (As shown). Alternatively, multiple attitude sensors can be pre-set on a wearable device (e.g., an exoskeleton or data glove), which is then worn on the operator's body, thereby positioning the attitude sensors at predetermined key locations on the operator's body (see attached figure).

[0081] The controller 150 is used to execute the teleoperation display control method described in the embodiments of this application, etc.

[0082] In an optional embodiment, the controller is configured to acquire a working screen; the working screen includes a collaborative robot and / or a status indicator of the collaborative robot; and send the working screen to a display for display; wherein the working screen is an initial working screen or a working screen obtained by processing an initial working screen.

[0083] It should be noted that the controller described in the embodiments of the present application can be a controller of the entire teleoperation system, a controller of the remote-controlled robot and / or the collaborative robot, a controller of the display, and the like. For the convenience of understanding, the controller is collectively referred to as a controller in the embodiments of the present application. Each controller can be integrated, or can be separately arranged in the respective robots, displays, and the like.

[0084] The teleoperation display control method provided by the embodiments of the present application can be applied to a personal computer (PC), an industrial personal computer (IPC), a mobile terminal, a server, a system including a terminal and a server, and is implemented through interaction of the terminal and the server, a programmable logic controller (PLC), a field-programmable gate array (FPGA), a digital signal processor (DSP), or a microcontroller unit (MCU) and the like. The controller generates program instructions according to a pre-fixed program in combination with data collected by an external IMU 110 and the like. The specific definition of the controller can be referred to the definition of the teleoperation display control method in the embodiments below.

[0085] Specifically, it can be applied to a computer device as shown in the figure. Figure 7 The computer device can be a terminal or a server. The computer device includes a processor, a memory, a communication interface, a display screen, and an input device connected through a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operating system and the computer program in the non-volatile storage medium to run. The communication interface of the computer device is used to communicate with external terminals in a wired or wireless manner. The wireless manner can be achieved through WIFI, operator network, NFC (near field communication), or other technologies. The computer program is executed by the processor to implement a pose recognition method for teleoperation. The display screen of the computer device can be a liquid crystal display screen or an electronic ink display screen. The input device of the computer device can be a touch layer overlaid on the display screen, or a security check, trackball, or touchpad arranged on the shell of the computer device. It can also be an external keyboard, touchpad, or mouse, and the like.

[0086] The server can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms. The terminal can be a smartphone, tablet, laptop, desktop computer, smart speaker, smartwatch, etc., but is not limited to these. The terminal and server can be directly or indirectly connected via wired or wireless communication, which is not limited herein.

[0087] It should be noted that the remote operation display control method provided in this application embodiment is generally executed by the controller 150, and correspondingly, the remote operation display control device is generally set in the controller 150.

[0088] like Figure 4 As shown, Figure 4 This is a flowchart illustrating one embodiment of the teleoperation display control method of this application.

[0089] Step 210: Obtain the working screen of the remotely controlled robot; the working screen includes the collaborative robot and / or the status indicators of the collaborative robot.

[0090] like Figure 3A As shown, Figure 3A This is a schematic diagram of one embodiment of the collaborative robot's working screen in this application. In an optional embodiment, when the collaborative robot 120 is fixed, the viewing angle of the image sensor 130 can be pre-adjusted. The controller retrieves the image portion 210 of the collaborative robot directly included in the initial working screen acquired and transmitted by the image sensor 130 from the memory or server according to a preset storage address. This initial working screen can then be directly used as the aforementioned working screen. The operator can directly understand the working status of the collaborative robot through the collaborative robot image portion 210.

[0091] like Figure 3B and Figure 3C As shown, Figure 3B This is a schematic diagram of an embodiment in which the working screen of this application includes the status indication of the collaborative robot; Figure 3C This is a schematic diagram of another embodiment of the working screen in this application that includes a status indication of the collaborative robot.

[0092] In another optional embodiment, when the collaborative robot 120 is in motion, in some cases, the collaborative robot 120 in motion can disappear from the work picture, and the controller obtains the work picture including the state indication of the collaborative robot from the storage or server according to a preset storage address. The work picture including the state indication of the collaborative robot is a work picture obtained by the controller through certain processing on the initial work picture collected and sent by the image sensor 130, which will be further described in the following embodiments.

[0093] The step 220 sends the work picture to the display for display.

[0094] In an optional embodiment, before the step 220, the following steps can also be included:

[0095] The step 230 converts the work picture into a work picture available for display by a virtual display device.

[0096] In an optional embodiment, the controller converts the work picture into a virtual three-dimensional work picture available for display by a virtual display device (such as VR or AR), so as to improve the real experience of the operator, thereby facilitating the operator to more accurately perform the target task based on the work picture.

[0097] The embodiments of the present application display the work condition of the collaborative robot in the work picture, and still generate the state indication of the work condition of the collaborative robot after the robot disappears from the work picture, so as to facilitate the operator to understand the work condition of the collaborative robot, thereby more intuitively judging the execution condition of the target task, and facilitating better completion of the teleoperation task. For example, the operator can make decisions and take actions more quickly, reduce errors caused by lack of information of the collaborative robot, and better understand the relative positions, motion trajectories and work states of the robots, thereby avoiding safety problems such as collision and erroneous operation between the robots.

[0098] In an optional embodiment, in addition to the collaborative robot, the work picture can also include the remote-controlled robot. For example, the image sensor 130 is fixed to a certain preset position outside the remote-controlled robot 110, so that the remote-controlled robot 110 and the collaborative robot 120 are both located in the field of view of the image sensor.

[0099] In another optional embodiment, before the step 220 sends the work picture to the display for display, the following steps can also be included:

[0100] The step 240 obtains the state information of the remote-controlled robot.

[0101] Specifically, the state information of the remotely controlled robot may include, but is not limited to: the posture information of the remotely controlled robot, the posture information of the joints of the remotely controlled robot, or the trajectory information of the remotely controlled robot.

[0102] Step 250 generates a status indication of the remotely controlled robot in the working screen based on the coordinate transformation relationship between the image sensor and the remotely controlled robot.

[0103] In an optional embodiment, the coordinate transformation relationship between the image sensor 130 and the remotely controlled robot 110 can be calibrated in advance. Based on the calibration result, the controller can convert the attitude and other state information of the remotely controlled robot 110 into the working screen to generate state indications of the remotely controlled robot and the collaborative robot in the working screen.

[0104] For example, taking a remotely controlled robot as a manipulator, the posture of the manipulator's end effector can be obtained based on the kinematic equations. Based on the calibration results, the posture of the manipulator's end effector can be transformed into the image sensor coordinate system. Then, based on the calibration parameters of the image sensor itself, the display coordinates of the manipulator's end effector in the image can be obtained.

[0105] This application embodiment also displays the remotely controlled robot or its status indicator on the working screen, which helps the operator to better understand the collaboration between the remotely controlled robot and the collaborating robot, thereby better completing multi-robot remote operation control.

[0106] like Figure 5 As shown, Figure 5 This is a flowchart illustrating another embodiment of the teleoperation display control method of this application.

[0107] In an optional embodiment, when the collaborative robot is in motion, it may disappear from the initial working image in certain situations. Therefore, before step 210 acquires the working image of the remotely controlled robot, the following steps may also be included:

[0108] Step 260: Obtain the initial working screen of the remotely controlled robot.

[0109] In an optional embodiment, the controller retrieves the initial working image captured and transmitted by the image sensor or the initial working image after some preprocessing from the memory or server according to a preset storage address.

[0110] Step 270: Determine whether a collaborative robot exists in the initial working screen.

[0111] In an optional embodiment, the controller can identify the collaborative robot in the work picture based on traditional image processing or artificial intelligence, etc. If the collaborative robot cannot be identified, it is considered that the collaborative robot leaves the work picture. Through this method, it can be directly judged whether the collaborative robot exists in the work picture.

[0112] In another optional embodiment, the controller can determine whether the collaborative robot exists in the work picture based on the field of view range of the image sensor, etc. The following embodiments will be described in further detail.

[0113] Step 280, if the initial work picture does not contain the collaborative robot, the state information of the collaborative robot is acquired.

[0114] Specifically, the state information can be, but is not limited to, the posture information of the robot, the posture information of the joint of the robot, and / or the trajectory information of the robot.

[0115] For example, the state information can be the posture information or the motion trajectory information of the humanoid robot.

[0116] In an optional embodiment, the controller can acquire the state information of the collaborative robot collected and sent by the position sensor. For example, the position information (i.e. the posture information of the robot) of the robot can be acquired based on the position sensor (such as a positioning device) installed on the robot base.

[0117] In another optional embodiment, taking the collaborative robot as a mechanical hand, the state information of the robot can also refer to the posture information of the end of the robot. Specifically, the controller can acquire the motion amount information of the joint of the mechanical hand based on the encoder, and calculate the posture of the end of the robot based on the kinematics equation of the robot.

[0118] Step 290, based on the state information, the state indication of the robot is generated in the initial work picture, so as to obtain the work picture containing the state indication of the collaborative robot.

[0119] Specifically, the state indication can be, but is not limited to, a robot motion pointing arrow or a mark corresponding to the image boundary, which will be described in further detail in the following embodiments.

[0120] In an optional embodiment, before step 210, the following method steps can also be included:

[0121] Step 300, if the initial work picture contains the collaborative robot, the initial work picture is taken as the work picture.

[0122] This application embodiment displays the working status of the collaborative robot on the working screen, and continues to generate status indicators related to the working status of the collaborative robot even after the robot disappears from the working screen. This allows the operator to understand the working status of the collaborative robot, thereby enabling a more intuitive judgment on the execution of the target task and facilitating the better completion of remote operation tasks.

[0123] In another optional embodiment, step 270, determining whether a collaborative robot exists in the initial work screen, may specifically include the following method steps:

[0124] Step 271: Obtain the posture information of the collaborative robot.

[0125] The method for generating posture information for collaborative robots can be found in the above embodiments, and will not be repeated here.

[0126] Step 272: Obtain the field of view of the image sensor.

[0127] In an optional embodiment, the controller may predetermine the coordinates of the image sensor's field of view boundary based on the image sensor's position coordinates and the image sensor's own calibration parameters.

[0128] Step 273 determines whether the collaborative robot's posture is outside the field of vision.

[0129] In an optional embodiment, the controller can convert the collaborative robot's posture in the robot coordinate system to the image sensor coordinate system, and then determine whether the robot's posture in the image sensor coordinate system is outside the preset field of view of the image sensor.

[0130] If the robot leaves the field of view in step 274, it is considered that there is no collaborative robot in the initial working screen.

[0131] like Figure 2 As shown, Figure 2 This is a schematic diagram of an embodiment in which the collaborative robot moves from within the field of view of the image sensor to outside the field of view. For example, if the viewing angle of the image sensor 130 is to the right of the corresponding object sensor 130, then taking the lens dividing line 0 through the sensor as the boundary, when the collaborative robot moves to a certain posture and, after being transformed into the image sensor coordinate system, is located to the left of the dividing line 0, it can be considered that the collaborative robot 12 is outside the field of view of the image sensor 130.

[0132] The embodiment of the present application can more accurately determine whether the collaborative robot exists in the work picture in the case that the collaborative robot is blocked, by determining whether the collaborative robot leaves the field of view of the image sensor 130, for example, the case that the robot cannot be displayed in the work picture due to being blocked by an object, and the applicability is wider than the above-mentioned method of identifying the robot based on the image.

[0133] In an optional embodiment, taking the collaborative robot 120 as a manipulator, the step 280 of acquiring the state information of the collaborative robot can include the following method steps:

[0134] The step 281 acquires joint motion information of the collaborative robot.

[0135] In an optional embodiment, the controller can acquire joint motion information collected and sent by a motion encoder located at a joint of the manipulator.

[0136] The step 282 obtains the pose of the end of the collaborative robot based on the joint motion information.

[0137] In an optional embodiment, the controller can obtain the pose of the end of the manipulator based on the joint motion information and in combination with a robot kinematics equation.

[0138] The step 283 takes the pose of the end of the collaborative robot as the state information of the collaborative robot.

[0139] The embodiment of the present application can obtain the pose information of the end of the robot based on the above-mentioned method after the end of the robot disappears from the current work picture, and further generate a state indication of the end of the robot based on the pose information.

[0140] In an optional embodiment, the step 290 of generating a state indication of the collaborative robot in the work picture based on the state information can specifically include the following steps:

[0141] The step 291 acquires a current position of the collaborative robot in the current frame of the work picture.

[0142] The step 293 acquires a previous position of the collaborative robot in the previous frame of the work picture.

[0143] The step 295 constructs an indication mark from the previous position to the current position in the current frame of the work picture.

[0144] Specifically, the above-mentioned indication mark can be, but is not limited to, an arrow indication mark, a connection line indication mark.

[0145] For example, based on the method described in the above embodiment, an arrow indication mark as shown in Figure 2 is generated.

[0146] In the embodiments of the present application, when the collaborative robot disappears from the work picture, an indication mark from the last position to the current position is constructed in the current frame work picture, so that even if the collaborative robot disappears from the work picture, the status indication about the working condition of the collaborative robot can still be obtained, and the operator can conveniently understand the working condition of the collaborative robot.

[0147] In another optional embodiment, the step 290 of generating the status indication of the collaborative robot in the work picture based on the status information can specifically include the following steps:

[0148] The step 292 acquires the current position of the collaborative robot in the current frame work picture.

[0149] The step 294 extracts the picture boundary closest to the current position in the current frame work picture.

[0150] In an optional embodiment, a perpendicular line can be drawn from the current position to the work picture, and the intersection of the perpendicular line and the work picture is the picture boundary closest to the current position as described in the above embodiment.

[0151] The step 296 marks the indication mark at the picture boundary.

[0152] Specifically, the indication mark can be, but is not limited to, an arrow mark of a preset direction displayed at the work picture boundary, a boundary point selection mark, or a boundary box selection mark, and the like.

[0153] The embodiments of the present application can extract the picture boundary closest to the current position in the current frame work picture when the collaborative robot disappears from the work picture, mark the indication mark at the picture boundary, so that even if the collaborative robot disappears from the work picture, the status indication about the working condition of the collaborative robot can still be obtained, and the operator can conveniently understand the working condition of the collaborative robot.

[0154] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be completed by a computer program instructing related hardware, and the computer program can be stored in a computer readable storage medium. When the program is executed, the processes of the above-mentioned embodiments can be included. The storage medium can be a non-volatile storage medium such as a magnetic disc, an optical disc, a read-only memory (ROM), or a random access memory (RAM).

[0155] It should be understood that although each step in the flowchart of the accompanying drawings is shown in sequence according to the direction of the arrow, these steps are not necessarily executed in sequence according to the direction of the arrow. Unless otherwise explicitly stated herein, the execution of these steps is not strictly limited in sequence, and they can be executed in other sequences. Moreover, at least part of the steps in the flowchart of the accompanying drawings can include multiple sub-steps or multiple stages, which are not necessarily executed at the same time, but can be executed at different times, and the execution sequence is not necessarily sequential, but can be alternately executed with at least part of other steps or sub-steps or stages of other steps.

[0156] Further referring to Figure 6 , as an implementation of the method shown in the above Figure 4 , the present application provides an embodiment of a teleoperation display control device, which corresponds to the method embodiment shown in Figure 4 , and the device can be applied to various controllers.

[0157] As shown in Figure 6 , the teleoperation display control device 400 of the embodiment includes an image acquisition module 410 and an image sending module 420. Wherein:

[0158] The image acquisition module 410 is configured to acquire a working picture of the teleoperated robot; the working picture includes a collaborative robot and / or a state indication of the collaborative robot.

[0159] The image sending module 420 is configured to send the working picture to a display for display.

[0160] The embodiment of the present application displays the working condition of the collaborative robot in the working picture, and still generates the state indication about the working condition of the collaborative robot after the robot disappears from the working picture, which facilitates the operator to understand the working condition of the collaborative robot, so that the execution condition of the target task can be more intuitively judged, and the teleoperation task can be better completed.

[0161] In one embodiment, the working picture can further include a teleoperated robot and / or a state indication of the teleoperated robot.

[0162] In an optional embodiment, the teleoperation display control device 400 can further include:

[0163] A state acquisition module configured to acquire state information of the teleoperated robot;

[0164] An indication generation module configured to generate, based on the state information, a state indication of the teleoperated robot in the working picture in combination with a coordinate conversion relationship between the image sensor and the teleoperated robot.

[0165] The embodiment of the present application helps the operator to better understand the cooperation of the remote-controlled robot and the collaborative robot, and thus better completes the multi-robot teleoperation control, by further displaying the remote-controlled robot or the state indication of the remote-controlled robot in the working picture.

[0166] In an optional embodiment, the teleoperation display control device 400 can further include:

[0167] an initial acquisition module, configured to acquire an initial working picture of the remote-controlled robot;

[0168] a collaboration judgment module, configured to judge whether the initial working picture includes the collaborative robot;

[0169] an indication determination module, configured to, if the initial working picture does not include the collaborative robot, acquire state information of the collaborative robot; and based on the state information, generate a state indication of the collaborative robot in the initial working picture to obtain a working picture including the state indication of the collaborative robot;

[0170] a picture determination module, configured to, if the initial working picture includes the collaborative robot, take the initial working picture as a working picture including the collaborative robot.

[0171] The embodiment of the present application helps the operator to better understand the cooperation of the remote-controlled robot and the collaborative robot, and thus better completes the multi-robot teleoperation control, by further displaying the remote-controlled robot or the state indication of the remote-controlled robot in the working picture.

[0172] In an optional embodiment, the collaboration judgment module can specifically include:

[0173] a collaboration judgment sub-module, configured to judge whether the initial working picture identifies the collaborative robot;

[0174] an image determination sub-module, configured to, if the collaborative robot is not identified, regard the initial working picture as not including the collaborative robot.

[0175] The embodiment of the present application can intuitively judge whether the collaborative robot leaves the working picture.

[0176] In another optional embodiment, the collaboration judgment module can specifically include:

[0177] a pose obtaining sub-module, configured to obtain the pose of the collaborative robot;

[0178] a field of view acquisition sub-module, configured to acquire the field of view range of the image sensor;

[0179] The posture judgment submodule is configured to judge whether the posture of the collaborative robot is outside the field of view range.

[0180] The image determination submodule is configured to determine that the collaborative robot does not exist in the initial work image if the collaborative robot is outside the field of view range.

[0181] The embodiment of the present application can more accurately determine whether the collaborative robot leaves the work image in the case of robot occlusion, and has wider applicability.

[0182] In an optional embodiment, the indication determination module can specifically include:

[0183] The information acquisition submodule is configured to acquire joint motion information of the collaborative robot.

[0184] The posture acquisition submodule is configured to acquire the posture of the end of the collaborative robot based on the joint motion information.

[0185] The state determination submodule is configured to take the posture of the end of the collaborative robot as the state information of the collaborative robot.

[0186] The embodiment of the present application can acquire the posture information of the end of the robot based on the above method after the end of the robot disappears from the current work image, and further generate the state indication of the end of the robot based on the posture information.

[0187] In another optional embodiment, the indication determination module can specifically include:

[0188] The position acquisition submodule is configured to acquire the position information of the collaborative robot collected and sent by the position sensor.

[0189] The embodiment of the present application can directly acquire the position information of the collaborative robot based on the position sensor.

[0190] In an optional embodiment, the indication determination module can further include:

[0191] The current acquisition submodule is configured to acquire the current position of the collaborative robot in the current frame work image.

[0192] The last acquisition submodule is configured to acquire the last frame position of the collaborative robot in the last frame work image.

[0193] The marker construction submodule is configured to construct an indication marker from the last position to the current position in the current frame work image.

[0194] In the embodiment of the present application, when the collaborative robot disappears from the work picture, an indication mark from the last position to the current position is constructed in the current frame work picture, so that even if the collaborative robot disappears from the work picture, the state indication about the working condition of the collaborative robot can still be obtained, and the operator can conveniently understand the working condition of the collaborative robot.

[0195] In another optional embodiment, the indication determination module specifically can further include:

[0196] The current acquisition submodule is configured to acquire a current position of the collaborative robot in the current frame work picture.

[0197] The boundary extraction submodule is configured to extract a picture boundary closest to the current position in the current frame work picture.

[0198] The indication marking submodule is configured to mark the indication mark on the picture boundary.

[0199] In the embodiment of the present application, when the collaborative robot disappears from the work picture, the controller extracts the picture boundary closest to the current position in the current frame work picture, and marks the indication mark on the picture boundary, so that even if the collaborative robot disappears from the work picture, the state indication about the working condition of the collaborative robot can still be obtained, and the operator can conveniently understand the working condition of the collaborative robot.

[0200] To solve the above technical problems, the embodiment of the present application further provides a computer device. For details, please refer to Figure 7 , Figure 7 The basic structure block diagram of the computer device in the embodiment is shown in the figure.

[0201] The computer device 6 includes a memory 61, a processor 62 and a network interface 63 which are connected to each other through a system bus. It should be pointed out that only the computer device 6 with components 61-63 is shown in the figure, but it should be understood that it is not required to implement all the shown components, and more or fewer components can be alternatively implemented. Among them, those skilled in the art can understand that the computer device here is a device capable of automatically performing numerical calculation and / or information processing according to pre-set or stored instructions, and its hardware includes but is not limited to microprocessor, application specific integrated circuit (ASIC), field programmable gate array (FPGA), digital signal processor (DSP), embedded device, etc.

[0202] The computer device can be a desktop computer, a notebook computer, a palm computer, a cloud server, or the like. The computer device can interact with a user through a keyboard, a mouse, a remote controller, a touchpad, a voice control device, or the like.

[0203] The memory 61 can include at least one type of readable storage medium, such as a flash memory, a hard disk, a multimedia card, a card-type memory (e.g., an SD or DX memory, or the like), a random access memory (RAM), a static random access memory (SRAM), a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), a programmable read-only memory (PROM), a magnetic memory, a magnetic disk, an optical disk, or the like. In some embodiments, the memory 61 can be an internal storage unit of the computer device 6, such as a hard disk or a memory of the computer device 6. In other embodiments, the memory 61 can also be an external storage device of the computer device 6, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, or the like. Of course, the memory 61 can include both an internal storage unit and an external storage device of the computer device 6. In this embodiment, the memory 61 is generally used to store an operating system and various application software installed in the computer device 6, such as program codes of the teleoperation display control method, or the like. In addition, the memory 61 can also be used to temporarily store various data that have been output or will be output.

[0204] The processor 62 can be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chip in some embodiments. The processor 62 is generally used to control the overall operation of the computer device 6. In this embodiment, the processor 62 is used to run program codes or process data stored in the memory 61, such as program codes of the teleoperation display control method.

[0205] The network interface 63 can include a wireless network interface or a wired network interface, and is generally used to establish a communication connection between the computer device 6 and other electronic devices.

[0206] The present application also provides another embodiment, i.e., a computer readable storage medium storing a teleoperation display control program. The teleoperation display control program can be executed by at least one processor to enable the at least one processor to perform the steps of the teleoperation display control method as described above.

[0207] Through the description of the above embodiments, those skilled in the art can clearly understand that the above-mentioned example methods can be realized by means of software and a necessary general hardware platform, and of course, they can also be realized by hardware, but in many cases, the former is a better embodiment. Based on such understanding, the technical solutions of the present application can be embodied in the form of a software product in essence or in the form of a part of the prior art that contributes to the present application. The computer software product is stored in a storage medium (such as a ROM / RAM, a magnetic disk, or an optical disk) and includes a plurality of instructions for causing an end device (which can be a mobile phone, a computer, a server, an air conditioner, or a network device) to execute the methods described in the various embodiments of the present application.

[0208] Obviously, the above-described embodiments are only some of the embodiments of the present application, not all the embodiments, and the drawings show the preferred embodiments of the present application, but do not limit the patent scope of the present application. The present application can be implemented in many different forms, and conversely, the purpose of providing these embodiments is to make the disclosure of the present application more thorough and comprehensive. Although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions recorded in the foregoing specific embodiments or equivalently replace some of the technical features. Any equivalent structure made by using the content of the specification and drawings, directly or indirectly applied to other related technical fields, is also within the scope of the patent protection of the present application.

Claims

1. A multi-machine teleoperation display control method, the multi-machine including a teleoperated robot and a collaborative robot outside the teleoperated robot, characterized by, The method comprises: acquiring a working picture of the remote-controlled robot; the working picture comprising a collaborative robot and / or a state indication of the collaborative robot; sending the working picture to a display for display; before the acquiring the working picture of the remote-controlled robot, further comprising the following steps: acquiring an initial working picture of the remote-controlled robot; determining whether the initial working picture comprises the collaborative robot; if the initial working picture does not comprise the collaborative robot, acquiring state information of the collaborative robot; based on the state information, generating a state indication of the collaborative robot in the initial working picture to obtain the working picture comprising the state indication of the collaborative robot; if the initial working picture comprises the collaborative robot, taking the initial working picture as the working picture comprising the collaborative robot.

2. The multi-robot teleoperation display control method according to claim 1, characterized by, the working picture further comprising the remote-controlled robot; or before the sending the working picture to the display for display, further comprising the following steps: acquiring state information of the remote-controlled robot; based on the state information, generating a state indication of the remote-controlled robot in the working picture in combination with a coordinate conversion relationship between the image sensor and the remote-controlled robot.

3. The multi-robot teleoperation display control method according to claim 1, characterized by, the determining whether the initial working picture comprises the collaborative robot comprises the following steps: acquiring a pose of the collaborative robot; acquiring a field of view range of the image sensor; determining whether the pose of the collaborative robot is outside the field of view range; if the pose of the collaborative robot is outside the field of view range, regarding the initial working picture as not comprising the collaborative robot; or determining whether the initial working picture identifies the collaborative robot; if the initial working picture does not identify the collaborative robot, regarding the initial working picture as not comprising the collaborative robot.

4. The multi-robot teleoperation display control method according to claim 1, characterized by, the acquiring the state information of the collaborative robot comprises the following steps: acquiring joint motion information of the collaborative robot; based on the joint motion information, acquiring a pose of an end of the collaborative robot; taking the pose of the end of the collaborative robot as the state information of the collaborative robot; or acquiring the state information of the collaborative robot collected and sent by a position sensor.

5. The multi-robot teleoperation display control method according to claim 1, characterized by, the generating the state indication of the collaborative robot in the working picture based on the state information comprises the following steps: acquiring a current position of the collaborative robot in a current frame working picture; acquiring a last frame position of the collaborative robot in a last frame working picture; constructing an indication mark from the last frame position to the current position in the current frame working picture; or acquiring a current position of the collaborative robot in a current frame working picture; extracting a picture boundary closest to the current position in the current frame working picture; labeling the picture boundary with an indication mark.

6. A multi-robot teleoperation display control device, characterized by comprising: The multiple machines comprise the remote-controlled robot and the collaborative robot, and comprise: an initial acquisition module, configured to acquire an initial working picture of the remote-controlled robot; a collaboration determination module, configured to determine whether the initial working picture comprises the collaborative robot; The indication determination module is configured to: if the collaborative robot does not exist in the initial work picture, acquire state information of the collaborative robot; and based on the state information, generate a state indication of the collaborative robot in the initial work picture to obtain a work picture including the state indication of the collaborative robot. The picture determination module is configured to: if the collaborative robot exists in the initial work picture, use the initial work picture as the work picture including the collaborative robot. The image acquisition module is configured to acquire a work picture of the remote-controlled robot; the work picture includes the collaborative robot and / or a state indication of the collaborative robot. The image display module is configured to send the work picture to a display for display. The collaboration determination module is configured to determine whether the collaborative robot exists in the initial work picture.

7. A teleoperation system, characterized by The image sensor, the remote-controlled robot, the collaborative robot, the display, and the controller are included. The image sensor is configured to acquire an initial work picture of the remote-controlled robot and send the initial work picture to the controller. The controller is configured to acquire a work picture; the work picture includes the collaborative robot and / or a state indication of the collaborative robot; send the work picture to a display for display; and when the collaborative robot is in motion, before acquiring the work picture of the remote-controlled robot, further include acquiring an initial work picture of the remote-controlled robot. Determine whether the collaborative robot exists in the initial work picture. If the collaborative robot does not exist in the initial work picture, acquire state information of the collaborative robot; and based on the state information, generate a state indication of the collaborative robot in the initial work picture to obtain a work picture including the state indication of the collaborative robot. If the collaborative robot exists in the initial work picture, use the initial work picture as the work picture including the collaborative robot; wherein the work picture is the initial work picture or a work picture obtained after processing the initial work picture.

8. A computer device, including a memory and a processor, the memory stores a computer program, and the processor implements the steps of the multi-machine teleoperation display control method according to any one of claims 1 to 5 when executing the computer program. The computer readable storage medium stores a computer program, and the computer program implements the steps of the multi-machine teleoperation display control method according to any one of claims 1 to 5 when executed by a processor.

9. A computer-readable storage medium, characterized in that, ​

Images