Robotic remote guidance method, device and system

Abstract

The application is suitable for the field of medical imaging technology, and provides a robot remote guidance method, device and system. The method comprises the following steps: a first electronic device of an operation end acquires a left-eye image group sequence and a right-eye image group sequence collected by a binocular camera, determines a target disparity map of each left-eye image group and right-eye image group, takes an integer part and a decimal part of a pixel value of each pixel in the target disparity map as a first channel value and a second channel value respectively, takes a preset placeholder as a third pixel value of the pixel, obtains a three-channel disparity map, and compresses the three-channel disparity map and a reference image into a target compressed file, so as to send the target compressed file and parameter information of the binocular camera to a head-mounted device of a guidance end; the head-mounted device restores target three-dimensional coordinates and color information of each spatial point from the target compressed file according to the parameter information, and renders and displays all spatial points according to the target three-dimensional coordinates and the color information, so that the data transmission efficiency and data quality are improved under the premise of ensuring the accuracy of remote surgery guidance.

Description

Technical Field

[0001] This application belongs to the field of medical imaging technology, and in particular relates to a method, device and system for remote guidance of a robot. Background Technology

[0002] Currently, with the widespread application of robot-assisted surgery technology, the demand for teaching and remote guidance of surgical robots is becoming increasingly urgent. Traditional remote guidance methods for surgical robots typically use single-format information such as voice, text, or two-dimensional (2D) video for remote interaction. This not only prevents instructors from fully immersing themselves in the three-dimensional surgical scene but also results in poor interactivity, thereby reducing the efficiency and accuracy of remote guidance for surgical robots and hindering the improvement of surgical safety.

[0003] Based on this, related technologies provide a method for remote guidance of surgical robots using augmented reality (AR) / virtual reality (VR) devices. This method configures the guidance device as an AR / VR device and the operating device to send 3D surgical scene data to the guidance device, enabling the guidance device to reconstruct a corresponding 3D virtual surgical scene, allowing the instructor to immerse themselves in the 3D surgical scene. Furthermore, this method improves the accuracy of remote surgical robot guidance by overlaying guidance content as virtual images onto the 3D surgical scene. However, because 3D surgical scene data typically contains massive amounts of irregular point cloud data, and point cloud data has low encoding and decoding efficiency, this method suffers from low data transmission efficiency and high data latency, making it difficult to meet the high real-time and high-quality requirements of remote interaction. Summary of the Invention

[0004] In view of this, embodiments of this application provide a method, device, and system for remote guidance of a robot to solve the technical problem that existing remote guidance methods for surgical robots based on AR / VR devices cannot meet the high real-time and high-quality requirements of remote interaction.

[0005] In a first aspect, embodiments of this application provide a robot remote guidance method, applied to a robot remote guidance system. The robot remote guidance system includes a first electronic device disposed at an operating end and a head-mounted device disposed at a guidance end; the first electronic device is communicatively connected to the robot at the operating end and the head-mounted device; the robot's robotic arm end is equipped with a binocular camera; the method includes:

[0006] The first electronic device acquires the left and right eye image group sequences captured by the binocular camera, and for each group of original left eye images and original right eye images in the left and right eye image group sequences, determines a target disparity map based on the left or right camera of the binocular camera.

[0007] For each pixel in each target disparity map, the first electronic device uses the integer part and the fractional part of the pixel value as the first channel value and the second channel value of the pixel, respectively, and uses a preset placeholder as the third channel value of the pixel, to obtain a three-channel disparity map corresponding to each target disparity map.

[0008] The first electronic device uses a preset image compression algorithm to compress the three-channel disparity map and the reference image to obtain a target compressed file, and sends the target compressed file and the parameter information of the binocular camera to the head-mounted device; the reference image is either a left-eye image or a right-eye image;

[0009] After parsing the three-channel disparity map and the reference image from the received target compressed file, the head-mounted device determines the target three-dimensional coordinates of the spatial point corresponding to each pixel in the three-channel disparity map in the head-mounted device's head-mounted display coordinate system according to the parameter information, and determines the pixel value of the corresponding pixel in the reference image as the color information of the spatial point. Based on the target three-dimensional coordinates and color information of each spatial point, the device renders and displays all the spatial points.

[0010] In one alternative implementation of the first aspect, determining a target disparity map based on the left or right camera of the binocular camera includes:

[0011] Based on the parameter information of the binocular camera, the original left-eye image and the original right-eye image are corrected to obtain a row-aligned target left-eye image and target right-eye image;

[0012] Based on the target's left-eye image and the target's right-eye image, a target disparity map is calculated, using either the target's left-eye image or the target's right-eye image as the reference image; the reference image corresponds to a reference camera.

[0013] In one optional implementation of the first aspect, determining the target three-dimensional coordinates of the spatial point corresponding to each pixel in the three-channel disparity map in the head-mounted display coordinate system of the head-mounted device, based on the parameter information, includes:

[0014] Construct a reprojection matrix based on the parameter information;

[0015] For each pixel in the three-channel disparity map, the sum of the first channel value and the second channel value of the pixel is determined as the disparity value of the pixel. Based on the image coordinates of the pixel, the disparity value, and the reprojection matrix, the original three-dimensional coordinates of the spatial point corresponding to the pixel in the camera coordinate system are calculated. The original three-dimensional coordinates are then transformed using a target transformation matrix from the camera coordinate system to the head-mounted display coordinate system to obtain the target three-dimensional coordinates of the spatial point.

[0016] In one optional implementation of the first aspect, the parameter information includes the principal point coordinates of the reference camera of the binocular camera, the focal length of the reference camera, and the baseline of the binocular camera; the reference camera is either the left or right camera of the binocular camera; correspondingly, based on the image coordinates of the pixel, the disparity value, and the reprojection matrix, the original three-dimensional coordinates of the spatial point corresponding to the pixel in the camera coordinate system of the binocular camera are calculated, including:

[0017] Based on the image coordinates of the pixel, the disparity value, and the reprojection matrix, the homogeneous coordinate vector corresponding to the pixel is constructed using the following formula:

[0018] [ X , Y , Z , W ] T = Q ×[ u , v , d ( u , v ),1] T ;

[0019] in,[ X , Y , Z , W ] T Let be the homogeneous coordinate vector corresponding to the pixel. Q Let be the reprojection matrix, ( u , v ) represents the image coordinates of the pixel. d(u , v The disparity value is ; the reprojection matrix is:

[0020] ;

[0021] in,( c x , c y ) represents the principal point coordinates of the reference camera. fThe focal length of the reference camera is... b The baseline of the binocular camera;

[0022] Based on the homogeneous coordinate vector, the original three-dimensional coordinates of the spatial point corresponding to the pixel in the camera coordinate system of the stereo camera are calculated using the following formula:

[0023] ( p x , p y , p z )=( X / W , Y / W , Z / W );

[0024] in,( p x , p y , p z ) represents the original three-dimensional coordinates of the spatial point corresponding to the pixel.

[0025] In one optional implementation of the first aspect, the original three-dimensional coordinates are transformed using a target transformation matrix from the camera coordinate system to the head-mounted display coordinate system to obtain the target three-dimensional coordinates of the spatial point, including:

[0026] Obtain the spatial point pose control matrix, the camera rendering matrix of the binocular camera, and the current pose matrix of the head-mounted device;

[0027] Based on the camera rendering matrix, the spatial point pose control matrix, and the current pose matrix, the target transformation matrix from the camera coordinate system to the head-mounted display coordinate system is constructed using the following formula:

[0028] T E H = T W H × T P W × T E P ;

[0029] in, T E H Let be the target transformation matrix. T WH The current pose matrix, T P W The spatial point pose control matrix, T E P The camera rendering matrix;

[0030] Based on the original three-dimensional coordinates of the spatial point, the target transformation matrix, and the spatial point scale parameter, the target three-dimensional coordinates of the spatial point are calculated using the following formula:

[0031] P fn = a × T E H × P on ;

[0032] in, P fn The target's three-dimensional coordinates are the spatial points. a The spatial point scale parameter, P on The original three-dimensional coordinates of the spatial point.

[0033] In one alternative implementation of the first aspect, it also includes:

[0034] The head-mounted device receives and recognizes the target virtual gesture input by the user;

[0035] The head-mounted device performs the corresponding target operation based on the target virtual gesture.

[0036] In one optional implementation of the first aspect, the head-mounted device performs a corresponding target operation based on the target virtual gesture, including:

[0037] The head-mounted device controls the target virtual object to move in the three-dimensional space under the head-mounted display coordinate system according to the target virtual gesture, so as to draw the surgical guidance trajectory or mark the surgical lesion area;

[0038] Correspondingly, the method further includes:

[0039] The head-mounted device generates surgical guidance information based on the motion information of the target virtual object, and sends the surgical guidance information to the first electronic device.

[0040] In a second aspect, embodiments of this application provide an electronic device, including a memory and a computer program stored in the memory and executable on a processor. When the processor executes the computer program, it implements the steps corresponding to the first electronic device in the method described in any optional implementation of the first aspect described above.

[0041] Thirdly, embodiments of this application provide a head-mounted device, including a memory and a computer program stored in the memory and executable on a processor. When the processor executes the computer program, it implements the steps corresponding to the head-mounted device in any optional implementation of the first aspect described above.

[0042] Fourthly, embodiments of this application provide a robot remote guidance system, including a robot and a first electronic device disposed at the operating end, and a head-mounted device disposed at the guidance end; the first electronic device is communicatively connected to the robot and the head-mounted device; the robot's robotic arm end is provided with a binocular camera;

[0043] The first electronic device is used to perform the steps corresponding to the first electronic device in the method of any optional implementation of the first aspect described above;

[0044] The head-mounted device is used to perform the steps corresponding to the head-mounted device in any optional implementation of the first aspect described above.

[0045] Implementing the robot remote guidance method, device, and system provided in the embodiments of this application has the following beneficial effects:

[0046] The robot remote guidance method provided in this application calculates the target disparity map for each left and right eye image group acquired by a binocular camera. It then splits the pixel value of each pixel in the target disparity map into an integer part and a fractional part, using the integer part, fractional part, and a preset placeholder as the first channel value, second channel value, and third channel value of that pixel, respectively, thereby constructing a corresponding three-channel disparity map. Since the three-channel disparity map can be efficiently compressed using image compression technology compared to the traditional floating-point form of the disparity map, a preset image compression algorithm is used to compress the three-channel disparity map and the reference image into a target compressed file. This target compressed file, along with the binocular camera's parameter information, is then sent to the head-mounted device at the guidance end. This not only improves data transmission efficiency and reduces data latency but also enables the head-mounted device to obtain the depth information of the operation end scene represented by the disparity map and reconstruct the operation end scene accordingly, thereby improving the accuracy of remote guidance. Therefore, the robot remote guidance method provided in this application can simultaneously meet the high real-time and high-quality requirements of remote interaction. Attached Figure Description

[0047] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0048] Figure 1 This is a schematic diagram of the structure of a robot remote guidance system provided in an embodiment of this application;

[0049] Figure 2 A schematic flowchart illustrating a remote robot guidance method provided in an embodiment of this application;

[0050] Figure 3 A schematic diagram of a left and right eye image group sequence provided in an embodiment of this application;

[0051] Figure 4 A schematic diagram illustrating the implementation process of S21 in a robot remote guidance method provided in this application embodiment;

[0052] Figure 5 A schematic diagram of the structure of a reference image and a three-channel disparity map provided in an embodiment of this application;

[0053] Figure 6 A schematic diagram illustrating the implementation process of S24 in a robot remote guidance method provided in this application embodiment;

[0054] Figure 7 A schematic diagram of coordinate transformations involved in the implementation of a robot remote guidance method provided in this application embodiment;

[0055] Figure 8 A schematic flowchart illustrating a remote robot guidance method according to another embodiment of this application;

[0056] Figure 9 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application;

[0057] Figure 10 This is a schematic diagram of the structure of a head-mounted device provided in an embodiment of this application. Detailed Implementation

[0058] The following embodiments are only used to illustrate the technical solutions of this application more clearly, and are therefore only examples and should not be used to limit the scope of protection of this application.

[0059] In the description of the embodiments of this application, the technical terms "comprising," "including," "having," and any variations thereof all mean "including but not limited to," unless otherwise specifically emphasized. In the description of the embodiments of this application, unless otherwise stated, the technical term "multiple" refers to two or more, and the technical terms "at least one" or "one or more" refer to one, two, or more than two. The technical terms "first," "second," etc., are only used to distinguish different objects and should not be construed as indicating or implying relative importance or implicitly specifying the number, specific order, or primary / secondary relationship of the indicated technical features. The technical term "and / or" is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. Additionally, the character " / " in this document generally indicates that the preceding and following related objects have an "or" relationship.

[0060] This application first provides a robot remote guidance system. Figure 1 This is a schematic diagram of the structure of a robot remote guidance system provided in an embodiment of this application.

[0061] like Figure 1 As shown, a robot remote guidance system may include a first electronic device 10 located at the operating end and a head-mounted device 20 located at the guidance end. The operating end may refer to the operating space, such as a surgical space. The guidance end may refer to other spaces outside the operating space, such as the instructor's office space.

[0062] The first electronic device 10 can communicate with the robot 11 at the operating end and the head-mounted device 20. The robot 11 can also communicate with the head-mounted device 20. This communication connection can be a wireless communication connection or a wired communication connection. The embodiments of this application do not limit the specific type of the above-mentioned communication connection.

[0063] Optionally, the end effector of the robotic arm of robot 11 may be equipped with a binocular camera 110. The binocular camera 110 may include a left camera and a right camera. The left camera and the right camera can be analogous to the left and right eyes of a human, respectively. Based on this, for ease of understanding, the image captured by the left camera is referred to as the left-eye image, and the image captured by the right camera is referred to as the right-eye image.

[0064] Optionally, in a mobile phone scenario, robot 11 can be a surgical robot, and binocular camera 110 can be a binocular endoscope. Optionally, in other scenarios, robot 11 can be other types of robots, and binocular camera can be other types of binocular cameras. This application embodiment does not limit the specific types of robot 11 and binocular camera 110.

[0065] Optionally, the first electronic device 10 can be a mobile phone, tablet computer, laptop computer, or desktop computer, etc. This application embodiment does not limit the type of the first electronic device 10.

[0066] Optionally, the head-mounted device 20 may include an AR device, a VR device, a mixed reality (MR) device, or smart glasses, etc. This application embodiment does not limit the type of head-mounted device 20.

[0067] Optionally, the first electronic device 10 can be used to perform the various steps executed by the first electronic device in subsequent method embodiments, and the head-mounted device 20 can be used to perform the various steps executed by the head-mounted device in subsequent method embodiments. Therefore, the specific functions of the first electronic device 10 and the head-mounted device 20 can be referred to the description in subsequent method embodiments, and will not be detailed here.

[0068] Optionally, both the first electronic device 10 and the head-mounted device 20 can be equipped with audio acquisition devices and audio playback devices. The first electronic device 10 can acquire first audio data and send the first audio data to the head-mounted device 20 based on the Web real-time communication (WebRTC) protocol, and the head-mounted device 20 can play the first audio data. Similarly, the head-mounted device can acquire second audio data and send the second audio data to the first electronic device 10 based on the WebRTC protocol, and the first electronic device 10 can play the second audio data.

[0069] Optionally, the binocular camera 110 can also capture target image video in video mode and send the target image video to the head-mounted device 20 based on the WebRTC protocol.

[0070] Based on the above-described robot remote guidance system, this application also provides a robot remote guidance method applied to the system. Figure 2 This is a schematic flowchart illustrating a remote robot guidance method provided in an embodiment of this application. Figure 2 As shown, the method may include S21 to S24, which are detailed below:

[0071] S21, the first electronic device acquires the left and right eye image group sequences captured by the binocular camera, and for each group of original left eye images and original right eye images in the left and right eye image group sequences, determines the target disparity map based on the left or right camera of the binocular camera.

[0072] Optionally, the binocular camera can acquire a set of raw left-eye images and raw right-eye images at first time intervals. The raw left-eye images and raw right-eye images can refer to the unprocessed left-eye and right-eye images acquired by the binocular camera. Both the raw left-eye and raw right-eye images can be color images; that is, the pixel value of each pixel in the raw left-eye and raw right-eye images can be composed of the color values ​​of that pixel in the red, green, and blue channels.

[0073] It should be understood that multiple sets of original left-eye and right-eye images acquired by a binocular camera can be arranged sequentially according to the acquisition time to obtain the left and right eye image sequence. For example, Figure 3 This is a schematic diagram of a left and right eye image group sequence provided in an embodiment of this application. Figure 3 As shown, the left and right eye image group sequence can include multiple left and right eye image groups arranged in chronological order of acquisition time. Each left and right eye image group includes one original left eye image and one original right eye image.

[0074] Optionally, the first electronic device can acquire a sequence of left and right eye images from the binocular camera every second time interval. Both the first and second time intervals can be set according to actual needs.

[0075] It should be understood that the pixel value (i.e., the disparity value) of each pixel in the target disparity map can be used to describe the horizontal deviation between the left and right cameras at the corresponding spatial point.

[0076] It should be understood that since the original left-eye and right-eye images captured by binocular cameras usually have optical distortion, and the imaging planes of the left and right cameras may have positional deviations, if the disparity map is determined directly based on the original left-eye and right-eye images, the pixel matching step of the original left-eye and right-eye images in the disparity map determination process requires a lot of computing resources, resulting in low efficiency in determining the disparity map, and consequently, poor real-time performance of remote robot guidance.

[0077] Based on this, optionally, in S21, a target disparity map is determined based on the left or right camera of the binocular camera, which may specifically include, for example... Figure 4 S211 to S212 are described in detail below:

[0078] S211, based on the parameter information of the binocular camera, correct the original left-eye image and the original right-eye image to obtain the row-aligned target left-eye image and target right-eye image.

[0079] Optionally, the parameter information of the binocular camera can be obtained by controlling the binocular camera to take images of a calibration reference object containing known geometric features from different angles, and by using a camera calibration algorithm to perform mathematical calculations based on the correspondence between the pixel coordinates of feature points in the image and the real physical coordinates.

[0080] Optionally, the parameter information of the binocular camera may include intrinsic and extrinsic parameters.

[0081] Internal parameter information may include the principal point coordinates of the left camera, the focal length of the left camera, the distortion coefficient of the left camera, the principal point coordinates of the right camera, the focal length of the right camera, and the distortion coefficient of the right camera.

[0082] The principal point coordinates can refer to the two-dimensional coordinates of the intersection of the camera's optical axis and the imaging plane in the pixel coordinate system (i.e., the image coordinate system).

[0083] Focal length refers to the distance from the optical center of a camera lens to the imaging plane.

[0084] Distortion coefficients can include radial distortion coefficients and tangential distortion coefficients. Radial distortion coefficients can be used to correct barrel or pincushion distortion caused by lens deformation in the camera. Tangential distortion coefficients can be used to correct centrifugal deformation caused by non-parallel lens assembly in the camera.

[0085] Extrinsic parameter information can include rotation matrices and translation vectors.

[0086] The rotation matrix can be used to describe the rotation angle of the right camera relative to the left camera, or the rotation angle of the left camera relative to the right camera. For example, the rotation matrix can be a 3×3 matrix.

[0087] A translation vector can be used to describe the spatial displacement of the optical center of the right camera relative to the optical center of the left camera, or vice versa. For example, the translation vector can be a 3×1 vector. It should be understood that the magnitude of the translation vector is the baseline of the stereo camera. The baseline of the stereo camera can be used to describe the straight-line distance between the optical centers of the left and right cameras.

[0088] Based on this, the first electronic device can use the distortion coefficients of the left camera and the right camera to perform distortion correction on the original left-eye image and the original right-eye image respectively, so as to eliminate the optical distortion caused by the lens deformation of the left and right cameras; and can use the rotation matrix and translation vector of the left camera and the rotation matrix and translation vector of the right camera to perform epipolar correction on the distortion-corrected original left-eye image and the original right-eye image respectively, to obtain the row-aligned target left-eye image and target right-eye image, that is, the pixels at the same position in the target left-eye view and the target right-eye view are on the same horizontal scan line.

[0089] It should be noted that the specific processes of the above distortion correction and epipolar correction can be found in the descriptions in related technologies, and will not be detailed in the embodiments of this application.

[0090] S212, Calculate the target disparity map based on the target left-eye image and the target right-eye image, using either the target left-eye image or the target right-eye image as the reference image.

[0091] The reference image corresponds to the reference camera; that is, the reference image can be either the left-eye image or the right-eye image of the target. Optionally, when the target disparity map is determined using the left camera as a reference, the left-eye image of the target is the reference image. Optionally, when the target disparity map is determined using the right camera as a reference, the right-eye image of the target is the reference image.

[0092] It should be understood that the reference image is a three-channel image, such as... Figure 5 As shown, the pixel value of each pixel in the reference image can be composed of the red, green and blue values ​​of that pixel in the RGB channels.

[0093] Optionally, the first electronic device may employ a semi-global matching (SGM) algorithm that balances disparity map calculation accuracy and real-time performance to calculate the target disparity map.

[0094] Optionally, the first electronic device may employ a deep learning algorithm to compute the target disparity map. The deep learning algorithm may include, for example, an algorithm based on a stereo matching network (StereoNet), an algorithm based on a lightweight stereo matching network (LightStereo), or an algorithm based on a hierarchical iterative tile refinement network (HITNet), etc. This application does not limit the specific type of deep learning algorithm used in its embodiments.

[0095] S22, for each pixel in each target disparity map, the first electronic device uses the integer part and the fractional part of the pixel value as the first channel value and the second channel value of the pixel, respectively, and uses a preset placeholder as the third channel value of the pixel, thus obtaining a three-channel disparity map corresponding to each target disparity map.

[0096] It should be understood that the pixel values ​​of each pixel in the target disparity map are usually floating-point numbers, and floating-point numbers are generally difficult to compress and transmit efficiently using image compression algorithms. Therefore, in order to improve data transmission efficiency while completely preserving disparity data, the first electronic device can split the pixel value of each pixel in each target disparity map into an integer part and a fractional part, and use the integer part and fractional part of each pixel as the first channel value and the second channel value of that pixel, respectively, and use a preset placeholder as the third channel value of that pixel, thereby obtaining a three-channel disparity map corresponding to each target disparity map.

[0097] Optionally, the three-channel disparity map can occupy a total of 3 bytes, or 24 bits. For example, as shown... Figure 5 As shown, the first channel value can occupy 12 bits, the second channel value can occupy 4 bits, and the preset placeholder can occupy 8 bits. It should be noted that the number of bits occupied by the first channel value, the second channel value, and the preset placeholder can also be other numbers, and this application embodiment does not limit them.

[0098] Optionally, the preset placeholder can be either an all-zero placeholder or an all-one placeholder. This application does not limit the specific type of the preset placeholder in its embodiments.

[0099] S23, the first electronic device uses a preset image compression algorithm to compress the three-channel disparity map and the reference image to obtain the target compressed file, and sends the target compressed file and the parameter information of the binocular camera to the head-mounted device.

[0100] Optionally, the preset image compression algorithm can be set according to actual needs. This application does not limit the specific type of the preset image compression algorithm in its embodiments.

[0101] Optionally, to improve the reliability of data transmission, the first electronic device may send the target compressed file and the parameter information of the binocular camera to the head-mounted device based on the transmission control protocol (TCP) / internet protocol (IP).

[0102] It should be understood that since the three-channel disparity map and the reference image can be directly compressed using image compression algorithms, this embodiment of the application uses a three-channel disparity map to represent depth information, which can reduce the amount of data transmission while maintaining high image quality. Specifically:

[0103] One traditional method for remote robot guidance is to send the acquired 3D coordinates and color information of various spatial points to the guiding device. However, the coordinate values ​​of each dimension in the 3D coordinates are usually floating-point numbers, and each floating-point number occupies 4 bytes, so the 3D coordinates require a total of 12 bytes; the color information usually includes color values ​​of three channels, and each channel's color value occupies 1 byte, so the color information requires a total of 3 bytes. That is, the information for each spatial point requires 15 bytes.

[0104] A second traditional method for remote robot guidance typically involves sending color and depth information for each spatial point to the guiding device. However, color information usually includes three channels of color values, with each channel occupying one byte, thus requiring a total of three bytes. Depth values ​​are usually floating-point numbers, requiring four bytes. Therefore, the information for each spatial point requires seven bytes.

[0105] Unlike traditional remote robot guidance methods, this embodiment does not directly send the three-dimensional coordinates or depth values ​​of spatial points to the guidance device. Instead, it represents the depth value of spatial points using a processed three-channel disparity map that occupies only 3 bytes. Including the 3 bytes for color information, each spatial point only requires 6 bytes of information, thus reducing data transmission volume. Furthermore, because traditional methods transmit data containing floating-point numbers, it is difficult to compress using ordinary image compression algorithms. However, since the three-channel disparity map and reference image in this embodiment are regular two-dimensional images with strong temporal and spatial correlation, they can be directly compressed efficiently using image compression algorithms. This not only shortens computation time but also reduces image quality loss, thereby meeting the real-time and high-quality requirements of remote surgical guidance.

[0106] S24, after parsing the three-channel disparity map and reference image from the received target compressed file, the head-mounted device determines the target three-dimensional coordinates of the spatial point corresponding to each pixel in the three-channel disparity map in the head-mounted device's head display coordinate system based on the parameter information of the binocular camera, and determines the pixel value of the corresponding pixel in the reference image as the color information of the spatial point. Based on the target three-dimensional coordinates and color information of each spatial point, all spatial points are rendered and displayed.

[0107] Optionally, after receiving the target compressed file from the first electronic device, the head-mounted device can use a preset image decompression algorithm corresponding to the preset image compression algorithm to parse the three-channel disparity map and reference image from the target compressed file.

[0108] Alternatively, the head-mounted device can store the decompressed channel disparity map and reference image in a pre-allocated structured buffer in its graphics processing unit (GPU).

[0109] Optionally, in S24, based on the parameter information of the binocular camera, the target's three-dimensional coordinates in the head-mounted display coordinate system of the head-mounted device are determined for each pixel in the three-channel disparity map. Specifically, this may include, for example... Figure 6 S241 to S242 are described in detail below:

[0110] S241, construct the reprojection matrix based on the parameter information of the binocular camera.

[0111] Alternatively, the reprojection matrix constructed by the head-mounted electronic device can be:

[0112]

[0113] in, Q For the reprojection matrix, ( c x , c y () represents the principal point coordinates of the reference camera. f The focal length of the reference camera. b This is the baseline for the binocular camera.

[0114] S242, for each pixel in the three-channel disparity map, the sum of the first channel value and the second channel value of the pixel is determined as the disparity value of the pixel. Based on the image coordinates, disparity value and reprojection matrix of the pixel, the original three-dimensional coordinates of the spatial point corresponding to the pixel in the camera coordinate system of the binocular camera are calculated. The original three-dimensional coordinates are transformed by the target transformation matrix from the camera coordinate system to the head-mounted display coordinate system to obtain the target three-dimensional coordinates of the spatial point.

[0115] The image coordinates of a pixel can be represented by the number of rows and columns of that pixel in a three-channel disparity map.

[0116] The spatial point corresponding to a pixel can refer to the imaging point that the pixel corresponds to in three-dimensional space.

[0117] Optionally, in step S242, the original three-dimensional coordinates of the spatial point corresponding to the pixel in the camera coordinate system of the stereo camera are calculated based on the image coordinates, disparity value, and reprojection matrix of the pixel. This may specifically include the following steps 1.1 to 1.2:

[0118] Step 1.1: Based on the image coordinates, disparity value, and reprojection matrix of the pixel, construct the homogeneous coordinate vector corresponding to the pixel using the following formula (1):

[0119] [ X , Y , Z , W ] T = Q ×[ u , v , d ( u , v ),1] T ;Formula (1)

[0120] in,[ X , Y , Z , W ] T This is the homogeneous coordinate vector corresponding to the pixel. Q For the reprojection matrix, ( u , v ) represents the image coordinates of that pixel. d(u , v ) represents the disparity value of that pixel.

[0121] also, W The scale factor is the homogeneous coordinate vector mentioned above. W = d(u , v ) / b .

[0122] Step 1.2: Based on the homogeneous coordinate vector, calculate the original three-dimensional coordinates of the spatial point corresponding to the pixel in the camera coordinate system of the stereo camera using the following formula (2):

[0123] ( p x , p y , p z )=( X / W , Y / W , Z / W ); formula (2)

[0124] in,(p x , p y , p z ) represents the original three-dimensional coordinates of the spatial point corresponding to this pixel.

[0125] As can be seen, the three-channel disparity map can be reprojected into the camera coordinate system using the above formulas (1) and (2). Furthermore, by using the pixel values ​​of each pixel in the reference image as the color information of the corresponding spatial point, the information of the original point cloud data composed of all spatial points can be obtained.

[0126] It should be understood that in order to recreate the surgical scene in the head-mounted device, the head-mounted device also needs to transform the original three-dimensional coordinates of each spatial point to its head-mounted display coordinate system.

[0127] Specifically, the head-mounted device can transform the original three-dimensional coordinates of each spatial point in the camera coordinate system to the head-mounted device's head-mounted display coordinate system through the following steps 1.3 to 1.5:

[0128] Step 1.3: Obtain the spatial point pose control matrix, the camera rendering matrix of the stereo camera, and the current pose matrix of the head-mounted device.

[0129] For example, Figure 7 This diagram illustrates the coordinate transformations involved in implementing a remote robot guidance method according to an embodiment of this application. Figure 7 As shown, the camera rendering matrix describes the spatial transformation from the camera coordinate system to the point cloud rendering coordinate system. It is typically a fixed value and can be determined using the parameters of the stereo camera. The spatial point pose control matrix describes the spatial transformation from the point cloud rendering coordinate system to the world coordinate system of the guidance space. It can be set by the user, for example, it can be an identity matrix. The current pose matrix of the head-mounted device describes the spatial transformation from the world coordinate system of the guidance space to the head-mounted device's head-mounted display coordinate system.

[0130] Optionally, the camera coordinate system can be represented by {E}, the point cloud rendering coordinate system by {P}, the world coordinate system by {W}, and the head-mounted display coordinate system by {H}. Based on this, the camera rendering matrix can be... T E P This indicates that the spatial point pose control matrix can be achieved through... T P W This indicates that the current pose of the head-mounted device can be determined through... T W H This means that the target transformation matrix can be obtained through...T E H express.

[0131] Step 1.4: Based on the camera rendering matrix, the spatial point pose control matrix, and the current pose matrix of the head-mounted device, construct the target transformation matrix from the camera coordinate system to the head-mounted display coordinate system using the following formula (3):

[0132] T E H = T W H × T P W × T E P ;Formula (3)

[0133] in, T E H Let be the target transformation matrix. T W H This is the current pose matrix of the head-mounted device. T P W For spatial point pose control matrix, T E P Render the matrix for the camera.

[0134] Step 1.5: Based on the original three-dimensional coordinates of the spatial point, the target transformation matrix, and the spatial point scale parameters, calculate the target three-dimensional coordinates of the spatial point using the following formula (4):

[0135] P fn = a × T E H × P on ;Formula (4)

[0136] in, P fn The target's three-dimensional coordinates are given by the spatial point. a For spatial point scale parameters, P on These are the original three-dimensional coordinates of the point in space.

[0137] Spatial point scale parameters can be used to control the scaling of point clouds composed of all spatial points, thereby facilitating the magnification of various details (such as lesion areas) for observation and guidance.

[0138] As can be seen from the above, this embodiment calculates the target disparity map of each left and right eye image group acquired by the binocular camera, and splits the pixel value of each pixel in the target disparity map into an integer part and a fractional part. The integer part, fractional part, and a preset placeholder are used as the first channel value, second channel value, and third channel value of the pixel, respectively, thereby constructing a corresponding three-channel disparity map. Since the three-channel disparity map can be efficiently compressed using image compression technology compared to the traditional floating-point form of the disparity map, a preset image compression algorithm is used to compress the three-channel disparity map and the reference image into a target compressed file. This target compressed file and the parameter information of the binocular camera are then sent to the head-mounted device at the guidance end. This not only improves data transmission efficiency and reduces data latency, but also enables the head-mounted device to obtain the depth information of the operation end scene represented by the disparity map and reconstruct the operation end scene accordingly, thereby improving the accuracy of remote guidance. Therefore, the robot remote guidance method provided in this embodiment can simultaneously meet the high real-time and high-quality requirements of remote interaction.

[0139] Figure 8 This is a schematic flowchart illustrating a remote robot guidance method according to another embodiment of this application. Figure 8 As shown, this embodiment is similar to Figure 2 Compared to the corresponding embodiments, it may also include S81 to S82:

[0140] S81, the head-mounted device receives and recognizes the target virtual gesture input by the user.

[0141] Target virtual gestures may include, for example, pinch gestures, drag gestures, zoom gestures, and selection gestures. For instance, a head-mounted device may employ a preset gesture recognition algorithm to recognize the user's input of the target virtual gesture. This preset gesture recognition algorithm can be set according to actual needs and is not limited here.

[0142] S82, the head-mounted device executes the corresponding target operation based on the target virtual gesture.

[0143] Optionally, the head-mounted device can control the movement of a corresponding target virtual tool (such as a paintbrush) in the head-mounted display coordinate system based on the target virtual gestures, in order to draw surgical guidance trajectories or mark surgical lesion areas.

[0144] Optionally, the head-mounted device can also generate surgical guidance information based on the motion trajectory of the target virtual tool in the coordinates of the virtual tool, and can send the surgical guidance information to the first electronic device based on the TCP / IP protocol or the user datagram protocol (UDP).

[0145] The aforementioned motion trajectory can be represented by the coordinates of all trajectory points in the virtual tool coordinate system. For example, as shown... Figure 7 As shown, the virtual tool coordinate system can be represented by {D}. The transformation relationship from the virtual tool coordinate system to the head-mounted display coordinate system can be obtained through... T D H express.

[0146] Optionally, after receiving the surgical guidance information, the first electronic device can use the following formula (5) to calculate the motion trajectory of the target virtual tool in the camera coordinate system:

[0147] T D E = T H E × T D H ;Formula (5)

[0148] in, T D E The trajectory of the target virtual tool in the camera coordinate system. T H E This is the transformation matrix from the head-mounted display coordinate system to the camera coordinate system. T D H This is the transformation matrix from the virtual tool coordinate system to the head-mounted display coordinate system.

[0149] Optionally, after the first electronic device calculates the motion trajectory of the target virtual tool in the camera coordinate system, it can overlay the motion trajectory onto the 3D image it displays to obtain real-time visual guidance.

[0150] As can be seen from the above, this embodiment can improve the efficiency of remote guidance of the robot by allowing the instructor to input guidance operations using virtual gestures; by superimposing the instructor's guidance information onto the 3D image of the first electronic device, the operator can immerse themselves in viewing the guidance information, thereby further improving the efficiency and quality of remote guidance of the robot.

[0151] It is understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0152] This application also provides an electronic device. Figure 9This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Figure 9 As shown, the electronic device 9 may include: a processor 90, a memory 91, and a computer program 92 stored in the memory 91 and executable on the processor 90. When the processor 90 executes the computer program 92, it implements the steps corresponding to the first electronic device in the above-described robot remote guidance method embodiment, for example... Figure 2 S21~S23 are shown.

[0153] For example, computer program 92 may be divided into one or more modules / units, one or more of which are stored in memory 91 and executed by processor 90. One or more modules / units may be a series of computer program instruction segments capable of performing a specific function, which describe the execution process of computer program 92 in electronic device 9.

[0154] It should be understood that Figure 9 This is merely an example of electronic device 9 and does not constitute a limitation on electronic device 9. It may include more or fewer components than shown, or combine certain components, or use different components.

[0155] The processor 90 can be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor.

[0156] The memory 91 can be an internal storage unit of the electronic device 9, such as a hard disk or RAM. The memory 91 can also be an external storage device of the electronic device 9, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, or flash card. Furthermore, the memory 91 can include both internal and external storage units of the electronic device 9. The memory 91 is used to store computer programs and other programs and data required by the electronic device. The memory 91 can also be used to temporarily store data that has been output or will be output.

[0157] This application also provides a head-mounted device. Figure 10 This is a schematic diagram of the structure of a head-mounted device provided in an embodiment of this application. Figure 10 As shown, the head-mounted device 20 may include a processor 100, a memory 101, and a computer program 102 stored in the memory 101 and executable on the processor 100. When the processor 100 executes the computer program 102, it implements the steps corresponding to the head-mounted device in the above-described robot remote guidance method embodiment, for example... Figure 2 S24 is shown.

[0158] For example, computer program 102 may be divided into one or more modules / units, one or more of which are stored in memory 101 and executed by processor 100. One or more modules / units may be a series of computer program instruction segments capable of performing a specific function, which describe the execution process of computer program 102 in head-mounted device 20.

[0159] It should be understood that Figure 10 This is merely an example of head-mounted device 20 and does not constitute a limitation on head-mounted device 20. It may include more or fewer components than illustrated, or combine certain components, or use different components.

[0160] Processor 100 can be a CPU, or other general-purpose processors, DSPs, ASICs, FPGAs, or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor can be a microprocessor or any conventional processor.

[0161] The memory 101 can be an internal storage unit of the head-mounted device 20, such as a hard disk or RAM of the head-mounted device 20. The memory 101 can also be an external storage device of the head-mounted device 20, such as a plug-in hard disk, SMC, SD card, or flash card provided on the head-mounted device 20. Furthermore, the memory 101 can include both internal and external storage units of the head-mounted device 20. The memory 101 is used to store computer programs and other programs and data required by the electronic device. The memory 101 can also be used to temporarily store data that has been output or will be output.

[0162] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, refer to the relevant descriptions of other embodiments.

[0163] It should be noted that, unless otherwise specified, all technical terms used in the embodiments of this application have the same meaning as commonly understood by those skilled in the art to which this application belongs. The technical terms used in the embodiments of this application are only used to explain specific embodiments of this application and are not intended to limit this application.

[0164] The term "embodiment" as used in the description of embodiments in this application means that a specific feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places in the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0165] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0166] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. A method of robot remote guidance, characterized by, The application is applied to a robot remote guidance system, which comprises a first electronic device arranged at an operation end and a head-mounted device arranged at a guidance end; the first electronic device is in communication connection with a robot at the operation end and the head-mounted device; The mechanical arm end of the robot is provided with a binocular camera; the method comprises: The first electronic device acquires a left-eye image group sequence collected by the binocular camera, and determines a target disparity map with a left camera or a right camera of the binocular camera as a reference for each original left-eye image and original right-eye image in the left-eye image group sequence; The first electronic device takes an integer part and a decimal part of a pixel value of each pixel in each target disparity map as a first channel value and a second channel value of the pixel respectively, and takes a preset placeholder as a third channel value of the pixel, to obtain a three-channel disparity map corresponding to each target disparity map respectively; The first electronic device adopts a preset image compression algorithm to compress the three-channel disparity map and a reference image to obtain a target compressed file, and sends the target compressed file and parameter information of the binocular camera to the head-mounted device; the reference image is a left-eye image or a right-eye image; the parameter information comprises a principal point coordinate of a reference camera of the binocular camera, a focal length of the reference camera and a baseline of the binocular camera; the reference camera is a left camera or a right camera of the binocular camera; The head-mounted device parses the three-channel disparity map and the reference image from the received target compressed file, and constructs a re-projection matrix according to the parameter information; The head-mounted device determines a disparity value of each pixel in the three-channel disparity map as a sum of the first channel value and the second channel value of the pixel, and constructs a homogeneous coordinate vector corresponding to the pixel by using the following formula according to an image coordinate of the pixel, the disparity value and the re-projection matrix: [ X , Y , Z , W ] T = Q ×[ u , v , d ( u , v ),1] T ； wherein, X , Y , Z , W ] T is a homographic coordinate vector corresponding to the pixel, Q is the re-projection matrix, u , v is an image coordinate of the pixel, d(u , v is the disparity value; and the re-projection matrix is: ； wherein, c x , c y ) is a principal point coordinate of the reference camera, f is a focal length of the reference camera, b is a base line of the binocular camera; The head-mounted device calculates original three-dimensional coordinates of a space point corresponding to the pixel in a camera coordinate system of the binocular camera by using the following formula according to the homogeneous coordinate vector: ( p x , p y , p z )=( X / W , Y / W , Z / W )； wherein, p x , p y , p z are the original three-dimensional coordinates of the spatial point corresponding to the pixel; The head-mounted device performs coordinate transformation on the original three-dimensional coordinates by using a target transformation matrix from the camera coordinate system to a head-mounted device coordinate system to obtain target three-dimensional coordinates of the space point; The head-mounted device determines a pixel value of a corresponding pixel in the reference image as color information of the space point, and renders and displays all the space points according to the target three-dimensional coordinates and the color information of each space point.

2. The method of claim 1, wherein, The method for determining a target disparity map with a left camera or a right camera of the binocular camera as a reference comprises: According to the parameter information of the binocular camera, the original left-eye image and the original right-eye image are corrected to obtain target left-eye images and target right-eye images in line alignment; The method for determining a target disparity map with a left camera or a right camera of the binocular camera as a reference comprises: According to the parameter information of the binocular camera, the original left-eye image and the original right-eye image are corrected to obtain target left-eye images and target right-eye images in line alignment; According to the target left-eye image and the target right-eye image, a target disparity map is calculated, taking the target left-eye image or the target right-eye image as the reference image; the reference image corresponds to a reference camera.

3. The method of claim 1, wherein, The original three-dimensional coordinates are subjected to coordinate transformation by using a target transformation matrix from the camera coordinate system to the head-mounted device coordinate system, to obtain target three-dimensional coordinates of the spatial point, including: A spatial point pose control matrix, a camera rendering matrix of the binocular camera, and a current pose matrix of the head-mounted device are obtained; According to the camera rendering matrix, the spatial point pose control matrix, and the current pose matrix, a target transformation matrix from the camera coordinate system to the head-mounted device coordinate system is constructed by using the following formula: T E H = T W H × T P W × T E P ； wherein, T E H is the target transformation matrix, T W H is the current pose matrix, T P W is the spatial point pose control matrix, T E P is the camera rendering matrix; According to the original three-dimensional coordinates of the spatial point, the target transformation matrix, and a spatial point scale parameter, target three-dimensional coordinates of the spatial point are calculated by using the following formula: P fn = a × T E H × P on ； wherein, P fn is the target three-dimensional coordinate for the spatial point, a is a scale parameter for the spatial point, P on is the original three-dimensional coordinate for the spatial point.

4. The method according to any one of claims 1 to 3, characterized in that, Further comprising: The head-mounted device receives and recognizes a target virtual gesture input by a user; The head-mounted device performs a corresponding target operation according to the target virtual gesture.

5. The method of claim 4, wherein, The head-mounted device performs a corresponding target operation according to the target virtual gesture, including: The head-mounted device controls a target virtual object to move in a three-dimensional space under the head-mounted device coordinate system according to the target virtual gesture, to draw a surgical guidance track or mark a surgical lesion area; Correspondingly, the method further comprises: The head-mounted device generates surgical guidance information according to the movement information of the target virtual object, and sends the surgical guidance information to the first electronic device.

6. An electronic device, comprising: A computer program is stored in the memory and executable on the processor, and the processor executes the computer program to realize each step corresponding to the first electronic device in the method of any one of claims 1-5.

7. A head-mounted device, comprising: A computer program is stored in the memory and executable on the processor, and the processor executes the computer program to realize each step corresponding to the head-mounted device in the method of any one of claims 1-5.

8. A robotic remote guidance system characterized by, A robot and a first electronic device are arranged at an operating end, and a head-mounted device is arranged at a guidance end; the first electronic device is in communication connection with the robot and the head-mounted device; a mechanical arm of the robot is provided with a binocular camera; The first electronic device is used to perform each step corresponding to the first electronic device in the method of any one of claims 1-5; The head-mounted device is used to perform each step corresponding to the head-mounted device in the method of any one of claims 1-5.

Images