Image processing method and device of robot control terminal and robot control terminal

Abstract

The invention provides an image processing method and device of a robot control terminal and the robot control terminal. On the basis of the method, a robot control terminal can determine form adjustment data for a target image according to first operation data while receiving the first operation data of a user for an endoscope image in an image interface, responding to the first operation data and displaying the adjusted endoscope image in the image interface; according to the endoscope image, content adjustment data for the target image is determined through feature matching; and synchronously displaying an adjusted target image matched with the adjusted endoscope image in an image interface according to the form adjustment data and / or the content adjustment data. Therefore, when the user adjusts the endoscope image, the target image displayed in the same image interface with the endoscope image at the same time can be intelligently and automatically adjusted in a relatively matched manner, the user operation is effectively simplified, and the user can obtain relatively good interaction experience.

Description

Technical Field

[0001] This specification belongs to the technical field of medical robots, and particularly relates to an image processing method, apparatus, and robot control terminal for a robot control terminal. Background Art

[0002] In order to facilitate a user to clearly observe the operation area where a medical robot docks and accurately control the medical robot to complete related operations in this area, usually while a robot control terminal connected to the medical robot displays the real image (for example, an endoscope image, etc.) collected through a display screen to the user, it will also display a virtual image (for example, a nuclear magnetic image, etc.) associated with the real image in a form such as picture-in-picture to assist the user in analysis and judgment.

[0003] Specifically, during the process of a user controlling a medical robot to perform related operations through a control terminal, the user also needs to adjust the size and / or position of the real image displayed on the display screen from time to time. Based on existing methods, after the user adjusts the size and / or position of the real image, the user often needs to separately perform corresponding manual adjustments on the virtual image displayed on the same screen so that the virtual image can be adapted to the adjusted real image, which is more convenient for the user to observe and use. This increases the operation burden of the user and also affects the user's interaction experience.

[0004] Regarding the above problems, no effective solution has been proposed yet. Summary of the Invention

[0005] This specification provides an image processing method, apparatus, and robot control terminal for a robot control terminal. When a user adjusts an endoscope image, the target image displayed in the same image interface as the endoscope image can be automatically adjusted in a relatively matching and adaptive manner, enabling the user to obtain a better interaction experience.

[0006] This specification provides an image processing method for a robot control terminal, including:

[0007] Receiving first operation data for an endoscope image in an image interface; wherein, the target image is also displayed in the image interface; the target image is a virtual image associated with the endoscope image;

[0008] Determining form adjustment data for the target image according to the first operation data; wherein, the form adjustment data at least includes position adjustment data and / or size adjustment data;

[0009] Determining content adjustment data for the target image through feature matching according to the endoscope image;

[0010] Adjust the data according to the form and / or the content adjustment data to generate an adjusted target image; and display the adjusted target image in the image interface.

[0011] This specification also provides an image processing device for a robot control terminal, including:

[0012] A receiving module, configured to receive first operation data for an endoscopic image in an image interface; wherein, the target image is also displayed in the image interface; the target image is a virtual image associated with the endoscopic image;

[0013] A first determination module, configured to determine form adjustment data for the target image according to the first operation data; wherein, the form adjustment data at least includes position adjustment data and / or size adjustment data;

[0014] A second determination module, configured to determine content adjustment data for the target image by feature matching according to the endoscopic image;

[0015] A processing module, configured to generate an adjusted target image according to the form adjustment data and / or the content adjustment data; and display the adjusted target image in the image interface.

[0016] This specification also provides a robot control terminal, at least including a display screen, a control arm, and a processor; wherein,

[0017] The display screen is configured to display an image interface; the endoscopic image and the target image are simultaneously displayed in the image interface; the target image is a virtual image associated with the endoscopic image;

[0018] The control arm is configured to receive first operation data for the endoscopic image in the image interface;

[0019] The processor is configured to correspondingly adjust the size and / or position of the endoscopic image according to the first operation data to obtain an adjusted endoscopic image; the processor is further configured to determine form adjustment data for the target image according to the first operation data; wherein, the form adjustment data at least includes position adjustment data and / or size adjustment data; determine content adjustment data for the target image by feature matching according to the endoscopic image; generate an adjusted target image according to the form adjustment data and / or the content adjustment data;

[0020] The display screen is further configured to display an updated image interface; the adjusted endoscopic image and the adjusted target image are simultaneously displayed in the updated image interface.

[0021] The present specification also provides an image processing method for a robot control terminal, including: receiving first operation data for a first image in an image interface; wherein, the target image is also displayed in the image interface; the target image is associated with the first image; determining form adjustment data for the target image according to the first operation data; wherein, the form adjustment data at least includes position adjustment data and / or size adjustment data; determining content adjustment data for the target image according to the first image through feature matching; generating an adjusted target image according to the form adjustment data and / or the content adjustment data; and displaying the adjusted target image in the image interface.

[0022] Based on the image processing method, apparatus, and robot control terminal provided in the present specification, when the robot control terminal receives first operation data of the user for the endoscope image in the image interface and responds to the first operation data, while displaying the adjusted endoscope image in the image interface, it can determine form adjustment data for the target image according to the first operation data; and determine content adjustment data for the target image through feature matching according to the endoscope image; then synchronously display the corresponding adjusted target image in the image interface according to the form adjustment data and / or the content adjustment data. Thus, while the user adjusts the endoscope image, the target image displayed in the same image interface as the endoscope image can be automatically adjusted in a relatively matching and adaptive manner, which can effectively simplify the user operation and enable the user to obtain a better interaction experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] In order to more clearly illustrate the embodiments of the present specification, the drawings required for the embodiments will be briefly introduced below. The drawings in the following description are only some embodiments recorded in the present specification. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0024] Figure 1 is a schematic flowchart of an image processing method for a robot control terminal provided by an embodiment of the present specification;

[0025] Figure 2 is a schematic diagram of an embodiment of an image interface displayed by applying the robot control terminal provided in the present specification in a scenario example;

[0026] Figure 3 is a schematic diagram of a scenario in which the robot control terminal provided in the present specification is connected to a medical robot and an image processing device in a scenario example;

[0027] Figure 4It is a schematic diagram of the overall structural composition of a medical operation system to which the image processing method of the robot control terminal provided in this specification is applied in a scenario example;

[0028] Figure 5 It is a schematic diagram of the structural composition of a doctor's console in a medical operation system in a scenario example;

[0029] Figure 6 It is a schematic diagram of the data processing flow when switching to the image adjustment mode by applying the image processing method of the robot control terminal provided in the embodiments of this specification in a scenario example;

[0030] Figure 7 It is an example schematic diagram when switching to the image adjustment mode by applying the image processing method of the robot control terminal provided in the embodiments of this specification in a scenario example;

[0031] Figure 8 In a scenario example, it is a schematic diagram of a refined example of the image interface displayed by the robot control terminal provided in this specification;

[0032] Figure 9 It is a schematic diagram of the data processing flow when performing feature matching by applying the image processing method of the robot control terminal provided in the embodiments of this specification in a scenario example;

[0033] Figure 10 It is a schematic diagram of the data processing flow when performing virtual enhancement by applying the image processing method of the robot control terminal provided in the embodiments of this specification in a scenario example;

[0034] Figure 11 It is an example schematic diagram when performing virtual enhancement by applying the image processing method of the robot control terminal provided in the embodiments of this specification in a scenario example;

[0035] Figure 12 It is an example schematic diagram of the display method when displaying an image in the image interface by applying the image processing method of the robot control terminal provided in the embodiments of this specification in a scenario example;

[0036] Figure 13 It is an example schematic diagram when performing image exchange by applying the image processing method of the robot control terminal provided in the embodiments of this specification in a scenario example;

[0037] Figure 14 It is a schematic diagram of the structural composition of a computer device provided by an embodiment of this specification;

[0038] Figure 15It is a schematic diagram of the structural composition of an image processing device for a robot control terminal provided by an embodiment of this specification;

[0039] Figure 16 It is a schematic diagram of the structural composition of a robot control terminal provided by an embodiment of this specification;

[0040] Figure 17 It is a schematic diagram of an embodiment for determining an effective blank area in case 1 by applying the image processing method of the robot control terminal provided by the embodiment of this specification in a scenario example;

[0041] Figure 18 It is a schematic diagram of an embodiment for determining an effective blank area in case 2 by applying the image processing method of the robot control terminal provided by the embodiment of this specification in a scenario example. Detailed implementation manners

[0042] In order to enable those skilled in the art to better understand the technical solutions in this specification, the following will clearly and completely describe the technical solutions in the embodiments of this specification with reference to the accompanying drawings in the embodiments of this specification. Obviously, the described embodiments are only a part of the embodiments of this specification, rather than all the embodiments. Based on the embodiments in this specification, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the scope of protection of this specification.

[0043] Refer to Figure 1 As shown, the embodiment of this specification provides an image processing method for a robot control terminal. Specifically, the method may include the following content:

[0044] S101: Receive first operation data for an endoscopic image in an image interface; wherein, the target image is also displayed in the image interface; the target image is a virtual image associated with the endoscopic image;

[0045] S102: Determine form adjustment data for the target image according to the first operation data; wherein, the form adjustment data at least includes position adjustment data and / or size adjustment data;

[0046] S103: Determine content adjustment data for the target image through feature matching according to the endoscopic image;

[0047] S104: Generate an adjusted target image according to the form adjustment data and / or the content adjustment data; and display the adjusted target image in the image interface.

[0048] Among them, the above endoscopic image (or main picture) can specifically be a real image collected in real time or at regular intervals by an endoscope for a target area. The above target area can specifically be understood as the operation area (such as the surgical area) of a medical robot that is connected to the robot control terminal and controlled by the robot control terminal.

[0049] The above target image (or picture-in-picture) can specifically be a virtual image for the target area generated based on a preset image data source different from the endoscope and associated with the above endoscopic image. The above preset image data source can specifically include at least one of the following: nuclear magnetic imaging data source, ultrasonic imaging data source, X-ray imaging data source, computer simulation imaging data source, etc.

[0050] Specifically, for example, the above target image can be an ultrasonic image for the target area generated based on the ultrasonic imaging data source, a nuclear magnetic image for the target area generated based on the nuclear magnetic imaging data source, a simulation image for the target area generated based on the computer simulation imaging data source, and so on. Of course, the above-listed target images are only illustrative. The specific content and type of the above target images are not limited in this specification.

[0051] Refer to Figure 2 As shown, the above endoscopic image and target image can be displayed in the same image interface simultaneously. Specifically, two different windows can also be arranged in the above image interface, respectively denoted as the main window (or main picture window) and the secondary window (or picture-in-picture window). Usually, the main window can be used to display the endoscopic image, and the secondary window can be used to display the target image. Among them, the positions and sizes of the above main window and secondary window can change respectively according to the positions and sizes of the endoscopic image and target image.

[0052] The above first adjustment data can specifically be adjustment data for the position and / or size of the endoscopic image actively initiated by a user through the robot control terminal.

[0053] During specific implementation, the position and / or size of the endoscopic image can be adjusted accordingly according to the above first adjustment data to obtain an adjusted endoscopic image.

[0054] The above form adjustment data for the target image can specifically be understood as adjustment data for the position and / or size of the target image. Among them, the form adjustment data of the target image can match the position and / or size of the adjusted endoscopic image based on the image interface. The above content adjustment data for the target image can specifically be understood as adjustment data for the image content of the target image. Among them, the content adjustment data of the target image can match the image content of the adjusted endoscopic image.

[0055] In specific implementation, the form adjustment data and / or content adjustment data of the corresponding target image can be determined according to the first adjustment data; alternatively, the endoscopic image can be adjusted according to the first adjustment data to obtain the adjusted endoscopic image, and then the form adjustment data and / or content adjustment data of the corresponding target image can be determined according to the adjusted endoscopic image.

[0056] In specific implementation, different situations can be distinguished. The adjusted target image can be generated by separately determining and according to the form adjustment data; or the adjusted target image can be generated by separately determining and according to the content adjustment data; or the adjusted target image can be generated by simultaneously determining and according to the form adjustment data and the content adjustment data.

[0057] In specific implementation, after obtaining the adjusted endoscopic image and the adjusted target image, the adjusted endoscopic image and the adjusted target image can be displayed together on the image interface according to the preset layout rules.

[0058] In some embodiments, the image processing method of the above-mentioned robot control terminal can be specifically applied to one side of the robot control terminal. Specifically, reference can be made to Figure 3 as shown.

[0059] The above-mentioned robot control terminal can at least include structures such as a control arm, a processor, and a display screen. Further, a control joint can be provided on the above-mentioned control arm.

[0060] The above-mentioned robot control terminal can also be connected to a medical robot (such as a patient operation platform, a surgical robot, a human internal detection robot, etc.) and an image processing device respectively in a wired or wireless manner. Among them, the above-mentioned image processing system is connected to an endoscope and a preset image data source (such as a nuclear magnetic resonance image data source, an ultrasonic image data source, etc.). The above-mentioned endoscope can specifically extend into the surgical environment of the medical robot to collect endoscopic images in the surgical environment. The above-mentioned preset image data source is used to generate a virtual image associated with the endoscopic image. The above-mentioned medical robot is provided with a plurality of robotic arms, and corresponding operating instruments (such as surgical instruments) can be mounted on the above-mentioned robotic arms.

[0061] Specifically, the control arm and the control joint of the robot control terminal can be associated with the robotic arm of the medical robot. Correspondingly, the user can operate the control arm and the control joint of the robot control terminal to control the robotic arm of the medical robot to use the operating instrument to complete specific operations.

[0062] In specific implementation, the robot control terminal can receive the endoscope image sent by the image processing device and the image data obtained based on a preset image data source. After the robot control terminal processes the above data through a processor, the endoscope image and the target image associated with the endoscope image are simultaneously displayed in the image interface shown on the display screen.

[0063] In specific implementation, the user can, according to the endoscope image and the target image shown in the image interface, control the medical robot to perform specific operations through the control arm of the robot control terminal and the control joints on the control arm.

[0064] Based on the image processing method of the robot control terminal provided in the embodiments of this specification, the user can initiate the first operation data for the endoscope graphic by manipulating the control arm. While the robot control terminal makes corresponding adjustments to the endoscope graphic according to the first operation data, it will also automatically make corresponding adjustments to the target image according to the first operation data. Furthermore, the adjusted endoscope image and the adjusted target image can be simultaneously displayed in the image interface.

[0065] In this way, when the user uses the robot control terminal to make corresponding adjustments to the endoscope image, the robot control terminal can intelligently and automatically make adaptive adjustments to the target image associated with the endoscope image in a matching manner; then display the adjusted endoscope image and the adjusted target image to the user simultaneously. Thus, it can effectively simplify the user operation and enable the user to obtain a relatively good interaction experience.

[0066] In a specific scenario example, refer to Figure 4 As shown, the above robot control terminal can be a doctor's console. Specifically, refer to Figure 5 As shown, the above doctor's console at least includes a left control arm, a right control arm, and a monitor. Among them, the user can drag the left control arm and the right control arm to move within a certain area range. Further, one or more control joints allowing the user to perform pinching and releasing operations can be respectively arranged on the above left control arm and right control arm. The above monitor can also be provided with a display screen. In addition, the above doctor's console can also be provided with a control pedal for switching modes.

[0067] Refer to Figure 4 As shown, the above doctor's console can also be respectively connected to a patient operation platform (a kind of medical robot) and an image platform (or called an image trolley, a kind of image processing device). Among them, the above doctor's console, image platform, and patient operation platform can form a medical operation system.

[0068] Specifically, multiple robotic arms can be arranged on the above-mentioned patient operation platform. Among them, the above-mentioned robotic arms can be used to mount equipment such as surgical instruments and endoscopes required during surgical operations. Among them, the above-mentioned surgical instruments can specifically include any one of the following: duckbill grasping forceps, rat-tooth grasping forceps, strong duckbill grasping forceps, and so on.

[0069] The image platform at least includes devices such as an image host and a display. Further, the above-mentioned image platform can also be connected to an endoscope (which can be denoted as the first signal source). In addition, the above-mentioned image platform can also be connected to other relevant detection devices such as nuclear magnetic detection devices and ultrasonic detection devices (which can be denoted as the second signal source).

[0070] Correspondingly, the image host of the image platform can collect corresponding endoscope images in real time or at regular intervals through the connected endoscope; through the connected relevant detection devices, obtain the corresponding preset image data sources (for example, three-dimensional scanning models based on nuclear magnetic images, three-dimensional scanning models based on ultrasonic images, or computer simulation models, etc.).

[0071] During specific implementation, the image host can generate corresponding endoscope images (a kind of real image) by processing the image data collected by the endoscope; at the same time, it can also generate a target image (a kind of virtual image) associated with the above-mentioned endoscope image by using the preset image data source.

[0072] Further, the image platform can simultaneously display the endoscope image and the target image in an image interface through the display; at the same time, the image platform can also send the above-mentioned endoscope image and target image to the doctor's console. The doctor's console can simultaneously display the endoscope image and the target image to the user in an image interface through the display screen in the monitor.

[0073] During specific implementation, the above-mentioned doctor's console can be connected to the patient operation platform in a wired or wireless manner. Specifically, the left control arm, the right control arm, and the control joints provided on the left control arm and the right control arm can be associated with the robotic arms on the patient operation platform.

[0074] During specific implementation, the user can, according to the endoscope image and the target image displayed on the monitor of the doctor's console, control the robotic arms on the patient operation platform to use the mounted equipment to complete relevant surgical operations by manipulating the left control arm, the right control arm, and the control joints.

[0075] When the user wants to adjust the currently displayed endoscope image, the user can initiate a switching trigger action by stepping on the control pedal. When the doctor console detects that the user has initiated this switching trigger action, it can automatically switch to the image adjustment mode. In the image adjustment mode, the communication connection between the left control arm, the right control arm, the control joint of the doctor console and the robotic arm of the patient operating platform will be paused. That is, in the image adjustment mode, the user will no longer be able to control the robotic arm of the patient operating platform to perform corresponding movements by manipulating the left control arm, the right control arm, and the control joint.

[0076] Specifically, in the image adjustment mode, the user can initiate the first operation data for the endoscope image by manipulating the left control arm, the right control arm, and the control joint of the doctor console.

[0077] Correspondingly, the doctor console can adjust the size and / or position of the endoscope image according to the above first operation data to obtain an adjusted endoscope image. At the same time, the doctor console can also automatically adaptively adjust the target image based on the form dimension and / or content dimension in a matching manner according to the above first operation data, or based on the adjusted endoscope image obtained from the first operation data, to obtain an adjusted target image. Then, according to the preset layout rules, the adjusted endoscope image and the adjusted target image are simultaneously displayed in the image interface displayed on the monitor screen, so that the user can more clearly and comprehensively observe and analyze the current surgical situation.

[0078] When the user needs to re-control the robotic arm of the patient operating platform to continue the surgical operation, the user can step on the control pedal of the doctor console again to initiate a recovery trigger action. When the doctor console detects that the user has initiated this recovery trigger action, it can automatically switch back to the robotic arm control mode. In the robotic arm control mode, the communication connection between the left control arm, the right control arm, the control joint of the doctor console and the robotic arm of the patient operating platform will be restored. At this time, the user can again control the robotic arm of the patient operating platform to continue the surgical operation by manipulating the left control arm, the right control arm, and the control joint.

[0079] It should be noted that the robot connected to the above robot control terminal can be not only a medical robot applied to the medical scenario, but also a production robot applied to the production line scenario, such as an automobile production robot, a mobile phone production robot, a laptop production robot, and so on. It should be noted that the above-listed robot devices are only illustrative. Specifically, in practice, according to the specific situation and processing needs, the image processing method of the robot control terminal provided in this specification can also be applied to other application scenarios. This specification does not make any limitations in this regard.

[0080] In some embodiments, after receiving the first operation data for the endoscopic image in the image interface, when the method is specifically implemented, the following may further be included:

[0081] S1: According to the first operation data, correspondingly adjust the size and / or position of the endoscopic image to obtain an adjusted endoscopic image;

[0082] S2: Display the adjusted endoscopic image in the image interface.

[0083] Wherein, the above first operation data is specifically the operation data for the endoscopic image initiated by the user by using the control arm (or main hand) and / or control joint of the main control terminal when the control terminal switches to the image adjustment mode.

[0084] Specifically in implementation, the control terminal can collect and, according to the first operation data, parse to obtain a corresponding adjustment instruction; and then, according to the corresponding adjustment instruction, locally adjust the size and / or position of the endoscopic image to obtain an adjusted endoscopic image.

[0085] The control terminal can also send the collected first operation data to the image processing device, and the image processing device correspondingly adjusts the size and / or position of the endoscopic image according to the first operation data; the control terminal receives the adjusted endoscopic image returned by the image processing device.

[0086] In some cases, the above first operation data may further include a content adjustment operation for the endoscopic image (for example, an operation indicating to display the previous frame of the endoscopic image). Correspondingly, specifically in implementation, the image content of the endoscopic image can also be correspondingly adjusted according to the first operation data (for example, using the previous frame of the endoscopic image to replace the current endoscopic image).

[0087] In some embodiments, the above receiving the first operation data for the endoscopic image in the image interface, specifically in implementation, may include the following:

[0088] S1: When it is detected that the user initiates a switching trigger action for the control pedal of the control terminal, switch the control terminal to the image adjustment mode;

[0089] S2: In the image adjustment mode, by collecting the action data of the user for the control arm and / or control joint of the control terminal, obtain the first operation data for the endoscopic image.

[0090] Taking the control terminal as the doctor's console as an example, the control terminal may include a control pedal, and a pressure sensor may also be provided on the control pedal. Further, the control arm of the control terminal may include a left control arm and a right control arm; and one or more control joints may be respectively provided on the left control arm and the right control arm.

[0091] In specific implementation, when the control terminal monitors that the user steps on the control pedal through the above pressure sensor, it can determine that the user has initiated a switching trigger action; and then it can generate a corresponding switching instruction, and switch the control terminal to the image adjustment mode based on this switching instruction.

[0092] In the image adjustment mode, the user can initiate first operation data for the endoscope image by performing actions such as moving the control arm and / or pinching and releasing the control joints according to the voice guidance prompt played by the control terminal.

[0093] In some embodiments, after switching the control terminal to the image adjustment mode, refer to Figure 6 As shown, when the method is specifically implemented, the following content may further be included:

[0094] S1: Obtain the current position parameter of the image interface and the current position parameter of the control arm;

[0095] S2: According to the preset coordinate system mapping relationship, use the current position parameter of the image interface and the current position parameter of the control arm to determine the position parameter of the identification icon associated with the control arm based on the image interface;

[0096] S3: Display the identification icon associated with the control arm in the image interface according to the position parameter of the identification icon based on the image interface.

[0097] Among them, the above preset coordinate system mapping relationship may specifically be the corresponding relationship between the image interface coordinate system and the control arm coordinate system determined in advance through calibration.

[0098] The above identification icon may specifically be understood as the identification graphic corresponding to the control arm displayed in the image interface. Specifically, when the user moves the control arm, the identification icon corresponding to the control arm will also move correspondingly in the image interface as the control arm moves.

[0099] Taking the control terminal as the doctor's console as an example. The control arm may include two control arms, a left control arm and a right control arm. Correspondingly, refer to Figure 7 As shown, in the image adjustment mode, the identification icons displayed in the image interface may include: a left identification icon associated with the left control arm, and a right identification icon associated with the right control arm.

[0100] During specific implementation, after switching to the image adjustment mode, the control terminal can obtain the current position parameters of the image interface and the current position data of the control arm through corresponding sensors or encoders; then, based on the preset coordinate system mapping relationship, using the current position parameters of the image interface and the current position parameters of the control arm, calculate the position parameters of the identification icon associated with the control arm based on the image interface (i.e., based on the image interface coordinate system); then, according to the position parameters of the identification icon based on the image interface, display the identification icon associated with the control arm at the corresponding position in the image interface to obtain an initialized image interface in the image adjustment mode. Furthermore, the user can initiate first operation data by operating the control arm and / or control joints based on the above-mentioned initialized image interface to perform corresponding adjustments on the endoscopic image in the initialized image interface.

[0101] In some embodiments, after switching the control terminal to the image adjustment mode, when the method is specifically implemented, it may further include the following: obtaining the current position parameters of the image interface and the current position parameters of the control arm; generating corresponding control instructions according to the preset initialization position parameters, the current position parameters of the image interface, and the current position parameters of the control arm; and using the control instructions to automatically move the control arm to the preset initial position, and at the same time, display the identification icon corresponding to the control arm currently located at the preset initial position at the preset initial position (for example, the 0-point position) in the image interface to obtain another initialized image interface in the image adjustment mode.

[0102] In some embodiments, the first operation data may specifically include an action sequence for the control arm and / or control joints; where the action sequence specifically includes: a moving action for the control arm and / or a pinching and releasing action for the control joints, etc.

[0103] Of course, it should be noted that the above-listed actions are only illustrative. According to the specific structure and processing requirements of the control arm, the above action sequence may further include other types of actions. In this regard, this specification does not make any limitations.

[0104] In some embodiments, when specifically implementing the corresponding adjustment of the size and / or position of the endoscopic image according to the first operation data, it may include the following:

[0105] S1: Move the identification icon to the control point position of the endoscopic image according to the first moving action in the first operation data;

[0106] S2: Lock the control point according to the second pinching and releasing action in the first operation data;

[0107] S3: According to the third movement action in the first operation data, translate and / or scale the endoscopic image based on this control point.

[0108] During specific implementation, corresponding control points can also be set on the displayed endoscopic image. For details, please refer to Figure 8 as shown. Specifically, control points can be set at the midpoints of the long sides, the midpoints of the wide sides, and the diagonal points of the endoscopic image respectively. When the user moves the control arm and moves the identification icon associated with the control arm to the control point or within the adjacent range area of this control point, this control point will light up (or display a different color). At this time, the user can pinch the control joint on this control arm to lock out this control point. Furthermore, the user can move the control arm to drive the entire side to which the locked control point belongs to move accordingly, so as to perform translation and / or scaling operations on the endoscopic image and achieve adjustment of the position and / or size of the endoscopic image. After completing the adjustment of the endoscopic image, the user can release the previously pinched control joint and initiate the fourth pinch-and-release action to end the adjustment of the endoscopic image.

[0109] Specifically, for example, refer to Figure 8 as shown. The user can move the left control arm and the right control arm respectively to initiate the first movement action, and move the left identification icon and the right identification icon to the control point 1 at the midpoint of the left wide side and the control point 2 at the midpoint of the right wide side respectively; then pinch the left control joint on the left control arm and the right control joint on the right control arm simultaneously to initiate the second pinch-and-release action to lock the control point 1 and the control point 2 respectively; then, on the premise of pinching the left control joint and the right control joint, move the left control arm and the right control arm inward and towards each other simultaneously to initiate the third movement action to perform a reduction operation on the endoscopic image in the horizontal direction; of course, the user can also not move the left control arm and only move the right control arm inward to initiate the third movement action to reduce the endoscope. When the endoscopic image is reduced to meet the user's requirements, the user can release the previously pinched left control joint and right control joint and initiate the fourth pinch-and-release action to end the adjustment of this endoscopic image. At this time, an adjusted endoscopic image that is reduced relative to the original endoscopic image will be displayed in the image interface.

[0110] After determining that the adjustment of the endoscopic image is completed, the user can step on the control pedal again to trigger the control terminal to automatically exit the image adjustment mode.

[0111] In some embodiments, the above-mentioned determining the form adjustment data for the target image according to the first operation data may specifically include the following content during specific implementation:

[0112] S1: According to the first operation data, determine the position information and size information of the adjusted endoscopic image;

[0113] S2: Determine the blank area in the image interface according to the position information and size information of the adjusted endoscope image;

[0114] S3: Determine the form adjustment data for the target image according to the blank area in the image interface.

[0115] Among them, the form adjustment data of the target image may specifically include the adjustment data for the position information of the target image and / or the adjustment data for the size information of the target image. The adjustment data for the above size information may further include the adjustment data for the long side of the target image and / or the adjustment data for the wide side of the target image.

[0116] In specific implementation, the robot control terminal may first adjust the endoscope image according to the first operation data to obtain the adjusted endoscope image; after obtaining the adjusted endoscope image, determine the blank area in the image interface; and then determine the form adjustment data for the target image according to the blank area. The robot control terminal may also predict the blank area in the image interface according to the first operation data and the size information of the image interface before obtaining the adjusted endoscope image; and then determine the corresponding form adjustment data according to the predicted blank area.

[0117] In some embodiments, specifically, for example, reference may be made to Figure 12 As shown, the size information of the image interface includes: the length of the long side of the image interface (denoted as Lx) and the length of the wide side (denoted as Ly). The size information of the adjusted endoscope includes: the length of the long side of the adjusted endoscope image (denoted as Lex) and the length of the wide side (denoted as Ley). The size information of the adjusted target image includes: the length of the long side of the adjusted target image (denoted as Lsx) and the length of the wide side (denoted as Lsy). To ensure the user's interaction experience, a minimum value for the size information of the target image is also preset, including the minimum value of the long side length (denoted as Lsx org ) and the minimum value of the wide side length (denoted as Lsy ory ). Furthermore, according to the minimum value of the long side length, the minimum value of the wide side length, the length of the long side and the length of the wide side of the image interface, and the minimum value of the long side length and the minimum value of the width length, different situations can be distinguished, and the appropriate long side length and wide side length of the adjusted target image corresponding to different situations can be calculated respectively.

[0118] Specifically, different situations can be distinguished according to the following formulas to determine the appropriate long side length and wide side length of the adjusted target image:

[0119]

[0120] According to the appropriate long side length and short side length of the adjusted target image described above, adjustment data for the size information of the target image can be obtained.

[0121] Furthermore, by combining the long side length, short side length, and position information of the adjusted endoscopic image, as well as the long side length and short side length of the image interface, a blank area can be determined in the image interface. Then, based on the long side length and short side length of the adjusted target image, size information matching is performed in the blank area to determine the position coordinates suitable for placing the adjusted target image in the blank area, so that adjustment data for the position information of the target image can be obtained.

[0122] In addition, taking the user's interaction experience with the adjusted target image as a constraint, based on the adjusted endoscopic image, the largest area image region that can satisfy the user's interaction experience can be further found in the determined blank area in the image interface as the effective blank area; then, based on this effective blank area, form adjustment data for the target image is further determined.

[0123] Specifically, for example, the area of the image interface can be denoted as: S0 = Lx * Ly, the area of the adjusted endoscopic image can be denoted as: S1 = Lex * Ley, and the area of the adjusted target image can be denoted as: S2 = Lsx * Lsy. Among them, Lx, Ly, Lex, and Ley are known.

[0124] Furthermore, to ensure the user's interaction experience, there are the following constraint conditions: it is required that the long side and short side of the adjusted target image satisfy a preset aspect ratio, for example, 4:3, that is, Lsx:Lsy = 4:3; and, the area of the adjusted target image is greater than the minimum value of the preset image area, for example, That is,

[0125] Furthermore, based on the above constraint conditions, according to the minimum value of the long side length, the minimum value of the short side length, the long side length and short side length of the image interface, different situations are distinguished, and the area of the corresponding effective blank area is determined and calculated for different situations respectively.

[0126] Specifically, reference can be made to Figure 17 and Figure 18 As shown, two situations can be distinguished as: Situation 1, Situation 2, The area of the corresponding effective blank area is calculated for the above two situations respectively.

[0127] Specifically, the area of the effective blank area in the above two situations can be calculated according to the following formulas respectively:

[0128] When When

[0129] When When

[0130] Furthermore, according to the area of the effective blank area and in combination with relevant constraint conditions, the appropriate long side length, wide side length and specific position of the adjusted target image can be finally determined.

[0131] In some embodiments, as shown in Figure 9 The content adjustment data for the target image determined by feature matching according to the endoscopic image may specifically include the following when implemented:

[0132] S1: Extract the edge contour features from the endoscopic image;

[0133] S2: Perform feature matching in a preset image data source according to the edge contour features to obtain corresponding feature matching results;

[0134] S3: According to the feature matching results, extract the target image data with a matching degree greater than a preset matching degree threshold from the preset image data source as the content adjustment data for the target image.

[0135] When specifically implemented, a morphological edge extraction algorithm can be used to process the endoscopic image to extract the morphological gradients responded by different structure elements of the edge contour in the endoscopic image; then combine the morphological gradients of multiple different structure elements (for example, perform weighted calculation on the morphological gradients responded by multiple structure elements) to obtain the edge contour features of the endoscopic image.

[0136] Specifically, for example, the morphological gradients of multiple different structure elements can be combined according to the following formula to obtain the edge contour features of the endoscopic image: IGrad[A]=c1IGrad1[A]+c2IGrad2[A]……. Where IGrad[A] is the edge contour feature of the endoscopic image, IGrad1[A] and IGrad2[A] are the morphological gradients of the structure elements numbered 1 and 2 in the edge contour of the endoscopic image respectively, and c1 and c2 are the corresponding weight coefficients.

[0137] When specifically implemented, a neural network model capable of automatically extracting image edge contour features can also be trained in advance with a large number of sample images; then input the endoscopic image into the neural network model, and process the input endoscopic image by running the neural network model to automatically extract and output the corresponding edge contour features.

[0138] In specific implementation, according to the edge contour features of the endoscope image, feature matching can be performed in a preset image data source to find content adjustment data for the target image that matches the endoscope image.

[0139] In some embodiments, after performing feature matching in a preset image data source according to the edge contour features to obtain corresponding feature matching results, when the method is specifically implemented, the following content may further be included:

[0140] S1: When it is determined according to the feature matching results that no target image data with a matching degree greater than a preset matching degree threshold can be extracted from the preset image data source, determine the current endoscope viewing angle according to the endoscope image;

[0141] S2: Adjust the preset image data source according to the current endoscope viewing angle to obtain an adjusted image data source;

[0142] S3: Perform feature matching again in the adjusted image data source according to the edge contour features to obtain corresponding feature matching results.

[0143] When specifically implementing, when it is impossible to directly find matching target image data from the preset image data source, the viewing angle used in the preset image data source can be adjusted according to the current endoscope viewing angle; then, based on the adjusted viewing angle, feature matching is performed again in the preset image data source according to the edge contour features to find matching target image data from the preset image data source.

[0144] When specifically implementing, when it is impossible to directly find matching target image data from the preset image data source, the image data with the highest matching degree with the edge contour features of the endoscope image can also be determined as approximate image data from the preset image data source based on the current viewing angle according to the feature matching results; then, according to the difference between the approximate image data and the contour features of the endoscope image, combined with the endoscope viewing angle, an adjustment plan for the current viewing angle is generated. Furthermore, a viewing angle adjustment prompt can be generated according to this adjustment plan and presented to the user to prompt the user to adjust the viewing angle for the preset image data source more accurately. In addition, the viewing angle of the preset image data source can also be automatically adjusted according to this adjustment plan.

[0145] In some embodiments, after extracting image data with a matching degree greater than a preset matching degree threshold from the preset image data source, when the method is specifically implemented, the following content may further be included:

[0146] Adjust the data according to the form, and intercept the image data within the preset regional range containing the target image data from the preset image data source as the content adjustment data for the target image.

[0147] In specific implementation, it is possible to determine whether the regional range included in the adjusted target image is larger than the regional range included in the target image data according to the form adjustment data for the target image; in the case where it is determined that the regional range included in the adjusted target image is larger than the regional range included in the target image data, it is possible to intercept, according to the corresponding interception rules, the image data within the preset regional range that contains the target image data and matches the regional range included in the adjusted target image data from the preset image data source as the content adjustment data for the target image.

[0148] In some embodiments, after generating the adjusted target image according to the form adjustment data and / or the content adjustment data, refer to Figure 10 As shown, when the method is specifically implemented, the following content may further be included:

[0149] S1: Determine the instrument type of the operating instrument in the endoscopic image, and obtain the position parameters of the operating instrument based on the endoscopic image;

[0150] S2: Determine the relative pose of the virtual instrument based on the landmark points in the adjusted target image according to the position parameters of the operating instrument; query the preset instrument model database according to the instrument type of the operating instrument to obtain the model data of the virtual instrument;

[0151] S3: Determine the display position of the virtual instrument in the adjusted target image according to the relative pose of the virtual instrument;

[0152] S4: Display the virtual instrument corresponding to the operating instrument at the display position in the adjusted target image according to the model data of the virtual instrument.

[0153] Specifically, since the endoscopic image is a real image collected in real time or at regular intervals by an endoscope or other devices, the endoscopic image may also include the specific operating instrument (for example, a surgical instrument) used. However, since the target image is a virtual image generated based on a preset image data source. Therefore, the above-mentioned operating instrument often does not appear in the directly generated target image. Thus, it will interfere with the user's analysis and decision-making.

[0154] In specific implementation, the user can also initiate a virtual enhancement instruction through the operation control terminal. The control terminal can respond to the virtual enhancement instruction, perform image recognition on the endoscopic image to determine whether there is an operating instrument in the endoscopic image; in the case of determining that there is an operating instrument, a model of the corresponding operating instrument, that is, a virtual instrument, can be further added at the corresponding position in the adjusted target image, so that the adjusted target image is closer to the real scene, and a virtual enhanced target image is obtained.

[0155] In specific implementation, refer to Figure 11 As shown, taking the operating instrument Ot1 in the endoscopic image as an example. The control terminal can use a preset mechanical recognition model to process the endoscopic image to detect and identify the instrument type of the operating instrument in the endoscopic image; at the same time, determine the position parameters of the operating instrument based on the endoscopic image. Among them, the position parameters of the operating instrument based on the endoscopic image can specifically be the position coordinates of the key points (such as the center point, edge contour points, etc.) of the operating instrument based on the coordinate system of the endoscopic image. For example, it can be recorded as (p1x, p1y).

[0156] Then, the center point of the endoscopic image can be selected as the landmark point (or focus, which can be recorded as O0), and the relative pose of the operating instrument based on the endoscopic image relative to the landmark point is determined, expressed in the following form: Among them, the coordinate system based on the landmark point in the endoscopic image can be recorded as {O0}; the coordinate system based on the operating instrument in the endoscopic image can be recorded as {Ot1}.

[0157] Next, the pose of the landmark point (which can be recorded as O1) in the adjusted target image relative to the landmark point in the endoscopic image can be calculated, which can be recorded as

[0158] Assume that the position coordinates of the key points of the virtual instrument ({Ot1`}) corresponding to the operating instrument based on the coordinate system of the adjusted target image are recorded as (px, py). Then, the relative pose of the virtual instrument relative to the landmark point ({O1}) in the adjusted target image can be expressed in the following form: Then there is

[0159] By jointly solving the relative poses of the operating instrument in the endoscopic image, the position coordinates of the key points of the virtual instrument based on the coordinate system of the adjusted target image can be calculated, that is, the following results are obtained:

[0160]

[0161] And, if p1x = 0.

[0162] Wherein, k is the scaling factor, which can be determined based on the ratio of the adjusted target image size to the original target image size.

[0163] In this way, the position coordinates of the key points of the virtual device based on the coordinate system of the adjusted target image can be calculated as the display position of the virtual device based on the adjusted target image.

[0164] At the same time, the preset instrument model database can be queried according to the instrument type of the operating instrument to obtain the model data of the virtual instrument.

[0165] The preset instrument model database may store a plurality of preset instrument model data, each preset instrument model data corresponding to an instrument type. Each preset instrument model data includes at least one key point and the structural relationship of the instrument contour relative to the key point.

[0166] Correspondingly, after obtaining the model data of the virtual instrument, the key points of the virtual instrument can be marked at the display position in the adjusted target image, and then the corresponding instrument outline can be drawn based on the key points of the virtual instrument according to the structural relationship between the instrument outline in the model data of the virtual instrument and the key points. Thus, the corresponding virtual instrument can be displayed at the display position corresponding to the real operating instrument in the adjusted target image.

[0167] In some embodiments, after the adjusted target image is displayed in the image interface, the method may further include the following contents when implemented:

[0168] S1: monitor and obtain the change data of the adjusted endoscopic image;

[0169] S2: According to the change data of the adjusted endoscopic image, the displayed adjusted target image is adjusted in linkage.

[0170] During specific implementation, the user can also initiate a tracking instruction by operating the control terminal. The control terminal can respond to the tracking instruction and detect in real time or periodically whether the adjusted endoscopic image has changed; if it is determined that a change has occurred, the changed image point and the corresponding change data are determined; then, according to the preset coordinate system mapping relationship, the corresponding image point is determined in the adjusted target image, and according to the change data, the image point is modified accordingly, so that the change data of the adjusted endoscopic image can be tracked, and the displayed adjusted target image can be adjusted in a linked manner, so that the user can obtain a relatively better user experience.

[0171] In some embodiments, after the adjusted endoscopic image is displayed in the image interface, the method may further include the following contents when implemented:

[0172] S1: Detect whether the area of the current blank area in the image interface is less than or equal to the threshold area;

[0173] S2: When it is determined that the area of the current blank area is less than or equal to the threshold area, adjust the data according to the content to generate an adjusted target image;

[0174] S3: According to the preset layout rules, display the adjusted target image in the endoscopic image of the image interface in an embedded manner.

[0175] In specific implementation, the above threshold area can be set to 1 / 16 of the area of the image interface. Of course, the above-listed threshold area is only an illustrative description. In specific implementation, according to the specific situation and processing requirements, the threshold area can also be set to other values. This specification does not make any limitations in this regard.

[0176] In specific implementation, refer to Figure 12 As shown, in the normal default state, the default area of the target image can be the threshold area. Correspondingly, the target image can be displayed to the user in an embedded manner at the corner position of the endoscopic image.

[0177] In specific implementation, when the area of the current blank area other than the adjusted endoscopic image in the image interface is less than or equal to the threshold area, the size and / or position of the target image may not be adjusted. However, according to the endoscopic image, the content adjustment data for the target image can be determined through feature matching; then, according to the content adjustment data, an adjusted target image is generated. Finally, the adjusted target image is displayed in the endoscopic image of the image interface in an embedded manner.

[0178] In some embodiments, after detecting whether the area of the current blank area in the image interface is less than or equal to the threshold area, when the method is specifically implemented, the following content may further be included:

[0179] S1: When it is determined that the area of the current blank area is greater than the threshold area, generate an adjusted target image according to the form adjustment data and / or the content adjustment data;

[0180] S2: According to the preset layout rules, display the adjusted target image in the blank area of the image interface in a corresponding other display manner.

[0181] Among them, refer to Figure 12 As shown, the above corresponding other display manners include at least one of the following: vertical side-by-side display, horizontal side-by-side display, diagonal display, etc.

[0182] During specific implementation, the display method that can be adapted to the adjusted endoscope image and the adjusted target image simultaneously can be determined based on the size parameters of the image interface, the size parameters of the adjusted endoscope image, and the data for adjusting the form of the target image.

[0183] Before specific implementation, a large amount of historical image interface evaluation data of users for the robot control terminal can be collected; then, based on the evaluation data, the historical image interfaces with image layouts meeting the preferences of most users can be selected as sample image interfaces; then, according to the display method, the sample image interfaces can be divided into multiple sample groups (for example, divided into three sample groups, corresponding to top-bottom side-by-side display, left-right side-by-side display, diagonal display, etc.); then, the size parameters of the image interfaces of the sample image interfaces in each sample group, the size parameters of the adjusted endoscope image, and the data for adjusting the form of the target image (size adjustment data) are respectively statistically analyzed to find out the size parameter interval with the relatively highest proportion, which is used as the preset reference size parameter interval corresponding to each sample group. Among them, each preset reference size parameter interval corresponds to a display method.

[0184] In this way, when determining the display method, the size parameters of the image interface, the size parameters of the adjusted endoscope image, and the data for adjusting the form of the target image can be detected to find out which preset reference size parameter interval they hit; furthermore, the display method corresponding to the preset reference size parameter interval that is hit can be determined as the display method that can be adapted to the adjusted endoscope image and the adjusted target image simultaneously.

[0185] During specific implementation, when determining the display method, the robot control terminal can also receive the custom parameters input by the user, determine the display method that matches the custom parameters of the user from multiple adaptable display methods; then, based on this display method, the adjusted endoscope image and the adjusted target image are displayed in the image interface to meet the personalized needs of the user.

[0186] In some embodiments, after the adjusted endoscope image and the adjusted target image are displayed in the image interface, the user can also initiate an image exchange instruction by operating the control terminal. Correspondingly, referring to Figure 13 As shown, the control terminal can respond to this exchange instruction and perform an image exchange on the image content displayed by the endoscope image and the target image in the current image interface to meet the personalized usage needs of the user.

[0187] In some embodiments, after the control terminal switches to the image adjustment mode, the control terminal can monitor the code value generated by the encoder of the mechanical arm of the medical robot (for example, the patient surgical platform) in real time; and judge whether the mechanical arm has abnormal movement due to misoperation or other reasons in the image adjustment mode according to the code value. In the case where it is determined that the mechanical arm has abnormal movement according to the code value, the control terminal can play the abnormal risk prompt to the user through the voice player, and / or display the abnormal risk prompt to the user through the display screen, so as to prompt the user in time. Thereby, the operation safety in the image adjustment mode can be effectively ensured.

[0188] In some embodiments, after determining the form adjustment data for the target image according to the first operation data, the method may further include the following: predicting whether the content data of the target image needs to be adjusted according to the size adjustment data in the form adjustment data for the target image and / or the content data of the adjusted endoscopic image. If it is determined that the content data of the target image does not need to be adjusted, feature matching may be skipped, and the adjusted target image may be generated directly according to the form adjustment data.

[0189] In a specific implementation, according to the size adjustment data in the form adjustment data for the target image and / or the content data of the adjusted endoscopic image, when it is detected that the size adjustment data of the target image is zero, and / or when it is detected that the difference between the content data of the adjusted endoscopic image and the content data of the original endoscopic image is greater than a preset difference threshold, it is determined that the content data of the target image needs to be adjusted.

[0190] As can be seen from the above, based on the image processing method of the robot control terminal provided in the embodiment of this specification, when the robot control terminal receives the first operation data of the user for the endoscopic image in the image interface and responds to the first operation data, while displaying the adjusted endoscopic image in the image interface, it can determine the form adjustment data for the target image according to the first operation data; and determine the content adjustment data for the target image through feature matching based on the endoscopic image; and then synchronously display the corresponding adjusted target image in the image interface according to the form adjustment data and / or the content adjustment data. Therefore, while the user adjusts the endoscopic image, the target image displayed simultaneously with the endoscopic image in the same image interface can be intelligently and automatically adjusted in a more matching adaptive manner, which effectively simplifies the user operation and enables the user to obtain a better interactive experience.

[0191] This manual also provides another image processing method for a robot control terminal, which may include the following contents during specific implementation:

[0192] S1: Receive first operation data for a first image in an image interface; wherein, the target image is also displayed in the image interface; the target image is associated with the first image;

[0193] S2: Determine form adjustment data for the target image according to the first operation data; wherein, the form adjustment data at least includes position adjustment data and / or size adjustment data;

[0194] S3: Determine content adjustment data for the target image through feature matching according to the first image;

[0195] S4: Generate an adjusted target image according to the form adjustment data and / or the content adjustment data; and display the adjusted target image in the image interface.

[0196] Among them, the above image processing method of the robot control terminal can be specifically used on the side of the robot control terminal. The above robot control terminal is connected to a robot (such as a medical robot, etc.) and is used to control the operation of the robot in the operation area.

[0197] Among them, the above first image can specifically be an image of a different type from the target image. Specifically, the above first image can be an endoscopic image or a virtual image generated based on a preset image data source. For example, the above first image can be an ultrasound image generated based on a preset ultrasound image data source, and the above target image can be a nuclear magnetic resonance image generated based on a preset nuclear magnetic resonance image data source. The above first image and target image can be images related to the operation area of the robot.

[0198] Based on the above embodiments, when the robot control terminal monitors that the user adjusts the first image, it can intelligently and automatically adjust the target image displayed simultaneously in the same image interface as the first image in a relatively matching and adaptive manner, thereby effectively simplifying the user operation and enabling the user to obtain a better interaction experience.

[0199] The embodiments of this specification also provide a computer device, as shown in Figure 14 shown. Among them, the computer device includes a network communication port 1401, a processor 1402, and a memory 1403. The above structures are connected by internal cables so that each structure can perform specific data interactions.

[0200] Among them, the network communication port 1401 can specifically be used to receive first operation data for an endoscopic image in an image interface; wherein, the target image is also displayed in the image interface; the target image is a virtual image associated with the endoscopic image.

[0201] The processor 1402 can be specifically configured to determine form adjustment data for a target image according to first operation data; wherein, the form adjustment data at least includes position adjustment data and / or size adjustment data; determine content adjustment data for the target image through feature matching according to the endoscopic image; generate an adjusted target image according to the form adjustment data and / or the content adjustment data; and display the adjusted target image in the image interface.

[0202] The memory 1403 can be specifically configured to store corresponding instruction programs.

[0203] In this embodiment, the network communication port 1401 can be bound to different communication protocols, so as to send or receive different data virtual ports. For example, the network communication port can be a port responsible for web data communication, can also be a port responsible for FTP data communication, and can also be a port responsible for mail data communication. In addition, the network communication port can also be a physical communication interface or communication chip. For example, it can be a wireless mobile network communication chip, such as GSM, CDMA, etc.; it can also be a Wifi chip; it can also be a Bluetooth chip.

[0204] In this embodiment, the processor 1402 can be implemented in any suitable manner. For example, the processor can take the form of, for example, a microprocessor or a processor and a computer-readable medium storing computer-readable program code (such as software or firmware) executable by the (micro)processor, logic gates, switches, application specific integrated circuits (ASICs), programmable logic controllers, and embedded microcontrollers, etc. This specification does not make any limitations.

[0205] In this embodiment, the memory 1403 can include multiple levels. In a digital system, anything that can store binary data can be a memory; in an integrated circuit, a circuit without a physical form but with a storage function is also called a memory, such as RAM, FIFO, etc.; in a system, a storage device with a physical form is also called a memory, such as a memory stick, a TF card, etc.

[0206] The embodiment of this specification also provides a computer-readable storage medium for an image processing method based on the above robot control terminal. The computer-readable storage medium stores computer program instructions, and when the computer program instructions are executed, the following steps are implemented: receiving first operation data for an endoscope image in an image interface; wherein, a target image is further displayed in the image interface; the target image is a virtual image associated with the endoscope image; determining form adjustment data for the target image according to the first operation data; wherein, the form adjustment data at least includes position adjustment data and / or size adjustment data; determining content adjustment data for the target image through feature matching according to the endoscope image; generating an adjusted target image according to the form adjustment data and / or the content adjustment data; and displaying the adjusted target image in the image interface.

[0207] In this embodiment, the above storage medium includes but is not limited to Random Access Memory (RAM), Read-Only Memory (ROM), Cache, Hard Disk Drive (HDD), or Memory Card. The memory can be used to store computer program instructions. The network communication unit can be set according to the standards stipulated by the communication protocol and is used for the interface of network connection communication.

[0208] In this embodiment, the functions and effects specifically implemented by the program instructions stored in the computer-readable storage medium can be explained by comparison with other embodiments and will not be elaborated here.

[0209] Refer to Figure 15 As shown, at the software level, the embodiment of this specification also provides an image processing device for a robot control terminal. The device can specifically include the following structural modules:

[0210] A receiving module 1501 can specifically be used to receive first operation data for an endoscope image in an image interface; wherein, a target image is further displayed in the image interface; the target image is a virtual image associated with the endoscope image;

[0211] A first determination module 1502 can specifically be used to determine form adjustment data for the target image according to the first operation data; wherein, the form adjustment data at least includes position adjustment data and / or size adjustment data;

[0212] A second determination module 1503 can specifically be used to determine content adjustment data for the target image through feature matching according to the endoscope image;

[0213] The processing module 1504 can be specifically used to generate an adjusted target image according to the form adjustment data and / or the content adjustment data; and display the adjusted target image in the image interface.

[0214] In some embodiments, after receiving the first operation data for the endoscopic image in the image interface, when the device is specifically implemented, it can also be used to: adjust the size and / or position of the endoscopic image according to the first operation data to obtain an adjusted endoscopic image; and display the adjusted endoscopic image in the image interface.

[0215] In some embodiments, when the receiving module 1501 is specifically implemented, it can receive the first operation data for the endoscopic image in the image interface in the following manner: when it detects that the user initiates a switching trigger action on the control pedal of the control terminal, switch the control terminal to the image adjustment mode; in the image adjustment mode, obtain the first operation data for the endoscopic image by collecting the action data of the user on the control arm and / or control joint of the control terminal.

[0216] In some embodiments, after switching the control terminal to the image adjustment mode, when the device is specifically implemented, it can also be used to: obtain the current position parameter of the image interface and the current position parameter of the control arm; according to the preset coordinate system mapping relationship, use the current position parameter of the image interface and the current position parameter of the control arm to determine the position parameter of the identification icon associated with the control arm based on the image interface; and display the identification icon associated with the control arm in the image interface according to the position parameter of the identification icon based on the image interface.

[0217] In some embodiments, the first operation data can specifically include an action sequence for the control arm and / or control joint; wherein, the action sequence includes: a moving action for the control arm and / or a pinching and releasing action for the control joint.

[0218] In some embodiments, when the device is specifically implemented, it can adjust the size and / or position of the endoscopic image according to the first operation data in the following manner: move the identification icon to the control point position of the endoscopic image according to the first moving action in the first operation data; lock the control point according to the second pinching and releasing action in the first operation data; and perform translation and / or scaling of the endoscopic image based on the control point according to the third moving action in the first operation data.

[0219] In some embodiments, when the above-mentioned first determination module 1502 is specifically implemented, the form adjustment data for the target image can be determined according to the following method: according to the first operation data, determine the position information and size information of the adjusted endoscopic image; according to the position information and size information of the adjusted endoscopic image, determine the blank area in the image interface; according to the blank area in the image interface, determine the form adjustment data for the target image.

[0220] In some embodiments, when the above-mentioned second determination module 1503 is specifically implemented, the content adjustment data for the target image can be determined through feature matching according to the endoscopic image in the following way: extract the edge contour features from the endoscopic image; perform feature matching in the preset image data source according to the edge contour features to obtain the corresponding feature matching result; according to the feature matching result, extract the target image data with a matching degree greater than the preset matching degree threshold from the preset image data source as the content adjustment data for the target image.

[0221] In some embodiments, after performing feature matching in the preset image data source according to the edge contour features to obtain the corresponding feature matching result, when the device is specifically implemented, it can also be used for: in the case where it is determined according to the feature matching result that no target image data with a matching degree greater than the preset matching degree threshold can be extracted from the preset image data source, determine the current endoscopic view according to the endoscopic image; adjust the preset image data source according to the current endoscopic view to obtain the adjusted image data source; perform feature matching again in the adjusted image data source according to the edge contour features to obtain the corresponding feature matching result.

[0222] In some embodiments, after generating the adjusted target image according to the form adjustment data and / or the content adjustment data, when the device is specifically implemented, it can also be used for: determine the instrument type of the operating instrument in the endoscopic image, and obtain the position parameter of the operating instrument based on the endoscopic image; according to the position parameter of the operating instrument, determine the relative pose of the virtual instrument based on the landmark points in the adjusted target image; according to the instrument type of the operating instrument, query the preset instrument model database to obtain the model data of the virtual instrument; according to the relative pose of the virtual instrument, determine the display position of the virtual instrument in the adjusted target image; according to the model data of the virtual instrument, display the virtual instrument corresponding to the operating instrument at the display position in the adjusted target image.

[0223] In some embodiments, after the adjusted target image is displayed in the image interface, the device can also be used to: monitor and obtain the change data of the adjusted endoscopic image; and adjust the displayed adjusted target image in a linked manner according to the change data of the adjusted endoscopic image.

[0224] In some embodiments, after the adjusted endoscopic image is displayed in the image interface, the device can also be used to: detect whether the area of ​​the current blank area in the image interface is less than or equal to the threshold area; if it is determined that the area of ​​the current blank area is less than or equal to the threshold area, adjust the data according to the content to generate an adjusted target image; and display the adjusted target image in the endoscopic image of the image interface in an embedded manner according to a preset layout rule.

[0225] In some embodiments, after detecting whether the area of ​​the current blank area in the image interface is less than or equal to the threshold area, the device can also be used to: generate an adjusted target image according to the form adjustment data and / or the content adjustment data when it is determined that the area of ​​the current blank area is greater than the threshold area; and display the adjusted target image in the blank area of ​​the image interface in other corresponding display modes according to preset layout rules.

[0226] It should be noted that the units, devices or modules described in the above embodiments can be implemented by computer chips or entities, or by products with certain functions. For the convenience of description, the above devices are described separately by dividing them into various modules according to their functions. Of course, when implementing this specification, the functions of each module can be implemented in the same or more software and / or hardware, or the modules that implement the same function can be implemented by a combination of multiple sub-modules or sub-units. The device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0227] From the above, it can be seen that the image processing device of the robot control terminal provided in the embodiment of this specification can automatically adjust the target image displayed simultaneously with the endoscopic image in the same image interface in an intelligent and adaptive manner while the user adjusts the endoscopic image, which effectively simplifies the user operation and enables the user to obtain a better interactive experience.

[0228] SeeFigure 16 As shown, this specification also provides a robot control terminal, which at least includes a display screen 1601, a control arm 1602, and a processor 1603; among them, the processor 1603 is electrically connected to the display screen 1601 and the control arm 1602 respectively;

[0229] The display screen 1601 is used to display an image interface; the image interface simultaneously displays an endoscope image and a target image; the target image is a virtual image associated with the endoscope image;

[0230] The control arm 1602 is used to receive first operation data for the endoscope image in the image interface;

[0231] The processor 1603 is used to correspondingly adjust the size and / or position of the endoscope image according to the first operation data to obtain an adjusted endoscope image; the processor 1603 is also used to determine form adjustment data for the target image according to the first operation data; wherein, the form adjustment data at least includes position adjustment data and / or size adjustment data; determine content adjustment data for the target image through feature matching according to the endoscope image; generate an adjusted target image according to the form adjustment data and / or the content adjustment data;

[0232] The processor 1603 can send the above-mentioned adjusted endoscope image and adjusted target image to the display screen 1801;

[0233] The display screen 1601 is also used to display an updated image interface; the updated image interface simultaneously displays the adjusted endoscope image and the adjusted target image.

[0234] Although this specification provides method operation steps as described in the embodiments or flowcharts, more or fewer operation steps may be included based on conventional or non-creative means. The step order listed in the embodiments is only one way among the execution orders of numerous steps and does not represent the only execution order. When actually executed by a device or client product, it can be executed in the order of the embodiments or the method shown in the drawings or executed in parallel (for example, in an environment of parallel processors or multi-threaded processing, or even in a distributed data processing environment). The term "comprising", "including" or any other variant thereof is intended to cover non-exclusive inclusion, so that a process, method, product or device including a series of elements not only includes those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such a process, method, product or device. Without more limitations, it does not exclude the existence of additional identical or equivalent elements in the process, method, product or device including the said elements. The words such as first, second, etc. are used to represent names and do not represent any specific order.

[0235] Those skilled in the art also know that, in addition to implementing the controller in the form of pure computer-readable program code, the method steps can be logically programmed to enable the controller to implement the same functions in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, embedded microcontrollers, etc. Therefore, such a controller can be regarded as a hardware component, and the devices included therein for implementing various functions can also be regarded as the structures within the hardware component. Or even, the devices for implementing various functions can be regarded as either software modules for implementing the method or the structures within the hardware component.

[0236] This specification can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, classes, etc. that perform specific tasks or implement specific abstract data types. This specification can also be practiced in a distributed computing environment where tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer-readable storage media including storage devices.

[0237] From the description of the above embodiments, those skilled in the art can clearly understand that this specification can be implemented by means of software plus a necessary general hardware platform. Based on such an understanding, the technical solution of this specification can essentially be embodied in the form of a software product, and this computer software product can be stored in a storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to enable a computer device (which can be a personal computer, mobile terminal, server, or network device, etc.) to execute the methods described in various embodiments or some parts of the embodiments of this specification.

[0238] The various embodiments in this specification are described in a progressive manner. For the same or similar parts between the various embodiments, reference can be made to each other. Each embodiment focuses on the differences from other embodiments. This specification can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multi-processor systems, microprocessor-based systems, set-top boxes, programmable electronic devices, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc.

[0239] Although this specification is depicted through embodiments, those of ordinary skill in the art know that this specification has many variations and changes without departing from the spirit of this specification, and it is hoped that the appended claims will cover these variations and changes without departing from the spirit of this specification.

Claims

1. An image processing method for a robot control terminal, characterized in that: include: Receiving first operation data for an endoscopic image in an image interface; wherein the image interface also displays a target image; the target image is a virtual image associated with the endoscopic image; Determine, according to the first operation data, form adjustment data for the target image; wherein the form adjustment data at least includes position adjustment data and / or size adjustment data; According to the endoscopic image, content adjustment data for the target image is determined through feature matching; Generate an adjusted target image according to the form adjustment data and / or the content adjustment data; and display the adjusted target image in the image interface.

2. The method according to claim 1, characterized in that After receiving first operation data for the endoscopic image in the image interface, the method further includes: According to the first operation data, the size and / or position of the endoscopic image is adjusted accordingly to obtain an adjusted endoscopic image; The adjusted endoscopic image is displayed in the image interface.

3. The method according to claim 1, characterized in that: Receiving first operation data for an endoscopic image in an image interface includes: When detecting that the user initiates a switching trigger action on the control terminal, the control terminal is switched to the image adjustment mode; In the image adjustment mode, first operation data for the endoscopic image is acquired by collecting action data of the user on the control arm and / or the control joint of the control terminal.

4. The method according to claim 2 or 3, characterized in that: After switching the control terminal to the image adjustment mode, the method further includes: Get the current position parameters of the image interface and the current position parameters of the control arm; According to the preset coordinate system mapping relationship, using the current position parameters of the image interface and the current position parameters of the control arm, determine the position parameters of the identification icon associated with the control arm based on the image interface; According to the position parameters of the identification icon based on the graphic interface, the identification icon associated with the control arm is displayed in the graphic interface.

5. The method according to claim 4, characterized in that According to the first operation data, the size and / or position of the endoscopic image is adjusted accordingly, including: According to the first moving action in the first operation data, the identification icon is moved to the control point position of the endoscope image; According to the second pinch-and-release action in the first operation data, locking the control point; According to the third movement action in the first operation data, the endoscopic image is translated and / or scaled based on the control point.

6. The method according to claim 2, characterized in that Determining form adjustment data for a target image according to the first operation data includes: Determining position information and size information of the adjusted endoscopic image according to the first operation data; Determine a blank area in the image interface according to the adjusted position information and size information of the endoscopic image; According to the blank area in the image interface, form adjustment data for the target image is determined.

7. The method according to claim 2, characterized in that According to the endoscopic image, content adjustment data for the target image is determined through feature matching, including: extracting edge contour features from the endoscopic image; According to the edge contour features, feature matching is performed in a preset image data source to obtain corresponding feature matching results; According to the feature matching result, target image data with a matching degree greater than a preset matching degree threshold is extracted from the preset image data source as content adjustment data for the target image.

8. The method according to claim 7, characterized in that After performing feature matching in a preset image data source according to the edge contour feature and obtaining a corresponding feature matching result, the method further includes: When it is determined according to the feature matching result that target image data having a matching degree greater than a preset matching degree threshold cannot be extracted from the preset image data source, a current endoscope viewing angle is determined according to the endoscope image; According to the current endoscope viewing angle, a preset image data source is adjusted to obtain an adjusted image data source; According to the edge contour features, feature matching is re-performed in the adjusted image data source to obtain corresponding feature matching results.

9. The method according to claim 2, characterized in that: After generating an adjusted target image according to the form adjustment data and / or the content adjustment data, the method further includes: Determining the instrument type of the operating instrument in the endoscopic image, and obtaining position parameters of the operating instrument based on the endoscopic image; According to the position parameters of the operating instrument, the relative position and posture of the virtual instrument based on the marker points in the adjusted target image are determined; according to the instrument type of the operating instrument, a preset instrument model database is queried to obtain the model data of the virtual instrument; According to the relative position and posture of the virtual device, the display position of the virtual device is determined in the adjusted target image; Based on the model data of the virtual tool, a virtual tool corresponding to the operating tool is displayed at a display position in the adjusted target image.

10. The method according to claim 2, characterized in that After displaying the adjusted target image in the image interface, the method further includes: Monitor and obtain change data of adjusted endoscopic images; According to the change data of the adjusted endoscopic image, the displayed adjusted target image is adjusted in linkage.

11. The method according to claim 6, characterized in that After obtaining the adjusted endoscopic image, the method further includes: Detect whether the area of ​​the current blank area in the image interface is less than or equal to the threshold area; When it is determined that the area of ​​the current blank area is less than or equal to the threshold area, adjusting the data according to the content to generate an adjusted target image; According to preset layout rules, the adjusted target image is displayed in the endoscopic image of the image interface in an embedded manner.

12. The method according to claim 11, characterized in that After detecting whether the area of ​​the current blank area in the image interface is less than or equal to the threshold area, the method further includes: When it is determined that the area of ​​the current blank area is greater than the threshold area, generating an adjusted target image according to the form adjustment data and / or the content adjustment data; According to the preset layout rule, the adjusted target image is displayed in a blank area of ​​the image interface in other corresponding display modes.

13. An image processing device for a robot control terminal, characterized in that: include: A receiving module, configured to receive first operation data for an endoscopic image in an image interface; wherein the image interface also displays a target image; and the target image is a virtual image associated with the endoscopic image; A first determining module, configured to determine form adjustment data for a target image according to the first operation data; wherein the form adjustment data at least includes position adjustment data and / or size adjustment data; A second determination module is used to determine content adjustment data for a target image through feature matching based on the endoscopic image; The processing module is used to generate an adjusted target image according to the form adjustment data and / or the content adjustment data; and display the adjusted target image in the image interface.

14. A robot control terminal, characterized in that: At least includes a display screen, a control arm, and a processor; wherein, The display screen is used to display an image interface; the image interface simultaneously displays an endoscopic image and a target image; the target image is a virtual image associated with the endoscopic image; The control arm is used to receive first operation data for the endoscopic image in the image interface; The processor is used to adjust the size and / or position of the endoscopic image accordingly according to the first operation data to obtain an adjusted endoscopic image; the processor is also used to determine form adjustment data for a target image according to the first operation data; wherein the form adjustment data at least includes position adjustment data and / or size adjustment data; according to the endoscopic image, content adjustment data for the target image is determined by feature matching; and the adjusted target image is generated according to the form adjustment data and / or the content adjustment data; The display screen is also used to display an updated image interface; the updated image interface simultaneously displays the adjusted endoscopic image and the adjusted target image.

15. An image processing method for a robot control terminal, characterized in that: include: Receiving first operation data for a first image in an image interface; wherein the image interface also displays a target image; the target image is associated with the first image; Determine, according to the first operation data, form adjustment data for the target image; wherein the form adjustment data at least includes position adjustment data and / or size adjustment data; Determining content adjustment data for a target image through feature matching according to the first image; Generate an adjusted target image according to the form adjustment data and / or the content adjustment data; and display the adjusted target image in the image interface.

Images