An apparatus, method and system for positioning

Abstract

The embodiment of the specification provides an instrument positioning method and system, the method comprises the following steps: acquiring an external recognition image and an endoscope acquisition image respectively, the external recognition image and the endoscope acquisition image are respectively captured by a robot and an endoscope; the endoscope and the instrument are respectively installed on a mechanical arm of the robot; determining a key point space position of the mechanical arm according to the external recognition image; determining an endoscope space pose and an instrument space pose respectively based on the key point space position; determining a positionable space range by combining the endoscope space pose and the endoscope acquisition image; and controlling the instrument to move to the positionable space range based on the instrument space pose. The above scheme guarantees the safety of positioning the instrument before operation, improves the operation efficiency and precision of the robot, and has high practical application value.

Description

Technical Field

[0001] The embodiments in this specification relate to the field of instrument control technology, and in particular to an instrument placement method, system and storage medium. Background Technology

[0002] Endoscopic surgery is a relatively new surgical technique. It involves making several small incisions on the patient's skin and inserting an endoscope and surgical instruments through these incisions. The endoscope allows for visualization of the patient's internal condition, while the surgical instruments are used to perform the necessary surgical procedures. Endoscopic surgery offers a safe and efficient way to perform procedures, minimizing surgical trauma and demonstrating significant practical value.

[0003] During endoscopic surgery, it's crucial to ensure the instruments remain within the endoscope's field of view. This requires positioning the instruments and adjusting their orientation to ensure they are visible before proceeding with subsequent procedures. Since the endoscope cannot provide information about the instruments' position in the working environment, this process often necessitates manual insertion based on the user's experience. Manual insertion demands a high level of skill and carries inherent risks. Furthermore, it hinders subsequent high-precision robotic manipulation. Therefore, a method for automatically and accurately controlling instrument positioning is urgently needed. Summary of the Invention

[0004] The purpose of the embodiments in this specification is to provide a method and system for instrument placement to solve the problem of how to automatically and accurately control instrument placement.

[0005] To address the aforementioned technical problems, embodiments of this specification propose an instrument placement method, comprising: acquiring an external recognition image captured by a robot and an endoscopic acquisition image captured by an endoscope; the endoscope is mounted on a robotic arm of the robot; instruments are also mounted on other robotic arms of the robot; determining the spatial positions of key points of the robotic arm based on the external recognition image; determining the spatial pose of the endoscope and the instrument based on the spatial positions of the key points; determining a placement space range by combining the spatial pose of the endoscope and the endoscopic acquisition image; and controlling the instrument to move to the placement space range based on the instrument spatial pose.

[0006] In some implementations, the positioning space is within the field of view of the endoscope.

[0007] In some embodiments, controlling the movement of the instrument to the available positioning space based on the instrument's spatial pose includes: determining a target point within the available positioning space based on images acquired by the endoscope; and controlling the movement of the instrument to the target point based on the instrument's spatial pose.

[0008] In some implementations, controlling the movement of the instrument to the available positioning space based on the instrument's spatial pose includes: during the movement of the instrument, tracking and acquiring the real-time spatial pose and the real-time available positioning space of the instrument; and adjusting the movement path of the instrument based on the real-time spatial pose and the real-time available positioning space.

[0009] In some embodiments, the endoscope includes a binocular endoscope; determining the possible placement space range by combining the spatial pose of the endoscope and the images acquired by the endoscope includes: determining the coordinates of a safety point in the images acquired by the endoscope; determining the depth of the safety point coordinates based on a binocular image algorithm; determining the spatial position of the lens according to the spatial pose of the endoscope; and determining the possible placement space range by combining the safety point coordinates, the depth of the safety point coordinates, and the spatial position of the lens.

[0010] In some embodiments, the robot is equipped with multiple instruments; the multiple instruments are respectively mounted on different robotic arms; the step of controlling the movement of the instruments to the available positioning space based on the spatial pose of the instruments includes: determining a preset position for each instrument according to the instrument spatial pose of each instrument and the available positioning space; adjusting the preset positions of each instrument based on the conflict relationship between the preset positions to obtain a target positioning position; and moving each instrument to the corresponding target positioning position.

[0011] Based on the aforementioned embodiments, determining the preset position of each instrument according to its instrument spatial pose and the available space range includes: dividing the available space range into a corresponding number of sub-space regions according to the number of instruments; and determining the preset position of the corresponding instrument in each sub-space region.

[0012] In some implementations, determining the key point spatial position of the robotic arm based on the external recognition image includes: receiving status information of the robotic arm's mounting device sent by the robot; determining the robotic arm to be identified based on the status information; and determining the key point spatial position of the robotic arm to be identified from the external recognition image.

[0013] In some embodiments, before determining the key point spatial position of the robotic arm based on the external recognition image, the method further includes: obtaining the model parameters of the robot; correspondingly, determining the key point spatial position of the robotic arm based on the external recognition image includes: determining the key point spatial position of the robotic arm based on the model parameters and the external recognition image.

[0014] In some embodiments, controlling the movement of the device to the available positioning space based on the device's spatial pose includes: stopping the movement of the device when an abnormal movement state is detected; the abnormal movement state includes one of the following: the device's pose exceeding the limit, the device deviating from the movement path, or the device contacting the patient's tissue; and replanning the movement path of the device based on the current position of the device and the available positioning space.

[0015] This specification also proposes an instrument placement system, including a robot, an external optical platform, and a computing device. An endoscope and an instrument are mounted on the robot's robotic arm. The robot moves the endoscope and instrument by controlling the movement of its robotic arm. The endoscope is used to capture images. The external optical platform is used to acquire external recognition images corresponding to the robot. The computing device is used to acquire the endoscope-captured images and the external recognition images, and is also used to perform the following steps: determining the key point spatial positions of the robotic arm based on the external recognition images; determining the endoscope spatial pose and the instrument spatial pose based on the key point spatial positions; determining a placement space range by combining the endoscope spatial pose and the endoscope-captured images; and controlling the instrument to move to the placement space range based on the instrument spatial pose.

[0016] This specification also proposes a computer-readable storage medium storing a computer program / instruction thereon, which, when executed, implements the above-described instrument placement method.

[0017] As can be seen from the technical solutions provided in the embodiments of this specification above, the method determines the key point spatial position of the robotic arm based on the external recognition image captured by the robot, and then determines the spatial pose of the endoscope and instruments held by the robotic arm based on the key point spatial position. Subsequently, it determines the possible placement space range based on the spatial pose of the endoscope and the images acquired by the endoscope, thereby enabling the instrument to be moved into the possible placement space range according to its spatial pose. This method ensures the safety of preoperative instrument placement, avoids reliance on user experience in this process, improves the robot's operating efficiency and accuracy, and has high practical application value. Attached Figure Description

[0018] To more clearly illustrate the technical solutions in the embodiments or prior art of this specification, the drawings used in the description of the embodiments or prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this specification. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1This is a schematic diagram of a device positioning system according to an embodiment of this specification;

[0020] Figure 2 This is a flowchart illustrating the placement of an instrument according to an embodiment of this specification;

[0021] Figure 3 This is a schematic diagram of the interface for selecting a robot version according to an embodiment of this specification;

[0022] Figure 4 This is a schematic diagram of the interface of an instrument insertion state according to an embodiment of this specification;

[0023] Figure 5 This is a schematic diagram illustrating a process for determining the coordinates of a safety point in the endoscope coordinate system according to an embodiment of this specification.

[0024] Figure 6 This is a schematic diagram illustrating the division of sub-control areas for different instruments according to an embodiment of this specification;

[0025] Figure 7 This is a schematic diagram of the interface showing different states of a device according to an embodiment of this specification.

[0026] Figure 8 This is a schematic diagram of a device posture correction process according to an embodiment of this specification;

[0027] Figure 9 This is a schematic diagram of the interface for a device insertion / removal state according to an embodiment of this specification;

[0028] Figure 10 This is a schematic diagram of the insertion position of an instrument according to an embodiment of this specification. Detailed Implementation

[0029] The technical solutions in the embodiments of this specification will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this specification, and not all embodiments. Based on the embodiments in this specification, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of this specification.

[0030] To better understand the inventive concept of this application, a device positioning system according to an embodiment of this specification will first be described. The device positioning system includes a robot, an external optical platform, and a computing device.

[0031] The robot may include a robot for performing endoscopic surgery. The robot includes multiple robotic arms, each of which can be equipped with and secured instruments or endoscopes. Each robotic arm has a corresponding motor module capable of moving the instruments or endoscopes to appropriate positions to perform corresponding operations or capture images within a corresponding field of view. The methods of securing the robotic arms and instruments / endoscopes are not detailed here.

[0032] Endoscopes are generally long and thin tubular structures, with a lens at the front and a clamping end at the rear. The clamping end can be held by a robotic arm, which can then control the spatial position of the endoscope. Preferably, the endoscope is a binocular endoscope, meaning it has two cameras at the front. Based on the images captured by the two cameras and the distance between the cameras, the depth information of the target area in the image can be determined.

[0033] Furthermore, depending on the task requirements, different types of instruments can be installed on the robotic arm, or corresponding instruments can be installed on multiple robotic arms to combine and complete the corresponding operation.

[0034] An external optical platform can acquire images corresponding to the robot based on optical sensing. The external optical platform can include a general optical camera, and the corresponding images can be pictures captured by the optical camera; alternatively, the external optical platform can include other optical sensing devices such as infrared detectors and point cloud scanners, allowing images to be acquired through infrared, point cloud, or other methods. The purpose of the images acquired by the external optical platform is to determine the positions of the robot's various robotic arms, and thus the positions of the endoscopes and instruments held by the robotic arms.

[0035] The computing device can communicate with the robot, the external optical platform, and the endoscope to obtain images transmitted by the external optical platform and the endoscope, as well as relevant information transmitted by the robot.

[0036] In actual minimally invasive surgical scenarios, the endoscope is connected to an imaging carriage to transmit the acquired image data to the carriage for display. Correspondingly, the computing device acquires the data from the images captured by the endoscope by communicating with the imaging carriage.

[0037] The computing device is equipped with corresponding computer programs / instructions for implementing the instrument placement method. The specific implementation process can be found in the following description of the instrument placement method.

[0038] The computing device may be a computing device installed on a robot, an external optical platform, or an image trolley, or it may be a computing device independent of these devices, or it may be a server installed in the cloud, etc. There are no restrictions on this.

[0039] like Figure 1 The diagram shows a scenario of the instrument positioning system. The control cart is responsible for controlling the movement of the robotic arm. The robot contains multiple robotic arms, which can be used to hold different instruments or endoscopes. While the image cart displays the endoscope image, the external optical platform can take corresponding images of the robot to complete the position determination of the robotic arm and the corresponding instrument / endoscope.

[0040] To address the aforementioned technical problems, embodiments of this specification propose a method for instrument placement. The instrument placement method can be executed by a corresponding computing device, such as a user control terminal, image trolley, or cloud server, or other computing devices that can be integrated into the environment. Figure 2 As shown, the instrument placement method includes the following specific implementation steps.

[0041] S210: Acquire external recognition images captured by the robot and endoscopic images captured by the endoscope, respectively; the endoscope is mounted on the robot's robotic arm; instruments are also mounted on other robotic arms of the robot.

[0042] Before positioning the equipment, external recognition images taken by the robot and endoscopic images taken by the endoscope can be acquired.

[0043] The external recognition image can be an image captured by an external optical platform for the robot. The spatial position of different robotic arms on the robot can be determined by the external recognition image, and then the spatial pose of the endoscope and instruments can be determined based on the size and status information of the endoscope and instruments.

[0044] The images acquired by the endoscope are those captured by the endoscope. The endoscope should be in position when acquiring both the endoscopic and external recognition images. Because the endoscope has a camera at its tip, the path of the endoscope can be monitored in real time during positioning, thus ensuring the safety of the endoscope insertion process.

[0045] Because the images captured by the endoscope correspond to the internal condition, it is possible to obtain condition information based on the images captured by the endoscope and determine the space within which the instrument can be safely inserted.

[0046] It should be noted that, in order to ensure that the external recognition image and the endoscopic acquisition image correspond, the external recognition image and the endoscopic acquisition image should be images acquired at the same time, or the external recognition image and the endoscopic acquisition image should be images acquired during the time period when the robotic arm of the surgical robot is not moving.

[0047] In some implementations, the external optical platform has a calibration function, which can pre-calibrate the robot and perform real-time pose tracking of the instrument after calibration to determine the coordinates of the corresponding key points of the instrument.

[0048] The specific methods for acquiring images can be implemented based on the communication relationships between different devices in the system, which will not be elaborated here.

[0049] S220: Determine the spatial position of the key points of the robotic arm based on the external recognition image.

[0050] To achieve the positioning of endoscopes and instruments, the spatial positions of key points on the robotic arm can first be determined based on external recognition images. These key point positions can correspond to key points set on the robotic arm. The positions of these key points can be set as needed, preferably at the end of the robotic arm or at the location where the instrument or endoscope is held, to better position the held endoscope or instrument.

[0051] In practical applications, in order to ensure that the spatial pose of instruments and endoscopes can be determined based on the spatial location of key points, at least two key points should be set on a robotic arm, and the acquired external recognition image should contain at least two key points for a robotic arm.

[0052] Specifically, keypoint identification can be achieved by pre-marking keypoints on the robotic arm, enabling identification from images and determining their spatial locations; alternatively, keypoint features can be pre-recorded, and feature analysis of the image can be used to identify the keypoints. When using other sensing methods, special processing can be applied to keypoints to acquire different sensor information corresponding to each keypoint, thereby determining their position in the image. The specific method for keypoint identification can be set according to actual application requirements and is not limited thereto.

[0053] After identifying key points from the external recognition image, it is necessary to determine the spatial location of each key point. The location of a key point on the external recognition image is a two-dimensional plane. Further determination of the distance between the key point and the camera of the external optical platform is required. This distance can be achieved by setting up a binocular camera and using a binocular algorithm to determine the distance. Alternatively, when using infrared, point cloud, or other sensing methods, after identifying the key points, the corresponding distance data can be directly obtained from the sensing results to determine the spatial location of the key points. In practical applications, the appropriate method can be adopted to determine the spatial location of the key points based on the specific device specifications; there are no restrictions on this.

[0054] It should be noted that the determined spatial position of the key point corresponds to the position in the coordinate system of the external optical platform. For ease of description, this coordinate system is referred to as the real coordinate system.

[0055] Furthermore, since the external recognition image is acquired using optical sensing, in practical applications, obstructions between robotic arms may prevent the capture of key points for the corresponding robotic arm. Therefore, in practical applications, the external optical platform's imaging module is equipped with a corresponding motor module to drive its movement until the key points of the corresponding robotic arm can be effectively captured in the image. The movement of the external optical platform's imaging module can be manually controlled or automatically controlled by the external optical platform after analyzing the captured image; there are no restrictions on this.

[0056] In some implementations, robot model parameters can be obtained before identifying the spatial location of key points. The computing device can pre-record key point identification methods for different robot models, thereby enabling better identification of the spatial location of key points on the robotic arm based on the model parameters.

[0057] In practical applications, different robot models may differ. For example, some robots may have high-precision joint encoders or different numbers of robotic arms, and different types of robots have different recognition requirements. Recognition methods for key points can be pre-labeled for each robot model. Furthermore, different robot models may have different types of endoscope lenses (e.g., plano lenses, oblique lenses), which can be further applied to the subsequent recognition process of images acquired by the endoscope.

[0058] The model parameters obtained can be manually entered by the operator, or they can be obtained based on the connection between the computing device and the robot, for example, Figure 3 As shown, users can manually select different pre-entered robot versions to obtain the corresponding model parameters. There are no restrictions on the specific method for obtaining the model parameters.

[0059] In some implementations, before identifying the spatial location of key points, the state information of the instruments installed on the robotic arm on the robot can be determined, thereby identifying the robotic arm to be identified, and then determining the spatial location of the key points of the robotic arm to be identified, so as to ensure the accuracy of the identification results.

[0060] Because the robot can obtain the connection status of the equipment during installation, for example... Figure 4As shown, the insertion status of the instrument can be determined for different tool arms, thereby identifying the robotic arm that needs to identify key points. Preferably, the state parameters of different robotic arms or corresponding key points can be recorded simultaneously, thereby ensuring that the key points and spatial positions of the corresponding robotic arms can be identified in a targeted manner.

[0061] S230: Determine the spatial pose of the endoscope and the spatial pose of the instrument based on the spatial positions of the key points.

[0062] After determining the spatial positions of the keypoints, the spatial poses of the endoscope and instruments can be further determined. These poses describe the spatial positions of the endoscope and instruments. Since the positions of the keypoints on the robotic arm are predetermined, the spatial poses of the endoscope and instruments can be determined based on the positions of the keypoints and the shape characteristics of the endoscope / instrument when it is mounted on the robotic arm.

[0063] Specifically, the spatial positional relationship between the endoscope / instrument and key points when held by the robotic arm can be predetermined. When at least two key points exist, the spatial pose of the endoscope and the instrument can be directly determined based on the spatial positions of the key points and the aforementioned spatial relationship. The specific process for determining the spatial pose of the endoscope and the instrument can be implemented based on corresponding coordinate derivation processes, which will not be elaborated here.

[0064] Furthermore, given the spatial pose of the endoscope and the instrument, the spatial position of any component on the endoscope / instrument can also be determined based on the overall spatial pose.

[0065] Correspondingly, the coordinates corresponding to the endoscope spatial pose and the instrument spatial pose also correspond to the real coordinate system.

[0066] S240: Determine the possible placement space range by combining the endoscope's spatial pose and the images acquired by the endoscope.

[0067] Based on the endoscopic spatial orientation and the images captured by the endoscope, the possible placement space range can be determined. The possible placement space range is the area within which instruments can be safely inserted. Inserting instruments within this range minimizes the risk of damage to tissues or organs and is generally closer to the location where the procedure needs to be performed.

[0068] Specifically, based on the criteria for determining the available space in the images acquired by the endoscope, the corresponding available image area can be determined. These criteria can be set according to requirements, such as ensuring the area does not touch the patient's body surface or is close to the area where the task needs to be performed. These criteria can be translated into requirements for image recognition, thereby enabling the determination of the available space range from the images acquired by the endoscope.

[0069] In some implementations, such as Figure 5 As shown, the coordinates of safety points can be determined first in the images acquired by the endoscope. These safety point coordinates can be the corresponding coordinates of a specific safety point. Safety points can be points determined after uniformly identifying the entire image area, or points that satisfy the boundary of the possible placement space. In short, the spatial area of ​​the possible placement space can be determined based on these safety points.

[0070] The safety point coordinates are coordinates on the image plane, i.e., two-dimensional coordinates, which cannot be applied to spatial range. Therefore, it is also necessary to determine the depth of each safety point coordinate. For example, when the endoscope is a binocular endoscope, two cameras can simultaneously acquire images from different shooting points, and then the depth of the safety point coordinates can be determined by a binocular image algorithm. The specific implementation process can be set according to the actual application requirements, and will not be elaborated here.

[0071] The safety point coordinates and corresponding depth at this point correspond to the endoscope's imaging point, that is, the coordinates in the endoscope coordinate system, and cannot be directly applied to the real-world coordinate system. Therefore, the lens spatial position can also be determined based on the endoscope's spatial pose, which is the spatial position coordinate of the endoscope's camera in the real-world coordinate system.

[0072] Finally, by combining the safety point coordinates, the depth of the safety point coordinates, and the spatial position of the lens, the safety point coordinates can be converted into coordinates in the real coordinate system, thereby obtaining the possible positioning space range.

[0073] To ensure the smooth progress of subsequent surgical procedures, the available positioning space is wholly or partially within the field of view of the endoscope. This ensures that after the instruments are moved into the available positioning space, they are included within the endoscope's field of view. During subsequent operations, if the moving instruments can be captured by the endoscope, the spatial position of the instruments can be monitored in real time, thereby ensuring the normal execution of the procedure.

[0074] It should be noted that the execution order of steps S220-S240 is not restricted in practical applications. For example, the possible positioning space corresponding to the endoscope coordinate system can be determined first based on the images acquired by the endoscope, and then the endoscope spatial pose can be determined based on the external recognition images, thereby converting the possible positioning space corresponding to the endoscope coordinate system into the possible positioning space corresponding to the real coordinate system. Alternatively, the endoscope spatial pose can be determined first based on the external recognition images, and then the possible positioning space corresponding to the real coordinate system can be determined directly from the endoscope images based on the spatial position of the endoscope. Two processes can also be used to process the aforementioned two steps simultaneously. The specific execution order can be selected according to requirements in practical applications.

[0075] S250: Based on the spatial pose of the instrument, control the movement of the instrument to the available positioning space range.

[0076] After determining the available positioning space range and the instrument's spatial pose, since both positions correspond to the real coordinate system, the instrument can be controlled to move to the available positioning space range based on the positional correspondence between the instrument's spatial pose and the available positioning space range.

[0077] Specifically, based on the correspondence between the available space range and the device's spatial pose, the movement path of the device and its own pose adjustment can be planned, and the device can be moved to the available space range according to this pre-planned path.

[0078] In some implementations, the robot may have multiple robotic arms, each equipped with a corresponding instrument, and all of these instruments need to be operated. In this case, the preset position of each instrument can be determined first based on its spatial pose and the available space range. The preset position can be a position corresponding to each instrument determined in a general manner without considering conflicts between instruments.

[0079] After determining the preset positions, the preset positions of each instrument are adjusted according to the conflict relationships between the preset positions to obtain the target positioning position. The target positioning position ensures that multiple instruments can move to their corresponding positions without conflict or interference. In addition, depending on the actual situation, the order in which different instruments move to their corresponding target positioning positions can be set to avoid interference between instruments.

[0080] When determining the position of the preset value, such as Figure 6As shown, the available space can be pre-divided into a corresponding number of sub-space regions based on the number of instruments, and then the pre-set position of the corresponding instrument can be determined in each sub-space region. The position of the sub-control region can also correspond to the position of the required instrument to ensure the final effect after the instrument is moved. By pre-dividing the sub-control regions, it is possible to ensure that the instruments do not interfere with each other as much as possible, thus optimizing the subsequent process of determining the target positioning position.

[0081] Furthermore, in some embodiments, during the movement of the instrument to the designated positioning space, the real-time spatial pose and the real-time positioning space range of the instrument can be acquired in real time. Specifically, an external optical platform can acquire external recognition images corresponding to the robot in real time, while an endoscope can acquire endoscopic images of the patient's body in real time. The real-time spatial pose and the real-time positioning space range of the instrument are then identified from the external recognition images using the aforementioned method.

[0082] Accordingly, given the real-time spatial pose and the real-time possible positioning space range, the movement path of the instrument can be adjusted based on these parameters. For example, if the soft tissue shifts position during the process, inserting the instrument along the original path might cause damage to the soft tissue. Therefore, the instrument's movement path can be adjusted and planned in real time to ensure safe positioning. The specific adjustment and planning methods can be determined based on pre-set route planning methods, and will not be elaborated upon here.

[0083] In some implementations, the status of the device can be monitored in real time during its movement. This monitoring process can be performed by a robot, which then transmits the data to a computing device. For example... Figure 7 The diagram shows an interface for the connection status of instruments on different tool arms. The corresponding statuses can include "in place", "in position", "over-limit adjustment", etc.

[0084] If an abnormal movement is detected during the movement of an instrument, the movement can be stopped. Abnormal movement includes instrument position exceeding limits, instrument deviating from the movement path, and instrument contacting tissue. When an abnormal movement occurs, it may hinder the continuation of the instrument's movement, necessitating stopping the instrument's original movement path and making adjustments. Figure 8 The diagram shown is a flowchart illustrating an example of adjusting the pose of an instrument.

[0085] Specifically, the movement path of the instrument can be replanned based on its current position and the available space to ensure it moves normally to the designated space. If a malfunction is detected, the instrument can be repaired or removed and reinstalled. This can also be reflected in the instrument's status, for example... Figure 9 As shown, this is the status information determined for the insertion and removal of instruments, where tool arms 2 and 4 have corresponding status information for the removal or insertion of instruments.

[0086] In practical applications, other types of abnormal action states can also be set according to requirements, and there are no restrictions on this.

[0087] Furthermore, in some implementations, the path of the instrument entering the body can be recorded. After execution, the corresponding withdrawal path can be deduced based on this path to ensure the safety of the procedure. Accordingly, the insertion path can be saved for each robotic arm to ensure its effective use in the future.

[0088] Furthermore, since the subsequent processes are controlled by the user, therefore... Figure 10 As shown, the placement space range for the instrument can correspond to an intermediate depth, without needing to move it to contact soft tissue. The specific depth range can be set according to needs and is not limited. Correspondingly, to facilitate operation, a highlight can be added to the edge of the display image to ensure that the user can monitor whether the instrument has entered the field of view in real time based on the captured image. Based on the above embodiments and examples, it can be seen that the method determines the key point spatial position of the robotic arm based on the external recognition image captured by the robot, and then determines the spatial pose of the endoscope and instrument held by the robotic arm based on the key point spatial position. Then, the placement space range is determined based on the spatial pose of the endoscope and the image captured by the endoscope, thereby enabling the instrument to be moved into the placement space range according to the spatial pose of the instrument. Through the above method, the safety of inserting the instrument into the working environment before surgery is ensured, avoiding the situation where this process relies purely on the user's practical experience, improving the operating efficiency and accuracy of the robot, and having high practical application value.

[0089] based on Figure 1 The corresponding instrument placement method, according to embodiments of this specification, provides a computer-readable storage medium storing a computer program / instruction. The computer-readable storage medium can be read by a processor via the device's internal bus, and the processor can then implement the program instructions in the computer-readable storage medium.

[0090] In this embodiment, the computer-readable storage medium can be implemented in any suitable manner. The computer-readable storage medium includes, but is not limited to, random access memory (RAM), read-only memory (ROM), cache, hard disk drive (HDD), memory card, etc. The computer storage medium stores computer program instructions. When the computer program instructions are executed, this specification is implemented. Figure 1 The program instructions or modules corresponding to the embodiments.

[0091] It should be noted that the above-mentioned instrument placement method, system and storage medium can be applied to the field of instrument control technology, and can also be applied to other technical fields without limitation.

[0092] Although the process described above includes multiple operations that occur in a specific order, it should be clearly understood that these processes may include more or fewer operations, which may be executed sequentially or in parallel (e.g., using parallel processors or a multithreaded environment).

[0093] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this specification. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0094] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0095] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0096] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0097] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0098] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0099] Those skilled in the art will understand that the embodiments of this specification can be provided as methods, systems, or computer program products. Therefore, the embodiments of this specification can take the form of entirely hardware embodiments, entirely software embodiments, or embodiments combining software and hardware aspects. Furthermore, the embodiments of this specification can take the form of computer program products implemented on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0100] The embodiments described in this specification can be described in the general context of computer-executable instructions, such as program modules, that are executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform a specific task or implement a specific abstract data type. The embodiments of this specification can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0101] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, system embodiments are basically similar to method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments. In the description of this specification, the terms "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., refer to specific features, structures, materials, or characteristics described in connection with that embodiment or example, which are included in at least one embodiment or example of the embodiments in this specification. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described can be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification and the features of different embodiments or examples.

[0102] The above description is merely an embodiment of this application and is not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.

Claims

1. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer program instructions, which, when executed, implement an instrument positioning method, the instrument positioning method comprising: The system acquires external recognition images captured by the robot and endoscopic images captured by the endoscope. The endoscope is mounted on the robot's robotic arm. Other instruments are also mounted on the robot's other robotic arms. The external recognition images are images captured by an external optical platform on the robot. The endoscopic images and the external recognition images are acquired before the instruments are in place and after the endoscope is in place. The spatial position of the key points of the robotic arm in the coordinate system of the external optical platform is determined based on the external recognition image. Based on the spatial positions of the key points, the spatial poses of the endoscope and the instrument in the coordinate system of the external optical platform are determined respectively. The possible placement space range is determined by combining the endoscope's spatial pose and the images acquired by the endoscope. The possible placement space range is the space range in which the instrument can be safely placed. The endoscope includes a binocular endoscope. The determination of the possible placement space range by combining the endoscope's spatial pose and the images acquired by the endoscope includes: determining the coordinates of a safe point in the endoscope coordinate system in the images acquired by the endoscope; determining the depth of the safe point coordinates in the endoscope coordinate system based on a binocular image algorithm; determining the lens spatial position according to the endoscope's spatial pose; and determining the possible placement space range in the external optical platform coordinate system by combining the safe point coordinates, the depth of the safe point coordinates, and the lens spatial position. Based on the spatial pose of the instrument, the instrument is moved to the available positioning space.

2. The computer-readable storage medium as claimed in claim 1, characterized in that, The step of controlling the movement of the instrument to the available positioning space based on the instrument's spatial pose includes: Based on the images acquired by the endoscope, the target point is determined within the available placement space. Based on the spatial pose of the instrument, the instrument is moved to the target point.

3. The computer-readable storage medium as claimed in claim 1, characterized in that, Determining the spatial position of key points of the robotic arm in the external optical platform coordinate system based on the external recognition image includes: Receive status information of the robotic arm installing instruments sent by the surgical robot; The robotic arm to be identified is determined based on the aforementioned status information; The spatial location of key points of the robotic arm to be identified is determined from the external recognition image.

4. The computer-readable storage medium as claimed in claim 1, characterized in that, Before determining the spatial position of the key points of the robotic arm in the external optical platform coordinate system based on the external recognition image, the method further includes: Obtain the model parameters of the robot; Accordingly, determining the spatial location of the key points of the robotic arm based on the external recognition image includes: Based on the model parameters, the spatial positions of key points of the robotic arm are determined according to the external recognition image.

5. The computer-readable storage medium as claimed in claim 1, characterized in that, The robot is equipped with multiple instruments; each instrument is mounted on a different robotic arm; the control of moving the instrument to the available positioning space based on the instrument's spatial pose includes: The preset position of each instrument is determined according to the instrument spatial pose and the available space range. The target positioning position is obtained by adjusting the preset positions of each instrument based on the conflict relationship between the preset positions; Move each instrument to its corresponding target position.

6. The computer-readable storage medium as claimed in claim 5, characterized in that, The step of determining the preset position of each instrument based on its spatial pose and the available positioning space range includes: The available space is pre-divided into a corresponding number of sub-space regions based on the number of instruments; Determine the preset positions of the corresponding instruments in each subspace region.

7. The computer-readable storage medium as claimed in claim 1, characterized in that, The step of controlling the movement of the instrument to the available positioning space based on the instrument's spatial pose includes: During the movement of the instrument, the real-time spatial pose and real-time positioning space range of the instrument are tracked and acquired. The movement path of the instrument is adjusted based on the real-time spatial pose and the real-time available positioning space range.

8. The computer-readable storage medium as claimed in claim 1, characterized in that, The step of controlling the movement of the instrument to the available positioning space based on the instrument's spatial pose includes: When an abnormal movement of the instrument is detected, the movement of the instrument is stopped; the abnormal movement includes one of the following: the instrument's position exceeds the limit, the instrument deviates from the movement path, or the instrument comes into contact with tissue; Based on the current position and available space of the instrument, the movement path of the instrument is replanned.

9. An instrument placement system, characterized in that, This includes robots, external optical platforms, and computing devices; The robot's robotic arm is equipped with an endoscope and instruments; the robot moves the endoscope and instruments by controlling the movement of the robotic arm; the endoscope is used to capture images acquired by the endoscope. The external optical platform is used to acquire external recognition images corresponding to the robot; The computing device is used to acquire the images captured by the endoscope and the external recognition images; It is also used to perform the following steps: determining the spatial position of key points of the robotic arm in the coordinate system of an external optical platform based on the external recognition image; determining the spatial pose of the endoscope and the spatial pose of the instrument in the coordinate system of the external optical platform based on the spatial position of the key points; determining the possible placement space range by combining the spatial pose of the endoscope and the image acquired by the endoscope, wherein the possible placement space range is the space range in which the instrument can be safely placed; the endoscope includes a binocular endoscope, and the step of determining the possible placement space range by combining the spatial pose of the endoscope and the image acquired by the endoscope includes: The coordinates of a safe point in the endoscopic coordinate system are determined from the images acquired by the endoscope. The depth of the safe point in the endoscopic coordinate system is determined based on a binocular image algorithm. The spatial position of the lens is determined based on the spatial pose of the endoscope. The possible positioning space range in the external optical platform coordinate system is determined by combining the safe point coordinates, the depth of the safe point coordinates, and the spatial position of the lens. The instrument is moved to the possible positioning space range based on the instrument's spatial pose. The endoscopic images and the external recognition images are acquired before the instrument is positioned and after the endoscope is already in place.

Images