Puncture method and related products based on three-dimensional reconstruction

Abstract

The present application discloses a puncture method based on three-dimensional reconstruction and related products. The method includes: obtaining at least two first images acquired by an image acquisition device when it is fixedly connected to the target structure of a robotic arm, and at least two first images both include target tissues; obtaining at least two first transformation relationships, and the first transformation relationships represent the transformation relationship between the first pixel coordinate system of the first image and the base coordinate system of the robotic arm; obtaining at least two first positions by performing the following steps on each of the at least two first images: determining the position of the target tissue in the first image in the first pixel coordinate system; based on the at least two first transformation relationships, converting the at least two first positions into the position of the target tissue in the base coordinate system, obtaining at least two second positions; based on the at least two second positions, obtaining a three-dimensional model of the target tissue in the base coordinate system.

Description

Technical Field

[0001] The present application relates to the field of medical image processing technology, and in particular to a puncture method based on three-dimensional reconstruction and related products. Background Art

[0002] In the medical field, puncture surgery refers to puncturing the lesion of the object to be punctured along the puncture path with a puncture needle so that the puncture needle penetrates the lesion. Before puncturing, it is necessary to plan the puncture path based on the images collected by the image acquisition device, and then control the puncture needle to move along the puncture path to complete the puncture.

[0003] Since the body of the object to be punctured includes tissues, and some tissues may be damaged by the puncture needle, the position of the tissue needs to be considered when planning the puncture path. However, the images captured by the image acquisition device are usually two-dimensional images, and the information of the tissue contained in the two-dimensional image has limitations, which leads to low accuracy of the puncture path planned based on the images captured by the image acquisition device, resulting in poor puncture effect. Therefore, there is an urgent need for a three-dimensional model of tissue based on the two-dimensional image captured by the image acquisition device. Summary of the invention

[0004] The present application provides a puncture method based on three-dimensional reconstruction and related products, so as to construct a three-dimensional model of the target tissue based on a two-dimensional image of the target tissue.

[0005] In a first aspect, a puncture method based on three-dimensional reconstruction is provided, the method comprising:

[0006] Acquire at least two first images acquired by the image acquisition device when the image acquisition device is fixedly connected to the target structure of the robotic arm, wherein the at least two first images both include the target tissue;

[0007] Acquire at least two first conversion relationships, where the first conversion relationships correspond to the first image one by one, and the first conversion relationships represent a conversion relationship between a first pixel coordinate system of the first image and a base coordinate system of the robotic arm;

[0008] At least two first positions are obtained by performing the following steps on each of the at least two first images: determining the position of the target tissue in the first image in the first pixel coordinate system;

[0009] Based on at least two first conversion relationships, converting the at least two first positions into positions of the target tissue in the base coordinate system to obtain at least two second positions;

[0010] Based on the at least two second positions, a three-dimensional model of the target tissue in the base coordinate system is obtained.

[0011] In combination with any embodiment of the present application, the at least two first images include a second image, and the at least two first conversion relationships include a second conversion relationship corresponding to the second image;

[0012] The obtaining of at least two first conversion relationships includes:

[0013] Acquire a third conversion relationship, where the third conversion relationship represents a conversion relationship between a pixel coordinate system of an image acquired by the image acquisition device and a capture device coordinate system, where the capture device coordinate system is a coordinate system constructed based on the image acquisition device;

[0014] Acquire a fourth conversion relationship between the position and posture of the target structure and the position and posture of the image acquisition device;

[0015] Acquire a first pose of the target structure in the base coordinate system when the image acquisition device acquires the second image;

[0016] The second conversion relationship is obtained based on the product of the third conversion relationship, the fourth conversion relationship and the first posture.

[0017] In combination with any embodiment of the present application, after obtaining the three-dimensional model of the target tissue in the base coordinate system based on the at least two second positions, the method further includes:

[0018] Acquire a third image, wherein the third image is an image acquired by the image acquisition device when the image acquisition device is fixedly connected to the target structure of the robotic arm, the third image includes the target tissue, and the acquisition time of the third image is later than the acquisition time of any one of the at least two first images;

[0019] Acquire a fifth transformation relationship between the second pixel coordinate system of the third image and the base coordinate system;

[0020] Determine a second position range of the third image in the base coordinate system based on the fifth conversion relationship and the first position range of the third image in the second pixel coordinate system;

[0021] The target tissue in the third image is determined based on the target tissue located within the second position range in the three-dimensional model.

[0022] In combination with any embodiment of the present application, the target tissue includes tissue in a target organ of the object to be punctured, the target organ includes a lesion, and the third image includes the lesion;

[0023] After determining the target tissue in the third image based on the target tissue located within the second position range in the three-dimensional model, the method further includes:

[0024] Based on the lesion and the target tissue in the third image, a puncture path for the lesion is determined.

[0025] In combination with any embodiment of the present application, determining the target tissue in the third image based on the target tissue located within the second position range in the three-dimensional model includes:

[0026] Based on the three-dimensional model, determining a third position of the target tissue within the range of the second position;

[0027] Acquire a second posture of the target structure when the image acquisition device acquires the third image;

[0028] Based on the product of the third conversion relationship, the fourth conversion relationship and the second posture, a sixth conversion relationship is obtained, where the sixth conversion relationship represents the conversion relationship between the second pixel coordinate system and the base coordinate system;

[0029] Determining a fourth position of the target tissue in the second pixel coordinate system based on the inverse relationship of the sixth conversion relationship and the third position;

[0030] Based on the fourth position, the target tissue in the third image is determined.

[0031] In combination with any embodiment of the present application, the target tissue includes a blood vessel, and the puncture path does not pass through the second position.

[0032] In a second aspect, a puncture device based on three-dimensional reconstruction is provided, and the puncture device based on three-dimensional reconstruction comprises:

[0033] an acquisition unit, configured to acquire at least two first images acquired by the image acquisition device when the image acquisition device is fixedly connected to the target structure of the robot arm, wherein the at least two first images both include the target tissue;

[0034] The acquisition unit is further used to acquire at least two first conversion relationships, where the first conversion relationships correspond to the first images one by one, and the first conversion relationships represent the conversion relationship between the first pixel coordinate system of the first image and the base coordinate system of the robotic arm;

[0035] A determination unit, configured to obtain at least two first positions by performing the following steps on each of the at least two first images: determining a position of the target tissue in the first image in the first pixel coordinate system;

[0036] A conversion unit, configured to convert the at least two first positions into positions of the target tissue in the base coordinate system based on at least two first conversion relationships, to obtain at least two second positions;

[0037] A processing unit is used to obtain a three-dimensional model of the target tissue in the base coordinate system based on the at least two second positions.

[0038] In combination with any embodiment of the present application, the at least two first images include a second image, and the at least two first conversion relationships include a second conversion relationship corresponding to the second image;

[0039] The acquisition unit is specifically used for:

[0040] Acquire a third conversion relationship, where the third conversion relationship represents a conversion relationship between a pixel coordinate system of an image acquired by the image acquisition device and a capture device coordinate system, where the capture device coordinate system is a coordinate system constructed based on the image acquisition device;

[0041] Acquire a fourth conversion relationship between the position and posture of the target structure and the position and posture of the image acquisition device;

[0042] Acquire a first pose of the target structure in the base coordinate system when the image acquisition device acquires the second image;

[0043] The second conversion relationship is obtained based on the product of the third conversion relationship, the fourth conversion relationship and the first posture.

[0044] In combination with any implementation manner of the present application, the acquisition unit is further used to:

[0045] Acquire a third image, wherein the third image is an image acquired by the image acquisition device when the image acquisition device is fixedly connected to the target structure of the robotic arm, the third image includes the target tissue, and the acquisition time of the third image is later than the acquisition time of any one of the at least two first images;

[0046] Acquire a fifth transformation relationship between the second pixel coordinate system of the third image and the base coordinate system;

[0047] The determining unit is further used to determine a second position range of the third image in the base coordinate system based on the fifth conversion relationship and the first position range of the third image in the second pixel coordinate system;

[0048] The determination unit is further configured to determine the target tissue in the third image based on the target tissue located within the second position range in the three-dimensional model.

[0049] In combination with any embodiment of the present application, the target tissue includes tissue in a target organ of the object to be punctured, the target organ includes a lesion, and the third image includes the lesion;

[0050] The determination unit is further configured to determine a puncture path for the lesion based on the lesion and the target tissue in the third image.

[0051] In combination with any implementation manner of the present application, the determining unit is specifically configured to:

[0052] Based on the three-dimensional model, determining a third position of the target tissue within the range of the second position;

[0053] Acquire a second posture of the target structure when the image acquisition device acquires the third image;

[0054] Based on the product of the third conversion relationship, the fourth conversion relationship and the second posture, a sixth conversion relationship is obtained, where the sixth conversion relationship represents the conversion relationship between the second pixel coordinate system and the base coordinate system;

[0055] Determining a fourth position of the target tissue in the second pixel coordinate system based on the inverse relationship of the sixth conversion relationship and the third position;

[0056] Based on the fourth position, the target tissue in the third image is determined.

[0057] In combination with any embodiment of the present application, the target tissue includes a blood vessel, and the puncture path does not pass through the second position.

[0058] In a third aspect, an electronic device is provided, comprising: a processor and a memory, the memory being used to store computer program code, the computer program code comprising computer instructions, and when the processor executes the computer instructions, the electronic device executes the method as described in the first aspect above and any possible implementation thereof.

[0059] In a fourth aspect, another electronic device is provided, comprising: a processor, a sending device, an input device, an output device and a memory, wherein the memory is used to store computer program code, and the computer program code includes computer instructions. When the processor executes the computer instructions, the electronic device executes the method as described in the first aspect above and any possible implementation method thereof.

[0060] In a fifth aspect, a computer-readable storage medium is provided, in which a computer program is stored. The computer program includes program instructions, and when the program instructions are executed by a processor, the processor is caused to execute the method as described in the first aspect above and any possible implementation method thereof.

[0061] In a sixth aspect, a computer program product is provided, which includes a computer program or instructions, and when the computer program or instructions are run on a computer, the computer is enabled to execute the method of the above-mentioned first aspect and any possible implementation thereof.

[0062] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application.

[0063] In the present application, at least two first images are acquired by the image acquisition device when it is fixedly connected to the target structure of the manipulator, and at least two first images both include the target tissue. After acquiring at least two first images, the puncture device can obtain at least two first positions by performing the following steps on each of the at least two first images: determining the position of the target tissue in the first image in the first pixel coordinate system. After acquiring at least two first conversion relationships, the puncture device converts the at least two first positions into the position of the target tissue in the base coordinate system based on the at least two first conversion relationships, and obtains at least two second positions, wherein the first conversion relationship represents the conversion relationship between the first pixel coordinate system of the first image and the base coordinate system of the manipulator. Finally, the puncture device can obtain a three-dimensional model of the target tissue in the base coordinate system based on the at least two second positions. In this way, it is possible to obtain a three-dimensional model of the target tissue in the base coordinate system based on at least two two-dimensional images of the target tissue, thereby realizing three-dimensional reconstruction of the target tissue. BRIEF DESCRIPTION OF THE DRAWINGS

[0064] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the background technology, the drawings required for use in the embodiments of the present application or the background technology will be described below.

[0065] The drawings herein are incorporated into the specification and constitute a part of the specification. These drawings illustrate embodiments consistent with the present application and are used together with the specification to illustrate the technical solution of the present application.

[0066] Figure 1 A schematic diagram of a puncture method based on three-dimensional reconstruction provided in an embodiment of the present application;

[0067] Figure 2 A schematic diagram of a three-dimensional model of blood vessels in the kidney in a base coordinate system provided in an embodiment of the present application;

[0068] Figure 3a A schematic diagram of a display effect of a third image in a second pixel coordinate system provided by an embodiment of the present application;

[0069] Figure 3b A schematic diagram of a display effect of a third image in a base coordinate system provided in an embodiment of the present application;

[0070] Figure 4 A schematic diagram of an ultrasound probe provided in an embodiment of the present application;

[0071] Figure 5 A method for utilizing the embodiment of the present application Figure 4 A schematic diagram of an ultrasound image obtained by scanning with an ultrasound probe is shown;

[0072] Figure 6 A schematic diagram of a puncture scenario based on an ultrasound probe provided in an embodiment of the present application;

[0073] Figure 7 A schematic structural diagram of a puncture device based on three-dimensional reconstruction provided in an embodiment of the present application;

[0074] Figure 8 A schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0075] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0076] The terms "first", "second", etc. in the specification and claims of this application and the above-mentioned drawings are used to distinguish different objects, rather than to describe a specific order. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not limited to the listed steps or units, but optionally includes steps or units that are not listed, or optionally includes other steps or units inherent to these processes, methods, products or devices.

[0077] Mentioning "embodiment" in this article means that the specific features, structures or characteristics described in conjunction with the embodiment may be included in at least one embodiment of the present application. The appearance of this phrase in various places in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments. It should be understood that in this application, "at least one (item)" means one or more, "multiple" means two or more, and "at least two (items)" means two or three and more.

[0078] The execution subject of the embodiment of the present application is a puncture device based on three-dimensional reconstruction (hereinafter referred to as the puncture device), wherein the puncture device can be any electronic device that can execute the technical solution disclosed in the embodiment of the method of the present application. Optionally, the puncture device can be one of the following: a mobile phone, a computer, a tablet computer, and a wearable smart device.

[0079] It should be understood that the method embodiment of the present application can also be implemented by a processor executing a computer program code. The following describes the embodiment of the present application in conjunction with the drawings in the embodiment of the present application. Figure 1 , Figure 1 A schematic flow chart of a puncture method based on three-dimensional reconstruction provided in an embodiment of the present application.

[0080] 101. Acquire at least two first images captured by an image acquisition device when the image acquisition device is fixedly connected to a target structure of a robotic arm.

[0081] In an embodiment of the present application, the image acquisition device has the ability to acquire images of tissues inside the human body outside the human body. Optionally, the image acquisition device includes an ultrasonic probe, wherein the ultrasonic probe is used to acquire ultrasonic images. In this case, the first image is an ultrasonic image. The image acquisition device is fixedly connected to the target structure of the robotic arm, that is, the image acquisition device is fixed on the target structure of the robotic arm. The image acquisition device can be driven to move by controlling the movement of the robotic arm, and then the shooting angle of the image acquisition device can be adjusted, thereby adjusting the content of the image acquired by the image acquisition device. The target structure is the structure of the robotic arm. Optionally, the target structure is the structure of the robotic arm other than the base. For example, if the image acquisition device is fixed on the flange of the robotic arm, then the target structure is the flange.

[0082] In an embodiment of the present application, the number of first images is greater than 1, and each first image is acquired by the image acquisition device when it is fixedly connected to the target structure of the robotic arm. Optionally, the image acquisition device has different postures when acquiring any two first images. For example, at least two first images include a first image a and a first image b, wherein the posture of the image acquisition device when acquiring the first image a is posture c, and the posture of the image acquisition device when acquiring the first image b is posture d, and posture c is different from posture d. In a possible scenario, the image acquisition device is an ultrasound probe. The ultrasound probe is fixed on a robotic arm, and the robotic arm can be deployed in an operating room. By controlling the movement of the robotic arm through a device deployed outside the operating room, the image acquisition device can be driven to move, and then the shooting angle of the image acquisition device can be adjusted, so that the image acquisition device can shoot the object to be punctured from different shooting angles to obtain at least two first images.

[0083] In the embodiment of the present application, at least two first images include the target tissue, in other words, each first image includes the target tissue. The target tissue may be any tissue in the object to be punctured, for example, the target tissue is a blood vessel, another example, the target tissue is an organ, and another example, the target tissue is cartilage. Optionally, the target tissue is a tissue that is not expected to be punctured. In the case where the target tissue of the object to be punctured is pierced by the puncture needle, the object to be punctured will be damaged. For example, the object to be punctured is a patient. If the patient's blood vessel is pierced by the puncture needle, the patient will be damaged. If the patient's cartilage is pierced by the puncture needle, the human body will also be damaged. Therefore, the target tissue includes blood vessels and cartilage. Optionally, the lesion of the object to be punctured is located in the target organ, and the target tissue includes large blood vessels in the target organ, wherein the large blood vessels include large arteries and large veins. Since the damage to the large blood vessels may be fatal to the object to be punctured, it should be avoided as much as possible that the large blood vessels are pierced by the puncture needle during the puncture of the lesion of the target organ.

[0084] In an implementation of acquiring at least two first images, the puncture device receives at least two first images input by a user through an input component, wherein the input component includes: a mouse, a keyboard, a touch screen, a touch pad, and an audio input device.

[0085] In another implementation of acquiring at least two first images, the puncture device receives at least two first images sent by the user through a terminal, wherein the terminal includes: a mobile phone, a computer, a tablet computer, and a smart wearable device.

[0086] In yet another implementation of acquiring at least two first images, there is a communication connection between the puncture device and the image acquisition device, and the puncture device acquires the at least two first images acquired by the image acquisition device through the communication connection.

[0087] 102. Obtain at least two first conversion relationships.

[0088] In an embodiment of the present application, the first transformation relationship represents the transformation relationship between the first pixel coordinate system of the first image and the base coordinate system of the robotic arm, that is, the position in the first pixel coordinate system can be transformed into the position in the base coordinate system using the first transformation relationship, wherein the base coordinate system is a coordinate system constructed based on the base of the robotic arm.

[0089] Since the pixel coordinate systems of different first images are different when the image acquisition device acquires different first images at different postures, the first conversion relationship between the pixel coordinate systems of different first images and the base coordinate system is also different when the image acquisition device acquires different first images at different postures. In other words, the first conversion relationship corresponds one-to-one to the first image.

[0090] In one possible implementation, at least two first images include a second image, and at least two first transformation relationships include a second transformation relationship corresponding to the second image, wherein the second image is any one of the at least two first images, and the transformation relationship between the first pixel coordinate system of the second image and the base coordinate system is the second transformation relationship, that is, the position in the first pixel coordinate system can be converted into the position in the base coordinate system using the second transformation relationship.

[0091] The puncture device determines the second transformation relationship through the following steps: obtaining a third transformation relationship, wherein the third transformation relationship represents the transformation relationship between the pixel coordinate system of the image acquired by the image acquisition device and the acquisition device coordinate system, and the acquisition device coordinate system is a coordinate system constructed based on the image acquisition device. Obtaining a fourth transformation relationship between the posture of the target structure and the posture of the image acquisition device. Obtaining the third first posture of the target structure in the base coordinate system when the image acquisition device acquires the second image. Based on the product of the third transformation relationship, the fourth transformation relationship and the third first posture, the second transformation relationship is obtained.

[0092] Since the second image is an image captured by the image acquisition device, the conversion relationship between the pixel coordinate system of the second image and the acquisition device coordinate system is the third conversion relationship. In other words, the position of the second image in the first pixel coordinate system can be converted into the position in the acquisition device coordinate system using the third conversion relationship.

[0093] The fourth conversion relationship is a conversion relationship between the pose of the target structure and the pose of the image acquisition device, that is, the pose of the target structure can be converted into the pose of the image acquisition device using the fourth conversion relationship. For example, the pose of the target structure is pose A, and the pose of the image acquisition device can be obtained by converting pose A using the fourth conversion relationship. Optionally, the pose of the image acquisition device can be obtained by multiplying the pose of the target structure by the fourth conversion relationship.

[0094] The first position is the position of the target structure in the base coordinate system when the image acquisition device acquires the second image. Therefore, based on the product of the fourth conversion relationship and the first position, the position of the image acquisition device in the base coordinate system when acquiring the second image can be obtained. Because the acquisition device coordinate system is a coordinate system constructed based on the image acquisition device, the different positions of the image acquisition device in the base coordinate system will make the conversion relationship between the acquisition device coordinate system and the base coordinate system different. Therefore, the position of the image acquisition device in the base coordinate system can represent the conversion relationship between the acquisition device coordinate system and the base coordinate system. Based on this, when the third conversion relationship represents the conversion relationship between the pixel coordinate system of the image acquired by the image acquisition device and the acquisition device coordinate system, the second conversion relationship can be obtained based on the product of the third conversion relationship and the position of the image acquisition device in the base coordinate system when acquiring the second image. Therefore, the puncture device can obtain the second conversion relationship based on the product of the third conversion relationship, the fourth conversion relationship and the first position. Optionally, the puncture device determines the product of the third conversion relationship, the fourth conversion relationship and the first position to obtain the second conversion relationship.

[0095] It should be understood that in practical applications, for any one of the at least two first images, the puncture device can determine the first conversion relationship corresponding to the first image by the above method of determining the second conversion relationship of the second image, thereby obtaining at least two first conversion relationships.

[0096] 103. Obtain at least two first positions by performing the following steps on each of the at least two first images: determine the position of the target tissue in the first image in the first pixel coordinate system.

[0097] In the embodiment of the present application, the first position is the position of the target tissue in the first pixel coordinate system. For a first image, the puncture device can obtain a first position by determining the position of the target tissue in the first image in the first pixel coordinate system, and can obtain at least two first positions by determining the position of the target tissue in each first image in the first pixel coordinate system, that is, the first position corresponds to the first image one by one.

[0098] In one possible implementation, the puncture device determines pixels in the first image that are semantically target tissues by performing semantic segmentation on the first image, and then determines the position of the target tissue in the first pixel coordinate system based on the pixels that are semantically target tissues.

[0099] Optionally, the first image is a contrast image, wherein the contrast image includes information about the contrast agent, that is, the contrast image includes the development content of the contrast agent. Contrast agent (also called developer) is a chemical product injected into the tissue of the object to enhance the image observation effect, and the contrast agent is conducive to observing the tissue injected with the contrast agent in the contrast image. For example, the target tissue is a blood vessel, and the image acquisition device is an ultrasound probe. After the contrast agent is injected into the blood vessel of the object to be punctured, an ultrasound image including the blood vessel of the object to be punctured is acquired using an ultrasound probe, then in the ultrasound image, the artery injected with the contrast agent can be highlighted, that is, the acquired ultrasound image includes the development content of the contrast agent, and the ultrasound image at this time is a contrast image.

[0100] Since the target tissue in the contrast image can be highlighted by the contrast agent, after acquiring the contrast image, the puncture device determines the first position of the target tissue in the contrast image by performing semantic segmentation on the contrast image, thereby improving the accuracy of the first position and further improving the accuracy of at least two first positions.

[0101] 104. Based on at least two first conversion relationships, convert the at least two first positions into the positions of the target tissue in the base coordinate system to obtain at least two second positions.

[0102] Since the first conversion relationship represents the conversion relationship between the first pixel coordinate system of the first image and the base coordinate system of the robot, and the first position is the position of the target tissue in the first pixel coordinate system, the puncture device can convert the first position into the position of the target tissue in the base coordinate system based on the first conversion relationship to obtain the second position, that is, the second position represents the position of the target tissue in the base coordinate system. It should be understood that the second position obtained by the puncture device by executing step 104 corresponds to the first position one by one, that is, a first position can be converted into a second position based on a first conversion relationship.

[0103] 105. Based on the at least two second positions, obtain a three-dimensional model of the target tissue in the base coordinate system.

[0104] Since the target tissues in different first images are different, the positions of the target tissues in different first images in the base coordinate system are also different. Therefore, the puncture device can obtain the three-dimensional position of the target tissue in the base coordinate system based on the positions of the target tissues in different first images in the base coordinate system, and then obtain the three-dimensional model of the target tissue in the base coordinate system.

[0105] In one possible scenario, the image acquisition device is an ultrasound probe, and the target tissue is a blood vessel. At least two first images are at least two ultrasound images obtained by scanning the kidneys of the puncture subject with the ultrasound probe in different postures, wherein each first image includes blood vessels in the kidney, that is, at least two first images are obtained by scanning the blood vessels in the kidney from different angles using the ultrasound probe, and therefore different first images include different kidney contents. After obtaining at least two second positions based on steps 101 to 104, the puncture device can obtain a three-dimensional model of the blood vessels in the kidney in the base coordinate system based on the two second positions. For example, Figure 2 A schematic diagram of a three-dimensional model of blood vessels in the kidney in a base coordinate system provided in an embodiment of the present application.

[0106] In an embodiment of the present application, at least two first images are acquired by the image acquisition device when it is fixedly connected to the target structure of the robotic arm, and at least two first images both include the target tissue. After acquiring at least two first images, the puncture device can obtain at least two first positions by performing the following steps on each of the at least two first images: determining the position of the target tissue in the first image in the first pixel coordinate system. After acquiring at least two first conversion relationships, the puncture device converts the at least two first positions into the position of the target tissue in the base coordinate system based on the at least two first conversion relationships, and obtains at least two second positions, wherein the first conversion relationship represents the conversion relationship between the first pixel coordinate system of the first image and the base coordinate system of the robotic arm. Finally, the puncture device can obtain a three-dimensional model of the target tissue in the base coordinate system based on the at least two second positions. In this way, it is possible to obtain a three-dimensional model of the target tissue in the base coordinate system based on at least two two-dimensional images of the target tissue, thereby realizing three-dimensional reconstruction of the target tissue.

[0107] As an optional implementation, after obtaining the three-dimensional model of the target tissue in the base coordinate system, the puncture device further performs the following steps:

[0108] 201. Acquire a third image.

[0109] In an embodiment of the present application, the third image is an image captured by the image acquisition device when it is fixedly connected to the target structure of the robotic arm, the third image includes the target tissue, and the acquisition time of the third image is later than the acquisition time of any one of the at least two first images.

[0110] 202. Obtain a fifth transformation relationship between the second pixel coordinate system of the third image and the base coordinate system.

[0111] In the embodiment of the present application, the pixel coordinate system of the third image is the second pixel coordinate system. The fifth transformation relationship represents the transformation relationship between the second pixel coordinate system and the base coordinate system of the robot, that is, the position in the second pixel coordinate system can be transformed into the position in the base coordinate system using the fifth transformation relationship.

[0112] 203. Determine a second position range of the third image in the base coordinate system based on the fifth conversion relationship and the first position range of the third image in the second pixel coordinate system.

[0113] In the embodiment of the present application, the first position range is the position range of the image content of the third image in the second pixel coordinate system. For example, the minimum horizontal coordinate of the pixels in the third image in the second pixel coordinate system is 0, the maximum horizontal coordinate is 100, the minimum vertical coordinate is 0, and the maximum vertical coordinate is 80. Then the first position range is a horizontal coordinate of 0 to 100, and a vertical coordinate of 0 to 80. Based on the fifth conversion relationship, the puncture device can convert the first position range into a position range in the base coordinate system, that is, the second position range. Since the first position range is the position range of the image content of the third image in the second pixel coordinate system, the second position range is the position range of the image content of the third image in the base coordinate system.

[0114] For a better understanding of the first position range of the third image in the second pixel coordinate system and the second position range of the third coordinate system in the base coordinate system, please refer to Figure 3a and Figure 3b ,in, Figure 3a A schematic diagram of a display effect of a third image in a second pixel coordinate system provided in an embodiment of the present application, Figure 3b A schematic diagram of the display effect of a third image in a base coordinate system provided in an embodiment of the present application. Figure 3a and Figure 3b In the above figure, the third image is an ultrasound image. Figure 3a As shown, the second pixel coordinate system (i.e., the pixel coordinate system of the ultrasound image) is 1 X 1 Y 1 , where O 1 is the origin of the second pixel coordinate system, X 1 is the horizontal axis of the second pixel coordinate system, Y 1 is the vertical axis of the second pixel coordinate system. Figure 3a FIG. 4 shows the display effect of the third image in the two-dimensional plane represented by the second pixel coordinate system, and the position range of the third image in the two-dimensional plane is the first position range. Figure 3b As shown, the base coordinate system is O 2 X 2 Y 2 Z 2 , where O2 is the origin of the base coordinate system, X 2 is the horizontal axis of the base coordinate system, Y 2 is the vertical axis of the base coordinate system, Z 2 is the vertical axis of the base coordinate system. Figure 3b The display effect of the third image in the three-dimensional space represented by the base coordinate system is shown, and the position range of the third image in the three-dimensional space is the second position range.

[0115] 204. Determine the target tissue in the third image based on the target tissue located within the second position range in the three-dimensional model.

[0116] Since the target tissue in the third image is within the position range of the image content of the third image, the target tissue in the second position range in the three-dimensional model is the target tissue in the third image. Therefore, the puncture device can determine the target tissue in the third image based on the target tissue in the second position range in the three-dimensional model.

[0117] In one possible implementation, the puncture device determines a third position of the target tissue within the second position range based on the three-dimensional model. A second posture of the target structure when the image acquisition device acquires the third image is obtained. A sixth transformation relationship is obtained based on the product of the third transformation relationship, the fourth transformation relationship, and the second posture, wherein the sixth transformation relationship represents the transformation relationship between the second pixel coordinate system and the base coordinate system. Based on the inverse relationship of the sixth transformation relationship and the third position, a fourth position of the target tissue in the second pixel coordinate system is determined. Based on the fourth position, the target tissue in the third image is determined.

[0118] In this implementation, the target tissue within the second position range is the target tissue in the third image, and the position of the target tissue within the second position range in the base coordinate system is the third position, that is, the position of the target tissue in the third image in the base coordinate system is the third position.

[0119] The puncture device can determine the sixth transformation relationship based on the product of the third transformation relationship, the fourth transformation relationship and the second posture, wherein the sixth transformation relationship represents the transformation relationship between the second pixel coordinate system and the base coordinate system. The implementation principle of this step can refer to the implementation principle of "obtaining the second transformation relationship based on the product of the third transformation relationship, the fourth transformation relationship and the third and first postures" in step 102. That is to say, the position in the second pixel coordinate system can be converted to the position in the base coordinate system using the sixth relationship, and correspondingly, the position in the base coordinate system can be converted to the position in the second pixel coordinate system using the inverse relationship of the sixth relationship. Optionally, the sixth transformation relationship is a matrix, and the inverse relationship of the sixth transformation relationship is the inverse matrix of the matrix.

[0120] Therefore, after obtaining the sixth conversion relationship and determining the third position, the puncture device can determine the fourth position of the target tissue in the second pixel coordinate system based on the inverse relationship of the sixth conversion relationship and the third position. Specifically, the fourth position can be obtained by converting the third position into the position of the second pixel coordinate system using the inverse relationship of the sixth conversion relationship. Finally, based on the fourth position, the target tissue in the third image can be determined. Optionally, the image content at the fourth position in the third image is the target tissue. Optionally, after determining the target tissue in the third image, the target tissue in the third image can be highlighted, which is conducive to relevant personnel observing the target tissue in the third image.

[0121] In this embodiment, the third image is an image captured by the image acquisition device when it is fixedly connected to the target structure of the robotic arm, the third image includes the target tissue, and the acquisition time of the third image is later than the acquisition time of any one of the at least two first images. The fifth transformation relationship is the transformation relationship between the second pixel coordinate system of the third image and the base coordinate system. After acquiring the third image and the fifth transformation relationship, the puncture device determines the second position range of the third image in the base coordinate system based on the fifth transformation relationship and the first position range of the third image in the second pixel coordinate system. Furthermore, the target tissue in the third image can be determined based on the target tissue located within the second position range in the three-dimensional model. In this way, when the image captured by the image acquisition device does not include the image content of the target tissue, the position of the target tissue in the image captured by the image acquisition device can be determined based on the three-dimensional model.

[0122] As an optional embodiment, the target tissue includes tissue in a target organ of the object to be punctured, wherein the target organ includes a lesion, and the third image includes the lesion. For example, the target organ is a kidney, the target tissue is tissue in the kidney, and the lesion is a lesion in the kidney. In this embodiment, after executing step 204, the puncture device further performs the following steps:

[0123] 301. Determine a puncture path for the lesion based on the lesion and the target tissue in the third image.

[0124] After determining the third position of the target tissue in the third image, the puncture device can avoid the target tissue when planning the puncture path for the lesion, thereby reducing the probability of the puncture needle piercing the target tissue when the lesion is subsequently punctured based on the puncture path.

[0125] In a possible scenario, the target tissue is a blood vessel in the kidney of the object to be punctured, wherein the kidney is an organ in the body of the object to be punctured that needs to be punctured. The image acquisition device is an ultrasound probe, and at least two first images are at least two ultrasound images acquired by the ultrasound probe before the puncture operation is performed on the kidney of the object to be punctured. The third image is an ultrasound image acquired by the ultrasound probe during the puncture operation on the kidney of the object to be punctured. Before the puncture operation, a three-dimensional model of the blood vessels in the kidney in the base coordinate system can be constructed based on the at least two first images. Then, during the puncture operation, the blood vessels in the kidney in the third image can be determined based on the three-dimensional model. In this way, even if the third image only includes the kidney but not the blood vessels in the kidney, the position of the blood vessels in the kidney can be displayed in the third image, which is conducive to puncturing the kidney based on the third image. For example, before performing a puncture operation on a subject to be punctured, a contrast agent may be injected into the subject to be punctured, and then during the development time of the contrast agent, an ultrasonic probe may be used to scan the blood vessels to obtain at least two first images, so that each of the at least two first images is a contrast image, and each first image includes blood vessels in the kidney, and then a three-dimensional model of the blood vessels in the kidney in the base coordinate system may be constructed based on the at least two first images. During the puncture operation, the puncture device may determine the blood vessels in the kidney in the third image based on the three-dimensional model of the blood vessels in the kidney in the base coordinate system, so that when planning the puncture path based on the third image, the puncture path may be planned based on the position of the blood vessels in the kidney, so that the puncture path avoids the blood vessels in the kidney.

[0126] As an optional embodiment, the image acquisition device includes an ultrasonic probe, and the ultrasonic probe emits ultrasonic waves at a minimum emission angle passing through the origin of the acquisition device coordinate system, or the ultrasonic probe emits ultrasonic waves at a maximum emission angle passing through a reference point. For example, the ultrasonic probe emits ultrasonic waves at an emission angle range of 10 degrees to 90 degrees. Then, when the ultrasonic probe emits ultrasonic waves at an emission angle of 10 degrees, the emitted ultrasonic waves pass through the origin of the acquisition device coordinate system, or when the ultrasonic probe emits ultrasonic waves at an emission angle of 90 degrees, the emitted ultrasonic waves pass through the origin of the acquisition device coordinate system.

[0127] For example, Figure 4 A schematic diagram of an ultrasonic probe provided in an embodiment of the present application. Figure 4 In the ultrasound probe, the acoustic lens is located at the outermost side of the ultrasound probe. The acoustic lens is used to focus the ultrasound emitted by the ultrasound probe, thereby enhancing the penetration and sensitivity of the ultrasound. Figure 4 As shown, the origin of the acquisition device coordinate system is a point on the boundary of the acoustic lens of the ultrasonic probe, that is, the ultrasonic wave emitted by the ultrasonic probe at the minimum emission angle passes through the origin of the acquisition device coordinate system, or the ultrasonic wave emitted by the ultrasonic probe at the maximum emission angle passes through the origin of the acquisition device coordinate system.

[0128] Figure 5 A method for utilizing the embodiment of the present application Figure 4 Schematic diagram of an ultrasound image obtained by scanning with an ultrasound probe as shown, Figure 5 It is the display effect of the ultrasound image scanned by the ultrasound probe in B-ultrasound mode. Figure 5 In the figure, the pixel coordinate system of the ultrasound image is O 1 X 1 Y 1 , where O 1 is the origin of the pixel coordinate system of the ultrasound image, X 1 Y is the horizontal axis of the pixel coordinate system of the ultrasound image. 1 is the longitudinal axis of the pixel coordinate system of the ultrasound image, and the origin of the acquisition device coordinate system is the point with the largest longitudinal axis in the fan-shaped scanning area of ​​the ultrasound image, that is, the origin of the acquisition device coordinate system is the end point of the fan-shaped scanning area. It should be understood that since the acoustic lens of the ultrasound probe is an outwardly convex curved surface, in the ultrasound image scanned by the ultrasound probe, the side with the largest longitudinal coordinate in the scanning area (hereinafter referred to as the top side) is a curved surface. Moreover, for the point within the top side, the larger the longitudinal coordinate in the pixel coordinate system of the ultrasound image, the larger the emission angle of the ultrasound scanned at that point, or the larger the longitudinal coordinate in the pixel coordinate system of the ultrasound image, the smaller the emission angle of the ultrasound scanned at that point. Therefore Figure 4 The ultrasonic probe in the Figure 5 The origin of the acquisition device coordinate system in , or Figure 4 The ultrasonic probe in the Figure 5 The origin of the acquisition device coordinate system in .

[0129] Optionally, the angle between the horizontal axis of the pixel coordinate system of the ultrasound image collected by the ultrasound probe and the horizontal axis of the acquisition device coordinate system is 180 degrees, and the angle between the vertical axis of the pixel coordinate system of the ultrasound image collected by the ultrasound probe and the vertical axis of the acquisition device coordinate system is 180 degrees. Therefore, the rotation relationship between the pixel coordinate system of the ultrasound image collected by the ultrasound probe and the acquisition device coordinate system includes: rotating the horizontal axis of the pixel coordinate system and the vertical axis of the pixel coordinate system by 180 degrees around the origin of the pixel coordinate system, respectively, so that the horizontal axis of the pixel coordinate system is parallel to the horizontal axis of the acquisition device coordinate system, and the vertical axis of the pixel coordinate system is parallel to the vertical axis of the acquisition device coordinate system.

[0130] Since the ultrasonic image collected by the ultrasonic probe includes the origin of the acquisition device coordinate system, the translation relationship between the pixel coordinate system of the ultrasonic image collected by the ultrasonic probe and the acquisition device coordinate system can be obtained by aligning the origin of the acquisition device coordinate system in the ultrasonic image with the origin of the acquisition device coordinate system in the ultrasonic probe, and then the conversion relationship between the pixel coordinate system of the ultrasonic image collected by the ultrasonic probe and the acquisition device coordinate system can be determined based on the rotation relationship between the pixel coordinate system and the acquisition device coordinate system, and the translation relationship between the pixel coordinate system and the acquisition device coordinate system, which is the third conversion relationship mentioned above. This can improve the efficiency of determining the third conversion relationship.

[0131] Optionally, the puncture device obtains the fifth position and the seventh transformation relationship of the origin of the acquisition device coordinate system in the pixel coordinate system of the ultrasound image acquired by the ultrasound probe, wherein the seventh transformation relationship is the transformation relationship between the pixel coordinate system of the ultrasound image acquired by the ultrasound probe and the world coordinate system. The fifth position is converted into the sixth position of the origin of the acquisition device coordinate system in the world coordinate system using the seventh transformation relationship. Then the seventh position is obtained, wherein the seventh position is the position of the origin of the acquisition device coordinate system in the ultrasound probe in the world coordinate system determined by measurement. Based on the displacement between the sixth position and the seventh position, the translation relationship between the pixel coordinate system of the ultrasound image acquired by the ultrasound probe and the acquisition device coordinate system is determined.

[0132] As an optional implementation, in this scenario, after the puncture path is determined, puncture can be further performed based on the puncture path. Figure 6 , Figure 6 A schematic diagram of a puncture scenario based on an ultrasound probe provided in an embodiment of the present application. Figure 6 As shown, the scene includes an ultrasound probe, a mechanical arm, and a needle insertion device, wherein the ultrasound probe and the needle insertion device are both fixed on the mechanical arm, and specifically, the ultrasound probe is fixedly connected to the target structure of the mechanical arm. The ultrasound probe is used to obtain an ultrasound image (i.e., a third image) of the kidney of the object to be punctured by scanning and collecting the kidney of the object to be punctured, and the needle insertion device is used to control the puncture needle to perform puncture.

[0133] like Figure 6 As shown, the scanning area of ​​the ultrasound probe is a sector-shaped area. When the ultrasound probe scans the object to be punctured, the information in the scanning area can be imaged to obtain an ultrasound image. Figure 4 The acquisition device coordinate system is also shown, specifically, 3 X 3 Y 3 Z 3 Represents the acquisition device coordinate system, where O 3 is the origin of the acquisition device coordinate system, X 3 Y is the horizontal axis of the acquisition device coordinate system.3 is the vertical axis of the acquisition device coordinate system, Z 3 is the vertical axis of the acquisition device coordinate system. The origin of the acquisition device coordinate system is located in the ultrasonic probe as follows: Figure 4 As shown. Based on the acquisition device coordinate system and the scheme described above, a three-dimensional model of the target tissue in the base coordinate system can be constructed. After obtaining the ultrasound image, the position of the target tissue in the ultrasound image can be determined based on the three-dimensional model, and then the needle insertion point can be determined based on the position of the target tissue in the ultrasound image, and then the puncture path can be determined based on the needle insertion point and the lesion. Then, based on the puncture path, the robotic arm is controlled to drive the needle insertion device to move, so that the needle insertion device controls the puncture needle to puncture along the puncture path.

[0134] Those skilled in the art will appreciate that, in the above method of specific implementation, the order in which the steps are written does not imply a strict execution order and does not constitute any limitation on the implementation process. The specific execution order of the steps should be determined by their functions and possible internal logic.

[0135] If the technical solution of this application involves personal information, the product using the technical solution of this application has clearly informed the personal information processing rules and obtained the individual's voluntary consent before processing the personal information. If the technical solution of this application involves sensitive personal information, the product using the technical solution of this application has obtained the individual's separate consent before processing the sensitive personal information, and at the same time meets the "explicit consent" requirement. For example, on personal information collection devices such as cameras, clear and prominent signs are set to inform that the personal information collection scope has been entered and personal information will be collected. If the individual voluntarily enters the collection scope, it is deemed that he or she agrees to the collection of his or her personal information; or on the device that processes personal information, the personal information processing rules are notified by obvious signs / information, and the individual's authorization is obtained through pop-up information or by asking the individual to upload his or her personal information; among which, personal information processing may include information such as the personal information processor, the purpose of personal information processing, the processing method, and the type of personal information processed.

[0136] The method of the embodiment of the present application is described in detail above, and the device of the embodiment of the present application is provided below.

[0137] See also Figure 7 , Figure 7 A schematic diagram of the structure of a puncture device based on three-dimensional reconstruction provided in an embodiment of the present application, wherein the puncture device based on three-dimensional reconstruction 1 comprises: an acquisition unit 11, a determination unit 12, a conversion unit 13, and a processing unit 14, wherein:

[0138] An acquisition unit 11, used to acquire at least two first images acquired by an image acquisition device when the image acquisition device is fixedly connected to a target structure of the robot arm, wherein the at least two first images both include the target tissue;

[0139] The acquisition unit 11 is further used to acquire at least two first conversion relationships, where the first conversion relationships correspond to the first images one by one, and the first conversion relationships represent the conversion relationship between the first pixel coordinate system of the first image and the base coordinate system of the robotic arm;

[0140] The determination unit 12 is configured to obtain at least two first positions by performing the following steps on each of the at least two first images: determining a position of the target tissue in the first image in the first pixel coordinate system;

[0141] A conversion unit 13, configured to convert the at least two first positions into positions of the target tissue in the base coordinate system based on at least two first conversion relationships, to obtain at least two second positions;

[0142] The processing unit 14 is configured to obtain a three-dimensional model of the target tissue in the base coordinate system based on the at least two second positions.

[0143] In combination with any embodiment of the present application, the at least two first images include a second image, and the at least two first conversion relationships include a second conversion relationship corresponding to the second image;

[0144] The acquisition unit 11 is specifically used for:

[0145] Acquire a third conversion relationship, where the third conversion relationship represents a conversion relationship between a pixel coordinate system of an image acquired by the image acquisition device and a capture device coordinate system, where the capture device coordinate system is a coordinate system constructed based on the image acquisition device;

[0146] Acquire a fourth conversion relationship between the position and posture of the target structure and the position and posture of the image acquisition device;

[0147] Acquire a first pose of the target structure in the base coordinate system when the image acquisition device acquires the second image;

[0148] The second conversion relationship is obtained based on the product of the third conversion relationship, the fourth conversion relationship and the first posture.

[0149] In combination with any embodiment of the present application, the acquisition unit 11 is further used for:

[0150] Acquire a third image, wherein the third image is an image acquired by the image acquisition device when the image acquisition device is fixedly connected to the target structure of the robotic arm, the third image includes the target tissue, and the acquisition time of the third image is later than the acquisition time of any one of the at least two first images;

[0151] Acquire a fifth transformation relationship between the second pixel coordinate system of the third image and the base coordinate system;

[0152] The determining unit 12 is further configured to determine a second position range of the third image in the base coordinate system based on the fifth conversion relationship and the first position range of the third image in the second pixel coordinate system;

[0153] The determination unit 12 is further configured to determine the target tissue in the third image based on the target tissue located within the second position range in the three-dimensional model.

[0154] In combination with any embodiment of the present application, the target tissue includes tissue in a target organ of the object to be punctured, the target organ includes a lesion, and the third image includes the lesion;

[0155] The determining unit 12 is further configured to determine a puncture path for the lesion based on the lesion and the target tissue in the third image.

[0156] In combination with any implementation manner of the present application, the determining unit 12 is specifically configured to:

[0157] Based on the three-dimensional model, determining a third position of the target tissue within the range of the second position;

[0158] Acquire a second posture of the target structure when the image acquisition device acquires the third image;

[0159] Based on the product of the third conversion relationship, the fourth conversion relationship and the second posture, a sixth conversion relationship is obtained, where the sixth conversion relationship represents the conversion relationship between the second pixel coordinate system and the base coordinate system;

[0160] Determining a fourth position of the target tissue in the second pixel coordinate system based on the inverse relationship of the sixth conversion relationship and the third position;

[0161] Based on the fourth position, the target tissue in the third image is determined.

[0162] In combination with any embodiment of the present application, the target tissue includes a blood vessel, and the puncture path does not pass through the second position.

[0163] In an embodiment of the present application, at least two first images are acquired by the image acquisition device when it is fixedly connected to the target structure of the robotic arm, and at least two first images both include the target tissue. After acquiring at least two first images, the puncture device can obtain at least two first positions by performing the following steps on each of the at least two first images: determining the position of the target tissue in the first image in the first pixel coordinate system. After acquiring at least two first conversion relationships, the puncture device converts the at least two first positions into the position of the target tissue in the base coordinate system based on the at least two first conversion relationships, and obtains at least two second positions, wherein the first conversion relationship represents the conversion relationship between the first pixel coordinate system of the first image and the base coordinate system of the robotic arm. Finally, the puncture device can obtain a three-dimensional model of the target tissue in the base coordinate system based on the at least two second positions. In this way, it is possible to obtain a three-dimensional model of the target tissue in the base coordinate system based on at least two two-dimensional images of the target tissue, thereby realizing three-dimensional reconstruction of the target tissue.

[0164] In some embodiments, the functions or modules included in the device provided in the embodiments of the present application can be used to execute the method described in the above method embodiments. The specific implementation can refer to the description of the above method embodiments. For the sake of brevity, it will not be repeated here.

[0165] Figure 8 A schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present application. The electronic device 2 includes a processor 21 and a memory 22. Optionally, the electronic device 2 also includes an input device 23 and an output device 24. The processor 21, the memory 22, the input device 23 and the output device 24 are coupled via a connector, and the connector includes various interfaces, transmission lines or buses, etc., which are not limited in the embodiments of the present application. It should be understood that in each embodiment of the present application, coupling refers to mutual connection in a specific manner, including direct connection or indirect connection through other devices, for example, it can be connected through various interfaces, transmission lines, buses, etc.

[0166] The processor 21 may be one or more graphics processing units (GPUs). When the processor 21 is a GPU, the GPU may be a single-core GPU or a multi-core GPU. Optionally, the processor 21 may be a processor group consisting of multiple GPUs, and the multiple processors are coupled to each other via one or more buses. Optionally, the processor may also be other types of processors, etc., which are not limited in the embodiments of the present application.

[0167] The memory 22 can be used to store computer program instructions and various computer program codes including the program code for executing the program code of the present application. Optionally, the memory includes but is not limited to random access memory (RAM), read-only memory (ROM), erasable programmable read only memory (EPROM), or portable read only memory (CD-ROM), which is used for related instructions and data.

[0168] The input device 23 is used to input data and / or signals, and the output device 24 is used to output data and / or signals. The input device 23 and the output device 24 can be independent devices or an integrated device.

[0169] It can be understood that in the embodiment of the present application, the memory 22 can be used not only to store relevant instructions, but also to store relevant data. For example, the memory 22 can be used to store at least two first images obtained through the input device 23, or the memory 22 can also be used to store a three-dimensional model obtained by the processor 21, etc. The embodiment of the present application does not limit the specific data stored in the memory.

[0170] Understandably, Figure 8 Only a simplified design of an electronic device is shown. In practical applications, the electronic device may also include other necessary components, including but not limited to any number of input / output devices, processors, memories, etc., and all electronic devices that can implement the embodiments of the present application are within the protection scope of the present application.

[0171] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0172] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments, and will not be repeated here. Those skilled in the art can also clearly understand that the descriptions of the various embodiments of the present application have different focuses. For the convenience and brevity of description, the same or similar parts may not be repeated in different embodiments. Therefore, for parts not described or not described in detail in a certain embodiment, refer to the records of other embodiments.

[0173] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0174] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0175] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0176] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented by software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted through the computer-readable storage medium. The computer instructions may be transmitted from a website site, computer, server or data center to another website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The available medium may be a magnetic medium (eg, a floppy disk, a hard disk, a magnetic tape), an optical medium (eg, a digital versatile disc (DVD)), or a semiconductor medium (eg, a solid state disk (SSD)).

[0177] A person skilled in the art can understand that to implement all or part of the processes in the above-mentioned embodiments, the processes can be completed by a computer program to instruct the relevant hardware, and the program can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the above-mentioned method embodiments. The aforementioned storage medium includes: a read-only memory (ROM) or a random access memory (RAM), a magnetic disk or an optical disk, and other media that can store program codes.

Claims

1. A puncture path planning method based on three-dimensional reconstruction, characterized in that: The method comprises: Acquire at least two first images acquired by the image acquisition device when the image acquisition device is fixedly connected to the target structure of the robotic arm, wherein the at least two first images both include the target tissue; Acquire at least two first conversion relationships, where the first conversion relationships correspond to the first image one by one, and the first conversion relationships represent a conversion relationship between a first pixel coordinate system of the first image and a base coordinate system of the robotic arm; Acquire a third conversion relationship, where the third conversion relationship represents a conversion relationship between a pixel coordinate system of an image acquired by the image acquisition device and a capture device coordinate system, where the capture device coordinate system is a coordinate system constructed based on the image acquisition device; Acquire a fourth conversion relationship between the position and posture of the target structure and the position and posture of the image acquisition device; At least two first positions are obtained by performing the following steps on each of the at least two first images: determining the position of the target tissue in the first image in the first pixel coordinate system; Based on at least two first conversion relationships, converting the at least two first positions into positions of the target tissue in the base coordinate system to obtain at least two second positions; Based on the at least two second positions, obtaining a three-dimensional model of the target tissue in the base coordinate system; Acquire a third image, wherein the third image is an image acquired by the image acquisition device when the image acquisition device is fixedly connected to the target structure of the robotic arm, the third image includes the target tissue, and the acquisition time of the third image is later than the acquisition time of any one of the at least two first images; Acquire a fifth transformation relationship between the second pixel coordinate system of the third image and the base coordinate system; Determine a second position range of the third image in the base coordinate system based on the fifth conversion relationship and the first position range of the third image in the second pixel coordinate system; Based on the three-dimensional model, determine the third position of the target tissue within the second position range; obtain the second posture of the target structure when the image acquisition device acquires the third image; based on the product of the third transformation relationship, the fourth transformation relationship and the second posture, obtain a sixth transformation relationship, and the sixth transformation relationship represents the transformation relationship between the second pixel coordinate system and the base coordinate system; based on the inverse relationship of the sixth transformation relationship and the third position, determine the fourth position of the target tissue in the second pixel coordinate system; based on the fourth position, determine the target tissue in the third image.

2. The method according to claim 1, characterized in that The at least two first images include a second image, and the at least two first conversion relationships include a second conversion relationship corresponding to the second image; The obtaining of at least two first conversion relationships includes: Acquire a first pose of the target structure in the base coordinate system when the image acquisition device acquires the second image; The second conversion relationship is obtained based on the product of the third conversion relationship, the fourth conversion relationship and the first posture.

3. The method according to claim 1, characterized in that The target tissue includes tissue in a target organ of the object to be punctured, the target organ includes a lesion, and the third image includes the lesion; After determining the target tissue in the third image based on the target tissue located within the second position range in the three-dimensional model, the method further includes: Based on the lesion and the target tissue in the third image, a puncture path for the lesion is determined.

4. The method according to claim 3, characterized in that The target tissue includes a blood vessel, and the puncture path does not pass through the second position.

5. A puncture path planning device based on three-dimensional reconstruction, characterized in that: include: an acquisition unit, configured to acquire at least two first images acquired by the image acquisition device when the image acquisition device is fixedly connected to the target structure of the robot arm, wherein the at least two first images both include the target tissue; The acquisition unit is further used to acquire at least two first conversion relationships, where the first conversion relationships correspond to the first images one by one, and the first conversion relationships represent the conversion relationship between the first pixel coordinate system of the first image and the base coordinate system of the robotic arm; The acquisition unit is further used to acquire a third conversion relationship, wherein the third conversion relationship represents a conversion relationship between a pixel coordinate system of the image acquired by the image acquisition device and a collection device coordinate system, wherein the collection device coordinate system is a coordinate system constructed based on the image acquisition device; The acquisition unit is further used to acquire a fourth conversion relationship between the posture of the target structure and the posture of the image acquisition device; A determination unit, configured to obtain at least two first positions by performing the following steps on each of the at least two first images: determining a position of the target tissue in the first image in the first pixel coordinate system; A conversion unit, configured to convert the at least two first positions into positions of the target tissue in the base coordinate system based on at least two first conversion relationships, to obtain at least two second positions; A processing unit, configured to obtain a three-dimensional model of the target tissue in the base coordinate system based on the at least two second positions; The acquisition unit is further used to acquire a third image, wherein the third image is an image acquired by the image acquisition device when the image acquisition device is fixedly connected to the target structure of the robotic arm, the third image includes the target tissue, and the acquisition time of the third image is later than the acquisition time of any one of the at least two first images; The acquisition unit is further used to acquire a fifth transformation relationship between the second pixel coordinate system of the third image and the base coordinate system; The determining unit is further used to determine a second position range of the third image in the base coordinate system based on the fifth conversion relationship and the first position range of the third image in the second pixel coordinate system; The determination unit is further used to determine a third position of the target tissue located within the second position range based on the three-dimensional model; and obtain a second posture of the target structure when the image acquisition device acquires the third image; Based on the product of the third conversion relationship, the fourth conversion relationship and the second posture, a sixth conversion relationship is obtained, where the sixth conversion relationship represents the conversion relationship between the second pixel coordinate system and the base coordinate system; Based on the inverse relationship of the sixth conversion relationship and the third position, a fourth position of the target tissue in the second pixel coordinate system is determined; based on the fourth position, the target tissue in the third image is determined.

6. An electronic device, characterized in that: include: A processor and a memory, wherein the memory is used to store computer program codes, wherein the computer program codes include computer instructions, and when the processor executes the computer instructions, the electronic device executes the method according to any one of claims 1 to 4.

7. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, wherein the computer program includes program instructions. When the program instructions are executed by a processor, the processor is caused to execute the method according to any one of claims 1 to 4.

8. A computer program product, characterized in that The computer program product comprises a computer program; when the computer program is run on a computer, the computer is caused to execute the method according to any one of claims 1 to 4.

Images