SURGERY AID METHOD AND ASSOCIATED SYSTEM

Abstract

SURGERY AID METHOD AND ASSOCIATED SYSTEM The invention relates to a computer-implemented method (100) for assisting surgery comprising the following steps: Receiving (110) a medical image (10) of a predetermined body portion (11) of an individual in a first position; Receiving (121) an optical and / or three-dimensional image (25) of the body portion in a second position; Generating (140) a 3D model (14) comprising at least one set of anatomical elements (16) of said body portion of the individual; Generating (160) a virtual attribute (15), characteristic of the position and orientation of a selected anatomical element; Generating and transmitting (170) a display instruction to an augmented reality display to project a virtual representation of the virtual attribute (15). Figure for abstract: Fig.2

Description

Title of the invention: SURGERY AID METHOD AND ASSOCIATED SYSTEM Field of invention

[0001] The invention relates to a computer-implemented method for assisting surgery, in particular a method for generating a display instruction for a hand axis on an augmented reality device. The invention also relates to a system or computer program for implementing said method as well as a memory comprising said computer program. State of the art

[0002] During certain surgeries, certain anatomical parts of the patient such as bones are invisible to the surgeon.

[0003] Certain technologies make it possible, using an augmented reality headset worn by the surgeon, to superimpose virtual characteristics corresponding to physical characteristics hidden by the patient's anatomy on what the surgeon sees.

[0004] For example, WO2018164909 is known, proposing a solution for displaying knee ligaments through the patient's skin on the display of an augmented reality device to enable the surgeon to visualize these ligaments during an operation. The size and shape of the virtual ligaments can be modified based on the detection of movement of adjacent bones.

[0005] A disadvantage of this type of solution is that it does not allow the generation of surgery assistance images for body parts more complex than the knee such as the hand or the foot.

[0006] There is therefore a need for a method and a system for assisting surgery making it possible to represent physical characteristics of a portion of the body such as the hand or the foot. Summary of the invention

[0007] According to one aspect, the invention relates to a computer-implemented method or method for assisting surgery.

[0008] The method comprises receiving a medical image of a predetermined body portion of an individual in a first position.

[0009] The method comprises receiving an optical and / or three-dimensional image of said body portion in a second position different from the first position and determining position data of said body portion characteristic of said second position from said received image.

[0010] The method comprises comparing the second position with the first position to generate a transformation parameter.

[0011] The method comprises generating a 3D model of an anatomical structure of said portion of the body, associated with the second position, from the received medical image and from said transformation parameter.

[0012] In one embodiment, said anatomical structure comprises at least one set of anatomical elements of said portion of the individual's body.

[0013] In one embodiment, the method comprises selecting at least one anatomical element from said set of anatomical elements of said generated 3D model.

[0014] In one embodiment, the method comprises generating a virtual attribute, said virtual attribute comprising an element characteristic of the position and orientation of said selected anatomical element in the 3D model.

[0015] An advantage of the virtual attribute is to provide the practitioner with information on the position and / or orientation of an anatomical element of the subject not visible to the naked eye. This virtual attribute can thus be displayed on an augmented reality device in the practitioner's field of vision.

[0016] In one embodiment, the geometric element of the virtual attribute comprises an axis or a plane whose location and orientation are generated as a function of the position and orientation of said anatomical element.

[0017] An advantage of an axis or plane is to allow the practitioner to follow the orientation of an element, for example for the alignment or placement of a surgical tool.

[0018] In one embodiment, the method comprises generating and transmitting a display instruction to an augmented reality display to project a virtual representation of the generated virtual attribute into a surgical field comprising said body portion of said individual while allowing the surgical field to be viewed through the augmented reality display.

[0019] In one embodiment, the display instruction comprises an instruction to project a virtual representation of at least a portion of the 3D model into the surgical field by allowing the surgical field to be viewed through the augmented reality display.

[0020] An advantage of displaying a portion of the 3D model in the surgical field is to allow the visualization of anatomical elements under the skin such as bones or certain organs. The display of the 3D model is preferably displayed superimposed on the portion of the subject's body corresponding to said 3D model.

[0021] In one embodiment, the capture of position data of said portion of the body of said individual is carried out continuously and in which the display instruction is updated in real time.

[0022] In one embodiment, a preoperative 3D model of the body portion associated with the first position is generated from the received medical image, said 3D model associated with the second position being generated from the comparison and from said operative 3D model.

[0023] In one embodiment, the preoperative 3D model comprises depth information of each anatomical element of said set relative to the surface of the skin from the received medical image and the 3D model associated with the second position also comprises depth information of each anatomical element of said set relative to the surface of the skin from said information of the preoperative 3D model.

[0024] One advantage is that it makes it possible to retain in the 3D model the depth information of an anatomical element, for example the distance between said anatomical element and the surface of the skin. The augmented reality display of all or part of the 3D model is thus advantageously more precise, in particular the position of the anatomical element of the 3D model relative to the subject's skin.

[0025] In one embodiment, the display instruction further comprises an instruction for projecting a virtual representation of at least a portion of the anatomical elements of the 3D model associated with the second position, the display location of which is determined as a function of the hand surface depth information and the location of the skin surface of the subject's hand on the received optical and / or three-dimensional image.

[0026] In one embodiment, the preoperative 3D model is generated from a segmentation and labeling of each anatomical element of the body portion from the received medical image.

[0027] In one embodiment, the comparison step comprises generating a first set of points of interest of the body portion of the individual in the first position from the preoperative 3D model associated with the first position.

[0028] In one embodiment, the comparison step comprises generating a second set of points of interest of the body portion of the individual in the second position from the 3D model associated with the second generated position.

[0029] In one embodiment, the comparison step comprises comparing the first set of points of interest with the second set of points of interest.

[0030] In one embodiment, the anatomical elements comprise bones of a skeleton portion and in that the set of anatomical elements comprises at least one skeleton portion comprising a set of bones connected to each other.

[0031] In one embodiment, the virtual attribute further comprises a first parameter, such as an angular distance between two bones, calculated from the 3D model associated with the second position.

[0032] In one embodiment, the virtual attribute comprises a second parameter such as a screw diameter or length calculated from the 3D model associated with the second position.

[0033] According to another aspect, the invention relates to a system comprising hardware and / or software means for implementing the method according to the invention.

[0034] In one embodiment, the hardware means comprise information receiving means, information transmitting means, an optical acquisition device for acquiring optical and / or three-dimensional images, a display device and / or a human / machine interface.

[0035] In one embodiment, the software means comprise a processor and / or a memory for executing said method.

[0036] In one embodiment, the hardware means comprises at least one sensor for capturing data on the location and orientation of a body portion in the surgical field.

[0037] In one embodiment, the hardware means comprise a first processor associated with a first memory for implementing the method according to the invention.

[0038] In one embodiment, the system includes an augmented reality display for projecting a virtual representation of the generated virtual attribute into a surgical field while allowing the surgical field to be viewed through the augmented reality display.

[0039] In one embodiment, the system further comprises a robot arm comprising a head configured to support a surgical tool.

[0040] In one embodiment, the system comprises a second processor associated with a second memory for implementing a method for real-time guidance of a robot arm.

[0041] In one embodiment, the first processor and the second processor are the same processor and / or the second memory and the first memory are the same memory.

[0042] In one embodiment, the method for guiding a robot arm in real time comprises: - determination of the relative position of the head of the robot arm; - the calculation of a first guidance trajectory for the head of the robot arm in function of the relative position of the robot arm head and the virtual attribute; - activation of a kinematic of the robot arm to travel the guidance trajectory; - receiving a new 3D model and / or a new virtual attribute generated by the method according to the invention for calculating a new guidance trajectory and activating new kinematics based on said new 3D model and / or the new virtual attribute generated.

[0043] According to another aspect, the invention relates to a computer program product comprising instructions which cause the system according to the invention to execute the steps of the method according to the invention.

[0044] According to another aspect, the invention relates to a computer-readable medium such as a memory, in particular a non-transitory memory, on which the computer program according to the invention is recorded.

[0045] According to a final aspect, the invention relates to a computer-implemented method for assisting surgery comprising the following steps: - Receiving a medical image of a predetermined body portion of an individual in a first position; - Capturing position data of said portion of the body of said individual in a second position different from the first position; - Comparison between the second position and the first position to generate a transformation parameter; - The generation of a 3D model of the bones of said portion of the body, associated with the second position, from the received medical image and from said transformation parameter; - Selecting at least one bone from said generated 3D model and generating a virtual attribute, said virtual attribute comprising an element characteristic of the position and orientation of said selected bone in the 3D model; - Transmitting a display instruction to an augmented reality display to project a virtual representation of the generated virtual attribute into a surgical field comprising said body portion of said individual while allowing the surgical field to be seen through the augmented reality display. Brief description of the figures

[0046] Other characteristics and advantages of the invention will emerge on reading the detailed description which follows, with reference to the appended figures, which illustrate:

[0047] [Fig-1]: a schematic view of a device according to an embodiment of the invention.

[0048] [Fig.2]: a flowchart of the method according to one embodiment of the invention.

[0049] [Fig.3]: a top view of a subject's hand showing examples of points of interest.

[0050] [Fig.4]: an example of visualizations of the detected points of interest allowing to serve as a basis for the detection of the position and orientation of the hand.

[0051] [Fig.5]: a simplified view of an example of a display device according to an embodiment of the invention.

[0052] [Fig.6]: a flowchart of the method according to an embodiment of the invention where a model of the body portion associated with the first position is generated to generate the second body model associated with the second position.

[0053] [Fig.7]: a schematic view of a device according to an embodiment of the invention comprising a guidance system including a robotic arm. Description of the invention

[0054] The following description essentially concerns an example for the generation of a display instruction for an augmented reality display device; said display instruction being configured to display a geometric element characteristic of the position and location of a selected bone of the hand of a subject in the field of vision of the augmented reality device.

[0055] The proposed system and method can be adapted for example to other portions of the body such as the foot, the ankle, the wrist, the vertebrae or any other part of the body for which the precise visualization of a geometric element characteristic of a bone selected on an augmented reality display.

[0056] In another example, the system and the associated method make it possible to display such a guidance element in augmented reality characteristic of an organ or a portion of an organ such as the heart, the colon, the intestinal or venous system.

[0057] The system according to the invention comprises a calculation module comprising at least one calculator for generating a display instruction and an augmented reality device connected to said at least one calculator for displaying elements in response to said generated display instruction.

[0058] Preferably, the method described is implemented by computer. The steps of the method can thus be implemented by one or more processors or computers. Medical image acquisition

[0059] The invention comprises receiving 1100 a medical image 10 of the hand 14 of a subject.

[0060] A medical image 10 of the hand 14 is acquired by a medical image acquisition module.

[0061] Before proceeding with the capture of the medical image, it is preferable to prepare the individual's hand 14 to obtain optimal results. This may involve instructions to place the hand in the first predetermined specific position, possibly using props to keep the hand stable, and / or ensuring that the hand is clean and free of objects that could interfere with the capture of the medical image.

[0062] Preferably, the medical image 10 of the hand is obtained when the latter is arranged in a first predetermined position, for example, placed flat on a support. Preferably, the support comprises a solid-colored surface. In one embodiment, a specific support, for example comprising stop elements, to guide the individual to place his hand in the first predetermined position is used.

[0063] The medical image 10 preferably comprises a three-dimensional image of the body portion. For example, the medical image 10 comprises a series comprising a plurality of cross-sectional images of the hand 14 allowing a three-dimensional representation of the acquired body portion.

[0064] The medical image 10 preferably comprises an anatomical image of a portion of an individual's body. For example, the medical image is acquired by a CT scan technique, also called a "computed tomography." A CT scan is a non-invasive medical imaging technique that provides detailed images of the inside of the body. A CT scan is also known as a computed axial tomography (CAT) scan. This technique uses X-rays and a computer to produce cross-sectional images (slices) of the body, allowing physicians to visualize internal structures, such as organs, bones, blood vessels, and soft tissues.

[0065] In another embodiment, the acquired medical image 10 is a functional image of the body portion. For example, the medical image comprises an image acquired by an MRI (Magnetic Resonance Imaging) technique. MRI images advantageously allow bones, soft tissues and other anatomical structures to be visualized with high resolution without exposing the individual to ionizing radiation. The fundamental principle of MRI is based on the properties of hydrogen atoms present in the human body. When the patient is placed inside an MRI scanner, a strong magnetic field is applied, which aligns the hydrogen atoms in a specific direction. Then, short pulses of radiofrequency waves are emitted into the body, which disrupts the alignment of the hydrogen atoms. When these radiofrequency pulses die down, the hydrogen atoms return to their original aligned state.This relaxation process generates signals detected by special antennas in the . MRI scanner. These signals are then transformed into detailed 2D or 3D medical images of the body's internal structures by a computer.

[0066] Generally speaking, the medical image 10 includes any image making it possible to visualize the bones or skeleton of the body portion of an individual as well as the soft parts of the body portion (organs, blood network, nervous network or muscles).

[0067] In one embodiment, the term "medical image" may be understood as a series of cross-sectional images (such as MRI or CT scan images) acquired using a suitable medical device in which each cross-section corresponds to an image at a different depth.

[0068] In one embodiment, the acquired raw slice images undergo preprocessing to reduce noise and improve data quality. Advanced filtering and normalization techniques are applied to eliminate artifacts and make the images more consistent.

[0069] In one embodiment, the slice images may be subject to artifacts and disturbances, including noise, which may affect the quality and reliability of the extracted information. Different filtering methods, such as Gaussian filter, median filter, or bilateral filter, may be used to reduce noise while preserving important details.

[0070] In one embodiment, preprocessing the slice images includes applying contrast enhancement techniques, such as histogram equalization or linear contrast adjustment, to increase the brightness difference between tissues, thereby improving the visibility of details.

[0071] In one embodiment, preprocessing the slice images includes using artifact removal or correction methods to restore the integrity of the images and ensure their fidelity.

[0072] In one embodiment, preprocessing the medical images includes normalizing the intensities to make the slice images consistent by adjusting their grayscales or pixel values ​​so that they have similar intensity ranges.

[0073] In one embodiment, the preprocessing of the medical images comprises the implementation of a registration and registration technique to align the acquired section images of the same region of the body and correctly superimpose them to obtain a coherent medical image.

[0074] In the remainder of the description, the term “medical image” will be used to designate both an acquired raw medical image or a medical image having undergone a pre-processing step as described above.

[0075] Generation of a preoperative 3D model associated with the first position

[0076] In one embodiment, the method according to the invention comprises the generation of a first preoperative 3D model 22 of the hand or of a portion of the body of the individual in the first predetermined position in which the medical image 10 a was acquired.

[0077] The generation of a first preoperative 3D model 22 of the hand preferably includes a detection of the body portion on the medical image 10 to identify the region of interest of the image corresponding to the hand in the medical image 10, distinguishing it from other surrounding anatomical structures or from the background. For example, a thresholding segmentation or contour segmentation method is used.

[0078] The overall objective of the body portion detection step is to accurately isolate the region of interest corresponding to said body portion in the medical image. Once the body portion has been detected, the processed medical images 10 are ready to be used in subsequent steps of the process, such as 3D reconstruction and anatomical modeling of said body portion. Accurate detection of the body portion advantageously makes it possible to obtain more reliable and better quality results throughout the process of generating the 3D model(s).

[0079] Preferably, the preoperative 3D model 22 is also generated from an optical image of the body portion and / or a point cloud of said body portion acquired when the hand is in the first position.

[0080] The generation of a preoperative 3D model of the hand comprises a preoperative three-dimensional reconstruction to advantageously generate a three-dimensional volumetric representation of the hand associated with the first position. The first initial 3D model and the initial three-dimensional reconstruction are preferentially associated with the first position.

[0081] Preferably, the three-dimensional reconstruction of the first preoperative 3D model of the hand is generated from the 3D medical image or from a series of 2D cross-sectional images of the hand.

[0082] In one embodiment, the three-dimensional reconstruction comprises a voxel-based reconstruction. This method is based on the discretization of space into small cubic units called voxels. Each voxel is assigned an intensity value based on the corresponding medical image. Using these values, a volumetric 3D model of the hand is created, where each voxel represents a small volume element in three-dimensional space.

[0083] In one embodiment, the three-dimensional reconstruction comprises a contour reconstruction comprising the segmentation of the contours of the different hand structures in each cross-sectional image of the medical image and combine said contours to create a 3D representation of the body portion.

[0084] In one embodiment, the three-dimensional reconstruction comprises a reconstruction by depth-based approaches comprising using the depth information of each cross-sectional image of the medical image to reconstruct the three-dimensional geometry of the hand.

[0085] In one embodiment, the method further comprises the generation 112 of a preoperative anatomical model 23. The preoperative anatomical model 23 can be carried out from the medical image 10 and / or from the three-dimensional representation generated as described previously.

[0086] The anatomical modeling preferably comprises a three-dimensional modeling of an anatomical system of the body portion of which the optical image was acquired. For example, the anatomical modeling comprises a modeling of the bones of the user's hand.

[0087] Anatomical modeling makes it possible to classify the different anatomical structures present in the hand, in particular the bones, the joints, and the soft tissues of the skin or flesh.

[0088] In one embodiment, the generation of the preoperative anatomical modeling 23 comprises a segmentation of each anatomical structure of the medical image 10 and the classification of said segmented structure from the characteristics of said segmentation. In another embodiment, the segmentation and classification of each structure of the hand is implemented by a learning function such as a deep neural network or generative adversarial networks (GAN), can be used to create anatomical models of the hand from the medical image and / or from the generated three-dimensional representation.

[0089] The first preoperative 3D model of the individual associated with the first position is thus generated from the medical image 10 and comprises a 3D representation of the hand whose anatomical structures are classified. Preferably, the preoperative 3D model comprises a three-dimensional representation of the bony anatomical structures of the hand and optionally the surface of the skin. This preoperative 3D model can be obtained by subtracting certain anatomical structures from the three-dimensional representation according to their classification.

[0090] In an alternative embodiment, the first preoperative 3D model 22 comprises a three-dimensional representation of the bones of the hand in which the bones of the hand are labeled so that they can be identified among the bones of said first 3D model and optionally a virtual representation of the surface of the skin. In one embodiment, the first 3D model comprises depth information of the skin surface relative to the surface of the bone anatomical structures of the first 3D model.

[0091] The dimensions of each bone and the relative position of a bone with respect to adjacent bones can be determined from the acquired medical image. Preferably, these dimensional data can be recorded or stored, for example in a memory.

[0092] In one embodiment, the medical image 10 is transmitted to the PRO calculation module and the first preoperative 3D model is generated by the PRO calculation module. In an alternative embodiment, the first preoperative 3D model and / or the preoperative anatomical modeling associated with the first position are transmitted to the PRO calculation module and / or stored or recorded on a data storage unit such as a MEM memory (preferably non-transient). Hand position and location data

[0093] The method further comprises a step 120 of determining the position and optionally the location of the individual's hand.

[0094] The determination comprises the acquisition 121 of an image such as an optical image of the individual's hand when the individual's hand is in a second position different from the first position. The optical image may comprise a 2-dimensional or preferably a 3-dimensional optical image of the individual's hand. The image may be acquired by a remote image acquisition device. More preferably, the image is acquired by an image acquisition device integrated into an augmented reality headset as described below. Preferably, said image comprises an image of the individual's hand in an environment comprising objects which may make it possible to extract data on the location and orientation of the hand relative to said object in the environment.

[0095] In one embodiment, the determining comprises acquiring a three-dimensional image of the body portion such as a point cloud or a depth map.

[0096] The image acquisition device preferably comprises a 3D scanner. A 3D scanner means a device using one or more lasers or sensors to measure the three-dimensional geometry of the hand. It can provide a detailed 3D image of the hand, capturing its shape and contours accurately based on the principle of the laser's time of flight as it is reflected by the hand. The scanner is designed to scan the individual's hand to record the depth information of each pixel.

[0097] The image acquisition device may comprise a stereoscopic camera, i.e. a pair of stereoscopic cameras. A computing unit may then be used to extract acquired 2D images, using parallax or difference angles, depth information. By combining these images, it is possible to obtain a 3D image of the hand.

[0098] The term “optical image” therefore means an image in the visible or infrared range or a depth map or a point cloud.

[0099] In one embodiment, the image comprises a point cloud of at least a portion of the hand. In an alternative embodiment, a point cloud representing at least a portion of the hand is generated from the acquired image.

[0100] The method then comprises the generation of data 11, called “position data” or “position indicator” as a function of the second position of the hand on the acquired image. For example, a position indicator is generated from said acquired image of the hand in the second position. Said generated position indicator is characteristic of said second position. Preferably, it is thus possible to reconstruct the second position of the hand from said generated indicator.

[0101] For example, said generated indicator comprises pose information such as angular values ​​of at least a portion of the joints of the hand. In another example, the generated indicator is associated with said second position of the individual's hand.

[0102] In one embodiment, the position data makes it possible to characterize the second position and makes it possible to reconstruct a virtual model of a hand in said second position.

[0103] The position data 11 may describe the relative position of the joints of the hand and their spatial coordinates and / or the relative positions of the different fingers of the hand and the palm of the hand.

[0104] In one embodiment, the position data further comprises location and orientation data of the hand and / or each finger of the hand and / or each phalanx of the hand. The location data comprises relative coordinates of the hand in the environment. The orientation data comprises the relative orientation of the individual's hand in the environment.

[0105] According to one embodiment, points of interest 206, 208 of the user's hand are generated from the acquired image or from the processing of the point cloud El generated as illustrated in [Fig.3].

[0106] Said points of interest may comprise centers of mass 208 of the hand or of a portion of the hand. The centers of mass may be generated at coordinates corresponding substantially to the center of a predetermined area of ​​the hand. For example, a point of interest may correspond to the center of mass of the palm of the hand, of a phalanx, of a finger.

[0107] The points of interest may comprise deflection points 206. The deflection points 206 are generated at the boundary between two adjacent portions of the hand. For example, a point of interest can be generated between the portions of the hand corresponding to two adjacent phalanges of the same finger. The location of such a point of interest can then correspond to the location of a joint, for example between two phalanges.

[0108] The points of interest may comprise the tip or end of a finger. The different points of interest may be connected to each other by segments 207.

[0109] The points of interest may comprise coordinates of a point or area on the acquired image and labeling information corresponding to identification information. For example, the identification information makes it possible to identify a predetermined joint.

[0110] In one embodiment, the step of generating the points of interest comprises generating the coordinates of each point of interest on the acquired image or in the generated point cloud of the individual's hand in the second position.

[0111] Preferably, the position data are obtained from the coordinates of said points of interest. The pose of the hand in the second position can thus be determined from the angular differences between two segments 207 connected to the same point of interest 206; 208.

[0112] An example of a 3D model associated with a second position E2 is represented in [Fig.4] in which the points of interest 211, 212, 213 are generated from the transformation parameter as well as the segments 214 connecting said points of interest.

[0113] In another alternative or cumulative embodiment, the position indicator is generated from a trained learning function such as a trained neural network. Preferably, said learning function is trained from a collection of labeled training images. Each training image is an optical image of the hand in a position and comprises a label comprising the hand position data associated with said training image. Said learning function, once trained, is configured to receive as input an image of the hand and generate as output a position indicator of said hand on the image associated with said input image. In a variant, preprocessing of the hand image may be carried out, for example the generation of points of interest.In one embodiment, the learning function is configured to generate position data from the coordinates of the points of interest and / or the acquired image of the hand.

[0114] Comparison between the first position and the second position

[0115] From the generated position data or from the position indicator, the second position of the body portion is compared (130) to the first predetermined position in which the preoperative medical image of the body portion had been acquired.

[0116] This comparison step generates a transformation parameter 26. The transformation parameter 26 may for example comprise a transformation matrix making it possible to transform the position of the body portion into the second position from the first position.

[0117] In one embodiment, the transformation parameter 26 is obtained by mathematical operations of comparison between the generated position indicator associated with the second position and a pre-recorded position indicator associated with the first position and preferably different from the generated position indicator. In an alternative embodiment, the position indicator relating to the first position is determined from an optical and / or three-dimensional image of the body portion in the first position.

[0118] The comparison can be obtained by regression, subtraction or division operations.

[0119] The transformation parameter may be generated based on pre-recorded biomechanical rules. For example, the transformation parameter may include an angular value of rotation of each joint of the hand allowing a hand to move from the first to the second position. Preferably, the transformation parameter includes an angular value of each joint of the hand allowing it to move from the first to the second position.

[0120] Preferably, the transformation parameter is a parameter associated with the second position and converting the position index associated with the first position into a position index associated with the second position.

[0121] In one embodiment, the generation of the transformation parameter comprises the detection or calculation of angles of the different parts of the body portion around joints of said body portion (for example the angle between two adjacent phalanges) from the acquired optical images. Preferably, the transformation parameter is generated by comparing the angular values ​​of joints associated with the first pre-recorded position and those associated with the second position. Generation of an updated 3D model

[0122] The method comprises generating 140 a second updated 3D model 14 associated with the second position. The updated 3D model preferably comprises a 3D model as described for the preoperative 3D model. The updated 3D model 14 is however associated with the second position.

[0123] In a first embodiment illustrated in [Fig.2], the second updated 3D model 14 is generated from the acquired images of the body portion in the second position in the same way as the first initial 3D model is generated from the acquired images of the body portion in the first position.

[0124] The updated 3D model comprises an anatomical model comprising a three-dimensional model of said anatomical system associated with the second position. For example, the anatomical system comprises a representation of the skeletal system of the hand composed of the bones of the hand.

[0125] The generation of an updated 3D model comprises a three-dimensional reconstruction to advantageously generate a three-dimensional volumetric representation of the hand in the second position. Said three-dimensional reconstruction may be the same as that described for the first initial 3D model.

[0126] In a particularly advantageous embodiment illustrated in [Fig.6], the anatomical modeling associated with the second position is generated from the pre-recorded pre-operative anatomical modeling 23 associated with the first position and from the generated transformation parameter 26 associated with the transformation of the body portion in the second position from the first position.

[0127] In this mode, the transformation parameter 26 is applied to the preoperative anatomical model associated with the first position to obtain an anatomical model associated with the second position.

[0128] In one embodiment, the angular value of each joint of the first preoperative anatomical model associated with the first position is modified by the value of said joint according to the position data associated with the second position to generate the updated anatomical model associated with the second position.

[0129] Preferably, the shape of each anatomical element is generated from the previously acquired medical image associated with the first position of the body portion. Preferably, the position and orientation of each anatomical element relative to each other are generated from the transformation parameter and / or from the comparison between the first predetermined position and the second position.

[0130] The updated anatomical modeling associated with the second position preferably comprises a representation of the skeleton of the user's hand, i.e. a set of bones connected to each other. Preferably, the connection between two bones is symbolized by a joint labeled by an identifier.

[0131] In one embodiment, the updated 3D model associated with the second position is generated from the preoperative 3D model associated with the first position.

[0132] Preferably, the anatomical elements of the updated 3D model are identical to the anatomical elements of the preoperative 3D model.

[0133] In one embodiment, the updated anatomical modeling is generated from the preoperative anatomical modeling on which a spatial transformation in translation and / or rotation of the anatomical elements (for example bones) is applied according to the transformation parameter 16 previously calculated and optionally according to a set of pre-recorded biodynamic rules.

[0134] For example, biodynamic rules may set constraints on the angular displacement of certain joints. In particular, biodynamic rules define axes of rotation of each joint and / or ranges of authorized and / or prohibited values ​​of each joint of the body portion.

[0135] In one embodiment, the second updated anatomical model associated with the second position may comprise depth information. The depth information comprises distance information between the surface of a point of the updated anatomical model and the skin of the subject. Preferably, this information is generated from the acquired medical images. As such, the acquired medical images are processed so as to determine the distance between the surface of a point of a determined bone of the subject and the skin of the subject. For example, the distance between the surface of a bone and the skin of the subject is measured on the medical images and associated with the bone or point of the bone used for said measurement.

[0136] This depth information is preferably associated with coordinates of the point of the anatomical modeling from which the distance was measured relative to a reference point located on said anatomical element. Said coordinates can be calculated from the two-dimensional coordinates of the point on the medical image and as a function of the section image used or the depth associated with said section image.

[0137] Preferably, the depth distances between the surface of the skin and the anatomical elements of the second updated anatomical model associated with the second position are similar to the distances determined during the construction of the first preoperative anatomical model.

[0138] In one embodiment, the device comprises means for acquiring or receiving optical and / or three-dimensional images of the body portion continuously and for generating continuously and / or in real time a second 3D model updated from said images.

[0139] In a preferred embodiment, the anatomical modeling is generated by a trained learning function such as a trained neural network.

[0140] Said learning function is configured to receive as input: - a transformation parameter 26 associated with the second position and converting the position index associated with the first position into a position index associated with the second position; associated with - a medical image 10 of a body portion in a first position.

[0141] Said learning function is configured to generate as output an anatomical model associated with the second position.

[0142] Said learning function was preferably trained from a training database of a plurality of triplets each composed of medical images of the body portion in the first position, of transformation parameters associated with the transformation from the first position to a second position (different from the first position) and of an anatomical modeling of said body portion in the second position. Generating a virtual image

[0143] The method further comprises the selection 150, from the set of anatomical elements composing the updated anatomical modeling associated with the second position, of an anatomical element 16.

[0144] Said selection can be carried out via a human-machine interface INT.

[0145] Said human-machine interface INT further comprises one or more user interfaces.

[0146] The user interface(s) may for example include one or more computer mice, a keyboard, a touch screen, a trackball, a microphone for providing voice recognition software running on a processor with spoken instructions, a camera for providing images of captured gestures or the like to gesture recognition software running on a processor, and so on. It should be understood that any existing user interface device may be used in conjunction with the AFF display system described hereinafter.

[0147] In one embodiment, a display system includes a list of labels, each label being associated with a predetermined anatomical element.

[0148] In one embodiment, the method comprises the generation 170 of a virtual image 17 and / or the generation of a display instruction to display said virtual image. Said display instruction can then be received by a display module such as that of an augmented reality display computer device as described below to display said virtual image.

[0149] In one embodiment, the virtual image 17 may comprise a portion of the updated anatomical modeling. For example, the virtual image comprises at least a portion of the individual's hand bone system generated by the steps previously described. Preferably, the position of the hand is detected (second position) by the image capturing device and the virtual image comprising said portion of the anatomical model associated with the second position is displayed in real time.

[0150] Preferably, the virtual image is displayed in such a way that said portion of the anatomical model is superimposed with the portion of the individual's body.

[0151] In one embodiment, the virtual image is displayed so as to match the skin surface of the updated 3D model with the user's skin surface on the optical image. Preferably, the updated 3D model is displayed so that the skin surface of said 3D model is confused with the skin surface detected on the acquired optical image and / or with the point cloud of the body portion acquired by the image acquisition device.

[0152] The displayed virtual image including a portion of the 3D model is thus advantageously consistent with the position of the subject's hand. The user thus obtains an augmented reality display of the 3D model.

[0153] Preferably, the portion of the 3D model displayed does not include the skin surface but the internal anatomical elements of the subject such as the bones of the subject. Nevertheless, thanks to the depth information (distance between the anatomical element and the surface of the skin of the subject) said anatomical elements are displayed with a particularly advantageous localization precision, even in the event of movement of the portion of the subject's body. In one embodiment, the virtual image 17 is generated as a function of the selected anatomical element. The virtual image can thus comprise only the selected anatomical element 16 and / or the adjacent anatomical elements of the selected anatomical element 16. Generation of a virtual attribute

[0154] In one embodiment, the method further comprises generating a virtual attribute 15. The virtual attribute 15 comprises an element to be displayed reflecting a property of the selected anatomical element 16.

[0155] The virtual attribute 15 may comprise a direction axis. In one embodiment, the direction axis is generated as a function of the shape and / or orientation of the selected anatomical element 16 according to a rule or set of rules predefined and / or associated with said selected anatomical element.

[0156] In the example where the anatomical model comprises a system of bones of the hand, the virtual attribute 15 may comprise an axis merged with the longitudinal axis of said selected bone.

[0157] An advantage of displaying such an axis on an augmented reality display module is to allow the surgeon to visualize a specific axis relating to the anatomical element that he has previously selected.

[0158] In one embodiment, the virtual attribute comprises a pre-generated anatomical indicator.

[0159] The anatomical indicator may be displayed as a number near or superimposed on the body portion. The anatomical indicator may comprise the value of a distance between the surface of the skin and the surface of an anatomical element, for example at a predetermined and / or selected point.

[0160] In one embodiment, the virtual attribute 15 comprises a plane whose coordinates are a function of the selected anatomical element. In one embodiment, the computing device described below displays the plane of the virtual attribute as a section plane so as to display a portion of the modeling associated with the second position superimposed on the part of the individual's body in contact with said generated plane. Preferably, said generated plane is therefore seen as a section plane.

[0161] In one embodiment, the operator can select via the human-machine interface INT a virtual attribute from a list of virtual attributes such as those previously described to be displayed and the computing device will display in return the selected virtual attribute.

[0162] Preferably, the virtual image 17 and / or the virtual attribute 15 are generated and / or displayed in real time from the optical and / or three-dimensional images acquired and received by the system. These two pieces of information can thus evolve in real time depending on the movements of the subject.

[0163] Visualization of the anatomical model in real time via an augmented reality headset.

[0164] [Fig. 5] schematically represents a particularly preferred embodiment of the AFF display system, in which this system is implemented as a head-mounted computing device such that the virtual image 17 can be generated in the view of the doctor in the first instance. The device further comprises a location for augmenting the reality (i.e., the real view) of the doctor, e.g., by superimposing the virtual image 17 on this real view.

[0165] In the context of the present invention, a head-mountable computing device is a device that can be worn on the head of its user and provides the user with computing functionality.

[0166] The head-mountable computing device may be configured to perform specific computing tasks as specified in a software application (app) that may be retrieved from the Internet or other computer-readable medium.

[0167] Non-limiting examples of such head-mounted computing devices include smart headsets, e.g., glasses, goggles, a helmet, a hat, a visor, a headband, or any other device capable of being supported on or from the wearer's head, etc.

[0168] The head-mountable computing device may comprise the PRO processor and / or the computing unit 2, e.g., in a component package 122.

[0169] The head-mountable computing device may further include an image sensor or camera 127 as a detector of the position of the body portion for capturing an image in a field of view of a wearer of the wearable computing device. The image sensor may be arranged such that when the head-mountable computing device is worn as intended, the image sensor aligns with the eyes of its wearer, i.e., produces a forward-facing sensor signal corresponding to the field of view of its wearer.

[0170] Such an image sensor or camera may be integrated into the head-mountable computing device, for example integrated into a lens of a head-mountable computing device through which its wearer observes his or her field of view, into a lens holder or frame for such a lens, or into any other suitable structure of the head-mountable computing device in which the optical sensor aligns with the field of view of the wearer of the head-mountable computing device.

[0171] Alternatively, such an image sensor may be part of a modular wearable computing device, e.g., a head-mounted image sensor module communicatively coupled via a wired or wireless connection to one or more other modules of the head-mounted computing device, wherein at least some of the other modules may be worn on parts of the body other than the head, or wherein some of the other modules may not be wearable, but rather portable, for example.

[0172] The head-mountable computing device generally includes at least one display module 125, which may be a transparent display module, under the control of a discrete display controller (not shown).

[0173] Alternatively, the display controller may be implemented by a processor of the head-mounted computing device.

[0174] The at least one display module 125 is generally arranged such that a wearer of the head-mountable computing device, e.g., the physician present at the first location 100, can observe the virtual image 17 of the ultrasound probe 11 displayed on the at least one display module 125.

[0175] Preferably, the at least one display module 125 is a transparent display module such that the wearer can observe at least a portion of a field of vision through the display module 125, e.g., the subject's hand or other body portion, the doctor's tools required for surgery.

[0176] In one embodiment, the head-mountable computing device includes a pair of display modules 125 including a first module display module that can be viewed by the wearer's right eye and a second display module that can be viewed by the wearer's left eye.

[0177] Alternatively, at least one display module 125 may be an opaque display module on which an augmented reality scene of the field of vision of its wearer is displayed, for example a display module 125 making it possible to project a virtual image 17 into the augmented field of vision. To this end, the head-mountable computing device may comprise a camera for capturing the field of vision of its wearer.

[0178] The first display module and the second display module can be controlled to display different images, e.g. to generate a stereoscopic image as is well known in the art.

[0179] Alternatively, an image may be generated on one of the first and second display modules only such that the wearer can observe the generated image with one eye and the actual field of view with the other eye. The first and second display modules may be transparent or partially transparent display modules. Alternatively, one of the first and second display modules may be a transparent or partially transparent display module, while the other display module is an opaque display module, i.e., a display module that is not transparent such that the wearer cannot see through this display.

[0180] The at least one display module 125 may be provided in any suitable form, such as a transparent lens portion.

[0181] Alternatively, as shown in [Fig. 4], the head-mountable computing device may comprise a pair of such lens portions, i.e., one for each eye, as explained above. The one or more transparent lens portions may be sized such that substantially the entire field of vision of the wearer is obtained through the one or more transparent lens portions. For example, the at least one display module 125 may be in the form of a lens to be mounted within the frame of the head-mountable computing device. Any other configuration known to those skilled in the art may be contemplated.

[0182] It will be understood that the frame 128 may have any suitable shape and may be made of any suitable material, e.g., a metal, a metal alloy, a plastic, or a combination thereof.

[0183] Several components of the head-mountable computing device may be mounted in the frame 128, for example in a component housing 122 forming part of the frame 128. The component housing 122 may have any suitable shape, preferably an ergonomic shape that allows the head-mountable device to be worn by its wearer in a comfortable manner. Robotic arm

[0184] In one embodiment, illustrated in [Fig.7], the system further comprises a guidance system. The guidance system may comprise, for example, a robot arm ROB for guiding an OTL surgical tool.

[0185] The surgical tool may comprise a cutting tool, a device for injecting a medicinal or anesthetic product, a device for inserting a medical device such as a pin, plate, or a screw.

[0186] In this embodiment, the robot arm comprises a head adapted to receive a surgical tool.

[0187] Preferably, the system is configured to guide the robot arm head in real time according to a position and / or an orientation depending on the updated 3D model associated with the second position.

[0188] In a first example, the system is configured to control the robot arm so as to orient the head of the robot arm in real time according to the generated virtual attribute. For example, the head of the robot arm is controlled according to the axis 151 of the virtual attribute 15, preferably to align the longitudinal axis of the surgical tool with said axis 151 of the virtual attribute. The robot arm can thus be guided, manually by an operator or automatically to reach a point of interest of the body portion of the subject along a predetermined axis, even if the subject is moving.

[0189] In a second embodiment, the system is configured to detect the position and / or orientation of the robot arm head in its environment such as relative to a point of interest of the body portion of the subject. The system is then also configured to generate a guidance trajectory for moving the robot arm head from its initial position to a predetermined target point. The predetermined target point and the guidance trajectory may be continuously updated from the newly generated updated 3D anatomical model and the generated virtual attribute.

[0190] Preferably, the robot arm may comprise sensors such as optical and / or three-dimensional image sensors, for example for generating the updated three-dimensional model or for detecting obstacles on the guidance path. Software and computer

[0191] It should be noted, for the avoidance of doubt, that although the method has been described as a series of sequential steps, it will be immediately apparent to those skilled in the art that at least some of the steps may alternatively be performed simultaneously, i.e., in parallel.

[0192] Aspects of the described method may be provided in the form of a computer program product comprising a computer-readable storage medium. wherein there are incorporated computer-readable program instructions for, when executed on the system processor, causing such processors to perform the relevant steps of said method.

[0193] Aspects of the present invention may be implemented as a medical system 1, a display device control system 2, a display system 20.

[0194] Certain aspects of the present invention may take the form of a computer program product embodied in one or more computer-readable media having computer-readable program code embedded therein.

[0195] Any combination of one or more computer-readable media may be used.

[0196] The computer-readable medium may be a computer-readable signal medium or a computer-readable storage medium.

[0197] A computer-readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing.

[0198] Such a system, apparatus, or device may be accessible via any suitable network connection; for example, the system, apparatus, or device may be accessible over a network to retrieve computer-readable program code over the network.

[0199] Such a network may be, for example, the Internet, a mobile communications network or other.

[0200] More specific examples (a non-exhaustive list) of the computer-readable storage medium may include the following, a hard disk drive, random access memory (RAM), a drive, memory only (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any other suitable combination of the foregoing.

[0201] In the context of the present application, a computer-readable storage medium may be any tangible or non-transitory medium capable of containing or storing a program for use by or in connection with a system, apparatus, or device for executing instructions.

[0202] A computer-readable signal carrier may comprise a propagated data signal having computer-readable program code embedded therein, for example in baseband or as part of a carrier wave.

[0203] Such a propagated signal may take various forms, including, but not limited to, electromagnetic, optical, or any suitable combination thereof. A computer-readable signal carrier may be any computer-readable medium that is not a computer-readable storage medium and that can communicate, propagate, or carry a program for use by or in connection with an instruction-executing system, apparatus, or device. Program code embodied on a computer-readable medium may be transmitted using any suitable medium, including, but not limited to, wireless, wire, fiber optic cable, RF, etc., or any suitable combination of the foregoing.

Claims

Claims

1. A computer-implemented method (100) for assisting surgery comprising the following steps: • Receiving (110) a medical image (10) of a predetermined body portion (11) of an individual in a first position; • Receiving (121) an optical and / or three-dimensional image (25) of said body portion in a second position different from the first position and determining (120) position data of said body portion (11) characteristic of said second position from said received image (25); • Comparing (130) the second position with the first position to generate a transformation parameter; • Generating (140) a 3D model (14) of an anatomical structure of said body portion, associated with the second position, from the received medical image and from said transformation parameter;said anatomical structure comprising at least one set of anatomical elements (16) of said portion of the individual's body; • The selection (150) of at least one anatomical element (16) from said set of anatomical elements of said generated 3D model (14) and • The generation (160) of a virtual attribute (15), said virtual attribute comprising an element characteristic of the position and orientation of said anatomical element (16) selected in the 3D model (14).;

2. Method according to claim 1, characterized in that the characteristic element of the virtual attribute (15) comprises an axis (151) or a plane (152) whose location and orientation are generated as a function of the position and orientation of said anatomical element (16).

3. A method according to claim 1 or claim 2, further comprising generating and transmitting (170) a display instruction (29) to an augmented reality display (AFF) for projecting a virtual representation of the generated virtual attribute (15). in a surgical field comprising said body portion of said individual while allowing the surgical field to be viewed through the augmented reality display (ARD).

4. The method of claim 1, wherein a preoperative 3D model (22) of the body portion associated with the first position is generated from the received medical image, said 3D model associated with the second position being generated from the comparison and from said operative 3D model (22).

5. Method according to the preceding claim, characterized in that the preoperative 3D model comprises depth information of each anatomical element of said set relative to the surface of the skin from the received medical image and the 3D model associated with the second position also comprises depth information of each anatomical element of said set relative to the surface of the skin from said information of the preoperative 3D model.

6. Method according to claim 3 and claim 5, characterized in that the display instruction further comprises an instruction for projecting a virtual representation of at least part of the anatomical elements of the 3D model associated with the second position, the display location of which is determined as a function of the depth information relative to the hand surface and the location of the skin surface of the subject's hand on the received optical and / or three-dimensional image.

7. Method according to one of the preceding claims, characterized in that the anatomical elements comprise bones of a portion of skeleton and in that the set of anatomical elements comprises at least one portion of skeleton comprising a set of bones connected to each other.

8. System (1), comprising software and / or hardware means for implementing the steps of the method according to one of claims 1 to 7

9. d / . Computer program product comprising instructions which cause the system according to claim 8 to execute the steps of the method according to one of claims 1 to 7.

10. Computer-readable medium (MEM), on which the computer program according to claim 9 is recorded.

Images