Method for assisting with surgery and associated system

Abstract

The invention relates to a computer-implemented method (100) for assisting with surgery, which comprises the following steps: • receiving (110) a medical image (10) of a predetermined body portion (11) of an individual in a first position; • receiving (121) an optical and / or three-dimensional image (25) of the body portion in a second position; • generating (140) a 3D model (14) comprising at least one set of anatomical elements (16) of the body portion of the individual; • generating (160) a virtual attribute (15) which is characteristic of the position and orientation of a selected anatomical element; and • generating and transmitting (170) a display instruction for an augmented reality display to project a virtual representation of the virtual attribute (15).

Description

[0001] SURGERY AID METHOD AND ASSOCIATED SYSTEM

[0002] Field of invention

[0003] The invention relates to a computer-implemented method for assisting surgery, in particular a method for generating a display instruction for a hand axis on an augmented reality device. The invention also relates to a system or computer program for implementing said method and a memory comprising said computer program.

[0004] State of the art

[0005] During some surgeries, certain anatomical parts of the patient such as bones are invisible to the surgeon.

[0006] Some technologies make it possible, using an augmented reality headset worn by the surgeon, to superimpose virtual characteristics corresponding to physical characteristics hidden by the patient's anatomy on what the surgeon sees.

[0007] For example, WO2018164909 is known, proposing a solution for displaying knee ligaments through the patient's skin on the display of an augmented reality device to allow the surgeon to visualize these ligaments during an operation. The size and shape of the virtual ligaments can be modified based on the detection of movement of adjacent bones.

[0008] A disadvantage of this type of solution is that it does not allow the generation of surgery assistance images for body parts more complex than the knee, such as the hand or foot.

[0009] There is therefore a need for a method and a surgical assistance system that can represent physical characteristics of a body portion such as the hand or the foot.

[0010] Summary of the invention

[0011] In one aspect, the invention relates to a computer-implemented surgical assistance method or method.

[0012] The process includes:

[0013] ■ Receiving a medical image of a predetermined body portion of an individual in a first position; ■ Receiving an optical image of a scene comprising said body portion in a second position different from the first position;

[0014] ■ Comparison between the second position and the first position to generate a transformation parameter;

[0015] ■ The generation of a 3D model of an anatomical structure of said portion of the body, associated with the second position, said 3D model associated with the second position being generated from the transformation parameter and from said preoperative 3D model; said anatomical structure comprising at least one set of anatomical elements of said portion of the body of the individual; the 3D model associated with the second position also comprising depth information of each anatomical element of said set relative to the surface of the skin;

[0016] ■ the generation and transmission of a display instruction to an augmented reality display to project a virtual representation of at least part of the anatomical elements of the 3D model associated with the second position, the display location of which is determined based on the depth information of the 3D model associated with the second position.

[0017] In one embodiment, the display location is also determined based on the position of the skin of the body portion on the acquired optical image.

[0018] In one embodiment, the optical image allows the detection of a change in position of the subject relative to the first position. The 3D model is used to calculate the new positions of each anatomical element.

[0019] In one embodiment, dimensional features of the patient's anatomy are extracted from medical images to serve as parameters for fitting a biomechanical model to the patient's size and shape. These dimensions may be the 2D dimensions of major organs and bones in the axial, coronal, or sagittal planes, or specific distances between organs, bones, or skin in the axial, coronal, or sagittal planes. Another way to fit the model is to use atlas-based deformation techniques to determine a deformation field to be applied to one or more biomechanical models (from an atlas). The deformation field and the chosen final biomechanical model are determined after optimizing a cost function (similarity measures, strain energy) to match the actual patient's anatomy.

[0020] In one embodiment, the method comprises generating a preoperative 3D model of the body portion associated with the first position from the received medical image. In one embodiment, the preoperative 3D model comprises depth information of each anatomical element of said preoperative 3D model relative to the surface of the skin from the received medical image. In one embodiment, the depth information of the 3D model associated with the second position is generated from said information of the preoperative 3D model, in particular from the depth information of each anatomical element of said preoperative 3D model.

[0021] In one embodiment, the body portion comprises at least one hand of said individual. In one embodiment, the anatomical features comprise bones of the hand.

[0022] In one embodiment, the method comprises generating a pre-operative 3D model of the body portion associated with the first position from the received medical image.

[0023] In one embodiment, the method comprises a step of generating a virtual attribute, said virtual attribute comprising an element characteristic of the position and orientation of a predetermined or selected anatomical element in the 3D model.

[0024] In one aspect, the invention relates to a computer-implemented method or method for assisting surgery.

[0025] The method includes receiving a medical image of a predetermined body portion of an individual in a first position.

[0026] The method comprises receiving an optical and / or three-dimensional image of said body portion in a second position different from the first position and determining position data of said body portion characteristic of said second position from said received image. The method comprises comparing the second position with the first position to generate a transformation parameter.

[0027] The method comprises generating a 3D model of an anatomical structure of said portion of the body, associated with the second position, from the received medical image and from said transformation parameter.

[0028] In one embodiment, said anatomical structure comprises at least one set of anatomical elements of said portion of the individual's body.

[0029] In one embodiment, the method comprises selecting at least one anatomical element from said set of anatomical elements of said generated 3D model.

[0030] In one embodiment, the method comprises generating a virtual attribute, said virtual attribute comprising an element characteristic of the position and orientation of said selected anatomical element in the 3D model.

[0031] One advantage of the virtual attribute is that it provides the practitioner with information about the position and / or orientation of an anatomical element of the subject that is not visible to the naked eye. This virtual attribute can thus be displayed on an augmented reality device in the practitioner's field of vision.

[0032] In one embodiment, the geometric element of the virtual attribute comprises an axis or plane whose location and orientation are generated based on the position and orientation of said anatomical element.

[0033] One advantage of an axis or plane is that it allows the practitioner to track the orientation of an element, for example for the alignment or placement of a surgical tool.

[0034] In one embodiment, the method comprises generating and transmitting a display instruction to an augmented reality display to project a virtual representation of the generated virtual attribute into a surgical field comprising said body portion of said individual while allowing the surgical field to be viewed through the augmented reality display.

[0035] In one embodiment, the display instruction comprises an instruction to project a virtual representation of at least a portion of the 3D model into the surgical field by allowing the surgical field to be seen through the augmented reality display. An advantage of displaying a portion of the 3D model in the surgical field is to allow the visualization of anatomical elements under the skin such as bones or certain organs. The display of the 3D model is preferably displayed superimposed on the portion of the subject's body corresponding to said 3D model.

[0036] In one execution mode, the capture of position data of said portion of the body of said individual is carried out continuously and in which the display instruction is updated in real time.

[0037] In one embodiment, a preoperative 3D model of the body portion associated with the first position is generated from the received medical image, said 3D model associated with the second position being generated from the comparison and from said operative 3D model.

[0038] In one embodiment, the preoperative 3D model comprises depth information of each anatomical element of said set relative to the surface of the skin from the received medical image and the 3D model associated with the second position also comprises depth information of each anatomical element of said set relative to the surface of the skin from said information of the preoperative 3D model.

[0039] One advantage is that it allows the depth information of an anatomical element to be retained in the 3D model, for example the distance between said anatomical element and the surface of the skin. The augmented reality display of all or part of the 3D model is thus advantageously more precise, in particular the position of the anatomical element of the 3D model in relation to the subject's skin.

[0040] In one embodiment, the display instruction further comprises an instruction for projecting a virtual representation of at least a portion of the anatomical elements of the 3D model associated with the second position, the display location of which is determined based on the depth information and the location of the subject's skin surface on the received optical and / or three-dimensional image.

[0041] In one embodiment, the display instruction further comprises an instruction for projecting a virtual representation of at least a portion of the anatomical elements of the 3D model associated with the second position, the display location of which is determined based on the hand surface depth information and the location of the skin surface of the subject's hand on the received optical and / or three-dimensional image.

[0042] In one execution mode, the preoperative 3D model is generated from a segmentation and labeling of each anatomical element of the body portion from the received medical image.

[0043] In one embodiment, the comparison step comprises generating a first set of points of interest of the body portion of the individual in the first position from the preoperative 3D model associated with the first position.

[0044] In one embodiment, the comparison step comprises generating a second set of points of interest of the body portion of the individual in the second position from the 3D model associated with the generated second position.

[0045] In one execution mode, the comparison step comprises comparing the first set of points of interest with the second set of points of interest.

[0046] In one embodiment, the anatomical elements comprise bones of a skeleton portion and in that the set of anatomical elements comprises at least one skeleton portion comprising a set of bones connected to each other.

[0047] In an execution mode, the virtual attribute further comprises a first parameter, such as an angular distance between two bones, calculated from the 3D model associated with the second position.

[0048] In one execution mode, the virtual attribute includes a second parameter such as a screw diameter or length calculated from the 3D model associated with the second position.

[0049] According to another aspect, the invention relates to a system comprising hardware and / or software means for implementing the method according to the invention.

[0050] In one embodiment, the hardware means comprise information receiving means, information transmitting means, an optical acquisition device for acquiring optical and / or three-dimensional images, a display device and / or a human / machine interface. In one embodiment, the software means comprise a processor and / or a memory for executing said method.

[0051] In one embodiment, the hardware means comprises at least one sensor for capturing data on the location and orientation of a body portion in the surgical field.

[0052] In one embodiment, the hardware means comprise a first processor associated with a first memory for implementing the method according to the invention.

[0053] In one embodiment, the system includes an augmented reality display to project a virtual representation of the generated virtual attribute into a surgical field while allowing the surgical field to be viewed through the augmented reality display.

[0054] In one embodiment, the system further includes a robotic arm including a head configured to support a surgical tool.

[0055] In one embodiment, the system includes a second processor associated with a second memory to implement a method of real-time guidance of a robot arm.

[0056] In one embodiment, the first processor and the second processor are the same processor and / or the second memory and the first memory are the same memory.

[0057] In one embodiment, the method of real-time guidance of a robot arm comprises:

[0058] - determination of the relative position of the head of the robot arm;

[0059] - calculating a first guidance trajectory of the head of the robot arm as a function of the relative position of the head of the robot arm and the virtual attribute;

[0060] - activation of a kinematic of the robot arm to travel the guidance trajectory;

[0061] - receiving a new 3D model and / or a new virtual attribute generated by the method according to the invention for calculating a new guidance trajectory and activating new kinematics based on said new 3D model and / or the new virtual attribute generated. According to another aspect, the invention relates to a computer program product comprising instructions which cause the system according to the invention to execute the steps of the method according to the invention.

[0062] According to another aspect, the invention relates to a computer-readable medium such as a memory, in particular a non-transitory memory, on which the computer program according to the invention is recorded.

[0063] According to a final aspect, the invention relates to a computer-implemented method for assisting surgery comprising the following steps:

[0064] - Receiving a medical image of a predetermined body portion of an individual in a first position;

[0065] - Capturing position data of said portion of the body of said individual in a second position different from the first position;

[0066] - Comparison between the second position and the first position to generate a transformation parameter;

[0067] - The generation of a 3D model of the bones of said portion of the body, associated with the second position, from the received medical image and from said transformation parameter;

[0068] - Selecting at least one bone from said generated 3D model and generating a virtual attribute, said virtual attribute comprising an element characteristic of the position and orientation of said selected bone in the 3D model;

[0069] - Transmitting a display instruction to an augmented reality display to project a virtual representation of the generated virtual attribute into a surgical field comprising said body portion of said individual while allowing the surgical field to be seen through the augmented reality display.

[0070] Brief description of the figures

[0071] Other characteristics and advantages of the invention will emerge on reading the detailed description which follows, with reference to the appended figures, which illustrate:

[0072] Fig. 1: a schematic view of a device according to one embodiment of the invention. Fig. 2: a flowchart of the method according to one embodiment of the invention.

[0073] Fig. 3: A top view of a subject's hand showing examples of points of interest.

[0074] Fig. 4: An example of visualizations of the detected points of interest to serve as a basis for detecting the position and orientation of the hand.

[0075] Fig. 5: a simplified view of an example of a display device according to one embodiment of the invention.

[0076] Fig. 6: a flowchart of the method according to an embodiment of the invention where a model of the body portion associated with the first position is generated to generate the second body model associated with the second position.

[0077] Fig. 7: a schematic view of a device according to an embodiment of the invention comprising a guidance system including a robotic arm.

[0078] Fig. 8: a flowchart of the method according to one embodiment of the invention.

[0079] Description of the invention

[0080] The following description essentially concerns an example for the generation of a display instruction for an augmented reality display device; said display instruction being configured to display a geometric element characteristic of the position and location of a selected bone of the hand of a subject in the field of vision of the augmented reality device.

[0081] The proposed system and method can be adapted for example to other portions of the body such as the foot, ankle, wrist, vertebrae or any other part of the body for which the precise visualization of a geometric element characteristic of a bone selected on an augmented reality display.

[0082] In another example, the system and associated method make it possible to display such an augmented reality guidance element characteristic of an organ or a portion of an organ such as the heart, the colon, the intestinal or venous system.

[0083] The system according to the invention comprises a calculation module comprising at least one calculator for generating a display instruction and an augmented reality device connected to said at least one calculator for displaying elements in response to said generated display instruction. Preferably, the method described is implemented by computer. The steps of the method can thus be implemented by one or more processors or calculators.

[0084] Medical image acquisition

[0085] The invention comprises receiving 1100 a medical image 10 of a subject's hand 14.

[0086] A medical image 10 of the hand 14 is acquired by a medical image acquisition module.

[0087] Before proceeding with medical image capture, it is best to prepare the individual's hand 14 for optimal results. This may involve instructions to place the hand in the first predetermined specific position, possibly using props to keep the hand steady, and / or ensuring that the hand is clean and free of objects that could interfere with medical image capture.

[0088] Preferably, the medical image 10 of the hand is obtained when the latter is arranged in a first predetermined position, for example, placed flat on a support. Preferably, the support comprises a solid-colored surface. In one embodiment, a specific support, for example comprising stop elements, to guide the individual to place his hand in the first predetermined position is used.

[0089] The medical image 10 preferably comprises a three-dimensional image of the body portion. For example, the medical image 10 comprises a series comprising a plurality of cross-sectional images of the hand 14 allowing a three-dimensional representation of the acquired body portion.

[0090] The medical image 10 preferably comprises an anatomical image of a portion of an individual's body. For example, the medical image is acquired by a CT scan technique, also called a "computed tomography." A CT scan is a non-invasive medical imaging technique that provides detailed images of the inside of the body. A CT scan is also known as a computed axial tomography (CAT) scan. This technique uses X-rays and a computer to produce cross-sectional images (slices) of the body, allowing physicians to visualize internal structures, such as organs, bones, blood vessels, and soft tissues. In another embodiment, the acquired medical image 10 is a functional image of the body portion. For example, the medical image comprises an image acquired by an MRI (Magnetic Resonance Imaging) technique.MRI images advantageously allow for the visualization of bones, soft tissues, and other anatomical structures with high resolution without exposing the individual to ionizing radiation. The fundamental principle of MRI is based on the properties of hydrogen atoms present in the human body. When the patient is placed inside an MRI scanner, a strong magnetic field is applied, which aligns the hydrogen atoms in a specific direction. Then, short pulses of radiofrequency waves are emitted into the body, disrupting the alignment of the hydrogen atoms. When these radiofrequency pulses die down, the hydrogen atoms return to their original alignment. This relaxation process generates signals that are detected by special antennas in the MRI scanner. These signals are then transformed into detailed 2D or 3D medical images of the body's internal structures by a computer.

[0091] Generally speaking, the medical image 10 includes any image allowing the bones or skeleton of the body portion of an individual to be visualized as well as the soft parts of the body portion (organs, blood network, nervous network or muscles).

[0092] In one embodiment, the term "medical image" may be understood as a series of cross-sectional images (such as MRI or CT scan images) acquired using a suitable medical device in which each cross-section corresponds to an image at a different depth.

[0093] In one embodiment, the acquired raw slice images undergo preprocessing to reduce noise and improve data quality. Advanced filtering and normalization techniques are applied to eliminate artifacts and make the images more consistent.

[0094] In one embodiment, slice images may be subject to artifacts and disturbances, including noise, which may affect the quality and reliability of the extracted information. Different filtering methods, such as Gaussian filter, median filter, or bilateral filter, may be used to reduce noise while preserving important details.

[0095] In one embodiment, preprocessing the slice images includes applying contrast enhancement techniques, such as histogram equalization or linear contrast adjustment, to increase the brightness difference between tissues, thereby improving the visibility of details.

[0096] In one embodiment, preprocessing the slice images includes using artifact removal or correction methods to restore the integrity of the images and ensure their fidelity.

[0097] In one embodiment, preprocessing the medical images includes normalizing the intensities to make the slice images consistent by adjusting their grayscales or pixel values ​​so that they have similar intensity ranges.

[0098] In one embodiment, preprocessing medical images includes implementing a registration and registration technique to align acquired slice images of the same body region and properly overlay them to obtain a consistent medical image.

[0099] In the remainder of the description, the term “medical image” will be used to designate both an acquired raw medical image and a medical image having undergone a pre-processing step as described above.

[0100] Generation of a preoperative 3D model associated with the first

[0101] In one embodiment, the method according to the invention comprises generating a first preoperative 3D model 22 of the hand or a portion of the body of the individual in the first predetermined position in which the medical image 10 a was acquired.

[0102] The generation of a first preoperative 3D model 22 of the hand preferably comprises a detection of the body portion on the medical image 10 to identify the region of interest of the image corresponding to the hand in the medical image 10, distinguishing it from other surrounding anatomical structures or from the background. For example, a thresholding segmentation or contour segmentation method is used. The overall objective of the body portion detection step is to accurately isolate the region of interest corresponding to said body portion in the medical image. Once the body portion is detected, the processed medical images 10 are ready to be used in subsequent steps of the process, such as 3D reconstruction and anatomical modeling of said body portion.Accurate body portion detection advantageously allows for more reliable and better quality results throughout the 3D model generation process.

[0103] Preferably, the preoperative 3D model 22 is also generated from an optical image of the body portion and / or a point cloud of said body portion acquired when the hand is in the first position.

[0104] The generation of a preoperative 3D model of the hand comprises a preoperative three-dimensional reconstruction to advantageously generate a three-dimensional volumetric representation of the hand associated with the first position. The first initial 3D model and the initial three-dimensional reconstruction are preferentially associated with the first position.

[0105] Preferably, the three-dimensional reconstruction of the first preoperative 3D model of the hand is generated from the 3D medical image or from a series of 2D cross-sectional images of the hand.

[0106] In one embodiment, the three-dimensional reconstruction comprises a voxel-based reconstruction. This method relies on the discretization of space into small cubic units called voxels. Each voxel is assigned an intensity value based on the corresponding medical image. Using these values, a volumetric 3D model of the hand is created, where each voxel represents a small volume element in three-dimensional space.

[0107] In one embodiment, the three-dimensional reconstruction comprises a contour reconstruction comprising segmenting the contours of the different structures of the hand in each cross-sectional image of the medical image and combining said contours to create a 3D representation of the body portion.

[0108] In one embodiment, the three-dimensional reconstruction comprises a depth-based reconstruction approach comprising using the depth information of each cross-sectional image of the medical image to reconstruct the three-dimensional geometry of the hand.

[0109] In one embodiment, the method further comprises the generation 112 of a preoperative anatomical model 23. The preoperative anatomical model 23 can be carried out from the medical image 10 and / or from the three-dimensional representation generated as described previously.

[0110] Anatomical modeling preferably includes a three-dimensional modeling of an anatomical system of the body portion of which the optical image was acquired. For example, anatomical modeling includes a modeling of the bones of the user's hand.

[0111] Anatomical modeling allows us to classify the different anatomical structures present in the hand, including bones, joints, and soft tissues of the skin or flesh.

[0112] In one embodiment, the generation of the preoperative anatomical modeling 23 comprises a segmentation of each anatomical structure of the medical image 10 and the classification of said segmented structure from the characteristics of said segmentation. In another embodiment, the segmentation and classification of each structure of the hand is implemented by a learning function such as a deep neural network or generative adversarial networks (GAN), can be used to create anatomical models of the hand from the medical image and / or from the generated three-dimensional representation.

[0113] The first preoperative 3D model of the individual associated with the first position is thus generated from the medical image 10 and comprises a 3D representation of the hand whose anatomical structures are classified. Preferably, the preoperative 3D model comprises a three-dimensional representation of the bony anatomical structures of the hand and optionally the surface of the skin. This preoperative 3D model can be obtained by subtracting certain anatomical structures from the three-dimensional representation according to their classification.

[0114] In an alternative embodiment, the first preoperative 3D model 22 comprises a three-dimensional representation of the bones of the hand in which the bones of the hand are labeled to be able to be identified among the bones of said first 3D model and optionally a virtual representation of the surface of the skin. In one embodiment, the first 3D model comprises depth information of the surface of the skin relative to the surface of the anatomical bone structures of the first 3D model.

[0115] Said depth information can be generated from an analysis of the medical image used to generate the 3D model. In one example, the method for generating said depth information comprises measuring or calculating on the medical image the distance between the surface of the skin and the exterior surface of the anatomical element (bone, organ, muscle, etc.). Said calculated or measured distance is then converted into depth information. Preferably, each piece of depth information is associated with a point on the surface of the anatomical element.

[0116] The dimensions of each bone and the relative position of a bone with respect to adjacent bones can be determined from the acquired medical image. Preferably, these dimensional data can be recorded or stored, for example on a memory.

[0117] In one embodiment, the medical image 10 is transmitted to the PRO calculation module and the first preoperative 3D model is generated by the PRO calculation module. In an alternative embodiment, the first preoperative 3D model and / or the preoperative anatomical modeling associated with the first position are transmitted to the PRO calculation module and / or stored or recorded on a data storage unit such as a MEM memory (preferably non-transient).

[0118] Hand position and location data

[0119] In one embodiment, the method further comprises a step 120 of determining the position and optionally the location of the individual's hand.

[0120] The determination comprises the acquisition 121 of an image such as an optical image of the individual's hand when the individual's hand is in a second position different from the first position. The optical image may comprise a 2-dimensional or preferably a 3-dimensional optical image of the individual's hand. The image may be acquired by a remote image acquisition device. More preferably, the image is acquired by an image acquisition device integrated into an augmented reality headset as described below. Preferably, said image comprises an image of the individual's hand in an environment comprising objects which may make it possible to extract data on the location and orientation of the hand relative to said object in the environment.

[0121] In one embodiment, the determining comprises acquiring a three-dimensional image of the body portion such as a point cloud or a depth map.

[0122] The image acquisition device preferably comprises a 3D scanner. A 3D scanner is a device that uses multiple lasers or sensors to measure the three-dimensional geometry of the hand. It can provide a detailed 3D image of the hand, capturing its shape and contours accurately based on the principle of the laser's time of flight as it reflects from the hand. The scanner is designed to scan the individual's hand to record the depth information of each pixel.

[0123] The image acquisition device may comprise means for acquiring two-dimensional or three-dimensional optical images. For example, the acquisition device comprises a stereoscopic camera, i.e., a pair of stereoscopic cameras. A computing unit may then be used to extract depth information from the acquired 2D images, using parallax or angle difference. By combining these images, it is possible to obtain a 3D image of the hand. In another embodiment, the optical image acquisition device comprises a depth camera. In one embodiment, the optical image acquisition device comprises a LIDAR device for generating a point cloud or a depth map.

[0124] The term "optical image" therefore means an image in the visible or infrared range or a depth map or point cloud.

[0125] In one embodiment, the optical image comprises a point cloud of at least a portion of the hand. In an alternative embodiment, a point cloud representing at least a portion of the hand is generated from the acquired image.

[0126] The method then comprises the generation of data 11, called “position data” or “position indicator” as a function of the second position of the hand on the acquired image. For example, a position indicator is generated from said acquired image of the hand in the second position. Said generated position indicator is characteristic of said second position. Preferably, it is thus possible to reconstruct the second position of the hand from said generated indicator.

[0127] For example, said generated indicator comprises pose information such as angular values ​​of at least a portion of the joints of the hand. In another example, the generated indicator is associated with said second position of the individual's hand.

[0128] In one embodiment, the position data makes it possible to characterize the second position and makes it possible to reconstruct a virtual model of a hand in said second position.

[0129] The position data 11 may describe the relative position of the joints of the hand and their spatial coordinates and / or the relative positions of the different fingers of the hand and the palm of the hand.

[0130] In one embodiment, the position data further comprises location and orientation data of the hand and / or each finger of the hand and / or each phalanx of the hand. The location data comprises relative coordinates of the hand in the environment. The orientation data comprises the relative orientation of the individual's hand in the environment.

[0131] According to one embodiment, points of interest 206, 208 of the user's hand are generated from the acquired image or from the processing of the point cloud E1 generated as illustrated in FIG. 3.

[0132] Said points of interest may comprise centers of mass 208 of the hand or of a portion of the hand. The centers of mass may be generated at coordinates substantially corresponding to the center of a predetermined area of ​​the hand. For example, a point of interest may correspond to the center of mass of the palm of the hand, of a phalanx, of a finger.

[0133] The points of interest may comprise deflection points 206. The deflection points 206 are generated at the boundary between two adjacent portions of the hand. For example, a point of interest may be generated between the portions of the hand corresponding to two adjacent phalanges of the same finger. The location of such a point of interest may then correspond to the location of a joint, for example between two phalanges. The points of interest may comprise the end or tip of a finger. The different points of interest may be connected to each other by segments 207.

[0134] Points of interest may include coordinates of a point or area on the acquired image and labeling information corresponding to identification information. For example, the identification information makes it possible to identify a predetermined joint.

[0135] In one embodiment, the step of generating the points of interest comprises generating the coordinates of each point of interest on the acquired image or in the generated point cloud of the individual's hand in the second position.

[0136] Preferably, the position data are obtained from the coordinates of said points of interest. The pose of the hand in the second position can thus be determined from the angular differences between two segments 207 connected to the same point of interest 206; 208.

[0137] An example of a 3D model associated with a second position E2 is represented in FIG. 4 in which the points of interest 211, 212, 213 are generated from the transformation parameter as well as the segments 214 connecting said points of interest.

[0138] In another alternative or cumulative embodiment, the position indicator is generated from a trained learning function such as a trained neural network. Preferably, said learning function is trained from a collection of labeled training images. Each training image is an optical image of the hand in a position and comprises a label comprising the hand position data associated with said training image. Said learning function, once trained, is configured to receive as input an image of the hand and generate as output a position indicator of said hand on the image associated with said input image. In a variant, preprocessing of the hand image may be carried out, for example the generation of points of interest.In one embodiment, the learning function is configured to generate position data from the coordinates of the points of interest and / or the acquired image of the hand.

[0139] Comparison between the first position and the second position From the generated position data or from the position indicator, the second position of the body portion is compared (130) to the first predetermined position in which the preoperative medical image of the body portion was acquired.

[0140] This comparison step generates a transformation parameter 26. The transformation parameter 26 may for example comprise a transformation matrix making it possible to transform the position of the body portion into the second position from the first position.

[0141] In one embodiment, the transformation parameter 26 is obtained by mathematical operations of comparison between the generated position indicator associated with the second position and a pre-recorded position indicator associated with the first position and preferably different from the generated position indicator. In an alternative embodiment, the position indicator relating to the first position is determined from an optical and / or three-dimensional image of the body portion in the first position.

[0142] The comparison can be obtained by regression, subtraction or division operations.

[0143] The transformation parameter may be generated based on pre-recorded biomechanical rules. For example, the transformation parameter may include an angular value of rotation of each joint of the hand allowing a hand to move from the first to the second position. Preferably, the transformation parameter includes an angular value of each joint of the hand allowing it to move from the first to the second position.

[0144] Preferably, the transformation parameter is a parameter associated with the second position and converting the position index associated with the first position into a position index associated with the second position.

[0145] In one embodiment, the generation of the transformation parameter comprises the detection or calculation of angles of the different parts of the body portion around joints of said body portion (for example the angle between two adjacent phalanges) from the acquired optical images. Preferably, the transformation parameter is generated by comparing the angular values ​​of joints associated with the first pre-recorded position and those associated with the second position.

[0146] Generating an updated 3D model

[0147] The method comprises generating 140 a second updated 3D model 14 associated with the second position. The updated 3D model preferably comprises a 3D model as described for the preoperative 3D model. The updated 3D model 14 is however associated with the second position.

[0148] In a first embodiment illustrated in Figure 2, the second updated 3D model 14 is generated from the acquired images of the body portion in the second position in the same manner as the first initial 3D model is generated from the acquired images of the body portion in the first position.

[0149] The updated 3D model includes an anatomical model comprising a three-dimensional model of said anatomical system associated with the second position. For example, the anatomical system includes a representation of the skeletal system of the hand composed of the bones of the hand.

[0150] Generating an updated 3D model comprises a three-dimensional reconstruction to advantageously generate a three-dimensional volumetric representation of the hand in the second position. Said three-dimensional reconstruction may be the same as that described for the first initial 3D model.

[0151] In a particularly advantageous embodiment illustrated in FIG. 6, the anatomical modeling associated with the second position is generated from the pre-recorded pre-operative anatomical modeling 23 associated with the first position and from the generated transformation parameter 26 associated with the transformation of the body portion in the second position from the first position.

[0152] In this mode, transformation parameter 26 is applied to the preoperative anatomical model associated with the first position to obtain an anatomical model associated with the second position.

[0153] In one embodiment, the angular value of each joint of the first preoperative anatomical model associated with the first position is modified by the value of said joint according to the position data associated with the second position to generate the updated anatomical model associated with the second position.

[0154] Preferably, the shape of each anatomical element is generated from the previously acquired medical image associated with the first position of the body portion. Preferably, the position and orientation of each anatomical element relative to each other are generated from the transformation parameter and / or from the comparison between the first predetermined position and the second position.

[0155] In one embodiment, an anatomical element must be understood as an object of a 3D model defining a three-dimensional volume, itself defined by a surface, a point cloud, a point mesh.

[0156] A 3D model can thus be defined by one or more anatomical elements.

[0157] An anatomical element of the 3D model can include a representation of bones, an organ, a muscle, a tendon.

[0158] The updated anatomical modeling associated with the second position preferably includes a representation of the user's hand skeleton, i.e., a set of bones connected to each other. Preferably, the connection between two bones is symbolized by a joint labeled by an identifier.

[0159] In one embodiment, the updated 3D model associated with the second position is generated from the pre-operative 3D model associated with the first position.

[0160] Preferably, the anatomical elements of the updated 3D model are identical to the anatomical elements of the preoperative 3D model.

[0161] In one embodiment, the updated anatomical modeling is generated from the preoperative anatomical modeling on which a spatial transformation in translation and / or rotation of the anatomical elements (for example bones) is applied according to the transformation parameter 16 previously calculated and optionally according to a set of pre-recorded biodynamic rules.

[0162] For example, biodynamic rules may set constraints on the angular displacement of certain joints. In particular, biodynamic rules define axes of rotation for each joint and / or ranges of authorized and / or prohibited values ​​for each joint of the body portion.

[0163] In one embodiment, the second updated anatomical model associated with the second position may include depth information. The depth information includes distance information between the surface of a point of the updated anatomical model and the skin of the subject. Preferably, this information is generated from the acquired medical images. As such, the acquired medical images are processed so as to determine the distance between the surface of a point of a determined bone of the subject and the skin of the subject. For example, the distance between the surface of a bone and the skin of the subject is measured on the medical images and associated with the bone or point of the bone used for said measurement.

[0164] This depth information is preferably associated with coordinates of the point of the anatomical model from which the distance was measured relative to a reference point located on said anatomical element. Said coordinates can be calculated from the two-dimensional coordinates of the point on the medical image and as a function of the section image used or the depth associated with said section image.

[0165] Preferably, the depth distances between the skin surface and the anatomical elements of the second updated anatomical model associated with the second position are similar to the distances determined during the construction of the first preoperative anatomical model.

[0166] In one embodiment, the device comprises means for acquiring or receiving optical and / or three-dimensional images of the body portion continuously and for generating continuously and / or in real time a second 3D model updated from said images.

[0167] In a preferred embodiment, the anatomical modeling is generated by a trained learning function such as a trained neural network.

[0168] Said learning function is configured to receive as input: - a transformation parameter 26 associated with the second position and converting the position index associated with the first position into a position index associated with the second position; associated with

[0169] - a medical image 10 of a portion of the body in a first position.

[0170] Said learning function is configured to generate as output an anatomical model associated with the second position.

[0171] Said learning function was preferably trained from a training database of a plurality of triplets each composed of medical images of the body portion in the first position, of transformation parameters associated with the transformation from the first position to a second position (different from the first position) and of an anatomical modeling of said body portion in the second position.

[0172] Generating a virtual image

[0173] In one embodiment, the method further comprises selecting 150, from the set of anatomical elements comprising the updated anatomical modeling associated with the second position, an anatomical element 16.

[0174] Said selection can be carried out via a human-machine interface INT.

[0175] Said human-machine interface INT further comprises one or more user interfaces.

[0176] The user interface(s) may, for example, include one or more computer mice, a keyboard, a touchscreen, a trackball, a microphone for providing voice recognition software running on a processor with spoken instructions, a camera for providing images of captured gestures or the like to gesture recognition software running on a processor, and so on. It should be understood that any existing user interface device may be used in conjunction with the AFF display system described hereinafter.

[0177] In one embodiment, a display system comprises a list of labels, each label being associated with a predetermined anatomical element. In one embodiment, the method comprises the generation 170 of a virtual image 17 and / or the generation of a display instruction to display said virtual image, in particular by an augmented reality device. Said display instruction can then be received by a display module such as that of an augmented reality display computer device as described below to display said virtual image.

[0178] In one embodiment, the virtual image 17 may comprise a portion of the updated anatomical model. For example, the virtual image comprises at least a portion of the individual's hand bone system generated by the steps previously described. Preferably, the position of the hand is detected (second position) by the image capture device and the virtual image comprising said portion of the anatomical model associated with the second position is displayed in real time.

[0179] Preferably, the virtual image is displayed in such a way that said portion of the anatomical model is superimposed with the portion of the individual's body.

[0180] In one embodiment, the virtual image is displayed so as to match the skin surface of the updated 3D model with the user's skin surface on the optical image. Preferably, the updated 3D model is displayed so that the skin surface of said 3D model is confused with the skin surface detected on the acquired optical image and / or with the point cloud of the body portion acquired by the image acquisition device.

[0181] In another embodiment, the system is configured to display an anatomical element of the 3D model at a relative position of the subject's skin by taking into account the depth information of said anatomical element of the 3D model. An advantage is to be able to display an anatomical element such as a bone of a hand at a precise location, in particular through the subject's skin which can be captured by the acquisition device in real time.

[0182] The displayed virtual image, including a portion of the 3D model, is thus advantageously consistent with the position of the subject's hand. The user thus obtains an augmented reality display of the 3D model.

[0183] Preferably, the portion of the 3D model displayed does not include the skin surface but the internal anatomical elements of the subject such as the bones of the subject. Nevertheless, thanks to the depth information (distance between the anatomical element and the surface of the skin of the subject) said anatomical elements are displayed with a particularly advantageous localization precision, even in the event of movement of the body portion of the subject. In one embodiment, the virtual image 17 is generated according to the selected anatomical element. The virtual image can thus comprise only the selected anatomical element 16 and / or the adjacent anatomical elements of the selected anatomical element 16.

[0184] Generating a virtual attribute

[0185] In one embodiment, the method further comprises generating a virtual attribute 15. The virtual attribute 15 comprises an element to be displayed reflecting a property of the selected anatomical element 16.

[0186] The virtual attribute 15 may comprise a direction axis. In one embodiment, the direction axis is generated based on the shape and / or orientation of the selected anatomical element 16 according to a rule or set of rules predefined and / or associated with said selected anatomical element.

[0187] In the example where the anatomical model comprises a system of bones of the hand, the virtual attribute 15 may comprise an axis merged with the longitudinal axis of said selected bone.

[0188] An advantage of displaying such an axis on an augmented reality display module is that it allows the surgeon to visualize a specific axis relating to the anatomical element that he has previously selected.

[0189] In one embodiment, the virtual attribute comprises a pre-generated anatomical indicator.

[0190] The anatomical indicator may be displayed as a number near or superimposed on the body portion. The anatomical indicator may comprise the value of a distance between the surface of the skin and the surface of an anatomical element, for example at a predetermined and / or selected point.

[0191] In one embodiment, the virtual attribute 15 comprises a plane whose coordinates are a function of the selected anatomical element. In one embodiment, the computing device described below displays the plane of the virtual attribute as a section plane so as to display a portion of the modeling associated with the second position superimposed on the part of the individual's body in contact with said generated plane. Preferably, said generated plane is therefore seen as a section plane.

[0192] In one embodiment, the operator can select via the human-machine interface INT a virtual attribute from a list of virtual attributes such as those previously described to be displayed and the computing device will in return display the selected virtual attribute.

[0193] Preferably, the virtual image 17 and / or the virtual attribute 15 are generated and / or displayed in real time from the optical and / or three-dimensional images acquired and received by the system. These two pieces of information can thus evolve in real time depending on the movements of the subject.

[0194] Visualization of the anatomical model in real time via an augmented reality headset.

[0195] Figure 5 schematically represents a particularly preferred embodiment of the AFF display system, in which this system is implemented as a head-mounted computing device such that the virtual image 17 can be generated in the view of the doctor in the first instance. The device further comprises a location for augmenting the reality (i.e., the real view) of the doctor, e.g., by superimposing the virtual image 17 on this real view.

[0196] In the context of the present invention, a head-mountable computing device is a device that can be worn on the head of its user and provides the user with computing functionality.

[0197] The head-mounted computing device may be configured to perform specific computing tasks as specified in a software application (app) that may be retrieved from the Internet or other computer-readable medium.

[0198] Non-limiting examples of such head-mounted computing devices include smart headsets, e.g., glasses, goggles, a helmet, a hat, a visor, a headband, or any other device capable of being supported on or from the wearer's head, etc.

[0199] The head-mountable computing device may comprise the PRO processor and / or the computing unit 2, e.g., in a component package 122.

[0200] The head-mountable computing device may further include an image sensor or camera 127 as a detector of the position of the body portion for capturing an image in a field of view of a wearer of the wearable computing device. The image sensor may be arranged such that when the head-mountable computing device is worn as intended, the image sensor aligns with the eyes of its wearer, i.e., produces a forward-facing sensor signal corresponding to the field of view of its wearer.

[0201] Such an image sensor or camera may be integrated into the head-mounted computing device, for example, integrated into a lens of a head-mounted computing device through which its wearer views his or her field of view, into a lens holder or frame for such a lens, or into any other suitable structure of the head-mounted computing device in which the optical sensor aligns with the field of view of the wearer of the head-mounted computing device.

[0202] Alternatively, such an image sensor may be part of a modular wearable computing device, e.g., a head-mounted image sensor module communicatively coupled via a wired or wireless connection to one or more other modules of the head-mounted computing device, wherein at least some of the other modules may be worn on parts of the body other than the head, or wherein some of the other modules may not be wearable, but rather portable, for example.

[0203] The head-mountable computing device generally includes at least one display module 125, which may be a transparent display module, under the control of a discrete display controller (not shown).

[0204] Alternatively, the display controller may be implemented by a processor of the head-mounted computing device.

[0205] The at least one display module 125 is generally arranged such that a wearer of the head-mountable computing device, e.g., the physician present at the first location 100, can observe the virtual image 17 of the ultrasound probe 11 displayed on the at least one display module 125.

[0206] Preferably, the at least one display module 125 is a transparent display module such that the wearer can observe at least a portion of a field of vision through the display module 125, e.g., the subject's hand or other body portion, the doctor's tools required for surgery.

[0207] In one embodiment, the head-mountable computing device includes a pair of display modules 125 including a first display module that can be viewed by the wearer's right eye and a second display module that can be viewed by the wearer's left eye.

[0208] Alternatively, at least one display module 125 may be an opaque display module on which an augmented reality scene of the field of vision of its wearer is displayed, for example a display module 125 making it possible to project a virtual image 17 into the augmented field of vision. To this end, the head-mountable computing device may comprise a camera for capturing the field of vision of its wearer.

[0209] The first display module and the second display module can be controlled to display different images, e.g. to generate a stereoscopic image as is well known in the art.

[0210] Alternatively, an image may be generated on one of the first and second display modules only such that the wearer can observe the generated image with one eye and the actual field of view with the other eye. The first and second display modules may be transparent or partially transparent display modules. Alternatively, one of the first and second display modules may be a transparent or partially transparent display module, while the other display module is an opaque display module, i.e., a display module that is not transparent such that the wearer cannot see through that display.

[0211] The at least one display module 125 may be provided in any suitable form, such as a transparent lens portion.

[0212] Alternatively, the head-mountable computing device may include a pair of lens portions, i.e., one for each eye, as explained above. The one or more transparent lens portions may be sized such that substantially the entire field of vision of the wearer is obtained through the one or more transparent lens portions. For example, the at least one display module 125 may be shaped as a lens to be mounted within the frame of the head-mountable computing device. Any other configuration known to those skilled in the art may be contemplated. It will be understood that the frame 128 may have any suitable shape and may be made of any suitable material, e.g., a metal, a metal alloy, a plastic, or a combination thereof.

[0213] Several components of the head-mountable computing device may be mounted within the frame 128, for example, within a component housing 122 forming part of the frame 128. The component housing 122 may have any suitable shape, preferably an ergonomic shape that allows the head-mountable device to be worn by its wearer in a comfortable manner.

[0214] Robotic arm

[0215] In one embodiment, illustrated in Figure 7, the system further comprises a guidance system. The guidance system may comprise, for example, a robot arm ROB for guiding an OTL surgical tool.

[0216] The surgical tool may include a cutting tool, a device for injecting a medicinal or anesthetic product, a device for inserting a medical device such as a pin, plate, or screw.

[0217] In this embodiment, the robot arm includes a head configured to receive a surgical tool.

[0218] Preferably, the system is configured to guide the robot arm head in real time according to a position and / or an orientation based on the updated 3D model associated with the second position.

[0219] In a first example, the system is configured to control the robot arm so as to orient the head of the robot arm in real time according to the generated virtual attribute. For example, the head of the robot arm is controlled according to the axis 151 of the virtual attribute 15, preferably to align the longitudinal axis of the surgical tool with said axis 151 of the virtual attribute. The robot arm can thus be guided, manually by an operator or automatically to reach a point of interest of the body portion of the subject along a predetermined axis, even if the subject is moving.

[0220] In a second embodiment, the system is configured to detect the position and / or orientation of the robot arm head in its environment such as relative to a point of interest of the body portion of the subject. The system is then also configured to generate a guidance trajectory for moving the robot arm head from its initial position to a predetermined target point. The predetermined target point and the guidance trajectory may be continuously updated from the newly generated updated 3D anatomical model and the generated virtual attribute.

[0221] Preferably, the robot arm may comprise sensors such as optical and / or three-dimensional image sensors, for example for generating the updated three-dimensional model or for detecting obstacles on the guidance path.

[0222] Software and calculator

[0223] It should be noted, for the avoidance of doubt, that although the method has been described as a series of sequential steps, it will be immediately apparent to those skilled in the art that at least some of the steps may alternatively be performed simultaneously, i.e., in parallel.

[0224] Aspects of the disclosed method may be provided in the form of a computer program product comprising a computer-readable storage medium having embedded therein computer-readable program instructions for, when executed on the system processor, causing those processors to perform the relevant steps of said method.

[0225] Aspects of the present invention may be implemented as a medical system 1, a display device control system 2, a display system 20.

[0226] Certain aspects of the present invention may take the form of a computer program product embodied in one or more computer-readable media having computer-readable program code embedded therein.

[0227] Any combination of one or more computer-readable media may be used.

[0228] The computer-readable medium may be a computer-readable signal medium or a computer-readable storage medium.

[0229] A computer-readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination thereof. Such a system, apparatus, or device may be accessible via any suitable network connection; for example, the system, apparatus, or device may be accessible over a network to retrieve computer-readable program code over the network.

[0230] Such a network can be, for example, the Internet, a mobile communications network or other.

[0231] More specific examples (a non-exhaustive list) of the computer-readable storage medium may include the following, a hard disk drive, random access memory (RAM), a drive, memory only (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any other suitable combination of the foregoing.

[0232] In the context of the present application, a computer-readable storage medium may be any tangible or non-transitory medium capable of containing or storing a program for use by or in connection with an instruction executing system, apparatus, or device.

[0233] A computer-readable signal carrier may comprise a propagated data signal having computer-readable program code embedded therein, for example in baseband or as part of a carrier wave.

[0234] Such a propagated signal may take various forms, including, but not limited to, electromagnetic, optical, or any suitable combination thereof. A computer-readable signal carrier may be any computer-readable medium that is not a computer-readable storage medium and that can communicate, propagate, or carry a program for use by or in connection with an instruction-executing system, apparatus, or device. Program code embodied on a computer-readable medium may be transmitted using any suitable medium, including, but not limited to, wireless, wire, fiber optic cable, RF, etc., or any suitable combination of the foregoing.

[0235] In some cases, the processor or computer may communicate with one or more external devices via the network. The processor or computer may be connected to the network via a wired connection (e.g., via an Ethernet cable) and / or a wireless connection (e.g., via a WiFi network). These external devices may include servers, workstations, and / or databases. The processor or computer may communicate with these devices to, for example, offload computationally intensive tasks. For example, the processor or computer may send medical images or one or more optical images via the network to the server for analysis and receive the results of the analysis from the server (e.g., respectively, the preoperative 3D model or the 3D model associated with the second position).Additionally (or alternatively), the processor or computer may communicate with these devices to access information that is not available locally and / or update a central information repository.

[0236] The device may also include a plurality of processors, each associated with one or more memories, and configured to together execute such steps. In one embodiment, the processor(s) may be remote and connected to the display by a data network.

[0237] The device further comprises a memory or several memories for storing or recording the display instructions and / or the 3D models generated by the method according to the invention and / or for storing the computer programs allowing, when executed by one or more processors, to implement the method according to the invention. In one embodiment, the device further comprises a transmitter connected to said memory for transmitting said instructions and / or 3D models generated on said memory to a data network. In one embodiment, the device further comprises a second transmitter connected for transmitting said instruction to a display device (preferably an augmented reality display device).

[0238] The device may further comprise communication means such as transmitters and receivers for exchanging information with the probe and / or a remote device.

[0239] The display may include means for receiving the various information received by the various means of the device to generate the final image to be displayed, preferably in augmented reality.

Claims

CLAIMS 1. A computer-implemented method (100) for assisting surgery comprising the following steps: ■ receiving (110) a medical image (10) of a predetermined body portion (11) of an individual in a first position; ■ the generation of a preoperative 3D model (22) of the body portion associated with the first position from the medical image received; ■ receiving (121) an optical image (25) of a scene comprising said body portion in a second position different from the first position; ■ the comparison (130) between the second position and the first position to generate a transformation parameter; ■ the generation (140) of a 3D model (14) of an anatomical structure of said portion of the body, associated with the second position, said 3D model (14) associated with the second position being generated from the transformation parameter and from said preoperative 3D model (22); said anatomical structure comprising at least one set of anatomical elements (16) of said portion of the body of the individual; the 3D model associated with the second position comprising depth information of each anatomical element (16) of said set relative to the surface of the skin; ■ the generation and transmission (170) of a display instruction (29) to an augmented reality display (AFF) to project a virtual representation of at least part of the anatomical elements of the 3D model associated with the second position, the display location of which is determined as a function of the depth information of the 3D model associated with the second position.

2. Method according to claim 1, characterized in that the preoperative 3D model (22) comprises depth information of each anatomical element of said preoperative 3D model relative to the surface of the skin from the received medical image, and in that the depth information of the 3D model associated with the second position is generated from said information of the preoperative 3D model, in particular from the depth information of each anatomical element of said preoperative 3D model.

3. Method according to one of the preceding claims, characterized in that the anatomical elements comprise bones of a portion of skeleton and in that the set of anatomical elements comprises at least one portion of skeleton comprising a set of bones connected to each other.

4. Method according to one of claims 1 to 3, characterized in that the body portion comprises at least one hand of said individual.

5. Method according to one of claims 1 to 4, characterized in that it comprises a step of generating a virtual attribute (15), said virtual attribute comprising an element characteristic of the position and orientation of an anatomical element (16) predetermined or selected in the 3D model (14).

6. Method according to claim 5, characterized in that the characteristic element of the virtual attribute (15) comprises an axis (151) or a plane (152) whose location and orientation are generated as a function of the position and orientation of said anatomical element (16).

7. Method according to one of claims 1 to 6, characterized in that the display instruction (29) is generated in such a way that said virtual representation is generated in a surgical field comprising said portion of the body of said individual while allowing the surgical field to be seen through the augmented reality display (AFF).

8. System (1), comprising software and / or hardware means for implementing the steps of the method according to one of the claims 9. Computer program product comprising instructions which cause the system according to claim 8 to execute the steps of the method according to one of claims 1 to 7.

10. Computer-readable medium (MEM), on which the computer program according to claim 9 is recorded.

Images