Integrated method and system for assisting in the implementation of a personalized therapeutic approach for patients undergoing medical-surgical care.

Abstract

Cloud-based surgical support computing architecture (200) with a module for reconstructing a two-dimensional and / or three-dimensional digital model from raw volume medical imaging data of a patient stored in the architecture, a graphical display interaction interface (300) with an operator, comprising a surgical intervention planning module configured to implement the following steps: displaying a scene on the display from said digital model, determining (P2) one or more pathologies of said patient by means of an identification and measurement module (202104) applied to said reconstructed digital model and / or to data of interest extracted from said raw data; modeling a plurality of additional three-dimensional scenes by applying a sequence of surgical gestures (P3) to said digital model and / or to said data of interest;simulation and computer modeling of the virtual execution (P5) of said surgical procedure sequences on the patient. Figure for the abbreviation: Figure 6;

Description

Title of the invention: Method and integrated system for assisting in the implementation of a personalized therapeutic approach for patients subject to medical-surgical care. Technical field

[0001] The present invention relates to an integrated method and system for assisting in the implementation of a personalized therapeutic approach for patients subject to medical-surgical care. State of the prior art

[0002] Operators or surgeons are faced with the need to perform an increasing number of medical procedures without technological assistance to absorb this growth through a homogeneous and efficient capacity supply across all healthcare offerings. For example, while many operators can perform operations with the physical assistance of other healthcare workers, educational or training resources are not available during the operation. In addition, the healthcare sector is constantly facing a reduction in staff which is not only detrimental to operators, but also to patients.

[0003] The description of the background provided herein is intended to generally present the background of the disclosure. The works of the currently named inventors, to the extent described in this section, as well as aspects of the description that cannot be considered part of the prior art at the time of filing, are not expressly or implicitly admitted as part of the prior art for the purposes of this disclosure. Statement of this disclosure

[0004] According to a first aspect of the invention, a surgical assistance method is proposed, comprising the following steps: obtaining raw medical imaging data corresponding to a patient from a remote computer architecture; reconstructing a digital model from the obtained raw medical imaging data, in which the digital model is two-dimensional or three-dimensional; generating a plurality of three-dimensional scenes by applying a sequence of surgical acts to the digital model; simulating a virtual execution of the sequence of surgical acts on the digital model of the patient using the plurality of generated three-dimensional scenes; and displaying, on a display, at least one of the following: the reconstructed digital model, the pathology(ies), the plurality of generated three-dimensional scenes and the simulated virtual performance.

[0005] The method according to the invention may optionally comprise a determination of at least one pathology on the basis of the digital model. The determination of the sequence of surgical procedures may then be based on at least one pathology.

[0006] The method can be based on a single, cloud-type computing architecture.

[0007] The method may include projecting the plurality of three-dimensional scenes onto the patient to guide an operator, wherein the projection comprises a holographic projection. The holographic projection may be manipulated by the operator or may be positioned on the patient.

[0008] The plurality of generated three-dimensional scenes or simulation of a virtual execution is displayed in a collaborative mode to remotely assist the operator in performing a corresponding procedure requiring multiple evaluations or to train the operator in an observational mode.

[0009] The method may comprise an implementation of artificial intelligence and / or a simulation of physical systems by digital mathematical modeling to model the simulated virtual performance.

[0010] The method may comprise a step of modeling digital models of anatomical elements, and / or implantable medical devices and / or ancillary instrumentation for assisting the surgical procedure, for carrying out said sequence of surgical procedures, in the form of 3D objects added to or generated within at least one additional 3D scene modeled, from the digital model and / or extracted data of interest. The anatomical elements, and / or the implantable medical devices and / or the auxiliary instruments may be based on the anatomy of the patient.

[0011] The method may include projecting the digital model onto the patient to guide an operator, wherein the projection includes a holographic projection.

[0012] The method may further comprise a step of generating digital models of anatomical elements, and / or implantable and / or ancillary medical devices, specific or not to the anatomy of the patient.

[0013] The method may comprise manufacturing by an additive or subtractive manufacturing installation of the digital files generated from said generated digital models.

[0014] According to a second aspect of the invention, a cloud-based computing architecture for surgical support is proposed, equipped with a module for reconstructing a two-dimensional and / or three-dimensional digital model from raw medical imaging data of a patient's volume recorded in the architecture, an interaction interface with graphical display with an operator, comprising a module for planning a surgical intervention configured to implement the following steps: display on the display of a scene from said digital model, determination of one or more pathologies of said patient by means of an identification and measurement module applied to said reconstructed digital model and / or to data of interest extracted from said raw data; modeling of a plurality of additional three-dimensional scenes by applying a sequence of surgical gestures to said digital model and / or to said data of interest; simulation and computer modeling of the virtual realization of said sequences of surgical gestures on the patient. Brief description of the figures

[0015] Other characteristics and advantages of the present disclosure will become apparent upon reading the detailed description which follows, for the understanding of which reference will be made to the appended drawings in which:

[0016] [Fig-1] [Fig. 1] is a high-level block diagram of an embodiment of a system according to the present invention.

[0017] [Fig.2] [Fig.2] is a high-level functional diagram of an infrastructure data processing of an embodiment of a system according to the present invention.

[0018] [Fig.3] [Fig.3] is an example of a functional diagram of the architecture computer science.

[0019] [Fig.4] [Fig.4] is an example of a functional diagram of the processing module and of algorithms.

[0020] [Fig.5] [Fig.5] is an example of a functional diagram of the processing module and data display.

[0021] [Fig.6] [Fig.6] is an example of a functional diagram describing the connections between the different modules of the system.

[0022] [Fig.7] [Fig.7] is a flowchart describing the stages of planning surgical, surgical guidance and corresponding diffusion.

[0023] [Fig.8] [Fig.8] is another example of a flowchart describing the steps specific to the implementation of surgical planning.

[0024] [Fig.9] [Fig.9] is another example of a flowchart describing the steps specific to the implementation of surgical guidance.

[0025] [Fig. 10] [Fig. 10] is a high-level representation of an implementation of the system.

[0026] [Fig. 11] [Fig. 11] is a graphical representation of an operator's view of a guided surgery using the system.

[0027] In the drawings, reference numerals may be reused to identify similar and / or identical elements. Detailed description of embodiments

[0028] The embodiments described below being in no way limiting, it will be possible in particular to consider variants of the present disclosure comprising only a selection of described characteristics, subsequently isolated from the other described characteristics, if this selection of characteristics is sufficient to confer a technical advantage or to differentiate the present disclosure compared to the state of the prior art. This selection comprises at least one characteristic, preferably functional without structural details, or with only a part of the structural details if this part only is sufficient to confer a technical advantage or to differentiate the present disclosure compared to the state of the prior art.

[0029] In the remainder of the description, elements having an identical structure or similar functions will be designated by the same references. Preliminary definition

[0030] API: In computing, an application programming interface (often referred to as API) is a standardized set of classes, methods, functions, and constants that serves as a facade through which software provides services to other software.

[0031] CPU: A processor (Central Processing Unit) is a component present in many electronic devices which executes the machine instructions of computer programs.

[0032] Medical Device: According to the World Health Organization (“The WHO Model Global Regulatory Framework for Medical Devices Including In Vitro Diagnostic Medical Devices”, World Health Organization Medical Devices Technical Series, page 8, 2019), a medical device means any instrument, apparatus, equipment, machine, implant, in vitro reagent, software, hardware, or other similar or related article, intended by the manufacturer to be used alone or in combination for human beings, for one or more of the specific medical purposes of: (i) diagnosis, prevention, monitoring, treatment, or mitigation of disease; (ii) diagnosis, monitoring, treatment, relief, or repair of injury; (iii) investigation, replacement, modification, or support of anatomy or a physiological process, (iv) support or maintenance of life; (v) design control;(vi) disinfection of medical devices; or (vii) providing information by means of in vitro examination of specimens from the body; human, and in each case does not achieve its primary intended action by pharmacological, immunological or metabolic means in or on the human body, but may be assisted in its intended function by such means. Certain additional products may be considered medical devices by certain regulations, including: (i) disinfecting substances; (ii) aids for persons with disabilities; (iii) devices incorporating animal and / or human tissues; or (iii) devices for in vitro fertilization or assisted reproduction technologies. By extension, these definitions are considered applicable to animal medicine for the present invention;

[0033] Flops (from the English Floating-point operations per second): Measurement of the speed of a processor or one of the arithmetic calculation units of a processor by the number of floating point operations that can be carried out per second;

[0034] GPU: A graphics processor, or GPU (from the English Graphics Processing Unit) is an integrated circuit providing the display calculation functions;

[0035] Teraflops: One teraflop corresponds to 1012 flops.

[0036] Technical problems addressed by the present disclosure

[0037] The increase in the average life expectancy of the world population is a major cause calling for large-scale health action to improve the efficiency of patient care processes. Indeed, this increasing recourse to medicine is taking place in conditions of increasingly contrasting imbalance between the number of patients and the number of health professionals, the former increasing more significantly than the latter. Applied to surgery, this problem becomes significant, because this discipline of medicine involves, on the one hand, significant, specific and costly organization and resources, and, on the other hand, is potentially high risk for each of the parties.These risks are either medical, up to lethality in an extreme - but real - case for the patient, or of the order of criminal and financial responsibility for the practitioner as well as for the health establishment which hosts the care (private or public structure).

[0038] The treatment of this problem requires a multi-axis analytical approach: analysis of the current patient care cycle, analysis of the growing role of surgical planning, analysis of the growing place of personalized therapy in the surgical act, analysis of the relevance of current technological solutions and, finally, the solution in which the present invention is projected as an emanation of industry 4.0.

[0039] This methodology will highlight the complex and interdependent problems addressed by the present disclosure. It will subsequently be illustrated, in a non-limiting approach, with respect to the surgical specialty chosen for its emblematic character: orthopedic and trauma surgery.

[0040] Analysis of the patient care cycle in the current context

[0041] With a view to deploying a constantly improving healthcare offering for patients, fully meeting the expectations of surgical professionals requires combining an in-depth knowledge of healthcare establishments and their operation within state healthcare systems, digital technologies applied to healthcare, and finally the development of medical devices in accordance with applicable healthcare regulations. Applied to the care pathway, however, surgical performance is only one step in the process. Therapeutic management can in fact be broken down into four main phases.

[0042] The diagnostic phase: after a preliminary medical consultation, the suffering patient is referred to a specialist doctor, at the same time as medical examinations are prescribed to enable the specialist to be sufficiently informed about the nature and extent of the pathology causing the suffering to make a diagnosis. The vast majority of these examinations are of a biological and imaging nature, the typology of which varies according to the diagnostic needs (computed tomography, angiography, radiology, magnetic resonance imaging, etc.). The imaging examinations are under the responsibility of a radiologist who therefore proceeds, on the basis of the two-dimensional (qualifier abbreviated by the term 2D throughout this document) or three-dimensional (qualifier abbreviated by the term 3D throughout this document) data produced for this patient, to the diagnosis which will be transmitted to the prescribing doctor as well as to the specialist doctor.The specialist doctor, depending on his specialty and the results of medical examinations, may or may not decide on a therapeutic approach involving surgery for this patient. If he decides to intervene surgically, the process moves into a so-called "preoperative" phase.

[0043] The preoperative phase: once the decision to operate on the patient has been made, the specialist doctor initiates a process of preparing the intervention. The first theoretical step consists of planning the surgery in question, consisting of technically structuring the possible steps: surgical approach to optimize access to the surgical site, choice of ancillary instrumentation or assistance equipment to facilitate or perform part or all of the surgical procedures (forceps, forceps, surgical guides, cutting tools, camera, laparoscopic equipment, robotic arm, etc.), choice of implantable devices if applicable (prosthetic mesh, osteoarticular orthopedic hip or knee prosthesis, vascular endoprosthesis, osteosynthesis equipment for fractures, pacemaker, etc.), analysis and measurement via the data available in imaging, etc. Surgical planning therefore consists initially of simulating, by some means whether it is the surgery to be performed: on paper, by computer, or even informally by experience without any particular support. The second step of this phase follows from the first and is operational: consultation with the teams in charge of the operating rooms to check the availability in stock of the necessary equipment or if it must be ordered / recommended to allow the surgery to be scheduled. The equipment specific to the anatomy of the patient concerned which may be required for certain surgical interventions is ordered during this step. In this case, the devices are said to be "custom-made" or "specific", implantable or ancillary (for example in the case of non-standard morphology). The entire preoperative approach is also carried out in consultation with other medical disciplines that may be involved in the medical care process, such as anesthetic medicine for example.

[0044] The intraoperative phase: this phase consists of performing the actual surgery on the patient according to the technical and operational preparations validated and programmed during the two previous stages. The surgeon proceeds according to his planning with the identified material which will have been prepared by the teams affiliated with the operating room and / or by external service providers for the intervention (ordering, control, assembly, verification, sterilization, etc.). It is important to specify that different and accelerated arrangements can be made for phases 1 to 3 in urgent cases where the patient's survival is in question (so-called "hot" surgery, as opposed to so-called "cold" surgery which is programmable). The choice of accelerated care followed by a limited hospital stay can also be based on budget optimization objectives for reducing the costs of therapeutic treatments.

[0045] The postoperative phase: this phase includes care from the time the patient wakes up until they leave the facility, including intermediate phases that may include, for example, care, monitoring, or one or more additional or complementary interventions. An increasing number of interventions are now performed on an outpatient basis and do not require a post-surgery hospital stay. It is during this phase that the patient's monitoring and rehabilitation protocols are defined, if applicable.

[0046] Analysis of the growing role of surgical planning as a vector of performance and safety for patient care

[0047] In view of the intensification of demand for care, the imbalance between the number of patients and the number of surgeons destabilizes a chain which can become a succession of bottlenecks. The situation today translates into a significantly increased capacity requirement for each player in the process (radiologists and surgeons in the first place, but also nursing staff, operating room staff operating room, anesthesiologists, etc.) at the risk of sacrificing quality for numbers. If a decreasing performance resulting from an excessive need that is difficult to control and not proportionate to the available resources is to be avoided in any service activity, it is a major health risk that must be eliminated.

[0048] Demand for imaging services is thus constantly increasing (systematization of prevention, diagnosis and monitoring steps), exceeding the supply of qualified radiologists. Technological advances in equipment now allow several hundred images to be obtained per examination, which can mean that for a single radiologist, there is an injunction to process several tens of thousands of images per day, which is incompatible with human capacity without compromising the quality of the diagnosis.

[0049] In this highly constrained context, the use of the power of IT to accelerate treatments and analyses, facilitate and make diagnoses more reliable, secure the transmission of information between different stakeholders in the therapeutic chain or even increase surgical capacities is a rapidly growing approach to the point that the digitalization of the patient care pathway must be considered an unprecedented technological revolution. As part of this transformation, the digital conversion of a growing number of health procedures is a major challenge that is gradually structuring this paradigm shift for medical teams with the aim of providing ever more and ever better care.

[0050] At the heart of this transformation is the crux of the therapeutic care pathway composed of multiple phases, in particular, diagnostic actions, preoperative actions, intraoperative actions.

[0051] This node focuses on the challenges of medical performance (therapeutic outcome) and operational performance (organizational fluidity, financial burden), as well as health security issues (exposure to risks). All medical specialties face these problems, in all countries where structured health systems exist.

[0052] One of the preferred solutions today to address part of the problem is the adoption of artificial intelligence technologies which should make it possible to quickly exploit the colossal mass of information made available by medical image production systems, and therefore to improve the capabilities of radiologists by automating diagnostic assistance systems in certain disciplines. In addition to this action dedicated to phase 1 of diagnosis, a second action, complementary to the first, consists of giving the surgeon an increased capacity to exploit medical imaging data to strengthen the surgical planning stage. Without replacing radiologists, this approach has the strong advantage of increasing the surgeon's knowledge of the pathological case to be treated, with the prospect of optimal preparation of the operation for the benefit of the patient. Since the recent beginnings of additive manufacturing, the increasing integration of a structured surgical planning approach using simple methods such as 3D printing in polymer or plaster of the organ(s) to be operated on for use by the surgeon shows, within a still emerging literature, a reduction in the duration of the surgical interventions concerned. This reduction is to be correlated with better preparation, therefore knowledge of the surgical site, before surgical opening, but also with modifications of the therapeutic approach that can be decided by the surgeon due to the contribution of new data. Thus, sometimes drastic reductions in the expected duration of surgery have been reported in the event of a radical modification of the initial interventional strategy. This reduction in the duration of the intervention has several consequences on the therapeutic and operational levels.On the one hand, it helps reduce the patient's exposure to the risks inherent in the procedure: nosocomial infection, blood transfusion requirements or even anesthesia. Apart from the objective of avoiding as much as possible being held liable for health damage by a patient who considers himself to have been harmed following an intervention undergone within their structures, hospitals also have an operational and financial interest in this reduction in operating time: they can increase the daily number of surgeries, and therefore their profitability.

[0053] Towards an increasingly personalized therapy

[0054] Therapeutic personalization in surgery is defined as taking into account the physiological, morphological or anatomical specificities of a patient as input data for the preparation and performance of an intervention, with the aim of adapting a medical protocol to their particular case. The objective is twofold, with an equivalent benefit / risk ratio: better surgical performance for stable or reduced operational costs.

[0055] In the context of an instrumented intervention possibly requiring the use of one or more implantable devices and / or ancillary instrumentation specific to the patient, the operational gain described in the preceding paragraph can be further increased. Beyond the contribution in performance, this therapeutic choice makes it possible to optimize the operating results of health institutes by reducing stocks of devices within the establishments and by reducing operational management requirements (unit traceability, sterilization, control, etc.).

[0056] This personalized approach, which has existed for several years for certain medical specialties (orthopedic surgery for example), nevertheless suffers from an application that remains exceptional. The origin of this accessory nature is twofold.

[0057] The first is linked to the implementation between the surgeon and the device manufacturers of planning / manufacturing processes which are often unsuitable for unit production and therefore expensive and restrictive for the parties (use of equipment normally assigned to mass production, transfer of analysis and design to dedicated third-party teams).

[0058] The second origin is a direct consequence of the first: due to the high cost of custom-made devices according to the method described in the previous paragraph, the majority of health regulations persist in a requirement of exceptional use which must be justified by a medical prescription. The inertia of regulatory texts prevailing, it logically implies the persistence of the initial model.

[0059] Although a strong trend towards dual surgeon / engineer training is emerging within the medical profession (designated by the English neologism "surgineer", a mixture of the English "surgeon" for surgeon and the English "engineer" for engineer), it remains rare for the moment and the role of a third party remains predominant at several levels: recovery of patient files (loading by the practitioner onto a remote server or by sending to the third party a digital storage medium such as a CD-ROM for example), 3D reconstruction, preparation for modeling (for example: segmentation of the 3D file to extract specific tissues or the skeleton of an organism for example), software knowledge, methodology for understanding 3D, computing power adapted locally to process the information and produce a preliminary model that will be reviewed by the surgeon, etc.

[0060] With the advent of the additive manufacturing process, and more broadly due to the increasing digitalization of patient care cycles already mentioned in this document, the linearization of the processes for producing patient-specific medical devices is nevertheless underway, with a regulatory dynamic that gradually tends to integrate this powerful development into legal requirements. The linearization of the chain of links between diagnosis (phase 1) and surgery (phase 3) however implies full mastery of the second essential link which is surgical planning, and in particular the reality of the software architecture which must support it to make it compatible with all the performance, safety and cost requirements.

[0061] Analysis of the relevance of current technological solutions

[0062] The support provided to the analysis activity in medical imaging by artificial intelligence algorithms therefore increases, for certain specialties and certain examinations, the capacities of the radiologist and secures the diagnosis by avoiding congestion of the imaging services. As mentioned previously, the direct chaining with the planning of the surgical intervention can be validated consequently to the diagnosis is currently only slightly systematized outside of complex surgeries with high risk for the patient (for example: neurosurgery, cardiology, oncology). With a view to generalizing the activity of preoperative preparation to all surgical specialties, it is therefore critical to be able to develop solutions that allow this linear chain between the imaging examination and the performance of the surgery by means of planning as a gateway to surgical assistance (manufacturing of medical devices, implantable or ancillary, modeled to measure by the surgeon as well as assistance by surgical guidance).

[0063] The most direct solution is therefore to transfer part of the diagnostic analysis into the hands of the practitioners themselves by assisting them with software adapted to their objectives.The added value for surgeons is therefore less in visualization and exploitation for personalized diagnostic purposes than in supplementing this preliminary work with software functionalities that allow the surgical stages to be modeled in their entirety. This approach is already implemented in commercial solutions that meet the business objective of planning, some also allowing a link to be made with external computerized surgical assistance systems, or even to model implantable devices or ancillary instrumentation specific to the patient's anatomy. All of these products find their justification in the dynamic of digital automation of phases 1 to 3.The question of the business response is not, however, the only answer to be provided, because if we wish to generalize the chaining of diagnostic / modeling / surgical assistance solutions, it is fundamental to ask the question of the technical implementation in coherence with the constraints of the entire health ecosystem. A medical device must indeed present a favorable benefit / risk ratio, but also provide surgeons with a reliable, efficient, secure, and economically acceptable tool. It is therefore a question, in a context of using increasingly massive data sets to support the activities of phases 1 to 3, of clearly understanding the IT architectural challenge behind the business challenge of providing medical devices to treat a particular pathology.

[0064] An analysis of the technical principles supporting current solutions highlights certain limitations linked to technical choices that are not very compatible with lasting resistance to the test of time.

[0065] The availability of computing performance at the heart of the problem

[0066] The medical imaging files currently preferred for 3D modeling during surgical planning are those obtained by computer-assisted tomography, or the map files obtained by Magnetic Resonance Imaging. Although generated by different acquisition processes, The files obtained are encoded in a standardized format called DICOM (acronym for Digital Imaging and Communication in Medicine). These files, containing a significant amount of information relating to the three-dimensional structuring of the object of analysis, are large files (several hundred megabytes) which require high computing power to be used in their raw state. The CT acquisition process makes it possible, for example, to reconstruct, from series of two-dimensional sections obtained by measurements taken remotely outside an object, and by stacking them, three-dimensional volumes containing all the information detectable by the device depending on its precision and the acquisition program. The basic units of these reconstructed volumes are called "voxels" in reference to their status as the smallest discretionary volume element in a three-dimensional scene.Since the degree of precision of an analysis is directly correlated to the definition of the object of said analysis, the precision of a diagnosis is therefore logically directly dependent, at equivalent calibration, on the number of voxels per unit of volume, and therefore on the resolution. The higher the resolution, the more faithful the 3D reconstruction will be to the analyzed object, nevertheless resulting, at equivalent volume, in the delivery of increasingly massive files. The expectation of reliability and high performance, both at the level of diagnosis and at the level of surgical planning, combined with increasingly precise and efficient acquisition systems, therefore implies the availability for the actors concerned of high computing power within the chain of patient care.

[0067] The reality today is that the technical proposals aimed at supporting this chaining (with modeling of medical devices specific to the anatomical morphology of a patient or not), struggle to meet all the requirements of the health specifications. The pitfall essentially concerns the methods of achieving performance of each link in the chain, which can follow several paths whose specificities should be carefully examined.

[0068] The most obvious way to reduce computational requirements is to simplify the model that will serve as the basis for the entire process. Reducing the amount of data in a file can be achieved by applying a compression algorithm that will select sub-batches of data according to a predefined setting. Compression of DICOM volume files is achieved by converting the volume data into voxels into surface data via a meshing action of the object of interest, commonly referred to as a "mesh". The rendering of said object is thus simplified into a three-dimensional object that now only presents the envelopes of the initial volume, envelopes made up of vertices, edges and faces. This action algorithmic thus makes it possible to reduce the size of files and facilitate their manipulation to reduce computing power requirements.

[0069] This therefore amounts to considerably reducing the quantity of information contained within these envelopes before even having carried out any manipulation, in particular at the second stage common to all surgical planning: segmentation. Segmentation consists in fact of applying a computer algorithm to 2D images or to the volume reconstructed in 3D from these images to display and / or retain only certain specific data, chosen according to predefined criteria. This may consist, for example, of retaining within a CT scan only a single type of tissue or anatomical element depending on the medical specialty and the objective (bone tissue or individual bone for orthopedic or maxillofacial surgery, blood vessels for vascular surgery, liver for hepatic surgery, etc.).The quality of the material that will be used for subsequent planning operations can therefore be significantly impacted by the application of this algorithm to an already compressed file, with already reduced fidelity compared to the reconstruction from the original DICOM.

[0070] The choice of surface mesh resolution determining the compression rate of the file upstream of the segmentation, mathematical modeling is therefore critical and questions, below certain thresholds, the fidelity of the rendering for diagnostic or planning use, therefore modeling the stages of surgery with a view to real therapy in a patient. The model risks in fact representing only insufficiently the reality of its patient, which seems unsustainable within a health ecosystem where performance, therefore the fidelity of the model to reality, must become the norm in view of the technologies available today.

[0071] Due to the computational requirements inherent in voxel manipulation, the choice of data simplification by meshing is widespread: prior visualization by the surgeon assimilable as a complement to radiological diagnosis (soft tissue surgery: neurosurgery, vascular or hepatological surgery, etc.), modeling of ancillary surgical assistance instrumentation specific to the patient's morphology (guides in orthopedic surgery: positioning, drilling, cutting, orientation, etc.) or even decision support by allowing the surgeon to choose an implant model from a library of pre-designed volume elements to superimpose them on the anatomical object in 3D.

[0072] Solutions developed natively in the Cloud as a response to dissatisfaction

[0073] The provision of high-performance computing for the manipulation of 3D voxel data or simply to move from a software model installed locally on operators' computers — and therefore limited to the performance of these machines and the need for development specific to the computer's operating system (Windows®, MacOS®, etc.) — to an internet platform-type model for the mobilization of remote computer servers allowing the remote processing of this calculation in addition to mass data storage is currently growing rapidly in the medical field, which is a major consumer of 3D data for the reasons presented above. This growth is thus logically supported by the marketing of services providing "Cloud" or "Cloud Computing" solutions (a generic term used in many languages, including in French-speaking countries, which may sometimes prefer "cloud solution") dedicated to the remote processing of either data storage or high-performance graphics computing.These services allow access to remote computing machines equipped with high graphic processing power through the concentration of graphics processors (acronym for Graphics Processing Unit). The Microsoft® company, associated with the designer and manufacturer of graphics cards NVIDIA®, is for example currently a major player in the provision of these solutions through its Microsoft Azure® product.

[0074] Companies offering computerized surgical planning products using Cloud Computing to streamline the chain between phase 1 (radiological diagnosis) and phase 2 (surgery preparation) are mainly positioned on two services that can be unitary or combined: (1) a direct link with medical records management systems and (2) use of the file by the surgeon via a web browser.

[0075] In other words, the first option is a direct link with medical imaging file management systems managed by radiologists, and designated by the English acronym PACS (Picture Archiving Communication System) to store files on a new server which will then be processed by a third party for surgical planning preparation activities.

[0076] Furthermore, the second option is the possibility for the surgeon to use the file in question by means of software which is no longer executed locally on the practitioner's computer, but operates on a website accessible via a web browser such as Google™ Chrome™, Microsoft® Edge, Mozilla Firefox® or Opera™.

[0077] This second option, which is very attractive, is logically highly sought after in view of the objective sought. However, today there are software offers on the market for assisting with diagnosis / surgical planning which suffer from several major pitfalls in order to be sustainable:

[0078] Data security: health data is now subject to strict controls and must be hosted on certified servers, which does not prevent a certain opacity on the control of data duplication. This is particularly true during the transfer and transit phases, when this data undergoes several successive algorithmic treatments which require intermediate recordings, therefore storage.

[0079] Technical restrictions: several problems are emerging today in the use of a solution based on the use of Cloud Computing, with the common origin not being the construction of integrated solutions, but rather the sometimes forced arrangement of technological bricks which were not initially developed for these purposes.

[0080] Real-time rendering of 3D scenes: the objective being to allow a surgeon to carry out advanced 3D planning combining simple visualization of anatomical data with complex modeling of surgical steps, the rendering must be faithful, fluid and of good quality for all operators and observers of a scene and not dependent on the performance of the computer terminal (computer, tablet, smartphone, virtual or augmented reality visualization system, etc.), which is the most common situation and also explains the use of data simplification.

[0081] 3D visualization engines: the initial path followed by industries to meet needs in terms of interactive 3D visualization is the diversion of 3D engines from video games, a discipline very demanding in terms of performance. However efficient these engines are in video games, this efficiency is not achieved by simple transposition for use in other fields, especially in association with Cloud Computing technology. They were simply not designed with these needs in mind, and, without evolution, are not future-proof solutions.

[0082] The collaborative approach: one of the major interests of Cloud Computing is the sharing of data in real time. In the case of surgery, particularly in the perspective of surgical planning activities or remote expertise carried out by a group of surgeons or doctors, it is fundamental to offer an accessible and efficient technology for sharing a 3D scene. This is today a cornerstone in the expansion of solutions based on GPU Cloud Computing, the performance of which is the combination of quality and availability of infrastructures (GPU Cloud, telecommunications networks, operators' equipment) and choice of software architecture to support these solutions with the lowest possible latency (latency refers to the time required for a packet of computer data to pass from the source to the destination through a network).

[0083] Historically, and in the absence of solutions developed to exploit Cloud Computing technology natively, the preferred route has been to adapt applications by designing them to exploit the hardware resources of the computer terminal, to then port them to the Cloud via virtual machines (in English Virtual machine 5) finally synchronized to visualize the same three-dimensional scene (for example: a heart reconstructed in 3D obtained by MRI). A virtual machine is an illusion of a computer machine created by emulation software that simulates the presence of hardware resources. The problem encountered is that the software ported to the Cloud is not developed to make the most of cloud functionalities (this would require a colossal amount of reprogramming work to match the requirements of GPU technologies), therefore posing, in addition to synchronization problems, the problem of high operating costs by requiring the deployment of a virtual machine with a GPU per operator and per connection.

[0084] This synthetic analysis of the situation shows that while the product offering is substantial, the real solutions enabling the use of Cloud Computing technologies to support the streamlining of the diagnosis / surgical planning chain in compliance with operator expectations, reimbursement systems and health regulations currently appear largely unsatisfactory. Operating a cloud solution does not simply consist of transporting the initial solution in a container to be operated within a virtual machine rented on remote servers. It is in this context that new solutions, designed natively for the use of the GPU Cloud through an innovative and complex architecture allowing the shared distribution of 3D scenes.The pioneering solution in this area, natively exploiting the GPU Cloud since its commercialization in 2017, is the collaborative platform in the form of a cloud-type farm for 3D rendering and the calculation of 3DVerse® algorithms (www.3dverse.com). This solution thus acts as the first 3D engine purposely structured via Cloud Computing, replacing the model requiring the resources of individualized peripherals. The deployment of this new typology of solutions thus constitutes a favorable base for the implementation of transversal projects such as the linearization of solutions in support of the digitalization of surgical patient care, subject to the development of new applications and communication models compatible with the international regulatory requirements applicable to medical devices.

[0085] As mentioned previously, before addressing professional surgical issues, the question of the technological structure that allows the implementation of an "all-in-one" solution at the surgeon level must be asked. This implementation has so far been described through applications for diagnostic assistance (phase 1), surgical planning (phase 2) and finally assistance in the modeling of implantable medical devices or ancillary instrumentation for surgical assistance (phase 3). This path dedicated to personalized therapy, however, has as a corollary the need to provide the surgeon with a sufficient information allowing him to implant and / or use the specific instrumentation with maximum safety, and therefore to guide his surgical procedure. The two historical paths are computer-assisted surgery (also referred to as surgical "navigation") and surgical robotics, both of which are used and can be used from a surgical planning phase that focuses on all the considerations mentioned so far. Depending on their degree of complexity and autonomy, these systems can integrate artificial intelligence algorithms, mainly for anatomical recognition. With a view to a broader use of surgical planning and a less demanding implementation in the operating room than that required by navigation stations or robots, a third path has naturally opened up as an aid to the surgical procedure: augmented reality.

[0086] Holographic assistance at the service of the surgeon

[0087] In terms of assistance with surgical procedures, the main path taken for several years is that of so-called "computer-assisted" surgeries which allow the surgeon to assist and control his intervention using computer systems. This is called surgical "navigation" or "guidance", because these systems have the principle of allowing the surgeon to follow all or part of a validated preoperative plan by comparing it in real time using specific data measured on the patient during the surgical operation. Osteoarticular prosthetic surgery (hip, knee, spine...) thus saw the birth of this technology more than 30 years ago, resulting in interactive assistance for navigation stations, which operate, for example, on the principle of detection by sensors (possibly infrared type) of fixed anatomical landmarks put in place by the surgeon, allowing him initially to calibrate the spatial positions of the "elements" of the scene: patient, surgeon's instruments, anatomical elements. In a second step, the system allows the surgeon to record anatomical data of the patient in 3D by palpation using a specific tactile instrument, possibly detected by the same sensors.

[0088] The system having recorded the spatial coordinates of the fixed references, it is thus capable of reconstructing the areas treated by the surgeon dynamically and then comparing them to the initial planning carried out before the intervention and assisting in the decision-making phase focused mainly on the dimensional choices of implantable medical devices (elements of total knee prosthesis for example). This example can be extended to other specialties such as neurosurgery or cardiac surgery, the technologies used for detection being able to vary according to the manufacturer of the stations in question (QR code for example). The original examinations can also be different (Magnetic Resonance Imaging, angiography by computed tomography...but the principles remain comparable. One of the avenues under development for the improved realization of this surgical guidance involves the use of a future technology: augmented reality, also called mixed reality. For convenience, only the term "augmented reality" will be used in the rest of the document.

[0089] Augmented reality is defined as the superposition of reality and elements calculated by a computer system in real time. These elements can be of multiple natures: sound, video, haptic, two-dimensional, three-dimensional, etc. Notoriously identified for cinema or video games, an increasing reliability of support technologies has opened up demanding and often highly regulated professional avenues for augmented reality, such as industry or the medical sector. If the attraction of these fields was initially the direct consequence of the astonishment of first-time operators faced with a technology that was still recently categorized as fictional, this is no longer the case today.Indeed, the reliability of augmented reality visualization equipment (example: Microsoft® HoloLens®) combined with prospects for gains in performance and profitability have now led to the existence of numerous use cases of interest, such as technical control in industry or assistance with surgery in the medical sector.

[0090] These visualization systems consist of individual devices, possibly in the form of helmets, masks, glasses, and more generally of all systems composed of a frame resting on the head (such as a headband audio headset) by headrest support (ears, nose, skull, etc.) allowing to visualize through neutral or adaptive optical lenses or projection onto said lenses or via screens 3D scenes composed of holograms juxtaposed with reality. Unlike augmented reality, virtual reality has the principle of total immersion of the person equipped with a visualization system within a scene calculated in real time by a computer system without superimposition on reality, and therefore cannot be used in interventional situations on patients. Virtual reality can however be a technology perfectly suited to surgical planning.

[0091] Compared with computer-assisted surgery stations, augmented reality offers high potential added value for surgical guidance during phase 3 (intraoperative), by simplifying the principle and optimizing performance. Indeed, the use of holographic surgical navigation replaces the heavy and instrumented iterative operation of a passive assistance station by providing the surgeon, in his field of operative vision, with 3D scenes constructed during phase 2 of surgical planning superimposed on reality. It is therefore a matter of the surgeon manipulating holograms using of his hands or even voice to adapt the 3D scenes to his convenience and guide himself using the information provided. One of the major advantages of this technology is its safety for the patient: apart from the visualization system that the practitioner must wear, no additional equipment is required, reducing the risk of contamination within the operating room. It is of course necessary to validate the compatibility of the ergonomics of the augmented reality visualization system with the use concerned so as not to include any risk of hindering the surgical procedure by the presence of holograms.

[0092] The surgeon thus finds himself in a situation where he can integrate into his patient care process steps that make a direct link between the diagnosis and the guidance of his surgical procedure by means of surgical planning, using the patient data set from the imaging to carry out the entire process. He can also choose, depending on his needs, to use surgical planning to manufacture implantable medical devices or ancillary instrumentation to assist with the surgical procedure specific to the patient's anatomy. The structuring nature of the process of phases 1 to 3 therefore implies a strong interdependence. They must be the links in a chain that is as linear as possible to combine performance and safety, the two major requirements of medical device regulations. From Industry 4.0 to Surgery 4.0

[0093] This new approach to flow processes structuring patient care is thus characterized by a holistic understanding of successive operational issues, integrating both technical and medical requirements, while seeking to take advantage of the most advanced IT technologies. The resulting concept is thus very similar to that of Industry 4.0, which consists of making a strong and innovative industry sustainable by converging the fields of the virtual and the real around digital modeling.

[0094] Through permanent and real-time communication between the elements and actors of the operational chain, this convergence thus allows strong personalization at controlled costs, despite the low volumes produced. Industry 4.0, corresponding to the 4th industrial revolution after mechanization (1st), mass production (2nd) and automation (3rd), has as its focus the generalization of so-called "factories of the future" which natively integrate modeling, personalized production and communication technologies for its operation: 3D simulation, Cloud Computing, artificial intelligence, additive manufacturing, virtual reality, augmented reality... The transposition of these principles to the field of interest of the present invention to structure patient care from planning to guidance in the operating room, including or not by the manufacture of implantable or ancillary medical devices gives life to a concept that the authors designate as “Surgery 4.0”.

[0095] This concept is generalizable to any type of surgery using 3D imaging (generated directly or via the compilation of 2D elements). One of the most demanding medical specialties, among the most emblematic of the problem addressed by the present disclosure, is orthopedic and trauma surgery (hereinafter together being able to be designated by the single commonly accepted term of "orthopedic surgery"), mentioned throughout this analysis of the prior art. This non-limiting example is sufficient to concretely illustrate the need to remedy all or part of the drawbacks of the prior art described above, as well as the solutions provided by the present invention.

[0096] This introduction was written with a human patient in mind. The same drawbacks could be described with a patient of another animal species. The system

[0097] [Fig-1] illustrates a system 100 for assisting in the implementation of an approach personalized therapy for patients undergoing medical and surgical care. The system 100 comprises: (i) a data processing infrastructure 200 for processing data, (ii) a data processing and display module 300 for processing and displaying data, and (iii) a manufacturing facility module 500 for manufacturing medical devices such as implants or auxiliary instruments, whether standard or specific to the patient's anatomy, or anatomical elements of said patient.

[0098] The data processing infrastructure 200 comprises computing means and data storage means. The data processing infrastructure 200 may be connected to a distributed communication network, such as the Internet, in a cloud application platform enabling the operation and delivery of resources and services over the Internet by means of data storage on a remote server via a Cloud Computing solution. In other applications, the storage may be implemented in local data storage on individual computers.

[0099] [Fig.2] is an example of a functional diagram of the data processing infrastructure 200. The data processing infrastructure 200 comprises (i) a computing architecture 202, (ii) a data storage device, including medical data 204 (in English health database), (iii) a module for simulating physical systems using numerical mathematical modeling 206 (in English physical Systems simulation using numerical mathematical modeling), (iv) an artificial intelligence module 208 (in English artificial intelligence) containing computer programs intended to implement algorithmic functions capable of simulating the intelligence of an operator (v) interfaces 210, accessible by third-party systems, and (vi) a module for exporting digital files 212 for the manufacture of devices, in particular medical, implantable or ancillary devices, which may be specific to the anatomy of the patient or non-specific, or anatomical elements of said patient. In the remainder of the document, non-specific devices are defined as "standards".

[0100] In this description, a patient may be a human being or another animal species. According to at least one embodiment, the data processing architecture may exclude the module for simulations of physical systems by digital mathematical modeling 206. According to at least one embodiment, the data processing architecture may exclude the artificial intelligence module 208.

[0101] The computing architecture 202 is of the type with shared resources, unified data and remotely accessible by one or more simultaneous software clients. The sharing of resources allows scalability of the solution in response to increasing exploitation. The unification of the data makes it possible to secure their integrity, avoiding their intentional or involuntary alteration by fraudulent modification or degradation of the initial data by successive copying. The accessibility of the architecture is made possible by an open API (for Application Programming Interface) of the network type, making it possible to ensure the correct distribution of the architecture.

[0102] The computing architecture 202 may for example comprise a farm of machines, each of the machines comprising a CPU (Central Processing Unit) processor, and capable of integrating a GPU (Graphical Processing Unit) graphics processor, intended to operate a 3D rendering and cloud-based computing farm (example: 3DVerse solution www.3dverse.com).

[0103] GPUs deployed within the framework of architecture 202 may, for example, include professional NVIDIA Ampere-type graphics processors with integrated graphics processing capabilities to provide 3D rendering to meet the requirements of massively parallelized calculations.

[0104] [Fig.3] is an example of a functional diagram of the computer architecture 202. The computing architecture 202 more particularly comprises (i) a farm 2022 of virtual or physical machines, and (ii) a data storage device 2024. The computing architecture 202 aims to provide the following services: a 3D rendering engine 2026, a module for processing three-dimensional reconstruction of computer-assisted imaging data 2028, a module 20210 for processing data and algorithms.

[0105] The farm 2022 of virtual or physical machines may typically comprise virtual machines with a computing power of several teraflops intended to operate the rendering engine 2026 and the processing module and algorithms 20210. The machines are shared among several operators and their number can grow automatically according to the needs and the number of simultaneous operators. Typically, a machine can provide the services of the 2026 3D rendering engines and the 20210 module for one or more operators and the 2022 set is scaled according to the number of simultaneous operators.

[0106] The storage device 2024 allows the storage and management of the elements following management of the operator accounts (in English user account), the three-dimensional scenes (in English 3D Scenes) of the calculation architecture 202, the binary files of the basic resources (in English assets) constituting the 3D scene. These resources can be in particular 3D objects in voxels, in mesh, 2D or 3D textures or materials, the code files of the algorithms executed by the module 20210.

[0107] The 3D rendering engine 2026 can be implemented in the form of software or algorithmic functions that can be executed by special graphics cards (hardware type) that calculate a 3D scene by rendering the 3D projection, the textures (appearance of the surfaces of the objects viewed), the lighting effects (shadows, reflections, etc.), or even the physical behaviors of the deformation types of soft bodies, rigid bodies, particle behaviors or even the behaviors of fluids (liquids, gases, etc.). The whole constitutes a chain of functionalities to form a coherent channel for the distribution and successive processing of graphic information from the raw data to the operator's terminal (in English Graphics Pipeline).Types of 3D rendering engines include, but are not limited to, software-accelerated engines and hardware-accelerated engines.

[0108] In the context of the present disclosure, it is preferable to implement engines with hardware acceleration using the computing power of the set of machines or farm 2022. The 3D rendering engine of the computing architecture 202 combines voxel-type data and mesh-type data displayed in the same 3D scene, respecting their respective scales.

[0109] The 3D rendering engine 2026 is configured to generate a 3D scene from input data. The input data of the 3D engine may come from the storage device 2024. The input data of the 3D engine may come from data generated by the processing module 20210.

[0110] The 3D scene whose objects are contained in the storage device 2024 can be exported via an export interface 2002 of the computing architecture 202, which can be accessible through the module 210. The 3D scene generated by the rendering engine 3D 2026 can be streamed to one or more instances of module 300, via a stream generation interface 2004.

[0111] The computer-assisted three-dimensional reconstruction processing module for imaging data 2028 has an input interface for receiving data, as well as an output interface for transmitting processed data. The input data is received from the storage device or database 204, itself supplied with files from medical imaging examinations and downloaded to a remote server by the operator of the application. The input data is data from volume imaging examinations with 3D reconstruction via back-projection algorithms, for example of a DICOM type computed tomography nature (X-ray tomography or MRI for example).

[0112] The output data is voxel type data (3D texture). The processing module is also capable of transforming the voxel type data into mesh type data representing the exterior part of the object in voxel. The processing comprises the calculation of a three-dimensional scene corresponding to an area of ​​a body, human or animal, from for example DICOM type data. The voxel and mesh type objects obtained in the form of binary files at the output of the processing module as well as the composition information of the scene are hosted within the storage device 2024.

[0113] [Fig.4] is an example of a functional diagram of the processing module module and of algorithms 20210. The processing module and algorithms module 20210 is configured to receive input data, process the input data, and generate output data. The processed data is stored within the storage device 2024.

[0114] The data processing module 20210 comprises: a sub-module 202102 for segmenting the biological elements of interest, a sub-module 202104 of identification and measurement tools to assist in the characterization of the patient's pathology, a sub-module 202106 for 3D simulation of a surgical sequence as a unitary element of the surgical treatment, and a sub-module 202108 for generating volumes corresponding to implantable medical devices and / or ancillary surgical assistance instrumentation, both of which may be standard or specific to the patient's anatomy, or anatomical elements, from the output data of the sub-module 202106.

[0115] To carry out the various processing operations, the module 20210 uses the computing power of the farm 2022. The output data can be sent to the storage device 2024.

[0116] The sub-module 202102 for segmenting biological elements of interest from medical imaging data is configured to determine a segmentation from the input data. These input data constitute, after processing of the initial imaging examination of the patient in DICOM format by the 3D reconstruction module 2028, a 3D object preferably in voxels stored in the database 2024 to maintain a high level of fidelity to the patient's anatomy.

[0117] The sub-module 202102 performs the following operations: generating several 3D textures representing the different types of elements of interest, such as for example different types of tissues (bones, muscles, air, etc.) depending on the intensity level of each voxel, and extracting the individual anatomical elements, such as for example specific bones (femur, tibia, radius, vertebrae, jaws, etc.) in the context of an application to orthopedic or maxillofacial surgery, by different possible methods, a non-exhaustive list of which includes: clipping by selection of voxels by the operator with a manual tracing type tool, selection of contiguous voxels, propagation / dilation of the selection, identification of concave shapes to delimit the articular surfaces, shape recognition, statistical co-location of anatomical parts to predict the belonging of a voxel to an organic tissue.Optionally, the method can process and refine the segmentation with smoothing or correction operations of the 3D structure.

[0118] For example, the segmentation can concern tissues of any nature (bone, tendon, epidermal, dental, vascular tissues, etc.), anatomical elements such as individual bones (femur, tibia) or even an organ such as the liver, pancreas, prostate or heart depending on the medical specialty concerned. The segmentation sub-module 202102 is used in step A2 of the synopsis presented in [Fig.7].

[0119] The identification and measurement sub-module 202104 is configured to assist the operator in characterizing the patient's pathology, for example from the input data or from the input data segmented by the data segmentation sub-module 202102. This characterization is carried out from the observation of the original 3D object in voxels and / or from the segmented elements from the sub-module 202102, stored in the storage device 2024. The operator may nevertheless choose to work on non-segmented input data from the processing of the module 2028 and hosted in the storage device 2024.

[0120] The operator can also add input data at this stage to structure the rest of the process by entering additional information into the system. This can, for example, involve annotating the 3D anatomical scene (additional identification of pathology, identification of anatomical landmarks, etc.). The term “labeling” is also often used to describe the annotation action, particularly for the purpose of training artificial intelligence algorithms.

[0121] According to a particular feature, in application to orthopedic surgery, the identification and measurement sub-module 202104 provides, for example, the information necessary for the parameterization of a surgical treatment of bone deformation by corrective osteotomy, comprising: identification of the presence or absence of a bone deformation by comparing, if applicable, the bones of the same type on the right side and the left side of the patient. This is the case, for example, of the so-called "long" bones of the appendicular system in mammals such as humans or dogs, for example: tibia, femur, humerus, radius, ulna, and for the same type of bone, the bones of the same nature compared to models of healthy, non-deformed bones.

[0122] The identification and measurement sub-module 202104 further provides the information necessary for measuring the length of the pathological bone and the healthy bone (if applicable), identifying and measuring the anatomical axis of the pathological bone, identifying and measuring the mechanical axis of the corrected bone, measuring the articular angle between the corrected bone and the bone(s) attached to said joint, annotating anatomical landmarks on the pathological bone, annotating required information (possibly three-dimensional diagrams) for the rest of the planning process. The identification and measurement sub-module 202104 is used in particular in step A3 of the synoptic shown in [Fig.2].

[0123] The sub-module 202106 for simulating a surgical sequence S(i) is configured to calculate a 3D sequence corresponding to a step of the surgical intervention envisaged by the operator; sequence also identified by the expression “operating time”. The term surgical sequence is retained in the remainder of the document. The sub-module for modeling the surgical treatment 202106 uses the input data from the module 2028 or the input data segmented by the data segmentation sub-module 202102, or the patient identification and measurement data generated by the sub-module 202104.The sub-module 202106 for simulating a surgical sequence is configured to perform the following operations, based on the segmented elements of interest from the sub-modules 202102 and 202104, elements stored in the storage device 2024: (1) allow the operator to manipulate the different elements from the sub-modules 202102 and 202104, and (2) propose an assistance sequence S(i) for modeling the surgical treatment planned for the operation. This sequence must allow the operator to validate or adjust the parameterization of the modeling of said operation.

[0124] The simulation of a step, automated or not, is directly dependent on the medical intervention envisaged, each intervention being able to be natively broken down into distinct sub-phases from one medical specialty to another. Each sub-phase corresponds to a step S(i), the maximum value of the variable "i" corresponding to the total number of sequences to be simulated (by convention, we will write 1 < i < n).

[0125] Thus, according to a first particularity, in application to orthopedic surgery for the treatment of correction of a bone deformation by osteotomy, this could correspond to the following succession of sequences for n = 4. S(l) also called first sequence comprises a calculation and display of a 3D scene presenting the calculations (equation, positioning, etc.) and positioning on a deformed bone of one or more bone correction cutting plane(s).

[0126] S(2), also called second sequence, includes the steps: calculation and display of the result of the application of the bone cut displayed in S(l) on a deformed pathological bone to allow the operator to see the visual result of the virtual gesture of correction of the deformation. S(3), also called third sequence, includes the display of a 3D scene presenting the rate of contact surface between the different bone segments of interest resulting from the cut of the pathological bone according to the manipulation of these bone segments. This step makes it possible to virtually model the surgical gesture of adjustment of the correction of the bones to be carried out in the operating room. S(4), also called fourth sequence, includes the visualization of the bone after correction carried out in S(3) upstream of the modeling of implantable medical devices and ancillary instrumentation in view of the osteosynthesis phase which will make it possible to fix the correction in time.

[0127] For all scenes S(i), the operator's interaction is dynamic and allows the consequences of a modification of the calculation input data on the displayed result to be observed in real time.

[0128] According to a second particularity, in application to vascular surgery, this could correspond to the constituent stages of the treatment of a venous thrombosis (obstruction of a venous channel by the appearance of a blood clot called a thrombus) within a vascular system reconstructed in 3D. This treatment, called thrombolysis, can for example be simulated by a sequence S(1) incision, S(2) mechanical disintegration, S(3) aspiration of the thrombus by catheter, S(4) closure (n = 4).

[0129] According to a third possibility, in application to liver surgery, this could be the modeling of the resection treatment of liver tumors after characterization of the tumor (dimensions, anatomical location) and simulating the successive surgical gestures S(i) of resection by coelioscopic instrumentation (medical endoscopy technique for intervention in the abdominal cavity - also called laparoscopy). The sub-module 202106 for simulating a surgical sequence S(i) is used in step A4 of the synopsis presented in [Fig.7].

[0130] The 202108 sub-module for digital generation of standard or patient-specific DM(j) medical device is configured to transform the output data of sub-module 202104 and / or sub-module 202106. This sub-module allows the operator to generate a number "j" of implantable devices and / or ancillary surgical assistance instrumentation in the form of 3D objects added to the main scene S(n). This therefore involves generating a three-dimensional representation of a standard medical device or one specific to the morphology of the patient's anatomy, the shape of which may depend on the characteristics of the anatomical area of ​​interest defined by the operator by reproducing its topography with a high level of precision. Such reproduction may include the generation of geometric volumes of any kind, single or by assembly of these volumes, which may be of standard shapes with axes of symmetry (cube, sphere, cylinder, etc.) or specific and generated according to the output data from sub-modules 202104 and 202106.

[0131] According to a particular feature, the implantable devices generated are osteosynthesis plates adapted to the morphology of a pathological bone (fractured bone, deformed bone then corrected), or even vascular endoprostheses of the stent type of design adapted to the vascular anatomy to be treated following thrombolysis for example, the generation of recesses of adapted nature and geometry, opening or not, intended for example to receive medical devices generated by the same sub-module or resulting from the choice of the operator from a library of precalculated shapes, hosted from the storage device 2024 in anticipation of a permanent combination with the initial medical device at the time of surgical implantation.

[0132] This can also make it possible to optimize the 3D shape for the purpose of reducing the contact surface between the modeled medical device and the reference anatomical element. In the context of orthopedic and trauma surgery, this can be, for example, screw-type bone anchoring systems chosen from the constituent elements of a digital library, depending on their type and their dimensions, to be included within an osteosynthesis bone plate or temporary fixation devices of the surgical bone fixation pin type.According to a particularity, surgical assistance ancillaries can be designated by the generic term of "surgical guide" and possibly be composed of one or more elements integrating for example the functions of positioning and / or orientation of implantable devices, of cutting biological tissue(s), or even of bone drilling, depending on the type of surgery (orthopedic, traumatological, vascular, cardiac, neurosurgery, etc.).

[0133] According to a particularity in orthopedic and trauma surgery, this can be a custom-made cutting guide whose shape matches the surface of the bone at one or more location(s) defined by the operator by identical reproduction of its topography. including in particular one or more support planes for guiding a surgical cutting tool (example: oscillating saw), and / or a possibility of holes for temporary fixing devices such as surgical pins.

[0134] In various implementations, this may include a custom-made drilling guide whose shape matches the surface of the bone at one or more locations defined by the operator by identical reproduction of its topography, including in particular: a possibility of holes for guiding drilling of holes dedicated to receiving screw-type bone anchoring systems in the bone, and a possibility of holes for temporary fixation devices of the surgical pin type.

[0135] In various implementations, this may include generating a custom orientation guide whose shape matches the surface of the corrected bone at the location defined by the operator by identical reproduction of its topography, including in particular several holes for temporary fixation devices of the surgical pin type, and / or a possibility of counter-forming to guide the positioning of the custom bone plate.

[0136] The sub-module 202108 for digital generation of standard medical device DM(j) or specific to the patient's anatomy is used in step A5 of the synopsis presented in [Fig.7].

[0137] The medical data storage device 204 allows in particular the management of operator accounts to connect to the module 300 shown in [Fig.l]. The medical data storage device 204 also includes medical data associated with patients, derived from medical imaging data, metadata associated with the medical imaging data and information entered by the operating surgeon such as post-operative assessment, through the modules 302 and 304, illustrated in [Fig.5].

[0138] The medical data storage device 204 further comprises medical imaging data reconstructed in 3D by computer associated with patients and the post-operative assessment data entered in step G4. The medical data storage device 204 also comprises non-medical data, in particular for the management of operators, patient files, documents (orders, invoices, etc.).

[0139] The module for simulating physical systems by digital mathematical modeling 206 takes as input the output data of the modeling of the medical treatment from the module for simulating a surgical sequence S(i) 202106 or from the module for generating device volume 202108. It executes an algorithm which makes it possible to simulate and visualize the stresses and deformations induced within the assembly of the elements due to the mechanical stresses induced by the activity patient physics. The module uses the finite element analysis method to simulate the assembly behavior and identify the most stressed areas.

[0140] The sub-module 202106 for simulating a surgical sequence S(i) can operate in combination with the module 206 for simulating physical systems by digital mathematical modeling in order to take as input data the physical integrity of the anatomical part of interest. This combination makes it possible to map the areas of fragility of the anatomical part of interest and to provide the operator with additional information to take into consideration when planning his surgery. Applied to tumor resection surgery, whatever the organ or tissue concerned, this can thus make it possible to provide information on the impact of the resection procedure on the tissues surrounding the ablation zone by simulating the position of fragile areas when modeling the standard stressing of the anatomical portion concerned.The 202106 sub-module for simulating a surgical sequence S(i) combined with the 206 module for simulating physical systems by numerical mathematical modeling is used in step A4 of the synopsis presented in [Fig.7].

[0141] The sub-module 202108 for simulating a surgical sequence S(i) can operate in combination with the module 206 for simulating physical systems by numerical mathematical modeling in order to take as input data the physical integrity of the anatomical part of interest. This combination makes it possible to map the areas of fragility of the anatomical part of interest and to provide the operator with additional information to take into consideration when planning his surgery. Applied to trauma surgery for the treatment of a fracture, this can thus make it possible to model the implantable osteosynthesis device such as a bone plate, taking into account the quality of the recipient pathological bone. A multi-fragmented bone requires particular attention for the positioning of screw-type bone anchoring systems in solid areas.As a corollary, it is therefore appropriate to model a plate that is both specific to this positioning of the screws and which allows, once associated with the recipient bone, to maximize the long-term stability of the assembly {bone, screw-type bone anchoring systems, bone plate}. Modeling the constraints corresponding to normal use of the limb by a breast patient (walking for the lower limb for example) makes it possible to evaluate this stability over time and to validate — iteratively if necessary — the design of standard medical devices or those specific to the patient's anatomy. The sub-module 202108 for digital generation of standard medical devices DM(j) or those specific to the patient's anatomy combined with the module for simulations of physical systems by digital mathematical modeling 206 is used in step A5 of the synopsis presented in [Fig.7].

[0142] The artificial intelligence module 208 takes as input the data from the storage devices 204 and 2024. It integrates an annotation tool to categorize and label the information contained in the storage devices 204 and 2024, typically to classify the unitary elements constituting the data sets used by the artificial intelligence algorithms. In addition, machine learning and deep learning algorithms can be called via interfaces that can be of the API type by the sub-modules 202102, 202104, 202106 and 202108 to simulate in an automatic mode the complex tasks assigned in manual or semi-manual mode to the operator, therefore the surgeon.Module 208 thus makes it possible to categorize the actions of these sub-modules in an “intelligent” mode, to be broken down into sub-tasks or sets of automatic sub-tasks not requiring operator intervention to characterize input data and transform it into output data.

[0143] These subtasks include the automatic segmentation based on submodule 202102 of the different elements such as for example the different types of tissues (epidermal, vascular, bone, etc.) or the bones of the skeleton (femur, tibia, vertebra, pelvis, etc.) (steps A2 of [Fig.2]).

[0144] These subtasks also include automatic identification and measurement based on sub-module 202104, to perform analyses of the 3D model allowing the operator to accurately characterize the patient's pathology for the surgical intervention in an automated or semi-automated mode by partial automated assistance during the process. In one possibility of the present disclosure, this module provides the necessary information to the operator who will then be able, based on his training and experience of surgery, to strengthen his diagnostic thinking in order to continue the surgical planning process. In an alternative possibility, module 208 will directly provide the operator with a characterization of the pathology that it will be up to the operator to confirm or deny in order to move on to the following steps of surgical planning (step A3 of [Fig.7]).

[0145] These subtasks further comprise an automatic 3D simulation of a surgical sequence S(i) of the operation based on sub-module 202106 from the output data of sub-module 202104. This simulation allows the operator to simulate in an automated or semi-automated mode by partial automated assistance during the process. In one possibility of the present disclosure, this module allows the operator to predefine selection criteria, such as for example the preservation of the anatomical and mechanical axes of a fractured bone for the purpose of simulating the surgical sequence of osteosynthesis in the context of the orthopedic specialty (step A4 of [Fig.7]).

[0146] These subtasks further comprise an automatic digital generation of a standard medical device or one specific to the anatomy of the patient DM(j) on the basis of the sub-module 202108, from the output data of the sub-module 202106. The association with the module 208 thus makes it possible to generate and position in an automatic or semi-automatic mode (depending on the degree of assistance) implantable medical devices or ancillary instrumentation for assisting surgery, such as for example osteosynthesis systems {bone plate, bone anchoring systems] making it possible to obtain the best stability of the assembly in the context of a fracture repair, as well as all of the associated surgical guides. It may also involve automatically simulating a vascular endoprosthesis (step A5 of [Fig.7]).

[0147] In accordance with the content of Figures 7 to 9, the combination of one or more of these intelligent versions of the sub-modules 202102, 202104, 202106 and 202108 makes it possible to automate all or part of the surgical planning, comprising: characterization of the pathology, modeling of the surgical procedure, and modeling of a standard medical device or one specific to the patient's anatomy.

[0148] The interface module 210 consists of a set of functions enabling and facilitating communication between the data processing infrastructure 200 and third-party applications with the objective of mutually exchanging services or data. These interfaces may be, in a non-limiting manner, API, web service or file exchange type interfaces.

[0149] The digital file export module 212 for the manufacture of medical devices of the implant type or ancillary instrumentation, standard or specific to the anatomy of the patient, or anatomical elements of said patient. The digital file export module 212 for the manufacture of medical devices of the implant type, or ancillary instrumentation for assisting surgery, standard or specific to the anatomy of the patient 212, or anatomical elements, takes as input the data from the storage devices 204 and 2024, from steps A1, A2, A3, A4 and A5. It executes a set of algorithms which make it possible to generate files intended for the module 500, which may contain in particular: digital files, a metadata file, 2D representations.

[0150] The digital files represent implantable devices modeled from the patient's anatomy (e.g. bone plate in orthopedic surgery); ancillary surgical assistance instruments also modeled from the patient's anatomy, possibly of the "guide" type, custom-made; or even anatomical elements. These files are typically in a format which may be in particular STL (abbreviation for the English term Stereolithography), AMF (for English Additive Manufacturing File format) or OBJ (for English Object file).

[0151] The metadata file may include additional information necessary for manufacturing (dimensional references, specific geometric information, control tags, positions and typology of specific locations intended to receive permanent or temporary implantation systems for surgical purposes). The 2D representations make it possible to generate industrial type definition plans, directly usable for manufacturing. Module 212 is used in step A6 of the synopsis presented in [Fig.7].

[0152] [Fig. 5] is an example of a functional diagram of the data processing and display module 300. The module 300 for processing and displaying data intended for operators comprises a module 302 for processing and displaying data intended for any type of computer terminal. A non-limiting list includes: computer, smartphone, tablet, virtual reality device, augmented reality device. The data processing and display module 300 further comprises a module 304 for processing and displaying intended for any type of computer terminal allowing visualization in augmented reality.

[0153] [Fig.6] is a functional diagram describing the connections between the different modules of the system. Several simultaneous executions of the modules 302 or 304 can connect jointly to the module 200 and view and manipulate the same information and objects. This possibility offered by the system which is the subject of this document thus allows remote or local collaboration between several operators, possibly to share the experience of surgical planning and surgical guidance, but also for assistance between peers, professionals in the medical field, for the purposes of assistance in the expertise of pathological cases or even medical training.

[0154] The operator terminal module 302 is in the form of a heavy client (local) type application installed on the operating system or thin client accessible through an internet browser. This operator terminal module 302 is accessible for example from a computer terminal (computer, smartphone, tablet, virtual reality viewing device, augmented reality viewing device, etc.). This operator terminal module 302 comprises in particular: a 3D scene reception interface in the form of a stream 3022, a display and operating module 3024 allowing interaction with 200 through the interface 210.

[0155] The interface module 210 allows the operator to carry out surgical planning, regardless of the medical specialty, via the following operations, via a series of steps, shown in [Fig.7]. Step PI comprises: an upload of medical imaging files, for example of the DICOM type, to the device storage which may include medical data 204 via the interface 210. The data is then processed by the module 2028 to be generated in three dimensions and constitute the initial 3D scene of the entire process visualized in the form of a 3D scene by the module 202 and received through the flow generation interface 2004.

[0156] All of the steps P2 to P5 which constitute the continuation of the process allow the manipulation and creation of 3D objects of the 3D scene with the operator interface devices of the computer terminal used (computer, smartphone, tablet, etc.), relying on the interfaces 210 provided by the infrastructure 200.

[0157] Step P2 comprises a characterization of the pathology affecting the patient by successive and possibly iterative execution of the following two sub-modules: segmentation of the tissues by calling the sub-module 202102 by the interfaces 210 provided by the infrastructure 200, identification and measurement by calling the sub-module 202104 by the interfaces 210 provided by the infrastructure 200.

[0158] Step P3 comprises a modeling of the surgical gestures by “i” calls (1 < i < n) of the 3D simulation sub-module of a surgical sequence S(i) 202106 by the interfaces 210 provided by the infrastructure 200. As for step P2, several successive calls can thus be made to simulate the number of surgical sequences required by the planning of the pathology of interest. Each call generates a 3D scene different from the previous one (n scenes).

[0159] Step P4 (if applicable) comprises a modeling of a standard medical device or one specific to the patient's anatomy by "j" calls (1 < j < n), m corresponding to the number of devices modeled) of the digital generation sub-module of a standard medical device or one specific to the patient's anatomy DM(j) 202108 by the interfaces 210 provided by the infrastructure 200. As for steps P2 and P3, several successive calls can thus be made to simulate the number of medical devices required per modeled medical treatment. Step P5 comprises the review and validation of the planning by visualizing the different steps P2 to P4 (or P3).

[0160] The interface also makes it possible to enter information from the post-operative assessment at step G4 of the surgical guidance ([Fig.7]), which may in particular contain dictated or entered text transmitted to the medical data storage device 204 by the interface 210, or medical imaging files possibly of the DICOM type transmitted to the module 2028 by the interface 210.

[0161] According to one possibility of the present disclosure, step P2 can integrate a function for exporting in digital format the elements displayed or chosen by the operator for manufacturing by the installation 500 via the module 2002, and possibly those stated in the introductory part. According to one possibility of the present disclosure, step P4 is optional and planning the surgical procedure may be sufficient to then carry out surgical guidance.

[0162] Returning to [Fig.5], the augmented reality module 304 is in the form of a software application embedded in augmented reality systems. The module 304 comprises a 3D scene reception interface in the form of a stream 3042, a display and operating module 3044 for interacting with 200 through the interface 210, a 3D scene import interface 3046, connected to the export interface 2002 of the computing architecture 202, potentially through the interface 210, used in step G2, a data processing unit 3048 arranged to receive data from the 3D scene import interface 3046 and data from the 3D scene import interface in the form of a stream 3044.

[0163] The data processing unit 3048 is configured to generate an operator interface for display by the holographic viewing terminal. This unit also processes information from sensors which may be, for example, depth cameras, infrared sensors or transmitters to determine the correspondences between spatial reference frames and to enable the projection of holograms onto the anatomical reference areas of the patient to best guide the operator, and in particular the surgeon (for example: projection of information, measurements, 3D objects modeled during planning, etc.).

[0164] The interface allows the operator to carry out surgical planning in a manner similar to that described above, in particular via the following operations in holographic visualization for steps P2, P3, P4, P5. The holographic visualization comprises the visualization of 3D scenes generated by the module 202 and received through the stream reception interface 3042 and the stream generation interface 2004, or by importing a 3D scene generated by the module 202 and received through the 3D scene import interface in the form of a stream 3046 and the export interface 2002. The holographic visualization further comprises a manipulation of 3D objects of the 3D scene with the hands or dedicated devices, relying on the data processing unit 3048 and the interfaces 210 provided by the infrastructure 200.

[0165] Then, more specifically for each surgical planning step, step P2 comprises the characterization of the pathology affecting the patient by successive and possibly iterative execution of the following two sub-modules: segmentation of the tissues by calling the sub-module 202102 by the interfaces 210 provided by the infrastructure 200, identification and measurement by calling the sub-module 202104 by the interfaces 210 provided by the infrastructure 200.

[0166] Step P3 includes the modeling of surgical gestures by “i” calls (1 < i < n) of the 3D simulation sub-module of a surgical sequence S(i) 202106 by the interfaces 210 provided by the infrastructure 200. As for step P2, several successive calls can thus be made to simulate the number of surgical sequences required by the planning of the pathology of interest. Each call generates a 3D scene different from the previous one (n scenes).

[0167] Step P4 (if applicable) comprises a modeling of a standard medical device or one specific to the patient's anatomy by "j" calls (1 < j < m < n) of the sub-module for digital generation of a standard medical device or one specific to the patient's anatomy DM(j) 202108 by the interfaces 210 provided by the infrastructure 200. As for steps P2 and P3, several successive calls can thus be made to simulate the number of medical devices required per modeled medical treatment. Step P5 comprises the review and validation of the planning by viewing the different steps P2 to P4 (or P3). According to a possibility of the present disclosure, step P4 is optional and the planning of the surgical procedure can be sufficient in order to then carry out the surgical guidance.

[0168] The module 304 is also configured to provide the surgeon with a real-time visualization tool, possibly manual or in an automated mode, embedded on a preferably individual device as described above, compatible with use in an operating room during a surgical intervention, for the purposes of holographic guidance to accompany the operating practitioner during the performance of the operation planned in advance and validated in step P5. This surgical guidance tool takes the form of an operator interface which allows the following steps to be carried out.

[0169] Step G1 includes an initiation of the surgical guidance process by selecting during this first step a previously planned operation, possibly presented as a file relating to said operation, by referring explicitly or anonymously to the patient concerned by the operation, and / or to the planned date of the surgery. The module 304 then connects to the 3D rendering engine 2026 via the stream reception interface 3042 to display the objects from steps P1 to P5, possibly in a sequential mode, or optionally by importing the data from the export interface 2002 during this step (step A6 of [Fig.2]).

[0170] Step G2 includes the visualization of the surgical planning carried out beforehand and validated in step P5, typically in the form of holograms including in particular the 2D or 3D representation of the medical imaging examination, possibly in a combined presentation of the two types of formats, making it possible to navigate simultaneously in these multiple representations according to the operator's inputs, for example according to the multiplanar reconstruction model used in medical imaging for the analysis of files (for example CT scans). The hologram also includes the 3D scenes resulting from the pathology characterization step. as output data of the tissue segmentation sub-modules 202102 and identification and measurement sub-modules 202104. The hologram also includes the 3D scenes from the surgical gesture modeling step as output data of the 3D simulation sub-module 202106 of a surgical sequence S(i). This possibly includes the 3D view of the anatomical element of interest before and after surgical treatment if applicable (for example: bone before and after reconstruction in the treatment of a fracture in orthopedic surgery) as well as the information and notes entered or calculated during planning.

[0171] The holograms also include, if applicable, the 3D scenes resulting from the modeling step of standard medical devices or those specific to the patient's anatomy as output data from the sub-module 202108 for digital generation of standard medical devices or those specific to the patient's anatomy DM(j). In the context of fracture repair in trauma surgery, this corresponds, for example, to screw-type bone anchoring systems, temporary fixation devices of the surgical pin type (e.g.: Kirschner pin), a bone plate or even surgical guides.

[0172] The holograms further comprise the manipulation of 3D objects with the hands or using dedicated devices, relying on the interfaces 210 provided by the infrastructure 200, and performing measurement operations, for example of distance and angles on the holograms or on the patient.

[0173] Step G3 includes a functionality for automatic positioning on the patient during the intervention of the elements resulting from the surgical planning and represented in 3D in holographic form. In the context of fracture repair in trauma surgery, this corresponds for example to the automatic positioning of standard medical devices or those specific to the patient's anatomy: screw-type bone anchoring systems, temporary fixation devices of the surgical pin type (e.g.: Kirschner pin), osteosynthesis plate or even surgical guides for cutting, orientation or drilling for example. This so-called "recalibrated" positioning of holograms on the patient can be done for example by algorithms of the artificial intelligence module 208.These algorithms compare in real time the patient's anatomical data from the original medical imaging examinations used for surgical planning, with information from sensors which could be, for example, video systems from the augmented reality visualization device during the surgical intervention. These algorithms could, for example, operate on the principle of computer vision to determine the correspondences between spatial references and allow the projection of holograms onto the patient's anatomical reference areas to best guide the surgeon (for example: . projection of information, measurements, 3D objects modeled during planning, etc.).

[0174] Step G4 includes an entry of information from the surgical report, which may in particular contain: captures in the form of images or video sequences taken during the surgical operation by the augmented reality visualization system. These captures are obtained either by triggering by the operator (for example: by voice or by gestures), or automated by prior configuration carried out by the operator during surgical planning. These captures are transmitted to the medical data storage device 204 by the interface 210, text dictated or entered using a virtual keyboard transmitted to the medical data storage device 204 by the interface 210.

[0175] The installation 500 is for manufacturing medical devices of the implant or ancillary instrumentation type, standard or specific to the anatomy of the patient, or anatomical elements of said patient. In the embodiment previously described, the system 100 comprises the installation 500. According to a variant of the present disclosure, the system 100 does not comprise the installation 500. The installation 500 allows the use of files in digital format coming from the export module 212 for the on-demand manufacturing of medical devices of the implant or ancillary instrumentation type, the anatomy of the patient, or anatomical elements of said patient. The installation 500 can be a subtractive manufacturing process (for example machining) or additive (for example 3D printing) allowing the shaping of standard medical devices or specific to the anatomy of the patient.The materials used, compatible with health use, can be metallic, ceramic, plastic, organic or composite.

[0176] The installation 500 also integrates all the post-treatment steps of standard medical devices or those specific to the patient's anatomy to enable their future use in surgery. These steps are typically the following (but are not limited to) one or more heat treatment steps, one or more surface treatment steps, one or more medical grade cleaning steps, one or more packaging steps in controlled or uncontrolled atmosphere environments, a sterilization step of the finished products.

[0177] The installation also makes it possible, where appropriate, to manufacture anatomical elements such as organs or portions of organs, or even bone elements (fragments, individualized bones, etc.).

[0178] Use cases of an architecture according to the present disclosure

[0179] The architecture according to the present disclosure is implemented by a surgeon during the three stages of planning, manufacturing and guidance. During the manufacturing stage, one or more anatomical elements, one or more implantable medical devices and / or one or more ancillary instrumentation medical devices may be manufactured. The architecture according to the present disclosure may also be implemented in a collaborative form during one or more of the various steps mentioned above.

[0180] According to a first possibility, the architecture according to the present disclosure can be implemented for surgery with remote preoperative assistance for which the surgeon requests one or more third party(ies) for assistance in preparing the surgery, during the planning stage, remotely, for example one of his colleagues of the same specialty, a colleague of another specialty, for example a radiologist. The surgeon can also request the opinion of other technical experts (support in the use of new surgical equipment for example). All of the third party consultations can also be combined to help the surgeon operating the solution to take the best therapeutic direction.In some health systems such as the French system, the act of requesting third-party expertise in medical imaging to confirm or deny a diagnosis is called "teleexpertise" (remole assistance in English). This action is common in more traditional modes between radiologists or between surgeons and radiologists.

[0181] According to a second possibility, the architecture according to the present disclosure can be implemented for surgery with remote intraoperative assistance for which the surgeon requests one or more third party(ies) for remote surgical assistance during the operation, for example one of his colleagues of the same specialty, a colleague of another specialty, for example a radiologist. The surgeon can also request the opinion of other technical experts (support in the use of surgical equipment for example).

[0182] According to a third possibility, the preoperative assistance described according to the first possibility and the intraoperative assistance described according to the second possibility can be combined.

[0183] According to a fourth possibility, the architecture according to the present disclosure is implemented by the surgeon only during a planning step, with a manufacturing step reduced to the manufacturing of an anatomical element, and without an intraoperative guidance step. This use case allows the surgeon to explain, with the visual aid formed by the anatomical element, the consequences that he envisages for the patient. The consequence may be not to operate.

[0184] Each of the preceding possibilities, when it provides for the implementation of a manufacturing step, generates a new use case, in which the manufacturing step is removed from said possibility.

[0185] Of course, the present disclosure is not limited to the examples just described and many adjustments can be made to these examples without departing from the scope of the present disclosure. In addition, the various features, forms, variants and embodiments of the present disclosure can be associated with each other in various combinations to the extent that they are not incompatible or exclusive of each other.

[0186] [Fig. 8] is another example of a flowchart describing the specific steps for implementing surgical planning (P2 to P4). The control begins when a medical imaging file is downloaded at step 804, for example a CT file such as a scan. The control determines, at step 808, whether the scan is acceptable. If not, the control proceeds to 812 to request a new scan. The control then returns to 804. Otherwise, if the scan is acceptable, the control continues to step 816 to build a three-dimensional model based on the downloaded scan. The control proceeds to step 820 to visualize the three-dimensional model. At step 824, the control performs tissue segmentation on the three-dimensional model. At step 828, the control determines whether the operator has accepted the tissue segmentation. As noted above, the operator / surgeon can adjust the segmentation if necessary.If the segmentation is not accepted, control continues to step 832 to receive the operator's correction and return to step 824.

[0187] If the segmentation is accepted, the control continues to step 836 to perform a surgical simulation on the segmented three-dimensional model. The control continues to step 840 to determine if the operator has accepted the simulation. If not, the control continues to step 844 to allow for operator correction and return to step 836. Otherwise, the control continues to step 848 to model the implants using the surgical simulation. The control proceeds to step 852 to determine if the operator has accepted the modeled implants. If not, the control proceeds to step 856 to allow for operator correction and return to step 848. Otherwise, the control proceeds to step 860 to perform validation of the modeled implants.

[0188] At step 864, the control determines whether the operator has accepted the validation of the implants. If not, the control continues to 868 to receive the operator's correction and return to step 860. Otherwise, the control proceeds to step 872 to generate and transmit digital files for the manufacture of the implant. Then, the control ends.

[0189] [Fig.9] is another example of a flowchart describing the specific steps of implementing surgical guidance (G1 to G4). The control begins guidance, for example, by the operator selecting a start button on an operator interface. At step 904, the control reviews the scan that has been used for guidance. In step 908, the control reviews the tissue segmentation. The control continues to step 912 to position the three-dimensional objects on the patient's body. The control proceeds to step 916 to determine whether the operator has accepted the registration of the three-dimensional objects on the patient's body. If not, the control continues to step 920 to receive a correction from the operator and return to step 912. Otherwise, the control proceeds to step 924 to perform surgical guidance. The control continues to step 928 to perform implant positioning guidance.

[0190] [Fig. 10] is a high-level representation of an implementation of the system 100. As described throughout this communication, the system 100 is implemented via a distributed communications network or data processing infrastructure 200 comprising a processor and memory. The data processing infrastructure 200 receives medical imaging files from a scanning apparatus 1000, such as a scanner, and generates surgical guides and digital implant files. The surgical guidance may be transmitted to multiple operators or devices, such as the computer terminal module 302 and the augmented reality module 304 (such as a streaming headset).In addition, digital implant files, which provide instructions for constructing the implants (developed using the surgical guidance function applied to a particular patient) can be transmitted to the manufacturing facility's module 500, which can construct the implant on-site.

[0191] [Fig. 11] is a graphical representation of an operator view of a guided surgical procedure using the system 100. In particular, the augmented reality module 304 is shown in a mixed reality view, illustrating a surgical guide (which could also be an implant) on the particular patient to help guide the operator (shown as a hand). Conclusions

[0192] The foregoing description is purely illustrative in nature and is not intended in any way to limit the disclosure, its application, or its uses. The general teachings of the disclosure may be implemented in a variety of forms. Therefore, although this disclosure includes particular examples, its actual scope should not be so limited since other modifications will become apparent after a study of the following drawings, specification, and claims. It should be understood that one or more steps of a method may be performed in a different order (or simultaneously) without altering the principles of the present disclosure. Furthermore, although each of the embodiments is described above as having certain features, one or more of those features described with respect to any embodiment of the disclosure may be implemented in and / or combined with features of any other embodiment, even if such combination is not explicitly described. In other words, the described embodiments are not mutually exclusive and permutations of one or more embodiments with each other remain within the scope of the disclosure.

[0193] Spatial and functional relationships between elements (e.g., between modules, circuit elements, semiconductor layers, etc.) are described using various terms, including "connected," "engaged," "coupled," "adjacent," "beside," "above," "below," and "arranged." Unless explicitly described as "direct," when a relationship between a first and second element is described in the above disclosure, that relationship may be a direct relationship where no other intervening elements are present between the first and second elements, but may also be an indirect relationship where one or more intervening elements are present (either spatially or functionally) between the first and second elements.

[0194] In this disclosure, which includes the definitions below, the term "module" or the term "controller" may be replaced by the term "circuit." The term "module" may refer to, be part of, or include: an application-specific integrated circuit (ASIC); a digital, analog, or mixed analog / digital discrete circuit; a digital, analog, or mixed analog / digital integrated circuit; a combinational logic circuit; a field-programmable gate array (FPGA); a processor circuit (shared, dedicated, or group) that executes code; a memory circuit (shared, dedicated, or group) that stores the code executed by the processor circuit; other suitable hardware components that provide the described functionality; or a combination of some or all of the above, such as in a system-on-chip.

[0195] The module may include one or more interface circuits. In some examples, the one or more interface circuits may implement wired or wireless interfaces that connect to a local area network (LAN) or a wireless personal area network (WPAN). Examples of LANs are the Institute of Electrical and Electronics Engineers (IEEE) 802.11-2016 standard (also known as the WIFI wireless network standard) and the IEEE 802.3-2015 standard (also known as the ETHERNET wired network standard). Examples of WP ANs are the IEEE 802.15.4 standard (including the ZigBee Alliance's ZIGBEE standard) and the Bluetooth Special Interest Group (SIG) BLUETOOTH wireless networking standard (including versions 3.0, 4.0, 4.1, 4.2, 5.0, and 5.1 of the Bluetooth SIG Base Specification).

[0196] The module may communicate with other modules using the interface circuit(s). Although the module may be described in this communication as logically communicating directly with other modules, in various implementations, the module may actually communicate via a communications system. The communications system includes physical and / or virtual networking equipment such as hubs, switches, routers, and gateways. In some implementations, the communications system connects to or traverses a wide area network (WAN) such as the Internet. For example, the communications system may include multiple local area networks connected to each other over the Internet or point-to-point leased lines using technologies such as multiprotocol label switching (MPLS) and virtual private networks (VPNs).

[0197] In various implementations, the functionality of the module may be distributed among multiple modules that are connected via the communication system. For example, multiple modules may implement the same functionality distributed by a load balancing system. In another example, the functionality of the module may be distributed between a server module (also referred to as a remote or "cloud" module) and a client (or operator) module. For example, the client module may include a native or web application running on a client device and in network communication with the server module.

Claims

Claims

1. A method of surgical assistance, comprising the following steps: a. obtaining raw medical imaging data corresponding to a patient from a remote computer architecture; b. reconstructing a digital model from the obtained raw medical imaging data, wherein the digital model is two-dimensional or three-dimensional; c. generating a plurality of three-dimensional scenes by applying a sequence of surgical acts to the digital model; d. simulating a virtual execution of the sequence of surgical acts on the digital model of the patient using the plurality of generated three-dimensional scenes; and e.displaying, on a display, at least one of the following: the reconstructed digital model, the pathology(s), the plurality of generated three-dimensional scenes and the simulated virtual performance, comprising a projection of the plurality of three-dimensional scenes onto the patient to guide an operator, wherein the projection comprises a holographic projection.

2. A method according to the preceding claim, wherein the holographic projection can be manipulated by the operator or can be positioned on the patient.

3. A method according to any preceding claim, wherein the plurality of generated three-dimensional scenes or simulation of a virtual execution is displayed in a collaborative mode to remotely assist the operator in performing a corresponding procedure requiring multiple assessments or to train the operator in an observational mode.

4. Method according to any one of the preceding claims, comprising an implementation of artificial intelligence and / or a simulation of physical systems by digital mathematical modeling to model the simulated virtual performance.

5. Method according to any one of the preceding claims, further comprising a step of modeling (P4) digital models of anatomical elements, and / or implantable medical devices and / or ancillary instrumentation for assisting the surgical procedure, for carrying out said sequence of surgical procedures, in the form of 3D objects added to or generated within at least one additional modeled 3D scene, from the digital model and / or extracted data of interest.

6. Method according to the preceding claim, wherein the anatomical elements, and / or the implantable medical devices and / or the auxiliary instruments are based on the anatomy of the patient.

7. A method according to one of the two preceding claims, comprising a projection of the digital model onto the patient to guide an operator, wherein the projection comprises a holographic projection.

8. Method according to any one of the preceding claims, further comprising a step of generating digital models of anatomical elements, and / or implantable and / or ancillary medical devices, specific or not to the anatomy of the patient.

9. Method according to the preceding claim, further comprising manufacturing by a manufacturing installation (500), additive or subtractive, of the digital files generated from said generated digital models.

10. Cloud-type computing architecture (200) for surgical support provided with a module for reconstructing a two-dimensional and / or three-dimensional digital model from raw medical volume imaging data of a patient recorded in the architecture, a graphical display interaction interface (300) with an operator, comprising a module for planning a surgical intervention configured to implement the following steps: a. display on the display of a scene from said digital model, determination (P2) of one or more pathologies of said patient by means of an identification and measurement module (202104) applied to said reconstructed digital model and / or to data of interest extracted from said raw data, b. modeling a plurality of additional three-dimensional scenes by applying a sequence of surgical gestures (P3) to said digital model and / or to said data of interest, c. simulation and computer modeling of the virtual realization (P5) of said sequences of surgical gestures on the patient, d. projecting the plurality of three-dimensional scenes onto the patient to guide an operator, wherein the projection comprises a holographic projection.

Images