Method for evaluating an automatically generated medical report

Abstract

The invention relates to a method for evaluating a medical procedure report, comprising the following steps: (E2) providing a set of information (D, CRMi) relating to the medical procedure; (E3) generating a report (CRSj) by means of a machine learning algorithm (MLj) trained to generate a report using a training set (DSj) comprising information sets (CEk) associated with predetermined reports (CSk); (E4) estimating a first proximity value (V1) between said provided information set and at least one information set from the training set; (E5) estimating a second proximity value (V2,j) between the generated report and at least one predetermined report from the training set; (E6) estimating a confidence index (Ij) from the first and second proximity values;(E8) transmission of the generated report and the confidence index to a computer terminal (2) Figure to be published with the abbreviation: Fig. 1;

Description

Title of the invention: Method for evaluating an automatically generated medical report. Technical field

[0001] The invention relates to the technical field of health, and more specifically to the production of medical reports using machine learning algorithms. State of the art

[0002] Typically, during a patient's care pathway, one or more medical examinations are prescribed, such as a consultation with a specialist, a medical examination involving imaging, or a laboratory test. The practitioner conducting the examination generally writes a report or summary of the medical examination, possibly including observations or diagnostic conclusions. This report is then transmitted, either electronically via the patient's medical record or directly from the patient, to the next practitioner in the care pathway and / or to the physician coordinating the care pathway.

[0003] While this method effectively allows the various medical professionals to access the results of previous examinations and to effectively monitor the patient throughout the care pathway, it has the disadvantage of excluding the patient from the therapeutic strategy of the care pathway.

[0004] Indeed, a medical report generally contains complex medical terms that the patient may not be familiar with. Furthermore, the answers and conclusions provided by the practitioner in the report are difficult for the patient to interpret, for example because the practitioner uses clinical terms or acronyms, or because they refer to values ​​without providing an explanatory framework.

[0005] The patient may thus lose confidence in the proposed treatment strategy and reduce their adherence to and participation in the care pathway, which can lead to the failure of this pathway. Therefore, there is a need to simplify medical reports to adapt them to the patient's level of understanding.

[0006] In this context, machine learning algorithms, or artificial intelligence (AI), and in particular large language models, or LLMs, have shown significant potential in simplifying or translating complex texts. They can thus be trained using a corpus of texts representing different medical reports, labeled by Doctors, learned societies, and patient associations can use this technology to automatically generate a simplified medical report from a written report, which the practitioner can then distribute to the patient. This eliminates the need for the practitioner to write the report themselves, thus reducing their workload.

[0007] Despite the progress made in these algorithms, consistently producing satisfactory results remains difficult. One problem is the variability of the medical reports that can be provided to the algorithms, which may differ considerably from the texts in the training corpus. Furthermore, this training corpus may be incomplete or unbalanced, leading to a loss of information in the simplified report. Finally, LLMs are likely to produce hallucinations, which can compromise the reliability of the simplified report. This results in a need for a high level of trust among practitioners in the reliability and relevance of the simplified reports generated by the algorithms, in order to distribute them to patients. This need also clashes with the initial problem, as practitioners do not have the time to verify each AI-generated simplified report to meet this need for a high level of trust.

[0008] These needs exist more generally in a context of automatic generation of medical reports from information relating to a medical act, whether it is an expert medical report or data from the patient's file, and whether the automatically generated report is a simplified report or an expert report.

[0009] Thus, there is a need for a process to automatically generate, from a set of information relating to a medical procedure performed for a patient, a report and which allows the practitioner to know quickly whether he should check this report or whether, on the contrary, he can distribute it to the patient without verification.

[0010] The invention therefore falls within this context and seeks to resolve all the aforementioned drawbacks by meeting said need. Presentation of the invention

[0011] The invention thus relates to a method for evaluating a report of an act medical report carried out by a healthcare professional for a patient, the report being generated automatically, the process being implemented by a computer system and comprising the following steps: a. provision of a set of information relating to the medical procedure, to the computer system; b. generating, from said set of information provided, a report, using at least one machine learning algorithm trained to generate a report of a medical procedure from a set of information relating to that medical procedure, said at least one algorithm having been trained using a training set comprising a plurality of information sets, each relating to a given medical procedure, and a plurality of predetermined reports each associated with one of said information sets; c. estimation of a first proximity value between said provided information set and at least one information set from the training set of said machine learning algorithm; d. estimation of a second proximity value between the report generated by the machine learning algorithm from said provided information set and at least one predetermined report from the training set of said machine learning algorithm; e. estimation of a confidence index associated with the report generated by the machine learning algorithm from the first and second proximity values; f. transmission of at least the report generated by the machine learning algorithm and the confidence index to a computer terminal

[0012] The invention thus proposes that a set of information relating to a medical procedure performed by a healthcare professional on a patient be transmitted to a computer system, where it is automatically converted into a medical report. This set of information may be a medical report, referred to as an "expert" report, routinely written by the healthcare professional during or after the medical procedure, or a set of raw, processed, or enriched information relating to the medical procedure. In the invention, the conversion is performed by one or more machine learning algorithms implemented by the computer system and previously trained to generate a medical procedure report from a set of information relating to that procedure.This could be a simplified version of the expert report, or a simplified or expert report generated from data from the medical procedure and / or data from the patient's medical record.

[0013] Following this conversion, the computer system can then estimate two so-called proximity values, one of which indicates the proximity of the incident information set to the information sets of the training corpus, while the second indicates the proximity between the report inferred by the algorithm(s) and the reports of the training corpus labeling the information sets of the training corpus. It is thus possible to arrive at the calculation of a confidence index by a combination of these proximity values, indicating to the practitioner whether the incident information set is close to the information sets in the training set, and whether the report generated by the algorithm(s) is also close to one or more versions labeling these information sets.

[0014] The practitioner can thus, according to a threshold that they have set and / or that an association, such as a learned society or other professional organization, has set, determine whether the generated text warrants revision or whether they can trust it and distribute it to the patient without review. The invention thus makes it possible to limit the practitioner's intervention time while increasing their level of confidence in the automatic report generation tool. Furthermore, the confidence rating can also be transmitted to the patient to indicate whether they can trust the produced text or, conversely, encourage them to maintain a critical perspective on it.

[0015] Definitions

[0016] In the present invention, the term "computer system" means any set of interconnected hardware and software components designed to process, store, and manipulate digital data in order to implement the process according to the invention. A computer system may include one or more processing devices, storage means, input / output interfaces, applications, programs or software, and communication means. The computer system may operate autonomously or as a network, and may be physically located on a single site or geographically distributed.

[0017] The computer system may include, but is not limited to, a computer terminal such as a desktop computer, a laptop computer, a server, a tablet computer, a smartphone, a cell phone and / or computer network equipment; a local computer network comprising several computer terminals and one or more servers interconnected to the terminals; or a "cloud" type computer infrastructure comprising computer servers accessible by means of wireless communication and providing services to a remote computer terminal, in particular of the SaaS (Software As A Service) type.

[0018] Where appropriate, one or more computer applications, computer programs, or software, enabling the implementation of the method according to the invention, may be installed directly on a computer terminal that can be operated by the practitioner, and / or installed on a server of a local network so that they can be accessed by a computer terminal on that local network, or hosted on a computer server from which they are accessible through of software installed on a computer terminal, such as an internet browser or an application programming interface.

[0019] Preferably, in the case of a local area network or "cloud" infrastructure computer system, each computer terminal of the computer system is equipped with a communication unit capable of exchanging data with a communication unit of a processing server capable of implementing all or part of the process according to the invention, in particular according to one or more wireless communication protocols.

[0020] If desired, the computer system may be connected to or be part of one or more hospital computer systems to receive data relating to the medical procedure via other elements of that system, said computer system being, for example, a radiology information system (RIS), a laboratory management system (LMS), a results server, and / or a picture archiving and transmission system (PACS). Where applicable, the method according to the invention may be implemented by means of software programs or applications, some of which may be installed on a computer terminal that can be operated by the practitioner, particularly for the purpose of collecting or generating said set of information relating to the medical procedure.This set will thus be transmitted, once generated or collected on the computer terminal, to a processing server, on which is installed another part of the programs or software applications enabling the implementation of the process according to the invention.

[0021] Alternatively, the computer system may include one or more remote servers connected to a computer terminal that can be operated by the practitioner, the services of said processing server(s) then being accessible by this computer terminal using cloud computing methods. Where applicable, all the programs, software, or applications necessary for implementing the method according to the invention may be installed on the processing server(s), either centrally or in a distributed manner, all or part of these programs, software, or applications being accessible from the computer terminal via wireless communication protocols, such as, for example, through an internet browser and HTTPS and IP protocols, or through an application programming interface or API. Said set of information relating to the medical procedure will thus be collected or generated directly on the remote server.

[0022] In this case, it may be provided that the process according to the invention includes a prior step of identification and / or authentication of the person and / or application in charge of the collection or generation of said set of information, in particular through a strong authentication process implemented by the processing server or another server dedicated to this process.

[0023] In the present invention, the term "medical procedure information set" means any collection of data, structured or unstructured, relating to a medical procedure performed on or for a patient. This information may be generated manually and / or automatically following various types of medical interventions, by one or more healthcare professionals and / or by one or more computer programs, all at once or sequentially, and is not limited to a particular medical field.

[0024] This set may include, but is not limited to, an expert medical report written by a healthcare professional, raw data from medical examinations, laboratory test results, medical images and their interpretations, clinical notes, patient observations, physiological measurements, relevant medical history, or any combination of these elements. For example, the information set may include a detailed account of a medical consultation, or a data set including the results of a laboratory test, necessary for preparing a medical report.

[0025] In the present invention, "expert report" means a written or electronic document that records essential information relating to a medical procedure performed by a healthcare professional for a patient. This document can be generated following various types of medical interventions and is not limited to a particular medical field.

[0026] An expert report may include, but is not limited to, the following: patient identification information; date and nature of the medical procedure; reason for consultation or intervention; relevant medical history; symptoms reported by the patient; clinical observations of the practitioner; results of physical examinations; details of procedures performed; results of additional examinations, including imaging or laboratory tests; established diagnosis or diagnostic hypotheses; treatment administered or prescribed; prognosis; follow-up plan or future recommendations; any samples taken; or any other type of information provided to the patient.

[0027] The expert report may be generated by various methods, including: manual drafting by a healthcare professional, the handwritten document possibly being digitized and provided to an optical character recognition (OCR) solution; entry by a healthcare professional via word processing software on a computer terminal or via a dedicated application, local or accessible via a SaaS service; dictation by a healthcare professional and transcribed using specific software such as speech recognition; generation assisted by one or more machine learning algorithms from a prompt provided by a healthcare professional; automatic generation by one or more algorithms machine learning from recordings and / or results of the medical procedure.

[0028] The expert report may be transmitted to the computer server, for example, via a wireless communication protocol, or be directly generated using a software application on a processing server accessible from a computer terminal operated by the healthcare professional or installed directly on that computer terminal. The initial report is thus available and accessible from other computer terminals authorized to connect to the processing server, for example, to allow another healthcare professional, such as another practitioner, the attending physician, or other stakeholders in the care pathway, to review the report.

[0029] The medical procedure may be an act performed during a general or specialist medical consultation or teleconsultation, an act performed for investigative or diagnostic purposes, and may include taking a medical history, an inspection, palpation, auscultation, a sample collection, a psychotherapy session, an examination involving medical imaging, or the analysis of a sample and / or a medical image. The medical procedure may also be an act performed for therapeutic purposes, and may include an oral interview aimed at providing therapeutic advice, the administration of medication, or a surgical, radiological, or physical intervention.All or part of the medical procedure may be performed by general practitioners or doctors of various medical specialties, including a surgeon, a psychologist, a cardiologist, an oncologist, a radiologist; by midwives, dentists, nurses, medical secretaries, pharmacists, physiotherapists, medical biologists, laboratory technicians, nursing assistants, ambulance drivers.

[0030] In the present invention, a "simplified report" means a version of the expert report that has been modified to be more easily understood by non-experts, particularly patients. The simplification of the expert report may include, in particular: replacing complex medical or technical terms with more common synonyms; adding detailed explanations of medical terms and procedures; removing or summarizing non-essential or overly detailed information; and reorganizing the report's structure to highlight information considered essential for the patient, such as test results, diagnoses, and treatment recommendations.

[0031] It may be envisaged that the simplification of the expert report will be configurable by the healthcare professional or by the patient, in particular to adapt the simplified report to the patient's level of understanding. If applicable, the delivery step The expert report may include a sub-step for providing conversion parameters, notably through an input interface, which will be provided to the machine learning algorithm to generate the second, simplified report. For example, a query or prompt, perhaps pre-written, could be provided to the machine learning algorithm. This query would request, based on the presence of a given section in the expert report, the generation of a simplified section of the same type in the simplified report.

[0032] In the present invention, the term “machine learning algorithm,” also called an artificial intelligence (AI) model or a machine learning system, refers to a set of mathematical instructions and rules designed to analyze data, identify patterns, and make decisions or predictions with minimal human intervention. These algorithms are characterized by their ability to improve iteratively through a training process, during which they adjust their internal parameters based on input data and expected results in order to optimize their performance on a specific task.

[0033] In the context of the present invention, the report may be generated using one or more machine learning algorithms, implemented in parallel or sequentially, to which the said set of information relating to the medical procedure is provided as input. By way of non-limiting example, this or these machine learning algorithms may be chosen from: artificial neural networks, deep neural networks (DNNs), convolutional neural networks (CNNs), recurrent neural networks (RNNs), large language models (LLMs), support vector machines (SVMs), random forests, or any appropriate combination of these models.

[0034] Preferably, the report can be generated using one or more large language model machine learning algorithms, including in particular a predetermined generative transformer (GPT), which can be a semantic autoregression algorithm (GPT-3, GPT-4, Mistral Large or Small, LLaMa, LLaMa 2, Gemini, Claude), a transformer or only part of a transformer, such as an encoder (BERT).

[0035] The machine learning algorithm(s) may be trained using supervised training methods with the training set, and in particular using methods such as "few-shot learning", "fine tuning", and "transfer learning". Alternatively, other training methods may be used, or several training methods may be combined, including semi-supervised or even unsupervised training methods.

[0036] In an example of an embodiment of the invention, the machine learning algorithm(s) may have been previously trained to convert an expert report into a simplified report, from a training set comprising expert reports each associated with a simplified report.

[0037] Preferably, and particularly in the case of an LLM, the training set may include expert reports written by healthcare professionals, each of these expert reports then being labeled by an expert, such as a physician or a member of a patient association, with a simplified report. The machine learning algorithm(s) are thus trained to iteratively generate simplified reports from the expert reports in the training set and to evaluate an error metric between the generated simplified report and the simplified report labeling the expert report. At each iteration, the machine learning algorithm(s) adjust their internal parameters, such as weights and neuronal biases, to minimize this error metric over the course of the iterations.

[0038] The training may be implemented using all or part of the expert and / or simplified reports from the training set. These reports may be divided into several blocks or sections, such as a title, patient information, reasons for consultation, clinical context, medical history, clinical examination, diagnosis, treatment plan, clinical recommendations, and a conclusion. If necessary, the training may be implemented using a query or prompt, which may be pre-written, requesting, depending on the presence of a given section, the generation of a simplified section of the same type. All the sections thus generated are then combined to form the simplified report.

[0039] In the present invention, "a proximity value" means a quantitative measure representing the degree of similarity or resemblance between two sets of information.

[0040] In cases where the information sets include texts, this proximity value may, for example, be calculated using one or more text quality assessment metrics. These may include metrics for evaluating error, discrepancy, similarity, or correspondence between two texts, such as those used in LLM training methods. This proximity value may thus evaluate n-gram correspondence, precision, recall, or semantic similarity based on contextual representations, or any other measure of the distance between vector representations of the texts, particularly in a high-dimensional semantic space, such as Euclidean distance, cosine similarity, Levenshtein distance, or Manhattan distance.

[0041] For other types of information, said proximity value may be calculated from one or more statistical metrics, such as a Pearson correlation, a mean squared deviation, or any other metric capable of evaluating the error, deviation, similarity or correspondence between two data sequences or two time series.

[0042] Embodiments

[0043] In one embodiment of the invention, the first proximity value estimation step includes a substep for estimating a proximity index between the provided information set and each information set in a plurality of information sets in the training set, the first proximity value being calculated from said proximity indices. According to these characteristics, the first proximity value thus indicates the similarity between the information set provided as input to the machine learning algorithm and some, or even all, of the information sets in the training set of that algorithm, such that the confidence index reflects the proximity of the incident information set to the information sets used to train the machine learning algorithm(s).The healthcare professional thus has, through the confidence index, information enabling them to assess whether they should review the generated report due to a significant discrepancy with the training set, or whether they can instead distribute the generated report to the patient, or even to other healthcare professionals, or other medical information systems, such as a patient medical record management system or DMP.

[0044] Advantageously, the training set can be segmented into different batches of information sets, each labeled with a predetermined report, each batch being associated with a medical type, such as a medical specialty or a pathology. If necessary, a proximity index can be estimated for each information set in the training set associated with the medical type corresponding to the provided information set. This medical type can be pre-selected by the healthcare professional who wrote or collected the provided information set, or it can be automatically inferred from the provided information set. It is thus possible to specialize the training of the machine learning algorithm and the calculation of the confidence index to a given medical type.

[0045] Alternatively, a proximity index can be estimated for each set of information in the training set.

[0046] Advantageously, when said set of information provided is a first, so-called expert, report; and that the machine learning algorithm has been Trained to convert, from a training set containing expert reports each associated with a simplified report, an expert report into a simplified report, each proximity index is calculated according to at least one initial metric of semantic distance between two texts. This type of metric makes it possible to evaluate the similarity between two texts beyond simple lexical correspondences, particularly through metrics for evaluating machine translation or automatic summarization / synthesis.

[0047] Preferably, said semantic distance metric is chosen from one of the following translation or simplification evaluation metrics: BLUE, RED, METEOR, BERTScore, BLEURT, COMET, cosine similarity, Fl, SARI, BARTScore.

[0048] The BLEU metric (“Bilingual Evaluation Understudy”) is a metric for evaluating the quality of machine translation, measuring unigram, bigram, and even n-gram accuracy. The ROUGE metric (“Recall-Oriented Understudy for Gisting Evaluation”) is a metric for evaluating the quality of machine summarization, measuring both accuracy and recall of unigrams, bigrams, and even n-grams. The METEOR metric (“Metric for Evaluation of Translation with Explicit Ordering”) is a metric for evaluating the quality of machine translation, taking into account synonyms, stemming, and word order. The BERTScore metric measures the similarity of two texts by comparing their vector representations.The BLEURT (Bilingual Evaluation Understudy with Representations from Transformers) and COMET (Cross-lingual Optimized Metric for Evaluation of Translation) metrics are used to evaluate the quality of machine translation, based on a pre-trained model that takes into account meaning and semantic similarity. The cosine similarity metric evaluates the cosine between two vector representations of two texts. The SARI (System Output Against References and Against Inputs) metric evaluates the quality of a simplified text, based on the additions, deletions, and modifications introduced by the simplification process.The Fl metric is a measure of model accuracy that indicates the harmonic mean of accuracy and recall, thus taking into account both the proportion of correct positive predictions among all the model's positive predictions and the proportion of positive predictions that are actually positive. BARTScore is a text generation assessment measure that treats model evaluation as a text generation task, leveraging BART's pre-trained semantic embeddings (or "word embeddings") to return a score that measures the reliability, accuracy, recall, or F-score response of the main text generation model.

[0049] It may be possible to combine several semantic distance metrics and to employ other text comparison metrics, used in translation, simplification, or text summarization contexts, without departing from the scope of the present invention. It should be noted that combining several metrics can offer a more robust evaluation, for example, by identifying both synonyms and paraphrases.

[0050] In another example, where said information set provided is a data set from a clinical examination or laboratory analysis; and the machine learning algorithm has been trained to generate a medical report from a data set from a clinical examination or laboratory analysis, from a training set comprising data sets from a clinical examination or laboratory analysis each labeled with a medical report, each proximity index is calculated according to different metrics adapted to the types, categories and natures of the data in said data sets.

[0051] For example, in the case where the data sets contain qualitative or categorical data, such as a positive or negative result of a test, a classification or a grading on a given scale, each proximity index can be calculated from the Hamming distance between these qualitative or categorical data.

[0052] For example, in the case where the data sets contain quantitative data, such as a time sequence or numerical vectors, each proximity index can be calculated from a cosine similarity between these quantitative data.

[0053] For example, in the case where the data sets contain text-type data, such as a medical history, medical background, a description of symptoms, each proximity index can be calculated from a Levenshtein distance between these text-type data.

[0054] In one embodiment of the invention, said first proximity value is the maximum value of the proximity indices. This example allows for the rapid identification of the training set of information most similar to the incident information set. Alternatively, said first proximity value could be the average of the N highest values ​​of the proximity indices. In this example, the first proximity value reflects the proximity of the incident information set to the entire training set, these characteristics thus being of interest in sectors where the variability of information sets is significant.

[0055] In one embodiment of the invention, the step of estimating the first proximity value may be carried out prior to the step of generating the report from the provided information set using the machine learning algorithm(s). If so, if said first proximity value is a value indicating an exact match between the provided information set and one of the information sets in the training set, the steps of generating the report from the provided information set using the machine learning algorithm(s) and estimating the second proximity value are not carried out, and the method includes a step of selecting the report from the training set associated with said information set in the training set that corresponds exactly to the provided information set.This selected report is thus provided to the computer terminal with an indication of the exact match.

[0056] In this example, we thus avoid using the machine learning algorithm(s) in the case where the set of information provided already exists in the training set.

[0057] Alternatively, the steps for estimating the first and second proximity values ​​may be implemented after the report has been generated by the machine learning algorithm(s).

[0058] In one embodiment of the invention, said first proximity value corresponds to a given information set from the training set, and the second proximity value is an estimated proximity index between the report generated by the machine learning algorithm and a report from the training set associated with said given information set from the training set corresponding to the first proximity value. According to these characteristics, the second proximity value thus indicates the similarity between the automatically generated report and the training set report labeling the training set information set closest to the incident information set.Therefore, the confidence index combining these two values ​​reflects both the similarity of the incident data set to the data sets used to train the machine learning algorithm(s) and the relevance of the automatically generated report in relation to the training set. This further increases the healthcare professional's confidence and speed in judging whether to review the automatically generated report or whether, on the contrary, it can be shared with the patient, or even with other healthcare professionals or other medical information systems.

[0059] Advantageously, said proximity index is calculated according to at least a second metric of semantic distance between two texts.

[0060] Preferably, when the provided information set is a first, or expert, report, and the machine learning algorithm has been trained to convert, from a training set comprising expert reports each associated with a simplified report, an expert report into a simplified report, the second semantic distance metric is distinct from the first semantic distance metric. It can thus be assumed that the first proximity value reflects the similarity between the incidental expert report and the training set, while the second proximity value indicates the relevance of the transformation applied to this incidental expert report. If appropriate, the first distance metric could be a measure of semantic similarity, such as RED, while the second distance metric could be a measure evaluating the quality of a text simplification, such as SARI.

[0061] Alternatively, the second semantic distance metric may be the same metric as the first semantic distance metric

[0062] In one embodiment of the invention, the step of estimating a confidence index associated with the report generated by the machine learning algorithm includes a substep of estimating a penalty from one of the first and second proximity values, the confidence index corresponding to the other of said proximity values ​​penalized by said penalty.

[0063] The introduction of a penalty targeting one or the other of the proximity values ​​in the calculation of the confidence index makes it possible to introduce a tolerance or on the contrary a sanction in the discrepancies between the incident information set and the inferred report on the one hand and the training set on the other hand, depending on the sensitivity of the health professional with regard to machine learning algorithms.

[0064] Advantageously, the penalty can be calculated from the first proximity value, for example, the first proximity value raised to a given power such as 2 or more. Alternatively, or cumulatively, the penalty can be estimated by a combination of several first values ​​defined according to several metrics, such as RED-1 and BLUE-1.

[0065] Advantageously, the confidence index can be estimated by multiplying the penalty by the second proximity value.

[0066] Alternatively, the penalty may be applied to the second proximity value, in order to penalize errors in generating reports, or to the first and second proximity values.

[0067] In an alternative or cumulative embodiment of the invention, the confidence index may be determined from the first and second proximity values ​​and a weight determined from the machine learning algorithm used for generating the report and / or the training set. For example, the weight This can be determined from the type and / or number of hyperparameters of the LLM used to generate the report and / or the number of reports present in the training set. This weight allows the confidence index to be adjusted according to the reliability of the algorithm used and its training.

[0068] In one embodiment of the invention, the method includes, prior to the transmission step, a step for comparing the confidence index to a predetermined threshold value, the transmission step being conditional upon the result of said comparison. The results are thus filtered so as to transmit only the reports generated by the machine learning algorithm(s) that are likely to be disseminated by the healthcare professional.

[0069] For example, the report will only be transmitted if the confidence index is greater than said threshold value, for example set at 0.5.

[0070] In one embodiment of the invention, the generation step is a generation step, from said provided information set, of a plurality of reports, each being generated by means of a separate machine learning algorithm trained to generate a report of a medical act from a set of information relating to that medical act, each algorithm having been trained using a training set comprising a plurality of information sets, each relating to a given medical act, and a plurality of predetermined reports each associated with one of said information sets; and a second proximity value and a confidence index are estimated for each of said reports generated by the machine learning algorithms.

[0071] It may be provided that each of the machine learning algorithms is trained using the same training set or is trained using a dedicated training set. If so, each proximity value will be estimated from the dedicated training set.

[0072] Advantageously, it can be provided that only the report associated with the highest confidence index is transmitted during the transmission step. In this example, only the best result is submitted, in order to accelerate the healthcare professional's decision-making and increase their level of confidence.

[0073] Alternatively, it may be stipulated that the N reports associated with the highest confidence indices are transmitted during the transmission step, or even that all reports whose confidence index exceeds said threshold value are transmitted during the transmission step. In this alternative, the healthcare professional is free to choose which reports they wish to review and / or share with the patient, or even with other healthcare professionals, or other medical information systems.

[0074] Alternatively, each machine learning algorithm may be trained to convert an expert report into a simplified report according to a given level of understanding, which may be pre-established, selected, or indicated by the healthcare professional or patient via a selection interface or prompt. In this variant, the process generates several reports of the same medical procedure at different levels of understanding, thus enabling the healthcare professional to select the simplified report they deem most relevant based on the patient's understanding and the care pathway they are following.

[0075] In one embodiment of the invention, the method comprises an evaluation step, on said computer terminal, of the report generated by the machine learning algorithm, and a transmission step, to the computer system, of said evaluation. This evaluation step makes it possible to implement training of the machine learning algorithm(s) by reinforcement using human feedback (HFF). Reports rejected by the healthcare professional or the patient can thus be penalized in a subsequent retraining step, or even all or part of reports edited or corrected by the healthcare professional can be reintroduced into the training set.

[0076] The invention also relates to a computer system arranged to implement the process according to the invention.

[0077] The invention also relates to a computer program comprising program code which is designed to implement the method according to the invention.

[0078] The invention also relates to a data carrier on which the computer program according to the invention is recorded. Brief description of the figures

[0079] Other advantages and features of the present invention are now described by means of purely illustrative and in no way limiting examples of the scope of the invention, and from the accompanying figures, in which:

[0080] [Fig.l] represents, schematically and partially, a computer system for the implementation of a process according to an example of an embodiment of the invention;

[0081] [Fig.2] represents, schematically and partially, a devaluation process of a report of a medical act automatically generated according to an example of an embodiment of the invention, implemented using the computer system of [Fig.1];

[0082] [Fig.3] represents, schematically and partially, an expert report incident provided in a step of the process of [Fig.1];

[0083] [Fig. 4] schematically and partially represents part of an example training set for a machine learning algorithm used in the process of [Fig. 1]; and

[0084] [Fig.5] represents, schematically and partially, two simplified reports inferred in a step of the process of [Fig.1].

[0085] In the following description, identical elements, by structure or by function, appearing on different figures retain, unless otherwise specified, the same references.

[0086] The processes that will be described can also be implemented by software programs executable by a computer system. Furthermore, their implementation can be carried out interchangeably by distributed processing and / or parallel processing, in particular for processing several data points in parallel.

[0087] The figures described in this document are intended to provide a general understanding of the invention in various embodiments. These figures are not intended to serve as a complete description of all the elements and features of the devices, processors, and systems necessary for the invention. Many other embodiments of the invention, or combinations thereof, may be apparent to those skilled in the art upon reading this description, by combining the disclosed embodiments. Other embodiments may be derived from the description, so that structural and logical substitutions and changes may be made without departing from the scope of the present invention.

[0088] Furthermore, the description and figures should be considered illustrative rather than restrictive, and the appended claims are intended to cover all modifications, improvements, and other embodiments of the invention. Therefore, the scope of the following claims should be determined by the broadest possible interpretation of the claims and their equivalents and should not be restricted or limited by the preceding description. Description of the implementation methods

[0089] A computer system 1 for the implementation of a method for generating reports of a medical procedure is represented in [Fig.1] according to an example of an embodiment of the invention.

[0090] The computer system 1 comprises a first computer terminal 2, a processing server 3 and a second computer terminal 4.

[0091] In the example described, the first computer terminal 2 is a desktop computer, part of a hospital information system 5, for example of a hospital, clinic, radiology unit, surgical unit, consulting room or analysis laboratory.

[0092] The processing server 3 is remote from the hospital information system. The first computer terminal 2, or the hospital information system 5, and the processing server 3 are each equipped with a wireless communication unit, so that the services offered by the processing server 3 are accessible by the first computer terminal 2, in particular by a wireless communication network 6 implementing HTTPS and IP type communication protocols.

[0093] As will be described later, a software application 31 is installed on the processing server 3 and can in particular be executed on the first computer terminal 2 through an internet browser, in the form of a SaaS service.

[0094] The second computer terminal 4 is, in the example described, a patient's smartphone that can receive, through a software application 41 installed on this smartphone, data transmitted by the processing server 3.

[0095] Alternatively, the first computer terminal 2, the hospital information system 5, and the processing server 3 may be hosted in the same infrastructure, these different elements then being able to be connected to each other by a wired local communication network and / or a wireless local communication network. Alternatively, the software application 31 may be installed and executable locally on the first computer terminal 2.

[0096] In connection with [Fig.2], we will now describe a method for evaluating a report of a medical procedure carried out by a healthcare professional for a patient, and implemented using computer system 1.

[0097] In unrepresented steps, the patient follows a pathway within the hospital or laboratory where the medical procedure is performed. This pathway may include a reception stage, one or more examination stages involving medical imaging, sampling, analysis, surgery, care, or interviews with medical staff, and a discharge stage.A dataset D can thus be collected along the route by different components of the information system 5, these components being medical staff, including reception staff, an ambulance driver, a doctor or a nurse, medical equipment or a related information system such as a radiology information system (RIS), a laboratory management system (LMS), a protocol archiving and transmission system (PACS), or even through external information systems, such as the IT services of public bodies.

[0098] This dataset D may thus include information from a computerized patient record, clinical data, medical images acquired according to one or more medical imaging modalities.

[0099] During one of the stages of the patient's journey, a component of the information system 5 initiates the implementation of the process according to the invention. In the example described, this is the practitioner who performed the medical procedure for the patient. Alternatively, the process may be initiated by a non-human component of the information system 5, in particular by a computer program, or the process may be implemented by several components of the information system 5, in particular by different members of the medical staff and at different times in the patient's journey, both before and after the medical procedure.

[0100] In a first step 1, the practitioner connects via the first computer terminal 2 to the software application 31 of the remote server 3, through a strong authentication process implemented by another server dedicated to this process (not shown).

[0101] In this step El according to the example described, the practitioner usually writes a medical report, called "expert" relating to the medical act performed for the patient, through an input interface of the software application 31.

[0102] Figure 3 shows a simplified example of a CRMi expert medical report written during a human papillomavirus (HPV) screening performed on a patient. This CRMi expert report contains a title indicating the reason for the consultation, a section relating to the diagnosis of the consultation, and a section indicating a cytopathological commentary on the screening results as well as an indication of the cytopathological classification used.

[0103] The invention is not limited to this type of medical report or this type of medical procedure and may include, but is not limited to, other elements relating to the patient, their medical history and symptoms, the medical procedure and / or consultation, clinical observations, results of additional examinations, diagnosis and prognosis, treatment and / or follow-up plan; or any other type of information provided to the patient. It may also contain other documents, in the form of reports, examination results, images, or any document likely to originate from a medical procedure, examination, intervention, or consultation. Furthermore, the organization and structure of the report may differ from those shown in [Fig. 3].

[0104] Furthermore, the CRM expert report; may be generated by other healthcare professionals and in particular all or part of the CRM expert report; may be entered by other components, human, software or hardware, of the information system 5, during the patient's journey, before and / or during and / or after the medical procedure, in particular from the data set D.

[0105] It may also be envisaged that the CRM expert report; be generated by other methods, and in particular by: uploading into application 31 a scan of a handwritten report; the transcription of a report dictated orally and recognized by a speech recognition algorithm; a generation assisted by one or more machine learning algorithms from a prompt provided by a health professional; an automatic generation by one or more machine learning algorithms from a recording and / or results of the medical act.

[0106] In a second step E2, the CRM expert report; is transmitted to a data processing module of server 3; to be converted there in a third step E3 into a plurality of simplified CRSj reports.

[0107] In other variants not shown, it may be provided that all or part of the data D is transmitted directly to the data processing module of server 3, either by the practitioner via terminal 2, or directly by the information system 5. In this case, the objective of the data processing module of server 3 is to generate, from this data D, a report, which may be a simplified report or an expert report.

[0108] In the example described, the conversion is implemented by this data processing module using a plurality of machine learning algorithms MLj executed in parallel in sub-steps E3j and to which the CRM expert report is provided as input. Alternatively, each machine learning algorithm may be executed by one or more other servers, remote from server 3, to which server 3 connects. These servers may, for example, be organized into clusters, particularly within a cloud infrastructure.

[0109] In the example described, each MLj algorithm is a large language model or LLM, such as a GPT-4 type semantic autoregression algorithm and a Mistral Large type semantic autoregression algorithm.

[0110] It may be envisaged that other semantic autoregression algorithms, such as GPT-3, LLaMa, Gemini or Claude, or other encoder-type algorithms, such as BERT, or even other types of machine learning algorithms capable of converting a text into a simpler text, such as artificial neural networks, deep neural networks (DNNs), convolutional neural networks (CNNs), recurrent neural networks (RNNs), large language models (LLMs), support vector machines (SVMs), forests, may be used random. It is also possible to generate each simplified CRSj report using several algorithms executed sequentially.

[0111] In order to implement the conversion in each sub-step E3j, the CRM expert report is provided as input to each MLj algorithm with a query or prompt, which may be pre-written or, alternatively, written by the practitioner. This query may define instructions requesting the generation of a simplified CRSj report, excluding any use of technical or medical vocabulary.

[0112] Each machine learning algorithm MLj was previously trained, in EOj steps, to convert an expert report into a simplified report, that is to say a version of this expert report made understandable by non-experts by replacing complex medical or technical terms with more common synonyms; by adding detailed explanations of medical terms and procedures; by removing or summarizing non-essential or overly detailed information; or by reorganizing the structure of the report to highlight information considered essential for the patient, such as test results, diagnoses, and treatment recommendations.

[0113] To this end, a training set DSj was prepared beforehand for each machine learning algorithm MLj. Each training set DSj includes, on the one hand, expert reports CEk written by healthcare professionals. Each of these expert reports CEk is then labeled by an expert, in particular a physician or a member of a patient association, with a simplified report CSk. Alternatively, each of the machine learning algorithms could be trained using the same training set.

[0114] Each DSj training set is segmented into different sets of expert CEk and simplified CSk reports, each set being associated with a medical type, such as a medical specialty, or a pathology.

[0115] A simplified example of part of a DSj training game, comprising three expert reports CEi, CE2 and CE3, each labeled by a simplified report CSi, CS2 and CS3, is thus represented in [Fig.4].

[0116] In each step EOj, each machine learning algorithm MLj is thus trained to iteratively generate, from each expert report CEk of the training set and a pre-written prompt, a simplified report. At each iteration, an error metric is determined from a semantic distance between the generated simplified report and the simplified report CSk labeling the expert report CEk used to generate the simplified report. As a non-limiting example, said error metric may be a BLUE or RED metric.

[0117] The MLj machine learning algorithm can then adjust its internal parameters, such as weights and neuron biases, to minimize this error metric over iterations, for example by gradient descent.

[0118] In an LLM context, it may be foreseen that the machine learning algorithms MLj are pre-trained on a large training set not specific to the medical field, then trained by a "few-shot learning" or "fine tuning" method using a training set DSj comprising only a few pairs, such as a dozen, of expert report CEk-simplified report CSk.

[0119] Alternatively, other supervised training methods may be used or several training methods may be combined, including supervised and semi-supervised, or even unsupervised, training methods.

[0120] It may also be provided that the expert reports CEk of a training set DSj are segmented during labeling into several sections, such as a title, patient information, reasons for consultation, medical history, clinical examination, diagnosis, treatment plan, and a conclusion. Each section can thus be labeled by a section of a simplified report CSk. The training implemented in steps EOj can therefore be used to generate simplified sections from an expert report CEk, with all the sections thus generated being combined to form a simplified report which is compared to the simplified report CSk labeling the expert report CEk to adjust the MLj algorithm.

[0121] In a fourth step E4, a first proximity value Vj of the CRM expert report with the DSj training sets is estimated. This value Vi reflects the novelty of the CRM expert report with all the expert reports CEk in the batch of DSj training sets associated with the medical type corresponding to the CRM report. This medical type can be selected beforehand by the practitioner or be automatically inferred from the CRM expert report.

[0122] To this end, in a substep E41, a proximity index h > k is estimated from the CRM expert report; and each of the CEk expert reports. This proximity index h > k is calculated according to a semantic distance metric between the CRM and CEk reports, for example RED-1.

[0123] It may be envisaged that other translation or simplification evaluation metrics may be used, such as BLEU, METEOR, BERTScore, BLEURT, COMET, cosine similarity, Fl, SARI, or other measures of the distance between vector representations of texts, or even combining several semantic distance metrics or text comparison metrics.

[0124] In a substep E42, the first proximity value Vi is determined as the maximum value of the proximity indices h and thus indicates, on the one hand, which expert report CEk of the training corpus most closely resembles the incident report CRM; and on the other hand, the level of similarity between these two texts.

[0125] Alternatively, other methods of calculating the first proximity value Vj may be provided, such as the average value of the N highest values ​​of the proximity indices h k.

[0126] It should be noted that, in the examples in [Fig.3] and [Fig.4], the value Vi is 1, the CRM incident report; being identical to the CEi report of the DSj training game.

[0127] In a fifth step E5, for each simplified report CRSj, a second proximity value V2,j of this report CRSj with the training sets DSj is estimated. This value V2 j reflects the relevance of the simplified report CRSj with respect to the simplified report CSk labeling the expert report CEk of the DSj training set closest to the incident report CRM;

[0128] This second proximity value V2, j is calculated according to a semantic distance metric between the CRSj and CSk reports, distinct from that used in substep E41, for example SARI. Alternatively, the semantic distance metrics used in steps E41 and E5 may be identical.

[0129] Two simplified CRSj reports generated from the expert CRM report, represented in [Fig. 3], are shown in [Fig. 5], by two different LLM MLj, namely GPT-4 for CRSi and MISTRAL Large for CRS2, trained using the DSj training set shown in [Fig. 4].

[0130] In this example, the second proximity value V2,i between the simplified CRSi report and the CSi report is 0.97, while the second proximity value V2,2 between the simplified CRS2 report and the CSi report is 0.61.

[0131] In a sixth step E6, a confidence index Ij is estimated for each of the simplified reports CRSj inferred by the algorithms MLj from the first proximity value and the second proximity value V2j associated with the simplified report CRSj.

[0132] In the example described, a penalty Pn is calculated from the first value Vj raised to a given power, for example of 2. The penalty Pn is multiplied by the second value V2j to arrive at the confidence index Ij.

[0133] The confidence index Ij thus reflects both the similarity of the CRM expert incident report to the CEk reports of the DSj training set and the relevance of simplifying this incident report with regard to this training set. Furthermore, the penalty Pn allows for penalizing any discrepancies between the CRM report and the DSj training set.

[0134] It will thus be noted that, in the example of [Fig.3] to [Fig.5], the confidence index Ii associated with the simplified report CRSi is 0.97, while the confidence index I2 associated with the simplified report CRS2 is 0.61.

[0135] In undescribed examples, the penalty Pn may be estimated using other methods, for example, a combination of several first values ​​defined according to several metrics, such as RED-1 and BLUE-1, or the penalty Pn may be applied to the second proximity values ​​V2j, in order to penalize errors in generating the simplified report. Other combinations of the first and second values ​​VI and V2j may be considered, and a weight determined from the machine learning algorithm MLj used for generating the simplified report CRSj and / or the training set DSj may be introduced into this combination.

[0136] In a seventh step E7, each confidence index Ij is compared to a predetermined threshold value TS, for example 0.5, or even 0.75.

[0137] In an eighth step E8, all simplified reports CRSj whose confidence index Ij is greater than the threshold value TS are transmitted to the practitioner's computer terminal 2, along with their confidence index Ij, to be displayed on a viewing, editing and validation interface of the software application 31. Simplified reports CRSj that failed in step E7 are not transmitted, in order to avoid wasting time for the practitioner and risking providing unreliable information to the patient.

[0138] Therefore, the practitioner can thus view on his terminal 2 each of the simplified reports CRSj generated from his CRM report; In addition, he can also view the confidence index Ij of each of these simplified reports CRSj and evaluate both the proximity of his expert CRM report; with the training sets DSj of the machine learning algorithms MLj and the relevance of the simplification of this CRM report; with regard to this training set DSj.

[0139] The practitioner may, in particular, select the simplified CRSj report with the highest confidence index Ij and decide, by comparing this confidence index Ij to a second threshold value that he has set and / or that an association, such as a learned society or other professional organization, has recommended, whether this simplified CRSj report deserves to be revised or whether he can trust it and disseminate it to The patient can be reviewed without further review. The administrator can also evaluate each of the simplified CRSj reports to decide which one can be shared with the patient, or even with other healthcare professionals or other medical information systems, without in-depth review. The administrator can then edit the selected CRSj report to improve it before sharing it with the patient.

[0140] It may be foreseen that the fourth step E4 is implemented before step E3, and that, depending on the first proximity value Vb, steps E5 to E8 are replaced by other steps. In particular, as in the examples described in [Fig. 3] and [Fig. 4], when the CRM incident report is identical to one of the CEk reports in the DSj training set, the value Vj is 1. In this case, the simplified CRSj report can be directly selected from among the simplified CSk reports by selecting the one associated with the expert CEk report that is identical to the CRM incident report. The confidence index Ij can also be directly set to a maximum value of 1, so that only the selected simplified CRSj report and this confidence index Ij of 1 are transmitted to terminal 2.

[0141] In undescribed variants, it may be provided that only the simplified report CRSj associated with the highest confidence index Ij is transmitted during the transmission step E8, or that only the N simplified reports CRSj associated with the highest confidence indices Ij are transmitted during the transmission step E8.

[0142] Alternatively, each of the machine learning algorithms MLj could be trained to convert an expert report into a simplified report according to a given level of understanding, indicating, for example, whether certain technical terms are permitted or not, or whether specific sections should be deleted, simplified, or retained as is. In this variant, the process makes it possible to generate several CRSj reports of the same medical procedure according to different levels of understanding, so as to allow the healthcare professional to select the simplified CRSj report with a sufficiently high confidence index Ij that they deem most relevant in light of the patient's understanding and the care pathway they are following.

[0143] Alternatively, a single machine learning algorithm MLj may be used to convert the expert CRM report into a simplified CRSj report. In this case, only one simplified CRSj report will be transmitted to the practitioner's terminal 2, provided that its confidence index Ij is greater than the threshold value TS.

[0144] In a ninth step E9, the simplified CRSj reports rejected by the practitioner, as well as the modifications made to the simplified CRSj report selected by the practitioner, are transmitted to the remote server 3 in order to perform reinforcement retraining of the machine learning algorithms MLj using these different CRSj reports. It can thus be anticipated that, during this retraining, simplified reports generated by an MLj algorithm that are too similar to one of these modified or rejected CRSj reports will result in a penalty when the parameters of the MLj algorithm are adjusted.

[0145] In a tenth step E10, the expert CRM report, as well as the simplified CRSj report selected by the practitioner and possibly corrected, are transmitted to the patient's computer terminal 4. This transmission may be carried out by the transmission server 3, once these CRMi and CRSj reports have been retrieved by the server 3, following validation of their distribution by the practitioner.

[0146] In implementation examples of step E10, the processing server 3 may generate a digital identifier, for example in the form of a QR code, which can be communicated to the patient, for example via the screen of the computer terminal 2 or by being printed on a prescription form so that it can be scanned later by the patient. Entering this QR code on their computer terminal, in particular by scanning it via the software application 41 installed on the terminal 4, allows the patient to automatically download the CRM and CRSj reports from the processing server 3 into the software application 41.

[0147] It should be noted that different aspects of the various interfaces that have been represented may be provided for and / or other functionalities may be added, without going out of the scope of the present invention.

[0148] The preceding description clearly explains how the invention achieves its stated objectives, namely, to provide a method for exchanging medical reports between the various practitioners involved in a care pathway and the patient, according to appropriate levels of understanding. These objectives are achieved in particular by means of a processing server capable of generating simplified, abridged, and / or simplified reports from a medical report generated at a first computer terminal, and of transmitting all the reports to another computer terminal.

[0149] In any event, the invention cannot be limited to the embodiments specifically described in this document, and extends in particular to all equivalent means and to any technically operative combination of these means.

Claims

1. Demands Method for evaluating a report of a medical procedure performed by a healthcare professional for a patient, the report being generated automatically, the method being implemented by a computer system (1) and comprising the following steps: a. (E2) provision of a set of information (D, CRM;) relating to the medical act to the computer system; b. (E3) generation, from said provided information set, of a report (CRSj), by means of at least one machine learning algorithm (MLj) trained to generate a report of a medical act from a set of information relating to that medical act, said at least one algorithm having been trained using a training set (DSj) comprising a plurality of information sets (CEk), each relating to a given medical act, and a plurality of predetermined reports (CSk) each associated with one of said information sets; c. (E4) estimation of a first proximity value (Vi) between said provided information set and at least one information set from the training set of said machine learning algorithm; d. (E5) estimation of a second proximity value (V2j) between the report generated by the machine learning algorithm from said provided information set and at least one predetermined report from the training set of said machine learning algorithm; e. (E6) estimation of a confidence index (Ij) associated with the report generated by the machine learning algorithm from the first and second proximity values; f. (E8) transmission at least of the report generated by the machine learning algorithm and of the confidence index to a computer terminal (2).

2. A method according to the preceding claim, characterized in that the step (E4) of estimating the first proximity value (V) ) comprises a substep (E41) of estimating a proximity index (Iijk) between said provided information set (CRM;) and each information set (CEk) of a plurality of information sets of the training set (DSj), said first proximity value being calculated from said proximity indices.

3. A method according to the preceding claim, characterized in that said provided information set (CRM;) is a first report, called expert; in that the machine learning algorithm (MLj) has been trained to convert, from a training set (DSj) comprising expert reports (CEk) each associated with a simplified report (CSk), an expert report into a simplified report, and in that each proximity index (Iijk) is calculated according to at least a first metric of semantic distance between two texts.

4. A method according to the preceding claim, characterized in that said semantic distance metric is chosen from one of the following translation or simplification evaluation metrics: BLUE, RED, METEOR, BERTScore, BLEURT, COMET, cosine similarity, Fl, SARI, BARTScore.

5. A method according to any one of claims 2 to 4, characterized in that said first proximity value (VJ) is the maximum value of the proximity indices.

6. A method according to any one of the preceding claims, characterized in that said first proximity value (VJ) corresponds to a given information set (CEk) of the training set (DSj) and in that the second proximity value (V2j) is an estimated proximity index between the report (CRSj) generated by the machine learning algorithm (MLj) and a report (CSk) of the training set (DSj) associated with said given information set (CEk) of the training set corresponding to the first proximity value.

7. Method according to the preceding claim, characterized in that said proximity index (V2j) is calculated according to at least a second metric of semantic distance between two texts.

8. A method according to any one of the preceding claims, characterized in that the step (E6) of estimating a confidence index (Ij) associated with the report generated (CRSj) by the machine learning algorithm (MLj) comprises a substep of estimating a penalty (Pn) from one of the first and second proximity values ​​(Vb V2,j), the confidence index corresponding to the other of said proximity values ​​penalized by said penalty.

9. A method according to any one of the preceding claims, characterized in that it comprises, prior to the transmission step (E8), a step (E7) of comparing the confidence index (Ij) to a predetermined threshold value (TS), the transmission step being conditioned on the result of said comparison.

10. A method according to any one of the preceding claims, characterized in that the generation step (E3) is a generation step, from said provided information set (CRM;), of a plurality of reports (CRSj), each being generated by means of a separate machine learning algorithm (MLj) trained to generate a report of a medical procedure from a set of information relating to that medical procedure, each algorithm having been trained using a training set (DSj) comprising a plurality of information sets (CEk), each relating to a given medical procedure, and a plurality of predetermined reports (CSk) each associated with one of said information sets; and in that a second proximity value (V2j) and a confidence index (Ij) are estimated for each of said reports generated by the machine learning algorithms.

11. A method according to any one of the preceding claims, characterized in that it comprises a step of evaluating, on said computer terminal, the report generated by the machine learning algorithm, and a step (E9) of transmitting, to the computer system (1) said evaluation.

Images