A diagnosis and treatment auxiliary collaboration method based on multi-modal AI

Abstract

The application discloses a diagnosis and treatment auxiliary collaboration method based on a multi-modal AI, relates to the technical field of artificial intelligence auxiliary diagnosis and treatment, and comprises the following steps: carrying out a visit time back insertion on asynchronous access multi-modal clinical evidence fragments, searching for an associated candidate disease reasoning path along a diagnosis and treatment auxiliary evidence dependency index, extracting multi-modal clinical evidence fragments with a visit time in the associated candidate disease reasoning path, gathering the multi-modal clinical evidence fragments into a back insertion predecessor evidence set, extracting a back insertion direction support base around the back insertion predecessor evidence set, promoting the back insertion direction support base along an evidence dependency direction, converging the back insertion direction support base into a direction consistent reachable boundary at an evidence direction conflicting position, forming a cross-boundary incremental direction support, carrying out a local rollback on original direction support in the direction consistent reachable boundary, superimposing the local rollback original direction support and the cross-boundary incremental direction support, and splicing the local rollback original direction support and the cross-boundary incremental direction support along a visit time axis to form a rollback difference evidence sequence. The application improves the locality and traceability of evidence path rewriting.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence-assisted diagnosis and treatment technology, and in particular to a collaborative method for diagnosis and treatment assistance based on multimodal AI. Background Technology

[0002] With the continuous accumulation of electronic medical records, medical imaging, laboratory test data, and continuous physiological monitoring data in clinical diagnosis and treatment, multimodal AI-assisted diagnosis and treatment technologies are increasingly being used to jointly process clinical information from different sources. Conventional methods typically involve first accessing, temporally organizing, and normalizing the terminology of the multi-source data, and then constructing the association between clinical evidence and candidate diseases through feature fusion, knowledge graphs, disease entity matching, or inference models. The analysis results are then provided to clinicians for reference in the form of evidence chains, risk warnings, or candidate disease lists.

[0003] In the aforementioned conventional methods, candidate disease reasoning usually relies on the clinical evidence set that has been synchronously accessed. For multimodal clinical evidence such as examination reports and test results that are added asynchronously, the existing reasoning path often needs to be reorganized. At the same time, when there are differences in direction between the new evidence and the existing evidence, such as support, exclusion, or pending review, conventional processing methods make it difficult to limit the scope of local rewriting, and it is also inconvenient to trace the version differences before and after the reasoning path is rewritten. Summary of the Invention

[0004] In view of the aforementioned existing problems, the present invention is proposed.

[0005] Therefore, this invention provides a diagnostic and treatment assistance collaborative method based on multimodal AI to solve the problem that asynchronously supplemented multimodal clinical evidence is difficult to perform local incremental updates and version difference tracking in existing candidate disease inference paths.

[0006] To solve the above-mentioned technical problems, the present invention provides the following technical solution:

[0007] This invention provides a collaborative diagnostic and treatment assistance method based on multimodal AI, comprising: performing time registration and semantic normalization on multimodal clinical diagnostic and treatment data; extracting multimodal clinical evidence fragments and arranging them on the same timeline according to the consultation time to form a multimodal clinical event sequence; performing disease entity normalization and evidence direction labeling around the multimodal clinical event sequence; constructing a candidate disease evidence graph according to the evidence direction relationship between disease entities and multimodal clinical evidence fragments; labeling the candidate disease inference paths with evidence dependencies to form a diagnostic and treatment assistance evidence dependency index; performing consultation time back-interpolation on asynchronously accessed multimodal clinical evidence fragments; retrieving associated candidate disease inference paths along the diagnostic and treatment assistance evidence dependency index; extracting multimodal clinical evidence fragments with earlier consultation times from the associated candidate disease inference paths; and converging them into a back-interpolation method. The process involves inserting a precursor evidence set, extracting a support base for the insertion direction around the precursor evidence set, advancing directional support along the evidence-dependent direction, and converging at points where evidence directions contradict each other to form a reachable boundary with consistent direction, thus creating cross-boundary incremental directional support. The original directional support within the reachable boundary with consistent direction is then partially rolled back. This partially rolled-back original directional support is superimposed on the cross-boundary incremental directional support and spliced ​​along the timeline of the medical visit to form a rolled-back differential evidence sequence. Based on the rolled-back differential evidence sequence, the candidate disease evidence map is partially rewritten. Evidence differential docking is performed on the candidate disease reasoning paths before and after the partial rewriting. Version differential reversible mapping verification is performed according to the path correspondence before and after evidence differential docking. The path differences that pass the version differential reversible mapping verification are arranged into collaborative version differences, which are then incorporated into the diagnostic and treatment auxiliary collaborative record.

[0008] As a preferred embodiment of the multimodal AI-based collaborative diagnostic and treatment method of the present invention, the step of performing time registration and semantic normalization on multimodal clinical diagnostic and treatment data is as follows:

[0009] Collect multimodal clinical diagnosis and treatment data, verify the visit identification, merge records from different sources corresponding to the same visit process, and then unify the time field format and time sequence of the merged records from different sources to form time-registered clinical data.

[0010] Clinical concept standardization coding is performed on time-registered clinical data. Clinical expressions with the same semantics are merged into the same clinical concept. The attribution relationship between negation polarity and time attribute and clinical concept is corrected according to the dependency relationship in the sentence containing the clinical concept, so as to form semantically unified clinical data.

[0011] As a preferred embodiment of the multimodal AI-based diagnostic and therapeutic assistance collaborative method of the present invention, the formation of the multimodal clinical event sequence is specifically as follows:

[0012] Evidence content is extracted from semantically normalized clinical data, and clinical evidence content carrying clinical concepts, time attributes, source time and source location is extracted. Then, it is fragmented and organized according to the source modality corresponding to the clinical evidence content to form multimodal clinical evidence fragments.

[0013] The consultation time position is calibrated according to the time attribute and source time of the multimodal clinical evidence fragments. Multimodal clinical evidence fragments under the same consultation time position are grouped into clinical events, and the clinical events are arranged in chronological order of consultation time to form a multimodal clinical event sequence.

[0014] As a preferred embodiment of the multimodal AI-based collaborative diagnostic and treatment method of the present invention, the step of performing disease entity normalization and evidence direction annotation around the multimodal clinical event sequence is as follows:

[0015] According to the order of clinical events in the multimodal clinical event sequence, the clinical concepts carried by the multimodal clinical evidence fragments in each clinical event are screened for disease entities. The screened disease entities are coded and aligned with standard disease terms, and the disease entities are merged according to the coding consistency relationship and synonym relationship. At the same time, the clinical event position corresponding to the merged disease entities is retained to form disease entity normalized data.

[0016] Based on the clinical event location in the disease entity normalization data, the multimodal clinical event sequence is traced back. Within the corresponding clinical event, multimodal clinical evidence fragments carrying the corresponding clinical concepts of the normalized disease entity are located. Using the located multimodal clinical evidence fragments as anchors, multimodal clinical evidence fragments that participate in the negative polarity attribution correction and time attribute attribution correction within the same clinical event are extracted according to their source location. The located multimodal clinical evidence fragments and the extracted multimodal clinical evidence fragments are then correlated with the normalized disease entity, and the consultation time location corresponding to the multimodal clinical evidence fragments is retained to form disease evidence correspondence data.

[0017] Evidence direction labeling is performed on the data corresponding to disease evidence. First, the temporal attribute attribution carried by multimodal clinical evidence fragments is used to label the relationship between the unified disease entity and multimodal clinical evidence fragments with incomplete temporal attribute attribution as the direction to be reviewed. Then, the negative polarity attribution carried by multimodal clinical evidence fragments is used to label the relationship between the unified disease entity and multimodal clinical evidence fragments with complete temporal attribute attribution and not modified by negative polarity as the direction of support. The relationship between the unified disease entity and multimodal clinical evidence fragments with complete temporal attribute attribution and modified by negative polarity is labeled as the direction of exclusion, thus forming evidence direction labeling data.

[0018] As a preferred embodiment of the multimodal AI-based collaborative diagnostic and treatment assistance method of the present invention, the formation of the diagnostic and treatment assistance evidence dependency index is specifically as follows:

[0019] Based on the evidence direction annotation data, a candidate disease evidence graph is constructed. Candidate diseases are formed by normalized disease entities in the disease entity normalization data, and evidence vertices are formed by multimodal clinical evidence fragments. Evidence direction edges are established between candidate diseases and evidence vertices according to the evidence direction relationship, and the consultation time position corresponding to the multimodal clinical evidence fragments is retained to form a candidate disease evidence graph.

[0020] The candidate disease evidence graph is organized into a candidate disease reasoning path. The evidence vertices are connected according to the direction category and the location of the consultation time under the same candidate disease. Segmentation boundaries are set at the locations where the direction category of the evidence direction edge changes, thus forming the candidate disease reasoning path.

[0021] Evidence dependency annotation is performed on the reasoning path of candidate diseases, and the correspondence between the reasoning path of candidate diseases and multimodal clinical evidence fragments, evidence direction relationships, consultation time and location and segment boundaries is recorded to form a diagnostic and treatment auxiliary evidence dependency index.

[0022] As a preferred embodiment of the multimodal AI-based collaborative diagnostic and treatment assistance method of the present invention, the step of back-interpolating the consultation time of asynchronously accessed multimodal clinical evidence fragments and retrieving the inference path of associated candidate diseases along the diagnostic and treatment assistance evidence dependency index is as follows:

[0023] The appointment time of asynchronously accessed multimodal clinical evidence fragments is calibrated, and the time attributes and source time of the asynchronously accessed multimodal clinical evidence fragments are aligned with the appointment time axis in the multimodal clinical event sequence to form multimodal clinical evidence fragments with back-insertion time positions.

[0024] Following the diagnostic and treatment auxiliary evidence dependency index, candidate disease reasoning paths are retrieved based on the clinical concepts and interpolation time positions carried by multimodal clinical evidence fragments with interpolation time positions. According to the annotation method in the evidence direction annotation data, evidence direction annotation is performed between the multimodal clinical evidence fragments with interpolation time positions and the candidate diseases corresponding to the retrieved candidate disease reasoning paths, forming multimodal clinical evidence fragments with interpolation time positions and evidence direction relationships. The retrieved candidate disease reasoning paths are then used as the associated candidate disease reasoning paths.

[0025] As a preferred embodiment of the multimodal AI-based collaborative diagnostic and treatment method of the present invention, the formation of cross-boundary incremental directional support is specifically as follows:

[0026] Extract multimodal clinical evidence fragments that precede the time of visit relative to the insertion time from the inference path of associated candidate diseases, and retain the dependency order between the multimodal clinical evidence fragments according to the evidence dependency annotation in the inference path of associated candidate diseases, and aggregate them into an insertion precursor evidence set;

[0027] Evidence direction aggregation is performed on the pre-insertion evidence set. Multimodal clinical evidence fragments with the same evidence direction relationship as those with the inference path of the candidate disease and which depend on the sequentially continuous multimodal clinical evidence fragments with the inference time position and evidence direction relationship of the inversion are extracted to form the inversion direction support base.

[0028] Starting from the base of the back-insertion direction, the direction of directional support is advanced along the evidence-dependent direction in the inference path of the associated candidate disease. Positions with continuous and consistent evidence directions are included in the advancement range, and advancement is stopped at positions with contradictory evidence directions, thus converging to form a directionally consistent reachable boundary.

[0029] Multimodal clinical evidence fragments with interpolation time positions and evidence direction relationships are inserted into the interpolation time positions in the inference path of associated candidate diseases. Continuous evidence direction edges extending to the reachable boundary with consistent direction are extracted from the interpolation time positions along the inference path of associated candidate diseases. The continuous evidence direction edges and the corresponding multimodal clinical evidence fragments are then organized into cross-boundary incremental direction support.

[0030] As a preferred embodiment of the multimodal AI-based collaborative diagnostic and treatment method of the present invention, the original directional support within the reachable boundary with consistent direction is locally rolled back, and the locally rolled-back original directional support is superimposed with the cross-boundary incremental directional support, and spliced ​​along the medical treatment time axis to form a rollback differential evidence sequence, as follows:

[0031] Extract the existing continuous evidence direction edges and corresponding multimodal clinical evidence fragments within the reachable boundary of the associated candidate disease reasoning path as the original direction support. Perform local rollback on the original direction support, remove the original direction support parts that overlap with the cross-boundary incremental direction support in terms of consultation time and location and contradict each other in terms of evidence direction, and retain the original direction support parts that do not overlap or contradict each other to form the original direction support after local rollback.

[0032] The original directional support after local rollback is superimposed with the cross-boundary incremental directional support, and the superimposed directional support is sequentially spliced ​​according to the time axis of medical visits. At the same time, the corresponding candidate diseases, multimodal clinical evidence fragments, evidence directional relationships and medical visit time locations are associated to form a rollback differential evidence sequence.

[0033] As a preferred embodiment of the multimodal AI-based collaborative diagnostic and treatment method of the present invention, the step of locally rewriting the candidate disease evidence map based on the rollback differential evidence sequence and performing evidence differential docking on the candidate disease inference paths before and after the local rewriting is as follows:

[0034] Based on the candidate diseases, evidence direction relationships, and consultation time and location in the rollback differential evidence sequence, the local rewriting range in the candidate disease evidence map is located. The evidence vertices and evidence direction edges within the local rewriting range are deleted, written, and sequentially reconnected according to the rollback differential evidence sequence to form the locally rewritten candidate disease evidence map.

[0035] The candidate disease evidence graph after partial rewriting is organized into candidate disease inference paths. Evidence vertices are reconnected according to the candidate disease, the direction category of the evidence direction edge, and the location of the consultation time to form the candidate disease inference path after partial rewriting. Then, the path positions of the candidate disease inference paths before and after partial rewriting are aligned. The corresponding multimodal clinical evidence fragments, evidence direction relationships, and consultation time locations in the candidate disease inference paths before and after partial rewriting are subjected to evidence differential docking to form evidence differential docking data.

[0036] As a preferred embodiment of the multimodal AI-based collaborative diagnosis and treatment method of the present invention, the path differences verified by the version difference reversible mapping are arranged into collaborative version differences, and the collaborative version differences are incorporated into the collaborative diagnosis and treatment record, as follows:

[0037] According to the path correspondence in the evidence differential docking data, the version differential reversible mapping verification is performed. The candidate disease reasoning path after local rewriting is reverse mapped to the candidate disease reasoning path before local rewriting according to the candidate disease, the location of the consultation time, the evidence vertex and the evidence direction edge. The correspondence between the multimodal clinical evidence fragments, the evidence direction relationship and the location of the consultation time after the reverse mapping is verified.

[0038] Extract the path differences verified by the version difference reversible mapping, and arrange the path differences according to candidate diseases, evidence direction relationship, consultation time and location, and local rewriting scope to form a collaborative version difference;

[0039] The collaborative version difference is incorporated into the diagnostic and treatment assistance collaborative record, and the rewriting position of the candidate disease reasoning path, the difference in evidence before and after rewriting, and the reversible mapping verification content of the version difference are associated with the diagnostic and treatment assistance collaborative record.

[0040] The beneficial effects of this invention are as follows: by interpolating the consultation time of asynchronously accessed multimodal clinical evidence fragments, cross-boundary incremental directional support is formed along the dependency index of diagnostic and treatment auxiliary evidence, and the original directional support within the reachable boundary with consistent direction is locally rolled back, so that the rolled-back differential evidence sequence can be used for local rewriting of candidate disease evidence graphs; then, the path differences are arranged into collaborative version differences and incorporated into the diagnostic and treatment auxiliary collaborative record through version differential reversible mapping verification, which is used for diagnostic and treatment auxiliary collaborative updates after asynchronous evidence is added, thereby improving the locality and traceability of evidence path rewriting. Attached Figure Description

[0041] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0042] Figure 1 This is a flowchart of a collaborative diagnostic and treatment assistance method based on multimodal AI.

[0043] Figure 2 This is a flowchart for asynchronous evidence back-insertion, partial rollback, and version difference collaborative processing.

[0044] Figure 3 The graph shows the test results of update time for each method under different complexity quantile groups.

[0045] Figure 4 The graph shows the test results of the pass rate distribution for reversible mapping verification using different methods. Detailed Implementation

[0046] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0047] Many specific details are set forth in the following description in order to provide a full understanding of the invention. However, the invention may also be practiced in other ways different from those described herein, and those skilled in the art can make similar extensions without departing from the spirit of the invention. Therefore, the invention is not limited to the specific embodiments disclosed below.

[0048] Secondly, the term "one embodiment" or "embodiment" as used herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation of the present invention. The phrase "in one embodiment" appearing in different places in this specification does not necessarily refer to the same embodiment, nor is it a single or selective embodiment that is mutually exclusive with other embodiments.

[0049] Reference Figures 1-4 This is one embodiment of the present invention, which provides a collaborative diagnostic and treatment assistance method based on multimodal AI, comprising the following steps:

[0050] S1: Perform time registration and semantic normalization on multimodal clinical diagnosis and treatment data, extract multimodal clinical evidence fragments and arrange them into the same consultation time axis according to the consultation time to form a multimodal clinical event sequence;

[0051] S1.1: Collect multimodal clinical diagnosis and treatment data, verify the visit identification, merge records from different sources corresponding to the same visit process, and then unify the time field format and time sequence of the merged records from different sources to form time-registered clinical data.

[0052] Furthermore, when collecting multimodal clinical diagnosis and treatment data, electronic medical records, laboratory records, imaging reports, physiological monitoring records, and treatment records are received according to their source types, and patient identifiers, consultation numbers, hospitalization numbers, examination request numbers, laboratory request numbers, record generation time, and record sources are extracted from records from different sources.

[0053] The same medical process refers to a continuous medical process in which the patient identification is consistent and the consultation number, hospitalization number, examination request number, test request number and record generation time can correspond to each other. First, the medical identification is checked according to the patient identification and consultation number. Then, the hospitalization number, examination request number, test request number and record generation time are used to supplement and check the records from different sources. The records from different sources that are checked to the same medical process are merged into the same medical set.

[0054] For the same patient visit set, the field names of the collection time, examination time, report time, entry time and treatment time from different sources are standardized and the time format is converted. The time sequence of the clinical behavior between the application, collection, examination, report, treatment and entry is calibrated to form time-registered clinical data.

[0055] S1.2: Perform clinical concept standardization coding on time-registered clinical data, merge semantically identical clinical expressions into the same clinical concept, and correct the attribution relationship between negation polarity and time attributes and clinical concepts according to the dependency relationship in the statement containing the clinical concept, thus forming semantically unified clinical data;

[0056] Furthermore, when performing clinical concept standardization coding on time-registered clinical data, the text content in the time-registered clinical data is first segmented into sentences, words, and clinical named entity recognition to obtain the textual clinical expression; then, the structured fields in the time-registered clinical data are extracted by field name extraction, field value extraction, and unit name organization to obtain the field clinical expression.

[0057] The textual and field-based clinical expressions are coded and aligned with medical terminology dictionaries, hospital terminologies, and standard disease entries. Clinical expressions with consistent coding are merged, while those with inconsistent coding but identical semantics are merged according to synonym relationships. The merged clinical expressions are then written as the same clinical concept. A clinical concept is a medical expression formed after standardized coding; negation polarity indicates whether the clinical concept is modified by a negative expression, and temporal attribute indicates the time period to which the clinical concept belongs.

[0058] Dependency analysis is performed on the sentences containing clinical concepts to locate the scope of influence of negation and temporal expressions on clinical concepts. The negation polarity and temporal attributes covered by the scope of influence are then bound to the corresponding clinical concepts to form semantically normalized clinical data.

[0059] S1.3: Extract evidence content from semantically normalized clinical data, extract clinical evidence content carrying clinical concepts, time attributes, source time and source location, and fragment it according to the source modality corresponding to the clinical evidence content to form multimodal clinical evidence fragments.

[0060] Furthermore, when extracting evidence content from semantically normalized clinical data, the clinical concept is used as the positioning object. The sentence containing the clinical concept, adjacent conclusion sentences and corresponding paragraphs are extracted from the text content. The field name, field value and record source corresponding to the clinical concept are extracted from the structured fields. The image description and examination conclusion containing the clinical concept are extracted from the image report. The monitoring time period and monitoring record content corresponding to the clinical concept are extracted from the physiological monitoring record. The treatment name, treatment time and treatment record content corresponding to the clinical concept are extracted from the treatment record.

[0061] The extracted content is written as clinical evidence content, and a binding relationship is established between the clinical evidence content and the corresponding clinical concepts, time attributes, source time, and source location. Then, the clinical evidence content is fragmented according to its corresponding source modality, incorporating clinical concepts, time attributes, source time, source location, source modality, negation polarity attribution, and time attribute attribution into the fragmented clinical evidence content, forming multimodal clinical evidence fragments. These multimodal clinical evidence fragments serve as the evidence units for subsequent disease entity unification, evidence direction labeling, and consultation time back-interpolation.

[0062] S1.4: The consultation time position is calibrated according to the time attribute and source time of the multimodal clinical evidence fragments. Multimodal clinical evidence fragments under the same consultation time position are grouped into clinical events, and the clinical events are arranged in chronological order of consultation time to form a multimodal clinical event sequence.

[0063] Furthermore, when calibrating the location of the consultation time according to the time attribute and source time of the multimodal clinical evidence fragment, it is first determined whether the time attribute of the multimodal clinical evidence fragment points to the current consultation process or a previous time; for multimodal clinical evidence fragments whose time attribute points to the current consultation process, the corresponding position is located in the same consultation time axis according to the source time.

[0064] For multimodal clinical evidence fragments whose time attributes point to past times, the earlier position is located on the same consultation timeline according to the chronological order of the time attributes. For multimodal clinical evidence fragments with missing time attributes but complete source times, the corresponding position is located on the same consultation timeline according to the source time. The consultation time position is the position formed after the multimodal clinical evidence fragments are written into the same consultation timeline.

[0065] After completing the calibration of the consultation time and location, the multimodal clinical evidence fragments under the same consultation time and location are grouped into clinical events, and the clinical events are arranged in chronological order of consultation time to form a multimodal clinical event sequence.

[0066] S2: Perform disease entity normalization and evidence direction labeling around the multimodal clinical event sequence, construct candidate disease evidence graphs according to the evidence direction relationship between disease entities and multimodal clinical evidence fragments, label the evidence dependency of candidate disease reasoning paths, and form a diagnostic and treatment auxiliary evidence dependency index;

[0067] S2.1: According to the order of clinical events in the multimodal clinical event sequence, the clinical concepts carried by the multimodal clinical evidence fragments in each clinical event are screened for disease entities. The screened disease entities are coded and aligned with standard disease terms. Disease entities are merged according to coding consistency and synonymy relationship. At the same time, the clinical event position corresponding to the merged disease entities is retained to form disease entity normalization data.

[0068] Furthermore, following the order of clinical events in the multimodal clinical event sequence, the multimodal clinical evidence fragments within each clinical event are processed sequentially; from the clinical concepts carried by the multimodal clinical evidence fragments, clinical concepts pointing to disease names, disease diagnosis names, and disease-related descriptions are selected, and the selected clinical concepts are written as disease entities;

[0069] Standard disease terms are sets of disease names used for disease entity coding alignment and synonym merging. A standard disease term includes the standard disease name, synonyms, disease codes, and hierarchical relationships. Disease entities are matched with the standard disease names and codes in the standard disease terms; disease entities with the same codes are linked by a coding consistency relationship. Disease entities that have not yet completed coding matching are compared with synonyms in the standard disease terms; disease entities with the same name meaning are linked by a synonym relationship. Disease entities are merged in order of priority for coding consistency relationships, followed by synonym relationships, and the merged disease entities are associated with their corresponding clinical event locations to form unified disease entity data.

[0070] It should be noted that the standard disease entries are derived from the International Classification of Diseases (ICD), hospital disease dictionaries, and medical knowledge bases. The standard disease entries are used to align the codes of disease entities, merge synonyms, and verify hierarchical relationships.

[0071] S2.2: Based on the clinical event location in the disease entity normalization data, backtrack the multimodal clinical event sequence, locate the multimodal clinical evidence fragments carrying the corresponding clinical concepts of the normalized disease entity within the corresponding clinical event, use the located multimodal clinical evidence fragments as anchor points, extract the multimodal clinical evidence fragments that participate in the negative polarity attribution correction and time attribute attribution correction within the same clinical event according to the source location, establish the correspondence between the located multimodal clinical evidence fragments and the extracted multimodal clinical evidence fragments and the normalized disease entity, and retain the consultation time location corresponding to the multimodal clinical evidence fragments to form disease evidence correspondence data;

[0072] Furthermore, based on the location of clinical events in the disease entity normalization data, the multimodal clinical event sequence is traced back to locate the clinical event where the normalized disease entity is located; within the clinical event where the normalized disease entity is located, multimodal clinical evidence fragments carrying the corresponding clinical concepts of the normalized disease entity are searched, and the searched multimodal clinical evidence fragments are used as the location of the multimodal clinical evidence fragments.

[0073] Using the multimodal clinical evidence fragments obtained from the location as anchor points, multimodal clinical evidence fragments that are in the same sentence, adjacent sentence, same paragraph, adjacent field, or same record segment as the anchor point are searched within the same clinical event according to their source location; multimodal clinical evidence fragments that participate in the negative polarity attribution correction are extracted from the searched multimodal clinical evidence fragments, and multimodal clinical evidence fragments that participate in the time attribute attribution correction are also extracted.

[0074] The multimodal clinical evidence fragments located, the multimodal clinical evidence fragments involved in the negative polarity attribution correction, and the multimodal clinical evidence fragments involved in the time attribute attribution correction are correlated with the normalized disease entity. At the same time, the consultation time location corresponding to the multimodal clinical evidence fragments is retained to form disease evidence correspondence data.

[0075] S2.3: Perform evidence direction labeling on the data corresponding to disease evidence. First, use the time attribute attribution carried by the multimodal clinical evidence fragments to label the relationship between the unified disease entity and the multimodal clinical evidence fragment with incomplete time attribute attribution as the direction to be reviewed. Then, use the negative polarity attribution carried by the multimodal clinical evidence fragments to label the relationship between the unified disease entity and the multimodal clinical evidence fragment with complete time attribute attribution and not modified by negative polarity as the support direction. Label the relationship between the unified disease entity and the multimodal clinical evidence fragment with complete time attribute attribution and modified by negative polarity as the exclusion direction, thus forming evidence direction labeling data.

[0076] Furthermore, when annotating the evidence direction of the data corresponding to disease evidence, the time attribute of the multimodal clinical evidence fragments in the data corresponding to disease evidence is checked one by one. When the time attribute is missing a time pointer, the time pointer cannot be written into the consultation time position in the multimodal clinical event sequence, or the time pointer and the source time cannot be aligned sequentially, the relationship between the normalized disease entity and the corresponding multimodal clinical evidence fragment is marked as the direction to be reviewed.

[0077] For multimodal clinical evidence fragments with complete time attributes, the negative polarity attribute carried by the multimodal clinical evidence fragments is examined. When the multimodal clinical evidence fragments are not modified by negative expressions, the relationship between the normalized disease entity and the corresponding multimodal clinical evidence fragments is marked as the support direction. When the multimodal clinical evidence fragments are modified by negative expressions, the relationship between the normalized disease entity and the corresponding multimodal clinical evidence fragments is marked as the exclusion direction.

[0078] The evidence direction relationship is the support direction, exclusion direction, and direction to be reviewed between the unified disease entity and the multimodal clinical evidence fragment; the unified disease entity, multimodal clinical evidence fragment, consultation time and location, and evidence direction relationship are associated to form evidence direction annotation data.

[0079] S2.4: Construct a candidate disease evidence graph based on the evidence direction annotation data. Form candidate diseases with normalized disease entities in the disease entity normalization data, form evidence vertices with multimodal clinical evidence fragments, establish evidence direction edges between candidate diseases and evidence vertices according to the evidence direction relationship, and retain the consultation time position corresponding to the multimodal clinical evidence fragments to form a candidate disease evidence graph.

[0080] Furthermore, when constructing the candidate disease evidence graph based on the evidence direction annotation data, the normalized disease entities in the disease entity normalization data are first written as candidate diseases, and the candidate diseases are only used as intermediate objects in the processing of diagnostic and treatment auxiliary information; the multimodal clinical evidence fragments in the evidence direction annotation data are written as evidence vertices; then, according to the evidence direction relationship between the normalized disease entities and the multimodal clinical evidence fragments in the evidence direction annotation data, the evidence direction edge between the candidate disease and the evidence vertex is established.

[0081] The candidate disease evidence map consists of candidate diseases, evidence vertices, and evidence direction edges. The evidence direction edges record the evidence direction relationship between the candidate disease and the evidence vertex. The evidence vertex and evidence direction edges are both written with the consultation time position corresponding to the multimodal clinical evidence fragment, thus forming the candidate disease evidence map.

[0082] S2.5: Organize the candidate disease reasoning path for the candidate disease evidence graph, connect the evidence vertices according to the direction category and consultation time location of the evidence direction edge under the same candidate disease, and set segment boundaries at the position where the direction category of the evidence direction edge changes to form the candidate disease reasoning path;

[0083] Furthermore, when organizing the candidate disease inference path for the candidate disease evidence graph, the evidence vertices are first grouped according to the candidate disease; under the same candidate disease, the evidence vertices are sorted according to the consultation time position corresponding to the evidence vertex; then, adjacent evidence vertices are connected according to the sorting result, and the corresponding evidence direction edges between adjacent evidence vertices are written into the connection position to form the candidate disease inference path under the same candidate disease.

[0084] The candidate disease reasoning path is a path formed by connecting the vertices of evidence according to the location of the consultation time and the direction category of the evidence direction edge under the same candidate disease; when the direction category of adjacent evidence direction edges switches between the support direction, exclusion direction and the direction to be reviewed, the switching position is written as the segment boundary to form the candidate disease reasoning path.

[0085] S2.6: Add evidence dependency annotations to the reasoning paths of candidate diseases, record the correspondence between the reasoning paths of candidate diseases and multimodal clinical evidence fragments, evidence direction relationships, consultation time and location and segment boundaries, and form a diagnostic and treatment auxiliary evidence dependency index;

[0086] Furthermore, when annotating the candidate disease reasoning path with evidence dependencies, the multimodal clinical evidence fragments corresponding to each evidence vertex, the evidence direction relationships corresponding to each evidence direction edge, the location of the consultation time corresponding to each multimodal clinical evidence fragment, and the segment boundaries in the candidate disease reasoning path are recorded sequentially along the chronological order of the consultation time of the candidate disease reasoning path; then, an index relationship is established between the candidate disease reasoning path and the recorded multimodal clinical evidence fragments, evidence direction relationships, consultation time locations, and segment boundaries.

[0087] The diagnostic and treatment auxiliary evidence dependency index is used to reverse locate the corresponding multimodal clinical evidence fragments, evidence direction relationships, consultation time and location, and segment boundaries according to the reasoning path of candidate diseases, thus forming the diagnostic and treatment auxiliary evidence dependency index.

[0088] S3: Perform back-interpolation of the consultation time for asynchronously accessed multimodal clinical evidence fragments, retrieve the inference path of related candidate diseases along the diagnostic and treatment auxiliary evidence dependency index, extract the multimodal clinical evidence fragments with earlier consultation times from the inference path of related candidate diseases, and converge them into a back-interpolation precursor evidence set. Extract the back-interpolation direction support base around the back-interpolation precursor evidence set, advance the direction support along the evidence dependency direction, and converge at the position where the evidence directions are contradictory into a direction consistent reachable boundary to form cross-boundary incremental direction support. Perform local rollback on the original direction support within the direction consistent reachable boundary, superimpose the original direction support after local rollback with the cross-boundary incremental direction support, and splice them along the consultation time axis to form a rollback differential evidence sequence;

[0089] S3.1: Perform consultation time calibration on asynchronously accessed multimodal clinical evidence fragments, align the time attributes and source time of the asynchronously accessed multimodal clinical evidence fragments with the consultation time axis in the multimodal clinical event sequence, and form multimodal clinical evidence fragments with back-insertion time positions;

[0090] Furthermore, the asynchronously accessed multimodal clinical evidence fragments are multimodal clinical evidence fragments formed according to the aforementioned time registration, semantic normalization, and evidence content extraction methods. Asynchronous access means that the access time of the multimodal clinical evidence fragment is later than the formation time of the multimodal clinical event sequence and the diagnostic and treatment auxiliary evidence dependency index. Asynchronous access does not change the clinical concepts, time attributes, source time, and source location carried by the multimodal clinical evidence fragments.

[0091] When calibrating the consultation time of asynchronously accessed multimodal clinical evidence fragments, the time attributes and source time carried by the asynchronously accessed multimodal clinical evidence fragments are first extracted, and the time attributes are matched with the consultation time axis in the multimodal clinical event sequence for temporal semantic matching.

[0092] When the time attribute points to the current consultation process, the system searches for adjacent consultation time positions in the consultation timeline according to the source time, and writes the insertion position between adjacent consultation time positions as the back-insertion time position; when the time attribute points to a past time, the system searches for the earlier consultation time position in the consultation timeline according to the chronological relationship expressed by the time attribute, and writes the earlier consultation time position as the back-insertion time position; when the time attribute is missing but the source time is complete, the system searches for adjacent consultation time positions in the consultation timeline according to the source time, and writes the insertion position between adjacent consultation time positions as the back-insertion time position.

[0093] The time position of the back-insertion is the position formed after the asynchronously accessed multimodal clinical evidence fragment is written into the existing timeline of the visit. After the back-insertion time position is written, a multimodal clinical evidence fragment with the back-insertion time position is formed.

[0094] S3.2: Along the diagnostic and treatment auxiliary evidence dependency index, retrieve candidate disease reasoning paths based on the clinical concepts and back-injection time positions carried by the multimodal clinical evidence fragments with back-injection time positions. According to the annotation method in the evidence direction annotation data, perform evidence direction annotation between the multimodal clinical evidence fragments with back-injection time positions and the candidate diseases corresponding to the retrieved candidate disease reasoning paths to form multimodal clinical evidence fragments with back-injection time positions and evidence direction relationships. And use the retrieved candidate disease reasoning paths as the associated candidate disease reasoning paths.

[0095] Furthermore, when searching along the diagnostic and treatment auxiliary evidence dependency index, the clinical concepts and interpolation time positions carried by the multimodal clinical evidence fragments with interpolation time positions are first extracted; in the diagnostic and treatment auxiliary evidence dependency index, the clinical concepts carried by the multimodal clinical evidence fragments with interpolation time positions are used as the search objects, and the corresponding multimodal clinical evidence fragment records are retrieved according to the clinical concept coding consistency relationship and synonym name relationship, and then the candidate disease reasoning path is located in reverse from the multimodal clinical evidence fragment records;

[0096] Based on the interpolation time position, select candidate disease inference paths from the candidate disease inference paths obtained from the localization, which are adjacent in terms of visit time position, correspond to candidate diseases, and are adjacent in terms of segment boundaries.

[0097] When annotating the evidence direction of multimodal clinical evidence fragments with interpolated time positions, the time attribute is checked first, and then the negative polarity is checked, according to the annotation method in the evidence direction annotation data. The relationship between the multimodal clinical evidence fragments with interpolated time positions and the candidate diseases corresponding to the retrieved candidate disease inference paths is written as the support direction, exclusion direction, or direction to be reviewed, forming multimodal clinical evidence fragments with interpolated time positions and evidence direction relationships; the candidate disease inference paths corresponding to the completed screening and evidence direction annotation are used as the associated candidate disease inference paths.

[0098] S3.3: Extract multimodal clinical evidence fragments from the inference path of associated candidate diseases that come before the time of visit relative to the time of back-insertion, and retain the dependency order between the multimodal clinical evidence fragments according to the evidence dependency label in the inference path of associated candidate diseases, and aggregate them into a back-insertion precursor evidence set;

[0099] Furthermore, when extracting multimodal clinical evidence fragments from the inference path of associated candidate diseases, the multimodal clinical evidence fragments within the inference path of associated candidate diseases are first arranged sequentially according to the consultation time position recorded in the diagnostic and treatment auxiliary evidence dependency index.

[0100] Using the insertion time position as the boundary, extract multimodal clinical evidence fragments whose consultation time position is earlier than the insertion time position from the arranged multimodal clinical evidence fragments; then, according to the evidence dependency label in the inference path of associated candidate diseases, retain the path connection order, evidence direction relationship order and segment boundary order among the extracted multimodal clinical evidence fragments; and aggregate the multimodal clinical evidence fragments with retained dependency order into the insertion precursor evidence set.

[0101] S3.4: Aggregate the evidence direction of the back-insertion precursor evidence set, extract multimodal clinical evidence fragments that have the same evidence direction relationship with the multimodal clinical evidence fragments with back-insertion time position and evidence direction relationship and that are sequentially continuous in the inference path of the associated candidate disease, and form the back-insertion direction support base;

[0102] Furthermore, when performing evidence direction aggregation on the pre-insertion evidence set, the evidence direction relationship corresponding to the multimodal clinical evidence fragments with the insertion time position and evidence direction relationship is first obtained; then, the evidence direction relationship corresponding to the multimodal clinical evidence fragments is compared item by item in the pre-insertion evidence set, and multimodal clinical evidence fragments with the same evidence direction relationship as the multimodal clinical evidence fragments with the insertion time position and evidence direction relationship are extracted.

[0103] According to the evidence dependency annotations in the inference path of associated candidate diseases, check whether the dependency order of the extracted multimodal clinical evidence fragments in the path is continuous, and retain the multimodal clinical evidence fragments with continuous dependency order; arrange the retained multimodal clinical evidence fragments according to the order of consultation time and location to form the back-insertion direction support base.

[0104] S3.5: Starting from the base of the back-insertion direction support, advance the direction of directional support along the evidence dependence direction in the reasoning path of the associated candidate disease, include the positions with continuous and consistent evidence directions in the advancement range, and stop advancing at the positions with contradictory evidence directions, and converge to form a directionally consistent reachable boundary;

[0105] Furthermore, when advancing the direction support starting from the back-insertion direction support base, first locate the starting evidence vertex and the ending evidence vertex corresponding to the back-insertion direction support base in the related candidate disease reasoning path; then, starting from the ending evidence vertex, check the subsequent evidence direction edges item by item along the evidence dependency direction in the related candidate disease reasoning path.

[0106] When the evidence direction relationship corresponding to the subsequent evidence direction edge is continuous and consistent with the evidence direction relationship of the supporting base of the back-insertion direction, the subsequent evidence direction edge and the corresponding multimodal clinical evidence fragment are included in the advancement scope; when the evidence direction relationship corresponding to the subsequent evidence direction edge contradicts the evidence direction relationship of the supporting base of the back-insertion direction, the advancement of directional support is stopped. Contradictory evidence directions refer to a conflict between the supporting direction and the exclusion direction, or an uncertain conflict between the supporting direction, the exclusion direction, and the direction to be reviewed; the terminal evidence vertex and the terminal evidence direction edge before the stopping position are considered as the directional consistency reachable boundary. "Removing" forms the directional consistency reachable boundary.

[0107] S3.6: Insert multimodal clinical evidence fragments with interpolation time position and evidence direction relationship into the interpolation time position in the inference path of associated candidate diseases, and extract continuous evidence direction edges extending to the boundary outside the reachable boundary with consistent direction from the interpolation time position along the inference path of associated candidate diseases. Organize the continuous evidence direction edges and the corresponding multimodal clinical evidence fragments into cross-boundary incremental direction support.

[0108] Furthermore, the multimodal clinical evidence fragments with the back-insertion time position and evidence direction relationship are first inserted into the related candidate disease reasoning path according to the back-insertion time position. The evidence vertices before the back-insertion time position are connected to the evidence vertices corresponding to the multimodal clinical evidence fragments with the back-insertion time position and evidence direction relationship, and the evidence vertices corresponding to the multimodal clinical evidence fragments with the back-insertion time position and evidence direction relationship are connected to the evidence vertices after the back-insertion time position. The resulting evidence direction edges are written according to the evidence direction relationship corresponding to the multimodal clinical evidence fragments with the back-insertion time position and evidence direction relationship.

[0109] Starting from the evidence vertex corresponding to the interpolation time position, extract continuous evidence direction edges along the evidence dependency direction in the inference path of the associated candidate disease. When the evidence direction relationship corresponding to the continuous evidence direction edge is consistent with the evidence direction relationship corresponding to the multimodal clinical evidence fragment with interpolation time position and evidence direction relationship, continue to extract. When the continuous evidence direction edge extends beyond the reachable boundary of the consistent direction, organize the continuous evidence direction edges starting from the interpolation time position and extending beyond the reachable boundary of the consistent direction, as well as the multimodal clinical evidence fragments corresponding to the continuous evidence direction edges, to form cross-boundary incremental direction support.

[0110] S3.7: Extract the existing continuous evidence direction edges and corresponding multimodal clinical evidence fragments within the reachable boundary of the associated candidate disease reasoning path as the original direction support. Perform local rollback on the original direction support, remove the original direction support parts that overlap with the cross-boundary incremental direction support in terms of consultation time and location and contradict each other in terms of evidence direction, and retain the original direction support parts that do not overlap or contradict each other to form the original direction support after local rollback.

[0111] Furthermore, when extracting the original directional support from the inference path of associated candidate diseases, the existing continuous evidence directional edges within the reachable boundary with consistent direction are first located, and the multimodal clinical evidence fragments corresponding to the continuous evidence directional edges are extracted; the existing continuous evidence directional edges within the reachable boundary with consistent direction and the corresponding multimodal clinical evidence fragments are used as the original directional support.

[0112] When performing a partial rollback of the original directional support, first write the supporting direction, the direction to be reviewed, and the excluded direction as directional values ​​respectively. Then, calculate the rollback conflict value for each multimodal clinical evidence fragment in the original directional support. The expression is:

[0113] ;

[0114] in, Indicates the first support in the original direction The rollback conflict value corresponding to each multimodal clinical evidence fragment. This indicates the number of multimodal clinical evidence fragments supporting the cross-boundary incremental direction. Indicates the first support in the original direction The multimodal clinical evidence fragments and cross-boundary incremental directions support the first The overlap value of visit time and location among multimodal clinical evidence fragments, Indicates the first support in the original direction The numerical values ​​corresponding to the evidence direction of a multimodal clinical evidence fragment. Indicates the cross-boundary incremental directional support of the first The numerical values ​​of the evidence direction corresponding to each multimodal clinical evidence fragment;

[0115] When both patients visit the clinic at the same time and location. The value is one, when the two patients have different times and locations for their medical visits. The value is zero; the value of the evidence direction corresponding to the supporting direction is one, the value of the evidence direction corresponding to the direction to be reviewed is zero, and the value of the evidence direction corresponding to the exclusion direction is negative one; when When the value is greater than zero, the first support is removed from the original direction support. A multimodal clinical evidence fragment and its corresponding continuous evidence direction edge, when When equal to zero, retain the first... A multimodal clinical evidence fragment and its corresponding continuous evidence direction edge form the original direction support after local rollback.

[0116] It should be noted that the number of multimodal clinical evidence fragments in the cross-boundary incremental directional support is used to limit the scope of item-by-item comparison; the overlap value of the consultation time location is used to indicate whether the multimodal clinical evidence fragments in the original directional support and the multimodal clinical evidence fragments in the cross-boundary incremental directional support are located at the same consultation time location. If they are located at the same consultation time location, it is written as overlap; if they are not located at the same consultation time location, it is written as non-overlap; the evidence direction value is obtained by converting the evidence direction relationship, where the supporting direction is written as a positive value, the direction to be reviewed is written as an intermediate value, and the exclusion direction is written as a reverse value, so that the difference between different evidence direction relationships can be calculated.

[0117] The object of partial rollback is the original direction support. The triggering condition for partial rollback is that the original direction support and the cross-boundary incremental direction support overlap at the consultation time position and the evidence direction relationship is contradictory. Therefore, the cross-boundary incremental direction support content that is at the same consultation time position as the current multimodal clinical evidence fragment in the original direction support is first screened by the consultation time position overlap value. Then, the degree of evidence direction conflict at the same consultation time position is calculated by the difference between the evidence direction values. When the evidence direction relationship is consistent, the difference in evidence direction does not form a rollback conflict. When the evidence direction relationship is contradictory, the difference in evidence direction forms a rollback conflict. The denominator is used to retain the inhibitory effect of non-contradictory directions on the rollback conflict value and to avoid the calculation result losing its calculation basis due to the lack of overlapping objects. Thus, the rollback conflict value corresponding to the current multimodal clinical evidence fragment in the original direction support is obtained. When the rollback conflict value is greater than zero, partial rollback is performed to remove it. When the rollback conflict value is equal to zero, the corresponding content is retained.

[0118] S3.8: The original directional support after local rollback is superimposed with the cross-boundary incremental directional support, and the superimposed directional support is sequentially spliced ​​according to the time axis of medical treatment. At the same time, the corresponding candidate diseases, multimodal clinical evidence fragments, evidence directional relationships and medical treatment time positions are associated to form a rollback differential evidence sequence.

[0119] Furthermore, when superimposing the original directional support after local rollback with the cross-boundary incremental directional support, the original directional support after local rollback and the cross-boundary incremental directional support are first grouped under the same candidate disease according to the candidate disease; then, the multimodal clinical evidence fragments in the original directional support after local rollback and the multimodal clinical evidence fragments in the cross-boundary incremental directional support are spliced ​​sequentially according to the time axis of medical treatment.

[0120] For multimodal clinical evidence fragments with the same time and location of visit, the evidence direction relationship that was removed before the local rollback is covered according to the evidence direction relationship corresponding to the cross-boundary incremental direction support, and the evidence direction relationship that does not overlap or contradict each other in the original direction support after the local rollback is retained.

[0121] The spliced ​​multimodal clinical evidence fragments, evidence direction relationships, consultation time and location, and corresponding candidate diseases are associated, and the multimodal clinical evidence fragments are marked as originating from the original direction support after local rollback or the cross-boundary incremental direction support, forming a rollback differential evidence sequence.

[0122] S4: Based on the rollback differential evidence sequence, the candidate disease evidence map is partially rewritten, the candidate disease reasoning path before and after the partial rewriting is subjected to evidence differential docking, the version differential reversible mapping verification is performed according to the path correspondence before and after the evidence differential docking, the path differences that pass the version differential reversible mapping verification are arranged into collaborative version differences, and the collaborative version differences are incorporated into the diagnosis and treatment auxiliary collaborative record.

[0123] S4.1: Based on the candidate diseases, evidence direction relationships, and consultation time and location in the rollback differential evidence sequence, locate the local rewriting range in the candidate disease evidence graph. Delete, write, and sequentially reconnect the evidence vertices and evidence direction edges within the local rewriting range according to the rollback differential evidence sequence to form the locally rewritten candidate disease evidence graph.

[0124] Furthermore, when locally rewriting the candidate disease evidence map based on the rollback differential evidence sequence, the candidate disease, evidence direction relationship, consultation time and location, multimodal clinical evidence fragments, original direction support after local rollback, and cross-boundary incremental direction support are first extracted from the rollback differential evidence sequence.

[0125] Locate the evidence map position corresponding to the candidate disease in the candidate disease evidence map, and search for evidence vertices and evidence direction edges with the same consultation time position along the evidence map position corresponding to the candidate disease; include the parts of the retrieved evidence vertices and evidence direction edges that have overlapping consultation time positions with the original direction support and cross-boundary incremental direction support after local rollback in the local rewrite range.

[0126] After the local rewriting scope is located, the evidence vertices and evidence direction edges that were removed by the local rollback within the local rewriting scope are deleted. The multimodal clinical evidence fragments corresponding to the cross-boundary incremental direction support are written as new evidence vertices, and the new evidence direction edges are written according to the evidence direction relationship corresponding to the cross-boundary incremental direction support. Then, the evidence vertices retained within the local rewriting scope and the newly written evidence vertices are arranged in order according to the consultation time position in the rollback differential evidence sequence, and the evidence direction edges are reconnected according to the arrangement result to form the candidate disease evidence map after local rewriting.

[0127] S4.2: Organize the candidate disease reasoning path of the locally rewritten candidate disease evidence graph, reconnect the evidence vertices according to the candidate disease, the direction category of the evidence direction edge, and the location of the consultation time, and form the locally rewritten candidate disease reasoning path. Then, align the path positions of the candidate disease reasoning paths before and after the local rewriting, and perform evidence differential docking on the corresponding multimodal clinical evidence fragments, evidence direction relationships, and consultation time locations in the candidate disease reasoning paths before and after the local rewriting to form evidence differential docking data.

[0128] Furthermore, when organizing the candidate disease reasoning path in the locally rewritten candidate disease evidence map, first extract the evidence vertices and evidence direction edges in the locally rewritten candidate disease evidence map according to the candidate diseases; then sort the evidence vertices according to the consultation time position corresponding to the evidence vertices, and connect adjacent evidence vertices according to the direction category of the evidence direction edges.

[0129] When the direction category of adjacent evidence direction edges switches, the segment boundary is preserved at the switching position, forming a locally rewritten candidate disease reasoning path. Then, the candidate disease reasoning paths before and after the local rewriting are aligned in terms of path position. First, the candidate disease reasoning paths before and after the local rewriting are aligned according to the candidate disease, and then the evidence vertices before and after the local rewriting are aligned according to the location of the consultation time. For evidence vertices with the same consultation time location, the corresponding multimodal clinical evidence fragments and evidence direction relationships are compared.

[0130] For locations where the consultation time and location are inserted, deleted, or reconnected, record the changes in path location, evidence vertex, and evidence direction edge before and after the rewriting; perform evidence differential docking on the correspondence and differences between the multimodal clinical evidence fragments, evidence direction relationships, and consultation time and location before and after the local rewriting, to form evidence differential docking data.

[0131] S4.3: Perform version differential reversible mapping verification according to the path correspondence in the evidence differential docking data. Reverse map the candidate disease reasoning path after local rewriting to the candidate disease reasoning path before local rewriting according to the candidate disease, the location of the consultation time, the evidence vertex and the evidence direction edge. Verify the correspondence between the multimodal clinical evidence fragments, the evidence direction relationship and the location of the consultation time after reverse mapping.

[0132] Furthermore, when performing version differential reversible mapping verification based on the path correspondence in the evidence differential docking data, the following steps are taken: first extract the candidate disease reasoning path after local rewriting, the candidate disease reasoning path before local rewriting, path position changes, evidence vertex changes, and evidence direction edge changes from the evidence differential docking data.

[0133] Using the candidate disease as the starting point for mapping, the evidence vertices in the locally rewritten candidate disease reasoning path are mapped back to the candidate disease reasoning path before the local rewriting according to the location of the consultation time; for evidence vertices that can be found in the candidate disease reasoning path before the local rewriting with the same consultation time location, the correspondence between the multimodal clinical evidence fragments, evidence direction relationships and evidence direction edges after the reverse mapping is verified.

[0134] For evidence vertices originating from cross-boundary incremental direction support, the back-insertion time position and evidence direction relationship in the rollback differential evidence sequence are used to verify whether the content written within the local rewrite range corresponds before and after the reverse mapping; for evidence vertices originating from the original direction support after local rollback, the content retained within the local rewrite range is verified before and after the reverse mapping; through the above reverse mapping and correspondence verification, the version differential reversible mapping verification is completed.

[0135] It should be noted that when the evidence vertices, evidence direction edges, multimodal clinical evidence fragments, evidence direction relationships, and consultation time positions in the candidate disease reasoning path after local rewriting can respectively correspond to the candidate disease reasoning path before local rewriting, the written content within the scope of local rewriting, and the content retained within the scope of local rewriting, the corresponding path differences will be written as verification through version difference reversible mapping.

[0136] S4.4: Extract the path differences verified by the version difference reversible mapping, and arrange the path differences according to the candidate disease, the relationship of evidence direction, the time and location of the visit and the scope of local rewriting to form a collaborative version difference;

[0137] Furthermore, the path position changes, evidence vertex changes, evidence direction edge changes, and evidence direction relationship changes that have passed the version differential reversible mapping verification are read from the evidence differential docking data; then the path differences are collected according to the candidate diseases, and the path differences are arranged in order according to the time and location of the consultation.

[0138] For the deleted evidence vertices and evidence direction edges within the local rewriting range, record the corresponding multimodal clinical evidence fragments, evidence direction relationships, and consultation time and location; for the written evidence vertices and evidence direction edges within the local rewriting range, record the corresponding multimodal clinical evidence fragments, evidence direction relationships, and consultation time and location; for the sequentially reconnected evidence direction edges within the local rewriting range, record the connection position and evidence direction relationship before and after reconnection; arrange the above path differences according to candidate diseases, evidence direction relationships, consultation time and location, and local rewriting range to form a collaborative version difference.

[0139] S4.5: Incorporate the collaborative version difference into the diagnostic and treatment assistance collaborative record, and associate the rewriting position of the candidate disease reasoning path, the difference in evidence before and after rewriting, and the reversible mapping verification content of the version difference with the diagnostic and treatment assistance collaborative record;

[0140] Furthermore, when incorporating the collaborative version difference into the diagnostic and treatment assistance collaborative record, the candidate disease, local rewrite range, and consultation time and location corresponding to the collaborative version difference are first written into the diagnostic and treatment assistance collaborative record.

[0141] The rewritten position of the candidate disease reasoning path, the difference in evidence before and after rewriting, the deleted evidence vertices and evidence direction edges, the written evidence vertices and evidence direction edges, the evidence direction edges of sequential reconnection, and the version difference reversible mapping verification content are respectively associated with the collaborative version difference.

[0142] Based on the location of the visit and the path order of the candidate disease reasoning path, the collaborative version difference is incorporated into the diagnosis and treatment assistance collaborative record, so that the diagnosis and treatment assistance collaborative record can save the path differences, evidence direction relationship differences and version difference reversible mapping verification content of the candidate disease reasoning path before and after local rewriting.

[0143] To verify the local incremental update capability and version differential tracing capability of this invention for asynchronously accessing multimodal clinical evidence fragments, an event log test dataset was constructed. Comparative tests were conducted on conventional full reconstruction methods, sequential time appending methods, time back-insertion without boundaries methods, and the method of this invention. During the testing process, asynchronous evidence insertion events were configured around factors such as the number of evidence fragments, the number of candidate diseases, the length of the inference path, the evidence dependency density, the evidence direction conflict rate, the relative position of back-insertion, the number of direction switches, and the proportion of missing attributes. The update time, local rewrite ratio, evidence direction consistency maintenance rate, and reversible mapping verification pass rate of different methods under each asynchronous evidence insertion event were recorded.

[0144] See Figure 3 , Figure 3 Test results of update time for each method under different complexity quantiles are presented. As the complexity quantile increases, the update time of each method generally shows an upward trend, indicating that the burden of path rewriting increases synchronously with the increase in the evidence size, inference path length, and dependency complexity corresponding to asynchronous evidence insertion events. Compared to conventional full reconstruction methods, the method of this invention exhibits lower update time under all complexity quantiles. Compared to the time-interpolation-without-boundary method, the method of this invention limits the rewriting range by using a consistent reachable boundary and performs a local rollback on the original direction support, thus reducing the rewriting overhead of irrelevant evidence direction edges while maintaining the interpolation processing capability.

[0145] See Figure 4 , Figure 4 Test results of the pass rate distribution for reversible mapping verification using different methods are presented. Figure 4As can be seen, the reversible mapping verification pass rate distribution corresponding to the method of the present invention is generally in a high range, indicating that the method of the present invention can maintain the correspondence between the candidate disease inference paths before and after local rewriting relatively stably after asynchronous evidence supplementation. Compared with the conventional full reconstruction method, although the conventional full reconstruction method can complete path reconstruction, the stable correspondence between the paths before and after rewriting is weak due to the large rewriting range. Compared with the receiving time sequence appending method and the time back-insertion without boundary method, the method of the present invention performs local rewriting of the candidate disease evidence map by rolling back the differential evidence sequence and constrains the path differences by combining version differential reversible mapping verification, thus showing higher stability in version differential tracing.

[0146] Figure 3 This demonstrates that the present invention can reduce update time in asynchronous evidence supplementation scenarios with varying degrees of complexity. Figure 4 This indicates that the present invention has a more stable distribution performance in terms of the pass rate of reversible mapping verification. Therefore, the present invention can realize the local rewriting of candidate disease evidence maps and the stable tracing of path differences before and after rewriting under the condition of asynchronously supplementing multimodal clinical evidence.

[0147] In summary, this invention improves the locality and traceability of evidence path rewriting by: interpolating the consultation time of asynchronously accessed multimodal clinical evidence fragments, forming cross-boundary incremental directional support along the dependency index of diagnostic and treatment auxiliary evidence, and locally rolling back the original directional support within the reachable boundary with consistent direction, so that the rolled-back differential evidence sequence can be used for local rewriting of candidate disease evidence graphs; and then arranging the path differences into collaborative version differences and incorporating them into the diagnostic and treatment auxiliary collaborative record through version differential reversible mapping verification, which is used for diagnostic and treatment auxiliary collaborative updates after asynchronous evidence is added.

[0148] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A collaborative diagnostic and treatment assistance method based on multimodal AI, characterized in that, include: Time registration and semantic normalization are performed on multimodal clinical diagnosis and treatment data, multimodal clinical evidence fragments are extracted and arranged into the same time axis according to the time of consultation to form a multimodal clinical event sequence; Disease entities are normalized and evidence direction is labeled around multimodal clinical event sequences. Candidate disease evidence graphs are constructed according to the evidence direction relationship between disease entities and multimodal clinical evidence fragments. Evidence dependency labels are applied to the reasoning paths of candidate diseases to form a diagnostic and treatment auxiliary evidence dependency index. The process involves back-interpolating the visit time of asynchronously accessed multimodal clinical evidence fragments, retrieving the inference path of associated candidate diseases along the diagnostic and treatment auxiliary evidence dependency index, extracting multimodal clinical evidence fragments with earlier visit times from the inference path of associated candidate diseases, converging them into a back-interpolation precursor evidence set, extracting the back-interpolation direction support base around the back-interpolation precursor evidence set, advancing the direction support along the evidence dependency direction, and converging at the position where the evidence directions are contradictory into a direction-consistent reachable boundary to form cross-boundary incremental direction support, performing local rollback on the original direction support within the direction-consistent reachable boundary, superimposing the original direction support after local rollback with the cross-boundary incremental direction support, and splicing them along the visit time axis to form a rollback differential evidence sequence; Based on the rollback differential evidence sequence, the candidate disease evidence map is partially rewritten. The candidate disease reasoning paths before and after the partial rewriting are subjected to evidence differential docking. The version differential reversible mapping verification is performed according to the path correspondence before and after the evidence differential docking. The path differences that pass the version differential reversible mapping verification are arranged into collaborative version differences. The collaborative version differences are incorporated into the diagnostic and treatment auxiliary collaborative record.

2. The diagnostic and treatment assistance collaborative method based on multimodal AI as described in claim 1, characterized in that, The temporal registration and semantic normalization of multimodal clinical diagnosis and treatment data are performed as follows: Collect multimodal clinical diagnosis and treatment data, verify the visit identification, merge records from different sources corresponding to the same visit process, and then unify the time field format and time sequence of the merged records from different sources to form time-registered clinical data. Clinical concept standardization coding is performed on time-registered clinical data. Clinical expressions with the same semantics are merged into the same clinical concept. The attribution relationship between negation polarity and time attribute and clinical concept is corrected according to the dependency relationship in the sentence containing the clinical concept, so as to form semantically unified clinical data.

3. The diagnostic and treatment assistance collaborative method based on multimodal AI as described in claim 2, characterized in that, The formation of the multimodal clinical event sequence is as follows: Evidence content is extracted from semantically normalized clinical data, and clinical evidence content carrying clinical concepts, time attributes, source time and source location is extracted. Then, it is fragmented and organized according to the source modality corresponding to the clinical evidence content to form multimodal clinical evidence fragments. The consultation time position is calibrated according to the time attribute and source time of the multimodal clinical evidence fragments. Multimodal clinical evidence fragments under the same consultation time position are grouped into clinical events, and the clinical events are arranged in chronological order of consultation time to form a multimodal clinical event sequence.

4. The diagnostic and treatment assistance collaborative method based on multimodal AI as described in claim 1, characterized in that, The specific steps for disease entity normalization and evidence orientation annotation around multimodal clinical event sequences are as follows: According to the order of clinical events in the multimodal clinical event sequence, the clinical concepts carried by the multimodal clinical evidence fragments in each clinical event are screened for disease entities. The screened disease entities are coded and aligned with standard disease terms, and the disease entities are merged according to the coding consistency relationship and synonym relationship. At the same time, the clinical event position corresponding to the merged disease entities is retained to form disease entity normalized data. Based on the clinical event location in the disease entity normalization data, the multimodal clinical event sequence is traced back. Within the corresponding clinical event, multimodal clinical evidence fragments carrying the corresponding clinical concepts of the normalized disease entity are located. Using the located multimodal clinical evidence fragments as anchors, multimodal clinical evidence fragments that participate in the negative polarity attribution correction and time attribute attribution correction within the same clinical event are extracted according to their source location. The located multimodal clinical evidence fragments and the extracted multimodal clinical evidence fragments are then correlated with the normalized disease entity, and the consultation time location corresponding to the multimodal clinical evidence fragments is retained to form disease evidence correspondence data. Evidence direction labeling is performed on the data corresponding to disease evidence. First, the temporal attribute attribution carried by multimodal clinical evidence fragments is used to label the relationship between the unified disease entity and multimodal clinical evidence fragments with incomplete temporal attribute attribution as the direction to be reviewed. Then, the negative polarity attribution carried by multimodal clinical evidence fragments is used to label the relationship between the unified disease entity and multimodal clinical evidence fragments with complete temporal attribute attribution and not modified by negative polarity as the direction of support. The relationship between the unified disease entity and multimodal clinical evidence fragments with complete temporal attribute attribution and modified by negative polarity is labeled as the direction of exclusion, thus forming evidence direction labeling data.

5. The diagnostic and treatment assistance collaborative method based on multimodal AI as described in claim 4, characterized in that, The formation of the diagnostic and treatment support evidence dependency index is as follows: Based on the evidence direction annotation data, a candidate disease evidence graph is constructed. Candidate diseases are formed by normalized disease entities in the disease entity normalization data, and evidence vertices are formed by multimodal clinical evidence fragments. Evidence direction edges are established between candidate diseases and evidence vertices according to the evidence direction relationship, and the consultation time position corresponding to the multimodal clinical evidence fragments is retained to form a candidate disease evidence graph. The candidate disease evidence graph is organized into a candidate disease reasoning path. The evidence vertices are connected according to the direction category and the location of the consultation time under the same candidate disease. Segmentation boundaries are set at the locations where the direction category of the evidence direction edge changes, thus forming the candidate disease reasoning path. Evidence dependency annotation is performed on the reasoning path of candidate diseases, and the correspondence between the reasoning path of candidate diseases and multimodal clinical evidence fragments, evidence direction relationships, consultation time and location and segment boundaries is recorded to form a diagnostic and treatment auxiliary evidence dependency index.

6. The diagnostic and treatment assistance collaborative method based on multimodal AI as described in claim 1, characterized in that, The process of back-interpolating the visit time of asynchronously accessed multimodal clinical evidence fragments and retrieving the inference path of associated candidate diseases along the dependency index of diagnostic and treatment auxiliary evidence is as follows: The appointment time of asynchronously accessed multimodal clinical evidence fragments is calibrated, and the time attributes and source time of the asynchronously accessed multimodal clinical evidence fragments are aligned with the appointment time axis in the multimodal clinical event sequence to form multimodal clinical evidence fragments with back-insertion time positions. Following the diagnostic and treatment auxiliary evidence dependency index, candidate disease reasoning paths are retrieved based on the clinical concepts and interpolation time positions carried by multimodal clinical evidence fragments with interpolation time positions. According to the annotation method in the evidence direction annotation data, evidence direction annotation is performed between the multimodal clinical evidence fragments with interpolation time positions and the candidate diseases corresponding to the retrieved candidate disease reasoning paths, forming multimodal clinical evidence fragments with interpolation time positions and evidence direction relationships. The retrieved candidate disease reasoning paths are then used as the associated candidate disease reasoning paths.

7. The diagnostic and treatment assistance collaborative method based on multimodal AI as described in claim 6, characterized in that, The formation of cross-boundary incremental directional support is as follows: Extract multimodal clinical evidence fragments that precede the time of visit relative to the insertion time from the inference path of associated candidate diseases, and retain the dependency order between the multimodal clinical evidence fragments according to the evidence dependency annotation in the inference path of associated candidate diseases, and aggregate them into an insertion precursor evidence set; Evidence direction aggregation is performed on the pre-insertion evidence set. Multimodal clinical evidence fragments with the same evidence direction relationship as those with the inference path of the candidate disease and which depend on the sequentially continuous multimodal clinical evidence fragments with the inference time position and evidence direction relationship of the inversion are extracted to form the inversion direction support base. Starting from the base of the back-insertion direction, the direction of directional support is advanced along the evidence-dependent direction in the inference path of the associated candidate disease. Positions with continuous and consistent evidence directions are included in the advancement range, and advancement is stopped at positions with contradictory evidence directions, thus converging to form a directionally consistent reachable boundary. Multimodal clinical evidence fragments with interpolation time positions and evidence direction relationships are inserted into the interpolation time positions in the inference path of associated candidate diseases. Continuous evidence direction edges extending to the reachable boundary with consistent direction are extracted from the interpolation time positions along the inference path of associated candidate diseases. The continuous evidence direction edges and the corresponding multimodal clinical evidence fragments are then organized into cross-boundary incremental direction support.

8. The diagnostic and treatment assistance collaborative method based on multimodal AI as described in claim 7, characterized in that, The original directional support within the reachable boundary with consistent direction is partially rolled back. The rolled-back original directional support is then superimposed with the cross-boundary incremental directional support, and spliced ​​along the medical visit time axis to form a rollback differential evidence sequence, as detailed below: Extract the existing continuous evidence direction edges and corresponding multimodal clinical evidence fragments within the reachable boundary of the associated candidate disease reasoning path as the original direction support. Perform local rollback on the original direction support, remove the original direction support parts that overlap with the cross-boundary incremental direction support in terms of consultation time and location and contradict each other in terms of evidence direction, and retain the original direction support parts that do not overlap or contradict each other to form the original direction support after local rollback. The original directional support after local rollback is superimposed with the cross-boundary incremental directional support, and the superimposed directional support is sequentially spliced ​​according to the time axis of medical visits. At the same time, the corresponding candidate diseases, multimodal clinical evidence fragments, evidence directional relationships and medical visit time locations are associated to form a rollback differential evidence sequence.

9. The diagnostic and treatment assistance collaborative method based on multimodal AI as described in claim 1, characterized in that, The process involves partially rewriting the candidate disease evidence graph based on the rollback differential evidence sequence, and then performing evidence differential docking on the candidate disease inference paths before and after the partial rewriting, as detailed below: Based on the candidate diseases, evidence direction relationships, and consultation time and location in the rollback differential evidence sequence, the local rewriting range in the candidate disease evidence map is located. The evidence vertices and evidence direction edges within the local rewriting range are deleted, written, and sequentially reconnected according to the rollback differential evidence sequence to form the locally rewritten candidate disease evidence map. The candidate disease evidence graph after partial rewriting is organized into candidate disease inference paths. Evidence vertices are reconnected according to the candidate disease, the direction category of the evidence direction edge, and the location of the consultation time to form the candidate disease inference path after partial rewriting. Then, the path positions of the candidate disease inference paths before and after partial rewriting are aligned. The corresponding multimodal clinical evidence fragments, evidence direction relationships, and consultation time locations in the candidate disease inference paths before and after partial rewriting are subjected to evidence differential docking to form evidence differential docking data.

10. The diagnostic and treatment assistance collaborative method based on multimodal AI as described in claim 9, characterized in that, The path differences verified by the version difference reversible mapping are arranged into collaborative version differences, and the collaborative version differences are incorporated into the diagnostic and treatment auxiliary collaborative record, as follows: According to the path correspondence in the evidence differential docking data, the version differential reversible mapping verification is performed. The candidate disease reasoning path after local rewriting is reverse mapped to the candidate disease reasoning path before local rewriting according to the candidate disease, the location of the consultation time, the evidence vertex and the evidence direction edge. The correspondence between the multimodal clinical evidence fragments, the evidence direction relationship and the location of the consultation time after the reverse mapping is verified. Extract the path differences verified by the version difference reversible mapping, and arrange the path differences according to candidate diseases, evidence direction relationship, consultation time and location, and local rewriting scope to form a collaborative version difference; The collaborative version difference is incorporated into the diagnostic and treatment assistance collaborative record, and the rewriting position of the candidate disease reasoning path, the difference in evidence before and after rewriting, and the reversible mapping verification content of the version difference are associated with the diagnostic and treatment assistance collaborative record.

Images