Intelligent medical illusion detection method and system based on multi-modal data enhancement

Abstract

The invention provides an intelligent medical illusion detection method and system based on multi-modal data enhancement, and relates to the technical field of medical illusion detection. The method comprises the steps of collecting multi-modal medical data, and constructing an original multi-modal data set; performing data preprocessing on the original multi-modal data set, and generating a plurality of enhanced modal samples through semantic alignment and information fusion among modals by adopting a cross-modal data enhancement technology; modal feature vectors of the modal samples are extracted; according to the modal feature vector, determining a multi-modal consistency index; fusing the modal feature vectors, and determining an abnormal score; obtaining a semantic distance between the modal sample and a knowledge graph; determining illusion confidence; and according to the illusion confidence, judging whether medical illusion appears or not. According to the method and the device, the illusion content in the medical data can be detected according to the consistency of the multi-modal data, the abnormal score and the semantic distance between the multi-modal data and the knowledge graph, and the accuracy and the reliability of the medical data are improved.

Description

Technical Field

[0001] This invention relates to the field of medical hallucination detection technology, and in particular to an intelligent medical hallucination detection method and system based on multimodal data augmentation. Background Technology

[0002] While current technologies improve anomaly detection accuracy by integrating multimodal data and employing various techniques, they do not detect hallucinations that may exist in multimodal medical data. In other words, they cannot detect hallucination content in medical data based on the consistency of multimodal data, anomaly scores, and semantic distance from the knowledge graph.

[0003] The information disclosed in the background section of this application is intended only to enhance the understanding of the general background of this application and should not be construed as an admission or in any way implying that the information constitutes prior art known to those skilled in the art. Summary of the Invention

[0004] This invention provides an intelligent medical hallucination detection method and system based on multimodal data augmentation, which can solve the technical problem that related technologies cannot detect hallucination content in medical data based on the consistency, anomaly scoring and semantic distance of multimodal data with knowledge graphs.

[0005] According to a first aspect of the present invention, an intelligent medical hallucination detection method based on multimodal data augmentation is provided, comprising: collecting multimodal medical data to construct an original multimodal dataset, wherein the multimodal medical data includes medical image data, electronic medical record text data, and physiological signal data; performing data preprocessing on the original multimodal dataset and employing cross-modal data augmentation technology to generate multiple enhanced modal samples through semantic alignment and information fusion between modalities; constructing a multimodal feature extraction network to extract modal feature vectors of the modal samples respectively; determining a multimodal consistency index based on the modal feature vectors; fusing the modal feature vectors to determine an anomaly score; obtaining the semantic distance between the modal sample and a knowledge graph; determining a hallucination confidence level based on the multimodal consistency index, the anomaly score, and the semantic distance; and determining whether a medical hallucination has occurred based on the hallucination confidence level.

[0006] Further, determining a multimodal consistency index based on the modal feature vector includes: obtaining the L2 norm of the modal feature vector based on the modal feature vector; and determining the multimodal consistency index based on the modal feature vector and the L2 norm of the modal feature vector.

[0007] Further, based on the modal feature vectors and their L2 norms, a multimodal consistency index is determined, including: according to the formula: Determine the multimodal consistency index C, where, Let i be the modal feature vector of the i-th modal sample. Let j be the modal feature vector of the j-th modal sample. Let L2 norm be the modal eigenvector of the i-th modal sample. Let L2 norm be the modal eigenvector of the j-th modal sample. The modal feature vector of the i-th modal sample is subtracted from the L2 norm of the modal feature vector of the j-th modal sample, where M is the number of modal samples, i≤M-1, j≤M, and i, j, and M are all positive integers.

[0008] Further, fusing the modal feature vectors to determine anomaly scores includes: fusing the modal feature vectors based on different weights to obtain a final feature vector; obtaining the mean and standard deviation of all dimensions of the final feature vector based on the final feature vector; and determining anomaly scores based on the final feature vector, the mean, and the standard deviation.

[0009] Further, based on the final feature vector, the mean, and the standard deviation, an anomaly score is determined, including: according to the formula: Determine the anomaly score A, where, Let k be the k-th eigenvalue of the final eigenvector. The mean of the final feature vector. The standard deviation of the final feature vector. The L2 norm of the final eigenvector is the k-th eigenvalue minus the mean of the final eigenvector, where N is the number of eigendimensionalities of the final eigenvector, k ≤ N, and both k and N are positive integers.

[0010] Further, determining the hallucination confidence level based on the multimodal consistency index, the anomaly score, and the semantic distance includes: obtaining the maximum semantic distance based on the semantic distance; setting an anomaly score threshold; and determining the hallucination confidence level based on the maximum semantic distance, the anomaly score threshold, the multimodal consistency index, the anomaly score, and the semantic distance.

[0011] Further, the hallucination confidence is determined based on the maximum semantic distance, the anomaly scoring threshold, the multimodal consistency index, the anomaly score, and the semantic distance, including: according to the formula: Determine the confidence level Q of the hallucination, where, Let be the semantic distance between the s-th modality sample and the knowledge graph. Where A is the maximum semantic distance and A is the anomaly score. is the anomaly scoring threshold, C is the multimodal consistency index, M is the number of modal samples, s≤M, and both s and M are positive integers, and max is the maximum value function.

[0012] Further, based on the hallucination confidence level, determining whether medical hallucination has occurred includes: if the hallucination confidence level is greater than or equal to the hallucination confidence level threshold, then determining that medical hallucination has occurred; if the hallucination confidence level is less than the hallucination confidence level threshold, then determining that no medical hallucination has occurred.

[0013] According to a second aspect of the present invention, an intelligent medical hallucination detection system based on multimodal data augmentation is provided, comprising: a raw multimodal dataset module for collecting multimodal medical data and constructing a raw multimodal dataset, wherein the multimodal medical data includes medical image data, electronic medical record text data, and physiological signal data; a modal sample module for preprocessing the raw multimodal dataset and generating multiple enhanced modal samples by employing cross-modal data augmentation technology through semantic alignment and information fusion between modalities; a modal feature vector module for constructing a multimodal feature extraction network to extract modal feature vectors of the modal samples respectively; a multimodal consistency index module for determining a multimodal consistency index based on the modal feature vectors; an anomaly scoring module for fusing the modal feature vectors to determine an anomaly score; a semantic distance module for obtaining the semantic distance between the modal samples and a knowledge graph; a hallucination confidence module for determining a hallucination confidence based on the multimodal consistency index, the anomaly score, and the semantic distance; and a judgment module for judging whether a medical hallucination has occurred based on the hallucination confidence.

[0014] Technical Effects: According to this invention, cross-modal data augmentation technology generates multiple enhanced modal samples by utilizing semantic alignment and information fusion between modalities, enriching data diversity and enabling better mining of potential relationships between data. By determining hallucination confidence through multimodal consistency indices, anomaly scores, and semantic distance, the presence of hallucinations in medical data can be determined more comprehensively and accurately, improving detection precision and the accuracy and reliability of medical data. In determining multimodal consistency indices, pairwise comparisons between modal feature vectors are used to determine the indices, reflecting the degree of difference between samples from different modalities and the overall consistency of multimodal data, unaffected by differences in feature dimensions, thus improving robustness to anomalous modalities. In determining anomaly scores, the mean and standard deviation of the final feature vectors are used to determine the anomaly score, more sensitively capturing the asymmetry of feature distribution, effectively identifying progressive anomalies and local feature distortions, thereby achieving effective quantification and detection of anomalous information. When determining the confidence level of hallucination, the confidence level of hallucination can be determined by the maximum semantic distance, the anomaly score threshold, the multimodal consistency index, the anomaly score, and the semantic distance. The probability of hallucination is assessed from two aspects: the similarity between the modal sample and the knowledge graph, and the anomaly and inconsistency of the modal sample itself. This reduces false positives and false negatives caused by over-reliance on a single dimension and improves the accuracy of medical hallucination detection.

[0015] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only, and are not intended to limit the invention. Other features and aspects of the invention will become clearer from the following detailed description of exemplary embodiments with reference to the accompanying drawings. Attached Figure Description

[0016] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other embodiments can be obtained based on these drawings without creative effort. Figure 1 An exemplary flowchart of an intelligent medical hallucination detection method based on multimodal data augmentation according to an embodiment of the present invention is shown. Figure 2 An exemplary flowchart for calculating a multimodal consistency index according to an embodiment of the present invention is shown; Figure 3 An exemplary flowchart illustrating the calculation of anomaly scores according to an embodiment of the present invention is shown; Figure 4 An exemplary flowchart for calculating the confidence level of hallucination according to an embodiment of the present invention is shown; Figure 5An exemplary flowchart for determining medical hallucinations according to an embodiment of the present invention is shown; Figure 6 A block diagram of an intelligent medical hallucination detection system based on multimodal data augmentation according to an embodiment of the present invention is shown as an example. Detailed Implementation

[0017] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0018] The technical solution of the present invention will be described in detail below with reference to specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments.

[0019] Figure 1 An exemplary flowchart of an intelligent medical hallucination detection method based on multimodal data augmentation according to an embodiment of the present invention is shown. The method includes: Step S1, collecting multimodal medical data to construct an original multimodal dataset, wherein the multimodal medical data includes medical image data, electronic medical record text data, and physiological signal data; Step S2, preprocessing the original multimodal dataset and using cross-modal data augmentation technology to generate multiple enhanced modal samples through semantic alignment and information fusion between modalities; Step S3, constructing a multimodal feature extraction network to extract modal feature vectors of the modal samples respectively; Step S4, determining a multimodal consistency index based on the modal feature vectors; Step S5, fusing the modal feature vectors to determine an anomaly score; Step S6, obtaining the semantic distance between the modal sample and the knowledge graph; Step S7, determining the hallucination confidence level based on the multimodal consistency index, the anomaly score, and the semantic distance; Step S8, determining whether medical hallucination has occurred based on the hallucination confidence level.

[0020] The intelligent medical hallucination detection method based on multimodal data augmentation according to embodiments of the present invention generates multiple enhanced modal samples by utilizing cross-modal data augmentation technology and semantic alignment and information fusion between modalities, enriching data diversity and enabling better mining of potential relationships between data. By determining hallucination confidence through multimodal consistency indices, anomaly scores, and semantic distance, the method can more comprehensively and accurately determine whether hallucinations exist in medical data, improving detection accuracy and the accuracy and reliability of medical data.

[0021] According to one embodiment of the present invention, in step S1, multimodal medical data reflects the patient's health status from different perspectives. For example, medical imaging data can intuitively display the internal structure of the body, electronic medical record text data records the patient's medical history and treatment process, and physiological signal data reflects the body's real-time physiological state. The combination of multiple modal data makes the original multimodal dataset more comprehensive and richer, providing more sufficient information for subsequent analysis.

[0022] According to one embodiment of the present invention, in step S2, the original multimodal dataset undergoes data preprocessing, such as removing noisy data, processing missing values, and standardizing the data format. Medical image data may contain artifacts, electronic medical record text data may have spelling errors, non-standard formatting, etc., and physiological signal data may be subject to abnormal fluctuations due to external interference. These noisy data are identified and removed. For continuous physiological signal data, linear interpolation can be used to estimate missing values ​​based on signal values ​​at adjacent time points. Medical image data may have different file formats (such as DICOM, JPEG, etc.), and electronic medical record text data may have different encoding methods (such as UTF-8, GBK, etc.). All data is converted to a unified format for subsequent processing and analysis. For example, medical image data can be uniformly converted to DICOM format, ensuring consistent image resolution, color space, and other parameters; for electronic medical record text data, UTF-8 encoding is uniformly used. Cross-modal data augmentation techniques are employed, which involve constructing semantic mapping functions between modalities to establish transformation relationships between different modal feature spaces. For example, image features are mapped to text feature space, text features to physiological signal feature space, and physiological signal features back to image feature space. This is typically implemented using neural networks. An adversarial generative strategy is employed to generate multiple augmented modal samples while maintaining semantic consistency. For example, for each mapping pair (e.g., generating an image from text), a discriminator is assigned to determine whether the input image features are "real" (from a real dataset) or "fake" (generated by the generator). The generator is the mapping function, receiving a text feature and generating an image feature. Through this adversarial training, the generator is forced to continuously improve its generation capabilities until the generated image features are sufficiently realistic to be indistinguishable from the discriminator. Cross-modal cyclic consistency constraints, such as those using inverse mapping, ensure semantic coherence of generated samples across different modalities, i.e., semantic alignment. For example, a lung CT image (modal 1) and an electronic medical record text describing "lung nodules" (modal 2) are semantically equivalent. Information fusion refers to combining information from different modalities to create new, artificially synthesized data pairs, i.e., enhanced modal samples (e.g., image-text pairs, signal-text pairs).

[0023] According to one embodiment of the present invention, in step S3, a multimodal feature extraction network is constructed to extract high-dimensional features of multiple enhanced modal samples, and feature interaction and alignment are performed through a cross-modal attention mechanism, such as an image sub-network (convolutional neural network), a text sub-network (recurrent neural network), and a signal sub-network (temporal convolutional network). To fully utilize the complementary information between different modal data, an intermodal interaction mechanism needs to be designed in the multimodal feature extraction network, such as a cross-modal attention mechanism. This mechanism allows the model to "focus" on the most relevant part of another modality when processing information from one modality, and maps them to a shared subspace. For example, when processing medical images and electronic medical record text, the cross-modal attention mechanism can focus on the corresponding region in the image based on the symptom description in the text (i.e., calculate attention weights), projecting the original features of different enhanced modal samples through their respective linear transformation layers into a common subspace of the same dimension. In this space, similar semantic concepts (e.g., "pulmonary nodules," "cardiac hypertrophy") are mapped to nearby vector positions. Extract the modal feature vector of the modal sample, which is an abstract and condensed representation of the modal sample.

[0024] According to one embodiment of the present invention, in step S4, a multimodal consistency index is determined based on the modal feature vector.

[0025] Figure 2 A flowchart for calculating a multimodal consistency index according to an embodiment of the present invention is shown as an example.

[0026] According to an embodiment of the present invention, step S4 includes: step S41, obtaining the L2 norm of the modal feature vector based on the modal feature vector; step S42, determining a multimodal consistency index based on the modal feature vector and the L2 norm of the modal feature vector.

[0027] According to one embodiment of the invention, the L2 norm of the modal feature vector is the square root of the sum of the squares of the elements of the modal feature vector. A multimodal consistency index is determined based on the modal feature vector and its L2 norm. If the image and text describe the same thing (e.g., the image shows a nodule, and the text report also describes a nodule), after alignment using enhanced modal sample mapping, the modal feature vectors of the two enhanced modal samples will be very close in the common subspace, and the norm of the difference vector between the two enhanced modal feature vectors will be smaller. If an illusion occurs (e.g., the image is normal, but the text report incorrectly describes a tumor), even after alignment using enhanced modal sample mapping, the modal feature vectors of the two enhanced modal samples will still be far apart in the common subspace, and the norm of the difference vector between the two enhanced modal feature vectors will be larger.

[0028] According to one embodiment of the present invention, determining a multimodal consistency index based on the modal feature vector and the L2 norm of the modal feature vector includes: determining the multimodal consistency index C according to formula (1). (1), in, Let i be the modal feature vector of the i-th modal sample. Let j be the modal feature vector of the j-th modal sample. Let L2 norm be the modal eigenvector of the i-th modal sample. Let L2 norm be the modal eigenvector of the j-th modal sample. The modal feature vector of the i-th modal sample is subtracted from the L2 norm of the modal feature vector of the j-th modal sample, where M is the number of modal samples, i≤M-1, j≤M, and i, j, and M are all positive integers.

[0029] According to an embodiment of the present invention, in formula (1), The L2 norm of the modal feature vector of the i-th modal sample is subtracted from the L2 norm of the modal feature vector of the j-th modal sample. It represents the degree of numerical difference between the two modal feature vectors. The larger the L2 norm, the greater the difference between the two modal feature vectors, and the greater the possibility of medical hallucination. The sum of the L2 norms of the modal feature vectors of the i-th modal sample and the j-th modal sample is used to normalize the difference between the two modal feature vectors, making the differences between different modal sample pairs comparable. The relative difference reflects the proportion of the difference between the modal feature vectors of the i-th and j-th modal samples relative to their own sizes. Normalization eliminates the influence of the scale of the modal feature vectors themselves, making the differences between different modal sample pairs more fair and reasonable. There are a total of For modal samples, the average of the sum of the relative differences among all modal sample pairs reflects the average level of differences among all modal sample pairs. Subtracting the average relative difference from 1 yields the multimodal consistency index. The higher the multimodal consistency index, the smaller the differences between different modal samples, the more consistent the information among the modal samples, and the lower the likelihood of medical hallucinations.

[0030] In this way, a multimodal consistency index can be determined by pairwise comparison between modal feature vectors. This index reflects the degree of difference between samples of different modalities and the overall consistency of multimodal data. It is not affected by differences in feature dimensions and improves robustness to anomalous modalities.

[0031] According to one embodiment of the present invention, in step S5, the modal feature vectors are fused to determine an anomaly score.

[0032] Figure 3 A flowchart for calculating anomaly scores according to an embodiment of the present invention is shown as an example.

[0033] According to an embodiment of the present invention, step S5 includes: step S51, fusing the modal feature vectors based on different weights to obtain a final feature vector; step S52, obtaining the mean and standard deviation of all dimensions of the final feature vector based on the final feature vector; and step S53, determining an anomaly score based on the final feature vector, the mean, and the standard deviation.

[0034] According to one embodiment of the present invention, appropriate weights are assigned to the features of each modality, and after weighted summation and fusion, a final feature vector is obtained. For example, the modal feature vector of the enhanced modal sample generated from the image modality is assigned a weight of 3, the modal feature vector of the enhanced modal sample generated from the text modality is assigned a weight of 1, and the modal feature vector of the enhanced modal sample generated from the signal modality is assigned a weight of 2. The final feature vector is treated as a statistical sample, and the mean and standard deviation of all its dimensions are calculated. Anomaly scoring quantifies the degree of deviation of each dimension in the final feature vector.

[0035] According to one embodiment of the present invention, determining an anomaly score based on the final feature vector, the mean, and the standard deviation includes: determining an anomaly score A according to formula (2). (2), in, Let k be the k-th eigenvalue of the final eigenvector. The mean of the final feature vector. The standard deviation of the final feature vector. The L2 norm of the final eigenvector is the k-th eigenvalue minus the mean of the final eigenvector, where N is the number of eigendimensionalities of the final eigenvector, k ≤ N, and both k and N are positive integers.

[0036] According to an embodiment of the present invention, in formula (2), The L2 norm of the final eigenvector is the k-th eigenvalue minus the mean of the final eigenvector, representing the degree to which the k-th eigenvalue deviates from the mean. The relative deviation value is the ratio between the degree to which the k-th eigenvalue deviates from the mean and the standard deviation of the final eigenvector, ensuring comparability of deviations across different dimensions. Cube the relative deviation values ​​for all dimensions. Cube amplifies values ​​with larger deviations while shrinking those with smaller deviations. For eigendimensional dimensions with larger deviations from the mean, their corresponding values ​​will have a more significant impact on anomaly scoring. The average of the cubed values ​​for all dimensions yields the anomaly score. A higher anomaly score indicates that more eigenvalues ​​in the final eigenvector deviate significantly from the mean, meaning the data exhibits characteristics different from normal across multiple dimensions, thus increasing the likelihood of medical hallucinations.

[0037] In this way, anomaly scores can be determined by the mean and standard deviation of the final feature vector. This approach can more sensitively capture the asymmetry of feature distribution, effectively identify progressive anomalies and local feature distortions, thereby achieving effective quantification and detection of anomalous information.

[0038] According to one embodiment of the present invention, in step S6, a knowledge graph can be constructed. The construction process typically includes steps such as determining the knowledge domain and scope, collecting relevant data sources, performing knowledge extraction (extracting entities and relationships from text, databases, etc.), knowledge fusion (integrating knowledge from different sources, resolving conflicts and redundancies), and knowledge storage (selecting an appropriate storage method, such as a graph database). An appropriate mapping method is used to transform the extracted modal sample features into a semantic space consistent with the knowledge graph. Entities and relationships in the knowledge graph are appropriately represented; a common method is to represent entities and relationships as vectors. For example, graph embedding techniques (such as TransE, Node2Vec, etc.) can be used to map nodes (entities) and edges (relationships) in the knowledge graph into low-dimensional vectors, capturing the semantic associations between entities and relationships in the knowledge graph. An appropriate semantic distance metric is selected, such as Euclidean distance. For each modal sample, the semantic distance between its mapped semantic vector and the semantic distance between each entity vector in the knowledge graph is calculated. The minimum distance can be taken as the semantic distance, which is the distance between the modal sample and the nearest entity in the knowledge graph, representing the semantic proximity between the modal sample and the most relevant part of the knowledge graph.

[0039] According to one embodiment of the present invention, in step S7, the hallucination confidence level is determined based on the multimodal consistency index, the anomaly score, and the semantic distance.

[0040] Figure 4 A flowchart for calculating the confidence level of illusion is shown as an example according to an embodiment of the present invention.

[0041] According to an embodiment of the present invention, step S7 includes: step S71, obtaining the maximum semantic distance based on the semantic distance; step S72, setting an anomaly scoring threshold; step S73, determining the hallucination confidence level based on the maximum semantic distance, the anomaly scoring threshold, the multimodal consistency index, the anomaly score, and the semantic distance.

[0042] According to one embodiment of the present invention, the maximum value among multiple semantic distances is taken as the maximum semantic distance, reflecting the semantic gap between the modal sample and the most similar entity in the knowledge graph. An anomaly scoring threshold is set, for example, 4.3. Based on the maximum semantic distance, the anomaly scoring threshold, the multimodal consistency index, the anomaly score, and the semantic distance, the hallucination confidence is determined, medical hallucinations are identified, the understanding of multimodal information is corrected in a timely manner, and the accuracy of information is improved.

[0043] According to one embodiment of the present invention, determining the hallucination confidence level based on the maximum semantic distance, the anomaly scoring threshold, the multimodal consistency index, the anomaly score, and the semantic distance includes: determining the hallucination confidence level Q according to formula (3). (3), in, Let be the semantic distance between the s-th modality sample and the knowledge graph. Where A is the maximum semantic distance and A is the anomaly score. is the anomaly scoring threshold, C is the multimodal consistency index, M is the number of modal samples, s≤M, and both s and M are positive integers, and max is the maximum value function.

[0044] According to one embodiment of the present invention, in formula (3), The ratio of the semantic distance between the s-th modal sample and the knowledge graph to the maximum semantic distance is the larger the ratio. The larger the ratio, the lower the semantic proximity between the modal sample and the most relevant part of the knowledge graph. In other words, the further the modal sample deviates from reality, the greater the possibility of medical hallucination. This is the ratio between the abnormal score and the abnormal score threshold. The larger the ratio, the greater the possibility of abnormalities and distortions in local features, and the greater the likelihood of medical hallucinations. The value is 1 minus the multimodal consistency index, which represents the inconsistency of multimodalities. The larger the value, the greater the inconsistency of multimodalities and the greater the possibility of medical hallucinations. Indicates taking and The maximum of the two values ​​represents the likelihood of experiencing medical hallucinations. for and The product of and represents an index of the probability of hallucination in the s-th modality sample. The larger the index, the higher the probability that the modality sample is experiencing medical hallucination. Taking the square root of this index has a smoothing and adjustment effect. Then, the average of the square roots of the indices for all modalities is taken to obtain the hallucination confidence score. The larger the hallucination confidence score, the higher the probability of hallucination in the entire multimodal dataset.

[0045] In this way, the confidence level of hallucination can be determined by the maximum semantic distance, the anomaly scoring threshold, the multimodal consistency index, the anomaly score, and the semantic distance. The probability of hallucination is assessed from two aspects: the similarity between the modal sample and the knowledge graph, and the anomaly and inconsistency of the modal sample itself. This reduces false positives and false negatives caused by over-reliance on a single dimension and improves the accuracy of medical hallucination detection.

[0046] According to one embodiment of the present invention, in step S8, it is determined whether medical hallucination has occurred based on the hallucination confidence level.

[0047] Figure 5 An exemplary flowchart for determining medical hallucinations according to an embodiment of the present invention is shown.

[0048] According to an embodiment of the present invention, step S8 includes: step S81, if the hallucination confidence level is greater than or equal to the hallucination confidence level threshold, then it is determined that a medical hallucination has occurred; step S82, if the hallucination confidence level is less than the hallucination confidence level threshold, then it is determined that there is no medical hallucination.

[0049] According to one embodiment of the present invention, if the hallucination confidence level is greater than or equal to a hallucination confidence threshold (e.g., 0.8), the unreliability of the information has exceeded an acceptable safety boundary, and it is determined that a medical hallucination has occurred. If the hallucination confidence level is less than the hallucination confidence threshold, it is determined that there is no medical hallucination, that is, the current information has sufficient reliability.

[0050] The intelligent medical hallucination detection method based on multimodal data augmentation according to embodiments of the present invention generates multiple enhanced modal samples by utilizing cross-modal data augmentation technology and semantic alignment and information fusion between modalities, enriching data diversity and enabling better mining of potential relationships between data. Hallucination confidence is determined by multimodal consistency indices, anomaly scores, and semantic distance, allowing for a more comprehensive and accurate assessment of the presence of hallucinations in medical data, improving detection precision, accuracy, and reliability of medical data. When determining multimodal consistency indices, pairwise comparisons between modal feature vectors are used to determine the indices, reflecting the degree of difference between samples from different modalities and the overall consistency of multimodal data, unaffected by differences in feature dimensions, thus improving robustness to anomalous modalities. When determining anomaly scores, the mean and standard deviation of the final feature vectors are used to determine the anomaly scores, more sensitively capturing the asymmetry of feature distribution, effectively identifying progressive anomalies and local feature distortions, thereby achieving effective quantification and detection of anomalous information. When determining the confidence level of hallucination, the confidence level of hallucination can be determined by the maximum semantic distance, the anomaly score threshold, the multimodal consistency index, the anomaly score, and the semantic distance. The probability of hallucination is assessed from two aspects: the similarity between the modal sample and the knowledge graph, and the anomaly and inconsistency of the modal sample itself. This reduces false positives and false negatives caused by over-reliance on a single dimension and improves the accuracy of medical hallucination detection.

[0051] Figure 6 An exemplary block diagram of an intelligent medical hallucination detection system based on multimodal data augmentation according to an embodiment of the present invention is shown. The system includes: a raw multimodal dataset module for collecting multimodal medical data and constructing a raw multimodal dataset, wherein the multimodal medical data includes medical image data, electronic medical record text data, and physiological signal data; a modal sample module for preprocessing the raw multimodal dataset and generating multiple enhanced modal samples by employing cross-modal data augmentation technology through semantic alignment and information fusion between modalities; and a modal feature vector module. This system is used to construct a multimodal feature extraction network to extract modal feature vectors from the modal samples respectively; a multimodal consistency index module is used to determine a multimodal consistency index based on the modal feature vectors; an anomaly scoring module is used to fuse the modal feature vectors to determine an anomaly score; a semantic distance module is used to obtain the semantic distance between the modal sample and the knowledge graph; a hallucination confidence module is used to determine the hallucination confidence based on the multimodal consistency index, the anomaly score, and the semantic distance; and a judgment module is used to determine whether medical hallucination has occurred based on the hallucination confidence.

[0052] This invention can be a method, apparatus, system, and / or computer program product. The computer program product may include a computer-readable storage medium having computer-readable program instructions loaded thereon for performing various aspects of the invention.

[0053] Those skilled in the art should understand that the embodiments of the present invention described above and shown in the accompanying drawings are merely examples and do not limit the present invention. The objectives of the present invention have been fully and effectively achieved. The functions and structural principles of the present invention have been demonstrated and explained in the embodiments, and any variations or modifications may be made to the implementation of the present invention without departing from the stated principles.

Claims

1. A method for detecting intelligent medical hallucinations based on multimodal data augmentation, characterized in that, include: Multimodal medical data is collected to construct an original multimodal dataset, which includes medical image data, electronic medical record text data, and physiological signal data. The original multimodal dataset is preprocessed, and cross-modal data augmentation techniques are employed to generate multiple enhanced modal samples through semantic alignment and information fusion between modalities. A multimodal feature extraction network is constructed to extract modal feature vectors from the modal samples. A multimodal consistency index is determined based on the modal feature vectors. The modal feature vectors are fused to determine anomaly scores. The semantic distance between the modal samples and the knowledge graph is obtained. Hallucination confidence is determined based on the multimodal consistency index, the anomaly score, and the semantic distance. Finally, based on the hallucination confidence, it is determined whether medical hallucination has occurred.

2. The intelligent medical hallucination detection method based on multimodal data augmentation according to claim 1, characterized in that, Determining a multimodal consistency index based on the modal feature vector includes: obtaining the L2 norm of the modal feature vector; and determining the multimodal consistency index based on the modal feature vector and the L2 norm of the modal feature vector.

3. The intelligent medical hallucination detection method based on multimodal data augmentation according to claim 2, characterized in that, Based on the modal feature vectors and their L2 norms, a multimodal consistency index is determined, including: according to the formula: Determine the multimodal consistency index C, where, Let i be the modal feature vector of the i-th modal sample. Let j be the modal feature vector of the j-th modal sample. Let L2 norm be the modal eigenvector of the i-th modal sample. Let L2 norm be the modal eigenvector of the j-th modal sample. The modal feature vector of the i-th modal sample is subtracted from the L2 norm of the modal feature vector of the j-th modal sample, where M is the number of modal samples, i≤M-1, j≤M, and i, j, and M are all positive integers.

4. The intelligent medical hallucination detection method based on multimodal data augmentation according to claim 1, characterized in that, The process of fusing the modal feature vectors to determine anomaly scores includes: fusing the modal feature vectors based on different weights to obtain a final feature vector; obtaining the mean and standard deviation of all dimensions of the final feature vector; and determining anomaly scores based on the final feature vector, the mean, and the standard deviation.

5. The intelligent medical hallucination detection method based on multimodal data augmentation according to claim 4, characterized in that, Anomaly scores are determined based on the final feature vector, the mean, and the standard deviation, including: according to the formula: Determine the anomaly score A, where, Let k be the k-th eigenvalue of the final eigenvector. The mean of the final feature vector. The standard deviation of the final feature vector. The L2 norm of the final eigenvector is the k-th eigenvalue minus the mean of the final eigenvector, where N is the number of eigendimensionalities of the final eigenvector, k ≤ N, and both k and N are positive integers.

6. The intelligent medical hallucination detection method based on multimodal data augmentation according to claim 1, characterized in that, Determining the hallucination confidence level based on the multimodal consistency index, the anomaly score, and the semantic distance includes: obtaining the maximum semantic distance based on the semantic distance; setting an anomaly score threshold; and determining the hallucination confidence level based on the maximum semantic distance, the anomaly score threshold, the multimodal consistency index, the anomaly score, and the semantic distance.

7. The intelligent medical hallucination detection method based on multimodal data augmentation according to claim 6, characterized in that, Determining hallucination confidence based on the maximum semantic distance, the anomaly scoring threshold, the multimodal consistency index, the anomaly score, and the semantic distance includes: according to the formula: Determine the confidence level Q of the hallucination, where, Let be the semantic distance between the s-th modality sample and the knowledge graph. Where A is the maximum semantic distance and A is the anomaly score. is the anomaly scoring threshold, C is the multimodal consistency index, M is the number of modal samples, s≤M, and both s and M are positive integers, and max is the maximum value function.

8. The intelligent medical hallucination detection method based on multimodal data augmentation according to claim 1, characterized in that, Determining whether medical hallucinations occur based on the hallucination confidence level includes: if the hallucination confidence level is greater than or equal to the hallucination confidence level threshold, then it is determined that medical hallucinations occur; if the hallucination confidence level is less than the hallucination confidence level threshold, then it is determined that no medical hallucinations occur.

9. A smart medical hallucination detection system based on multimodal data augmentation, used to execute the smart medical hallucination detection method based on multimodal data augmentation as described in any one of claims 1-8, characterized in that, include: The system comprises the following modules: a raw multimodal dataset module for collecting multimodal medical data and constructing a raw multimodal dataset, wherein the multimodal medical data includes medical image data, electronic medical record text data, and physiological signal data; a modal sample module for preprocessing the raw multimodal dataset and generating multiple enhanced modal samples by employing cross-modal data augmentation techniques through semantic alignment and information fusion between modalities; a modal feature vector module for constructing a multimodal feature extraction network to extract modal feature vectors from the modal samples; a multimodal consistency index module for determining a multimodal consistency index based on the modal feature vectors; an anomaly scoring module for fusing the modal feature vectors to determine an anomaly score; a semantic distance module for obtaining the semantic distance between the modal sample and the knowledge graph; a hallucination confidence module for determining a hallucination confidence level based on the multimodal consistency index, the anomaly score, and the semantic distance; and a judgment module for determining whether medical hallucination has occurred based on the hallucination confidence level.

Images