Chest image diagnosis method and system based on multi-modal sign collection

Abstract

The invention discloses a chest image diagnosis method and system based on multi-modal sign collection, and relates to the technical field of medical image.The method comprises the steps that a chest image of a patient is obtained, electrocardiosignals and blood oxygen saturation data are synchronously collected, and an associated radiology report is obtained; the image lesion features and the frequency domain rhythm template are combined for processing, motion artifacts are eliminated through a frequency domain decoupling equation, and refined image features are output; inputting the refined image features and the text pathological semantic features into a bidirectional attention mechanism to generate fusion features, and splicing the oxyhemoglobin saturation data and the text pathological semantic features into a sign-text vector; and inputting the fusion feature and the sign-text joint vector into a multi-task loss function, and outputting a structured diagnosis report. According to the method, accurate elimination of motion artifacts is achieved through a frequency domain decoupling equation, and coupling calculation is conducted on an electrocardio rhythm template and image lesion features in a frequency domain space.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of medical imaging technology, in particular to a chest image diagnosis method and system based on multi-modal sign collection. BACKGROUND

[0002] In recent years, multi-modal medical data analysis technology has made significant progress, and the application of deep learning models in chest image diagnosis has developed from single-modal analysis to comprehensive diagnosis integrating images, physiological signals and text reports. The Transformer architecture has shown strong performance in text feature extraction, while visual models such as Faster R-CNN have achieved precise anatomical structure positioning. At the same time, frequency domain analysis methods have been continuously improved in electrocardiogram signal processing, providing new ideas for multi-modal time alignment.

[0003] The existing technology still has essential defects in the aspect of cross-modal feature fusion, especially the dynamic coupling problem of physiological rhythm and image features has not been effectively solved. Traditional methods usually use simple feature concatenation or static weighted fusion, ignoring the modulation effect of electrocardiogram rhythm on image artifacts, resulting in incomplete motion artifact elimination and low detection rate of small lesions, which seriously restricts the improvement of diagnosis accuracy. SUMMARY

[0004] In view of the above existing problems, the present application is proposed.

[0005] Therefore, the present application provides a chest image diagnosis method based on multi-modal sign collection to solve the problem of high residual rate of motion artifacts and missed diagnosis of small lesions caused by ignoring the dynamic modulation effect of physiological rhythm.

[0006] To solve the above technical problems, the present application provides the following technical solutions:

[0007] In a first aspect, the present application provides a chest image diagnosis method based on multi-modal sign collection, which includes acquiring a patient's chest image, synchronously collecting electrocardiogram signals and blood oxygen saturation data, and acquiring associated radiology reports;

[0008] The radiology reports are structured and processed, and a pre-trained Transformer-based Bert model is used to extract text pathology semantic features. The chest image is anatomically labeled by a pre-trained Faster R-CNN model to extract coarse-grained visual features.

[0009] The electrocardiogram signals are subjected to Fourier transform to generate a frequency domain rhythm template, and the coarse-grained visual features are input into a pre-trained Transformer-based Beit model to extract image lesion features.

[0010] The image lesion features are combined with the frequency domain rhythm template for processing, motion artifacts are eliminated through a frequency domain decoupling equation, and refined image features are output;

[0011] The refined image features and the text pathological semantic features are input into a bidirectional attention mechanism to generate fusion features, the blood oxygen saturation data and the text pathological semantic features are spliced into a sign-text vector;

[0012] The fusion features and the sign-text joint vector are input into a multi-task loss function, and a structured diagnosis report is output.

[0013] As a preferred scheme of the chest image diagnosis method based on multi-modal sign acquisition, the specific steps of obtaining the associated radiology report are as follows,

[0014] A synchronous trigger pulse is generated at the exposure starting time through a low-radiation digital X-ray image acquisition device to acquire a chest image of a subject;

[0015] The synchronous trigger pulse starts the electrocardio acquisition function and the blood oxygen saturation acquisition function to acquire electrocardio signals and blood oxygen saturation data;

[0016] The medical image archive platform is retrieved through the patient identifier to obtain the associated radiology report.

[0017] As a preferred scheme of the chest image diagnosis method based on multi-modal sign acquisition, the specific steps of extracting coarse-grained visual features are as follows,

[0018] The radiology report is parsed to extract the examination date, image findings description and diagnosis conclusion fields to generate structured report data;

[0019] The structured report data is input into a pre-trained Transformer-based Bert model to extract text pathological semantic features;

[0020] A region proposal network processing is performed on the chest image to label the lung lobe anatomical part bounding box coordinates through a pre-trained Faster R-CNN model;

[0021] Image blocks are extracted within the lung lobe anatomical part bounding box coordinate region, and coarse-grained visual features are extracted using a feature pyramid network.

[0022] As a preferred scheme of the chest image diagnosis method based on multi-modal sign acquisition, the specific steps of extracting image lesion features are as follows,

[0023] A Fourier transform is performed on the electrocardio signals to generate a frequency domain rhythm template;

[0024] Input the coarse-grained visual features into a pre-trained Beit model based on the Transformer, and extract initial image lesion features;

[0025] Calculate the heart rhythm main frequency component in the frequency domain rhythm template, and construct a rhythm modulation vector;

[0026] Perform Hadamard product operation on the initial image lesion features and the rhythm modulation vector to generate rhythm-aware image lesion features;

[0027] Perform nonlinear activation enhancement on the rhythm-aware image lesion features to output image lesion features.

[0028] As a preferred scheme of the chest image diagnosis method based on multi-modal sign collection, the output refined image features have the following specific steps,

[0029] Convert the input image lesion features into a quantum wave function, perform time domain second derivative calculation on the frequency domain rhythm template, and generate a rhythm acceleration field;

[0030] Construct a frequency domain decoupling equation, perform a curved surface integral operation in a phase consistency constraint area, and perform coupled calculation on the phase gradient of the quantum wave function and the frequency domain rhythm template to generate a phase integral result;

[0031] Input the rhythm acceleration field into a hyperbolic error modulator to generate a modulation factor;

[0032] Perform tensor product operation on the phase integral result and the modulation factor to output refined image features.

[0033] As a preferred scheme of the chest image diagnosis method based on multi-modal sign collection, the output refined image features have the following specific steps,

[0034] Map the refined image features to an image quantum state vector, and map the text pathology semantic features to a text quantum state vector;

[0035] Obtain a quantum entanglement strength parameter based on the blood oxygen saturation data, and construct a quantum entanglement gate operation matrix;

[0036] Use the quantum entanglement gate operation matrix to entangle and fuse the image quantum state vector and the text quantum state vector to generate a fusion feature, project it to a real number space, and output an image-text fusion feature;

[0037] Construct a sign tensor from the blood oxygen saturation data, and upgrade the text pathology semantic features to a text tensor. Fuse the sign tensor and the text tensor through tensor weaving product to generate a sign-text vector.

[0038] As a preferred scheme of the chest image diagnosis method based on multi-modal sign collection of the present application, the output structured diagnosis report has the following specific steps,

[0039] Map the fusion features and sign-text vectors to the symplectic manifold space, construct a quantum density matrix for each diagnosis task and obtain the von Neumann entropy;

[0040] Based on the quantum density matrix and the obtained von Neumann entropy, the quantum entropy and the annealing temperature are dynamically weighted multi-task loss, combined with the Riemannian manifold regularization term and the Lie algebra regularization term to construct the loss function, and output the multi-task features;

[0041] The multi-task features are generated into a structured diagnosis report through a long short-term memory decoder optimized by Lie group.

[0042] In a second aspect, the present application provides a chest image diagnosis system based on multi-modal sign collection,

[0043] It includes an image acquisition module, a feature extraction module, a rhythm analysis module, an artifact elimination module, a feature fusion module and a report generation module;

[0044] The image acquisition module is used to acquire the chest image of the patient, synchronously acquire the electrocardiogram signal and the blood oxygen saturation data, and acquire the associated radiology report;

[0045] The feature extraction module is used to structure the radiology report, extract the text pathology semantic features using the pre-trained Transformer-based Bert model, and extract the coarse-grained visual features by anatomical part labeling of the chest image through the pre-trained FasterR-CNN model;

[0046] The rhythm analysis module is used to perform Fourier transform on the electrocardiogram signal to generate a frequency domain rhythm template, and input the coarse-grained visual features into the pre-trained Transformer-based Beit model to extract image lesion features;

[0047] The artifact elimination module is used to combine and process the image lesion features and the frequency domain rhythm template, eliminate motion artifacts through a frequency domain decoupling equation, and output refined image features;

[0048] The feature fusion module is used to input the refined image features and the text pathology semantic features into a bidirectional attention mechanism to generate fusion features, and splice the blood oxygen saturation data and the text pathology semantic features into a sign-text vector;

[0049] The report generation module is used to input the fusion features and the sign-text joint vector into a multi-task loss function, and output a structured diagnosis report.

[0050] In a third aspect, the present application provides a computer device comprising a memory and a processor, wherein the memory stores a computer program, and wherein the computer program, when executed by the processor, implements any step of the method for diagnosing chest images based on multi-modal sign acquisition according to the first aspect of the present application.

[0051] In a fourth aspect, the present application provides a computer readable storage medium storing a computer program, wherein the computer program, when executed by a processor, implements any step of the method for diagnosing chest images based on multi-modal sign acquisition according to the first aspect of the present application.

[0052] The present application has the following beneficial effects: the motion artifact is accurately eliminated by the frequency domain decoupling equation, the electrocardiogram rhythm template and the image lesion feature are coupled and calculated in the frequency domain space, the periodic characteristic of the electrocardiogram signal is used to construct a dynamic attenuation factor, and the artifact component asynchronous with the physiological rhythm is separated by Hadamard product operation. The motion artifact elimination rate is improved, the feature integrity of the micro-lesion is preserved, the processing delay is reduced, the real-time requirement of the clinic is met, the quantum phase integration operation is used in the frequency domain decoupling process, and the separability of the lesion area and the artifact in the frequency domain is further enhanced, thereby providing high-quality feature input for multi-modal fusion. BRIEF DESCRIPTION OF DRAWINGS

[0053] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings needed in the embodiment description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0054] Fig. 1 The flowchart of the method for diagnosing chest images based on multi-modal sign acquisition.

[0055] Fig. 2 The module diagram of the chest image diagnosis system based on multi-modal sign acquisition.

[0056] Fig. 3 The flowchart of the frequency domain rhythm template generation and image feature extraction.

[0057] Fig. 4 The flowchart of the quantum feature fusion and report generation. DETAILED DESCRIPTION

[0058] In order to make the above-mentioned objects, features and advantages of the present application more apparent and easy to understand, the specific embodiments of the present application will be described in detail below with reference to the drawings.

[0059] In the following description, numerous specific details are set forth in order to provide a thorough understanding of the present application. However, it will be apparent to one skilled in the art that the present application can be practiced without the specific details set forth in this description. In other instances, well-known methods, procedures, components, and circuits have not been described in detail so as not to unnecessarily obscure aspects of the present application.

[0060] Secondly, the "one embodiment" or "embodiment" referred to herein means a specific feature, structure, or characteristic that can be included in at least one implementation of the present application. The "in one embodiment" appearing in different places in the specification does not mean the same embodiment, nor is it an embodiment that is independent of or mutually exclusive with other embodiments.

[0061] Reference Figs. 1-4 For one embodiment of the present application, the embodiment provides a chest image diagnosis method based on multi-modal sign acquisition, comprising the following steps:

[0062] S1, acquiring a patient's chest image, synchronously acquiring an electrocardiogram signal and blood oxygen saturation data, and acquiring an associated radiology report.

[0063] Further, through the low-radiation digital X-ray image acquisition device, a synchronous trigger pulse at the exposure starting moment is generated to acquire the chest image of the examinee;

[0064] Specifically, through the linkage of the high-voltage generator and the flat panel detector, a synchronous trigger pulse is generated at the same time when the high-voltage generator starts exposure, the trigger pulse signal is transmitted to the electrocardiogram gating device, the electrocardiogram gating device locks the exposure starting moment according to the R-wave peak moment, the flat panel detector starts to acquire the chest image of the examinee at the exposure starting moment, and the low-radiation digital X-ray image acquisition device automatically adjusts the tube voltage and tube current during the acquisition process to ensure that the chest image meeting the diagnostic requirements is acquired while the radiation dose is reduced, and the chest image is output.

[0065] The synchronous trigger pulse starts the electrocardiogram acquisition function and the blood oxygen saturation acquisition function to acquire the electrocardiogram signal and the blood oxygen saturation data;

[0066] Specifically, the electrocardiogram acquisition function and the blood oxygen saturation acquisition function are started at the same time through the hardware trigger interface, the electrocardiogram acquisition function uses a three-lead electrode to acquire the electrocardiogram signal and performs signal amplification through a differential amplification circuit, the blood oxygen saturation acquisition function uses a finger clip type photoelectric sensor to acquire red light and infrared light transmission signals and convert them into blood oxygen saturation data, the two acquisition functions are time-stamp aligned under the control of the synchronous trigger pulse, the electrocardiogram signal and the blood oxygen saturation data are uploaded in real time to a data processing terminal through a Bluetooth transmission protocol, the data processing terminal performs R-wave detection on the electrocardiogram signal and time alignment storage with the blood oxygen saturation data.

[0067] Retrieving a picture archiving and communication system by a patient identifier, and obtaining an associated radiology report.

[0068] Specifically, the patient identifier is input as a query condition into a retrieval interface of the picture archiving and communication system, the picture archiving and communication system matches stored image data through a patient ID field in a DICOM standard, retrieves all examination instances associated with the patient identifier, the picture archiving and communication system extracts radiology report text data in the examination instances, the radiology report text data is returned in a structured JSON format after format conversion, and the radiology report corresponding to the patient identifier is output.

[0069] S2, structuring the radiology report, using a pre-trained Transformer-based Bert model to extract text pathology semantic features, and using a pre-trained Faster R-CNN model to anatomically label chest images to extract coarse-grained visual features.

[0070] Further, the radiology report is parsed to extract examination date, image findings description, and diagnosis conclusion fields, and structured report data is generated.

[0071] Specifically, the radiology report is subjected to word segmentation and semantic analysis through natural language processing, the time format in the examination date field is matched using a regular expression, the anatomical sites and abnormal features in the image findings description are extracted using a named entity recognition method, the key diagnostic terms in the diagnosis conclusion are identified through semantic role labeling, the examination date, image findings description, and diagnosis conclusion fields are respectively mapped to the corresponding positions of the structured report template, and the structured report data is generated.

[0072] The structured report data is input into a pre-trained Transformer-based Bert model to extract text pathology semantic features.

[0073] It should be noted that a large-scale unlabeled medical text corpus is used to learn text context representation through a bidirectional Transformer architecture, a mask language task is used to predict the masked medical vocabulary, the logical relationship between two medical text segments is judged in combination with a next sentence prediction task, the optimizer adjusts the parameters of the Transformer-based Bert model to minimize the prediction error in the pre-training process, and a Transformer-based Bert model capable of capturing medical semantic relationships is obtained.

[0074] Specifically, the structured report data is converted into a word vector sequence after text preprocessing, the word vector sequence is input into a Transform-based Bert model, the text context relationship is encoded through a multi-layer self-attention mechanism, a hidden layer feature vector containing global semantic information is extracted, and a fixed-dimensional text pathological semantic feature representation is obtained after a pooling operation, and the text pathological semantic feature is output.

[0075] The region proposal network is used to perform region proposal network processing on the chest image, and the pre-trained Faster R-CNN model is used to label the lung lobe anatomical part bounding box coordinates.

[0076] It should be noted that a large-scale medical image dataset containing lung lobe anatomical part labels is used, image feature maps are extracted by a deep convolutional neural network, the region proposal network generates candidate region proposals on the image feature maps, the classification branch predicts the probability that the candidate region proposals contain lung lobe anatomical parts, the regression branch adjusts the bounding box coordinates of the candidate regions, a multi-task loss function is used in the pre-training process to simultaneously optimize the classification accuracy and the bounding box positioning accuracy, the parameters of the convolutional neural network and the region proposal network are iteratively updated by a backpropagation algorithm, and a Faster R-CNN model capable of accurately detecting lung lobe anatomical parts is obtained.

[0077] Specifically, the chest image generates candidate region proposals on the image feature maps through the region proposal network, and high-quality region proposals are obtained after the candidate region proposals are filtered by the classification and regression branches of the region proposal network. The Faster R-CNN model extracts features from each region proposal and performs classification and bounding box regression. The classification branch identifies the lung lobe anatomical part category, and the regression branch outputs the lung lobe anatomical part bounding box coordinate parameters to obtain the bounding box coordinates labeled with the lung lobe anatomical part.

[0078] Image blocks are extracted within the lung lobe anatomical part bounding box coordinates, and coarse-grained visual features are extracted using a feature pyramid network.

[0079] It should be noted that a medical image dataset containing multi-scale labels is used, different levels of image feature maps are extracted by a deep residual network, the low-level feature maps are connected to a 3x3 convolutional layer to preserve spatial details, the high-level feature maps are fused with up-sampled features after adjusting the channel dimension by a 1x1 convolutional layer, a cross-entropy loss function is used in the pre-training process to optimize the multi-scale feature representation capability, the convolutional neural network parameters are updated by a backpropagation algorithm, and a feature pyramid network capable of effectively fusing multi-scale information is obtained.

[0080] Specifically, the image block of the corresponding region is cropped from the chest image according to the lung lobe anatomical position bounding box coordinates, and the image block is input into the feature pyramid network after normalization processing. The feature pyramid network extracts feature maps of different resolutions through multi-scale convolution operations. The low-level feature map retains spatial detail information, and the high-level feature map captures semantic abstract information. The feature pyramid network fuses the multi-scale feature maps from top to bottom and extracts coarse-grained visual features.

[0081] S3, performing Fourier transform on the electrocardiosignal to generate a frequency domain rhythm template, and inputting the coarse-grained visual features into a pre-trained Transformer-based Beit model to extract image lesion features.

[0082] Further, performing Fourier transform on the electrocardiosignal to generate a frequency domain rhythm template;

[0083] Specifically, the electrocardiosignal is input into a fast Fourier transform algorithm after baseline drift removal, power frequency interference suppression and electromyographic noise filtering. The fast Fourier transform algorithm converts the time domain electrocardiosignal into a frequency domain power spectrum. For example, the power spectrum is filtered by a bandpass filter to retain the physiological rhythm frequency band of 0.5Hz to 40Hz. The filtered power spectrum is processed by Gaussian smoothing to eliminate high-frequency noise. The amplitude of the smoothed frequency spectrum is normalized to generate a frequency domain rhythm template.

[0084] The coarse-grained visual features are input into a pre-trained Transformer-based Beit model to extract initial image lesion features.

[0085] It should be noted that the Transformer-based Beit model parameters are trained using a large-scale medical image dataset through self-supervised learning. The mask image modeling task randomly masks the image block and predicts the coarse-grained visual features of the masked area. The coarse-grained visual features are extracted by the Transformer encoder to obtain the global context information. During the pre-training process, the optimizer minimizes the prediction error of the masked area, and the Transformer-based Beit model that can understand the semantics of medical images is obtained.

[0086] Specifically, the coarse-grained visual features are converted into a serialized feature vector through a linear projection layer. The serialized feature vector is added to a fixed vector generated by a sine and cosine function and input into the Transformer-based Beit model. The long-range dependency between features is captured through a multi-layer self-attention mechanism. The features output by the self-attention mechanism are further extracted by a feedforward neural network to extract high-order semantic information. The feature vector corresponding to the classification token extracted from the last layer of the Transformer-based Beit model is used as the initial image lesion feature.

[0087] A heart rhythm main frequency component in a frequency domain rhythm template is calculated, and a rhythm modulation vector is constructed.

[0088] Specifically, a heart rhythm main frequency component in a frequency domain rhythm template is calculated, and the expression is:

[0089]

[0090] In the formula, f represents the heart rhythm main frequency component, arg represents the parameter extraction operator, max represents the maximum value of the function, f1 represents the frequency variable, represents a Fourier transform function of the electrocardio signal.

[0091] A rhythm modulation vector is constructed, and the expression is:

[0092] M = cos(2πfG);

[0093] In the formula, M represents the rhythm modulation vector, cos represents the cosine function, 2π represents the constant multiple of the circular ratio, and G represents the normalized space grid coordinate.

[0094] Specifically, Hadamard product operation is performed on the initial image lesion feature and the rhythm modulation vector to generate a rhythm-aware image lesion feature, and the expression is:

[0095]

[0096] In the formula, F represents the rhythm-aware image lesion feature, ReLU represents the linear rectification function, F1 represents the initial image lesion feature, represents the Hadamard product, b represents the bias constant, and σ represents the Sigmoid function, represents the gradient operator, and ||·|| represents the Euclidean norm, represents the gradient Euclidean norm of the initial image lesion feature F1.

[0097] Nonlinear activation enhancement is performed on the rhythm-aware image lesion feature, and an image lesion feature is output.

[0098] Specifically, the rhythm-aware image lesion feature is subjected to Gaussian weighting enhancement processing, a gain coefficient is calculated through the local neighborhood feature mean and standard deviation, the gain coefficient and the rhythm-aware image lesion feature are multiplied element by element, and then a hyperbolic tangent activation function is input, the hyperbolic tangent activation function performs nonlinear mapping and compression on the feature value, and an image lesion feature is output.

[0099] It should be noted that the gain coefficient is derived from the local feature statistics, the ratio of the mean and the standard deviation, and the example value range is [1.2, 2.5].

[0100] S4, the image lesion feature is combined and processed with the frequency domain rhythm template, motion artifacts are eliminated through a frequency domain decoupling equation, and a refined image feature is output.

[0101] Further, the input image lesion feature is converted into a quantum wave function, the time domain second derivative calculation is performed on the frequency domain rhythm template to generate a rhythm acceleration field;

[0102] Specifically, the image lesion feature is normalized to eliminate dimensional differences, and the real number domain feature vector is mapped to a complex vector space through a linear transformation matrix. The feature representation in the complex space is normalized to obtain a normalized complex feature vector, which constitutes a quantum wave function.

[0103] The time domain second derivative calculation is performed on the frequency domain rhythm template to generate a rhythm acceleration field, and the expression is:

[0104]

[0105] In the formula, T represents the rhythm acceleration field, T1 represents the frequency domain rhythm template, t represents time, Δt represents the time sampling interval, T1(t) represents the value of the frequency domain rhythm template at time t, (t-Δt) represents the previous sampling time of the frequency domain rhythm template at time t, T1(t-Δt) represents the value of the frequency domain rhythm template at (t-Δt), and (Δt) 2 represents the square of the time sampling interval.

[0106] Specifically, a frequency domain decoupling equation is constructed, a curvilinear area operation is performed in a phase consistency constraint region, and a phase integral result is generated by coupling calculation of the quantum wave function and the phase gradient of the frequency domain rhythm template, and the expression is:

[0107]

[0108] In the formula, F2 represents the phase integral result, Ω represents the integral region, and ∫ Ω represents the integral operation on the region Ω, ψ represents the quantum wave function, F3 represents the input image lesion feature, ψ(F3) represents the input image lesion feature converted into a quantum wave function, exp represents the natural exponential function, r represents the imaginary unit, represents the phase gradient field, <·,·> represents the inner product operation, dμ(θ) represents the surface measure, μ represents the measure, and θ represents the phase angle.

[0109] Specifically, the rhythm acceleration field is input into a hyperbolic error modulator to generate a modulation factor, and the expression is:

[0110]

[0111] In the formula, represents the modulation factor, erf represents the error function, η represents the rhythm change standard deviation, tanh represents the hyperbolic tangent function, and κ represents the mutation sensitivity coefficient.

[0112] It should be noted that the modulation factor is derived from the quantum tunneling effect strength, and the example value range is [-1.2, 1.2]; the mutation sensitive coefficient is generated by the inner product of the phase gradient and the tissue elastic tensor, and the example value is 0.3.

[0113] Specifically, the phase integral result is subjected to tensor product operation with the modulation factor to output a refined image feature, and the expression is:

[0114]

[0115] In the formula, F4 represents the refined image feature, represents the tensor product.

[0116] S5, input the refined image feature and the text pathological semantic feature into the bidirectional attention mechanism to generate a fusion feature, and splice the blood oxygen saturation data and the text pathological semantic feature into a sign- text vector.

[0117] Further, the refined image feature is mapped into an image quantum state vector, and the text pathological semantic feature is mapped into a text quantum state vector;

[0118] Specifically, the refined image feature is projected to a Hilbert space through a complex linear transformation, and the feature vector after projection is subjected to normalization processing to form an image quantum state vector. The text pathological semantic feature is mapped to a Hilbert space of the same dimension through an independent complex linear transformation, and the transformed feature vector is also subjected to normalization processing to generate a text quantum state vector. The image quantum state vector and the text quantum state vector both satisfy the quantum mechanics normalization condition and have the same vector space structure.

[0119] Based on the blood oxygen saturation data, a quantum entanglement strength parameter is obtained, and a quantum entanglement gate operation matrix is constructed;

[0120] Specifically, after the blood oxygen saturation data is subjected to normalization processing, it is input into a parameter mapping function. The parameter mapping function calculates the quantum entanglement strength parameter according to the blood oxygen saturation fluctuation amplitude. The quantum entanglement strength parameter is input into a quantum gate generation algorithm as a rotation angle. The quantum gate generation algorithm generates a two-dimensional unitary matrix based on the rotation angle to represent the quantum entanglement gate operation matrix. The quantum entanglement gate operation matrix satisfies the unitary condition and can realize the controlled entanglement operation of the image quantum state vector and the text quantum state vector.

[0121] The quantum entanglement gate operation matrix is used to entangle and fuse the image quantum state vector and the text quantum state vector to generate a fusion feature, which is projected to a real number space to output an image-text fusion feature;

[0122] Specifically, the image quantum state vector and the text quantum state vector form a combined quantum state through tensor product operation, the combined quantum state is subjected to unitary transformation operation with a quantum entanglement gate operation matrix, the quantum state after the unitary transformation operation is subjected to partial trace operation to reserve a target quantum state component, the target quantum state component is converted into a real number feature vector through orthogonal measurement base projection, and the real number feature vector forms an image-text fusion feature representation after being adjusted in dimension through a fully connected layer. The image-text fusion feature representation simultaneously contains quantum correlation characteristics of visual features and semantic features.

[0123] The blood oxygen saturation data is constructed as a sign tensor, and the text pathology semantic features are dimensionally upgraded as a text tensor. The sign tensor and the text tensor are fused through tensor weaving product to generate a sign-text vector.

[0124] Specifically, the blood oxygen saturation data is subjected to time series standardization processing to form a sign tensor, and the text pathology semantic features are converted into a text tensor through linear dimension upgrading operation. The sign tensor and the text tensor are input into a tensor weaving product operation layer. The tensor weaving product operation layer interacts and fuses the two modal tensors in the feature dimension through multi-linear mapping. The fusion result is converted into a two-dimensional matrix representation through flattening operation, and a sign-text vector encoding physiological signs and pathology semantics is output.

[0125] S6, input the fusion feature and the sign-text joint vector into a multi-task loss function, and output a structured diagnosis report.

[0126] Further, the fusion feature and the sign-text vector are mapped to a symplectic manifold space, a quantum density matrix is constructed for each diagnosis task, and a von Neumann entropy is obtained.

[0127] Specifically, the fusion feature and the sign-text vector are mapped to a high-dimensional space through a linear projection matrix. The projected fusion feature representation is input into a symplectic transformation layer. The symplectic transformation layer performs structure-preserving transformation on the fusion feature using a learnable symplectic matrix. The invariance of the symplectic form is maintained during the transformation process. The fusion feature satisfies the geometric constraint condition of the symplectic manifold in the transformed space, and the feature representation in the symplectic manifold space is obtained. The feature representation in the symplectic manifold space retains the semantic association and geometric characteristics of the original feature.

[0128] A quantum density matrix is constructed for each diagnosis task, and the expression is:

[0129]

[0130] In the formula, ρ k represents the quantum density matrix corresponding to the kth diagnosis task, k represents the diagnosis task index, Z represents a normalization factor, β represents the reciprocal of the annealing temperature, H k represents the Hamiltonian corresponding to the kth diagnosis task.

[0131] It should be noted that the normalization factor is derived from the partition function in quantum statistical mechanics, and the example value range is [1.2, 5.0].

[0132] Get the von Neumann entropy, the expression is:

[0133] S(ρ k )=-β(ρ k logρ k );

[0134] Where, S(ρ k ) represents the von Neumann entropy of the quantum density matrix corresponding to the k-th diagnostic task, γ represents the trace of the matrix, and log represents the logarithm of the matrix.

[0135] Based on the quantum density matrix and von Neumann entropy, we dynamically weight the multi-task loss through quantum entropy and annealing temperature, and construct a loss function combining the Riemannian manifold regularization term and the Lie algebra regularization term to output multi-task features.

[0136] Specifically, the quantum density matrix calculates the entropy value of each task through the von Neumann entropy formula. The entropy value is combined with the annealing temperature parameter to generate a dynamic weight coefficient. The dynamic weight coefficient is multiplied by the basic loss function of each task to obtain the weighted task loss. The weighted task loss is added to the Riemann manifold regularization term. The Riemann manifold regularization term constrains the feature smoothness through the curvature on the symplectic manifold. The Lie algebra regularization term imposes sparse constraints on the manifold structure of the trainable parameters. The total loss function optimizes the network parameters through the backpropagation algorithm. During the optimization process, the multi-task features gradually adjust the geometric distribution in the symplectic manifold space, and output multi-task features that meet the multi-task collaboration constraints.

[0137] Combining the Riemann manifold regularization term and the Lie algebra regularization term to construct the loss function, the expression is:

[0138]

[0139] Where, represents the total loss function, represents a symplectic manifold, Representation on a symplectic manifold The integral on represents Riemann connection, F5 represents fusion feature, Represents a symplectic manifold The norm induced by the metric tensor g, dV g Represents a symplectic manifold The volume differential on , λ represents the hyperparameter, K represents the total number of diagnostic tasks, Indicates the traversal and summation of all diagnostic tasks, l k represents the basic loss function corresponding to the k-th diagnostic task, V represents the sign-text vector, y kYk represents the true label corresponding to the kth diagnosis task, and W represents a global trainable weight matrix.

[0140] It should be noted that the global trainable weight matrix contains submatrices of all tasks, W = [W1, W2, …, W k ];

[0141] The selection of the basic loss function depends on the specific type of diagnosis task (such as classification, regression, segmentation, etc.).

[0142] The multi-task features are generated into structured diagnostic reports through the Lie group optimized long short-term memory decoder.

[0143] Specifically, the multi-task features are input into the Lie group optimized long short-term memory decoder, which processes the feature sequence through a special orthogonal group constrained weight update mechanism, the gate unit calculates the forget gate and input gate state in the manifold space, and the hidden state is projected back to the Lie group after tangent vector update in the Lie algebra space. The decoder gradually generates a text token sequence containing diagnosis conclusions, image descriptions, and examination suggestions, and the text token sequence is converted into a structured diagnostic report conforming to the medical report specification through natural language processing.

[0144] It should be noted that the special orthogonal group constraint refers to a mathematical restriction imposed on the weight of the Lie group optimized long short-term memory decoder, i.e., a set of all rotation matrices that keep the length and angle unchanged; the feature sequence refers to a set of sequentially arranged feature vectors processed by the Lie group optimized long short-term memory decoder, usually corresponding to medical data at different time steps or different levels (such as image features, clinical indicators, etc.).

[0145] The embodiment also provides a chest image diagnosis system based on multi-modal sign collection, comprising: an image collection module, a feature extraction module, a rhythm analysis module, an artifact elimination module, a feature fusion module and a report generation module; the image collection module is used for acquiring a patient chest image, synchronously collecting an electrocardiogram signal and blood oxygen saturation data, and acquiring an associated radiology report; the feature extraction module is used for performing structured processing on the radiology report, extracting text pathology semantic features by using a pre-trained Transformer-based Bert model, and performing anatomical site labeling on the chest image by using a pre-trained Faster R-CNN model to extract coarse-grained visual features; the rhythm analysis module is used for performing Fourier transform on the electrocardiogram signal to generate a frequency domain rhythm template, and inputting the coarse-grained visual features into a pre-trained Transformer-based Beit model to extract image lesion features; the artifact elimination module is used for combining and processing the image lesion features and the frequency domain rhythm template, eliminating motion artifacts by using a frequency domain decoupling equation, and outputting refined image features; the feature fusion module is used for inputting the refined image features and the text pathology semantic features into a bidirectional attention mechanism to generate fusion features, and splicing the blood oxygen saturation data and the text pathology semantic features into a sign-text vector; and the report generation module is used for inputting the fusion features and the sign-text joint vector into a multi-task loss function to output a structured diagnosis report.

[0146] The embodiment also provides a computer device suitable for the chest image diagnosis method based on multi-modal sign collection, comprising: a memory and a processor; the memory is used for storing computer executable instructions, and the processor is used for executing the computer executable instructions to implement the chest image diagnosis method based on multi-modal sign collection proposed in the above embodiment.

[0147] The computer device can be a terminal, and the computer device comprises a processor, a memory, a communication interface, a display screen and an input device connected through a system bus. The processor of the computer device is used for providing computing and control capabilities. The memory of the computer device comprises a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for running the operating system and the computer program in the non-volatile storage medium. The communication interface of the computer device is used for wired or wireless communication with external terminals. The wireless communication can be realized through WIFI, an operator network, NFC (near field communication) or other technologies. The display screen of the computer device can be a liquid crystal display screen or an electronic ink display screen. The input device of the computer device can be a touch layer overlaid on the display screen, or can be a key, trackball or touchpad arranged on the shell of the computer device, or can be an external keyboard, touchpad or mouse, etc.

[0148] The embodiment also provides a storage medium on which a computer program is stored, the program being executed by a processor to implement the method for diagnosing chest images based on multi-modal sign acquisition as proposed in the above embodiment; the storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0149] To sum up, the application achieves accurate elimination of motion artifacts by using a frequency domain decoupling equation, performs coupling calculation on an electrocardiogram rhythm template and image lesion characteristics in a frequency domain space, constructs a dynamic attenuation factor by using periodic characteristics of an electrocardiogram signal, and separates out an artifact component that is asynchronous with physiological rhythm by using Hadamard product operation. The motion artifact elimination rate is improved, the integrity of characteristics of a small lesion is preserved, processing delay is reduced, and the real-time requirement of clinical application is met. The quantumized phase integration operation is used in the frequency domain decoupling process, which further enhances the separability of a lesion area and artifacts in the frequency domain, and provides high-quality feature input for multi-modal fusion.

[0150] It should be noted that the above embodiments are only used to illustrate the technical solutions of the application but not limit the application. Although the application has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the application can be modified or replaced equivalently without departing from the spirit and scope of the technical solutions of the application, and all of them should be covered in the scope of the claims of the application.

Claims

1. A chest imaging diagnostic method based on multimodal vital sign acquisition, characterized by: include, Obtain chest images of patients, simultaneously collect ECG signals and blood oxygen saturation data, and obtain associated radiology reports; Radiology reports are structured and pathology semantic features are extracted using a pre-trained Transformer-based Bert model. Anatomical parts of chest images are annotated using a pre-trained Faster R-CNN model to extract coarse-grained visual features. Perform Fourier transform on the ECG signal to generate a frequency domain rhythm template, and input the coarse-grained visual features into the pre-trained Transformer-based Beit model to extract image lesion features; Combine the image lesion features with the frequency domain rhythm template, eliminate motion artifacts through the frequency domain decoupling equation, and output refined image features; The refined image features and text pathology semantic features are input into the bidirectional attention mechanism to generate fusion features, and the blood oxygen saturation data and text pathology semantic features are spliced ​​into a sign-text vector; The fused features and the joint sign-text vector are input into the multi-task loss function to output a structured diagnosis report.

2. The chest imaging diagnostic method based on multimodal vital sign acquisition according to claim 1, characterized in that: The specific steps for obtaining the associated radiology report are as follows: A low-radiation digital X-ray image acquisition device is used to generate a synchronous trigger pulse at the start of exposure to obtain a chest image of the subject; Synchronous trigger pulse starts the ECG acquisition function and blood oxygen saturation acquisition function to collect ECG signals and blood oxygen saturation data; Search medical imaging archive platforms by patient identifier to obtain associated radiology reports.

3. The chest imaging diagnostic method based on multimodal vital sign acquisition according to claim 2, characterized in that: The specific steps of extracting coarse-grained visual features are as follows: Parse radiology reports, extract examination date, imaging description, and diagnosis conclusion fields, and generate structured report data; Input the structured report data into the pre-trained Transformer-based Bert model to extract text pathological semantic features; Perform region proposal network processing on chest images and annotate the bounding box coordinates of the anatomical parts of the lung lobes using the pre-trained Faster R-CNN model; Image blocks are extracted within the bounding box coordinate area of ​​the anatomical part of the lung lobe, and coarse-grained visual features are extracted using a feature pyramid network.

4. The chest imaging diagnostic method based on multimodal vital sign acquisition according to claim 3, characterized in that: The specific steps of extracting image lesion features are as follows: Perform Fourier transform on the ECG signal to generate a frequency domain rhythm template; Input the coarse-grained visual features into the pre-trained Transformer-based Beit model to extract the initial image lesion features; Calculate the main frequency component of the ECG rhythm in the frequency domain rhythm template and construct the rhythm modulation vector; Performing Hadamard product operation on the initial image lesion feature and the rhythm modulation vector to generate rhythm-aware image lesion features; Nonlinear activation enhancement is performed on the rhythm-aware image lesion features to output the image lesion features.

5. The chest imaging diagnostic method based on multimodal vital sign acquisition according to claim 4, characterized in that: The output refined image features, the specific steps are as follows: The input image lesion features are converted into quantum wave functions, and the second-order derivative of the frequency domain rhythm template is calculated in the time domain to generate the rhythm acceleration field; Construct a frequency domain decoupling equation, perform surface integral operations in the phase consistency constraint region, couple the quantum wave function with the phase gradient of the frequency domain rhythm template, and generate a phase integral result; Inputting the rhythmic acceleration field into the hyperbolic error modulator to generate the modulation factor; The phase integration result is subjected to a tensor product operation with the modulation factor to output the refined image features.

6. The chest imaging diagnostic method based on multimodal vital sign acquisition according to claim 5, characterized in that: The splicing is a sign-text vector, and the specific steps are as follows: Mapping refined image features into image quantum state vectors, and mapping text pathological semantic features into text quantum state vectors; Obtain quantum entanglement strength parameters based on blood oxygen saturation data and construct a quantum entanglement gate operation matrix; Use the quantum entanglement gate operation matrix to entangle and fuse the image quantum state vector and the text quantum state vector, generate fusion features, project them into the real space, and output image-text fusion features; The blood oxygen saturation data is constructed as a physical sign tensor, and the text pathological semantic features are upgraded to a text tensor. The physical sign tensor and text tensor are fused through tensor braiding convolution to generate a physical sign-text vector.

7. The chest imaging diagnostic method based on multimodal vital sign acquisition according to claim 6, characterized in that: The specific steps for outputting a structured diagnostic report are as follows: Map the fused features and the sign-text vectors to the symplectic manifold space, construct the quantum density matrix for each diagnostic task and obtain the von Neumann entropy; Based on the quantum density matrix and von Neumann entropy, we dynamically weight the multi-task loss through quantum entropy and annealing temperature, and construct a loss function by combining the Riemannian manifold regularization term and the Lie algebra regularization term to output multi-task features. The multi-task features are passed through a Lie group optimized long short-term memory decoder to generate a structured diagnosis report.

8. A chest imaging diagnostic system based on multimodal vital sign acquisition, based on the chest imaging diagnostic method based on multimodal vital sign acquisition according to any one of claims 1 to 7, characterized in that: Including image acquisition module, feature extraction module, rhythm analysis module, artifact elimination module, feature fusion module and report generation module; The image acquisition module is used to acquire chest images of patients, synchronously acquire electrocardiogram signals and blood oxygen saturation data, and obtain associated radiology reports; The feature extraction module is used to perform structured processing on radiology reports, extract textual pathological semantic features using a pre-trained Transformer-based Bert model, and annotate anatomical parts of chest images using a pre-trained Faster R-CNN model to extract coarse-grained visual features. The rhythm analysis module is used to perform Fourier transform on the ECG signal to generate a frequency domain rhythm template, and input the coarse-grained visual features into a pre-trained Transformer-based Beit model to extract image lesion features; The artifact removal module is used to combine the image lesion features with the frequency domain rhythm template, eliminate motion artifacts through the frequency domain decoupling equation, and output refined image features; The feature fusion module is used to input the refined image features and text pathology semantic features into a bidirectional attention mechanism to generate fusion features, and splice the blood oxygen saturation data and text pathology semantic features into a sign-text vector; The report generation module is used to input the fusion features and the joint sign-text vector into the multi-task loss function and output a structured diagnosis report.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the chest imaging diagnosis method based on multimodal vital sign acquisition according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the chest imaging diagnosis method based on multimodal vital sign acquisition according to any one of claims 1 to 7 are implemented.

Images