A method for phase synchronization of multimodal data from heart sounds, electrocardiograms, and echocardiograms.

Abstract

This invention discloses a method for phase synchronization of multimodal data from heart sounds, electrocardiograms (ECG), and echocardiograms, belonging to the field of medical image processing technology. The method includes the following steps: A1. Acquiring original echocardiogram video images and independently recorded heart sound audio data; A2. Preprocessing the acquired data and automatically constructing samples; A3. Constructing a dual-path neural network architecture and extracting ECG waveforms to obtain a digitized ECG signal sequence; A4. Identifying periodic feature points of heart sounds and ECGs based on the digitized ECG signals; A5. Aligning asynchronous data based on linear phase resampling. This invention uses a linear interpolation algorithm to forcibly map video frames, electrical signals, and heart sound signals at different sampling rates to a unified phase coordinate system, ultimately achieving absolute synchronization of the three in terms of the physiological logic of cardiac pulsation, eliminating the influence of individual heart rate differences on data alignment.

Description

Technical Field

[0001] This invention relates to the field of medical image processing technology, specifically a method for phase synchronization of multimodal data of heart sounds, electrocardiograms, and echocardiograms. Background Technology

[0002] Congenital heart disease (CHD) is a congenital malformation caused by abnormal development of the heart and great vessels during the embryonic period. Early detection and treatment are crucial for reducing the risk of serious complications such as heart failure and cardiogenic shock. Currently, the clinical diagnosis of CHD mainly relies on physical examination, echocardiography, electrocardiography, and heart sound auscultation. With the development of deep learning technology, automated disease classification and recognition based on echocardiographic videos has become a research hotspot. To enhance the influence of cardiac periodic features and improve recognition accuracy, fusing electrocardiogram (ECG) signals for multimodal alignment and information complementarity is particularly important.

[0003] In existing technical solutions, it is typically necessary to extract the built-in electrocardiogram waveform from keyframes of echocardiogram videos. However, existing extraction methods mainly suffer from the following problems and drawbacks:

[0004] 1. High dependence of embedded waveform extraction on color thresholds: Traditional algorithms mainly separate ECG pixels by setting a threshold for a specific color (such as green) in ultrasound video frames. This method is extremely sensitive to color deviations and has difficulty adapting to waveform color changes under different devices and acquisition environments, resulting in low robustness of extraction.

[0005] 2. Pixel-level extraction suffers from poor anti-interference capability and is prone to positional shifts: Existing algorithms are easily affected by characters, scales, or echo noise in ultrasound images, leading to positioning deviations in the vertical axis coordinates (amplitude position) of the ECG waveform. This pixel-level estimation algorithm lacks noise resistance and often results in distortion of the extracted ECG signal.

[0006] 3. Lack of time-series consistency modeling: Conventional mathematical modeling algorithms often ignore the continuity and logical dependence of ECG waveform changes over time, and lack effective utilization of time series information.

[0007] 4. Difficulty in handling waveform breaks and occlusion: Existing technologies are mainly based on the assumption of pixel domain continuity. When the ECG waveform in an echocardiogram is interrupted due to pixel discontinuity, occlusion by other text labels, or color fading, traditional algorithms (such as sliding window search) will fail to extract the waveform due to reaching the termination condition, resulting in missing or incomplete waveforms.

[0008] 5. Difficulty in synchronizing asynchronous data: When heart sound signals and echocardiogram data are not acquired simultaneously (asynchronous acquisition), due to the lack of effective means of extracting ECG references and in-depth mining of the periodic ripple characteristics of the heartbeat, it is difficult to achieve accurate alignment of multi-source heterogeneous data in the phase of the cardiac cycle.

[0009] The aforementioned drawbacks lead to the discarding of a large amount of ultrasound data with waveform defects, resulting in a waste of data resources and severely impacting the performance evaluation and clinical applicability of multimodal assisted diagnostic models. Therefore, how to accurately and completely extract electrocardiogram (ECG) data from complex ultrasound videos and achieve reliable alignment with other modal data such as heart sounds is a pressing technical problem that needs to be solved in the field. Summary of the Invention

[0010] To address the aforementioned problems, this invention provides a method for phase synchronization of multimodal data from heart sounds, electrocardiograms, and echocardiograms, comprising the following steps:

[0011] A1. Acquire raw echocardiographic images and videos, as well as independently recorded heart sound audio data;

[0012] A2. Preprocess the acquired data and automatically construct samples;

[0013] A3. Construct a dual-path neural network architecture and extract ECG waveforms to obtain digital ECG signal sequences;

[0014] A4. Identification of heart sounds and periodic feature points of electrocardiogram based on digital electrocardiogram signals;

[0015] A5. Align asynchronous data based on linear phase resampling.

[0016] Furthermore, step A2 specifically includes the following sub-steps:

[0017] A21. Parse the acquired image and video stream data, and generate a binary mask using HSV color filtering;

[0018] A22. A sliding window with motion trend prediction is used to track the waveform trajectory in both directions: the first waveform pixel that meets the color threshold is located by scanning column by column in the binary mask and used as the initial seed point. The initial seed point is used as the center to track the waveform trajectory in the direction of increasing horizontal coordinate of the image until the edge of the image. Then, the initial seed point is returned to the initial seed point to track the waveform trajectory in the direction of decreasing horizontal coordinate, so as to ensure that the entire waveform trajectory is completely acquired.

[0019] A23. Output the initial numerical sequence with breakpoint markers, and extract the failed samples to generate a negative sample training set;

[0020] When the effective pixel density detected by the sliding window within the detection area is lower than the preset continuity threshold, it indicates that there is character occlusion or pixel breakage. The frame sequence number is automatically marked as a breakpoint to be repaired, and an initial numerical sequence with defect features is output.

[0021] Furthermore, the calculation formula for generating the binarized mask through HSV color filtering in step A21 is as follows:

[0022] ;

[0023] In the formula, Mask(x,y) represents the binary mask output at coordinates (x,y), used to determine whether the point belongs to the target pixel, where 1 represents the target pixel and 0 represents the background pixel; H, S, and V represent the hue, saturation, and lightness channel component values ​​of the pixel at coordinates (x,y) in the input image, respectively; [H min H max ]、[S min ,S max ]、[V min V max ] These represent the preset color threshold ranges on the corresponding hue, saturation, and brightness channels, respectively, used to lock the waveform color and filter out background noise.

[0024] Furthermore, the construction of the dual-path neural network architecture in step A3 specifically includes the following sub-steps:

[0025] A31. Set up a confidence level determination module to determine the validity of image regions;

[0026] A32. Based on the autocorrelation of electrocardiogram signals, capture the dependence of the waveform position at the predicted moment on historical and future trajectories;

[0027] A33. Aggregate the dependencies between the predicted and past moments and the historical and future trajectories captured in step A32, and reconstruct them in the pixel breakage region;

[0028] A34. Output a digitized electrocardiogram (ECG) signal sequence that is continuous on the time axis and aligned with the video frame rate.

[0029] Furthermore, the determination function for the validity of the image region in step A31 is:

[0030] ;

[0031] In the formula, w i F represents the i-th local image sliding window of the input. conv The convolution operation represents feature extraction, θ c Let C(w) represent the network learning parameters, σ represent the activation function that maps the output to the interval [0,1], and C(w) represent the network learning parameters.i ) represents the confidence score of the sliding window for the i-th local image;

[0032] The validity of the image region in step A31 is determined as follows: when the confidence score is ≥0.8, the trajectory of the current image region is considered valid.

[0033] Furthermore, the formula for calculating the dependence of the waveform position at time step A32 on historical and future trajectories is as follows:

[0034] ;

[0035] In the formula, Y t X represents the amplitude of the waveform's vertical axis at time t to be predicted. t H represents the feature input at the current time. t-1 With H t+1 These represent the hidden layer states at historical and future moments, respectively. Represents a nonlinear mapping function, θ t Indicates network learning parameters;

[0036] In step A33, the dependence of the captured and predicted moments on the historical and future trajectories is multiplied point by point using a gated fusion formula to generate the repaired trajectory points and reconstruct them in the pixel breakage area.

[0037] The gating fusion formula is as follows: Output = w × v; where Output represents the repaired trajectory points; w represents the waveform prediction value; and v represents the validity mask.

[0038] Furthermore, step A4 specifically includes the following sub-steps:

[0039] A41. Use a five-point derivative filter to highlight the waveform slope and identify physiological anchor points in digitized electrocardiogram signal sequences;

[0040] A42. Environmental noise is filtered out using a bandpass filter from 20-500Hz, and heart sound anchor points are extracted;

[0041] A43. Time axis anchoring based on physiological anchor points, heart sound anchor points, and valve closure frames in echocardiogram videos;

[0042] A44. Output a synchronization matrix containing aligned images, ECG and heart sound data.

[0043] Furthermore, the calculation formula for the five-point derivative filter in step A41 is as follows:

[0044] ;

[0045] In the formula, x[n] represents the input digital sampling points, and y[n] represents the differential signal after five-point derivative filtering;

[0046] The A42 step specifically extracts heart sound anchor points using the average Shannon energy, and the calculation formula is as follows:

[0047] ;

[0048] In the formula, E shannon The calculated energy envelope value is represented by x, where N represents the total number of sampling points within the calculation window. i This represents the amplitude value of the i-th sampling point within the window.

[0049] Further, step A5 specifically involves defining each detected adjacent RR interval or adjacent S1-S1 interval as a standardized cardiac cycle interval, and performing phase mapping to match and align the echocardiogram video, electrocardiogram signal, heart sound signal, and sampling rate; where R represents the identified electrocardiogram anchor point; and S1 represents the extracted heart sound anchor point.

[0050] Furthermore, the formula for calculating the phase mapping is as follows:

[0051] ;

[0052] In the formula, Φ(t) represents the percentage of phase within the standardized cardiac cycle at time t; t represents the timestamp of the current sampling point; T S Indicates the starting anchor point of the current cardiac cycle, that is, the peak time of the current ECG anchor point or the starting time of the heart sound anchor point; T e Indicates the starting anchor point of the next cardiac cycle; (T) e -T S () indicates the absolute duration of the cardiac cycle.

[0053] This invention provides a method for phase synchronization of multimodal data from heart sounds, electrocardiograms, and echocardiograms, which has the following beneficial effects:

[0054] (1) In this invention, the absolute time domain based on the physical clock is mapped to a standardized physiological phase domain based on the cardiac cycle, which eliminates the influence of individual heart rate differences on data alignment. Through linear interpolation algorithm, video frames, electrical signals and heart sound signals at different sampling rates are forcibly mapped to a unified phase coordinate system, and finally the three are absolutely synchronized in the physiological logic of heartbeat.

[0055] (2) By constructing a dual-path neural network consisting of a confidence module and a time-dependent module, the ability to understand the features of complex visual backgrounds is enhanced, the defects of existing technologies in terms of sensitivity to environmental noise and image damage are improved, and intelligent repair of damaged waveforms is realized, thereby enhancing the utilization value of low-quality ultrasound video data.

[0056] (3) An association mechanism using digital electrocardiogram (ECG) signals as a common phase reference and linear phase resampling technology were adopted. By mapping data with different heart rates (BPM) and sampling rates to a standardized physiological phase interval, the correlation deviation caused by asynchronous acquisition was effectively mitigated, and the interference of heart rate fluctuations on the accuracy of multimodal fusion was reduced. Without increasing additional hardware costs, a high degree of synchronization of heart sounds, ECG, and ultrasound images in the cardiac physiological cycle was achieved, providing more reliable reference data for multimodal clinical diagnosis.

[0057] (4) By integrating classic signal enhancement algorithms (such as the Pan-Tompkins differential-integral operator for ECG and the average Shannon energy operator for heart sounds), the signal-to-noise ratio and detectability of key physiological features such as ECG R wave and first heart sound S1 are significantly improved, thereby ensuring the accuracy of multimodal synchronous reference point positioning.

[0058] (5) By remapping independently acquired time-domain signals to a unified, standardized physiological phase space using linear phase resampling technology, this invention effectively decouples data acquisition from specific hardware conditions. Clinicians are no longer entirely limited to expensive multi-channel real-time synchronous acquisition systems and can complete acquisition work at different times using independent equipment. This reduces the equipment configuration threshold for medical institutions to a certain extent, improves the convenience of large-scale congenital heart disease screening, and has good prospects for widespread application. Attached Figure Description

[0059] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the structures shown in these drawings without creative effort.

[0060] Figure 1 A flowchart of the method provided by the present invention;

[0061] Figure 2 A schematic diagram of the dual-path neural network architecture provided by this invention;

[0062] Figure 3 A schematic diagram of the ECG anchor point and heart sound anchor point identification and phase synchronization framework provided by the present invention;

[0063] Figure 4 This is a schematic diagram of the asynchronous data alignment architecture based on linear phase resampling provided by the present invention. Detailed Implementation

[0064] It should be understood that the specific embodiments described herein are for illustrative purposes only and are not intended to limit the scope of the invention.

[0065] The following detailed description of the implementation method of the present invention is in conjunction with the accompanying drawings. The description is only a partial embodiment and not all embodiments. For clarity, representations and descriptions unrelated to the present invention are omitted in the drawings and description.

[0066] To provide a clearer understanding of the technical features, objectives, and beneficial effects of this invention, the following detailed description of the technical solution is provided. Obviously, the described embodiments are only a portion of the embodiments of this invention, not all of them, and should not be construed as limiting the scope of implementation of this invention. Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without inventive effort are within the protection scope of this invention.

[0067] like Figure 1 As shown, the waveform trajectory is first initially tracked and breakpoints are marked in the video using HSV color filtering and an improved sliding window algorithm. Then, a dual-module neural network with confidence discrimination and Bi-LSTM time prediction is used to intelligently repair occlusions and omissions, reconstructing a continuous digital ECG sequence. Subsequently, the Pan-Tompkins algorithm and the average Shannon energy operator are applied to accurately extract the R-wave peak and S1 sound as physiological time anchors. Finally, a 0-100% standardized physiological phase coordinate system is established and linear interpolation resampling is performed to eliminate sampling rate differences and heart rate fluctuations between devices. The final output is a strictly aligned matrix containing video frames, ECG values, and heart sound amplitudes.

[0068] This invention provides a method for phase synchronization of multimodal data including heart sounds, electrocardiograms, and echocardiograms, comprising the following steps:

[0069] A1. Obtain raw echocardiogram images and videos, as well as independently recorded heart sound audio data.

[0070] A2. The acquired data is preprocessed and samples are automatically constructed. Clinical keyframes (such as end-diastolic (ED) frames and end-systolic (ES) frames) annotated by doctors in ultrasound videos are used as temporal localization references to help determine the active regions of the ECG waveform within a complete cardiac cycle.

[0071] A21. The acquired image and video stream data are parsed, and a binary mask is generated using HSV color filtering; the specific formula for calculating the binary mask generated by HSV color filtering is as follows:

[0072] ;

[0073] In the formula, Mask(x,y) represents the binary mask output at coordinates (x,y), used to determine whether the point belongs to the target pixel, where 1 represents the target pixel and 0 represents the background pixel; H, S, and V represent the hue, saturation, and lightness channel component values ​​of the pixel at coordinates (x,y) in the input image, respectively; [H min H max ]、[S min ,S max ]、[V min V max ] These represent the preset color threshold ranges on the corresponding hue, saturation, and brightness channels, respectively, used to lock the waveform color and filter out background noise.

[0074] A22. A sliding window approach with motion trend prediction is used for bidirectional waveform trajectory tracking. Waveform data is extracted based on a binary mask, and a motion trend prediction mechanism effectively avoids interference from background Chinese characters or ruler lines. Specifically, the algorithm first locates the first waveform pixel that meets the color threshold by scanning column by column in the binary mask, using it as the initial seed point. Centered on this seed point, the algorithm executes a bidirectional tracking strategy: first, it tracks forward along the direction of increasing horizontal coordinates until the image edge; then, it returns to the seed point and tracks backward along the direction of decreasing horizontal coordinates, ensuring complete acquisition of the entire waveform trajectory.

[0075] During sliding window tracking, the continuous temporal characteristics of the waveform are utilized to dynamically lock the search area by calculating the first derivative of the preceding trajectory. First, the algorithm maintains a historical state queue, recording the centroid ordinates of the waveform pixels tracked in the preceding N consecutive frames in real time. This data is then used to calculate the waveform motion trend at the current time t. The calculation formula is as follows:

[0076] ;

[0077] In the formula, This indicates the rate of change of the waveform's trend along the vertical axis at the current moment; This represents the length of the historical state queue; in this embodiment, a value of 5 is preferred. Indicates the current number of tracking steps; Indicates at time The centroid ordinate of the valid waveform pixels detected within the sliding window.

[0078] Based on the calculated motion trend, the algorithm calculates the next moment. Predicted value of the ordinate of the waveform centroid The calculation formula is as follows:

[0079] ;

[0080] In the formula, The predicted ordinate of the waveform centroid at the next moment is used to directly lock the center position of the sliding window at the next moment; Indicates the current time The actual detected centroid ordinate of the waveform.

[0081] Then, the center of the sliding window at the next moment is forcibly set at the calculated predicted value. At this location, the effective pixel density belonging to the waveform mask is counted within this window. The calculation formula is as follows:

[0082] ;

[0083] In the formula, This represents the effective pixel density within the sliding window; This represents the total number of target pixels within the current sliding window that satisfy the binarization mask conditions; Indicates the height of the sliding window; This indicates the width of the sliding window.

[0084] Finally, the calculated With the preset continuity threshold (For example, 0.3) for comparison. If Greater than or equal to If the tracking is successful, the new centroid coordinates of the pixels within the current window are added to the trajectory sequence and the historical state queue is updated; if Less than If the waveform is obstructed or broken, the algorithm will stop extending in the current direction and mark the position as a breakpoint to be repaired, to be completed in subsequent steps.

[0085] A23. Output the initial numerical sequence with breakpoint markers and extract the failed samples from it. Generate a negative sample training set by analyzing the features of pixel breaks, color fading, or occlusion.

[0086] When the effective pixel density detected by the sliding window within the detection area is lower than the preset continuity threshold, it indicates that there is character occlusion or pixel breakage. The frame sequence number is automatically marked as a breakpoint to be repaired, and an initial numerical sequence with defect features is output.

[0087] A dual-path neural network architecture can extract high-fidelity digital electrocardiogram signals from preprocessed HSV ultrasound images, such as... Figure 2 As shown, the processing flow begins with the input image being processed by a shared feature extraction backbone network that includes convolution (Conv), activation (ReLU), and pooling (MaxPool) operations to generate a high-dimensional feature map. Subsequently, the data flows in parallel into two functionally independent branches: the temporal trajectory regression module on the left (corresponding to the translation variability network) is supervised by MSELoss and uses historical context features at time t−n to intelligently predict and complete the waveform numerical missingness (w) caused by occlusion or pixel breakage; the confidence gating module on the right (corresponding to the translation invariance network) is supervised by BCELoss and uses a binary classifier and sigmoid activation function to generate a region validity mask (v) to accurately identify the true existence state of the waveform under color fading or background interference; finally, the two are combined by the gating fusion formula (Output=w×v), and the validity mask is used to adaptively filter the regression prediction value, thereby outputting a digital electrocardiogram waveform sequence that is continuous, complete, and denoised in the time domain.

[0088] A3. To address the issue of traditional algorithms being prone to interruption in complex contexts, a dual-path neural network architecture is constructed, and ECG waveforms are extracted to obtain digital ECG signal sequences. By introducing translation invariance modeling, a confidence-based validity gating mechanism is constructed, which can effectively avoid false termination of the algorithm due to the lack of local features even if the waveform is obscured by BPM values ​​or a white scale.

[0089] A31. A confidence score determination module is set to judge the validity of image regions: when the confidence score is ≥0.8, the current image region trajectory is considered valid. The determination function for the validity of an image region is:

[0090] ;

[0091] In the formula, w i F represents the i-th local image sliding window of the input. conv The convolution operation represents feature extraction, θ c Let C(w) represent the network learning parameters, σ represent the activation function that maps the output to the interval [0,1], and C(w) represent the network learning parameters. i ) represents the confidence score of the i-th local image sliding window.

[0092] A32. The autocorrelation of electrocardiogram (ECG) signals is used to capture the dependence of the waveform position at the predicted time on historical and future trajectories. The calculation formula is as follows: ;

[0093] In the formula, Y t X represents the amplitude of the waveform's vertical axis at time t to be predicted.t H represents the feature input at the current time. t-1 With H t+1 These represent the hidden layer states at historical and future moments, respectively. Represents a nonlinear mapping function, θ t This represents the network learning parameters.

[0094] A33. Aggregate the dependencies of the captured and predicted moments on the historical and future trajectories in step A32, and reconstruct them in the pixel fracture area. Specifically, multiply the dependencies of the captured and predicted moments on the historical and future trajectories point by point using a gated fusion formula to generate repaired trajectory points and reconstruct them in the pixel fracture area.

[0095] The gating fusion formula is as follows: Output = w × v; where Output represents the repaired trajectory points; w represents the waveform prediction value; and v represents the validity mask. Temporal logic completion is implemented using the continuity characteristics of physiological signals. Specifically, the pixel breakage regions are intelligently reconstructed by aggregating the trajectory features before and after the aggregation, ensuring the physical integrity of the digital ECG sequence on the time axis.

[0096] A34. Output a digitized electrocardiogram (ECG) signal sequence that is continuous on the time axis and aligned with the video frame rate.

[0097] like Figure 3 As shown, key physiological features are extracted and asynchronous data alignment is achieved through dual-path parallel processing: the left path pairs the reconstructed digitized electrocardiogram sequence ( The Pan-Tompkins algorithm (including second-order difference and sliding integral) was used to accurately locate the R-wave peak as an electrophysiological anchor point. The right-hand path performs 20-500Hz bandpass filtering and average Shannon energy calculation on the raw heart sound signal to identify the S1 sound as a mechanophysiological anchor. These two sets of anchor points are then input into the underlying linear phase resampling module, which converts the absolute time into a normalized physiological phase (0-100%) and uses linear interpolation to resample all data to the resolution. On a unified grid, the final output is a multimodal synchronization matrix containing video frames, ECG values, and heart sound amplitudes that are strictly aligned.

[0098] A4. Identification of heart sounds and periodic feature points of ECG based on digital ECG signals: The signal-to-noise ratio of electrical signals (R waves) and acoustic signals (S1 tones) is enhanced by mathematical operators, aiming to extract a time alignment reference with physiological consistency for data streams from different sources.

[0099] A41. A five-point derivative filter is used to highlight the waveform slope and identify physiological anchor points (R peaks) in digitized electrocardiogram (ECG) signal sequences. The formula for calculating the five-point derivative filter is as follows:

[0100] In the formula, x[n] represents the input digital sampling points, and y[n] represents the differential signal after five-point derivative filtering.

[0101] A42. Ambient noise is filtered out using a 20-500Hz bandpass filter, and the heart sound anchor point (the starting point of the first heart sound S1) is extracted. Specifically, the heart sound anchor point is extracted using the average Shannon energy, and the calculation formula is as follows: In the formula, E shannon The calculated energy envelope value is represented by x, where N represents the total number of sampling points within the calculation window. i This represents the amplitude value of the i-th sampling point within the window.

[0102] A43. Time axis anchoring based on physiological anchor points, heart sound anchor points, and valve closure frames in echocardiogram videos;

[0103] A44. By eliminating sampling delay between devices The output includes a synchronization matrix containing aligned images, ECG data, and heart sound data. Based on the high temporal correlation between the R peak and S1 in physiology, this scheme establishes these two feature points as common phase references for achieving multimodal synchronization.

[0104] A5. Alignment of asynchronous data based on linear phase resampling: Each detected adjacent RR interval or adjacent S1-S1 interval is defined as a standardized cardiac cycle interval, and phase mapping is performed to match and align the echocardiogram video, electrocardiogram signal, heart sound signal with the sampling rate; where R represents the identified electrocardiogram anchor point; S1 represents the extracted heart sound anchor point.

[0105] like Figure 4 As shown, the core processing architecture that unifies multi-source heterogeneous medical data into a standardized physiological phase begins by receiving three raw signals with vastly different sampling rates—including a sparse ultrasound video stream of approximately 30 FPS, a standard clinical ECG signal of approximately 500 Hz, and a medical-grade high-fidelity sound signal of 2000 Hz; the system first uses the extracted R wave or S1 tone as the starting anchor point ( Construct a mapping formula from the absolute time domain to the 0-100% relative phase domain. Then, a linear interpolation algorithm was used to overcome the huge frequency differences between the source data (from low-frequency video to high-frequency audio), and all modal data were resampled to a uniform resolution. On the phase grid, the final output contains strictly aligned video frames. ), ECG values ​​( ) and heart sound amplitude ( The multimodal physiological phase synchronization matrix.

[0106] The formula for calculating the phase mapping is as follows:

[0107] ;

[0108] In the formula, Φ(t) represents the percentage of phase within the standardized cardiac cycle at time t; t represents the timestamp of the current sampling point; T S Indicates the starting anchor point of the current cardiac cycle, that is, the peak time of the current ECG anchor point or the starting time of the heart sound anchor point; T e Indicates the starting anchor point of the next cardiac cycle; (T) e -T S () indicates the absolute duration of the cardiac cycle.

[0109] Example: Target Region Localization and Preliminary Waveform Tracking Implementation Phase. In implementing this solution, the input echocardiogram video stream is first parsed, and keyframe images pre-annotated by the physician are extracted. Image recognition technology is then used to locate the rectangular display area containing the electrocardiogram curve within the keyframes. To eliminate background interference, the RGB image in this area is converted to the HSV color space, and a color filtering operator is applied to extract the target green pixels. Specifically, this is done by setting the hue... saturation and brightness Threshold conditions generate binarization masks Subsequently, using the waveform pixels in the keyframes as initial seeds, a sliding window algorithm incorporating motion trend prediction is employed for bidirectional tracking. During window movement, the centroid trajectory changes of the pixels in the preceding five consecutive frames are calculated in real time, and the expected value of the waveform's ordinate at the current moment is predicted using the first derivative. When the effective pixel density detected by the window within the prediction area is lower than a preset continuity threshold. When a character is obscured or a pixel is broken, the frame sequence number is automatically marked as a breakpoint to be repaired, and an initial numerical sequence with defect characteristics is output.

[0110] The waveform reconstruction implementation stage is based on a deep learning dual-module neural network. The initial sequence with breakpoints obtained in the first step, along with its corresponding local image slices, is fed into a pre-trained deep neural network model. This network consists of a confidence module and a temporal dependency module, which work together to ensure the integrity of the waveform. In specific execution, the confidence module first processes the image sliding window region... Feature convolution extraction is performed using a function. Calculate the probability score that this region belongs to the true electrocardiogram trajectory, where The Sigmoid activation function is used. When scoring... At this time, the algorithm will forcibly recognize the current trajectory as valid, thus avoiding false termination due to BPM values ​​or scale line occlusion. Meanwhile, the time-dependent module employs a bidirectional long short-term memory (Bi-LSTM) network structure, utilizing state equations... This module fully integrates historical and future trend information before and after the breakpoint. Through nonlinear regression logic, it can identify missing voltage amplitudes. Perform accurate prediction and filling. This step ultimately outputs a line that is completely continuous on the timeline and synchronized with the video frame rate. Fully aligned digital ECG sequences .

[0111] The implementation phase involves the identification and extraction of physiological feature anchor points for multimodal signals. After obtaining continuous digitized electrocardiogram (ECG) signals, physiologically logically consistent time anchor points are sought in asynchronous signals from different sources. This is specifically for digitized ECG sequences. The Pan-Tompkins algorithm is used for processing, specifically including the use of second-order difference operators. Emphasize the R-wave edge and perform sliding window integration. Among these, The output differential signal, For the input digital sampling points, This represents the discrete-time index. By setting a dynamic threshold in the integral signal and searching for local maxima, the time sequence of the R-peak is accurately located. Synchronously, the sampling rate for asynchronous acquisition is... The WAV format heart sound signal was bandpass filtered from 20 to 500 Hz, and the energy envelope was extracted using the average Shannon energy algorithm. The calculation formula is as follows: .in, To calculate the energy envelope value, The original sampling frequency of the heart sound signal. To calculate the total number of sampling points within the window, This corresponds to the amplitude value of the sampling point. By identifying the steep maxima of the rising edge of the energy envelope, the sequence representing the onset of the first heart sound, signifying the beginning of the heart's mechanical contraction, is located. .

[0112] Multimodal data alignment and encapsulation stage based on linear phase resampling. Addressing heart rate fluctuations caused by different acquisition devices (e.g., ...). and To address the issue of differences in time parameters, a transformation from the absolute time domain to the physiological phase domain is performed. Each detected interval between adjacent R-peaks or adjacent first heart sounds is defined as a standardized cardiac cycle interval. For any sampling point within the cycle... Using the phase mapping formula This is converted to phase coordinates. During the alignment process, a uniform phase resolution is set. Using a linear interpolation algorithm, the original video frames, digitized ECG values, and heart sound sampling points are all resampled onto a unified phase scale axis. If a specific phase point... If the data is not present in the original samples, it is synthesized by linear weighting of the two nearest actual samples. Finally, the aligned trimodal data is indexed according to phase. Multi-dimensional matrix encapsulation is performed to ensure that when a user retrieves any phase point, the corresponding video image frame for that physiological instant can be obtained simultaneously. Electrical activity values and heart sound amplitude .

[0113] In this invention, the absolute time domain based on the physical clock is mapped to a standardized physiological phase domain based on the cardiac cycle, eliminating the influence of individual heart rate differences on data alignment. Through a linear interpolation algorithm, video frames, electrical signals and heart sound signals at different sampling rates are forcibly mapped to a unified phase coordinate system, ultimately achieving absolute synchronization of the three in the physiological logic of cardiac pulsation.

[0114] By constructing a dual-path neural network consisting of a confidence module and a temporal dependency module, the ability to understand features in complex visual backgrounds is enhanced, the shortcomings of existing technologies in being sensitive to environmental noise and image damage are improved, and intelligent repair of damaged waveforms is achieved, thereby increasing the utilization value of low-quality ultrasound video data.

[0115] A correlation mechanism using digital electrocardiogram (ECG) signals as a common phase reference and linear phase resampling technology were employed. By mapping data at different heart rates (BPM) and sampling rates to standardized physiological phase intervals, the correlation deviation caused by asynchronous acquisition was effectively mitigated, and the interference of heart rate fluctuations on the accuracy of multimodal fusion was reduced. Without increasing additional hardware costs, a high degree of synchronization between heart sounds, ECG, and ultrasound images in the cardiac physiological cycle was achieved, providing more reliable benchmark data for multimodal clinical diagnosis.

[0116] By integrating classic signal enhancement algorithms (such as the Pan-Tompkins differential-integral operator for ECG and the average Shannon energy operator for heart sounds), the signal-to-noise ratio and detectability of key physiological features such as the ECG R wave and the first heart sound S1 are significantly improved, thereby ensuring the accuracy of multimodal synchronous reference point localization.

[0117] By remapping independently acquired time-domain signals to a unified, standardized physiological phase space using linear phase resampling technology, this invention effectively decouples data acquisition from specific hardware conditions. Clinicians are no longer entirely limited to expensive multi-channel real-time synchronous acquisition systems and can complete acquisition work at different times using independent devices. This lowers the equipment configuration threshold for medical institutions to a certain extent, improves the convenience of large-scale congenital heart disease screening, and has good prospects for widespread application.

[0118] The above description is merely a preferred embodiment of the present invention. It should be understood that the present invention is not limited to the forms disclosed herein and should not be construed as excluding other embodiments. It can be used in various other combinations, modifications, and environments, and can be altered within the scope of the concept described herein through the above teachings or related technologies or knowledge. Modifications and variations made by those skilled in the art that do not depart from the spirit and scope of the present invention should be within the protection scope of the appended claims.

Claims

1. A method for phase synchronization of multimodal data from heart sounds, electrocardiograms, and echocardiograms, characterized in that, Includes the following steps: A1. Acquire raw echocardiographic images and videos, as well as independently recorded heart sound audio data; A2. Preprocess the acquired data and automatically construct samples; A3. Construct a dual-path neural network architecture and extract ECG waveforms to obtain digital ECG signal sequences; A4. Identification of heart sounds and periodic feature points of electrocardiogram based on digital electrocardiogram signals; A5. Aligning asynchronous data based on linear phase resampling; Step A2 specifically includes the following sub-steps: A21. Parse the acquired image and video stream data, and generate a binary mask using HSV color filtering; A22. A sliding window with motion trend prediction is used to track the waveform trajectory in both directions: the first waveform pixel that meets the color threshold is located by scanning column by column in the binary mask and used as the initial seed point. The initial seed point is used as the center to track the waveform trajectory in the direction of increasing horizontal coordinate of the image until the edge of the image. Then, the initial seed point is returned to the initial seed point to track the waveform trajectory in the direction of decreasing horizontal coordinate, so as to ensure that the entire waveform trajectory is completely acquired. A23. Output the initial numerical sequence with breakpoint markers, and extract the failed samples to generate a negative sample training set; When the effective pixel density detected by the sliding window within the detection area is lower than the preset continuity threshold, it indicates that there is character occlusion or pixel breakage. The frame sequence number is automatically marked as a breakpoint to be repaired, and an initial numerical sequence with defect features is output. The construction of the dual-path neural network architecture in step A3 specifically includes the following sub-steps: A31. Set up a confidence level determination module to determine the validity of image regions; A32. Based on the autocorrelation of electrocardiogram signals, capture the dependence of the waveform position at the predicted moment on historical and future trajectories; A33. Aggregate the dependencies between the predicted and past moments and the historical and future trajectories captured in step A32, and reconstruct them in the pixel breakage region; A34. Output a digitized electrocardiogram signal sequence that is continuous on the time axis and aligned with the video frame rate; The A4 step specifically includes the following sub-steps: A41. Use a five-point derivative filter to highlight the waveform slope and identify physiological anchor points in digitized electrocardiogram signal sequences; A42. Environmental noise is filtered out using a bandpass filter from 20-500Hz, and heart sound anchor points are extracted; A43. Time axis anchoring based on physiological anchor points, heart sound anchor points, and valve closure frames in echocardiogram videos; A44. Output a synchronization matrix containing aligned images, ECG and heart sound data; The A5 step specifically involves defining each detected adjacent RR interval or adjacent S1-S1 interval as a standardized cardiac cycle interval, and performing phase mapping to match and align the echocardiogram video, electrocardiogram signal, heart sound signal, and sampling rate; where R represents the identified electrocardiogram anchor point; and S1 represents the extracted heart sound anchor point.

2. The method for phase synchronization of multimodal data of heart sounds, electrocardiograms, and echocardiography according to claim 1, characterized in that, The specific calculation formula for generating the binary mask through HSV color filtering in step A21 is as follows: ； In the formula, Mask(x,y) represents the binary mask output at coordinates (x,y), used to determine whether the point belongs to the target pixel, where 1 represents the target pixel and 0 represents the background pixel; H, S, and V represent the hue, saturation, and lightness channel component values ​​of the pixel at coordinates (x,y) in the input image, respectively; [H min H max ]、[S min ,S max ]、[V min V max ] These represent the preset color threshold ranges on the corresponding hue, saturation, and brightness channels, respectively, used to lock the waveform color and filter out background noise.

3. The method for phase synchronization of multimodal data of heart sounds, electrocardiograms, and echocardiography according to claim 1, characterized in that, The determination function for the validity of the image region in step A31 is: ； In the formula, w i F represents the i-th local image sliding window of the input. conv The convolution operation represents feature extraction, θ c Let C(w) represent the network learning parameters, σ represent the activation function that maps the output to the interval [0,1], and C(w) represent the network learning parameters. i ) represents the confidence score of the sliding window for the i-th local image; The validity of the image region in step A31 is determined as follows: when the confidence score is ≥0.8, the trajectory of the current image region is considered valid.

4. The method for phase synchronization of multimodal data of heart sounds, electrocardiograms, and echocardiography according to claim 3, characterized in that, The formula for calculating the dependence of the waveform position at time step A32 on historical and future trajectories is as follows: ； In the formula, Y t X represents the amplitude of the waveform's vertical axis at time t to be predicted. t H represents the feature input at the current time. t-1 With H t+1 These represent the hidden layer states at historical and future moments, respectively. Represents a nonlinear mapping function, θ t Indicates network learning parameters; In step A33, the dependence of the captured and predicted moments on the historical and future trajectories is multiplied point by point using a gated fusion formula to generate the repaired trajectory points and reconstruct them in the pixel breakage area. The gating fusion formula is as follows: Output = w × v; where Output represents the repaired trajectory points; w represents the waveform prediction value; and v represents the validity mask.

5. The method for phase synchronization of multimodal data of heart sounds, electrocardiograms, and echocardiography according to claim 1, characterized in that, The calculation formula for the five-point derivative filter in step A41 is as follows: ； In the formula, x[n] represents the input digital sampling points, and y[n] represents the differential signal after five-point derivative filtering; The A42 step specifically extracts heart sound anchor points using the average Shannon energy, and the calculation formula is as follows: ； In the formula, E shannon The calculated energy envelope value is represented by x, where N represents the total number of sampling points within the calculation window. i This represents the amplitude value of the i-th sampling point within the window.

6. The method for phase synchronization of multimodal data of heart sounds, electrocardiograms, and echocardiography according to claim 1, characterized in that, The formula for calculating the phase mapping is as follows: ； In the formula, Φ(t) represents the percentage of phase within the standardized cardiac cycle at time t; t represents the timestamp of the current sampling point; T S Indicates the starting anchor point of the current cardiac cycle, that is, the peak time of the current ECG anchor point or the starting time of the heart sound anchor point; T e Indicates the starting anchor point of the next cardiac cycle; (T) e -T S () indicates the absolute duration of the cardiac cycle.

Images