Method and apparatus for processing echocardiograms

Abstract

The application provides an echocardiogram processing method and device, and relates to the technical field of artificial intelligence. The echocardiogram processing method comprises the following steps: preprocessing an original echocardiogram video to obtain an echocardiogram video sample; performing key frame labeling and report text labeling on the echocardiogram video sample to generate key frame positioning samples and report generation samples; constructing a hybrid sample dataset according to the key frame positioning samples and the report generation samples; performing multi-task autoregressive supervised training on a preset initial model by using the hybrid sample dataset to obtain an initial multi-task model; performing cycle perception reinforcement learning on the initial multi-task model by using a pre-constructed cycle perception composite reward function according to the hybrid sample dataset to obtain a multi-task processing model; and inputting a user-inputted to-be-processed task into the multi-task processing model to obtain a task processing result. The application has good interpretability, robustness and physiological consistency.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, specifically to a method and apparatus for processing echocardiograms. Background Technology

[0002] Currently, echocardiography is a commonly used basic cardiovascular examination method. Its core information cannot be fully expressed by static images but exists within the dynamic process of the heart's structure changing over time. In actual workflows, operators typically follow a path of "global observation first – then localizing evidence – finally drawing conclusions": on the one hand, they need to summarize comprehensive information such as chamber size, ventricular wall motion, and valvular activity from the entire video to form a structured report; on the other hand, they need to select verifiable keyframes at specific cardiac phases (e.g., end-diastole, end-systole) for measurement and evidence preservation. This process is rigorous but time-consuming and highly dependent on the operator's experience.

[0003] Some studies have considered applying visual language models or multimodal large models to echocardiography video scenarios. However, visual language models or multimodal large models tend to focus on overall prediction centered on "question-answering-classification," resulting in poor interpretability and making them unsuitable for echocardiography-related applications. Furthermore, the scarcity of labeled echocardiography data and the labeling problems caused by the periodicity of echocardiography data both lead to poor performance of the trained models. Summary of the Invention

[0004] Based on this, this application provides an echocardiogram processing method and apparatus to realize echocardiogram processing applicable to echocardiogram-related application scenarios.

[0005] According to one aspect of this application, a method for processing echocardiograms is proposed, comprising: preprocessing the original echocardiogram video to obtain echocardiogram video samples; annotating the echocardiogram video samples with keyframes to generate keyframe localization samples, and annotating the echocardiogram video samples with report text to generate report generation samples; constructing a mixed sample dataset based on the keyframe localization samples and the report generation samples; performing multi-task autoregressive supervised training on a pre-defined initial model using the mixed sample dataset to obtain an initial multi-task model; performing period-aware reinforcement learning on the initial multi-task model using a pre-constructed period-aware composite reward function based on the mixed sample dataset to obtain a multi-task processing model; and inputting the user-inputted task into the multi-task processing model to obtain the task processing result.

[0006] According to some embodiments, the echocardiogram video sample includes a first echocardiogram video sample and a second echocardiogram video sample; keyframe annotation is performed on the echocardiogram video sample to generate a keyframe location sample, and report text annotation is performed on the echocardiogram video sample to generate a report generation sample, including: generating a first keyframe location sample for non-pathological query task samples based on the first echocardiogram video sample; and generating a second keyframe location sample for pathological query task samples based on the second echocardiogram video sample.

[0007] According to some embodiments, generating a first keyframe localization sample for a non-pathological query task sample based on a first echocardiogram video sample includes: obtaining a structural measurement sequence of key physiological parameters of the first echocardiogram video sample based on the non-pathological query task sample; determining key cardiac phases and their corresponding image frames based on the structural measurement sequence, and obtaining multiple cardiac cycles; selecting at least one target cardiac cycle from the multiple cardiac cycles based on the non-pathological query task sample, and generating a first keyframe candidate set based on the target cardiac cycle; performing structured annotation on the first echocardiogram video sample based on the elements in the first keyframe candidate set and the non-pathological query task sample, generating a first query-localization keyframe localization sample; and performing structured annotation on the first echocardiogram video sample based on the elements in the first keyframe candidate set and explanatory data related to the elements in the non-pathological query task sample, generating a first localization-explanation keyframe localization sample, wherein the first keyframe localization sample includes a first query-localization keyframe localization sample and a first localization-explanation keyframe localization sample.

[0008] According to some embodiments, based on non-pathological query task samples, at least one target cardiac cycle is selected from multiple cardiac cycles, and a first keyframe candidate set is generated based on the target cardiac cycle, including: selecting at least one representative cardiac cycle and at least one valid and non-representative cardiac cycle from multiple cardiac cycles as the target cardiac cycle based on the non-pathological query task samples; generating a keyframe of the target cardiac cycle, and storing the keyframe of the target cardiac cycle in the first keyframe candidate set.

[0009] According to some embodiments, generating a second keyframe localization sample for a pathology query task sample based on a second echocardiogram video sample includes: obtaining the disease type corresponding to the second echocardiogram video sample based on the pathology query task sample; locating candidate keyframes based on the disease type and constructing a second keyframe candidate set based on the candidate keyframes; performing structured annotation on the second echocardiogram video sample based on the elements in the second keyframe candidate set and the pathology query task sample to generate a second query-localization keyframe localization sample; and performing structured annotation on the second echocardiogram video sample based on the elements in the second keyframe candidate set and explanatory data related to the elements in the pathology query task sample to generate a second localization-explanation keyframe localization sample, wherein the second keyframe localization sample includes a second query-localization keyframe localization sample and a second localization-explanation keyframe localization sample.

[0010] According to some embodiments, locating candidate keyframes based on disease type and constructing a second keyframe candidate set based on the candidate keyframes includes: locating at least one image frame as a first candidate keyframe based on disease type; selecting at least one valid cardiac cycle from the cardiac cycles of the second echocardiogram video sample; generating keyframes of the valid cardiac cycles as second candidate keyframes; and storing the first candidate keyframe and the second candidate keyframe into the second keyframe candidate set.

[0011] According to some embodiments, the echocardiogram video samples include a third echocardiogram video sample and a fourth echocardiogram video sample. Keyframe annotation is performed on the echocardiogram video samples to generate keyframe localization samples, and report text annotation is performed on the echocardiogram video samples to generate report generation samples. This includes: retrieving the text report corresponding to the third echocardiogram video sample, performing time window matching and redundancy removal; extracting ultrasound-related content from the processed text report as the examination report text; generating a first video-text pair based on the third echocardiogram video sample and the examination report text; obtaining multiple pseudo-reports corresponding to the fourth echocardiogram video sample, filtering the multiple pseudo-reports to obtain a target pseudo-report; generating a second video-text pair based on the fourth echocardiogram video sample and the target pseudo-report; obtaining a third video-text pair based on a pre-constructed dataset; and performing structured annotation on the corresponding echocardiogram video samples based on the first, second, and third video-text pairs to generate report generation samples.

[0012] According to some embodiments, based on a mixed sample dataset, a pre-built periodic-aware composite reward function is used to perform periodic-aware reinforcement learning on an initial multi-task model to obtain a multi-task processing model, including: S1: Extracting the current sample from the keyframe localization samples in the mixed sample dataset, wherein the current sample includes input data and labeled output data; S2: Inputting the input data of the current sample into the initial multi-task model, and sampling multiple candidate outputs based on the output of the initial multi-task model; S3: Calculating the composite reward value of the multiple candidate outputs based on the labeled output data of the current sample and the periodic-aware composite reward function, and standardizing the composite reward value of the multiple candidate outputs to obtain the relative advantage of the multiple candidate outputs; S4: Optimizing the objective using a pre-built group relative strategy based on the relative advantage of the multiple candidate outputs, and updating the parameters of the initial multi-task model; S5: Repeating steps S1-S4 until a preset termination condition is reached, and outputting the parameter update result of the last iteration as the multi-task processing model.

[0013] According to some embodiments, the periodic-aware composite reward function includes a format reward function, a soft-boundary time precision reward function, a measurement vector similarity reward function, and / or a consistency reward function.

[0014] According to one aspect of this application, an echocardiogram processing apparatus includes: a preprocessing unit for preprocessing raw echocardiogram videos to obtain echocardiogram video samples; a data annotation unit for annotating the echocardiogram video samples with keyframes to generate keyframe localization samples, and for annotating the echocardiogram video samples with report text to generate report generation samples; a dataset unit for constructing a mixed sample dataset based on the keyframe localization samples and the report generation samples; a first training unit for performing multi-task autoregressive supervised training on a preset initial model using the mixed sample dataset to obtain an initial multi-task model; a second training unit for performing period-aware reinforcement learning on the initial multi-task model using a pre-constructed period-aware composite reward function based on the mixed sample dataset to obtain a multi-task processing model; and a model inference unit for inputting user-inputted tasks into the multi-task processing model to obtain task processing results.

[0015] According to one aspect of this application, an electronic device is provided, comprising: one or more processors; a storage device for storing one or more programs; and, when the one or more programs are executed by the one or more processors, causing the one or more processors to implement the method as described above.

[0016] According to one aspect of this application, a computer-readable medium is provided that stores a computer program or instructions thereon, which, when executed by a processor, implement the method as described above.

[0017] Through the embodiments provided in this application, during the training phase, keyframe annotation and report text annotation are performed on the preprocessed original echocardiogram videos to construct a mixed sample dataset. The mixed sample dataset is used to perform a first-stage supervised fine-tuning training on a pre-set initial model, where the model learns the structured output of keyframe localization and the structured output of report generation. Based on the initial multi-task model obtained in the first stage of training, the model is further optimized using periodic-aware composite rewards to obtain a multi-task processing model with superior performance. During the application phase, the user inputs the task to be processed into the trained multi-task processing model, which directly outputs task processing results suitable for echocardiogram-related application scenarios. The task processing results include keyframe localization evidence and related interpretive reports, exhibiting good interpretability, robustness, and physiological consistency, making them suitable for echocardiogram-related application scenarios. Attached Figure Description

[0018] It should be understood that the above general description and the following detailed description are merely exemplary and do not limit this application.

[0019] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings, without exceeding the scope of protection claimed by this application.

[0020] Figure 1 A flowchart illustrating the echocardiogram processing method provided in this application embodiment;

[0021] Figure 2 A flowchart illustrating the generation of a first keyframe localization sample in the echocardiography processing method provided in this application embodiment; Figure 3 A flowchart illustrating the generation of a second keyframe localization sample in the echocardiogram processing method provided in this application embodiment; Figure 4 A flowchart illustrating the process of generating a report sample for the echocardiogram processing method provided in this application embodiment; Figure 5 A flowchart of the periodic-aware reinforcement learning process for the echocardiography processing method provided in the embodiments of this application; Figure 6 A block diagram of the echocardiography processing apparatus provided in the embodiments of this application; Figure 7 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0022] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this application. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0023] Furthermore, the described features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. Numerous specific details are provided in the following description to give a thorough understanding of embodiments of this application. However, those skilled in the art will recognize that the technical solutions of this application can be practiced without one or more of the specific details, or other methods, components, apparatuses, steps, etc., can be employed. In other instances, well-known methods, apparatuses, implementations, or operations are not shown or described in detail to avoid obscuring various aspects of this application.

[0024] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically independent entities. That is, these functional entities can be implemented in software, in one or more hardware modules or integrated circuits, or in different network and / or processor devices and / or microcontroller devices.

[0025] The flowcharts shown in the accompanying drawings are merely illustrative and do not necessarily include all content and operations / steps, nor do they necessarily have to be performed in the described order. For example, some operations / steps can be broken down, while others can be combined or partially combined; therefore, the actual execution order may change depending on the specific circumstances.

[0026] It should be understood that although the terms first, second, third, etc., may be used herein to describe various components, these components should not be limited by these terms. These terms are used to distinguish one component from another. Therefore, the first component discussed below may be referred to as the second component without departing from the teachings of this application. As used herein, the term "and / or" includes all combinations of any one and more of the associated listed items.

[0027] In existing technologies, visual language models or multimodal large models have made progress in general video understanding tasks, but still face significant gaps in echocardiography video scenarios. The main reasons include: 1) The task format is not aligned with the workflow of actual application scenarios: Many existing methods are biased towards overall prediction centered on "question-answering-classification", ignoring the rigid requirement of "evidence traceability" in actual application scenarios. As a result, even if the model gives textual conclusions, it is difficult to provide verifiable key frame evidence, which is difficult to be accepted by actual application scenarios.

[0028] 2) Scarcity of high-quality video instruction data: Due to privacy and annotation costs, publicly available video data often only provides video-level labels or a small number of frame-level annotations. There is a lack of unified instruction corpus that simultaneously covers "fine-grained time positioning instructions" and "professional report generation", making it difficult for models to learn the correspondence between semantics and dynamic evidence in actual application scenarios.

[0029] 3) Periodic temporal ambiguity leads to unsupervised poorness: Echocardiography exhibits significant periodicity: phases with very similar morphologies can recur across multiple cardiac cycles. If the training data only provides a single standard frame number as the sole ground truth, the model will be forced to learn "indexed memory" rather than "physiologically invariant features," resulting in biases in both training and evaluation. Especially under high frame rate sampling or heart rate variations, multiple frames of the same phase are all valid, and the traditional exact matching loss or reward will incorrectly penalize correct predictions, leading to unstable training and poor generalization.

[0030] 4) Lack of constraints on consistency and security: Application scenarios have high security requirements. If the model output has internal contradictions (such as inconsistencies in the given time period, timestamp, and frame number, or inconsistencies between the reported conclusions and findings), it will significantly reduce the usability of practical applications. Traditional methods lack explicit measures and optimization objectives for this structured consistency.

[0031] Based on this, this application proposes a method and apparatus for processing echocardiograms.

[0032] For specific implementation details, please refer to the following examples.

[0033] Figure 1 A flowchart illustrating the echocardiography processing method provided in this application embodiment. Figure 1 As shown, the method includes steps S110-S160.

[0034] In step S110, the original echocardiogram video is preprocessed to obtain an echocardiogram video sample.

[0035] The original echocardiographic video can be acquired directly by the echocardiographic acquisition device or collected from existing resources, and this application does not impose any restrictions on this.

[0036] Preprocessing includes decoding, frame sampling, size normalization, and grayscale / multi-view alignment.

[0037] Furthermore, preprocessing can incorporate noise reduction, cropping, and view recognition processes based on device differences.

[0038] The preprocessed original echocardiogram video is recorded as an echocardiogram video sample.

[0039] In step S120, keyframe annotation is performed on the echocardiogram video samples to generate keyframe localization samples, and report text annotation is performed on the echocardiogram video samples to generate report generation samples.

[0040] Keyframe annotation is essentially a process of transforming raw ultrasound video-level annotations, temporal physiological priors, and specialist model outputs into frame-level, periodically aware, and instruction-based supervised samples, resulting in keyframe localization samples. Its core objective is not merely to provide a frame number, but to construct training instances that support the model in completing the three-layer alignment of "query-localization-evidence interpretation," i.e., supervised signals that simultaneously contain relevant time periods, precise keyframes, and semantic explanations.

[0041] According to the example implementation, key phases such as end-diastolic / end-systolic are determined by using actual application scenario rules, publicly available annotations, or structural measurement sequences (such as intracardiac diameter, ventricular wall thickness, and cardiac chamber area) output by professional models. Candidate sets are generated on multiple cardiac cycles. Keyframe annotation is performed on echocardiogram video samples using candidate sets and constructed conditional queries to obtain keyframe localization samples.

[0042] The report text annotation is essentially a process of converting the original ultrasound video-level annotations and the report text of the actual application scenario into a unified video-report pair to obtain a report generation sample.

[0043] According to the example implementation, the video is matched with the corresponding actual application scenario text through time window matching and redundancy removal. The text extraction model is used to extract content related to ultrasound examination to form a high-quality video-report pair, which serves as a report generation sample.

[0044] In step S130, a hybrid sample dataset is constructed based on the keyframe localization samples and the report generation samples.

[0045] The constructed keyframe localization samples and the report generation samples are combined to form a hybrid training set, which serves as the data foundation for subsequent model training.

[0046] In step S140, the preset initial model is trained using a mixed sample dataset to obtain an initial multi-task autoregressive supervised training.

[0047] For multi-task supervised fine-tuning training, autoregressive supervised training is adopted on the basis of mixed sample datasets, so that the model can simultaneously learn to extract dynamic evidence from videos and map it to time localization structure and generate reports that conform to the format and terminology of actual application scenarios, thereby forming a synergy of "localization promotes report and report semantic feedback localization" under common parameters.

[0048] In practice, a multi-task autoregressive learning paradigm is adopted. Under a unified framework, a mixed sample dataset is used to perform joint learning through multi-task supervised fine-tuning.

[0049] According to the example implementation, based on the constructed mixed sample dataset, the preset initial model is uniformly optimized using standard autoregressive cross-entropy loss, so that the model can simultaneously obtain fine-grained temporal localization and clinical report generation capabilities.

[0050] The preset initial model can be selected according to the actual situation, and this application does not restrict the model type or structure.

[0051] In this embodiment, Synergistic Multi-Task SFT is performed during the first training phase. The model is trained under supervision on mixed data and learns two types of outputs: structured results of keyframe localization (such as time period / key timestamp / keyframe sequence number / description) and structured real-world application scenario text generated in the report (organized into segments according to heart chambers, valves, ventricular wall motion, etc.).

[0052] In step S150, based on the mixed sample dataset, the initial multi-task model is subjected to periodic-aware reinforcement learning using a pre-built periodic-aware composite reward function to obtain a multi-task processing model.

[0053] Based on the initial multi-task model, periodic-aware reinforcement learning is performed for further training to improve the model's fine localization capabilities and resolve periodic temporal ambiguities.

[0054] According to the example embodiment, post-training of periodic-aware reinforcement learning includes: introducing group-relative policy optimization for the keyframe localization task. Outputs are sampled from the old policy for the same input, and a reward is calculated for each output and then standardized within the group to obtain an advantage, which is used to update the policy and constrain policy drift through a divergence term.

[0055] Meanwhile, a periodic-aware composite reward function was designed to transform "periodic ambiguity" from a training challenge into an optimizable objective.

[0056] The training result obtained from periodic-aware reinforcement learning is denoted as the multi-task processing model.

[0057] In this embodiment, Cycle-Aware Reinforcement Learning is performed in the second training phase. A policy optimization based on relative advantage between groups (GRPO) is introduced for the keyframe localization task. Multiple candidate outputs are sampled for the same input, and their relative advantages are calculated for updating. Reference policy constraints are also added to stabilize training. A cycle-aware composite reward is designed to complement this phase.

[0058] In step S160, the task to be processed by the user is input into the multi-task processing model to obtain the task processing result.

[0059] During the application process, the user-inputted task is fed into the trained multi-task processing model, and the model outputs the task processing result.

[0060] For keyframe localization tasks, the input consists of echocardiographic video and query conditions. Query conditions may include periodic conditions (such as end-diastole, end-systole) and / or pathological conditions (such as a specific pathological type, functional classification, etc.). The model output adopts a hierarchical structure, including at least the predicted time period related to the query conditions (period-level segments), output key moment timestamps and keyframe numbers (frame-level localization), and a textual description of the structural / functional / state basis corresponding to the localization (evidence-level explanation).

[0061] For report generation, the input task to be processed includes echocardiogram video (multi-view video can be input), and the model outputs a structured report, which includes sections such as cardiac chamber structure, valves, ventricular wall motion, and overall conclusions, and meets the terminology and format requirements of actual application scenarios.

[0062] In the training phase of this application, keyframe annotation and report text annotation are performed on the preprocessed original echocardiogram videos to construct a mixed sample dataset. The mixed sample dataset is then used to perform a first-stage supervised fine-tuning training of a pre-set initial model, where the model learns the structured output of keyframe localization and the structured output of report generation. Based on the initial multi-task model obtained in the first stage of training, the model is further optimized using periodic-aware composite rewards to obtain a multi-task processing model with superior performance. In the application phase, the user inputs the task to be processed, and the trained multi-task processing model directly outputs task processing results suitable for echocardiography-related application scenarios. The task processing results include keyframe localization evidence and related interpretive reports, exhibiting good interpretability, robustness, and physiological consistency, making them suitable for echocardiography-related application scenarios.

[0063] This application aims to achieve "integrated keyframe localization and report generation". By constructing large-scale instruction data and introducing periodic-aware reinforcement learning post-training, and using "logical consistency and similarity to actual application scenarios" as the core reward signal, it replaces the rigid matching of a single frame index, fundamentally alleviating periodic time ambiguity, improving keyframe localization accuracy and report quality, and enhancing the verifiability of the output and its usability in actual application scenarios.

[0064] According to some embodiments, the echocardiogram video sample includes a first echocardiogram video sample and a second echocardiogram video sample.

[0065] The sources of the first and second echocardiographic video samples include multi-source echocardiographic data such as EchoNet-LVH, EchoNet-Dynamic, EchoCP, and HMC-QU.

[0066] Based on the above embodiments, in step S120, keyframe annotation is performed on the echocardiogram video samples to generate keyframe positioning samples, and report text annotation is performed on the echocardiogram video samples to generate report generation samples. This can be specifically achieved through steps S121-S122.

[0067] In step S121, a first keyframe localization sample for non-pathological query task samples is generated based on the first echocardiogram video sample.

[0068] Non-pathological query tasks include structure / function related tasks, such as the periodic condition keyframe localization task, which locates keyframes of specific cardiac phases (e.g., end-diastole, end-systole) within a representative cardiac cycle based on input periodic conditions (e.g., end-diastole, end-systole).

[0069] In step S122, a second keyframe localization sample for pathological query task samples is generated based on the second echocardiogram video sample.

[0070] Pathological query tasks include specific pathological tasks, which involve locating the most relevant keyframes based on input pathological conditions (such as a pathological type, functional grade, etc.).

[0071] According to some embodiments, refer to Figure 2 In step S121, a first keyframe localization sample for non-pathological query task samples is generated based on the first echocardiogram video sample, which can be implemented through steps S210-S250.

[0072] In step S210, based on the non-pathological query task sample, the structural measurement sequence of key physiological parameters of the first echocardiogram video sample is obtained.

[0073] Specifically, key physiological parameters of the first echocardiogram video sample are estimated frame by frame to obtain a structural measurement sequence.

[0074] Key physiological parameters include LVID (left ventricular internal diameter), IVS (interventricular septum), LVPW (left ventricular posterior wall thickness), and left ventricular area.

[0075] According to the example embodiment, structural measurement sequences are obtained using an open-source professional model from a data source.

[0076] In step S220, based on the structural measurement sequence, the key cardiac phases and their corresponding image frames are determined, and multiple cardiac cycles are obtained.

[0077] It needs to be explained that the left ventricle is full at end-diastole (ED), at which point the left ventricular area or diameter reaches its maximum value within a single heartbeat cycle; it reaches its minimum value at end-systole (ES).

[0078] Based on the numerical values ​​of the structural measurement sequence, the maximum area point (corresponding to the ED candidate frame) and the minimum area point (corresponding to the ES candidate frame) of each heartbeat cycle can be determined, which are the key phases such as end-diastole / end-systole, thereby obtaining multiple heartbeat cycles and their respective ED and ES candidate frame positions.

[0079] In step S230, at least one target cardiac cycle is selected from multiple cardiac cycles based on the non-pathological query task samples, and a first keyframe candidate set is generated based on the target cardiac cycle.

[0080] The target cardiac cycle includes at least one representative cardiac cycle.

[0081] Furthermore, in some embodiments, the target cardiac cycle also includes other valid cycles among multiple cardiac cycles to alleviate one-to-many temporal ambiguity caused by cardiac cycle repetition.

[0082] The screening strategy for representative cardiac cycles can be a pre-set optimization strategy (such as the Rank-Sum screening strategy), and this application embodiment does not limit this.

[0083] A first keyframe candidate set is generated based on the target cardiac cycle. According to the example embodiment, keyframes of the target cardiac cycle are selected and added to the set as the first keyframe candidate set.

[0084] The description of a keyframe in the first keyframe candidate set includes: the relevant time period (periodic segment), the output key moment timestamp, and the keyframe sequence number (frame-level positioning).

[0085] In step S240, the first echocardiogram video sample is structured and annotated based on the elements in the first keyframe candidate set and the non-pathological query task sample to generate the first query-location keyframe location sample.

[0086] Using the non-pathological query task samples related to the first echocardiogram video sample and the key frames in the corresponding first key frame candidate set, a "Query→Grounding" sample for query-location is constructed as the first query-location key frame location sample.

[0087] In step S250, the first echocardiogram video sample is structured and annotated according to the elements in the first keyframe candidate set and the explanatory data related to the elements and the non-pathological query task sample, and a first location-explanation keyframe location sample is generated. The first keyframe location sample includes a first query-location keyframe location sample and a first location-explanation keyframe location sample.

[0088] Using keyframes from the first keyframe candidate set related to the first echocardiogram video sample and description data related to the keyframes and non-pathological query task samples, a “Grounding→Description” sample is constructed to be located to the semantic interpretation, serving as the first localization-description keyframe localization sample.

[0089] The explanatory data includes a textual description (evidence-level explanation) of the structure / function / status corresponding to the location.

[0090] The first query - location keyframe location sample and the first location - description keyframe location sample constitute the first keyframe location sample.

[0091] The samples are constructed using an instruction-based approach, transforming them into "query-location" instruction forms. A reverse construction of "location-description" is also added to enhance robustness.

[0092] According to some embodiments, in step S230, at least one target cardiac cycle is selected from multiple cardiac cycles based on the non-pathological query task sample, and a first keyframe candidate set is generated based on the target cardiac cycle. This can be specifically implemented through steps S231-S232.

[0093] In step S231, based on the non-pathological query task sample, at least one representative cardiac cycle and at least one valid but non-representative cardiac cycle are selected from multiple cardiac cycles as the target cardiac cycle.

[0094] In some embodiments, based on non-pathological query task samples, at least one representative ED / ES cardiac cycle is determined as a representative cardiac cycle by combining pixel surface positive value and Rank-Sum screening strategy (such as Mann-Whitney U test).

[0095] In the specific implementation process, nonparametric statistical tests are performed on the left ventricular area measurements at the ED point (or ES point) of all cycles to obtain the cycle subset with the most concentrated and consistent measurement distribution as the representative cardiac cycle.

[0096] From multiple cardiac cycles other than the representative cardiac cycle, select at least one valid but non-representative cardiac cycle, and use it together with the representative cardiac cycle as the target cardiac cycle.

[0097] In step S232, keyframes of the target cardiac cycle are generated and stored in the first keyframe candidate set.

[0098] Based on the non-pathological query task samples, the most relevant frame in the target cardiac cycle is selected as the key frame of the target cardiac cycle and stored in the first key frame candidate set.

[0099] According to the example embodiment, the ED frame image and ES frame image of the target cardiac cycle are used as keyframes.

[0100] According to some embodiments, refer to Figure 3 In step S122, a second keyframe localization sample for pathological query task samples is generated based on the second echocardiogram video sample, which can be implemented through steps S310-S340.

[0101] In step S310, the disease type corresponding to the second echocardiogram video sample is obtained based on the pathology query task sample.

[0102] Disease types can be identified using open-source professional models from data sources, or they can be obtained using the expertise of technical personnel; this application does not impose any restrictions on this.

[0103] In step S320, candidate keyframes are located according to the disease type, and a second keyframe candidate set is constructed based on the candidate keyframes.

[0104] Candidate keyframes include at least one representative keyframe.

[0105] Furthermore, in some embodiments, candidate keyframes also include image frames in addition to representative keyframes to alleviate one-to-many temporal ambiguity caused by cardiac cycle repetition.

[0106] The selection strategy for representative keyframes can be based on dynamic analysis of grayscale intensity or expert prior knowledge, and this application does not limit this.

[0107] The candidate keyframes are stored in a set to obtain the second keyframe candidate set.

[0108] In step S330, the second echocardiogram video sample is structurally annotated based on the elements in the second keyframe candidate set and the pathological query task sample to generate the second query-location keyframe location sample.

[0109] Using the pathological query task samples related to the second echocardiogram video samples and the keyframes in the corresponding second keyframe candidate set, a "Query→Grounding" sample for query-location is constructed as the second query-location keyframe location sample.

[0110] In step S340, the second echocardiogram video sample is structured and annotated according to the elements in the second keyframe candidate set and the explanatory data related to the elements and the pathological query task sample, to generate a second location-explanation keyframe location sample. The second keyframe location sample includes the second query-location keyframe location sample and the second location-explanation keyframe location sample.

[0111] Using keyframes from the second keyframe candidate set related to the second echocardiogram video sample, and the descriptive data of the keyframes related to the pathology query task sample, a “Grounding→Description” sample is constructed to locate the semantic interpretation, serving as the second localization-description keyframe localization sample.

[0112] The explanatory data includes a textual description (evidence-level explanation) of the structure / function / status corresponding to the location.

[0113] Second query - locate keyframe location sample and second location - description: keyframe location sample constitutes second keyframe location sample.

[0114] In this application embodiment, the sample is constructed in an instructional form, which is a "query-location" instruction, and a "location-description" reverse construction is added to enhance robustness.

[0115] It is understandable that the construction process of the keyframe localization samples in this application can be summarized as follows: data collection and preprocessing - frame-level physiological quantity estimation / pathological clue extraction - representative cardiac cycle screening - keyframe and candidate cycle alignment and annotation - conditional query construction - bidirectional instruction sample organization, thereby constructing an instruction corpus covering "fine-grained temporal localization instructions". Specifically, the bidirectional instruction construction performed in this application involves formatting the same annotation into two complementary forms: one is "Query → Grounding", where given a diagnostic requirement or cycle / pathological query, the model outputs the keyframe localization result; the other is "Grounding → Description", where given the localized time period / keyframe information, the model is required to generate the corresponding explanatory semantic description, thus forming fine-grained supervised data that combines temporal localization, frame-level indexing, and clinical evidence description. This bidirectional construction can improve the diversity of training samples and enhance the model's bidirectional mapping ability between "localization" and "interpretation", thereby preventing the model from degenerating into a mere memorization of frame numbers.

[0116] According to some embodiments, in step S320, candidate keyframes are located according to the disease type, and a second keyframe candidate set is constructed based on the candidate keyframes. This can be specifically implemented through steps S321-S324.

[0117] In step S321, at least one image frame is located as a first candidate keyframe according to the disease type.

[0118] The first candidate keyframe is the image frame most relevant to the disease type, i.e., the main target frame. The first candidate keyframe can be located using dynamic analysis of grayscale intensity, a pre-trained localization model, or expert prior knowledge; this application does not impose any restrictions on this.

[0119] In step S322, at least one valid cardiac cycle is selected from the cardiac cycles of the second echocardiogram video sample.

[0120] In step S323, a valid keyframe of the cardiac cycle is generated as a second candidate keyframe.

[0121] Based on the disease type, the most relevant frame in the effective cardiac cycle is selected as the key frame of the effective cardiac cycle and as the second candidate key frame.

[0122] At the annotation level, in addition to the first candidate keyframe (main target frame), the embodiments of this application also simultaneously mark the corresponding frames in other valid cycles as second candidate keyframes to alleviate the one-to-many temporal ambiguity caused by the repetition of cardiac cycles.

[0123] The second candidate keyframe can be located using dynamic analysis of grayscale intensity, a pre-trained localization model, or expert prior knowledge; this application does not impose any restrictions on this.

[0124] In step S324, the first candidate keyframe and the second candidate keyframe are stored in the second keyframe candidate set.

[0125] Store the first and second candidate keyframes into a set to obtain the second keyframe candidate set.

[0126] According to some embodiments, the echocardiographic video samples include a third echocardiographic video sample and a fourth echocardiographic video sample.

[0127] The sources of the third and fourth echocardiographic video samples include time-aligned samples of MIMIC-IV-ECHO and MIMIC-IV-Note, and supplementary samples of MIMIC-EchoQA.

[0128] Based on the clinical definitions of diseases in different data, this application acquires, supplements, or infers frame-level indicators and key cycle information for each echocardiogram video sample, and then transcribes this information into fine-grained labels for the localization samples.

[0129] Based on the above embodiments, referring to Figure 4 In step S120, keyframe annotation is performed on the echocardiogram video samples to generate keyframe positioning samples, and report text annotation is performed on the echocardiogram video samples to generate report generation samples. This can be achieved through steps S410-S470.

[0130] In step S410, the text report corresponding to the third echocardiogram video sample is retrieved, and time window matching and redundancy removal are performed.

[0131] Ultrasound videos are extracted from third echocardiographic video samples with associated text reports (e.g., echocardiographic video samples from MIMIC-IV-ECHO). Corresponding scene-related records (including text reports) are retrieved from these samples. Check-level matching and redundancy removal are performed based on indexes and time windows to restore the true temporal correspondence between the videos and text reports as much as possible.

[0132] According to the example implementation, the check-level matching and redundancy removal process includes: using a 7-day main alignment window and deduplicating or filtering out redundant checks within 2 days.

[0133] In step S420, the content related to the ultrasound examination is extracted from the processed text report and used as the examination report text.

[0134] Extract relevant content directly related to the ultrasound examination from the original text report, filter out incomplete or irrelevant descriptions, and obtain a cleaned and task-relevant examination report text.

[0135] The inspection report text can be implemented based on rule-based semantic matching and / or model recognition, and this application does not impose any restrictions on this.

[0136] According to the example implementation, the Qwen3-32B model is used to extract the inspection report text.

[0137] In step S430, a first video-text pair is generated based on the third echocardiogram video sample and the examination report text.

[0138] The third echocardiogram video sample and the corresponding examination report text form a video-text pair, denoted as the first video-text pair.

[0139] In step S440, multiple pseudo reports corresponding to the fourth echocardiogram video sample are obtained, and the multiple pseudo reports are filtered to obtain the target pseudo report.

[0140] To broaden the coverage, a pseudo-report is generated for a fourth echocardiogram video sample that does not have an associated text report (e.g., an unpaired echocardiogram video sample from MIMIC-EchoQA). Then, high-quality samples are retained through rule filtering to obtain the target pseudo-report.

[0141] According to the example embodiment, a pseudo report is generated using EchoPrime to distill a fourth echocardiogram video sample.

[0142] In step S450, a second video-text pair is generated based on the fourth echocardiogram video sample and the target pseudo-report.

[0143] The fourth echocardiogram video sample and the corresponding examination report text form a video-text pair, denoted as the second video-text pair.

[0144] Furthermore, to enhance cross-view robustness, each processed target pseudo-report is paired with videos from multiple corresponding views to form multi-view video-report pair samples, which are then added to the second video-text pair.

[0145] In step S460, a third video-text pair is obtained based on a pre-built dataset.

[0146] Integrate existing video-text pairs from pre-built datasets (such as MIMIC-EchoQA) as third video-text pairs.

[0147] In step S470, the corresponding echocardiogram video samples are structurally annotated based on the first video-text pair, the second video-text pair, and the third video-text pair to generate report generation samples.

[0148] The text in the first video-text pair, the second video-text pair, and the third video-text pair is structured and annotated into the corresponding echocardiogram video samples to generate report samples.

[0149] This application embodiment constructs an instruction-based structure for echocardiogram video samples, transforming the samples into a "query-report" instruction format.

[0150] According to some embodiments, refer to Figure 5 In step S150, based on the mixed sample dataset, the initial multi-task model is subjected to periodic-aware reinforcement learning using a pre-constructed periodic-aware composite reward function to obtain a multi-task processing model, which can be specifically implemented through steps S1-S5.

[0151] In step S1, the current sample is extracted from the keyframe localization samples of the mixed sample dataset, wherein the current sample includes input data and labeled output data.

[0152] This application further introduces GRPO (Group Relative Policy Optimization) optimization based on group relative advantage for the keyframe localization task. Specifically, the current sample is first extracted from the keyframe localization samples in the mixed sample dataset.

[0153] In step S2, the input data of the current sample is input into the initial multi-task model, and multiple candidate outputs are sampled based on the output of the initial multi-task model.

[0154] In each policy update, for the given input data From the old strategy (i.e., the initial multi-task model) Sample a set of candidate outputs .

[0155] According to the example implementation, before inputting the current sample's input data into the initial multi-task model, the method further includes: distilling the CoT (thought chain) inference trajectory using a mixed sample dataset and an external strong model to perform a cold start on the policy, enabling the model to learn to output structured results in a "reasoning-response" format.

[0156] In step S3, the composite reward value of multiple candidate outputs is calculated based on the labeled output data of the current sample and the periodic perception composite reward function, and the composite reward value of multiple candidate outputs is standardized to obtain the relative advantage of multiple candidate outputs.

[0157] Using a periodic-aware composite reward function, calculate the composite reward for each candidate output. And within the group, mean-variance standardization was performed to obtain relative advantage. .

[0158] In step S4, the parameters of the initial multi-task model are updated by optimizing the objective using a pre-built group relative strategy based on the relative advantages of multiple candidate outputs.

[0159] The current policy is updated by optimizing the objective using a pre-built group-relative policy.

[0160] According to the example embodiment, the group relative policy optimization objective includes a GRPO objective with KL regularization, that is, to increase the probability of relatively superior outputs within the group through a dominance-weighted policy ratio term, while simultaneously... Constraint policy drift to prevent reward hacking in scenarios with repeated heartbeat cycles. Specifically, Indicates the current policy model. This represents the model after parameter updates.

[0161] The embodiments of this application enable the model to no longer mechanically memorize a single frame number, but instead learn a robust localization strategy that is consistent with multiple periodic candidates, temporal tolerance intervals, and clinical physiological characteristics.

[0162] In step S5, steps S1-S4 are repeated until the preset termination condition is met, and the parameter update result of the last iteration is output as the multi-task processing model.

[0163] The preset termination conditions can be set according to the actual situation, and this application does not impose any restrictions on them.

[0164] According to some embodiments, the periodic-aware composite reward function includes a format reward function, a soft-boundary time precision reward function, a measurement vector similarity reward function, and / or a consistency reward function.

[0165] The format reward function includes: giving a positive reward when the output strictly meets the structured format (e.g., giving the reasoning paragraph first, and then the structured answer field), thereby enhancing readability and parsability.

[0166] The soft-boundary time precision reward function includes: to avoid overly stringent exact matching at high frame rates, and a time period overlap IoU (Intersection over Union) reward, which calculates the overlap between the predicted time period and the labeled time period and gives a reward using a piecewise function, as well as a timestamp / frame number reward. A tolerance radius (related to the period length) is introduced, and a hierarchical reward is calculated based on the minimum normalized distance from the predicted value to the candidate set.

[0167] Furthermore, according to the example embodiment, a hierarchical reward mechanism was designed for both time-based and point-level predictions during the calculation of the soft-boundary time accuracy reward. For time-based predictions, the prediction interval is calculated first. and the marked interval intersection ratio Then, by using upper and lower thresholds and The piecewise function assigns rewards: high overlap receives full marks, medium overlap is assigned a linear score based on IoU, and low overlap receives no reward. For timestamp and frame number prediction, a tolerance radius related to cardiac cycle length is defined. (Take one-sixth of a period span) and use the predicted value to the set of valid candidate anchor points. minimum normalized distance The bias is measured, and then a four-segment hierarchical reward function is used for scoring. This design mitigates the strict one-to-one supervision bias caused by multiple repetitions in ultrasound videos using a periodic perception approach.

[0168] The measurement vector similarity reward function includes: using a specialized measurement module / model to extract actual application scenario measurement vectors (such as inner diameter, wall thickness, area, grayscale texture similarity, etc.) from the prediction frame and the labeled frame, and forming a reward based on the relative error or distance.

[0169] Furthermore, according to the example embodiment, in the calculation of the measurement vector similarity reward function, the clinical measurement vectors related to the task are first extracted from the predicted frame and the labeled frame using the specialty measurement module, and the difference between the two in the physiological quantity space is characterized by the average relative error. Specifically, for geometric / functional indicators, the mean of the relative errors of each measurement quantity is used as the difference measure; for texture-dependent pathology, the cosine similarity of the grayscale images is first calculated, and then converted into an error term. Subsequently, a reward is constructed by back-mapping the error, so that the more consistent the predicted frame and the labeled frame are in clinical measurement, the higher the model receives the reward, thereby avoiding relying solely on temporal index alignment and ignoring the physiological consistency related to the true diagnosis.

[0170] The consistency reward function includes: performing rule-based validation on the logical consistency between time periods, timestamps, and frame numbers in the output; awarding bonuses to mutually consistent field pairs; and suppressing outputs that are "internally contradictory".

[0171] Through the above design, the embodiments of this application achieve the following optimization objectives: shifting from "unique frame index matching" to "periodic equivalence - consistency with actual application scenarios - reviewable structure", thereby significantly reducing the training instability and reward speculation risks caused by periodic ambiguity, and improving the usability of key frame localization and report generation in actual application scenarios.

[0172] To provide a more detailed explanation of the echocardiogram processing method, three specific examples are given below.

[0173] In one embodiment, the model receives echocardiographic video data from EchoMM-120K as the raw echocardiogram video based on pre-trained Qwen2.5-VL-3B weights. The raw echocardiogram video is preprocessed by decoding, frame sampling, and size normalization. Then, the parameters of the trained multi-task processing model are loaded. A professional measurement model is used to estimate the cardiac chamber area / internal diameter sequence of candidate frames for similarity calculation in practical application scenarios.

[0174] The implementation steps include: 1) User input query: "Please locate the end-diastolic and end-systolic keyframes within a representative cardiac cycle in this video."

[0175] 2) The model outputs time periods, two-phase timestamps and frame numbers, and provides evidence-based descriptions such as "maximum filling at end of diastole and minimum chamber size at end of systole".

[0176] If a video has multiple periodically equivalent phases, this application embodiment uses soft matching scoring of candidate sets to avoid misjudgment due to missing a single frame number; at the same time, it selects the phase frame that truly reaches the extreme value based on the similarity reward tendency of the actual application scenario.

[0177] The embodiments of this application also include: an output consistency checker checks whether the two phases fall within the same period segment, whether the timestamp is within the time period, and whether the frame number and timestamp mapping are consistent. If they are inconsistent, resampling or correction is triggered.

[0178] In another embodiment, the model receives echocardiographic video data from EchoMM-120K as raw echocardiographic video based on pre-trained Qwen2.5-VL-3B weights. The raw echocardiographic video is preprocessed by decoding, frame sampling and size normalization. Then, the parameters of the trained multi-task processing model are loaded. The professional measurement model provides auxiliary indicators such as ventricular wall thickness, chamber size or texture features.

[0179] The implementation steps include: 1) Enter the query: "Please locate the end-diastolic and end-systolic keyframes within the single cardiac cycle that best reflects a certain pathological feature".

[0180] 2) The model first predicts the "critical time period" as the periodic level positioning, then provides two-phase frame level positioning, and explains the basis for wall thickness, chamber size and contraction function in the description.

[0181] In this embodiment, the model trained by reinforcement learning, compared to a supervised model, is more likely to output a self-consistent result of "time period-timestamp-frame number," and is less likely to output inconsistent frame pairs across periods under periodic ambiguity. The similarity reward in practical application scenarios suppresses speculative behavior of "only pursuing temporal distance," and avoids the model from selecting the wrong period or a phase without practical application significance in periodic repetition.

[0182] In another embodiment, the model, based on pre-trained Qwen2.5-VL-3B weights, receives echocardiographic video data (which may include multiple views) from EchoMM-120K as the raw echocardiographic video, performs preprocessing on the raw echocardiographic video including decoding, frame sampling, and size normalization; then loads the parameters of the trained multi-task processing model, accepts user input, and outputs a structured report; a consistency checker checks the consistency of the report structure, terminology, and conclusions.

[0183] The implementation steps include: 1) Input command: "Please generate an ultrasound examination report based on this echocardiogram video, describing it in sections according to heart chambers, valves, ventricular wall motion, and conclusions." 2) The model outputs a structured report, including necessary sections and standardized terminology.

[0184] Optionally, in this application embodiment, the keyframe localization result is output as "traceable evidence" along with the report, forming a "report + keyframe evidence" delivery form, which fits the retention and review needs of actual application scenarios.

[0185] In the traditional manual process of existing technology, starting from the input of the original echocardiogram video, the operator sequentially performs (1) observation and selection of key frames, (2) manual measurement and analysis based on the key frames (such as intracavitary diameter, ejection fraction, etc.), (3) comprehensive diagnostic information and complete report writing, and finally produces (4) a report on the actual application scenario containing key frame evidence. The echocardiogram processing method provided in this application embodiment, under the same original video input and text prompts (queries), the trained multi-task processing model first performs multimodal analysis and reasoning, then automatically completes key frame localization (such as end-diastolic ED, end-systolic ES), and further automatically generates a structured diagnostic report; the final output report includes the localized key frames as verifiable visual evidence.

[0186] The echocardiography processing method provided in this application has the following technical effects: First, it can simultaneously output keyframe localization and structured reports, providing verifiable visual evidence for the conclusions and improving the usability in practical application scenarios; Second, by using candidate keyframe sets and soft-matching rewards, it changes "single-point truth value" to "period-equivalent and acceptable", significantly reducing false penalties and unstable optimization; Third, it introduces a similarity reward for practical application scenarios, ensuring that the keyframes selected by the model are consistent with the true phase in terms of measurement indicators, avoiding errors caused by mere time proximity; Fourth, consistency rewards and structured format constraints reduce internal contradictions and unparseable outputs, improving system security and deployability.

[0187] The following describes an apparatus embodiment of this application, which can be used to perform the method embodiment of this application. For details not disclosed in the apparatus embodiment of this application, please refer to the method embodiment of this application.

[0188] Figure 6 A block diagram of an echocardiography processing apparatus according to an exemplary embodiment is shown.

[0189] Figure 6 The apparatus shown can perform the echocardiography processing method described above according to the embodiments of this application.

[0190] like Figure 6 The echocardiogram processing device shown may include: See Figure 6 Referring to the preceding description, the preprocessing unit 610 is used to preprocess the original echocardiogram video to obtain an echocardiogram video sample.

[0191] The data annotation unit 620 is used to annotate keyframes of echocardiogram video samples, generate keyframe positioning samples, and annotate report text of echocardiogram video samples to generate report generation samples.

[0192] Dataset unit 630 is used to construct a hybrid sample dataset by locating samples and generating samples based on keyframes and reports.

[0193] The first training unit 640 is used to perform multi-task autoregressive supervised training on the preset initial model using a mixed sample dataset to obtain the initial multi-task model.

[0194] The second training unit 650 is used to perform periodic-aware reinforcement learning on the initial multi-task model based on the mixed sample dataset and using a pre-built periodic-aware composite reward function to obtain a multi-task processing model.

[0195] The model inference unit 660 is used to input the user-inputted task into the multi-task processing model to obtain the task processing result.

[0196] The device performs functions similar to those described above; other functions are described in the preceding descriptions and will not be repeated here.

[0197] This application discloses an electronic device, including: a processor; and a memory storing a computer program, which, when executed by the processor, causes the processor to execute the above-described instruction generation method.

[0198] For example, refer to Figure 7 , Figure 7 The illustrated electronic device 700 includes a processor 701 and a memory 703. The processor 701 and the memory 703 are connected, for example, via a bus 702. Optionally, the electronic device 700 may also include a transceiver 704. It should be noted that in practical applications, the transceiver 704 is not limited to one type, and the structure of this electronic device 700 does not constitute a limitation on the embodiments of this application.

[0199] Processor 701 may be a CPU (Central Processing Unit), a general-purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. It can implement or execute the various exemplary logic blocks, modules, and circuits described in this application. Processor 701 may also be a combination that implements computational functions, such as including one or more microprocessor combinations, a combination of a DSP and a microprocessor, etc.

[0200] Bus 702 may include a pathway for transmitting information between the aforementioned components. Bus 702 may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus, etc. Bus 702 can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 7 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0201] The memory 703 may be a ROM (Read Only Memory) or other type of static storage device capable of storing static information and instructions, RAM (Random Access Memory) or other type of dynamic storage device capable of storing information and instructions, or an EEPROM (Electrically Erasable Programmable Read Only Memory), CD-ROM (Compact Disc Read Only Memory) or other optical disc storage, optical disc storage (including compressed optical discs, laser discs, optical discs, digital universal optical discs, Blu-ray discs, etc.), magnetic disk storage media or other magnetic storage devices, or any other storage medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but not limited thereto.

[0202] The memory 703 is used to store application code that executes the solution of this application, and its execution is controlled by the processor 701. The processor 701 is used to execute the application code stored in the memory 703 to implement the content shown in the foregoing method embodiments.

[0203] Figure 7 The electronic device shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of this application.

[0204] This application discloses a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, causes the processor to execute an instruction generation method.

[0205] It should be understood that although the steps in the flowcharts of the accompanying figures are shown sequentially as indicated by the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the accompanying figures may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the sub-steps or stages of other steps.

[0206] The above are only some embodiments of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. A method for processing echocardiograms, characterized in that, include: The original echocardiogram video was preprocessed to obtain echocardiogram video samples; The echocardiogram video samples are annotated with keyframes to generate keyframe localization samples, and the echocardiogram video samples are annotated with report text to generate report generation samples. A hybrid sample dataset is constructed based on the keyframe localization samples and the report generation samples. The initial model is trained using the mixed sample dataset to perform multi-task autoregressive supervised training, resulting in an initial multi-task model. Based on the mixed sample dataset, the initial multi-task model is subjected to periodic-aware reinforcement learning using a pre-constructed periodic-aware composite reward function to obtain a multi-task processing model. The user-inputted task is fed into the multi-task processing model to obtain the task processing result; The echocardiogram video samples include a first echocardiogram video sample and a second echocardiogram video sample. Keyframe annotation is performed on the echocardiogram video samples to generate keyframe localization samples, and report text annotation is performed on the echocardiogram video samples to generate report generation samples, including: Based on the first echocardiogram video sample, a first keyframe localization sample is generated for non-pathological query task samples. Based on the second echocardiogram video sample, a second keyframe localization sample is generated for the pathology query task sample. The echocardiogram video samples include a third echocardiogram video sample and a fourth echocardiogram video sample. Keyframe annotation is performed on the echocardiogram video samples to generate keyframe localization samples, and report text annotation is performed on the echocardiogram video samples to generate report generation samples, including: Retrieve the text report corresponding to the third echocardiogram video sample, and perform time window matching and redundancy removal; Extract the content related to the ultrasound examination from the processed text report, and use it as the examination report text. A first video-text pair is generated based on the third echocardiogram video sample and the examination report text. Multiple pseudo-reports corresponding to the fourth echocardiogram video sample are obtained, and the multiple pseudo-reports are filtered to obtain the target pseudo-report; A second video-text pair is generated based on the fourth echocardiogram video sample and the target pseudo-report; Based on a pre-built dataset, obtain third video-text pairs; Based on the first video-text pair, the second video-text pair, and the third video-text pair, the corresponding echocardiogram video samples are structurally annotated to generate the report generation sample.

2. The method according to claim 1, characterized in that, Based on the first echocardiogram video sample, a first keyframe localization sample for non-pathological query task samples is generated, including: Based on the non-pathological query task sample, obtain the structural measurement sequence of key physiological parameters of the first echocardiogram video sample; Based on the structural measurement sequence, key cardiac phases and their corresponding image frames are determined, and multiple cardiac cycles are obtained; Based on the non-pathological query task samples, at least one target cardiac cycle is selected from the plurality of cardiac cycles, and a first keyframe candidate set is generated based on the target cardiac cycle. Based on the elements in the first keyframe candidate set and the non-pathological query task sample, the first echocardiogram video sample is structurally annotated to generate the first query-location keyframe location sample. Based on the elements in the first keyframe candidate set and the descriptive data related to the elements and the non-pathological query task sample, the first echocardiogram video sample is structurally annotated to generate a first location-description keyframe location sample, wherein the first keyframe location sample includes the first query-location keyframe location sample and the first location-description keyframe location sample.

3. The method according to claim 2, characterized in that, Based on the non-pathological query task samples, at least one target cardiac cycle is selected from the plurality of cardiac cycles, and a first keyframe candidate set is generated based on the target cardiac cycle, including: Based on the non-pathological query task sample, at least one representative cardiac cycle and at least one valid but non-representative cardiac cycle are selected from the plurality of cardiac cycles as the target cardiac cycle; Generate keyframes of the target cardiac cycle and store the keyframes of the target cardiac cycle in the first keyframe candidate set.

4. The method according to claim 1, characterized in that, Based on the second echocardiogram video sample, a second keyframe localization sample for the pathology query task is generated, including: Based on the pathology query task sample, obtain the disease type corresponding to the second echocardiogram video sample; Based on the disease type, candidate keyframes are located, and a second keyframe candidate set is constructed based on the candidate keyframes; Based on the elements in the second keyframe candidate set and the pathological query task sample, the second echocardiogram video sample is structurally annotated to generate a second query-location keyframe location sample. Based on the elements in the second keyframe candidate set and the descriptive data related to the elements and the pathological query task sample, the second echocardiogram video sample is structurally annotated to generate a second location-description keyframe location sample, wherein the second keyframe location sample includes the second query-location keyframe location sample and the second location-description keyframe location sample.

5. The method according to claim 4, characterized in that, Based on the disease type, candidate keyframes are located, and a second keyframe candidate set is constructed based on the candidate keyframes, including: Based on the disease type, locate at least one image frame as a first candidate keyframe; Select at least one valid cardiac cycle from the cardiac cycles of the second echocardiographic video sample; Generate the keyframes of the effective cardiac cycle as the second candidate keyframes; Store the first candidate keyframe and the second candidate keyframe into the second keyframe candidate set.

6. The method according to claim 1, characterized in that, Based on the mixed sample dataset, a pre-constructed periodic-aware composite reward function is used to perform periodic-aware reinforcement learning on the initial multi-task model to obtain a multi-task processing model, including: S1: Extract the current sample from the keyframe localization samples in the mixed sample dataset, wherein the current sample includes input data and labeled output data; S2: Input the input data of the current sample into the initial multi-task model, and sample multiple candidate outputs based on the output of the initial multi-task model; S3: Based on the labeled output data of the current sample and the periodic perception composite reward function, calculate the composite reward value of the multiple candidate outputs, and standardize the composite reward value of the multiple candidate outputs to obtain the relative advantages of the multiple candidate outputs; S4: Based on the relative advantages of the multiple candidate outputs, the parameters of the initial multi-task model are updated using the pre-built group relative strategy optimization objective; S5: Repeat steps S1-S4 until the preset termination condition is met, and output the parameter update result of the last iteration as the multi-task processing model.

7. The method according to claim 1, characterized in that, The periodic perception composite reward function includes a format reward function, a soft boundary time accuracy reward function, a measurement vector similarity reward function, and / or a consistency reward function.

8. An echocardiogram processing device, characterized in that, include: The preprocessing unit is used to preprocess the original echocardiogram video to obtain echocardiogram video samples; The data annotation unit is used to annotate keyframes of the echocardiogram video samples to generate keyframe positioning samples, and to annotate report text of the echocardiogram video samples to generate report generation samples. A dataset unit is used to construct a hybrid sample dataset based on the keyframe location samples and the report generation samples; The first training unit is used to perform multi-task autoregressive supervised training on the preset initial model using the mixed sample dataset to obtain the initial multi-task model. The second training unit is used to perform periodic-aware reinforcement learning on the initial multi-task model based on the mixed sample dataset and using a pre-built periodic-aware composite reward function to obtain a multi-task processing model. The model inference unit is used to input the user-inputted task into the multi-task processing model to obtain the task processing result; The data annotation unit is used for: The echocardiogram video samples include a first echocardiogram video sample and a second echocardiogram video sample; Keyframe annotation is performed on the echocardiogram video samples to generate keyframe localization samples, and report text annotation is performed on the echocardiogram video samples to generate report generation samples, including: Based on the first echocardiogram video sample, a first keyframe localization sample is generated for non-pathological query task samples. Based on the second echocardiogram video sample, a second keyframe localization sample is generated for the pathology query task sample. The data annotation unit is used for: The echocardiogram video samples include a third echocardiogram video sample and a fourth echocardiogram video sample; Keyframe annotation is performed on the echocardiogram video samples to generate keyframe localization samples, and report text annotation is performed on the echocardiogram video samples to generate report generation samples, including: Retrieve the text report corresponding to the third echocardiogram video sample, and perform time window matching and redundancy removal; Extract the content related to the ultrasound examination from the processed text report, and use it as the examination report text. A first video-text pair is generated based on the third echocardiogram video sample and the examination report text. Multiple pseudo-reports corresponding to the fourth echocardiogram video sample are obtained, and the multiple pseudo-reports are filtered to obtain the target pseudo-report; A second video-text pair is generated based on the fourth echocardiogram video sample and the target pseudo-report; Based on a pre-built dataset, obtain third video-text pairs; Based on the first video-text pair, the second video-text pair, and the third video-text pair, the corresponding echocardiogram video samples are structurally annotated to generate the report generation sample.

9. An electronic device, characterized in that, include: One or more processors; Memory, used to store one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in any one of claims 1-7.

10. A computer-readable storage medium having a computer program or instructions stored thereon, characterized in that, When the computer program or instructions are executed by a processor, they implement the method as described in any one of claims 1-7.

Images