Key frame screening method and device for angiography sequence, equipment, medium and program product
Through encoding and decoding technology, combined with interactive prompts, the keyframes in the angiography sequence are screened out, which solves the problem of inaccurate screening in the prior art and improves the screening accuracy of keyframes in the angiography sequence.
Patent Information
- Application Number
- CN202411021455.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-29
- Publication Date
- 2024-11-22
AI Technical Summary
The existing keyframe screening methods in angiography sequences cannot guarantee accuracy, especially when the characteristics cannot be quantified or comprehensively judged, which affects the accuracy of the screening.
Sequence characteristics and prompt word characteristics are obtained by encoding the target angiography sequence, and decoded features are decoded using automated encoding and interactive prompting techniques, thereby screening out the target angiography image as keyframes.
It improves the accuracy of keyframe screening, avoids errors when features cannot be quantified and comprehensively judged, and ensures accurate screening of keyframes in angiography sequence.
Smart Images

Figure CN119006945A_ABST
Abstract
Description
Technical Field
[0001] The embodiments of the present invention relate to the field of medical image processing, and in particular to a key frame screening method, device, equipment, medium and program product for angiography sequence. Background Art
[0002] In recent years, the incidence of cardiovascular diseases has shown an upward trend and has attracted more and more attention. Coronary heart disease is a type of cardiovascular disease, which mainly refers to a series of clinical syndromes caused by coronary artery stenosis (i.e., coronary artery stenosis), such as palpitations, shortness of breath, angina pectoris, and myocardial infarction.
[0003] In response to this, some researchers have proposed a technology for detecting coronary artery stenosis based on quantitative flow ratio (QFR). This technology requires first selecting key frames from the digital subtraction angiography (DSA) sequence, and then performing coronary artery stenosis detection based on the key frames. The digital subtraction angiography sequence can be referred to as angiography sequence.
[0004] However, the currently used related solutions cannot guarantee the accuracy of key frame screening and need to be improved. Summary of the invention
[0005] The embodiments of the present invention provide a method, device, equipment, medium and program product for selecting key frames in angiography sequences, which solve the problem that accurate selection of key frames in angiography sequences cannot be guaranteed.
[0006] According to one aspect of the present invention, a key frame screening method for angiography sequence is provided, which may include:
[0007] Encode the target angiography sequence to obtain the sequence feature, and encode the target prompt word to obtain the prompt word feature;
[0008] Decoding sequence features and prompt word features to obtain decoding features;
[0009] According to the decoding characteristics, one or more target angiography images are selected from the target angiography sequence as key frames.
[0010] According to another aspect of the present invention, a key frame screening device for angiography sequence is provided, which may include:
[0011] A coding feature obtaining module is used for encoding a target angiography sequence to obtain a sequence feature, and encoding a target prompt word to obtain a prompt word feature;
[0012] A decoding feature obtaining module is used to decode sequence features and prompt word features to obtain decoding features;
[0013] The key frame screening module is used to screen one or more target angiography images as key frames from the target angiography sequence according to the decoding characteristics.
[0014] According to another aspect of the present invention, there is provided an electronic device, which may include:
[0015] at least one processor; and
[0016] a memory communicatively connected to at least one processor; wherein,
[0017] The memory stores a computer program executable by at least one processor. The computer program is executed by at least one processor so that the at least one processor implements the key frame screening method for angiography sequence provided by any embodiment of the present invention.
[0018] According to another aspect of the present invention, a computer-readable storage medium is provided, on which computer instructions are stored, and the computer instructions are used to enable a processor to implement the key frame screening method for angiography sequences provided by any embodiment of the present invention when executed.
[0019] According to another aspect of the present invention, a computer program product is provided, on which a computer program is stored. When the computer program is executed by a processor, the key frame screening method for angiography sequence provided by any embodiment of the present invention is implemented.
[0020] The technical solution of the embodiment of the present invention obtains sequence features by encoding the target DSA sequence and obtains cue word features by encoding the target cue words, thereby realizing automatic encoding to avoid the situation where some features cannot be quantified and / or all features cannot be quantified when comprehensively judged, and realizing interactive prompts related to key frame screening to improve the accuracy of key frame screening; decoding sequence features and cue word features to obtain at least decoding features that can be used for key frame screening; on this basis, one or more target DSA images are screened out from the target DSA sequence as key frames according to the decoding features. The above technical solution can ensure accurate screening of key frames in the target DSA sequence by utilizing the automatic encoding characteristics and the interactive prompt characteristics.
[0021] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present invention, nor are they intended to limit the scope of the present invention. Other features of the present invention will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS
[0022] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.
[0023] Figure 1 is a flowchart of a key frame screening method provided according to an embodiment of the present invention;
[0024] Figure 2 is a flowchart of an optional example of a key frame screening method provided in an embodiment of the present invention;
[0025] Figure 3 is a flowchart of another key frame screening method provided according to an embodiment of the present invention;
[0026] Figure 4 is a flowchart of another key frame screening method provided according to an embodiment of the present invention;
[0027] Figure 5 is a flowchart of another key frame screening method provided according to an embodiment of the present invention;
[0028] Figure 6 is a flowchart of an optional example of another key frame screening method provided according to an embodiment of the present invention;
[0029] Figure 7 is a structural block diagram of a key frame screening device provided according to an embodiment of the present invention;
[0030] Figure 8 It is a structural schematic diagram of an electronic device for implementing the key frame screening method according to an embodiment of the present invention. DETAILED DESCRIPTION
[0031] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.
[0032] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. The situations of "target", "original", etc. are similar and will not be repeated here. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.
[0033] Before introducing the embodiments of the present invention, an exemplary explanation is given of the currently used related scheme described in the background technology and the reasons why the related scheme cannot guarantee the accurate screening of key frames in the angiography (DSA) sequence, so as to better understand why the embodiments of the present invention can guarantee the accurate screening of key frames.
[0034] For example, currently, some features recommended by clinicians can be converted into quantifiable features, and then keyframes can be screened based on the quantifiable features. However, in actual applications, there are some features that cannot be quantified and / or all features cannot be quantified when comprehensively judged. These situations will affect the accuracy of keyframe screening, and thus the accurate screening of keyframes cannot be guaranteed.
[0035] In response to the above-mentioned technical problems to be solved, the following embodiments propose a technical solution for key frame screening based on a target model, especially a large model. This technical solution utilizes the automatic encoding characteristics and interactive prompt characteristics of the model to solve the above-mentioned technical problems, thereby ensuring the accuracy of key frame screening.
[0036] Next, the technical solution will be explained in detail.
[0037] Figure 1 This is a flowchart of a key frame screening method for angiography sequence provided by an embodiment of the present invention. This embodiment is applicable to the case of screening key frames in a DSA sequence, and is particularly applicable to the case of screening key frames using a model. The method can be executed by a key frame screening device for angiography sequence provided by an embodiment of the present invention, and the device can be implemented by software and / or hardware, and the device can be integrated on an electronic device, and the electronic device can be various user terminals or servers.
[0038] See also Figure 1 The method of the embodiment of the present invention specifically includes the following steps:
[0039] S110, encoding the target angiography sequence to obtain a sequence feature, and encoding the target prompt word to obtain a prompt word feature.
[0040] Among them, the target angiography sequence (i.e., target DSA sequence) can be understood as a sequence obtained after DSA imaging of the target blood vessel. The sequence may include multiple target angiography images (i.e., target DSA images), and one or more of the multiple target DSA images are used as key frames.
[0041] The target DSA sequence is feature encoded to obtain a sequence feature. In combination with the application scenarios that may be involved in the embodiments of the present invention, the sequence feature may be at least one of a global feature, a local feature, and a related motion feature. For example, it may be a corner feature and / or a texture feature. For another example, it may be a clear feature and / or an overlapping feature of the stenosis of the target blood vessel. No specific limitation is made here.
[0042] In practical applications, optionally, the encoding process of the above-mentioned target DSA sequence can be implemented by a sequence encoding module (Sequence Encoder Model), which can be understood as a module for feature encoding of the target DSA sequence. In combination with the application scenarios that may be involved in the embodiments of the present invention, the module can be a visual transformer (Visual Transformer, VIT) module and / or a convolutional neural network (Convolutional Neural Networks, CNN) module. On this basis, it can specifically be a detection (detection) VIT module and / or a residual network (resnet) module, etc., which are not specifically limited here. At this time, the specific type of the encoded sequence feature can be determined by the sequence encoding module during the training process.
[0043] The target prompt word can be understood as a prompt word set for the target DSA sequence, which is used to describe the information that the relevant personnel need to pay attention to. In combination with the subsequent steps, it can be further understood as a prompt word used to prompt the information that needs to be paid attention to for the target DSA sequence during the encoding process. This information can be reflected in the area and / or position, etc., which is related to the actual situation and is not specifically limited here. In combination with the application scenarios that may be involved in the embodiments of the present invention, optionally, the target prompt word can be obtained based on at least one of the following: prompt text, vascular segmentation map (i.e., vascular mask map), marking box, marked pixel points, and optical flow map (i.e., motion feature map). These items can be automatically obtained based on a preset algorithm, or can be obtained through manual operation, which is not limited here.
[0044] The target prompt word is feature encoded to obtain prompt word features. In combination with the application scenarios that may be involved in the embodiments of the present invention, optionally, the encoding process of the target prompt word can be implemented by a prompt word encoding module (Prompt EncoderModel), and the prompt word encoding module can be understood as a module for feature encoding the target prompt word. The network structure of the module can be the same as or different from the network structure of the sequence encoding module, which is related to the actual situation and is not specifically limited here.
[0045] S120, decoding sequence features and prompt word features to obtain decoding features.
[0046] The sequence feature and the prompt word feature are decoded to obtain a decoded feature. In the embodiment of the present invention, the decoded feature can at least be used for key frame screening, so the decoded feature can at least be a feature related to key frame screening, for example, it can be a feature that characterizes the probability that each target DSA image in the target DSA sequence is a key frame, or it can be a feature that characterizes the clarity of the target area where each target DSA image contains a vascular stenosis, etc., which is not specifically limited here.
[0047] In practical applications, optionally, the sequence features and the prompt word features can be decoded separately to obtain decoding features; or the sequence features and the prompt word features can be fused (combined) first to obtain fused features, and then the fused features are decoded to obtain decoding features. Compared with the former, the latter can comprehensively consider both the sequence features and the prompt word features in the decoding process through the fusion operation, which helps to obtain decoding features that are more conducive to key frame screening; and so on, which are not specifically limited here.
[0048] In combination with application scenarios that may be involved in the embodiments of the present invention, optionally, the above-mentioned feature decoding process can be implemented by a decoding module (Decoder Model). Exemplarily, the decoding module can be a decoding module in a segmentation everything model (SegmentAnything Model, SAM).
[0049] S130 , selecting one or more target angiography images from the target angiography sequence as key frames according to the decoding characteristics.
[0050] As mentioned above, the decoding features can be used to perform key frame screening, so one or more target DSA images can be screened out from the target DSA sequence as key frames based on the decoding features.
[0051] On this basis, in order to better understand the above-mentioned various technical solutions as a whole, the following is an exemplary description of them in combination with specific examples. Figure 2, input the target DSA sequence into the sequence encoding module to obtain the sequence features output by the sequence encoding module, and input the target cue word into the cue word encoding module to obtain the cue word features output by the cue word encoding module; then, fuse the sequence features and the cue word features to obtain the fused features; then, input the fused features into the decoding module to obtain the decoding features output by the decoding module, so as to perform the key frame screening task according to the decoding features, that is, to screen out one or more target DSA images as key frames from the target DSA sequence.
[0052] The technical solution of the embodiment of the present invention obtains sequence features by encoding the target DSA sequence and obtains cue word features by encoding the target cue words, thereby realizing automatic encoding to avoid the situation where some features cannot be quantified and / or all features cannot be quantified when comprehensively judged, and realizing interactive prompts related to key frame screening to improve the accuracy of key frame screening; decoding sequence features and cue word features to obtain at least decoding features that can be used for key frame screening; on this basis, one or more target DSA images are screened out from the target DSA sequence as key frames according to the decoding features. The above technical solution can ensure accurate screening of key frames in the target DSA sequence by utilizing the automatic encoding characteristics and the interactive prompt characteristics.
[0053] In an optional technical solution, the decoding feature includes the probability that each target angiography image in the target angiography sequence is a key frame; based on the decoding feature, one or more target angiography images are selected from the target angiography sequence as key frames, including:
[0054] Based on the magnitude of each probability, each target angiography image in the target angiography sequence is reordered to obtain a reordered sequence;
[0055] One or more target angiography images sorted at the target position in the re-sorted sequence are used as key frames.
[0056] Among them, the decoding features include the probability that each target DSA image in the target DSA sequence is a key frame, that is, the decoding features include the probability corresponding to each target DSA image. On this basis, the target DSA images in the target DSA sequence can be reordered based on the size of each probability, for example, reordered in order from large to small or from small to large, to obtain a reordered sequence. Further, one or more target DSA images sorted in the target position in the reordered sequence can be used as key frames, for example, when the reordering is performed in order from large to small, the target position is the first position, and when the reordering is performed in order from small to large, the target position is the last position, thereby achieving rapid and objective screening of key frames.
[0057] In the above technical solution, optionally, the key frame screening task can also be called a ranking task; and the probability can also be called a score.
[0058] Figure 3 Flow chart of another key frame screening method for angiography sequence provided in an embodiment of the present invention. This embodiment is optimized based on the above technical solutions. In this embodiment, optionally, the key frame screening method further includes: for a target angiography image in a target angiography sequence, in response to a marking operation on the target angiography image, obtaining a target prompt word. The explanations of the terms that are the same or corresponding to the above embodiments are not repeated here.
[0059] See also Figure 3 The method of this embodiment may specifically include the following steps:
[0060] S210 . For a target angiography image in a target angiography sequence, in response to a marking operation performed on the target angiography image, a target prompt word is obtained.
[0061] Among them, in combination with the application scenarios that may be involved in the embodiments of the present invention, when reading the target DSA sequence, the relevant personnel can mark at least part of the target DSA images in the target DSA sequence, and the marking operation can be implemented by at least one of a mouse, a keyboard, and a touch device, etc. Exemplarily, the marking operation can be to edit a marking box (i.e., a prompt box) using a mouse to represent the information that the relevant personnel are concerned about. In actual applications, the marking operation is the process of the relevant personnel selecting a key frame.
[0062] For a target DSA image in a target DSA sequence, especially a target DSA image with a marking operation, a target prompt word is obtained by responding to the marking operation on the target DSA image.
[0063] S220, encoding the target angiography sequence to obtain a sequence feature, and encoding the target prompt word to obtain a prompt word feature.
[0064] S230, decoding sequence features and prompt word features to obtain decoding features.
[0065] S240. Filter one or more target angiography images from the target angiography sequence as key frames based on the decoding features.
[0066] The technical solution of the embodiment of the present invention obtains the marking operation of relevant personnel when reading the target DSA sequence through human-computer interaction. Since the marking operation is closely related to key frame screening, the target prompt word can be obtained based on the marking operation subsequently, thereby further improving the accuracy of key frame screening.
[0067] An optional technical solution, in response to a marking operation on a target angiography image, obtains a target prompt word, including:
[0068] In response to a marking operation performed on the target angiography image, a marking frame is obtained, and a target area framed by the marking frame on the target angiography image is determined;
[0069] The target area is identified, the blood vessel information of the regional blood vessels located in the target area is obtained, and the target prompt word is obtained according to the blood vessel information.
[0070] Among them, the marking operation can be an operation of drawing a selection box on the target DSA image. Therefore, by responding to the marking operation, a marking box can be obtained, and then the target area framed by the marking box on the target DSA image can be determined. The target area can also be called the target position. In combination with the application scenarios that may be involved in the embodiments of the present invention, the target area can be the area where the blood vessel is narrowed. Furthermore, the part of the target blood vessel located in the target area is called the regional blood vessel. Then, by identifying the target area, the vascular information of the regional blood vessel can be obtained. The vascular information can be at least one of a vascular segment (i.e., the segment to which the regional blood vessel belongs), a vascular branch (i.e., the branch to which the regional blood vessel belongs), and a vascular stenosis condition (i.e., whether the regional blood vessel is narrowed). Then, based on the vascular information, the target prompt word is obtained.
[0071] The above technical solution obtains a marking frame by analyzing the marking operation, and then accurately obtains the target prompt word by analyzing the blood vessel information of the regional blood vessels in the target area selected by the marking frame.
[0072] Figure 4 It is a flowchart of another key frame screening method for angiography sequence provided in an embodiment of the present invention. This embodiment is optimized based on the above-mentioned technical solutions. In this embodiment, optionally, the above-mentioned key frame screening method includes: using a sequence encoding module to encode the target angiography sequence to obtain a sequence feature; using a prompt word encoding module to encode the target prompt word to obtain a prompt word feature; using a decoding module to decode the sequence feature and the prompt word feature to obtain a decoding feature; wherein the sequence encoding module, the prompt word encoding module and the decoding module are all located in the target model. Among them, the explanations of the terms that are the same or corresponding to the above-mentioned embodiments are not repeated here.
[0073] See also Figure 4 The method of this embodiment may specifically include the following steps:
[0074] S310 , using a sequence encoding module to encode a target angiography sequence to obtain sequence features.
[0075] S320 , using a prompt word encoding module to encode the target prompt word to obtain prompt word features.
[0076] S330, using a decoding module to decode the sequence features and the prompt word features to obtain a decoding feature;
[0077] Among them, the sequence encoding module, the prompt word encoding module and the decoding module are all located in the target model.
[0078] Among them, the target model includes a sequence encoding module, a prompt word encoding module and a decoding module. In practical applications, optionally, the target model can be a large model, which helps to use the interactive prompt characteristics of the large model for key frame screening. On this basis, further optionally, the large model can be a visual large model, and the remaining large models of the camera (such as a language large model, etc.), the visual large model pays more attention to the image, so it is more suitable for key frame screening scenarios, which helps to further improve the accuracy of key frame screening.
[0079] S340 , selecting one or more target angiography images from the target angiography sequence as key frames according to the decoding characteristics.
[0080] The technical solution of the embodiment of the present invention uses a sequence encoding module to encode the target DSA sequence to obtain sequence features, and uses a prompt word encoding module to encode the target prompt word to obtain prompt word features. Firstly, the two encoding modules are used to realize automatic encoding, which can avoid the situation that some features cannot be quantified and / or all features cannot be quantified when comprehensively judged. Secondly, the prompt word encoding module is used to realize interactive prompts related to key frame screening, which can improve the accuracy of key frame screening; the decoding module is used to decode the sequence features and the prompt word features to obtain at least decoding features that can be used for key frame screening; on this basis, one or more target DSA images can be screened out from the target DSA sequence as key frames according to the decoding features. The above technical solution can ensure the accurate screening of key frames in the target DSA sequence by utilizing the automatic encoding characteristics and interactive prompt characteristics of the module.
[0081] Figure 5 It is a flowchart of another key frame screening method for angiography sequence provided in an embodiment of the present invention. This embodiment is optimized based on the above-mentioned technical solutions. In this embodiment, optionally, the target model can be pre-trained by the following steps: obtaining multiple sets of training samples, wherein the training samples include sample angiography sequences, sample prompt words, and key frame labels for sample angiography sequences; based on the multiple sets of training samples, training the original model to obtain the target model. Among them, the explanations of the terms that are the same or corresponding to the above-mentioned embodiments are not repeated here.
[0082] See also Figure 5 The method of this embodiment may specifically include the following steps:
[0083] S410 , obtaining multiple groups of training samples, wherein the training samples include sample angiography sequences, sample prompt words, and key frame labels for the sample angiography sequences.
[0084] The target model at least includes a sequence encoding module, a prompt word encoding module and a decoding module, and the target model is obtained by training the original model, so the sequence encoding module, the prompt word encoding module and the decoding module are modules obtained after training. The target model can be obtained through training in S410 and S420.
[0085] Specifically, multiple groups of training samples are obtained, each of which includes a sample DSA sequence, a sample prompt word, and a key frame label for the sample DSA sequence. The key frame label can be understood as a label that characterizes the sample DSA image belonging to a key frame in the sample DSA sequence.
[0086] S420, training the original model based on multiple groups of training samples to obtain a target model, wherein the target model includes a sequence encoding module, a prompt word encoding module, and a decoding module.
[0087] Among them, the model structure of the original model is the same as the model structure of the target model. The original model can be trained based on multiple groups of training samples. Specifically, the sequence encoding module, prompt word encoding module and decoding module to be trained in the original model can be trained to obtain the target model.
[0088] S430 , using a sequence encoding module to encode the target angiography sequence to obtain sequence features.
[0089] S440 , using a prompt word encoding module to encode the target prompt word to obtain a prompt word feature.
[0090] S450 , using a decoding module to decode the sequence features and the prompt word features to obtain a decoding feature.
[0091] S460: Filter one or more target angiography images from the target angiography sequence as key frames according to the decoding characteristics.
[0092] The technical solution of the embodiment of the present invention realizes the effective training of the target model, specifically the three modules in the target model, which is an important preparatory work for realizing key frame screening.
[0093] An optional technical solution, when the sample prompt words are obtained based on the sample segmentation map of the sample blood vessel in the sample angiography sequence, the original model is trained based on multiple sets of training samples to obtain the target model, including:
[0094] For each group of training samples in the multiple groups of training samples, a sample angiography sequence and a sample prompt word in the training sample are input into the original model, and a first key frame screening feature and a blood vessel segmentation feature output by the original model are obtained;
[0095] According to the first key frame screening feature and the key frame labels in the training sample, as well as the blood vessel segmentation feature and the sample segmentation map, at least the parameters of the sequence encoding module to be trained and the parameters of the prompt word encoding module in the original model are adjusted to obtain the target model.
[0096] Among them, each sample DSA image in the sample DSA sequence corresponds to its own sample segmentation map, and the sample segmentation map can be understood as the result map obtained after the sample blood vessel segmentation of the corresponding sample DSA image. If the sample prompt word is obtained at least based on the sample segmentation map, then in the training process of the original model, the original model can at least output the first key frame screening feature and the blood vessel segmentation feature. The first key frame screening feature can be understood as the feature used to screen out the key frame from the sample DSA sequence, for example, it can be the probability mentioned above; the blood vessel segmentation feature can be understood as the feature used to perform sample blood vessel segmentation on each sample DSA image in the sample DSA sequence. Then, the key frame screening task is performed based on the first key frame screening feature and the key frame label, and the blood vessel segmentation task is performed according to the blood vessel segmentation feature and the sample segmentation map, so as to adjust the parameters of at least the two encoding modules to be trained in the original model to obtain the target model.
[0097] The above technical scheme, during the model training process, can guide the two encoding modules to better perform feature encoding through multimodal tasks (i.e., key frame screening tasks and vascular segmentation tasks) compared to the single-modal task of key frame screening tasks, thereby improving the richness of feature encoding, which helps to assist the key frame screening task in key frame screening, thereby further improving the accuracy of key frame screening.
[0098] Another optional technical solution, when the sample prompt word is obtained based on the vascular stenosis box marked on the sample angiography image in the sample angiography sequence, the original model is trained based on multiple sets of training samples to obtain the target model, including:
[0099] For each group of training samples in the plurality of groups of training samples, a sample angiography sequence and a sample prompt word in the training sample are input into the original model, and a second key frame screening feature and a vascular stenosis detection feature output by the original model are obtained;
[0100] According to the second key frame screening feature and the key frame label in the training sample, as well as the vascular stenosis detection feature and the vascular stenosis frame, at least the parameters of the sequence encoding module to be trained and the parameters of the prompt word encoding module in the original model are adjusted to obtain the target model.
[0101] Among them, the vascular stenosis frame can be understood as a frame marked on the sample DSA image that represents the current vascular stenosis. If the sample prompt word is obtained at least based on the vascular stenosis frame, then during the training process of the original model, the original model can at least output a second key frame screening feature and a vascular stenosis detection feature. The second key frame screening feature can be understood as a feature for screening key frames from the sample DSA sequence; the vascular stenosis detection feature can be understood as a feature for detecting the vascular stenosis of each sample DSA image. Further, the key frame screening task is performed based on the second key frame screening feature and the key frame label, and the vascular stenosis detection task is performed according to the vascular stenosis detection feature and the vascular stenosis frame, or according to the vascular stenosis detection feature and the vascular stenosis area / vascular stenosis position framed by the vascular stenosis frame, so as to adjust the parameters of at least the two encoding modules to be trained in the original model to obtain the target model.
[0102] The above technical scheme, during the model training process, can guide the two encoding modules to better perform feature encoding through multimodal tasks (i.e., key frame screening task and vascular stenosis detection task) compared with the single-modal task of key frame screening task, thereby improving the richness of feature encoding, which helps to assist the key frame screening task in key frame screening, thereby further improving the accuracy of key frame screening.
[0103] In order to better understand the above-mentioned technical solutions as a whole, the following is an exemplary description of them in combination with specific examples. Figure 6 The key frame screening process based on the large model shown in the figure shows the training process of the large model, which is as follows:
[0104] Step 1: Input the sample DSA sequence and use the sequence encoding module to be trained to extract sequence features.
[0105] Step 2: Input interactive prompts, such as sample segmentation maps, vascular stenosis locations, and optical flow maps, and use the prompt word encoding module to be trained to extract prompt word features to prompt the subsequent decoding module to focus on these areas.
[0106] Step 3: Fuse the sequence features obtained in step 1 and the prompt word features obtained in step 2, and input the fused features obtained so far into the decoding module to be trained to obtain decoding features, and then perform multimodal tasks (i.e., key frame screening task + vascular segmentation task + vascular stenosis detection task) based on the decoding features, so as to adjust the parameters of at least two encoding modules and obtain a large model.
[0107] On this basis, the reasoning process of the above large model is as follows:
[0108] Step 1: Input the target DSA sequence from which key frames need to be extracted.
[0109] Step 2: Input the obtained blood vessel segmentation result and the location of blood vessel stenosis.
[0110] Step 3: The target DSA image with the highest score is the key frame.
[0111] In the above example, multimodal tasks are used to effectively ensure the training effect of the large model. On this basis, the automatic encoding characteristics and interactive prompt characteristics of the large model are used to ensure the accurate screening of key frames.
[0112] Figure 7 This is a structural block diagram of a key frame screening device for angiography sequences provided in an embodiment of the present invention. The device is used to execute the key frame screening method for angiography sequences provided in any of the above embodiments. The device and the key frame screening method for angiography sequences in the above embodiments belong to the same inventive concept. For details not described in detail in the embodiment of the key frame screening device for angiography sequences, please refer to the embodiment of the key frame screening method for angiography sequences. Figure 7 The device may specifically include: a coding feature obtaining module 510, a decoding feature obtaining module 520 and a key frame screening module 530. Among them,
[0113] The encoding feature obtaining module 510 is used to encode the target angiography sequence to obtain the sequence feature, and encode the target prompt word to obtain the prompt word feature;
[0114] A decoding feature obtaining module 520 is used to decode the sequence feature and the prompt word feature to obtain a decoding feature;
[0115] The key frame screening module 530 is used to screen one or more target angiography images from the target angiography sequence as key frames according to the decoding characteristics.
[0116] Optionally, the target prompt word is obtained based on at least one of the following: prompt text, a blood vessel segmentation map, a marked box, marked pixels, and an optical flow map.
[0117] Based on any of the above technical solutions, optionally, the key frame screening device further includes:
[0118] The target cue word obtaining module is used to obtain a target cue word for a target angiography image in a target angiography sequence in response to a marking operation performed on the target angiography image.
[0119] On this basis, optionally, the target prompt word obtaining module includes:
[0120] a target region determination unit, for obtaining a marking frame in response to a marking operation on the target angiography image, and determining a target region framed by the marking frame on the target angiography image;
[0121] The target prompt word obtaining unit is used to identify the target area, obtain blood vessel information of regional blood vessels located in the target area, and obtain the target prompt word according to the blood vessel information.
[0122] On this basis, optionally, the blood vessel information includes at least one of a blood vessel segment, a blood vessel branch, and a blood vessel stenosis condition.
[0123] Optionally, the decoding feature includes the probability that each target angiography image in the target angiography sequence is a key frame; the key frame screening module 530 is specifically used to:
[0124] Based on the magnitude of each probability, each target angiography image in the target angiography sequence is reordered to obtain a reordered sequence;
[0125] One or more target angiography images sorted at the target position in the re-sorted sequence are used as key frames.
[0126] Optionally, the decoding feature obtaining module 520 is specifically used for:
[0127] Fuse the sequence features and the prompt word features to obtain fused features;
[0128] Decode the fused features to obtain the decoded features.
[0129] Optionally, the coding feature obtaining module 510 includes:
[0130] A sequence feature obtaining unit is used to encode the target angiography sequence using the sequence encoding module to obtain the sequence feature; and a prompt word feature obtaining unit is used to encode the target prompt word using the prompt word encoding module to obtain the prompt word feature;
[0131] The decoding feature obtaining module 520 is specifically used for:
[0132] The decoding module is used to decode the sequence features and the prompt word features to obtain the decoding features, wherein the sequence encoding module, the prompt word encoding module and the decoding module are all located in the target model.
[0133] On this basis, optionally, the target model is pre-trained through the following modules:
[0134] A training sample acquisition module, used to acquire multiple groups of training samples, wherein the training samples include sample angiography sequences, sample prompt words, and key frame labels for the sample angiography sequences;
[0135] The target model acquisition module is used to train the original model based on multiple groups of training samples to obtain the target model.
[0136] On this basis, optionally, when the sample prompt words are obtained based on the sample segmentation map of the sample blood vessel in the sample angiography sequence, the target model obtains a module, which is specifically used for:
[0137] For each group of training samples in the multiple groups of training samples, a sample angiography sequence and a sample prompt word in the training sample are input into the original model, and a first key frame screening feature and a blood vessel segmentation feature output by the original model are obtained;
[0138] According to the first key frame screening feature and the key frame labels in the training sample, as well as the blood vessel segmentation feature and the sample segmentation map, at least the parameters of the sequence encoding module to be trained and the parameters of the prompt word encoding module in the original model are adjusted to obtain the target model.
[0139] Alternatively, when the sample prompt word is obtained based on the vascular stenosis box marked on the sample angiography image in the sample angiography sequence, the target model obtaining module is specifically used for:
[0140] For each group of training samples in the plurality of groups of training samples, a sample angiography sequence and a sample prompt word in the training sample are input into the original model, and a second key frame screening feature and a vascular stenosis detection feature output by the original model are obtained;
[0141] According to the second key frame screening feature and the key frame label in the training sample, as well as the vascular stenosis detection feature and the vascular stenosis frame, at least the parameters of the sequence encoding module to be trained and the parameters of the prompt word encoding module in the original model are adjusted to obtain the target model.
[0142] Alternatively, the target model is a large model.
[0143] On this basis, optionally, the large model is a visual large model.
[0144] The key frame screening device for angiography sequence provided by the embodiment of the present invention encodes the target DSA sequence to obtain sequence features and encodes the target prompt words to obtain prompt word features through the encoding feature acquisition module, thereby realizing automatic encoding to avoid the situation that some features cannot be quantified and / or all features cannot be quantified when comprehensively judged, and realizing interactive prompts related to key frame screening to improve the accuracy of key frame screening; through the decoding feature acquisition module, the sequence features and the prompt word features are decoded to obtain decoding features that can at least be used for key frame screening; on this basis, through the key frame screening module, one or more target DSA images are screened out from the target DSA sequence as key frames according to the decoding features. The above device can ensure accurate screening of key frames in the target DSA sequence by utilizing the automatic encoding characteristics and the interactive prompt characteristics.
[0145] The key frame screening device for angiography sequence provided by the embodiment of the present invention can execute the key frame screening method for angiography sequence provided by any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.
[0146] It is worth noting that in the above-mentioned embodiment of the key frame screening device for angiography sequence, the various units and modules included are only divided according to functional logic, but are not limited to the above-mentioned division, as long as the corresponding functions can be achieved; in addition, the specific names of the functional units are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of the present invention.
[0147] Figure 8 A schematic diagram of the structure of an electronic device 10 that can be used to implement an embodiment of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices (such as helmets, glasses, watches, etc.) and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or required herein.
[0148] like Figure 8As shown, the electronic device 10 includes at least one processor 11, and a memory connected to the at least one processor 11, such as a read-only memory (ROM) 12, a random access memory (RAM) 13, etc., wherein the memory stores a computer program that can be executed by at least one processor, and the processor 11 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 12 or the computer program loaded from the storage unit 18 to the random access memory (RAM) 13. In the RAM 13, various programs and data required for the operation of the electronic device 10 can also be stored. The processor 11, the ROM 12, and the RAM 13 are connected to each other through a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.
[0149] A number of components in the electronic device 10 are connected to the I / O interface 15, including: an input unit 16, such as a keyboard, a mouse, etc.; an output unit 17, such as various types of displays, speakers, etc.; a storage unit 18, such as a disk, an optical disk, etc.; and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.
[0150] The processor 11 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The processor 11 executes the various methods and processes described above, such as a key frame screening method for angiography sequences.
[0151] In some embodiments, the key frame screening method for angiography sequences may be implemented as a computer program, which is tangibly contained in a computer-readable storage medium, such as a storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 10 via the ROM 12 and / or the communication unit 19. When the computer program is loaded into the RAM 13 and executed by the processor 11, one or more steps of the key frame screening method for angiography sequences described above may be performed. Alternatively, in other embodiments, the processor 11 may be configured to perform the key frame screening method for angiography sequences in any other appropriate manner (e.g., by means of firmware).
[0152] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.
[0153] Computer programs for implementing the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that when the computer program is executed by the processor, the functions / operations specified in the flow chart and / or block diagram are implemented. The computer program may be executed entirely on the machine, partially on the machine, partially on the machine as a stand-alone software package and partially on a remote machine, or entirely on a remote machine or server.
[0154] In the context of the present invention, a computer-readable storage medium may be a tangible medium that may contain or store a computer program for use by or in combination with an instruction execution system, device or equipment. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or equipment, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.
[0155] To provide interaction with a user, the systems and techniques described herein may be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or trackball) through which the user can provide input to the electronic device. Other types of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form (including acoustic input, voice input, or tactile input).
[0156] The systems and techniques described herein may be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.
[0157] A computing system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The client and server relationship is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services.
[0158] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in the present invention can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solution of the present invention can be achieved, and this document does not limit this.
[0159] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.
Claims
1. A key frame screening method for angiography sequence, characterized in that: include: Encode the target angiography sequence to obtain the sequence feature, and encode the target prompt word to obtain the prompt word feature; Decoding the sequence feature and the prompt word feature to obtain a decoding feature; According to the decoding feature, one or more target angiography images are selected from the target angiography sequence as key frames.
2. The method according to claim 1, characterized in that The target prompt word is obtained based on at least one of the following: prompt text, blood vessel segmentation map, marking box, marking pixel points and optical flow map.
3. The method according to claim 1 or 2, characterized in that: Also includes: For a target angiography image in the target angiography sequence, the target prompt word is obtained in response to a marking operation performed on the target angiography image.
4. The method according to claim 3, characterized in that: The step of obtaining the target prompt word in response to the marking operation on the target angiography image comprises: In response to a marking operation on the target angiography image, a marking frame is obtained, and a target area framed by the marking frame on the target angiography image is determined; The target area is identified, blood vessel information of regional blood vessels located in the target area is obtained, and the target prompt word is obtained according to the blood vessel information.
5. The method according to claim 4, characterized in that The blood vessel information includes at least one of a blood vessel segment, a blood vessel branch, and a blood vessel stenosis condition.
6. The method according to claim 1, characterized in that The decoded features include probabilities that each target angiography image in the target angiography sequence is the key frame; The step of selecting one or more target angiography images as key frames from the target angiography sequence according to the decoding feature includes: Reordering the target angiography images in the target angiography sequence based on the magnitude of each of the probabilities to obtain a reordered sequence; One or more target angiography images arranged at a target position in the re-arranged sequence are used as key frames.
7. The method according to claim 1, characterized in that The decoding of the sequence feature and the prompt word feature to obtain a decoding feature includes: Fusing the sequence feature and the prompt word feature to obtain a fused feature; The fused features are decoded to obtain decoded features.
8. The method according to claim 1, characterized in that The method comprises: Using a sequence encoding module to encode the target angiography sequence to obtain sequence features; Encode the target prompt word using the prompt word encoding module to obtain the prompt word feature; Decoding the sequence feature and the prompt word feature using a decoding module to obtain a decoding feature; Among them, the sequence encoding module, the prompt word encoding module and the decoding module are all located in the target model.
9. The method according to claim 8, characterized in that The target model is pre-trained through the following steps: Acquire multiple groups of training samples, wherein the training samples include sample angiography sequences, sample prompt words, and key frame labels for the sample angiography sequences; Based on the multiple groups of training samples, the original model is trained to obtain the target model.
10. The method according to claim 9, characterized in that When the sample prompt words are obtained based on the sample segmentation map of the sample blood vessel in the sample angiography sequence, the original model is trained based on the multiple groups of training samples to obtain the target model, including: For each group of training samples in the plurality of groups of training samples, a sample angiography sequence and a sample prompt word in the training sample are input into an original model, and a first key frame screening feature and a blood vessel segmentation feature output by the original model are obtained; According to the first key frame screening feature and the key frame labels in the training sample, as well as the blood vessel segmentation feature and the sample segmentation map, at least the parameters of the sequence encoding module to be trained and the parameters of the prompt word encoding module in the original model are adjusted to obtain the target model.
11. The method according to claim 9, characterized in that When the sample prompt words are obtained based on the vascular stenosis frame marked on the sample angiography image in the sample angiography sequence, the method of training the original model based on the multiple groups of training samples to obtain the target model includes: For each group of training samples in the plurality of groups of training samples, a sample angiography sequence and a sample prompt word in the training sample are input into the original model, and a second key frame screening feature and a vascular stenosis detection feature output by the original model are obtained; According to the second key frame screening feature and the key frame label in the training sample, as well as the vascular stenosis detection feature and the vascular stenosis frame, at least the parameters of the sequence encoding module to be trained and the parameters of the prompt word encoding module in the original model are adjusted to obtain the target model.
12. The method according to claim 8, characterized in that The target model is a large model.
13. The method according to claim 12, characterized in that The macro model is a visual macro model.
14. A key frame screening device for angiography sequence, characterized in that: include: A coding feature obtaining module is used for encoding a target angiography sequence to obtain a sequence feature, and encoding a target prompt word to obtain a prompt word feature; A decoding feature obtaining module, used for decoding the sequence feature and the prompt word feature to obtain a decoding feature; The key frame screening module is used to screen one or more target angiography images from the target angiography sequence as key frames according to the decoding characteristics.
15. An electronic device, characterized in that: include: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor to enable the at least one processor to perform the key frame screening method for angiography sequence according to any one of claims 1 to 13.
16. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the key frame screening method for angiography sequences as described in any one of claims 1 to 13 when executed.
17. A computer program product, characterized in that The computer program product comprises a computer program, which, when executed by a processor, implements the key frame screening method for angiography sequences according to any one of claims 1 to 13.
Citation Information
Cited By
Reference frame determination method and device, electronic equipment and computer storage medium
CN121305256A
