Image segmentation method and system based on parallel network framework and dynamic fusion, terminal and storage medium

Abstract

The invention discloses an image segmentation method and system based on a parallel network framework and dynamic fusion, a terminal and a storage medium, and the method comprises the steps: obtaining an original feature of a target image set, inputting the original feature into a parallel coding network, carrying out the dual-path convolution operation, obtaining a plurality of gating features, and constructing a spatial feature; performing global enhancement processing on the spatial features to obtain single-mode representation; a prediction segmentation result is generated after fusion; and constructing a plurality of loss functions to train the model, thereby outputting a segmentation result of the target image set. According to the method, a plurality of parallel encoder branches are arranged to fully learn own characteristics, the defect of insufficient fusion information caused by mode missing can be effectively avoided, an accurate image segmentation result is generated when the modes are missing, and when multiple modes are complete, each branch can fully extract the characteristics of the modes, and the overall image segmentation performance is enhanced.

Description

Technical Field

[0001] The present invention relates to the technical field of three-dimensional image segmentation, and in particular to an image segmentation method, system, terminal and computer-readable storage medium based on a parallel network framework and dynamic fusion. Background Art

[0002] With the rapid development of imaging technology, magnetic resonance imaging (MRI) plays a vital role in brain tumor image segmentation.

[0003] However, in actual scenarios, due to various reasons (such as differences in scanning equipment or protocols, patient allergies to contrast agents, imaging time or cost limitations, image damage, etc.), it is not always possible to obtain complete data of all modalities. If the missing modalities are simply regarded as uninformative or the relevant cases are directly deleted, the generalization ability of the deep learning model will be greatly weakened, thereby affecting the accurate segmentation of brain tumor images.

[0004] Therefore, the prior art still needs to be improved and developed. Summary of the invention

[0005] The main purpose of the present invention is to provide an image segmentation method, system, terminal and computer-readable storage medium based on a parallel network framework and dynamic fusion, aiming to solve the problems in the prior art of insufficient global dependence, high computational cost and lack of adaptability in the application process of the brain tumor segmentation modeling process for missing modalities.

[0006] To achieve the above object, the present invention provides an image segmentation method based on a parallel network framework and dynamic fusion, and the image segmentation method based on a parallel network framework and dynamic fusion comprises the following steps: Acquire multiple original features of the target image set, input all the original features into a parallel network framework for a two-way convolution operation, obtain the gated features corresponding to the original features of each sample, and construct the spatial features corresponding to each sample according to each gated feature; Performing global enhancement processing on each of the spatial features using a state space model, and obtaining all unimodal representations of each of the samples; Inputting all the single-modal representations into a dynamic shared model to obtain a plurality of predicted segmentation results of the target image set; Obtaining the original segmentation results of the target image set, constructing the original classification probability and the predicted classification probability, constructing a balance loss term according to the original segmentation results and the plurality of predicted segmentation results, and constructing a divergence loss term according to the original classification probability and the predicted classification probability; Constructing a dependency relationship between the distribution probability under all modalities and the distribution probability under the missing modality, extracting the deepest potential features of the distribution probability under all modalities and the distribution probability under the missing modality according to the dependency relationship, and constructing a mutual information loss term according to the deepest potential features; The parallel network framework is trained using the balance loss term, the divergence loss term and the mutual information loss term to obtain a target parallel network framework, and a final segmentation result of the target image set is output through the target parallel network framework.

[0007] Optionally, the image segmentation method based on a parallel network framework and dynamic fusion, wherein the step of acquiring multiple original features of a target image set, inputting all the original features into a parallel network framework for a two-way convolution operation, obtaining a gated feature corresponding to the original feature of each sample, and constructing a spatial feature corresponding to each sample according to each gated feature, specifically includes: Obtaining the original features corresponding to each sample in the target image set, and inputting all the original features into multiple encoders in the parallel network framework respectively; The local features corresponding to the original features of each sample are extracted by the gated convolution module of each encoder, and a two-way convolution operation is performed on the multiple original features according to each local feature to output the corresponding gated features: ; in, Indicates The gated features output by the layer encoder, Indicates input to Features of the gated convolutional modules in the layer encoder, represents the gated convolution module, represents a nonlinear activation function, express The convolution kernel, express The convolution kernel, Indicates the multiplication of corresponding elements; All the gated features are input into the state space model in the parallel network framework, and the state space model performs residual connection on each pair of the gated features and the features output by the state space model to obtain the corresponding spatial features: ; in, Indicates The spatial features output by the layer encoder, Represents a state-space model.

[0008] Optionally, the image segmentation method based on parallel network framework and dynamic fusion, wherein the state space model is used to perform global enhancement processing on each of the spatial features and obtain all single-modal representations of each of the samples, specifically includes: Inputting samples corresponding to each of the spatial features into multiple state space models of the parallel network framework in parallel for analysis; Each of the state-space models analyzes the feature connection relationships between different levels of a single modality in each of the samples, and constructs global dependency information corresponding to each modality in each sample based on all of the feature connection relationships; Obtaining the global information enhancement channel constructed respectively by each of the state space models, and constructing a feature sequence of each mode in each of the samples according to each of the spatial features; Inputting all the feature sequences in each sample into the corresponding global information enhancement channel to update the corresponding global dependency information in real time, and performing global enhancement processing on each of the spatial features according to the real-time updated global dependency information; The missing modality information input by the user is obtained, and a single modality representation corresponding to each sample is obtained according to the missing modality information.

[0009] Optionally, the image segmentation method based on parallel network framework and dynamic fusion, wherein all the single-modal representations are input into a dynamic shared model to obtain multiple predicted segmentation results of the target image set, specifically includes: Inputting all the single modality representations and the missing modality information into a dynamic sharing model, and constructing an available modality set in the dynamic sharing model according to all the single modality representations and the missing modality information: ; ; in, Indicates The sample A unimodal representation, Indicates the available modal index, Indicates the number of modes; The dynamic sharing model determines multiple missing modalities of each sample according to the available modality set, and performs fusion processing on the available modality set to obtain the predicted segmentation result of the available modality set under each missing modality: ; in, Indicates The predicted segmentation results of samples in available modalities.

[0010] Optionally, the image segmentation method based on a parallel network framework and dynamic fusion, wherein the obtaining of the original segmentation results of the target image set, constructing the original classification probability and the predicted classification probability, and constructing a balance loss term according to the original segmentation results and the plurality of predicted segmentation results, and constructing a divergence loss term according to the original classification probability and the predicted classification probability, specifically includes: Obtaining the original segmentation results and the original classification results of the target image set, constructing the original classification probability according to the original segmentation results and the original classification results, and constructing the predicted classification probability according to all the predicted segmentation results and the original classification results: ; ; ; ; ; in, represents the original classification probability, represents the predicted classification probability, represents the gamma function, Indicates the number of categories in the original classification result, Parameters representing the original classification probabilities, , and Represents the first, second and The parameters of the original classification probability, Indicates The parameters of the original classification probability, denoting the parameter of the predicted classification probability, , and Represents the first, second and parameters to predict the classification probability, Indicates standard dimensional simplex, Indicates parameters to predict the classification probability, Represents the predicted classification result in the predicted classification probability, Represents the predicted classification result of the first category in the predicted classification probability, Represents the predicted classification result of the second category in the predicted classification probability, Represents the predicted classification probability The predicted classification results of the categories, Represents the predicted classification probability The predicted classification results of the categories, represents the original classification result in the original classification probability, Represents the original classification result of the first category in the original classification probability, Represents the original classification result of the second category in the original classification probability, Represents the original classification probability The original classification results of the categories, Represents the original classification probability The original classification results of the categories; Analyze the overlap between the original segmentation result and the plurality of predicted segmentation results to construct a balanced loss term: ; in, represents the balance loss term, Indicates the original segmentation result The true label of each pixel, Indicates the predicted segmentation result The predicted probability of pixels; A divergence loss term is constructed based on the original classification probability and the predicted classification probability: ; ; ; in, and represents the divergence loss term, Indicates the difference, and The relationship between them is conjugate exponents. , and All represent hyperparameters, represents the original classification probability, represents the predicted classification probability, represents the original segmentation result, Represents the predicted segmentation result.

[0011] Optionally, the image segmentation method based on a parallel network framework and dynamic fusion, wherein the dependency relationship between the distribution probability under all modalities and the distribution probability under the missing modalities is constructed, the deepest potential features of the distribution probability under all modalities and the distribution probability under the missing modalities are extracted according to the dependency relationship, and the mutual information loss term is constructed according to the deepest potential features, specifically including: Constructing the distribution probability under all modalities and the distribution probability under the missing modalities according to the available modality set and all the missing modalities of each sample; Extracting the true entropy of the distribution probability under the full modality and the predicted entropy of the distribution probability under the missing modality from the parallel network framework, and quantifying the dependency between the distribution probability under the full modality and the distribution probability under the missing modality according to the true entropy and the predicted entropy: ; in, Indicates dependency. represents the true entropy, represents the prediction entropy, represents the distribution probability under all modes, represents the probability of distribution under the missing mode; According to the dependency relationship and the true entropy, the deepest potential features of the distribution probability under the full modality and the distribution probability under the missing modality are extracted to obtain the joint probability distribution between the distribution probability under the full modality and the distribution probability under the missing modality, and the mutual information loss term is constructed according to the joint probability distribution: ; ; in, Denotes the random variables P and The joint probability distribution of represents the mutual information loss term, It represents the probability of P under the condition that Q occurs.

[0012] Optionally, the image segmentation method based on the parallel network framework and dynamic fusion, wherein the use of the balance loss term, the divergence loss term and the mutual information loss term to train the parallel network framework to obtain a target parallel network framework, and outputting the final segmentation result of the target image set through the target parallel network framework, specifically includes: A total loss function is constructed based on the balance loss term, the divergence loss term and the mutual information loss term: ; in, represents the total loss function, represents the balance loss term, represents the mutual information loss term, represents the divergence loss term, express The weight adjustment coefficient, express The weight adjustment coefficient of Inputting the total loss function into the parallel network framework for training to obtain a target parallel network framework; The target image set is segmented through the decoding network in the target parallel network framework, and the final segmentation result is output.

[0013] In addition, to achieve the above-mentioned purpose, the present invention also provides an image segmentation system based on a parallel network framework and dynamic fusion, wherein the image segmentation system based on a parallel network framework and dynamic fusion comprises: A spatial feature construction module is used to obtain multiple original features of the target image set, input all the original features into a parallel network framework for a two-way convolution operation, obtain a gated feature corresponding to the original feature of each sample, and construct a spatial feature corresponding to each sample according to each gated feature; A modality extraction module, used for performing global enhancement processing on each of the spatial features using a state space model, and obtaining all single modality representations of each of the samples; A result prediction module, used for inputting all the single-modal representations into a dynamic shared model to obtain a plurality of predicted segmentation results of the target image set; A first loss term construction module is used to obtain the original segmentation result of the target image set, construct the original classification probability and the predicted classification probability, and construct a balance loss term according to the original segmentation result and the plurality of predicted segmentation results, and construct a divergence loss term according to the original classification probability and the predicted classification probability; A second loss term construction module is used to construct a dependency relationship between the distribution probability under the full modality and the distribution probability under the missing modality, extract the deepest potential features of the distribution probability under the full modality and the distribution probability under the missing modality according to the dependency relationship, and construct a mutual information loss term according to the deepest potential features; A model training module is used to train the parallel network framework using the balance loss term, the divergence loss term and the mutual information loss term to obtain a target parallel network framework, and output a final segmentation result of the target image set through the target parallel network framework.

[0014] In addition, to achieve the above-mentioned purpose, the present invention also provides a terminal, wherein the terminal includes: a memory, a processor, and an image segmentation program based on a parallel network framework and dynamic fusion stored in the memory and run on the processor, and when the image segmentation program based on a parallel network framework and dynamic fusion is executed by the processor, the steps of the image segmentation method based on a parallel network framework and dynamic fusion as described above are implemented.

[0015] In addition, to achieve the above-mentioned objectives, the present invention also provides a computer-readable storage medium, wherein the computer-readable storage medium stores an image segmentation program based on a parallel network framework and dynamic fusion, and when the image segmentation program based on a parallel network framework and dynamic fusion is executed by a processor, the steps of the image segmentation method based on a parallel network framework and dynamic fusion as described above are implemented.

[0016] In the present invention, multiple original features of the target image set are obtained, all of the original features are input into a parallel coding network for a two-way convolution operation to obtain gated features corresponding to multiple samples, and spatial features corresponding to each of the samples are constructed based on all of the gated features; each of the spatial features is globally enhanced using a state-space model to obtain multiple unimodal representations of the multiple samples; all of the unimodal representations are input into a dynamic shared model to obtain multiple predicted segmentation results of the target image set; the original segmentation results of the target image set are obtained, the original classification probability and the predicted classification probability are constructed, and a balanced loss term is constructed based on the original segmentation results and the predicted segmentation results, and a divergence loss term is constructed based on the original classification probability and the predicted classification probability; a dependency relationship between the original classification probability and the predicted classification probability is constructed, and the true posterior of the predicted classification probability is estimated based on the dependency relationship to obtain a mutual information loss term; the parallel coding network is trained using the balanced loss term, the divergence loss term and the mutual information loss term to obtain a target parallel coding network, and the final segmentation result of the target image set is output through the target parallel coding network. The present invention sets multiple parallel encoder branches to fully learn their own features, which can effectively avoid the disadvantage of insufficient fusion information caused by missing modalities and produce accurate image segmentation results when the modality is missing. When the multi-modality is complete, each branch can fully extract the features of the modality and enhance the overall image segmentation performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 It is a flow chart of a preferred embodiment of the image segmentation method based on a parallel network framework and dynamic fusion of the present invention; Figure 2 It is a schematic diagram of a parallel network framework of a preferred embodiment of an image segmentation method based on a parallel network framework and dynamic fusion of the present invention; Figure 3 It is a schematic diagram of image segmentation results of a preferred embodiment of the image segmentation method based on a parallel network framework and dynamic fusion of the present invention; Figure 4 It is a schematic diagram of the whole tumor area segmentation result of a preferred embodiment of the image segmentation method based on the parallel network framework and dynamic fusion of the present invention; Figure 5It is a schematic diagram of the tumor core area segmentation result of a preferred embodiment of the image segmentation method based on the parallel network framework and dynamic fusion of the present invention; Figure 6 It is a schematic diagram of enhanced tumor segmentation results of a preferred embodiment of the image segmentation method based on a parallel network framework and dynamic fusion of the present invention; Figure 7 It is a cardiac image segmentation result diagram of a preferred embodiment of the image segmentation method based on a parallel network framework and dynamic fusion of the present invention; Figure 8 It is a conventional scanning segmentation result diagram of a preferred embodiment of the image segmentation method based on a parallel network framework and dynamic fusion of the present invention; Fig. 9 It is an abnormal state segmentation result diagram of a preferred embodiment of the image segmentation method based on a parallel network framework and dynamic fusion of the present invention; Fig.10 It is a specific segmentation result diagram of a preferred embodiment of the image segmentation method based on a parallel network framework and dynamic fusion of the present invention; Fig.11 It is a structural diagram of a preferred embodiment of the image segmentation system based on a parallel network framework and dynamic fusion of the present invention; Fig.12 It is a structural diagram of a preferred embodiment of the terminal of the present invention. DETAILED DESCRIPTION

[0018] In order to make the purpose, technical solution and advantages of the present invention clearer and more specific, the present invention is further described in detail below with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0019] The image segmentation method based on parallel network framework and dynamic fusion described in the preferred embodiment of the present invention is as follows: Figure 1 As shown, the image segmentation method based on parallel network framework and dynamic fusion includes the following steps: Step S10, obtaining multiple original features of the target image set, inputting all the original features into a parallel network framework for a two-way convolution operation, obtaining the gated features corresponding to the original features of each sample, and constructing the spatial features corresponding to each sample according to each gated feature.

[0020] The structure of the robust missing modality MRI segmentation method framework (parallel network framework) based on Mamba (selective state space model) state space modeling and information theory standards proposed in this embodiment is as follows: Figure 2 As shown (where Figure 2 In Indicates Layer encoder, Indicates Layer encoder), where the Mamba-based encoder extracts features from multiple modalities ( Respectively represent the characteristics of the mode, among which, is the mean of these features), it can be used to construct the subsequent mutual information loss term (That is, The specific process is described in the following steps). The Mamba-based decoder contains two activation functions, softmax and softplus. After decoding the features through these two activation functions, the outputs are used to construct the balanced loss term. (That is, ) segmentation features (i.e. ) and construct the divergence loss term (That is, )’s classification features (i.e. ), and then use the DS sharing mechanism (Dynamic Sharing, DS) to perform weighted average fusion on the segmentation results of the available modalities to construct the predicted labels of the output target image set ( ), the true label ( ), and for building The fusion features of ) and for building The fusion features of ).

[0021] Among them, the target image set (such as Figure 2 In ) is input into the encoder of the parallel network framework for convolution operation, and the encoder extracts features from each modality through independent sub-encoders. Each encoder extracts features according to the different modalities of each sample and generates a preliminary feature representation. The parallel network framework can handle the combination of different modal data, thereby coping with the challenge of modality loss and ensuring that each modal information is independently and effectively utilized.

[0022] Specifically, the original features corresponding to each sample in the target image set are obtained, and all the original features are respectively input into multiple encoders in the parallel network framework; the local features corresponding to the original features of each sample are extracted through the gated convolution module of each encoder, and a two-way convolution operation is performed on the multiple original features according to each local feature, and the corresponding gated features are output: ; in, Indicates The gated features output by the layer encoder, Indicates input to Features of the gated convolutional modules in the layer encoder, represents the gated convolution module, represents a nonlinear activation function, express The convolution kernel, express The convolution kernel, Indicates the multiplication of corresponding elements; all the gated features are input into the state space model in the parallel network framework, and the state space model performs residual connection on each pair of the gated features and the features output by the state space model to obtain the corresponding spatial features: ; in, Indicates The spatial features output by the layer encoder, Represents a state-space model.

[0023] In each layer of the encoder, a GSC (Gated Spatial Convolution) module is introduced to enhance the model's ability to extract local features and fuse spatial information. The GSC module combines 3×3×3 convolution kernels and 1×1×1 convolution kernels through dual-path convolution operations to obtain feature expressions at different scales. These features are fused through a gating mechanism to effectively capture spatial information while extracting local features. The gating mechanism can generate gated features by element-by-element multiplication of the features output by the two path convolutions, and combine these gated features with the original features of the input GSC through residual connections. This design effectively improves the expression of local features, especially the ability to capture spatial features at different scales.

[0024] Step S20: perform global enhancement processing on each of the spatial features using a state space model, and obtain all unimodal representations of each of the samples.

[0025] Among them, in the image segmentation task, traditional convolutional networks often rely on local receptive fields to obtain features, and the limited size of the convolution kernel causes the model to pay more attention to local texture, boundary and other information. Although the receptive field can be expanded to a certain extent by stacking multiple layers of convolution or using pyramid pooling, the modeling ability for large-scale and long-distance dependencies is still insufficient. For example, in medical images, the contextual information of certain anatomical structures or lesions often spans a large spatial range, and it is difficult to learn the global association of the entire image by relying solely on local convolution. Therefore, in an embodiment of the present application, an SSM (State Space Model) is introduced between the encoder and the decoder for global modeling features to capture long-distance dependencies.

[0026] Specifically, the samples corresponding to each of the spatial features are respectively input in parallel into the multiple state-space models of the parallel network framework for analysis; each of the state-space models respectively analyzes the feature connection relationships between different levels of a single modality in each of the samples, and constructs the global dependency information corresponding to each modality in each sample based on all the feature connection relationships; the global information enhancement channel constructed by each of the state-space models is obtained, and a feature sequence of each modality in each of the samples is constructed based on each of the spatial features; all of the feature sequences in each sample are input into the corresponding global information enhancement channel to update the corresponding global dependency information in real time, and each of the spatial features is globally enhanced based on the real-time updated global dependency information; the missing modality information input by the user is obtained, and the single modality representation corresponding to each sample is obtained based on the missing modality information.

[0027] Among them, the state space model maintains a "state vector" or "state representation" that can be continuously updated during the feature flow process of the network. Traditional convolution only performs convolution operations locally, while the state space model establishes a global or long-distance dependency between feature maps (or feature sequences), and updates features distributed at different locations or different resolution levels to a shared global "state". When the SSM module is embedded in the convolutional segmentation network, the corresponding position can get the representation of "feature aggregation performed within the entire image range" after one or more information interactions. In this way, the relationship between different anatomical parts can be better distinguished, and the capture of long-distance information (such as organ boundaries, lesion contexts) can be enhanced. By introducing SSM, the parallel network framework in this embodiment no longer relies only on the "small range" features seen by the local convolution kernel, but can rely on the "global perspective" provided by SSM to make richer and more global reasoning, thereby improving the image segmentation accuracy and significantly improving the accuracy and robustness of segmentation.

[0028] Step S30: input all the single-modal representations into a dynamic shared model to obtain multiple predicted segmentation results of the target image set.

[0029] Among them, there is a situation where modality is missing in the medical image segmentation task, and the decoder needs to dynamically fuse the segmentation results of each modality according to the available modality combination. Therefore, the embodiment of the present application introduces a DS (Dynamic Sharing) mechanism to dynamically adjust the fusion strategy according to the number of available modalities.

[0030] Specifically, all the single modal representations and the missing modal information are input into a dynamic sharing model, and an available modality set is constructed in the dynamic sharing model according to all the single modal representations and the missing modal information: ; ; in, Indicates The sample A unimodal representation, Indicates the available modal index, represents the number of modalities; the dynamic sharing model determines multiple missing modalities of each sample according to the available modal set, and performs fusion processing on the available modal set to obtain the predicted segmentation result of the available modal set under each missing modality: ; in, Indicates The predicted segmentation results of samples in available modalities.

[0031] Among them, for each sample, the decoder will choose a weighted average method to fuse the segmentation results of each modality according to the different available modalities, so as to ensure that the model can output valid segmentation results even when some modalities are missing. Since the input modality may be missing, a dynamic fusion method is needed to merge the segmentation results of the available modalities. Set a set of segmentation results of available modalities, and use a weighted and average method to fuse the segmentation results of the available modalities. The fused segmentation results are used as the final prediction output, so that the model can flexibly adapt to different available modal combinations, thereby enhancing the robustness in the case of modality missing.

[0032] Furthermore, when performing fusion, if only the first and second modalities are available and the other modalities are missing, the single-modal output results of the first channel and the second channel are weighted and averaged, and the result is considered to be the missing modality segmentation image obtained when only the first and second modalities are available. By flexibly adapting to the combination of different modalities, the robustness of the model in the case of modality missing is enhanced, ensuring the accuracy of the segmentation result.

[0033] Step S40, obtaining the original segmentation results of the target image set, constructing the original classification probability and the predicted classification probability, and constructing a balance loss term based on the original segmentation results and the multiple predicted segmentation results, and constructing a divergence loss term based on the original classification probability and the predicted classification probability.

[0034] Among them, in order to further optimize the parallel network framework in this application, the loss function is designed by combining Dice loss (an indicator for measuring the similarity of two sets), Hölder divergence (Kullback-Leibler divergence, KL divergence, a method for measuring the difference between two probability distributions) loss based on Dirichlet distribution (Dirichlet distribution) and mutual information loss.

[0035] Specifically, the original segmentation results and the original classification results of the target image set are obtained, the original classification probabilities are constructed according to the original segmentation results and the original classification results, and the predicted classification probabilities are constructed according to all the predicted segmentation results and the original classification results: ; ; ; ; ; in, represents the original classification probability, represents the predicted classification probability, represents the gamma function, Indicates the number of categories in the original classification result, Parameters representing the original classification probabilities, , and Represents the first, second and The parameters of the original classification probability, Indicates The parameters of the original classification probability, denoting the parameter of the predicted classification probability, , and Represents the first, second and parameters to predict the classification probability, Indicates standard dimensional simplex, Indicates parameters to predict the classification probability, Represents the predicted classification result in the predicted classification probability, Represents the predicted classification result of the first category in the predicted classification probability, Represents the predicted classification result of the second category in the predicted classification probability, Represents the predicted classification probability The predicted classification results of the categories, Represents the predicted classification probability The predicted classification results of the categories, represents the original classification result in the original classification probability, Represents the original classification result of the first category in the original classification probability, Represents the original classification result of the second category in the original classification probability, Represents the original classification probability The original classification results of the categories, Represents the original classification probability The original classification results of the categories; analyzing the overlap between the original segmentation results and the plurality of predicted segmentation results to construct a balanced loss term: ; in, represents the balance loss term, Indicates the original segmentation result The true label of each pixel, Indicates the predicted segmentation result The predicted probability of pixels; construct a divergence loss term based on the original classification probability and the predicted classification probability: ; ; ; in, and represents the divergence loss term, Indicates the difference, and The relationship between them is conjugate exponents. , and All represent hyperparameters, represents the original classification probability, represents the predicted classification probability, represents the original segmentation result, Represents the predicted segmentation result.

[0036] Among them, by introducing Dirichlet distribution, the parallel network framework can accurately capture and represent the uncertainty between multiple modalities, effectively improving the robustness of the network.

[0037] Furthermore, in this embodiment, the evidence value It is obtained by applying the softplus activation function (a mathematical function commonly used in the field of deep learning) to the logits (logistic units): ; Among them, represents the index of the category in the original classification result.

[0038] Furthermore, the Hölder divergence loss based on the Dirichlet distribution can flexibly and robustly quantify the difference between probability distributions, thereby calculating the difference between the true Dirichlet distribution and the predicted Dirichlet distribution in this embodiment, thereby calculating the divergence-based loss.

[0039] Step S50: construct a dependency relationship between the distribution probability under all modalities and the distribution probability under the missing modality, extract the deepest potential features of the distribution probability under all modalities and the distribution probability under the missing modality according to the dependency relationship, and construct a mutual information loss term according to the deepest potential features.

[0040] Among them, the mutual information loss function can enhance the model's ability to extract relevant information from available modalities, especially when some modalities are missing.

[0041] Specifically, according to the available modality set and all the missing modalities of each sample, the distribution probability under the full modality and the distribution probability under the missing modality are constructed; the true entropy of the distribution probability under the full modality and the predicted entropy of the distribution probability under the missing modality are extracted from the parallel network framework, and the dependency between the distribution probability under the full modality and the distribution probability under the missing modality is quantified according to the true entropy and the predicted entropy: ; in, Indicates dependency. represents the true entropy, represents the prediction entropy, represents the distribution probability under all modes, represents the distribution probability under the missing modality; according to the dependency relationship and the true entropy, the deepest potential features of the distribution probability under the full modality and the distribution probability under the missing modality are extracted to obtain the joint probability distribution between the distribution probability under the full modality and the distribution probability under the missing modality, and the mutual information loss term is constructed according to the joint probability distribution: ; ; in, Denotes the random variables P and The joint probability distribution of represents the mutual information loss term, It represents the probability of P under the condition that Q occurs.

[0042] Among them, in this embodiment, by randomly quantizing the dependency between the distribution probability under the full modality and the distribution probability under the missing modality, the correlation between different modalities or different feature branches is emphasized, and the network is encouraged to extract relevant effective information from the remaining modalities in the case of missing modalities, that is, to extract the deepest potential features of the two distribution probabilities.

[0043] Step S60, using the balance loss term, the divergence loss term and the mutual information loss term to train the parallel network framework to obtain a target parallel network framework, and outputting a final segmentation result of the target image set through the target parallel network framework.

[0044] Specifically, a total loss function is constructed according to the balance loss term, the divergence loss term and the mutual information loss term: ; in, represents the total loss function, represents the balance loss term, represents the mutual information loss term, represents the divergence loss term, express The weight adjustment coefficient, express The weight adjustment coefficient is ; the total loss function is input into the parallel network framework for training to obtain a target parallel network framework; the target image set is segmented through the decoding network in the target parallel network framework, and the final segmentation result is output.

[0045] In addition to the Dice loss, the Dirichlet-based Hölder divergence (HD) loss and mutual information (MI) loss are introduced. This combination is not common in multimodal medical image segmentation, especially the Dirichlet-based Hölder divergence is used to characterize the uncertainty of prediction classification, and the mutual information term can better balance accuracy and uncertainty mining during training.

[0046] Furthermore, if Figure 3 As shown, a comparison of image segmentation performance under different missing modalities is shown. Figure 3(a) shows the segmentation results based on the CHAOS dataset (Combined (CT-MR) Healthy Abdominal Organ Segmentation, mainly used for the challenge of abdominal organ segmentation). Each row (from top to bottom) shows the results under IN (input), OUT (output) and full modality input, and each column (from left to right) compares ACN (first method, Automatic Control Network), SMU-Net (second method, Style Matching U-Net), M³AE (third method, ‌Multimodal Masked Autoencoders, where ‌Multimodal means multimodal, Masked means mask, and Autoencoders means autoencoder) under different missing modalities, MMCFormer (fourth method, Missing Modality Compensation Transformer, missing modality compensation transformer), MML-MM-SSF (fifth method, ‌Multimodal Machine Learning - Multimodal Semantic FusionModel, multimodal machine learning-multimodal semantic fusion model), GGMD (the sixth method, Gradient-GuidedModality Decoupling for Missing-Modality Robustness, "Gradient-guided modality decoupling: improving the robustness of missing modalities") and the segmentation output of the method used in this embodiment, and the last column shows the real image. Figure 3 (b) is the segmentation result based on the BraTS 2018 dataset (an authoritative open source dataset for brain tumor segmentation in the field of medical imaging). Each row (from top to bottom) shows the results of T1 (first modality), T2 (second modality), T1ce (third modality), Flair (fourth modality) and full modality input. Each column compares the segmentation outputs of different methods under various missing modalities. The last column shows the real image. The figure highlights the robustness and effectiveness of the method proposed in this embodiment compared with the existing technology in various missing modality scenarios.

[0047] Specifically, the effectiveness of the method used in this embodiment can be determined by the data in Tables 1, 2 and 3 below: Table 1: WT (Whole Tumor, whole tumor area) segmentation result evaluation table

[0048] Table 2: ET (Enhancing Tumor) segmentation result evaluation table

[0049] Table 3: TC (Tumor Core) segmentation result evaluation table

[0050] Among them, the horizontal elements in Tables 1, 2 and 3 represent different single modalities, combinations of different single modalities and full modal inputs. The last column represents the mean of all modal segmentation results. In the first column, multiple methods (models) are divided. U-HVED represents the Unruh-Hybrid Variable Energy Detection Model, RFNet represents the Region-aware Fusion Network, mmFormer represents a deep learning network designed specifically for medical image segmentation tasks (especially brain tumor segmentation), EMRM represents the Enhanced Mobile Radio Module, SSFM represents the "Multi-modal Learning with Missing Modality via Shared-Specific Feature Modelling" method, QuMo represents the quantum mobility network, and GSS represents the "Scratch Each Other's Collaboration: Incomplete Multimodal Brain Tumor Segmentation Based on Category-aware Group Self-support Learning" method. Back: Incomplete Multi-modal Brain Tumor Segmentation Via Category Aware Group Self-Support Learning), MFTrans represents a multi-functional neural network model (Multi-Function Transformer Network) based on the Transformer architecture, MTI represents the “Multimodal Transformer of Incomplete MRI Data for Brain Tumor Segmentation” method, GGDM represents the “Gradient-Guided Modality Decoupling for Missing-Modality Robustness” method, and OUR represents the parallel network framework used in the present invention.

[0051] Furthermore, if Figure 4 , Figure 5 and Figure 6 As shown, in this embodiment, the quantitative comparison of the segmentation results of the BraTS 2018 and BraTS2020 datasets (authoritative open source datasets for brain tumor segmentation in the field of medical imaging) under different missing modes is also investigated. Specifically, the segmentation results of the whole tumor area, the tumor core area, and the enhanced tumor are compared. Figure 4-Figure 6 The results in show that the proposed method is effective in processing incomplete data, showing superior reconstruction quality compared to the baseline method.

[0052] Furthermore, if Figure 7 , Figure 8 , Fig. 9 and Fig.10 As shown, in this embodiment, the quantitative comparison of segmentation results of the CHAOS dataset (Combined (CT-MR) Healthy Abdominal Organ Segmentation, mainly used for abdominal organ segmentation challenges) under different missing modalities is investigated, and the different methods are specifically compared for cardiac images (such as Figure 7 As shown, LV-dice, Left Ventricle dice coefficient), conventional scan (such as Figure 8 As shown, Routine K-space, medical imaging scans), abnormal states (such as Fig. 9 as shown, LK, Lucas-Kanade method) and specificity (as Fig.10 As shown, the quantitative comparison results of SP (Standardized Patients) were obtained. Figure 7-Figure 10 The reconstruction performance of the CHAOS dataset in various missing modalities is demonstrated, proving that the method used in this embodiment can achieve high-quality segmentation even when the input data is incomplete, and the performance exceeds the baseline technology.

[0053] The present invention sets multiple parallel encoder branches to fully learn their own features, which can effectively avoid the disadvantage of insufficient fusion information caused by missing modalities and produce accurate image segmentation results when the modality is missing. When the multi-modality is complete, each branch can fully extract the features of the modality and enhance the overall image segmentation performance.

[0054] Furthermore, if Fig.11 As shown, based on the above-mentioned image segmentation method based on parallel network framework and dynamic fusion, the present invention also provides an image segmentation system based on parallel network framework and dynamic fusion, wherein the image segmentation system based on parallel network framework and dynamic fusion includes: The spatial feature construction module 51 is used to obtain multiple original features of the target image set, input all the original features into the parallel coding network for a two-way convolution operation, obtain gated features corresponding to multiple samples, and construct a spatial feature corresponding to each sample according to all the gated features; A modality extraction module 52, configured to perform global enhancement processing on each of the spatial features using a state space model, and obtain all single modality representations of each of the samples; A result prediction module 53, used for inputting all the single-modal representations into a dynamic shared model to obtain a plurality of predicted segmentation results of the target image set; A first loss term construction module 54 is used to obtain the original segmentation result of the target image set, construct the original classification probability and the predicted classification probability, and construct a balance loss term according to the original segmentation result and the predicted segmentation result, and construct a divergence loss term according to the original classification probability and the predicted classification probability; A second loss term construction module 55 is used to construct a dependency relationship between the original classification probability and the predicted classification probability, and estimate the true posterior of the predicted classification probability according to the dependency relationship to obtain a mutual information loss term; The model training module 56 is used to train the parallel coding network using the balance loss term, the divergence loss term and the mutual information loss term to obtain a target parallel coding network, and output a final segmentation result of the target image set through the target parallel coding network.

[0055] Furthermore, if Fig.12 As shown, based on the above-mentioned image segmentation method and system based on parallel network framework and dynamic fusion, the present invention also provides a terminal accordingly, and the terminal includes a processor 10, a memory 20 and a display 30. Fig.12 Only some components of the terminal are shown, but it should be understood that it is not required to implement all of the components shown, and more or fewer components may be implemented instead.

[0056] In some embodiments, the memory 20 may be an internal storage unit of the terminal, such as a hard disk or memory of the terminal. In other embodiments, the memory 20 may also be an external storage device of the terminal, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (SecureDigital, SD) card, a flash card (Flash Card), etc. equipped on the terminal. Further, the memory 20 may also include both an internal storage unit of the terminal and an external storage device. The memory 20 is used to store application software and various types of data installed in the terminal, such as the program code of the installation terminal. The memory 20 may also be used to temporarily store data that has been output or is to be output. In one embodiment, an image segmentation program 40 based on a parallel network framework and dynamic fusion is stored on the memory 20, and the image segmentation program 40 based on a parallel network framework and dynamic fusion can be executed by the processor 10, thereby realizing the image segmentation method based on a parallel network framework and dynamic fusion in the present application.

[0057] In some embodiments, the processor 10 may be a central processing unit (CPU), a microprocessor or other data processing chip, used to run the program code or process data stored in the memory 20, such as executing the image segmentation method based on the parallel network framework and dynamic fusion.

[0058] In some embodiments, the display 30 may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, an OLED (Organic Light-Emitting Diode) touch device, etc. The display 30 is used to display information on the terminal and to display a visual user interface. The components of the terminal communicate with each other via a system bus.

[0059] In one embodiment, when the processor 10 executes the image segmentation program 40 based on the parallel network framework and dynamic fusion in the memory 20, the steps of the image segmentation method based on the parallel network framework and dynamic fusion as described above are implemented.

[0060] The present invention also provides a computer-readable storage medium, wherein the computer-readable storage medium stores an image segmentation program based on a parallel network framework and dynamic fusion, and when the image segmentation program based on a parallel network framework and dynamic fusion is executed by a processor, the steps of the image segmentation method based on a parallel network framework and dynamic fusion as described above are implemented.

[0061] In summary, the present invention provides an image segmentation method and related equipment based on a parallel network framework and dynamic fusion, the method comprising: obtaining multiple original features of a target image set, inputting all the original features into a parallel coding network for a two-way convolution operation, obtaining gated features corresponding to multiple samples, and constructing a spatial feature corresponding to each sample based on all the gated features; performing global enhancement processing on each of the spatial features using a state space model to obtain multiple single-modal representations of multiple samples; inputting all the single-modal representations into a dynamic shared model to obtain multiple predicted segmentation results of the target image set; obtaining the original features of the target image set; Segmentation result, construct the original classification probability and the predicted classification probability, and construct the balance loss term according to the original segmentation result and the predicted segmentation result, and construct the divergence loss term according to the original classification probability and the predicted classification probability; construct the dependency relationship between the original classification probability and the predicted classification probability, and estimate the true posterior of the predicted classification probability according to the dependency relationship to obtain the mutual information loss term; use the balance loss term, the divergence loss term and the mutual information loss term to train the parallel coding network to obtain the target parallel coding network, and output the final segmentation result of the target image set through the target parallel coding network. The present invention sets multiple parallel encoder branches to fully learn their own features, which can effectively avoid the disadvantage of insufficient fusion information caused by missing modalities, and produce accurate image segmentation results when the modality is missing. When the multi-modality is complete, each branch can fully extract the features of the modality to enhance the overall image segmentation performance.

[0062] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or terminal including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or terminal. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the presence of other identical elements in the process, method, article or terminal including the element.

[0063] Of course, those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing related hardware (such as a processor, a controller, etc.) through a computer program, and the program can be stored in a computer-readable storage medium that can be read by a computer, and the program can include the processes of the above-mentioned method embodiments when executed. The computer-readable storage medium can be a memory, a disk, an optical disk, etc.

[0064] It should be understood that the application of the present invention is not limited to the above examples. For ordinary technicians in this field, improvements or changes can be made based on the above description. All these improvements and changes should fall within the scope of protection of the claims attached to the present invention.

Claims

1. An image segmentation method based on a parallel network framework and dynamic fusion, characterized in that: The image segmentation method based on parallel network framework and dynamic fusion includes: Acquire multiple original features of the target image set, input all the original features into a parallel network framework for a two-way convolution operation, obtain the gated features corresponding to the original features of each sample, and construct the spatial features corresponding to each sample according to each gated feature; Performing global enhancement processing on each of the spatial features using a state space model, and obtaining all unimodal representations of each of the samples; Inputting all the single-modal representations into a dynamic shared model to obtain a plurality of predicted segmentation results of the target image set; Obtaining the original segmentation results of the target image set, constructing the original classification probability and the predicted classification probability, constructing a balance loss term according to the original segmentation results and the plurality of predicted segmentation results, and constructing a divergence loss term according to the original classification probability and the predicted classification probability; Constructing a dependency relationship between the distribution probability under all modalities and the distribution probability under the missing modality, extracting the deepest potential features of the distribution probability under all modalities and the distribution probability under the missing modality according to the dependency relationship, and constructing a mutual information loss term according to the deepest potential features; The parallel network framework is trained using the balance loss term, the divergence loss term and the mutual information loss term to obtain a target parallel network framework, and a final segmentation result of the target image set is output through the target parallel network framework.

2. The image segmentation method based on parallel network framework and dynamic fusion according to claim 1, characterized in that: The method of obtaining multiple original features of the target image set, inputting all the original features into a parallel network framework for a two-way convolution operation, obtaining a gated feature corresponding to the original feature of each sample, and constructing a spatial feature corresponding to each sample according to each gated feature, specifically includes: Obtaining original features corresponding to each sample in the target image set, and inputting all the original features into multiple encoders in the parallel network framework respectively; The local features corresponding to the original features of each sample are extracted by the gated convolution module of each encoder, and a two-way convolution operation is performed on the multiple original features according to each local feature to output the corresponding gated features: ； in, Indicates The gated features output by the layer encoder, Indicates input to Features of the gated convolutional modules in the layer encoder, represents the gated convolution module, represents a nonlinear activation function, express The convolution kernel, express The convolution kernel, Indicates the multiplication of corresponding elements; All the gated features are input into the state space model in the parallel network framework, and the state space model performs residual connection on each pair of the gated features and the features output by the state space model to obtain the corresponding spatial features: ； in, Indicates The spatial features output by the layer encoder, Represents a state-space model.

3. The image segmentation method based on parallel network framework and dynamic fusion according to claim 1, characterized in that: The utilizing the state space model to perform global enhancement processing on each of the spatial features and obtaining all single-modal representations of each of the samples specifically includes: Inputting samples corresponding to each of the spatial features into multiple state space models of the parallel network framework in parallel for analysis; Each of the state-space models analyzes the feature connection relationships between different levels of a single modality in each of the samples, and constructs global dependency information corresponding to each modality in each sample based on all of the feature connection relationships; Obtaining the global information enhancement channel constructed respectively for each of the state space models, and constructing a feature sequence of each mode in each of the samples according to each of the spatial features; Inputting all the feature sequences in each sample into the corresponding global information enhancement channel to update the corresponding global dependency information in real time, and performing global enhancement processing on each of the spatial features according to the real-time updated global dependency information; The missing modality information input by the user is obtained, and a single modality representation corresponding to each sample is obtained according to the missing modality information.

4. The image segmentation method based on parallel network framework and dynamic fusion according to claim 3, characterized in that: The step of inputting all the single-modal representations into a dynamic shared model to obtain a plurality of predicted segmentation results of the target image set specifically includes: Inputting all the single modality representations and the missing modality information into a dynamic sharing model, and constructing an available modality set in the dynamic sharing model according to all the single modality representations and the missing modality information: ； ； in, Indicates The sample A unimodal representation, Indicates the available modal index, Indicates the number of modes; The dynamic sharing model determines multiple missing modalities of each sample according to the available modality set, and performs fusion processing on the available modality set to obtain the predicted segmentation result of the available modality set under each missing modality: ； in, Indicates The predicted segmentation results of samples in available modalities.

5. The image segmentation method based on parallel network framework and dynamic fusion according to claim 1, characterized in that: The obtaining of the original segmentation results of the target image set, constructing the original classification probability and the predicted classification probability, constructing a balance loss term according to the original segmentation results and the plurality of predicted segmentation results, and constructing a divergence loss term according to the original classification probability and the predicted classification probability, specifically includes: Obtaining the original segmentation results and the original classification results of the target image set, constructing the original classification probability according to the original segmentation results and the original classification results, and constructing the predicted classification probability according to all the predicted segmentation results and the original classification results: ； ； ； ； ； in, represents the original classification probability, represents the predicted classification probability, represents the gamma function, Indicates the number of categories in the original classification result, Parameters representing the original classification probabilities, , and Represent the parameters of the 1st, 2nd and Kth original classification probabilities respectively, Indicates The parameters of the original classification probability, , which represents the parameter of the predicted classification probability, , and Respectively represent the first, second and parameters to predict the classification probability, Indicates standard dimensional simplex, Indicates parameters to predict the classification probability, Represents the predicted classification result in the predicted classification probability, Represents the predicted classification result of the first category in the predicted classification probability, Represents the predicted classification result of the second category in the predicted classification probability, Represents the predicted classification probability The predicted classification results of the categories, Represents the predicted classification probability The predicted classification results of the categories, represents the original classification result in the original classification probability, Represents the original classification result of the first category in the original classification probability, Represents the original classification result of the second category in the original classification probability, Represents the original classification probability The original classification results of the categories, Represents the original classification probability The original classification results of the categories; Analyze the overlap between the original segmentation result and the plurality of predicted segmentation results to construct a balanced loss term: ； in, represents the balance loss term, Indicates the original segmentation result The true label of each pixel, Indicates the predicted segmentation result The predicted probability of pixels; A divergence loss term is constructed based on the original classification probability and the predicted classification probability: ； ； ； in, and represents the divergence loss term, Indicates the difference, and The relationship between them is conjugate exponents. , and All represent hyperparameters, represents the original classification probability, represents the predicted classification probability, represents the original segmentation result, Represents the predicted segmentation result.

6. The image segmentation method based on parallel network framework and dynamic fusion according to claim 4, characterized in that: The step of constructing a dependency relationship between the distribution probability under all modalities and the distribution probability under the missing modalities, extracting the deepest potential features of the distribution probability under all modalities and the distribution probability under the missing modalities according to the dependency relationship, and constructing a mutual information loss term according to the deepest potential features, specifically includes: Constructing the distribution probability under all modalities and the distribution probability under the missing modalities according to the available modality set and all the missing modalities of each sample; Extracting the true entropy of the distribution probability under the full modality and the predicted entropy of the distribution probability under the missing modality from the parallel network framework, and quantifying the dependency between the distribution probability under the full modality and the distribution probability under the missing modality according to the true entropy and the predicted entropy: ； in, Indicates dependency. represents the true entropy, represents the prediction entropy, represents the distribution probability under all modes, represents the probability of distribution under the missing mode; According to the dependency relationship and the true entropy, the deepest potential features of the distribution probability under the full modality and the distribution probability under the missing modality are extracted to obtain the joint probability distribution between the distribution probability under the full modality and the distribution probability under the missing modality, and the mutual information loss term is constructed according to the joint probability distribution: ； ； in, Denotes the random variables P and The joint probability distribution of represents the mutual information loss term, It represents the probability of P under the condition that Q occurs.

7. The image segmentation method based on parallel network framework and dynamic fusion according to claim 1, characterized in that: The using the balance loss term, the divergence loss term and the mutual information loss term to train the parallel network framework to obtain a target parallel network framework, and outputting a final segmentation result of the target image set through the target parallel network framework specifically includes: A total loss function is constructed based on the balance loss term, the divergence loss term and the mutual information loss term: ； in, represents the total loss function, represents the balance loss term, represents the mutual information loss term, represents the divergence loss term, express The weight adjustment coefficient, express The weight adjustment coefficient of Inputting the total loss function into the parallel network framework for training to obtain a target parallel network framework; The target image set is segmented through the decoding network in the target parallel network framework, and the final segmentation result is output.

8. An image segmentation system based on a parallel network framework and dynamic fusion, characterized in that: The image segmentation system based on parallel network framework and dynamic fusion is applied to the image segmentation method based on parallel network framework and dynamic fusion according to any one of claims 1 to 7, and the image segmentation system based on parallel network framework and dynamic fusion comprises: A spatial feature construction module is used to obtain multiple original features of the target image set, input all the original features into a parallel network framework for a two-way convolution operation, obtain a gated feature corresponding to the original feature of each sample, and construct a spatial feature corresponding to each sample according to each gated feature; A modality extraction module, used for performing global enhancement processing on each of the spatial features using a state space model, and obtaining all single modality representations of each of the samples; A result prediction module, used for inputting all the single-modal representations into a dynamic shared model to obtain a plurality of predicted segmentation results of the target image set; A first loss term construction module is used to obtain the original segmentation result of the target image set, construct the original classification probability and the predicted classification probability, and construct a balance loss term according to the original segmentation result and the plurality of predicted segmentation results, and construct a divergence loss term according to the original classification probability and the predicted classification probability; A second loss term construction module is used to construct a dependency relationship between the distribution probability under the full modality and the distribution probability under the missing modality, extract the deepest potential features of the distribution probability under the full modality and the distribution probability under the missing modality according to the dependency relationship, and construct a mutual information loss term according to the deepest potential features; A model training module is used to train the parallel network framework using the balance loss term, the divergence loss term and the mutual information loss term to obtain a target parallel network framework, and output a final segmentation result of the target image set through the target parallel network framework.

9. A terminal, characterized in that: The terminal includes: a memory, a processor, and an image segmentation program based on a parallel network framework and dynamic fusion stored in the memory and executable on the processor. When the image segmentation program based on a parallel network framework and dynamic fusion is executed by the processor, the steps of the image segmentation method based on a parallel network framework and dynamic fusion as described in any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores an image segmentation program based on a parallel network framework and dynamic fusion. When the image segmentation program based on a parallel network framework and dynamic fusion is executed by a processor, the steps of the image segmentation method based on a parallel network framework and dynamic fusion as described in any one of claims 1 to 7 are implemented.

Images