Image segmentation method, device, electronic device and storage medium

Abstract

The present invention discloses an image segmentation method, device, electronic device, and storage medium. The method includes: obtaining an input image; inputting the input image into a preset multi-decision aggregation network to perform image segmentation prediction and obtain a segmentation prediction map; constructing a first loss function based on weighted scores, prediction results, and annotation information, and pre-training the multi-decision aggregation network; using the segmentation model in the pre-trained multi-decision aggregation network to obtain segmentation data; performing an indicator evaluation on all segmentation results based on preset indicators to obtain a reward score for each segmentation result; constructing a preference data pair based on the segmentation result based on the reward score; constructing a second loss function based on the preference data pair, and performing a second training on the segmentation model; and using the multi-decision aggregation network after the second training to perform image segmentation on the image to be processed to obtain an image segmentation result. The present invention can balance multiple indicators to achieve accurate image segmentation and can be widely applied in the field of image processing technology.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to an image segmentation method, device, electronic device and storage medium. Background Art

[0002] The goal of medical image segmentation is to classify each pixel in an input medical image into a class from a set of categories. In this task, various evaluation metrics (such as pixel accuracy, Dice similarity coefficient, Hausdorff distance, sensitivity, etc.) are used to assess segmentation performance from different perspectives. Existing methods generally design specific differentiable surrogate loss functions for different metrics to optimize the segmentation model.

[0003] However, in the field of medical image segmentation, evaluation metrics are extremely diverse, and some are non-differentiable, making it challenging to design a specific proxy loss function for each metric. Furthermore, optimizing multiple metrics simultaneously during training is also difficult, requiring a careful balance of different loss functions during training. Summary of the Invention

[0004] The main purpose of the embodiments of the present invention is to propose an image segmentation method, device, electronic device and storage medium in order to solve at least one problem of the prior art. The present invention can balance the loss optimization of different indicators to achieve accurate image segmentation.

[0005] To achieve the above objectives, an embodiment of the present invention provides an image segmentation method, comprising:

[0006] Get an input image; the input image carries the category annotation information of each pixel;

[0007] Input the input image into the preset multi-decision aggregation network to perform image segmentation prediction and obtain a segmentation prediction map;

[0008] The multi-decision aggregation network includes a segmentation model and a sparse router; the segmentation model includes a visual encoder and a preset number of decision sub-networks connected in parallel after the visual encoder; the decision sub-networks use a decoder; the sparse router is used to assign a weighted score to the data results of each decision sub-network; the segmentation prediction map includes the prediction result of the category of each pixel in the input image;

[0009] A first loss function is constructed based on the weighted scores, prediction results, and annotation information, and then a multi-decision aggregation network is pre-trained;

[0010] Based on the input image, the segmentation model in the pre-trained multi-decision aggregation network is used to obtain segmentation data; the segmentation data includes the segmentation results of each decision sub-network;

[0011] Perform an indicator evaluation on all segmentation results based on the preset indicators to obtain the reward score for each segmentation result;

[0012] Based on the reward score, a preferred data pair is constructed according to the segmentation results; the preferred data pair includes positive samples and negative samples, and the reward score of the positive sample is greater than the reward score of the negative sample;

[0013] Constructing a second loss function based on the preference data pair, and then performing a second training on the segmentation model;

[0014] The image to be processed is segmented using the multi-decision aggregation network after the second training to obtain an image segmentation result.

[0015] In some embodiments, inputting an input image into a preset multi-decision aggregation network to perform image segmentation prediction to obtain a segmentation prediction map includes the following steps:

[0016] Visually encode the input image through the visual encoder and extract the visual embedding;

[0017] Based on visual embedding, a corresponding first segmentation probability map is generated through each decision sub-network; random deactivation operation is used in each decision sub-network;

[0018] A weighted score is assigned to the first segmentation probability map corresponding to each decision subnetwork through a sparse router, and the products of all first segmentation probability maps and their corresponding weighted scores are accumulated and summed to obtain a second segmentation probability map; the sparse router includes a full connection and an activation function;

[0019] Find the maximum value index in the channel dimension of the second segmentation probability map to obtain the segmentation prediction map.

[0020] In some embodiments, constructing a first loss function based on the weighted scores, the prediction results, and the annotation information includes the following steps:

[0021] Construct an entropy penalty loss function based on the weighted scores;

[0022] Construct a cross entropy loss function based on the prediction results and annotation information;

[0023] The first loss function is obtained by adding the entropy penalty loss function and the cross entropy loss function.

[0024] In some embodiments, constructing an entropy penalty loss function based on the weighted scores includes the following steps:

[0025] Solve the logarithmic function for the weighted scores assigned to each decision subnetwork;

[0026] The entropy penalty loss function is obtained by summing up the products of the weighted scores assigned to all decision sub-networks and their corresponding logarithmic function solutions.

[0027] In some embodiments, performing an indicator evaluation on all segmentation results based on a preset indicator to obtain a reward score for each segmentation result includes the following steps:

[0028] Perform an index evaluation on the segmentation result based on the preset index, and obtain the function value of the evaluation index function corresponding to each preset index of the segmentation result;

[0029] The function values ​​of the segmentation results in all evaluation index functions are weighted and summed to obtain the reward score corresponding to the segmentation result.

[0030] In some embodiments, constructing a second loss function based on the preference data pair includes the following steps:

[0031] Construct contrast loss and cross entropy loss functions based on preference data pairs;

[0032] The contrast loss is balanced based on the preset hyperparameters, and then the second loss function is constructed by combining it with the cross entropy loss function.

[0033] In some embodiments, constructing a contrastive loss based on a preference data pair includes the following steps:

[0034] Get the generation probability of each segmentation result;

[0035] Construct contrast loss based on the generation probability of positive and negative samples corresponding to all preference data pairs;

[0036] Among them, the expression of contrast loss is:

[0037]

[0038] Where, represents contrast loss; Y a represents a positive sample; Y b represents negative samples; π φ (Y a |I) means that when given an input image I, the segmentation model generates Y a The generation probability of π φ (Y b |I) means that when given an input image I, the segmentation model generates Y b Ω represents the preference ranking set composed of all segmentation results, and the segmentation results in the preference ranking set are arranged in the order of their corresponding reward scores.

[0039] To achieve the above-mentioned object, another aspect of an embodiment of the present invention provides an image segmentation device, the device comprising:

[0040] The first module is used to obtain an input image; the input image carries the labeling information of the category of each pixel;

[0041] The second module is used to input the input image into a preset multi-decision aggregation network to perform image segmentation prediction and obtain a segmentation prediction map;

[0042] The multi-decision aggregation network includes a segmentation model and a sparse router; the segmentation model includes a visual encoder and a preset number of decision sub-networks connected in parallel after the visual encoder; the decision sub-networks use a decoder; the sparse router is used to assign a weighted score to the data results of each decision sub-network; the segmentation prediction map includes the prediction result of the category of each pixel in the input image;

[0043] The third module is used to construct a first loss function based on the weighted scores, prediction results and annotation information, and then pre-train the multi-decision aggregation network;

[0044] The fourth module is used to obtain segmentation data based on the input image using the segmentation model in the pre-trained multi-decision aggregation network; the segmentation data includes the segmentation results of each decision sub-network;

[0045] The fifth module is used to evaluate all segmentation results based on preset indicators and obtain the reward score for each segmentation result;

[0046] The sixth module is used to construct a preference data pair based on the segmentation result based on the reward score; the preference data pair includes a positive sample and a negative sample, and the reward score of the positive sample is greater than the reward score of the negative sample;

[0047] A seventh module is used to construct a second loss function based on the preference data, thereby performing a second training on the segmentation model;

[0048] The eighth module is used to perform image segmentation on the image to be processed using the multi-decision aggregation network after the second training to obtain an image segmentation result.

[0049] To achieve the above object, another aspect of an embodiment of the present invention provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor implements the above method when executing the computer program.

[0050] To achieve the above object, another aspect of an embodiment of the present invention provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the above method is implemented.

[0051] The embodiments of the present invention include at least the following beneficial effects: the present invention provides an image segmentation method, device, electronic device and storage medium, which obtains an input image; the input image carries the labeling information of the category of each pixel; the input image is input into a preset multi-decision aggregation network for image segmentation prediction to obtain a segmentation prediction map; wherein the multi-decision aggregation network includes a segmentation model and a sparse router; the segmentation model includes a visual encoder and a preset number of decision sub-networks connected in parallel after the visual encoder; the decision sub-network adopts a decoder; the sparse router is used to assign a weighted score to the data result of each decision sub-network; the segmentation prediction map includes the prediction result of the category of each pixel of the input image; based on the weighted score and the prediction The first loss function is constructed based on the results and annotation information, and the multi-decision aggregation network is pre-trained; based on the input image, the segmentation data is obtained using the segmentation model in the pre-trained multi-decision aggregation network; the segmentation data includes the segmentation results of each decision sub-network; all segmentation results are evaluated based on preset indicators to obtain a reward score for each segmentation result; based on the reward score, a preference data pair is constructed according to the segmentation result; the preference data pair includes positive samples and negative samples, and the reward score of the positive sample is greater than the reward score of the negative sample; a second loss function is constructed based on the preference data pair, and the segmentation model is trained for the second time; the multi-decision aggregation network after the second training is used to perform image segmentation on the processed image to obtain the image segmentation result. The embodiment of the present invention integrates preference optimization and expert mixing strategy, and uses a unified method to optimize any evaluation index; the multi-decision aggregation network is used as the backbone network of the segmentation task, which is responsible for generating diversified segmentation results of the input image, and then constructing segmentation data and optimizing the segmentation model accordingly. The embodiment of the present invention can balance multiple indicators to achieve accurate image segmentation. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Figure 1 is a flow chart of an image segmentation method provided by an embodiment of the present invention;

[0053] Figure 2 is a schematic diagram of a multi-decision preference optimization network structure provided by an embodiment of the present invention;

[0054] Figure 3 Schematic diagram of a multi-decision preference optimization process provided by an embodiment of the present invention;

[0055] Figure 4 1 is a schematic diagram of the architecture of an image segmentation device provided by an embodiment of the present invention;

[0056] Figure 5 It is a schematic diagram of the hardware structure of the electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0057] In order to make the objects, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the embodiments of the present invention. They are merely examples of devices and methods consistent with some aspects of the embodiments of the present invention as detailed in the appended claims.

[0058] It will be understood that the terms "first," "second," and the like used in the present invention may be used to describe various concepts in the present invention, but unless otherwise specified, these concepts are not limited by these terms. These terms are merely used to distinguish one concept from another. For example, without departing from the scope of the embodiments of the present invention, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the terms "if" and "if" as used herein may be interpreted as "at the time of," "when," or "in response to a determination."

[0059] The terms "at least one", "plurality", "each", "any", etc. used in the present invention include "at least one", "two" or "more than two", "plurality" or "more than two", "each" or "more than two", "each" or "more than two", and "any" or "more than one" in the present invention.

[0060] Unless otherwise defined, all technical and scientific terms used in the present invention have the same meaning as those commonly understood by those skilled in the art to which the present invention pertains. The terms used in the present invention are for the purpose of describing the embodiments of the present invention only and are not intended to limit the present invention.

[0061] The image segmentation method provided by the embodiment of the present invention relates to the field of image processing technology. The image segmentation method provided by the embodiment of the present invention can be applied to a terminal, can be applied to a server, or can be software running in a terminal or a server. In some embodiments, the terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart watch, and a car terminal, etc., but is not limited to this; the server side can be configured as an independent physical server, or as a server cluster or distributed system composed of multiple physical servers, or as a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. The server can also be a node server in a blockchain network; the software can be an application that implements the image segmentation method, etc., but is not limited to the above forms.

[0062] The present invention can be used in a wide variety of general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, and the like. The present invention can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present invention can also be practiced in distributed computing environments in which tasks are performed by remote processing devices connected via a communications network. In a distributed computing environment, program modules can be located in local and remote computer storage media, including storage devices.

[0063] Figure 1 is an optional flowchart of the image segmentation method provided by an embodiment of the present invention. Figure 1 The method may include but is not limited to steps S100 to S800.

[0064] S100, obtaining an input image;

[0065] The input image carries the labeling information of the category of each pixel;

[0066] S200, inputting the input image into a preset multi-decision aggregation network to perform image segmentation prediction to obtain a segmentation prediction map;

[0067] The multi-decision aggregation network includes a segmentation model and a sparse router; the segmentation model includes a visual encoder and a preset number of decision sub-networks connected in parallel after the visual encoder; the decision sub-networks use a decoder; the sparse router is used to assign a weighted score to the data results of each decision sub-network; the segmentation prediction map includes the prediction result of the category of each pixel in the input image;

[0068] It should be noted that, in some embodiments, step S200 may include the following steps: visually encoding the input image through a visual encoder to extract a visual embedding; based on the visual embedding, generating a corresponding first segmentation probability map through each decision sub-network; using a random deactivation operation in each decision sub-network; assigning a weighted score to the first segmentation probability map corresponding to each decision sub-network through a sparse router, and accumulating and summing the products of all first segmentation probability maps and their corresponding weighted scores to obtain a second segmentation probability map; the sparse router includes a full connection and an activation function; searching for the maximum value index in the channel dimension of the second segmentation probability map to obtain a segmentation prediction map.

[0069] For example, in some specific embodiments, Figure 2 The process on the left, the processing flow of the multi-decision aggregation network can be implemented as follows:

[0070] Input a medical image I, the multi-decision preference optimization network first passes through the visual encoder Extracting visual embeddings:

[0071]

[0072] in, is the visual embedding of the input image I. Here, S and C represent the number of tokens and latent dimension, respectively. is a visual encoder. Then, the present invention designs multiple decoder decision sub-networks, defined as {ε m (·)|m=1,…,M}, used to generate diverse segmentation results:

[0073]

[0074] in, Y represents the segmentation probability map generated by the mth decision sub-network for X. m Represents the corresponding segmentation result prediction map. Find the index of the maximum value in the channel dimension, and get the corresponding segmentation prediction map Y m Each decision subnetwork ε m (·) are expected to adaptively learn expertise for medical image segmentation, thus producing diverse segmentation results Y for X m In order to further enhance the diversity of the decision sub-network, the present invention adds m (·) Dropout operation (random inactivation) is used.

[0075] Since each decision subnetwork ε m (·) can capture diverse information from X and generate different segmentation results. Therefore, this paper designs a sparse router s(·). The router function s(·) is implemented by two layers of full connection + Softmax (FC-ReLU-FC-ReLU-Softmax) layers. It is used to aggregate ε from different decision sub-networks. m Different segmentation results of (·):

[0076]

[0077] in is the final segmentation probability map, and Y is the final prediction map. The router function s(·) is for each corresponding decision subnetwork ε m (·) Assign score sm (X) (i.e., a weighted score), which indicates how suitable the mth decision subnetwork is for processing X.

[0078] S300, constructing a first loss function based on the weighted scores, the prediction results, and the annotation information, and then pre-training the multi-decision aggregation network;

[0079] It should be noted that, in some embodiments, constructing a first loss function based on weighted scores and prediction results and annotation information may include the following steps: constructing an entropy penalty loss function based on the weighted scores; constructing a cross-entropy loss function based on the prediction results and annotation information; and obtaining the first loss function based on the result of adding the entropy penalty loss function and the cross-entropy loss function.

[0080] For example, in some specific implementations, in order to encourage efficient sparse routing, the present invention designs an entropy penalty loss function to minimize the entropy of router scores in different decision subnetworks:

[0081]

[0082] The entropy penalty loss function can make the decision subnetwork ε that is most suitable for processing X m (·) is highlighted. Therefore, during the training process, different decision sub-networks focus on different images, so that specific and complementary medical knowledge can be learned. The final prediction map Y is obtained through the probability map {ε m (X)|m=1,…,M} are aggregated.

[0083] During the training process, the present invention applies the cross entropy loss function to the final prediction graph Y:

[0084]

[0085] Among them, N and They represent the number of pixels and the number of categories of image I respectively. represents the true label that the i-th pixel belongs to category c.

[0086] The present invention utilizes multiple parallel decision sub-networks to generate diverse segmentation results, which can be used to construct subsequent segmentation data. It is worth noting that the sparse router and dropout operations ensure that for the same input, the {Y m |m=1,…,M} has sufficient diversity.

[0087] S400, based on the input image, using the segmentation model in the pre-trained multi-decision aggregation network to obtain segmentation data;

[0088] Among them, the segmentation data includes the segmentation results of each decision sub-network;

[0089] For example, in some embodiments, a trained multi-decision aggregation network is used to construct segmentation data.

[0090] S500, performing an indicator evaluation on all segmentation results based on preset indicators to obtain a reward score for each segmentation result;

[0091] It should be noted that, in some embodiments, performing an index evaluation on all segmentation results based on preset indicators to obtain a reward score for each segmentation result may include the following steps: performing an index evaluation on the segmentation results based on preset indicators to obtain the function value of the evaluation index function corresponding to each preset indicator of the segmentation result; performing a weighted summation on the function values ​​of all evaluation index functions of the segmentation result to obtain a reward score corresponding to the segmentation result.

[0092] For example, in some specific implementations, for multiple segmentation results {Y m |m=1,…,M}, the present invention proposes a segmentation optimization module based on index preference to learn the knowledge of evaluation preference. The segmentation optimization module based on index preference is used to calculate the value of each result Y. m The reward score can be used to construct segmentation data for various target evaluation indicators. This segmentation data is then used for preference supervision to guide the optimization of the segmentation model through contrastive learning, encouraging the generation of high-quality results and suppressing poor-quality results.

[0093] The present invention first designs a reward function to evaluate different segmentation results {Y m |m=1,…,M}. This reward function can be used to score each segmentation result based on any evaluation metric and is defined as follows:

[0094]

[0095] Among them, r m Represents the segmentation result Y of the input image I m Bonus score. It can be any differentiable or non-differentiable evaluation index function, such as Dice coefficient, Hausdorff distance, etc. The calculation results of these evaluation index functions are normalized to the range of 0 to 1. The higher the score, the better the quality of the segmentation result. k Represents the kth evaluation index function The weight of .

[0096] S600: Based on the reward score, construct a preference data pair according to the segmentation result;

[0097] Among them, the preference data pairs include positive samples and negative samples, and the reward score of the positive sample is greater than the reward score of the negative sample.

[0098] For example, in some specific implementations, according to {r m |m=1,…,M}, the present invention divides the segmentation result {Y m |m=1,…,M} are sorted and rewritten as where t m represents the order of sorting, and Finally, the present invention constructs a segmentation preference sorting set Ω for the input image I, which contains a series of preference data pairs (for example, one of the preference data pairs is

[0099] Using the constructed preference ranking set Ω, the present invention optimizes the segmentation model of the present invention by contrastive learning. Ω contains multiple preference data pairs, such as in, represents the high-quality segmentation result of a given input image I, as a positive sample, and Indicates a relatively poor segmentation result. As a negative sample, the definition of positive and negative samples is relative. For two results, the one with a higher evaluation index is the positive sample, and the poor one is the negative sample.

[0100] S700, constructing a second loss function based on the preference data, and then performing a second training on the segmentation model;

[0101] It should be noted that, in some embodiments, constructing a second loss function based on the preference data pair may include the following steps: constructing a contrast loss and a cross-entropy loss function based on the preference data pair; balancing the contrast loss based on preset hyperparameters, and then constructing a second loss function in combination with the cross-entropy loss function.

[0102] In some embodiments, constructing a contrast loss based on the preference data pairs may include the following steps: obtaining a generation probability of each segmentation result; constructing a contrast loss based on the generation probabilities corresponding to the positive samples and negative samples in all the preference data pairs;

[0103] Among them, the expression of contrast loss is:

[0104]

[0105] Where, represents contrast loss; Y a represents a positive sample; Y b represents negative samples; π φ (Y a |I) means that when given an input image I, the segmentation model generates Y a The generation probability of π φ (Y b|I) means that when given an input image I, the segmentation model generates Y b Ω represents the preference ranking set composed of all segmentation results, and the segmentation results in the preference ranking set are arranged in the order of their corresponding reward scores.

[0106] For example, in some specific implementations, the network is optimized using the following contrast loss:

[0107]

[0108] Here, π φ (Y t |I) means that when given an input image I, the segmentation model generates Y t probability. The purpose is to ensure that the network generates The probability of generating The probability of effectively combining positive and negative supervisory signals. This formula can be further extended to the entire preference ranking set Ω:

[0109]

[0110] Different from the loss functions generally used in medical image segmentation, By ensuring that the probability of generating better results is higher than the probability of generating worse results, the segmentation prediction is aligned with the evaluation preference. This loss function is independent of the specific form of the evaluation metric and provides flexibility to adapt to various metrics.

[0111] S800 , performing image segmentation on the image to be processed using the multi-decision aggregation network after the second training to obtain an image segmentation result.

[0112] In order to explain the principle of the technical solution of the present invention in detail, the overall process of the present invention is described below in combination with some specific embodiments. It is easy to understand that the following is an explanation of the technical principle of the present invention and cannot be regarded as a limitation of the present invention.

[0113] First, it's important to note that in medicine, image segmentation technology plays a crucial role in accurate disease diagnosis and subsequent treatment strategies. Numerous evaluation metrics have emerged to measure image segmentation quality. However, existing methods require designing specific proxy loss functions to optimize these metrics. This process is both cumbersome and time-consuming, especially when the metrics are non-differentiable and complex.

[0114] In view of this, the present invention proposes a multi-decision preference optimization network, such as Figure 3As shown, the network adopts preference optimization and hybrid expert algorithm to design a unified optimization scheme that can optimize any evaluation index. Unlike existing methods, the multi-decision preference optimization network does not design a specific proxy loss function for each evaluation index, but guides the segmentation model through comparative learning to generate results consistent with the evaluation index preference. In addition, the multi-decision preference optimization network uses a single preference optimization for supervised optimization of all evaluation indicators, eliminating the problem of balancing different loss functions during training in existing methods. On this basis, the multi-decision preference optimization network further constructs segmentation data that can reflect the evaluation index preference, designs a multi-decision aggregation network and segmentation optimization based on index preference to achieve effective segmentation preference optimization. Experiments on multiple public benchmark datasets have demonstrated the advanced nature of the multi-decision preference optimization network in segmentation performance. The contributions of this invention can be summarized as follows:

[0115] 1. A multi-decision preference optimization network is proposed, which designs a unified framework through preference optimization and hybrid expert algorithm to optimize any evaluation index.

[0116] 2. Two key modules, multi-decision aggregation network and segmentation optimization based on indicator preference, are introduced to establish an effective preference optimization paradigm for medical image segmentation.

[0117] 3. Experiments on multiple public benchmarks have verified the effectiveness of the method of the present invention, and the method of the present invention can achieve the most advanced medical segmentation performance.

[0118] The goal of medical image segmentation is to classify each pixel in the input medical image I into a set of categories In this task, various evaluation metrics (such as pixel accuracy, Dice similarity coefficient, Hausdorff distance, sensitivity, etc.) are used to evaluate segmentation performance from different perspectives. Existing methods generally design specific differentiable proxy loss functions for different metrics to optimize the segmentation model.

[0119] However, in the field of medical image segmentation, evaluation metrics are extremely diverse, and some are non-differentiable, making it challenging to design a specific proxy loss function for each metric. Furthermore, optimizing multiple metrics simultaneously during training is also difficult, requiring a careful balance of different loss functions during training.

[0120] In order to overcome these difficulties, the present invention proposes an innovative medical image segmentation framework - multi-decision preference optimization network, which integrates preference optimization and expert mixture strategy to optimize any evaluation index in a unified way. The network consists of two core parts: multi-decision aggregation network and segmentation optimization module based on index preference. As the backbone network of the segmentation task, the multi-decision aggregation network is responsible for generating diversified segmentation results of the input image. The segmentation optimization module based on index preference focuses on constructing segmentation data and optimizing the segmentation model accordingly. The entire workflow of the multi-decision preference optimization network is described in Figure 2 The details are shown in . The specific implementation can be as follows:

[0121] Input a medical image I, the multi-decision preference optimization network first passes through the visual encoder Extracting visual embeddings:

[0122]

[0123] in, is the visual embedding of the input image I. Here, S and C represent the number of tokens and latent dimension, respectively. is a visual encoder. Then, the present invention designs multiple decoder decision sub-networks, defined as {ε m (·)|n=1,…,M}, used to generate diverse segmentation results:

[0124]

[0125] in, Y represents the segmentation probability map generated by the mth decision sub-network for X. m Represents the corresponding segmentation result prediction map. Find the index of the maximum value in the channel dimension, and get the corresponding segmentation prediction map Y m Each decision subnetwork ε m (·) are expected to adaptively learn expertise for medical image segmentation, thus producing diverse segmentation results Y for X m In order to further enhance the diversity of the decision sub-network, the present invention adds m (·) Dropout operation (random inactivation) is used.

[0126] Since each decision subnetwork ε m (·) can capture diverse information from X and generate different segmentation results. Therefore, this paper designs a sparse router s(·). The router function s(·) is implemented by two layers of full connection + Softmax (FC-ReLU-FC-ReLU-Softmax) layers. It is used to aggregate ε from different decision sub-networks.m Different segmentation results of (·):

[0127]

[0128] in is the final segmentation probability map, and Y is the final prediction map. The router function s(·) is for each corresponding decision subnetwork ε m (·) Assign score s m (X), the score indicates how suitable the mth decision subnetwork is for processing X. In order to encourage efficient sparse routing, the present invention designs an entropy penalty loss function to minimize the entropy of the router scores of different decision subnetworks:

[0129]

[0130] The entropy penalty loss function can make the decision subnetwork ε that is most suitable for processing X m (·) is highlighted. Therefore, during the training process, different decision sub-networks focus on different images, so that specific and complementary medical knowledge can be learned. The final prediction map Y is obtained through the probability map {ε m (X)|m=1,…,M} are aggregated.

[0131] During the training process, the present invention applies the cross entropy loss function to the final prediction graph Y:

[0132]

[0133] Among them, N and They represent the number of pixels and the number of categories of image I respectively. Represents the true label that the i-th pixel belongs to category c

[0134] The present invention utilizes multiple parallel decision sub-networks to generate diverse segmentation results, which can be used to construct subsequent segmentation data. It is worth noting that the sparse router and dropout operations ensure that for the same input, the {Y m |m=1,…,M} has sufficient diversity.

[0135] Next, for multiple segmentation results {Y m |m=1,…,M}, the present invention proposes a segmentation optimization module based on index preference to learn the knowledge of evaluation preference. The segmentation optimization module based on index preference is used to calculate the value of each result Y. m The reward score can be used to construct segmentation data for various target evaluation indicators. This segmentation data is then used for preference supervision to guide the optimization of the segmentation model through contrastive learning, encouraging the generation of high-quality results and suppressing poor-quality results.

[0136] The present invention first designs a reward function to evaluate different segmentation results {Y m |m=1,…,M}. This reward function can be used to score each segmentation result based on any evaluation metric and is defined as follows:

[0137]

[0138] Among them, r m Represents the segmentation result Y of the input image I m Bonus score. It can be any differentiable or non-differentiable evaluation index function, such as Dice coefficient, Hausdorff distance, etc. The calculation results of these evaluation index functions are normalized to the range of 0 to 1. The higher the score, the better the quality of the segmentation result. k Represents the kth evaluation index function The weight of .

[0139] Then, according to {r m |m=1,…,M}, the present invention divides the segmentation result {Y m |m=1,…,M} are sorted and rewritten as where t m represents the order of sorting, and Finally, the present invention constructs a segmentation preference sorting set Ω for the input image I, which contains a series of preference data pairs (for example, one of the preference data pairs is

[0140] Using the constructed preference ranking set Ω, the present invention optimizes the segmentation model of the present invention by contrastive learning. Ω contains multiple preference data pairs, such as in, represents the high-quality segmentation result of a given input image I, as a positive sample, and Relatively poor segmentation results are used as negative samples. The definition of positive and negative samples is relative. For two results, the one with a higher evaluation index is the positive sample, and the one with a poor evaluation index is the negative sample. The present invention optimizes the network through the following contrast loss:

[0141]

[0142] Here, π φ (Y t |I) means that when given an input image I, the segmentation model generates Y t probability. The purpose is to ensure that the network generates Y t1 The probability of generating Y is higher than t2The probability of effectively combining positive and negative supervisory signals. This formula can be further extended to the entire preference ranking set Ω:

[0143]

[0144] Different from the loss functions generally used in medical image segmentation, By ensuring that the probability of generating better results is higher than the probability of generating worse results, the segmentation prediction is aligned with the evaluation preference. This loss function is independent of the specific form of the evaluation metric and provides flexibility to adapt to various metrics.

[0145] Ultimately, the overall framework of the present invention follows a two-stage training process, consisting of a pre-training phase and a preference alignment phase. During the pre-training phase, the present invention uses the sum of the cross-entropy loss function (Equation 5) and the entropy penalty loss function (Equation 4) as the total loss function to train the multi-decision aggregation network. This training method enables the multi-decision aggregation network to produce diverse segmentation results, laying the foundation for subsequent segmentation data construction.

[0146] In the preference alignment stage, the present invention uses the trained multi-decision aggregation network to construct segmentation data and combines the contrast loss in formula (8) with the cross entropy loss function in formula (5) to learn the evaluation preference knowledge and optimize the network:

[0147]

[0148] Here, γ is a hyperparameter used to balance different loss terms. In formula (9) The purpose is to stabilize the training process and ensure that the network of the present invention does not significantly deviate from its pre-trained state during the preference alignment phase.

[0149] After a total of 200 epochs of training, the network of the present invention achieved the current best results in three widely used medical image segmentation benchmark datasets. The present invention uses the Dice similarity coefficient (DSC), Hausdorff distance (HD), sensitivity (SE), specificity (SP) and accuracy (ACC) indicators. Specifically, the method of the present invention is verified on the following three medical image segmentation benchmark datasets: Synapse multi-organ CT dataset, ISIC 2017 and ISIC 2018. The Synapse multi-organ CT dataset consists of 30 abdominal CT scan cases, a total of 3,779 slices, covering eight abdominal organs such as the aorta, gallbladder, spleen, left kidney, right kidney, liver, pancreas and stomach. The dataset is divided into a training set (18 cases, 2,212 axial slices) and a test set (12 cases, 1,567 axial slices). The ISIC 2017 and ISIC 2018 datasets focus on skin lesion segmentation and provide annotations of cancerous and non-cancerous lesions. The ISIC 2017 dataset contains 2,000 training dermatoscope images, 150 validation images, and 600 test images. Following the setup of existing research, the ISIC 2018 dataset is divided into 1,815 training images, 259 validation images, and 520 test images.

[0150] The performance on all datasets surpassed the existing methods. In particular, on the Synapse multi-organ CT dataset, the method of the present invention outperformed other methods in all five evaluation indicators, with improvements of 0.31%, 0.01%, 0.97%, 0.01% and 1.0% respectively. On the ISIC 2017 dataset. The method of the present invention was ahead of the second-place method in DSC, HD, SE and SP indicators, with improvements of 0.82%, 0.2172, 0.09% and 0.04% respectively, and also showed the best performance comparable to the best existing method in the ACC indicator. On the ISIC 2018 dataset, the method of the present invention achieved the highest performance in DSC, HD, SP and ACC indicators, with improvements of 0.59%, 0.1552, 0.69% and 0.02% respectively, and also showed performance comparable to the best method in SE indicator.

[0151] In summary, the present invention proposes an innovative medical image segmentation framework - a multi-decision preference optimization network, which can optimize various evaluation indicators in a unified framework without the need to design a specific proxy loss function for each evaluation indicator. It is simpler and more practical. Specifically, the multi-decision preference optimization network designs multiple parallel decision sub-networks to generate multiple segmentation results, and scores and ranks these segmentation results through a reward function based on the evaluation indicator. The ranking result can reflect the preference of the evaluation indicator for the result. Based on this evaluation preference, the multi-decision preference optimization network designs a segmentation preference optimization mechanism, which guides the segmentation model through comparative learning, so that the segmentation model can ultimately generate results consistent with the evaluation indicator. The present invention has conducted experiments on multiple challenging public datasets and proved that the multi-decision preference optimization network can achieve superior segmentation performance compared with existing methods.

[0152] like Figure 4 As shown, the embodiment of the present invention further provides an image segmentation device 900, which may include:

[0153] The first module 901 is used to obtain an input image; the input image carries the labeling information of the category of each pixel;

[0154] The second module 902 is used to input the input image into a preset multi-decision aggregation network to perform image segmentation prediction and obtain a segmentation prediction map;

[0155] The multi-decision aggregation network includes a segmentation model and a sparse router; the segmentation model includes a visual encoder and a preset number of decision sub-networks connected in parallel after the visual encoder; the decision sub-networks use a decoder; the sparse router is used to assign a weighted score to the data results of each decision sub-network; the segmentation prediction map includes the prediction result of the category of each pixel in the input image;

[0156] The third module 903 is used to construct a first loss function based on the weighted scores, the prediction results and the annotation information, and then pre-train the multi-decision aggregation network;

[0157] The fourth module 904 is configured to obtain segmentation data based on the input image using the segmentation model in the pre-trained multi-decision aggregation network; the segmentation data includes the segmentation results of each decision sub-network;

[0158] The fifth module 905 is used to perform an indicator evaluation on all segmentation results based on a preset indicator to obtain a reward score for each segmentation result;

[0159] The sixth module 906 is configured to construct a preference data pair based on the segmentation result based on the reward score; the preference data pair includes a positive sample and a negative sample, and the reward score of the positive sample is greater than the reward score of the negative sample;

[0160] The seventh module 907 is used to construct a second loss function based on the preference data, and then perform a second training on the segmentation model;

[0161] The eighth module 908 is used to perform image segmentation on the image to be processed using the second trained multi-decision aggregation network to obtain an image segmentation result.

[0162] The contents of the method embodiments of the present invention are all applicable to the device embodiments. The functions specifically implemented by the device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0163] An embodiment of the present invention further provides an electronic device comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements the above-described image segmentation method when executing the computer program. The electronic device can be any intelligent terminal, such as a tablet computer or an in-vehicle computer.

[0164] It can be understood that the contents of the above method embodiments are applicable to the present device embodiments, the functions specifically implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0165] See also Figure 5 , Figure 5 The hardware structure of an electronic device 1000 according to another embodiment is shown. The electronic device 1000 includes:

[0166] The processor 1001 may be implemented as a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is configured to execute relevant programs to implement the technical solutions provided by the embodiments of the present invention.

[0167] The memory 1002 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 1002 can store an operating system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program codes are stored in the memory 1002 and are called by the processor 1001 to execute the image segmentation method of the embodiments of the present invention.

[0168] Input / output interface 1003, used to implement information input and output;

[0169] Communication interface 1004, used to implement communication interaction between this device and other devices, which can be achieved through wired means (such as USB, network cable, etc.) or wireless means (such as mobile network, WiFi, Bluetooth, etc.);

[0170] Bus 1005 , which transmits information between various components of the device (e.g., processor 1001 , memory 1002 , input / output interface 1003 , and communication interface 1004 );

[0171] The processor 1001 , the memory 1002 , the input / output interface 1003 and the communication interface 1004 are connected to each other in communication within the device via the bus 1005 .

[0172] An embodiment of the present invention further provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the image segmentation method described above is implemented.

[0173] It can be understood that the contents of the above method embodiments are all applicable to the present storage medium embodiment, the functions specifically implemented by the present storage medium embodiment are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0174] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely arranged relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0175] The image segmentation method, image segmentation device, electronic device and storage medium provided by the embodiment of the present invention are as follows: obtaining an input image; the input image carries the labeling information of the category of each pixel; inputting the input image into a preset multi-decision aggregation network to perform image segmentation prediction, and obtaining a segmentation prediction map; wherein the multi-decision aggregation network includes a segmentation model and a sparse router; the segmentation model includes a visual encoder and a preset number of decision sub-networks connected in parallel after the visual encoder; the decision sub-network uses a decoder; the sparse router is used to assign a weighted score to the data result of each decision sub-network; the segmentation prediction map includes the prediction result of the category of each pixel in the input image; based on the weighted score and the prediction result and the labeling information Construct a first loss function, and then pre-train the multi-decision aggregation network; based on the input image, use the segmentation model in the pre-trained multi-decision aggregation network to obtain segmentation data; the segmentation data includes the segmentation results of each decision sub-network; perform an indicator evaluation on all segmentation results based on preset indicators to obtain a reward score for each segmentation result; based on the reward score, construct a preference data pair according to the segmentation result; the preference data pair includes positive samples and negative samples, and the reward score of the positive sample is greater than the reward score of the negative sample; construct a second loss function based on the preference data pair, and then perform a second training on the segmentation model; use the multi-decision aggregation network after the second training to perform image segmentation on the processed image to obtain the image segmentation result. The embodiment of the present invention integrates preference optimization and expert mixing strategy, and uses a unified method to optimize any evaluation index; uses the multi-decision aggregation network as the backbone network of the segmentation task, which is responsible for generating diversified segmentation results of the input image, and then constructs segmentation data and optimizes the segmentation model accordingly. The embodiment of the present invention can balance multiple indicators to achieve accurate image segmentation.

[0176] The embodiments described in the embodiments of the present invention are intended to more clearly illustrate the technical solutions of the embodiments of the present invention and do not constitute a limitation on the technical solutions provided by the embodiments of the present invention. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of the present invention are equally applicable to similar technical problems.

[0177] Those skilled in the art will understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present invention, and may include more or fewer steps than shown in the figures, or a combination of certain steps, or different steps.

[0178] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, i.e., they may be located in one place or distributed across multiple network units. Some or all of the modules may be selected based on actual needs to achieve the objectives of this embodiment.

[0179] Those skilled in the art will appreciate that all or some of the steps, devices, and functional modules / units in the methods disclosed above may be implemented as software, firmware, hardware, or appropriate combinations thereof.

[0180] The terms "first," "second," "third," "fourth," and the like (if any) in the description of the present invention and the accompanying drawings are used to distinguish similar objects and are not necessarily used to describe a particular order or precedence. It should be understood that the terms used in this manner are interchangeable where appropriate so that the embodiments of the present invention described herein can be implemented in orders other than those illustrated or described herein. In addition, the terms "including" and "having," and any variations thereof, are intended to cover non-exclusive inclusions. For example, a process, method, apparatus, product, or device that includes a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units that are not explicitly listed or that are inherent to such processes, methods, products, or devices.

[0181] It should be understood that in the present invention, "at least one (item)" refers to one or more, and "plurality" refers to two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can represent: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the previous and next associated objects are in an "or" relationship. "At least one of the following items" or similar expressions refers to any combination of these items, including any combination of single items or plural items. For example, at least one of a, b or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0182] In the several embodiments provided by the present invention, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the division of the above units is merely a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. In addition, the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0183] The units described above as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0184] In addition, the functional units in the various embodiments of the present invention may be integrated into a single processing unit, each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0185] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes multiple instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the method of each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), magnetic disk or optical disk, and other media that can store programs.

[0186] The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but are not intended to limit the scope of the present invention. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and essence of the present invention should be within the scope of the present invention.

Claims

1. An image segmentation method, characterized in that: The method comprises the following steps: Obtain an input image; the input image carries the labeling information of the category of each pixel; Inputting the input image into a preset multi-decision aggregation network to perform image segmentation prediction to obtain a segmentation prediction map; The multi-decision aggregation network includes a segmentation model and a sparse router; the segmentation model includes a visual encoder and a preset number of decision sub-networks connected in parallel after the visual encoder; the decision sub-networks use a decoder; the sparse router is used to assign a weighted score to the data results of each decision sub-network; the segmentation prediction map includes a prediction result for the category of each pixel in the input image; Constructing a first loss function based on the weighted score, the prediction result, and the annotation information, and then pre-training the multi-decision aggregation network; Based on the input image, using the segmentation model in the pre-trained multi-decision aggregation network to obtain segmentation data; the segmentation data includes the segmentation results of each decision sub-network; Performing an indicator evaluation on all the segmentation results based on preset indicators to obtain a reward score for each segmentation result; Based on the reward score, constructing a preference data pair according to the segmentation result; the preference data pair includes a positive sample and a negative sample, and the reward score of the positive sample is greater than the reward score of the negative sample; Constructing a second loss function based on the preference data, and then performing a second training on the segmentation model; The multi-decision aggregation network after the second training is used to perform image segmentation on the image to be processed to obtain an image segmentation result.

2. The image segmentation method according to claim 1, wherein: The step of inputting the input image into a preset multi-decision aggregation network to perform image segmentation prediction to obtain a segmentation prediction map comprises the following steps: Visually encode the input image using the visual encoder to extract a visual embedding; Based on the visual embedding, generating a corresponding first segmentation probability map through each of the decision sub-networks; a random deactivation operation is used in each of the decision sub-networks; Assigning a weighted score to the first segmentation probability map corresponding to each decision subnetwork through the sparse router, and accumulating and summing the products of all the first segmentation probability maps and their corresponding weighted scores to obtain a second segmentation probability map; the sparse router includes a full connection and an activation function; The maximum value index is found in the channel dimension of the second segmentation probability map to obtain the segmentation prediction map.

3. The image segmentation method according to claim 1 or 2, characterized in that: The constructing of a first loss function based on the weighted score, the prediction result, and the annotation information comprises the following steps: constructing an entropy penalty loss function based on the weighted scores; Constructing a cross entropy loss function based on the prediction result and the annotation information; The first loss function is obtained according to the result of adding the entropy penalty loss function and the cross entropy loss function.

4. The image segmentation method according to claim 3, characterized in that The step of constructing an entropy penalty loss function based on the weighted score comprises the following steps: Solving a logarithmic function for the weighted scores obtained by allocating each decision subnetwork; The products of the weighted scores obtained by all the decision sub-networks and the corresponding logarithmic function solution values ​​are accumulated and summed to obtain the entropy penalty loss function.

5. The image segmentation method according to claim 1, wherein: The step of performing an indicator evaluation on all the segmentation results based on a preset indicator to obtain a reward score for each segmentation result includes the following steps: Performing an index evaluation on the segmentation result based on preset indicators to obtain a function value of an evaluation index function corresponding to each preset indicator of the segmentation result; A weighted sum is performed on the function values ​​of the segmentation result in all the evaluation index functions to obtain the reward score corresponding to the segmentation result.

6. The image segmentation method according to claim 1, wherein: The constructing of a second loss function based on the preference data comprises the following steps: Constructing contrast loss and cross entropy loss functions based on the preference data pairs; The contrast loss is balanced based on preset hyperparameters, and then the second loss function is constructed in combination with the cross entropy loss function.

7. The image segmentation method according to claim 6, characterized in that: The step of constructing the contrast loss based on the preference data comprises the following steps: Obtaining a generation probability of each segmentation result; Constructing the contrast loss based on the generation probabilities corresponding to the positive samples and the negative samples in all the preference data pairs; The expression of the contrast loss is: ； Where, represents contrast loss; represents a positive sample; represents negative samples; Indicates that given an input image When , the segmentation model is generated The probability of generation; Indicates that given an input image When , the segmentation model is generated The probability of generation; Represents a preference sorted set consisting of all segmentation results, and the segmentation results in the preference sorted set are arranged in the order of their corresponding reward scores.

8. An image segmentation device, characterized in that: The device comprises: The first module is configured to obtain an input image, wherein the input image carries the labeling information of the category of each pixel; The second module is used to input the input image into a preset multi-decision aggregation network to perform image segmentation prediction and obtain a segmentation prediction map; The multi-decision aggregation network includes a segmentation model and a sparse router; the segmentation model includes a visual encoder and a preset number of decision sub-networks connected in parallel after the visual encoder; the decision sub-networks use a decoder; the sparse router is used to assign a weighted score to the data results of each decision sub-network; the segmentation prediction map includes a prediction result for the category of each pixel in the input image; A third module is configured to construct a first loss function based on the weighted score, the prediction result, and the annotation information, thereby pre-training the multi-decision aggregation network; A fourth module is configured to obtain segmentation data based on the input image using the segmentation model in the pre-trained multi-decision aggregation network; the segmentation data includes a segmentation result of each decision subnetwork; A fifth module is configured to perform an indicator evaluation on all the segmentation results based on a preset indicator to obtain a reward score for each segmentation result; A sixth module is configured to construct a preference data pair based on the segmentation result based on the reward score; the preference data pair includes a positive sample and a negative sample, and the reward score of the positive sample is greater than the reward score of the negative sample; A seventh module is configured to construct a second loss function based on the preference data, thereby performing a second training on the segmentation model; An eighth module is configured to perform image segmentation on the image to be processed using the multi-decision aggregation network after the second training to obtain an image segmentation result.

9. An electronic device, characterized in that: The electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the method according to any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images