Model determination method and device based on image segmentation, equipment, medium and product

Abstract

The invention relates to the technical field of computers, and discloses a model determination method and device based on image segmentation, equipment, a medium and a product, and the method comprises the steps: obtaining training data of image segmentation; inputting the training data into an initial model for feature extraction and feature alignment to obtain an initial feature and a first loss function; inputting the initial features into an initial model to obtain a context information queue; inputting the initial feature and the context information queue into an initial model to obtain a target feature; inputting the target features into a classifier to obtain target prediction data and a second loss function; and adjusting model parameters of the initial model by using the first loss function and the second loss function to obtain a target model. According to the method and the device, the initial features are obtained by performing feature extraction on the training data and then performing feature alignment, then the context information queue is obtained according to the initial features, and the target features are further obtained, so that the universality of the image segmentation model to a real scene and the generalization of the image segmentation model to an unknown scene are improved.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, and in particular to a method, device, equipment, medium and product for determining a model based on image segmentation. Background Art

[0002] Colon cancer is the third most common malignant tumor in the world, and the number of deaths caused by colon cancer is increasing year by year. Polyps are considered to be a precursor to colon cancer, so effective examination and removal of polyps are crucial. Clinically, the diagnosis of polyps is done by experienced doctors using endoscopes, but this traditional method has the disadvantages of high labor costs and missed diagnosis rate.

[0003] To address the above problems, related technologies usually adopt methods based on convolutional neural networks, mainly following the encoder-decoder framework, to complete the segmentation task in an end-to-end manner; or adopt methods based on attention networks to extract multiple layers of feature maps with different resolutions, obtain preliminary prediction results of each layer through attention-guided multi-layer aggregation strategies, and aggregate them into final predictions. However, the above methods have the following limitations: (1) The above methods are based on the premise that the test set and the training set are in the same data distribution (image color, texture and other styles, equipment parameters, and patient conditions are the same). However, in reality, due to factors such as data collection by different institutions, the use of different equipment and the detection of different patients, there are large differences in the endoscopic images of intestinal polyps. The size, color, texture and other appearances of polyps are often different, making it difficult to apply to real scenarios; (2) During training, it is necessary to collect test set images in advance. In real scenarios, it is difficult to collect polyp images of patients to be diagnosed in advance. When facing unknown scenarios, it is necessary to reorganize the test set images for retraining, and the generalization ability is poor.

[0004] Therefore, the polyp segmentation scheme in the related art has the problems of poor universality for real scenes and poor generalization for unknown scenes. Summary of the invention

[0005] In view of this, the present disclosure provides a model determination method, device, equipment, medium and product based on image segmentation to solve the problem that the polyp segmentation solution of the related art has poor universality for real scenes and poor generalization for unknown scenes.

[0006] In a first aspect, the present disclosure provides a model determination method based on image segmentation, the method comprising:

[0007] Get training data for image segmentation;

[0008] Input the training data into the initial model for feature extraction and feature alignment to obtain initial features and a first loss function, wherein the initial features are used to characterize pixel information of the training data image;

[0009] Inputting the initial features into the initial model to obtain a context information queue, wherein the context information queue includes current context information obtained from the initial features and historical context information obtained from the historical features;

[0010] Input the initial features and context information queue into the initial model to obtain the target features;

[0011] Inputting the target feature into the classifier to obtain target prediction data and a second loss function, wherein the target prediction data is used to represent the final prediction probability that the training data belongs to the first data type and the second data type;

[0012] The model parameters of the initial model are adjusted using the first loss function and the second loss function to obtain the target model.

[0013] In the disclosed embodiment, by acquiring training data for image segmentation; inputting the training data into the initial model for feature extraction and feature alignment, obtaining initial features and a first loss function, wherein the initial features are used to characterize the pixel information of the training data image; inputting the initial features into the initial model to obtain a context information queue, wherein the context information queue contains current context information obtained from the initial features and historical context information obtained from historical features; inputting the initial features and the context information queue into the initial model to obtain target features; inputting the target features into the classifier to obtain target prediction data and a second loss function, wherein the target prediction data is used to represent the final prediction probability that the training data belongs to the first data type and the second data type; adjusting the model parameters of the initial model using the first loss function and the second loss function to obtain the target model. Since the disclosed method extracts features from the training data and then performs feature alignment to obtain initial features, then obtains a context information queue based on the initial features, and further obtains the target features based on the initial features and the context information queue, the universality of the image segmentation model for real scenes and the generalization for unknown scenes are improved.

[0014] In an optional implementation, the training data carries label data, wherein the label data is used to characterize that the training data is a first data type or a second data type. Before the training data is input into the initial model for feature extraction and feature alignment to obtain initial features and a first loss function, the method further includes:

[0015] Preprocess the label data to obtain the target label;

[0016] Perform data preprocessing and data enhancement on the training data to obtain processed training data;

[0017] The processed training data is input into the initial model for feature extraction.

[0018] In the disclosed embodiment, by preprocessing the label data and the training data to obtain the target label and the processed training data, the data format can be unified, which is convenient for the model to process the label data and the training data later. By performing data enhancement on the training data, the training data can be made more robust, and the generalization ability of the model for different scenes when performing image segmentation can be improved.

[0019] In an optional implementation, the training data is input into the initial model for feature extraction and feature alignment to obtain initial features and a first loss function, including:

[0020] According to the target label, obtain the label feature;

[0021] Input the training data into the feature extraction network and the feature alignment network of the initial model to obtain initial features, wherein the initial model includes a plurality of first processing units and a second processing unit, the first processing unit includes a feature extraction network and a corresponding feature alignment network, and the second processing unit includes a feature extraction network;

[0022] Extracting first features output by each first processing unit;

[0023] According to the first feature and the label feature, a first loss function is obtained.

[0024] In the disclosed embodiment, by using the feature alignment network to align the features extracted by the feature extraction network to obtain the initial features, it is possible to alleviate the feature mismatch problem caused by scene differences while maintaining the differences between different scene images. By using the obtained first loss function for the parameter update of the initial model, the initial model can be optimized and the image segmentation effect of the initial model can be improved.

[0025] In an optional implementation, the initial features are input into the initial model to obtain a context information queue, including:

[0026] Passing the initial features through the first network, the second network, and the third network of the initial model in sequence to obtain initial prediction data, wherein the initial prediction data is used to indicate the initial prediction probability that the training data belongs to the first data type and the second data type;

[0027] According to the initial features and initial prediction data, the current context information is obtained;

[0028] A context information queue is obtained according to the current context information and the historical context information.

[0029] In the embodiment of the present disclosure, the context information is stored by obtaining the current context information according to the initial features and combining it with the historical context information to obtain a context information queue.

[0030] In an optional implementation, the initial features and the context information queue are input into the initial model to obtain the target features, including:

[0031] According to the initial features and the context information queue, a relationship matrix is ​​obtained, wherein the relationship matrix is ​​used to represent the similarity relationship between the initial features and the context information queue;

[0032] According to the context information queue and relationship matrix, intermediate features are obtained;

[0033] According to the initial features and intermediate features, the target features are obtained.

[0034] In the disclosed embodiment, by obtaining a relationship matrix based on the initial features and the context information queue, and further obtaining intermediate features based on the context information queue and the relationship matrix, the initial features can be enhanced using the context information queue to improve the generalization of the image segmentation model for unknown scenes. By obtaining the target features based on the initial features and the intermediate features, feature aggregation can be achieved to make the target features more recognizable.

[0035] In an optional implementation, the target feature is input into a classifier to obtain target prediction data and a second loss function, including:

[0036] Input the target features into the classifier to obtain the target prediction data;

[0037] According to the target prediction data and the target label, the second loss function is obtained.

[0038] In the disclosed embodiment, by inputting the target feature into the classifier to obtain the target prediction data, the image can be accurately segmented. By using the obtained second loss function for updating the parameters of the initial model, the initial model can be optimized and the image segmentation effect of the initial model can be improved.

[0039] In a second aspect, the present disclosure provides a model determination device based on image segmentation, the device comprising:

[0040] An acquisition module, used to acquire training data for image segmentation;

[0041] A first obtaining module is used to input the training data into the initial model for feature extraction and feature alignment to obtain initial features and a first loss function, wherein the initial features are used to characterize pixel point information of the training data image;

[0042] The second obtaining module is used to input the initial features into the initial model to obtain a context information queue, wherein the context information queue contains current context information obtained from the initial features and historical context information obtained from the historical features;

[0043] The third module is used to input the initial features and context information queue into the initial model to obtain the target features;

[0044] A fourth obtaining module is used to input the target feature into the classifier to obtain target prediction data and a second loss function, wherein the target prediction data is used to represent the final prediction probability that the training data belongs to the first data type and the second data type;

[0045] The fifth obtaining module is used to adjust the model parameters of the initial model using the first loss function and the second loss function to obtain the target model.

[0046] In a third aspect, the present disclosure provides a computer device, comprising: a memory and a processor, the memory and the processor being communicatively connected to each other, the memory storing computer instructions, and the processor executing the image segmentation-based model determination method of the above-mentioned first aspect or any corresponding embodiment thereof by executing the computer instructions.

[0047] In a fourth aspect, the present disclosure provides a computer-readable storage medium having computer instructions stored thereon, the computer instructions being used to enable a computer to execute the image segmentation-based model determination method of the above-mentioned first aspect or any corresponding embodiment thereof.

[0048] In a fifth aspect, the present disclosure provides a computer program product, including computer instructions, which are used to enable a computer to execute the image segmentation-based model determination method of the above-mentioned first aspect or any corresponding embodiment. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] In order to more clearly illustrate the specific embodiments of the present disclosure or the technical solutions in the prior art, the drawings required for use in the specific embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0050] Figure 1 is a flow chart of a model determination method based on image segmentation according to an embodiment of the present disclosure;

[0051] Figure 2 is a flow chart of a model determination method based on image segmentation according to another embodiment of the present disclosure;

[0052] Figure 3 is a structural block diagram of a model determination device based on image segmentation according to an embodiment of the present disclosure;

[0053] Figure 4 It is a schematic diagram of the hardware structure of the computer device of the embodiment of the present disclosure. DETAILED DESCRIPTION

[0054] In order to make the purpose, technical solution and advantages of the embodiments of the present disclosure clearer, the technical solution in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are part of the embodiments of the present disclosure, rather than all the embodiments. Based on the embodiments in the present disclosure, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present disclosure.

[0055] Colon cancer is the third most common malignant tumor in the world, and the number of deaths caused by colon cancer is increasing year by year. Polyps are considered to be a precursor to colon cancer, so effective examination and removal of polyps are crucial. Clinically, the diagnosis of polyps is done by experienced doctors using endoscopes, but this traditional method has the disadvantages of high labor costs and missed diagnosis rate.

[0056] To address the above problems, related technologies usually adopt methods based on convolutional neural networks, mainly following the encoder-decoder framework, to complete the segmentation task in an end-to-end manner; or adopt methods based on attention networks to extract multiple layers of feature maps with different resolutions, obtain preliminary prediction results of each layer through attention-guided multi-layer aggregation strategies, and aggregate them into final predictions. However, the above methods have the following limitations: (1) The above methods are based on the premise that the test set and the training set are in the same data distribution (image color, texture and other styles, equipment parameters, and patient conditions are the same). However, in reality, due to factors such as data collection by different institutions, the use of different equipment and the detection of different patients, there are large differences in the endoscopic images of intestinal polyps. The size, color, texture and other appearances of polyps are often different, making it difficult to apply to real scenarios; (2) During training, it is necessary to collect test set images in advance. In real scenarios, it is difficult to collect polyp images of patients to be diagnosed in advance. When facing unknown scenarios, it is necessary to reorganize the test set images for retraining, and the generalization ability is poor.

[0057] In order to solve the above problems, according to an embodiment of the present disclosure, an embodiment of a model determination method based on image segmentation is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although the logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0058] In this embodiment, a model determination method based on image segmentation is provided. Figure 1 As shown, Figure 1: is a flow chart of a model determination method based on image segmentation according to an embodiment of the present disclosure. The flow chart can be applied to a server and includes the following steps:

[0059] Step S101, obtaining training data for image segmentation.

[0060] Optionally, in the disclosed embodiment, the server first selects part of the data sets of polyp segmentation public data sets Kvasir-SEG, CVC-ClinicDB, CVC-ColonDB and ETIS as training data for image segmentation, wherein the training data for image segmentation may be some color photos.

[0061] Step S102: input the training data into the initial model for feature extraction and feature alignment to obtain initial features and a first loss function, wherein the initial features are used to characterize pixel information of the training data image.

[0062] Alternatively, if Figure 2 As shown, in the embodiment of the present disclosure, the initial model includes a feature extraction and alignment module (DeeplabV2), and the server uses the feature extraction and alignment module to extract and align features of the training data to obtain the initial features F and the first loss function L SAN , where the initial feature F∈T (c×H×W) , where c is the number of channels of the initial feature, H is the height of the initial feature, and W is the width of the initial feature.

[0063] Step S103 , inputting the initial features into the initial model to obtain a context information queue, wherein the context information queue contains current context information obtained from the initial features and historical context information obtained from the historical features.

[0064] Alternatively, if Figure 2 As shown, in the embodiment of the present disclosure, the initial model includes an information storage module, and the server uses the information storage module to calculate the initial feature F to obtain the context information queue

[0065] Specifically, the context information queue Wherein, S is the capacity of the context information queue E in the storage pool. The context information queue E is stored in the storage pool and includes the current context information E and the historical context information. When the storage pool reaches the maximum capacity S, the earliest historical context information is removed and the current context information E is added.

[0066] Step S104, inputting the initial features and the context information queue into the initial model to obtain the target features.

[0067] Alternatively, if Figure 2As shown, in the embodiment of the present disclosure, the initial model includes a feature aggregation module, and the server uses the feature aggregation module to aggregate the initial features F and the context information queue. Calculate and get the target feature F agg .

[0068] Step S105, inputting the target feature into the classifier to obtain target prediction data and a second loss function, wherein the target prediction data is used to represent the final prediction probability that the training data belongs to the first data type and the second data type.

[0069] Optionally, in the embodiment of the present disclosure, the server sets the target feature F agg Input the classifier to obtain the target prediction data P (h,w) and the second loss function L seg , where the target prediction data P (h,w) The final prediction probability used to represent the training data as the first data type (normal tissue category) and the second data type (polyp category), the target prediction data P (h,w) The value range is any value between 0 and 1. The target prediction data P (h,w) The higher it is, the more likely the training data is a polyp. 0.5 is used as the division threshold, h is the height of the target feature, and w is the width of the target feature.

[0070] Specifically, the classifier can use Atrous Spatial Pyramid Pooling (ASPP) and can perform upsampling during classification. Upsampling refers to the process of enlarging a low-resolution feature map to the original resolution, including upsampling methods such as bilinear interpolation, nearest neighbor interpolation, and deconvolution.

[0071] Step S106, using the first loss function and the second loss function to adjust the model parameters of the initial model to obtain the target model.

[0072] Optionally, in the embodiment of the present disclosure, the server uses the first loss function L SAN and the second loss function L seg Adjust the model parameters of the initial model to obtain the target model.

[0073] Specifically, by transforming the first loss function L SAN and the second loss function L seg By performing linear combination, we can get the loss function L, which is defined as follows:

[0074] L=λ1L SAN +λ2L seg

[0075] Among them, λ1 and λ2 are used to balance the first loss function LSAN and the second loss function L seg The weight relationship between them, the preferred values ​​selected in this embodiment are λ1=0.5, λ2=0.5.

[0076] Then, the server uses the loss function L to iteratively train the initial model through the gradient descent algorithm to update all weight parameters and obtain the target model. The optimizer used is the Adam optimizer, and the learning rate is set to 1×10 -4 , weight decay is set to 1×10 -4 , the training cycle is 300 epochs.

[0077] In the disclosed embodiment, by acquiring training data for image segmentation; inputting the training data into the initial model for feature extraction and feature alignment, obtaining initial features and a first loss function, wherein the initial features are used to characterize the pixel information of the training data image; inputting the initial features into the initial model to obtain a context information queue, wherein the context information queue contains current context information obtained from the initial features and historical context information obtained from historical features; inputting the initial features and the context information queue into the initial model to obtain target features; inputting the target features into the classifier to obtain target prediction data and a second loss function, wherein the target prediction data is used to represent the final prediction probability that the training data belongs to the first data type and the second data type; adjusting the model parameters of the initial model using the first loss function and the second loss function to obtain the target model. Since the disclosed method extracts features from the training data and then performs feature alignment to obtain initial features, then obtains a context information queue based on the initial features, and further obtains the target features based on the initial features and the context information queue, the universality of the image segmentation model for real scenes and the generalization for unknown scenes are improved.

[0078] In some optional embodiments, the training data carries label data, wherein the label data is used to characterize that the training data is a first data type or a second data type. Before the training data is input into the initial model for feature extraction and feature alignment to obtain initial features and a first loss function, the method further includes:

[0079] Preprocess the label data to obtain the target label;

[0080] Perform data preprocessing and data enhancement on the training data to obtain processed training data;

[0081] The processed training data is input into the initial model for feature extraction.

[0082] Optionally, in the embodiment of the present disclosure, the label data includes 0 (normal tissue) and 1 (polyp), which is used to characterize that the training data is a first data type (normal tissue category) or a second data type (polyp category).

[0083] Specifically, before inputting the training data into the initial model, the server may perform data preprocessing on the training data, and specifically adjust the size of the training data to 320×320. After preprocessing the training data, the server may also perform data enhancement on the preprocessed training data, and specifically may perform horizontal and vertical random flipping, scaling, or rotation on the preprocessed training data. The server may also perform data preprocessing on the label data, first converting all label data into grayscale image mode, and then setting the labels with pixel values ​​less than or equal to 150 in the label data to 0 (normal tissue), and setting the labels with pixel values ​​greater than 150 to 1 (polyp), and adjusting the size of the label data to 320×320 to obtain the target label y.

[0084] It should be noted that when the server inputs the processed training data into the initial model, the server can set the number of training data input in each batch. The preferred value selected in this embodiment is 12.

[0085] In the disclosed embodiment, by preprocessing the label data and the training data to obtain the target label and the processed training data, the data format can be unified, which is convenient for the model to process the label data and the training data later. By performing data enhancement on the training data, the training data can be made more robust, and the generalization ability of the model for different scenes when performing image segmentation can be improved.

[0086] In some optional implementations, inputting the training data into the initial model for feature extraction and feature alignment to obtain initial features and a first loss function includes:

[0087] According to the target label, obtain the label feature;

[0088] Input the training data into the feature extraction network and the feature alignment network of the initial model to obtain initial features, wherein the initial model includes a plurality of first processing units and a second processing unit, the first processing unit includes a feature extraction network and a corresponding feature alignment network, and the second processing unit includes a feature extraction network;

[0089] Extracting first features output by each first processing unit;

[0090] According to the first feature and the label feature, a first loss function is obtained.

[0091] Alternatively, if Figure 2As shown, in the disclosed embodiment, the feature extraction and alignment module (DeeplabV2) includes a feature extraction network and a feature alignment network, the first processing unit includes a feature extraction network and a corresponding feature alignment network, and the second processing unit includes a feature extraction network. Preferably, the feature extraction network selected in the disclosed embodiment is a residual network (Residual Networks, ResNet), and the feature alignment network is a standardization module (Semantic-AwareNormalization, SAN).

[0092] It should be noted that, in the first processing unit, the standardization module is used to perform a standardization operation on the features, and transform the features into a distribution with a mean of 0 and a standard deviation of 1, that is, the features are subtracted from the mean and then divided by the standard deviation.

[0093] Specifically, the server extracts shallow features by inputting the training data into the first processing unit for feature extraction and feature alignment, and obtains the first feature F0. The server further inputs the first feature F0 into the second processing unit for feature extraction, extracts deep features, and obtains the initial feature F.

[0094] Then, the server obtains the label feature F according to the target label obj The server calculates the first feature F0 and the label feature F obj , get the first loss function L SAN , the first loss function L SAN The calculation formula is as follows:

[0095] L SAN =||F0-F obj ||1

[0096] In the disclosed embodiment, by using the feature alignment network to align the features extracted by the feature extraction network to obtain the initial features, it is possible to alleviate the feature mismatch problem caused by scene differences while maintaining the differences between different scene images. By using the obtained first loss function for the parameter update of the initial model, the initial model can be optimized and the image segmentation effect of the initial model can be improved.

[0097] In some optional implementations, the initial features are input into the initial model to obtain a context information queue, including:

[0098] Passing the initial features through the first network, the second network, and the third network of the initial model in sequence to obtain initial prediction data, wherein the initial prediction data is used to indicate the initial prediction probability that the training data belongs to the first data type and the second data type;

[0099] According to the initial features and initial prediction data, the current context information is obtained;

[0100] A context information queue is obtained according to the current context information and the historical context information.

[0101] Alternatively, if Figure 2 As shown, in the embodiment of the present disclosure, the first network is a 1×1 convolutional layer, the second network is batch normalization (Batch Normalization, BN), and the third network is an activation function (ReLu, Rectified Linear Unit).

[0102] Specifically, the server performs feature dimensionality reduction on the initial feature F through the first network, batch normalization through the second network, and image segmentation prediction through the third network to obtain the initial prediction data M. In the second network, batch normalization is used to standardize the features of a batch and transform the features into a distribution with a mean of 0 and a standard deviation of 1, that is, the original feature is subtracted from the mean and then divided by the standard deviation.

[0103] It should be noted that M∈R (2×H×W) The initial prediction data M is used to represent the initial prediction probability that the training data is of the first data type (normal tissue category) and the second data type (polyp category). The value range of the initial prediction data M is any value between 0 and 1. The higher the initial prediction data M, the greater the possibility that the training data is a polyp. 0.5 is used as the division threshold, H is the height of the initial feature, and W is the width of the initial feature.

[0104] Then, the server obtains the current context information E based on the initial feature F and the initial prediction data M, stores the current context information E in the storage pool, and obtains the context information queue based on the current context information E and the historical context information. The current context information E is defined as follows:

[0105] E=φ(M) T ·φ(F)

[0106] Here, φ(·) refers to the operation of flattening multidimensional data into a one-dimensional array, and · refers to matrix multiplication.

[0107] In the embodiment of the present disclosure, the context information is stored by obtaining the current context information according to the initial features and combining it with the historical context information to obtain a context information queue.

[0108] In some optional implementations, the initial features and the context information queue are input into the initial model to obtain the target features, including:

[0109] According to the initial features and the context information queue, a relationship matrix is ​​obtained, wherein the relationship matrix is ​​used to represent the similarity relationship between the initial features and the context information queue;

[0110] According to the context information queue and relationship matrix, intermediate features are obtained;

[0111] According to the initial features and intermediate features, the target features are obtained.

[0112] Optionally, in the disclosed embodiment, the server first flattens the initial features F of a batch to obtain the flattened features in B represents the number of training data input in each batch. The preferred value selected in this embodiment is 12. Then, the server calculates the number of training data according to the flattened features. and context information queue The relationship matrix A is obtained, where the relationship matrix A is composed of relationship data a, and a is defined as follows:

[0113]

[0114] Among them, a zji Used to measure the characteristics of pixel z on the jth training data image and the i-th context feature information The more similar the features of the two images are, the better the zji The bigger.

[0115] Furthermore, the server performs feature enhancement based on the context information queue E and the relationship matrix A to obtain the intermediate feature F aug , the intermediate feature F aug The definition is as follows:

[0116]

[0117] The δ operation refers to adjusting the order of data dimensions to restore the original spatial dimensions (B, C, H, W), B represents the number of training data input in each batch, and the preferred value selected in this embodiment is 12, C represents the number of channels of the intermediate feature, H is the height of the intermediate feature, and W is the width of the intermediate feature.

[0118] Next, the server calculates the initial feature F and the intermediate feature F aug Perform feature aggregation to obtain the target feature F agg , target feature F agg The definition is as follows:

[0119] F agg =concat(concat(F aug ,F),F aug +F,F aug F)

[0120] Among them, concat() is a function that connects multiple strings.

[0121] In the disclosed embodiment, by obtaining a relationship matrix based on the initial features and the context information queue, and further obtaining intermediate features based on the context information queue and the relationship matrix, the initial features can be enhanced using the context information queue to improve the generalization of the image segmentation model for unknown scenes. By obtaining the target features based on the initial features and the intermediate features, feature aggregation can be achieved to make the target features more recognizable.

[0122] In some optional implementations, the target feature is input into a classifier to obtain target prediction data and a second loss function, including:

[0123] Input the target features into the classifier to obtain the target prediction data;

[0124] According to the target prediction data and the target label, the second loss function is obtained.

[0125] Optionally, in the embodiment of the present disclosure, the server sets the target feature F agg Input the classifier to obtain the target prediction data P (h,w) , according to the target prediction data P (h,w) and the target label y, we get the second loss function L seg , where the second loss function L seg By the cross entropy loss function L BCE and Dice loss function L Dice Composition, defined as follows:

[0126] L seg =L BCE (L (h,w) ,y)+L Dice (P (h,w) ,y)

[0127] In the disclosed embodiment, by inputting the target feature into the classifier to obtain the target prediction data, the image can be accurately segmented. By using the obtained second loss function for updating the parameters of the initial model, the initial model can be optimized and the image segmentation effect of the initial model can be improved.

[0128] In the present embodiment, a model determination device based on image segmentation is also provided, which is used to implement the above-mentioned embodiments and preferred implementation modes, and the descriptions that have been made will not be repeated. As used below, the term "module" can implement a combination of software and / or hardware of a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceivable.

[0129] This embodiment provides a model determination device based on image segmentation, such as Figure 3As shown, including:

[0130] An acquisition module 301 is used to acquire training data for image segmentation;

[0131] A first obtaining module 302 is used to input the training data into the initial model for feature extraction and feature alignment to obtain initial features and a first loss function, wherein the initial features are used to characterize pixel information of the training data image;

[0132] The second obtaining module 303 is used to input the initial features into the initial model to obtain a context information queue, wherein the context information queue contains current context information obtained from the initial features and historical context information obtained from the historical features;

[0133] The third obtaining module 304 is used to input the initial features and the context information queue into the initial model to obtain the target features;

[0134] A fourth obtaining module 305 is used to input the target feature into the classifier to obtain target prediction data and a second loss function, wherein the target prediction data is used to represent the final prediction probability that the training data belongs to the first data type and the second data type;

[0135] The fifth obtaining module 306 is used to adjust the model parameters of the initial model using the first loss function and the second loss function to obtain the target model.

[0136] In the disclosed embodiment, by acquiring training data for image segmentation; inputting the training data into the initial model for feature extraction and feature alignment, obtaining initial features and a first loss function, wherein the initial features are used to characterize the pixel information of the training data image; inputting the initial features into the initial model to obtain a context information queue, wherein the context information queue contains current context information obtained from the initial features and historical context information obtained from historical features; inputting the initial features and the context information queue into the initial model to obtain target features; inputting the target features into the classifier to obtain target prediction data and a second loss function, wherein the target prediction data is used to represent the final prediction probability that the training data belongs to the first data type and the second data type; adjusting the model parameters of the initial model using the first loss function and the second loss function to obtain the target model. Since the disclosed method extracts features from the training data and then performs feature alignment to obtain initial features, then obtains a context information queue based on the initial features, and further obtains the target features based on the initial features and the context information queue, the universality of the image segmentation model for real scenes and the generalization for unknown scenes are improved.

[0137] In some optional embodiments, the device further comprises:

[0138] The sixth obtaining module is used to pre-process the label data to obtain the target label;

[0139] The seventh obtaining module is used to perform data preprocessing and data enhancement on the training data to obtain processed training data;

[0140] The extraction module is used to input the processed training data into the initial model for feature extraction.

[0141] In some optional implementations, the first obtaining module 302 includes:

[0142] The first obtaining unit is used to obtain a label feature according to a target label;

[0143] The second obtaining unit is used to input the training data into the feature extraction network and feature alignment network of the initial model to obtain initial features, wherein the initial model includes multiple first processing units and second processing units, the first processing unit includes a feature extraction network and a corresponding feature alignment network, and the second processing unit includes a feature extraction network.

[0144] An extraction unit, used to extract the first features output by each first processing unit;

[0145] A third obtaining unit is used to obtain a first loss function according to the first feature and the label feature;

[0146] In some optional implementations, the second obtaining module 303 includes:

[0147] a fourth obtaining unit, configured to sequentially pass the initial features through the first network, the second network, and the third network of the initial model to obtain initial prediction data, wherein the initial prediction data is used to indicate an initial prediction probability that the training data belongs to the first data type and the second data type;

[0148] A fifth obtaining unit, used for obtaining current context information according to the initial features and the initial prediction data;

[0149] The sixth obtaining unit is used to obtain a context information queue according to the current context information and the historical context information.

[0150] In some optional implementations, the third obtaining module 304 includes:

[0151] A seventh obtaining unit is used to obtain a relationship matrix according to the initial features and the context information queue, wherein the relationship matrix is ​​used to represent the similarity relationship between the initial features and the context information queue;

[0152] An eighth obtaining unit is used to obtain intermediate features according to the context information queue and the relationship matrix;

[0153] The ninth obtaining unit is used to obtain the target feature according to the initial feature and the intermediate feature.

[0154] In some optional implementations, the fourth obtaining module 305 includes:

[0155] The tenth obtaining unit is used to input the target feature into the classifier to obtain the target prediction data;

[0156] The eleventh obtaining unit is used to obtain a second loss function according to the target prediction data and the target label.

[0157] The further functional description of each of the above modules and units is the same as that of the above corresponding embodiments and will not be repeated here.

[0158] The image segmentation-based model determination device in this embodiment is presented in the form of a functional unit, where the unit refers to an ASIC (Application Specific Integrated Circuit) circuit, a processor and memory that executes one or more software or fixed programs, and / or other devices that can provide the above-mentioned functions.

[0159] The present disclosure also provides a computer device having the above Figure 3 The model determination device based on image segmentation is shown.

[0160] See also Figure 4 , Figure 4 is a schematic diagram of a computer device provided by an optional embodiment of the present disclosure, such as Figure 4 As shown, the computer device includes: one or more processors 10, a memory 20, and interfaces for connecting various components, including high-speed interfaces and low-speed interfaces. Various components are connected to each other using different buses for communication, and can be installed on a common mainboard or installed in other ways as needed. The processor can process the instructions executed in the computer device, including instructions stored in or on the memory to display the graphical information of the GUI on an external input / output device (such as, a display device coupled to the interface). In some optional embodiments, if necessary, multiple processors and / or multiple buses can be used together with multiple memories and multiple memories. Similarly, multiple computer devices can be connected, and each device provides some necessary operations (for example, as a server array, a group of blade servers, or a multi-processor system). Figure 4 A processor 10 is taken as an example.

[0161] The processor 10 may be a central processing unit, a network processor or a combination thereof. The processor 10 may further include a hardware chip. The hardware chip may be a dedicated integrated circuit, a programmable logic device or a combination thereof. The programmable logic device may be a complex programmable logic device, a field programmable gate array, a general purpose array logic or any combination thereof.

[0162] The memory 20 stores instructions executable by at least one processor 10, so that the at least one processor 10 executes the method shown in the above embodiment.

[0163] The memory 20 may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created according to the use of the computer device, etc. In addition, the memory 20 may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some optional embodiments, the memory 20 may optionally include a memory remotely arranged relative to the processor 10, and these remote memories may be connected to the computer device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0164] The memory 20 may include a volatile memory, such as a random access memory; the memory may also include a non-volatile memory, such as a flash memory, a hard disk or a solid state drive; the memory 20 may also include a combination of the above types of memory.

[0165] The computer device further comprises a communication interface 30 for the computer device to communicate with other devices or a communication network.

[0166] The embodiments of the present disclosure also provide a computer-readable storage medium. The above-mentioned method according to the embodiments of the present disclosure can be implemented in hardware, firmware, or can be implemented as a computer code that can be recorded in a storage medium, or can be implemented as a computer code that is originally stored in a remote storage medium or a non-temporary machine-readable storage medium and will be stored in a local storage medium and downloaded through a network, so that the method described herein can be stored in such software processing on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. Among them, the storage medium can be a magnetic disk, an optical disk, a read-only storage memory, a random access memory, a flash memory, a hard disk or a solid-state drive, etc.; further, the storage medium can also include a combination of the above-mentioned types of memory. It can be understood that a computer, a processor, a microprocessor controller, or programmable hardware includes a storage component that can store or receive software or computer code. When the software or computer code is accessed and executed by a computer, a processor, or hardware, the method shown in the above embodiment is implemented.

[0167] A part of the present disclosure may be applied as a computer program product, such as a computer program instruction, which, when executed by a computer, can call or provide the method and / or technical solution according to the present disclosure through the operation of the computer. Those skilled in the art should understand that the existence of computer program instructions in computer-readable media includes, but is not limited to, source files, executable files, installation package files, etc., and accordingly, the way in which computer program instructions are executed by a computer includes, but is not limited to: the computer directly executes the instruction, or the computer compiles the instruction and then executes the corresponding compiled program, or the computer reads and executes the instruction, or the computer reads and installs the instruction and then executes the corresponding installed program. Here, the computer-readable medium can be any available computer-readable storage medium or communication medium accessible to the computer.

[0168] Although the embodiments of the present disclosure have been described in conjunction with the accompanying drawings, those skilled in the art may make various modifications and variations without departing from the spirit and scope of the present disclosure, and such modifications and variations are all within the scope defined by the appended claims.

Claims

1. A model determination method based on image segmentation, characterized in that: The method comprises: Get training data for image segmentation; Inputting the training data into an initial model for feature extraction and feature alignment to obtain initial features and a first loss function, wherein the initial features are used to characterize pixel information of the training data image; Inputting the initial features into the initial model to obtain a context information queue, wherein the context information queue includes current context information obtained from the initial features and historical context information obtained from historical features; Inputting the initial features and the context information queue into the initial model to obtain target features; Inputting the target feature into a classifier to obtain target prediction data and a second loss function, wherein the target prediction data is used to represent the final prediction probability that the training data belongs to the first data type and the second data type; The first loss function and the second loss function are used to adjust the model parameters of the initial model to obtain a target model.

2. The method according to claim 1, characterized in that The training data carries label data, wherein the label data is used to characterize that the training data is the first data type or the second data type. Before inputting the training data into the initial model for feature extraction and feature alignment to obtain initial features and a first loss function, the method further includes: Performing data preprocessing on the label data to obtain a target label; Performing data preprocessing and data enhancement on the training data to obtain processed training data; The processed training data is input into the initial model for feature extraction.

3. The method according to claim 2, characterized in that The step of inputting the training data into the initial model for feature extraction and feature alignment to obtain initial features and a first loss function includes: According to the target label, obtain label features; Inputting the training data into the feature extraction network and the feature alignment network of the initial model to obtain the initial features, wherein the initial model includes a plurality of first processing units and a second processing unit, the first processing unit includes the feature extraction network and the corresponding feature alignment network, and the second processing unit includes the feature extraction network; Extracting first features output by each of the first processing units; The first loss function is obtained according to the first feature and the label feature.

4. The method according to claim 2, characterized in that: The step of inputting the initial features into the initial model to obtain a context information queue includes: Passing the initial features through the first network, the second network, and the third network of the initial model in sequence to obtain initial prediction data, wherein the initial prediction data is used to represent the initial prediction probability that the training data belongs to the first data type and the second data type; Obtaining the current context information according to the initial features and the initial prediction data; The context information queue is obtained according to the current context information and the historical context information.

5. The method according to claim 2, characterized in that: The step of inputting the initial features and the context information queue into the initial model to obtain target features includes: According to the initial features and the context information queue, a relationship matrix is ​​obtained, wherein the relationship matrix is ​​used to characterize the similarity relationship between the initial features and the context information queue; Obtaining intermediate features according to the context information queue and the relationship matrix; The target feature is obtained according to the initial feature and the intermediate feature.

6. The method according to claim 2, characterized in that The step of inputting the target feature into a classifier to obtain target prediction data and a second loss function comprises: Inputting the target feature into the classifier to obtain the target prediction data; The second loss function is obtained according to the target prediction data and the target label.

7. A model determination device based on image segmentation, characterized in that: The device comprises: An acquisition module, used to acquire training data for image segmentation; A first obtaining module is used to input the training data into an initial model for feature extraction and feature alignment to obtain initial features and a first loss function, wherein the initial features are used to characterize pixel information of the training data image; A second obtaining module is used to input the initial features into the initial model to obtain a context information queue, wherein the context information queue contains current context information obtained from the initial features and historical context information obtained from historical features; A third obtaining module is used to input the initial features and the context information queue into the initial model to obtain target features; A fourth obtaining module is used to input the target feature into a classifier to obtain target prediction data and a second loss function, wherein the target prediction data is used to represent the final prediction probability that the training data belongs to the first data type and the second data type; The fifth obtaining module is used to adjust the model parameters of the initial model using the first loss function and the second loss function to obtain the target model.

8. A computer device, characterized in that: include: A memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the model determination method based on image segmentation according to any one of claims 1 to 6 by executing the computer instructions.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a computer to execute the model determination method based on image segmentation according to any one of claims 1 to 6.

10. A computer program product, characterized in that The method comprises computer instructions for causing a computer to execute the model determination method based on image segmentation according to any one of claims 1 to 6.

Images