Optimal transmission-based boundary correction method and system, medium and device

Abstract

The application discloses a boundary correction method based on optimal transmission, which comprises the following steps: a general semantic segmentation step: performing feature extraction, feature enhancement and pixel-by-pixel classification operation based on a general semantic segmentation network, and outputting a loss value of semantic segmentation supervision; a boundary prediction step: introducing a boundary branch after the general semantic segmentation network, calculating an initial position of a predicted boundary based on the boundary branch, and performing supervision through a semantic boundary loss; and a boundary correction step: normalizing each pixel prediction result of the boundary branch, calculating a semantic boundary positioning loss through optimal transmission, correcting the semantic boundary loss through continuous training, and achieving the optimal accuracy of boundary region segmentation. The method models the boundary prediction as an optimal transmission problem, supervises the result of the boundary prediction by calculating the minimum transmission cost of the predicted boundary to the real boundary, and effectively improves the accuracy of the boundary segmentation.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of computer vision, and in particular, to a method for improving semantic segmentation performance by improving semantic boundary prediction accuracy and a system thereof. BACKGROUND

[0002] Currently, the semantic boundary region segmentation accuracy is not high, and accurate boundary segmentation is an important method to improve the segmentation performance of small objects. Therefore, the current semantic segmentation research focuses on improving the accuracy of semantic boundaries to improve the performance of semantic segmentation. The common method is to introduce a boundary branch and supervise the predicted boundary through binary cross entropy (BCE) loss:

[0003] Existing solution one: Gated-SCNN

[0004] As shown in Figure 1 , Gated-SCNN proposes a double-flow network as shown in Figure 1 , the upper branch is used to extract semantic features, and the lower branch is used to extract boundary features. In the inference process, the semantic features and boundary features are fused for semantic segmentation prediction, and the real semantic boundary is used to supervise the predicted boundary through binary cross entropy loss:

[0005] Existing solution two: DecoupleSegNets

[0006] As shown in Figure 2 , DecoupleSegNets proposes to decompose the image features extracted by the network into object internal features and object boundary features for separate supervision, and then combine the features for the final semantic segmentation. Similarly, this method also uses binary cross entropy to supervise the predicted semantic boundary:

[0007] The above methods all use binary cross entropy loss to supervise the predicted semantic boundary, and the loss is defined as follows:

[0008]

[0009] This loss considers the boundary prediction problem as a pixel-by-pixel classification problem, and evaluates the boundary segmentation result by calculating the pixels that are misclassified in the predicted boundary. However, this loss does not consider the spatial distance between the predicted boundary and the real boundary, and the supervision is not accurate enough. An example is given to illustrate the above problem, as shown in Figure 3 :

[0010] Wherein bpred1, bpred2 represent the results of two predicted boundaries, and bgt represents the true semantic boundary. From the spatial position, the predicted result bpred1 is better than bpred2 because bpred1 is closer to the true semantic boundary bgt. However, for the binary cross-entropy loss, bpred1 and bpred2 are all wrong in the local area boundary, so it is difficult to distinguish the advantages and disadvantages of the two boundary prediction results, which shows that using the binary cross-entropy loss to supervise the boundary prediction result is not accurate and comprehensive.

[0011] Therefore, it is urgent to propose a new boundary correction method based on optimal transmission, which can solve the defects in the prior art that the binary cross-entropy loss function for the boundary does not consider the distance between the predicted boundary and the true boundary, the boundary supervision is not accurate enough, and the boundary segmentation accuracy is not high, which affects the semantic segmentation performance. SUMMARY

[0012] In order to solve the defects of the prior art, the embodiment of the present application provides a boundary correction method based on optimal transmission to solve the problem that the binary cross-entropy loss is not accurate for semantic boundary supervision, and proposes a boundary correction method based on optimal transmission.

[0013] In the first aspect, the embodiment of the present application provides a boundary correction method based on optimal transmission, which comprises:

[0014] The general semantic segmentation step: performing feature extraction, feature enhancement and pixel-by-pixel classification operation based on the general semantic segmentation network, and outputting the loss value of semantic segmentation supervision;

[0015] The boundary prediction step: introducing a boundary branch after the general semantic segmentation network, calculating the initial position of the predicted boundary based on the boundary branch, and supervising by the semantic boundary loss;

[0016] The boundary correction step: normalizing each pixel prediction result of the boundary branch, calculating the semantic boundary positioning loss by optimal transmission, correcting the semantic boundary loss by continuous training, and achieving the optimal accuracy of the boundary region segmentation.

[0017] Preferably, in the embodiment of the present application, the general semantic segmentation step further comprises: performing semantic segmentation supervision based on the loss function L ce of the predicted semantic result L pred and the true semantic result L gt , and outputting the loss value of semantic segmentation supervision.

[0018] Preferably, in the embodiments of the present application, the boundary prediction step comprises: performing boundary prediction through a boundary prediction branch after the general semantic segmentation network, and the boundary prediction branch is composed of a convolution layer and a Sigmoid activation layer, and the prediction result is supervised by a binary cross-entropy loss.

[0019] Preferably, in the embodiments of the present application, the boundary correction step comprises:

[0020] The result calculation step of the predicted boundary: calculating b pred The normalized result of (i,j) Wherein, b pred (i,j) is the result of the predicted boundary.

[0021] The target distribution calculation step of the boundary region: calculating b target The normalized result of (i,j) Wherein, b target (i,j) is the result of the target boundary.

[0022] The transmission cost calculation step: calculating the Euclidean distance of each position pixel p s on the source distribution to the target distribution p t , as the transmission cost C st =(p s -p t ) 2 , to complete the construction of the transmission cost matrix.

[0023] The boundary supervision step: calculating the target value of the boundary optimal transport problem Wherein, π is the optimal transport scheme, C is the transmission cost matrix, and U is the set of optimal transport schemes, representing the set of transmission matrices from the source distribution to the target distribution.

[0024] The constraint condition of the boundary optimal transport is: Wherein, n s represents the existence of discrete source distribution, and n t represents the existence of discrete target distribution.

[0025] The minimum transmission cost supervises the predicted boundary, and the semantic boundary correction loss function Lopt is defined as:

[0026] Wherein, bpred is the result of the predicted boundary, bgt represents the real semantic boundary, and α and β represent the transition probability matrix of the target distribution and the source distribution.

[0027] Preferably, in the embodiments of the present application, the boundary correction step further comprises:

[0028] The network loss function is:

[0029]

[0030] wherein, L ce is the loss function of semantic segmentation supervision, L opt is the semantic boundary correction loss function, L BCE is the binary cross-entropy loss, representing the semantic boundary positioning loss function, y pred is the semantic segmentation prediction result, y gt is the real semantic segmentation result, λ1 and λ2 are weight parameters.

[0031] Preferably, in the embodiment of the present application, the above boundary supervision step further comprises:

[0032] The objective of the boundary optimal transport problem is:

[0033] wherein, α represents the transition probability matrix of the target distribution, β represents the transition probability matrix of the source distribution, R n represents the transition matrix.

[0034] Preferably, in the embodiment of the present application, the above boundary branch comprises: a convolution layer, a regularization layer and an activation layer stacked together, the activation function of the activation layer is Sigmoid, the result of each pixel prediction of the boundary branch is normalized to the range of (0, 1), and the value of each pixel represents the probability of belonging to the boundary.

[0035] In the second aspect, the embodiment of the present application provides a boundary correction system based on optimal transport, which adopts the boundary correction method based on optimal transport as described above, and the system comprises:

[0036] A general semantic segmentation module: performing feature extraction, feature enhancement and pixel-by-pixel classification operation based on a general semantic segmentation network, and outputting the loss value of semantic segmentation supervision;

[0037] A boundary prediction module: introducing a boundary branch after the general semantic segmentation network, calculating the initial position of the predicted boundary based on the boundary branch, and supervising through the semantic boundary loss;

[0038] A boundary correction module: normalizing the result of each pixel prediction of the boundary branch, calculating the semantic boundary positioning loss through optimal transport, correcting the semantic boundary loss through continuous training, and achieving the optimal accuracy of the boundary region segmentation.

[0039] In the third aspect, the embodiment of the present application provides a computer readable storage medium, which stores a computer program, and the program is characterized in that when the program is executed by a processor, the steps of the boundary correction method based on optimal transport as described above are realized.

[0040] In a fourth aspect, an electronic device is provided, which comprises a memory, a processor, and a computer program stored in the memory and capable of running on the processor, and wherein the processor implements the steps of the boundary correction method based on optimal transport as described above when executing the program.

[0041] Compared with the related prior art, the present application has the following outstanding beneficial effects:

[0042] 1. The present application models the semantic boundary prediction as an optimal transport problem, calculates the minimum transport cost of the predicted boundary to the real boundary to more accurately supervise the semantic boundary, improves the accuracy of the boundary region segmentation, and further improves the performance of the semantic segmentation.

[0043] 2. The present application proposes a semantic segmentation method based on boundary correction of optimal transport, which only provides additional supervision for the semantic boundary through optimal transport in the training stage, and the boundary branch can be removed in the inference stage, so as to not increase the calculation amount and parameter amount in the inference stage. BRIEF DESCRIPTION OF DRAWINGS

[0044] The accompanying drawings, which are included to provide a further understanding of the application and constitute a part of this application, illustrate embodiments of the application and together with the description serve to explain the application. In the drawings:

[0045] Figure 1 It is a schematic diagram of the prior art Gated-SCNN method;

[0046] Figure 2 It is a schematic diagram of the prior art DecoupleSegNets method;

[0047] Figure 3 It is a schematic diagram of the prior art supervision precision;

[0048] Figure 4 It is a schematic diagram of the boundary correction method based on optimal transport of the present application;

[0049] Figure 5 It is a schematic diagram of the boundary correction network of the specific embodiment of the present application;

[0050] Figure 6 It is a schematic diagram of the segmentation result of the specific embodiment of the present application;

[0051] Figure 7 It is a schematic diagram of the boundary correction system based on optimal transport of the present application;

[0052] Figure 8 It is a schematic diagram of the computer hardware of the present application.

[0053] In the above drawings:

[0054] 10 general semantic segmentation module 20 boundary prediction module

[0055] 30 boundary correction module. DETAILED DESCRIPTION

[0056] In order to make the objectives, technical solutions and advantages of the present application clearer, the present application is described and explained below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and should not be used to limit the present application. Based on the embodiments provided by the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort belong to the scope of protection of the present application.

[0057] Obviously, the accompanying drawings in the following description are only some examples or embodiments of the present application, and for those of ordinary skill in the art, the present application can be applied to other similar scenarios without creative effort based on these drawings. In addition, it can be understood that although the efforts made in this development process can be complex and lengthy, for those of ordinary skill in the art related to the content disclosed in the present application, some design, manufacture or production changes based on the technical content disclosed in the present application are only routine technical means and should not be understood as insufficient disclosure of the content disclosed in the present application.

[0058] The details of one or more embodiments of the present application are presented in the following drawings and description to make other features, objectives and advantages of the present application more apparent.

[0059] In the present application, the phrase "embodiment" means that the specific features, structures or characteristics described in conjunction with the embodiment can be included in at least one embodiment of the present application. The appearance of this phrase at various places in the specification does not necessarily mean the same embodiment, nor is it an independent or alternative embodiment to other embodiments. It is explicitly and implicitly understood by those of ordinary skill in the art that the embodiments described in the present application can be combined with other embodiments without conflict.

[0060] Unless otherwise defined, technical terms and scientific terms used in the present application shall have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terms "a", "an", "one", "this", and similar referents in the context of describing the application are to be construed to be inclusive of both the singular and the plural, unless otherwise indicated. The terms "comprising", "comprises" and "comprised of" as well as "containing", "containing" and "contains" are inclusive and do not exclude other steps, elements or ingredients. The terms "connected" and "coupled" are not restricted to physical or mechanical connections or couplings. The term "multiple" means two or more. The term "and / or" includes all possible combinations of the listed terms. The term "first", "second", "third", etc. are used to distinguish elements with similar or identical properties, but do not necessarily represent a specific order.

[0061] The present application aims to improve the accuracy of semantic segmentation, and finds that the existing binary cross-entropy loss function for the boundary does not consider the distance between the predicted boundary and the real boundary, and the boundary supervision is not accurate enough, resulting in low boundary segmentation accuracy and affecting the performance of semantic segmentation. To solve this defect, the inventors model the boundary prediction as an optimal transport problem, and supervise the boundary prediction result by calculating the minimum transport cost from the predicted boundary to the real boundary, effectively improving the accuracy of boundary segmentation.

[0062] The present application models the semantic boundary prediction as an optimal transport problem, calculates the minimum transport cost from the predicted boundary to the real boundary to more accurately supervise the semantic boundary, improves the accuracy of boundary region segmentation, and further improves the performance of semantic segmentation.

[0063] The present application provides a semantic segmentation method based on optimal transport boundary correction. This method only provides additional supervision for semantic boundaries through optimal transport in the training stage, and the boundary branch can be removed in the inference stage, thereby not increasing the calculation amount and parameter amount in the inference stage.

[0064] As shown in Figure 4 The present application provides a boundary correction method based on optimal transport, which includes:

[0065] The general semantic segmentation step S10: based on the general semantic segmentation network, feature extraction, feature enhancement and pixel-by-pixel classification operation are carried out, and the loss value of semantic segmentation supervision is outputted;

[0066] The boundary prediction step S20: a boundary branch is introduced after the general semantic segmentation network, the initial position of the predicted boundary is calculated based on the boundary branch, and the semantic boundary loss is supervised;

[0067] The boundary correction step S30: the prediction result of each pixel of the boundary branch is normalized, the semantic boundary positioning loss is calculated through the optimal transmission, the semantic boundary loss is corrected through continuous training, and the optimal accuracy of the boundary region segmentation is achieved.

[0068] In the embodiment of the present application, the general semantic segmentation step S10 further comprises: based on the loss function L ce The predicted semantic result L pred And the real semantic result L gt , semantic segmentation supervision is carried out, and the loss value of semantic segmentation supervision is outputted.

[0069] Preferably, in the embodiment of the present application, the boundary prediction step S20 comprises: a boundary prediction branch is introduced after the general semantic segmentation network, which is composed of a convolution layer and a Sigmoid activation layer, and the prediction result is supervised by binary cross-entropy loss.

[0070] Preferably, in the embodiment of the present application, the boundary correction step S30 comprises:

[0071] The result calculation step of the predicted boundary: the normalized result of b pred (i,j) is calculated Wherein, b pred (i,j) is the result of the predicted boundary.

[0072] The target distribution calculation step of the boundary region: the normalized result of b target (i,j) is calculated Wherein, b target (i,j) is the result of the target boundary.

[0073] The transmission cost calculation step: the Euclidean distance of each position pixel p s on the source distribution to each position pixel p t on the target distribution is taken as the transmission cost, C st =(p s -p t ) 2 , and the construction of the transmission cost matrix is completed.

[0074] Boundary supervision step: calculating the target value of the boundary optimal transport problem Wherein, π is the optimal transport scheme, C is the transport cost matrix, U is the set of optimal transport schemes, represents the set of transport matrices from the source distribution to the target distribution.

[0075] The constraint condition of the boundary optimal transport is: Wherein, n s represents the existence of discrete source distribution, n t represents the existence of discrete target distribution.

[0076] The minimum transport cost supervises the predicted boundary, and the semantic boundary correction loss function Lopt is defined as:

[0077] Wherein, bpred is the result of the predicted boundary, bgt represents the real semantic boundary, and α and β represent the transition probability matrix of the target distribution and the source distribution.

[0078] Preferably, in the embodiment of the present application, the above boundary correction step S30 further comprises:

[0079] The network loss function is:

[0080]

[0081] Wherein, L ce is the loss function for semantic segmentation supervision, L opt is the semantic boundary correction loss function, L BCE is the binary cross entropy loss, represents the semantic boundary positioning loss function, y pred is the semantic segmentation prediction result, y gt is the real semantic segmentation result, and λ1 and λ2 are weight parameters.

[0082] Preferably, in the embodiment of the present application, the above boundary supervision step further comprises:

[0083] The target of the boundary optimal transport problem is:

[0084] Wherein, α represents the transition probability matrix of the target distribution, β represents the transition probability matrix of the source distribution, and R n represents the transition matrix.

[0085] Preferably, in the embodiment of the present application, the above boundary branch comprises: a convolution layer, a regularization layer and an activation layer stacked, the activation function of the activation layer is Sigmoid, the result predicted by each pixel of the boundary branch is normalized to the range of (0, 1), and the value of each pixel represents the probability of belonging to the boundary.

[0086] The application is described in detail below with reference to the drawings.

[0087] 1. Network structure

[0088] As shown in the drawings, the overall structure of the boundary correction network based on optimal transmission proposed by the application is shown in the following figure, including a general semantic segmentation module and a boundary branch. The general semantic segmentation network generally includes a backbone network for feature extraction, a context module for feature enhancement, and a classifier composed of several convolution layers for pixel-by-pixel classification. Figure 5

[0089] In order to predict semantic boundaries and perform supervision, the application additionally introduces a boundary branch after the backbone network for boundary prediction, which is stacked by convolution layers, regularization (BatchNorm) layers, and activation layers. The last layer activation function is Sigmoid, which normalizes the result of each pixel prediction of the boundary branch to the range of (0, 1), and the value of each pixel represents its probability of belonging to the boundary. As can be seen from the overall network structure, the boundary correction network proposed by the application is a general plug-and-play module suitable for various mainstream semantic segmentation network structures. Since the application does not explicitly fuse the features extracted by the boundary branch into the semantic branch, the boundary branch can be removed during the inference stage of the network, so the branch does not increase the computational complexity of the inference stage.

[0090] 2. Modeling of boundary optimal problem

[0091] 1) The goal of the boundary optimal transmission problem is:

[0092]

[0093] Where π represents the optimal transmission scheme, C represents the transmission cost matrix, and π represents the transmission scheme. The dual problem of this problem is:

[0094]

[0095] 2) Model the predicted boundary and the real boundary as source distribution and target distribution: first, for the result of the predicted boundary, the application directly normalizes it:

[0096]

[0097] Since the original segmentation dataset only includes semantic segmentation labels, in order to construct the target distribution, the semantic boundary label needs to be extracted from it first. Therefore, for each class in the dataset, the application first calculates the minimum distance from other classes to the pixels of the class, and the distance less than 2 pixels is the semantic boundary. Further normalization of the boundary region target distribution:

[0098] ​

[0099] 3) In order to reflect the spatial distance between the predicted boundary and the real boundary, the present application first calculates the Euclidean distance from each position pixel ps on the source distribution to each position pixel pt on the target distribution as the cost of transmission, that is, Cst=(ps-pt)2, and the construction of the cost matrix is completed in this way. Then the minimum cost after solving the optimal transmission reflects the spatial distance from the predicted boundary to the real boundary, and based on this, additional supervision is provided for the predicted boundary.

[0100]

[0101] 4) The constraint condition of the boundary optimal transmission is:

[0102]

[0103] The first constraint condition: the sum of the source distribution transition probability matrix is the source distribution probability value;

[0104] The second constraint condition: the sum of the target distribution transition probability matrix is the target distribution probability value;

[0105] The third constraint condition: the transition probability from any source distribution to target distribution is not less than 0.

[0106] 5) The solution of the boundary optimal transmission problem: from the above modeling, it can be seen that the boundary optimal transmission is obviously a linear programming problem, and the solution complexity of the linear programming problem is often high. The present application introduces the constraint of entropy regularization, and uses the Sinkhorn-Knopp algorithm to iteratively solve the above problem. And the predicted boundary is supervised based on the minimum transmission cost, which is defined as follows:

[0107]

[0108] In general, the above network loss function is,

[0109]

[0110] Wherein, Lce is the loss function for semantic segmentation supervision, LBCE and Lopt are the loss functions for boundary prediction, LBCE is the semantic boundary positioning loss, and Lopt is the semantic boundary correction loss. λ1 and λ2 are weight parameters, and in the specific embodiments of the present application, λ1 and λ2 are set to 15 and 0.1 respectively, but the present application is not limited thereto, and λ1 and λ2 can also be set to other numerical ranges.

[0111]

[0112]

[0113] Wherein, WH represents a characteristic size, K represents a number of semantic categories, and sigma represents a sigmoid function. A real semantic category, p ik A predicted semantic segmentation probability, b gti A real semantic boundary, b predi A predicted semantic boundary.

[0114] The application improves the accuracy of boundary region segmentation and the performance of semantic segmentation without increasing the amount of calculation and the number of parameters in the reasoning stage.

[0115] 1. The application effectively improves the accuracy of semantic segmentation.

[0116] The application first performs an ablation experiment on the Camvid dataset, and it can be seen that the introduction of the BCE supervision increases the segmentation accuracy mIoU from 70.64% to 71.49%, and the further introduction of the optimal transport loss increases the segmentation accuracy to 72.37%, which shows that the application effectively improves the accuracy of semantic segmentation. In addition, the application does not fuse the boundary features and semantic features, so it does not increase the complexity of the reasoning stage.

[0117]

[0118] 2. The application effectively improves the accuracy of semantic boundary segmentation.

[0119] The application calculates the accuracy of the boundary F-boundary under different thresholds, and the experimental results prove that the application effectively improves the accuracy of boundary region segmentation.

[0120]

[0121] 3. The application effectively improves the qualitative segmentation result.

[0122] As Figure 6 shown, from left to right are the original image, the baseline method, the application, and the real label, and it can be seen that the application forms more accurate semantic boundaries, thereby improving the accuracy of semantic segmentation.

[0123] In a second aspect, the application embodiments provide a boundary correction system based on optimal transport, which adopts the boundary correction method based on optimal transport as described above, as Figure 7 shown, the system comprises:

[0124] A general semantic segmentation module 10: performing feature extraction, feature enhancement, and pixel-by-pixel classification operation based on a general semantic segmentation network, and outputting a loss value of semantic segmentation supervision;

[0125] The boundary prediction module 20: a boundary branch is introduced after the general semantic segmentation network, the initial position of the predicted boundary is calculated based on the boundary branch, and the semantic boundary loss is supervised;

[0126] The boundary correction module 30: the prediction result of each pixel of the boundary branch is normalized, the semantic boundary positioning loss is calculated by optimal transport, the semantic boundary loss is corrected by continuous training, and the optimal accuracy of the boundary region segmentation is achieved.

[0127] In a third aspect, an embodiment of the present application provides a computer readable storage medium, which stores a computer program, and the program is executed by a processor to implement the steps of the boundary correction method based on optimal transport.

[0128] In a fourth aspect, an embodiment of the present application provides an electronic device, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor implements the steps of the boundary correction method based on optimal transport when executing the program.

[0129] In addition, the boundary correction method based on optimal transport of the embodiments of the present application is described in combination with Figure 1 The boundary correction method based on optimal transport of the embodiments of the present application described above can be implemented by a computer device. Figure 8 The hardware structure of the computer device according to the embodiments of the present application is shown in the figure.

[0130] The computer device can include a processor 81 and a memory 82 storing computer program instructions.

[0131] Specifically, the processor 81 described above can include a central processing unit (CPU), or an application specific integrated circuit (ASIC), or can be configured as one or more integrated circuits implementing the embodiments of the present application.

[0132] The memory 82 can include a mass storage for data or instructions. By way of example, and without limitation, the memory 82 can include a hard disk drive (HDD), a floppy disk drive, a solid-state drive (SSD), a flash drive, a compact disc (CD) or DVD, a tape, a magnetic or optical or other recording medium, a universal serial bus (USB) drive, or two or more of these or other mass storage devices. The memory 82 can be removable and / or non-removable (or fixed). The memory 82 can be internal or external to the data processing device. In particular embodiments, the memory 82 is a non-volatile memory. In particular embodiments, the memory 82 includes read-only memory (ROM) and random access memory (RAM). The ROM can be mask programmed ROM, programmable ROM (PROM), erasable PROM (EPROM), electrically erasable PROM (EEPROM), electrically alterable ROM (EAROM), or FLASH, or two or more of these or other types of ROM, as appropriate. The RAM can be static random-access memory (SRAM) or dynamic random-access memory (DRAM), which can be Fast Page Mode DRAM (FPM DRAM), Extended Data Output DRAM (EDO DRAM), synchronous dynamic random-access memory (SDRAM), or the like, as appropriate.

[0133] The memory 82 can be used to store or buffer various data files needed for processing and / or communication, and possible computer program instructions executed by the processor 81.

[0134] The processor 81 reads and executes the computer program instructions stored in the memory 82 to implement any of the above-described embodiments of the optimal transmission based boundary correction method.

[0135] In some embodiments, the computer device can further include a communication interface 83 and a bus 80. In which, as shown, the processor 81, the memory 82, the communication interface 83 are connected through the bus 80 and complete the communication between each other. Figure 8

[0136] The communication interface 83 is used to realize the communication between the modules, devices, units and / or devices in the embodiments of the present application. The communication interface 83 can also realize the data communication between other components, such as: external devices, image / data acquisition devices, databases, external storage and image / data processing workstations, etc.

[0137] ​Bus 80 includes hardware, software, or both, to couple components of the computer device to each other and to couple components to other components in the environment. Bus 80 includes, but is not limited to, at least one of the following: a data bus, an address bus, a control bus, an expansion bus, a local bus, etc. By way of example and not limitation, bus 80 can include an Accelerated Graphics Port (AGP) or other graphics bus, an Extended Industry Standard Architecture (EISA) bus, a Front Side Bus (FSB), a Hyper Transport (HT) interconnect, an Industry Standard Architecture (ISA) bus, an InfiniBand interconnect, a Low Pin Count (LPC) bus, a memory bus, a Micro Channel Architecture (MCA) bus, a Peripheral Component Interconnect (PCI) bus, a PCI-Express (PCI-X) bus, a Serial Advanced Technology Attachment (SATA) bus, a Video Electronics Standards Association Local Bus (VLB) bus, or another suitable bus or combination of two or more of these. Where appropriate, bus 80 can include one or more buses. Although the present embodiments describe and show a particular bus, the present application contemplates any suitable bus or interconnect.

[0138] Compared with the prior art, the method of the present application models the semantic boundary prediction as an optimal transmission problem, calculates the minimum transmission cost of the predicted boundary to the real boundary to more accurately supervise the semantic boundary, improves the accuracy of the boundary region segmentation, and further improves the performance of the semantic segmentation; the present application proposes a semantic segmentation method based on optimal transmission boundary correction, which only provides additional supervision for the semantic boundary through optimal transmission in the training stage, and the boundary branch can be removed in the inference stage, so as to not increase the calculation amount and parameter amount in the inference stage.

[0139] Any combination of the technical features in the above-described embodiments can be made, and for the sake of brevity, not all possible combinations are described, however, as long as there is no conflict, any combination of the technical features should be considered within the scope of the present disclosure.

[0140] The above-described embodiments only express several implementation manners of the present application, and the description is relatively specific and detailed, but it should not be understood as a limitation on the patent scope of the present application. It should be pointed out that, for ordinary skilled persons in the art, several modifications and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the patent protection scope of the present application should be subject to the appended claims.

Claims

1. A method for boundary correction based on optimal transport, characterized in that, The method comprises: a general semantic segmentation step: performing feature extraction, feature enhancement and pixel-by-pixel classification operation based on a general semantic segmentation network, and outputting a loss value of semantic segmentation supervision; a boundary prediction step: introducing a boundary branch after the general semantic segmentation network, calculating an initial position of a predicted boundary based on the boundary branch, and performing supervision through a semantic boundary loss; Boundary correction steps: Normalize the prediction results of each pixel in the boundary branch, calculate the semantic boundary localization loss through optimal transmission, and correct the semantic boundary loss through continuous training to achieve the optimal accuracy of boundary region segmentation; where the objective function of optimal transmission is... The dual function of the objective function is The constraint condition for optimal transmission at the boundary is: , To predict boundary results The normalization result, , For the target boundary result The normalization result of , π represents the optimal transmission scheme, U is the set of optimal transmission schemes, and C is the transmission cost matrix. st =(p s -p t ) 2 p s p represents the location pixels of the source distribution. t Let be the location pixels of the target distribution, α represent the transition probability matrix of the target distribution, and β represent the transition probability matrix of the source distribution. Represents the transition matrix. This represents the existence of discrete source distributions. This represents the existence of discrete target distributions; the overall loss function is... , It is the loss function used for semantic segmentation supervision. It is the semantic boundary correction loss function. , It is the binary cross-entropy loss function representing the semantic boundary localization loss. The semantic segmentation prediction results This is the actual semantic segmentation result. and For weight parameters, For the true semantic boundary.

2. The method of claim 1, wherein the optimal transmission based boundary correction method further comprises: The general semantic segmentation step further comprises: based on the loss function L of semantic segmentation supervision ce The result L of the predicted semantics pred And the real semantic result L gt , semantic segmentation supervision is performed, and the loss value of semantic segmentation supervision is output.

3. The method of claim 1, wherein the optimal transmission based boundary correction is performed by a processor. the boundary prediction step comprises: performing boundary prediction through a boundary prediction branch after the general semantic segmentation network, the boundary prediction branch being composed of a convolution layer and a Sigmoid activation layer, and the prediction result being supervised through a binary cross-entropy loss.

4. The method of claim 1, wherein the optimal transmission based boundary correction method further comprises: the boundary branch comprises: a convolution layer, a regularization layer and an activation layer stacked together, an activation function of the activation layer being Sigmoid, the prediction result of each pixel of the boundary branch being normalized to a range of (0, 1), and a value of each pixel representing a probability of belonging to a boundary.

5. A system for optimal transport based boundary correction, employing the optimal transport based boundary correction method of any one of claims 1-4, characterized in that, The system comprises: a general semantic segmentation module: performing feature extraction, feature enhancement and pixel-by-pixel classification operation based on a general semantic segmentation network, and outputting a loss value of semantic segmentation supervision; a boundary prediction module: introducing a boundary branch after the general semantic segmentation network, calculating an initial position of a predicted boundary based on the boundary branch, and performing supervision through a semantic boundary loss; a boundary correction module: normalizing the prediction result of each pixel of the boundary branch, calculating a semantic boundary positioning loss through optimal transport, correcting the semantic boundary loss through continuous training, and achieving optimal accuracy of boundary region segmentation.

6. A computer-readable storage medium having stored thereon a computer program, characterized in that, The program is executed by the processor to implement the steps of the optimal transport-based boundary correction method according to any one of claims 1-4.

7. An electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The processor executes the program to implement the steps of the optimal transport-based boundary correction method according to any one of claims 1-4.

Images