A panoramic segmentation method, device, equipment and storage medium

Abstract

The application discloses a panoramic segmentation method and device, equipment and storage medium, and relates to the technical field of image segmentation. The method comprises the following steps: dividing semantic categories into several semantic levels according to importance, and weighting learning of the several semantic levels by using a preset initial semantic loss function to obtain a semantic enhancement loss function; determining a target semantic segmentation loss function by using a preset importance matrix based on the semantic enhancement loss function; processing a preset instance segmentation loss function and the target semantic segmentation loss function by using a preset weight coefficient to obtain a target semantic enhancement loss function; and processing an original image based on a preset panoramic segmentation network by using the target semantic enhancement loss function to obtain an instance segmentation result and a semantic segmentation result. Therefore, the application can classify semantic importance, learn semantic levels, optimize a semantic segmentation loss function, and improve the segmentation effect of semantic information with high priority in a panoramic segmentation network.

Description

Technical Field

[0001] This invention relates to the field of image segmentation technology, and in particular to a panoramic segmentation method, apparatus, device, and storage medium. Background Technology

[0002] Image segmentation is a crucial research area in computer vision, with research spanning decades. Traditional segmentation algorithms rely solely on intuitive human visual features, segmenting and classifying images based on color distribution, texture features, and point features. Their segmentation effectiveness is severely limited by the specific scene. With the widespread adoption and development of deep learning, image segmentation has entered a new era. In 2018, Facebook AI Research proposed the concept of panoptic segmentation and provided relevant benchmarks. Compared to traditional semantic and instance segmentation algorithms, panoptic segmentation requires classifying image pixels while simultaneously distinguishing different instances and assigning identification numbers. Panoptic segmentation effectively combines the advantages of semantic and instance segmentation, obtaining classification results for all objects in the image while also distinguishing individual object instances—that is, simultaneously processing image background semantic information and foreground object segmentation.

[0003] Most current segmentation algorithms focus on improving the average segmentation accuracy of all types. As a result, these algorithms have limited effectiveness for segmentation tasks with different semantic segmentation requirements. This leads to suboptimal segmentation accuracy for high-priority information in panoptic segmentation, resulting in limited practicality of the segmentation results. Summary of the Invention

[0004] In view of this, the purpose of this invention is to provide a panoramic segmentation method, apparatus, device, and storage medium that can improve the segmentation accuracy and reliability of important targets based on the importance of semantic information. The specific solution is as follows:

[0005] Firstly, this application provides a panoramic segmentation method, including:

[0006] The semantic categories are divided into several semantic levels according to their importance, and the semantic levels are weighted and learned using a preset initial semantic loss function to obtain a semantic enhancement loss function.

[0007] The target semantic segmentation loss function is determined based on the semantic enhancement loss function using a preset importance matrix;

[0008] The target semantic enhancement loss function is obtained by processing the preset instance segmentation loss function and the target semantic segmentation loss function with preset weight coefficients;

[0009] Based on the preset panoramic segmentation network, the target semantic enhancement loss function is used to process the original image to obtain instance segmentation results and semantic segmentation results.

[0010] Optionally, the step of dividing semantic categories into several semantic levels according to their importance, and then using a preset initial semantic loss function to perform weighted learning on the several semantic levels to obtain a semantic enhancement loss function, includes:

[0011] The semantic categories are divided into several semantic levels according to their importance, and the loss weights corresponding to the object classification in the several semantic levels are recorded using a preset vector.

[0012] Based on a preset initial semantic loss function, the IF-IDF principle is used to perform weighted learning on the classification elements corresponding to the object classifications in the several levels to obtain the semantic enhancement loss function.

[0013] Optionally, determining the target semantic segmentation loss function based on the semantic enhancement loss function using a preset importance matrix includes:

[0014] The importance coefficients corresponding to the several semantic levels are determined using the preset importance matrix;

[0015] The semantic enhancement loss function is processed using the importance coefficient to obtain the target semantic segmentation loss function.

[0016] Optionally, the step of processing the original image using the target semantic enhancement loss function based on the preset panoramic segmentation network to obtain instance segmentation results and semantic segmentation results includes:

[0017] The instance segmentation result is obtained in the Mask R-CNN network based on a fully convolutional neural network and a preset instance segmentation loss function.

[0018] Based on the target semantic enhancement loss function, the semantic segmentation result is obtained by processing the original image using a feature pyramid network in the Mask R-CNN network.

[0019] Optionally, the semantic segmentation result is obtained by processing the original image using a feature pyramid network within the Mask R-CNN network based on the target semantic enhancement loss function, including:

[0020] The feature information of the pyramid levels in the feature pyramid network is merged to obtain the semantic segmentation result.

[0021] Optionally, before obtaining instance segmentation results and semantic segmentation results by processing the original image using the target semantic enhancement loss function based on the preset panoramic segmentation network, the method further includes:

[0022] A preset label class is added to the semantic segmentation function branch corresponding to the target semantic enhancement loss function so as to be output as additional pixels.

[0023] Optionally, after obtaining instance segmentation results and semantic segmentation results by processing the original image using the target semantic enhancement loss function based on the preset panoramic segmentation network, the method further includes:

[0024] Determine whether there is overlap between different instances in the instance segmentation results;

[0025] If there is overlap between different instances in the instance segmentation results, the instances with lower confidence among the overlapping instances will be removed from the instance segmentation results to obtain the processed instance segmentation results.

[0026] Determine whether there is any overlap between the processed instance segmentation result and the semantic segmentation result;

[0027] If the segmentation results overlap, the semantic segmentation results in the overlapping segmentation results are removed to obtain the processed semantic segmentation results.

[0028] Determine whether a preset tag class exists in the processed instance segmentation result and the processed semantic segmentation result;

[0029] If the preset tag class exists in the processed instance segmentation result and the processed semantic segmentation result, the preset tag class is removed to output the removed instance segmentation result and semantic segmentation result.

[0030] If the preset tag class is not present in the processed instance segmentation result and the processed semantic segmentation result, then the processed instance segmentation result and the processed semantic segmentation result are output.

[0031] Secondly, this application provides a panoramic segmentation device, comprising:

[0032] The first function optimization module is used to divide semantic categories into several semantic levels according to their importance, and to use a preset initial semantic loss function to perform weighted learning on the several semantic levels to obtain a semantic enhancement loss function;

[0033] The second function optimization module is used to determine the target semantic segmentation loss function based on the semantic enhancement loss function using a preset importance matrix;

[0034] The objective function determination module is used to process the preset instance segmentation loss function and the target semantic segmentation loss function using preset weight coefficients to obtain the target semantic enhancement loss function;

[0035] The image segmentation module is used to process the original image based on a preset panoramic segmentation network and the target semantic enhancement loss function to obtain instance segmentation results and semantic segmentation results.

[0036] Thirdly, this application provides an electronic device, comprising:

[0037] Memory, used to store computer programs;

[0038] A processor for executing the computer program to implement the panoramic segmentation method described above.

[0039] Fourthly, this application provides a computer-readable storage medium for storing a computer program that, when executed by a processor, implements the above-described panoramic segmentation method.

[0040] Therefore, this application first divides semantic categories into several semantic levels according to their importance, and then uses a preset initial semantic loss function to perform weighted learning on the semantic levels to obtain a semantic enhancement loss function. Based on the semantic enhancement loss function, a preset importance matrix is ​​used to determine the target semantic segmentation loss function. Preset weight coefficients are used to process the preset instance segmentation loss function and the target semantic segmentation loss function to obtain the target semantic enhancement loss function. Based on a preset panoramic segmentation network, the target semantic enhancement loss function is used to process the original image to obtain instance segmentation results and semantic segmentation results. Preset overlap result elimination rules are used to process the instance segmentation results and the semantic segmentation results, and the processed target segmentation result is output. In this way, this application can group semantic categories according to their importance, then perform weighted learning on the corresponding semantic levels and determine the target semantic segmentation loss function based on a preset importance matrix. This weighted learning can effectively suppress classification imbalance problems, and the obtained target semantic segmentation loss function can improve the segmentation accuracy and reliability of important targets. Furthermore, this application can pre-set a panoramic segmentation network and then apply the obtained target semantic enhancement loss function to the preset panoramic segmentation network to improve the segmentation accuracy of the desired object types. Attached Figure Description

[0041] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0042] Figure 1 This is a flowchart of a panoramic segmentation method disclosed in this application;

[0043] Figure 2 This is a flowchart of a specific panoramic segmentation method disclosed in this application;

[0044] Figure 3 This is a schematic diagram of a semantic hierarchy grouping disclosed in this application;

[0045] Figure 4 This is a schematic diagram of an importance matrix disclosed in this application;

[0046] Figure 5 This is a diagram illustrating the structure for calculating the importance coefficient as disclosed in this application;

[0047] Figure 6 This is a flowchart of a specific panoramic segmentation method disclosed in this application;

[0048] Figure 7 This is a diagram of a panoramic segmentation network structure disclosed in this application;

[0049] Figure 8 This is a diagram of an example segmented subnetwork structure disclosed in this application;

[0050] Figure 9 This is a semantic segmentation subnetwork structure diagram disclosed in this application;

[0051] Figure 10 This is a comparison image of a semantic segmentation result disclosed in this application;

[0052] Figure 11 This is a schematic diagram of the structure of a panoramic segmentation device disclosed in this application;

[0053] Figure 12 This is a structural diagram of an electronic device disclosed in this application. Detailed Implementation

[0054] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0055] Existing literature shows that most current image segmentation algorithms focus on improving the average segmentation accuracy across all categories, neglecting the varying requirements and emphasis placed on semantic segmentation results by different tasks. This leads to unsatisfactory accuracy and limited practicality in many panoramic segmentations. This application optimizes existing semantic segmentation loss functions for different semantic categories to obtain semantic enhancement loss functions, and utilizes a panoramic segmentation network with the added semantic enhancement loss function for image segmentation, thereby improving the segmentation accuracy for important semantic categories.

[0056] See Figure 1 As shown, this embodiment of the invention discloses a panoramic segmentation method, including:

[0057] Step S11: Divide the semantic categories into several semantic levels according to their importance, and use a preset initial semantic loss function to perform weighted learning on the several semantic levels to obtain a semantic enhancement loss function.

[0058] It is understandable that the loss function in panoptic segmentation networks is usually composed of a semantic segmentation sub-branch loss function and an instance segmentation sub-branch loss function. However, traditional loss functions give the same loss cost for all semantic categories, thus ignoring the different needs and levels of importance for different semantic segmentation results in different segmentation tasks, leading to unsatisfactory panoptic segmentation accuracy in many cases. In this embodiment, semantic categories can be divided into several semantic levels according to their importance. Then, based on the preset initial semantic loss function, the loss weights corresponding to the object categories in the several semantic levels are weighted and learned. The semantic enhancement loss function obtained in this way will have higher segmentation accuracy for specific objects and backgrounds in image segmentation than that of the preset initial semantic loss function. It should be noted that the preset initial semantic loss function can include commonly used semantic segmentation loss functions such as cross-entropy loss function and Focal Loss. In this application, the cross-entropy loss function is selected as the preset initial semantic loss function. The cross-entropy loss function represents the distance between the actual output and the expected output, used to characterize the similarity between the predicted value and the true value. The smaller the cross-entropy, the closer the two probability distributions are. The traditional definition is:

[0059]

[0060] q i,j and p i,j All are one-hot encoded data of length C, where C is the total number of categories and q is the number of categories. i,j Let p be the truth vector at (i,j), where the position with the correct semantic label is marked as 1, and the others as 0; i,j Let be the prediction vector at (i,j), where each array element corresponds to the corresponding classification prediction probability, and H and W are the height and width of the image, respectively.

[0061] Step S12: Determine the target semantic segmentation loss function based on the semantic enhancement loss function using a preset importance matrix.

[0062] In this embodiment, after obtaining the semantic enhancement loss function, the semantic enhancement loss function can be further processed using the preset importance matrix. It should be noted that the preset importance matrix can determine the grouping coefficients corresponding to different semantic levels, and then the target semantic segmentation loss function can be obtained by calculating the semantic enhancement loss function using the grouping coefficients.

[0063] Step S13: Process the preset instance segmentation loss function and the target semantic segmentation loss function using preset weight coefficients to obtain the target semantic enhancement loss function.

[0064] In this embodiment, it is understood that the method of this application still adopts a structure of separately calculating the loss function. The preset instance segmentation loss function and the target semantic segmentation loss function can be processed using the preset weight coefficients to obtain the target semantic enhancement loss function. In this way, different emphases can be achieved on the semantic segmentation network and the instance segmentation network by controlling the preset weight coefficients. It is understood that the two independent image segmentation networks can also be trained separately, which can halve the computational load. Furthermore, when the weight coefficient corresponding to the instance segmentation network is 0, the semantic segmentation network can be trained; correspondingly, when the weight coefficient corresponding to the semantic segmentation network is 0, the instance segmentation network can be trained.

[0065] Step S14: Based on the preset panoramic segmentation network, the original image is processed using the target semantic enhancement loss function to obtain instance segmentation results and semantic segmentation results.

[0066] In this embodiment, after obtaining the semantic enhancement loss function, the original image can be processed using the target semantic enhancement loss function based on a preset panoramic segmentation network to obtain instance segmentation results and semantic segmentation results. It is understood that the network structure for completing the panoramic segmentation task must meet the following conditions: sufficiently high resolution to resolve minute structures; sufficient semantic encoding to accurately predict object classification; and multi-scale information to perform predictions at different resolutions. The Feature Pyramid Network (FPN) possesses the ability to extract high-resolution, rich multi-scale features, therefore, the panoramic segmentation task can be completed by adding a semantic segmentation network.

[0067] It should be noted that, in a specific embodiment, after obtaining the instance segmentation result and the semantic segmentation result, the process may further include: determining whether there is overlap between different instances in the instance segmentation result; specifically, if there is overlap between different instances in the instance segmentation result, the instances with low confidence among the overlapping different instances are removed from the instance segmentation result to obtain the processed instance segmentation result; further, determining whether there is overlap between the processed instance segmentation result and the semantic segmentation result; specifically, if there is overlap between the segmentation results, the semantic segmentation result in the overlapping segmentation result is removed to obtain the processed semantic segmentation result; it can be understood that it can be determined whether there is a preset marker class in the processed instance segmentation result and the processed semantic segmentation result; specifically, if there is the preset marker class in the processed instance segmentation result and the processed semantic segmentation result, the preset marker class is removed to output the removed instance segmentation result and the semantic segmentation result; if there is no preset marker class in the processed instance segmentation result and the processed semantic segmentation result, the processed instance segmentation result and the processed semantic segmentation result are output. It is understandable that the panoramic output format requires assigning a class label and instance ID to each image pixel; in order to avoid the problem of overlapping outputs of instances and semantic segmentation branches in the network structure, this application can remove overlapping segmentation results to avoid overlapping segmentation results.

[0068] Therefore, this application can optimize a preset initial semantic loss function based on the importance of semantic categories, loss weights, and a preset importance matrix to obtain a target semantic segmentation loss function. Furthermore, it weights the preset instance segmentation loss function and the target semantic segmentation loss function based on preset weight coefficients to obtain a target semantic enhancement loss function for the panoramic segmentation network. Further, this application can obtain instance segmentation results and semantic segmentation results based on a preset panoramic segmentation network, and then remove overlapping segmentation results from the obtained instance segmentation results and semantic segmentation results to obtain the target segmentation result. In this way, grouping semantic categories according to importance and performing weighted learning in this application can improve the classification accuracy of specific objects and backgrounds. Moreover, the optimized target semantic enhancement loss function can enhance the sensitivity to semantic information in image segmentation tasks, meeting the different segmentation accuracy requirements for different semantic categories in segmentation tasks.

[0069] The following examples will provide a detailed description of the steps for obtaining the target semantic enhancement loss function. (See also...) Figure 2 As shown, this embodiment of the invention discloses a panoramic segmentation method, including:

[0070] Step S21: Divide the semantic categories into several semantic levels according to their importance, and use a preset vector to record the loss weights corresponding to the object classification in the several semantic levels.

[0071] In this embodiment, semantic categories can be divided into several semantic levels according to their importance; it is understood that, as Figure 3 As shown, in a specific embodiment, semantic categories can be divided into four levels, with importance decreasing from R4 to R1. The classification is based on a loop closure detection task as an example: compared to dynamic objects R2 and R1, static objects R4 and R3 provide more reliable and robust reference information; static instance objects (things) R4 are more valuable as references and signposts than static background (stuff) R3. Among dynamic objects, compared to objects like vehicles (R2), which are sometimes stationary, objects like people and animals (R1) are high-frequency moving objects and occur frequently, thus representing interference information.

[0072] It should be noted that, in one specific embodiment, the cross-entropy loss function can be selected as the preset initial semantic loss function. Correspondingly, after grouping semantic categories according to their importance, vectors can be defined to store the cross-entropy loss based on the semantic level grouping. For example, after dividing semantic categories into four levels, vectors V1, V2, V3, and V4 can be defined to store the cross-entropy loss of object classification in the four groups, respectively. For example, the loss value of the j-th pixel in the i-th group is defined as:

[0073]

[0074] Among them, O c,i,j Let q be a tensor that outputs c classification probabilities at image (i,j). c It is a one-hot encoding where the c-th element is 1. Similarly, four vectors W1, W2, W3, and W4 can be defined using the preset vectors to record the loss weights corresponding to the object classification in the four groups, and the loss weights can be used to effectively suppress the classification imbalance problem. W i The definition of is:

[0075]

[0076] Where s i,c α represents the total number of times the c-th category appears in the training set pixels, and α and β represent control parameters.

[0077] Step S22: Based on the preset initial semantic loss function, the classification elements corresponding to the object classification in the several levels are weighted and learned using the IF-IDF principle to obtain the semantic enhancement loss function.

[0078] In this embodiment, the IF-IDF principle can be used for the weighted classification element process. For semantic categories with high frequency, the loss weight needs to be reduced to decrease the adjustment magnitude of each iteration and reduce training divergence. For categories with low frequency, the weight is increased to improve the learning process of the iteration, thus accelerating the learning process. It is understood that, in a specific embodiment, after weighted learning, the enhancement loss function can be defined as W. i T L Pi (i = 1, 2, 3, 4), where i is the number of semantic levels according to importance, which can be a number of groups in this application.

[0079] Step S23: Determine the importance coefficients corresponding to the several semantic levels using a preset importance matrix.

[0080] In this embodiment, after obtaining the enhancement loss function, importance coefficients corresponding to the several semantic levels can be determined based on the preset importance matrix. For example... Figure 4 As shown, in one specific embodiment, a preset importance matrix M is used. t It can include three parts, each of which is H×W in size, that is, the height of the image multiplied by its width. The horizontal rectangular areas in the matrix represent the aforementioned grouping levels, corresponding from top to bottom to the above... Figure 3 The matrix is ​​composed of R1, R2, R3, and R4, where 1 represents high importance and 0 represents low importance. For example, R4 contains the most 1s, indicating that the object category in R4 is the most important. This can be based on M... t Define the importance coefficient θ(M) t (t=1,2,3):

[0081]

[0082] Where E is an all-one matrix (one-matrix), For parameter tuning, in this embodiment, a value of 0.5 can be used, and G is the output value. c,i,j The predicted probability corresponding to the ground truth classification label at image (i,j) is represented by the ⊙ operation, which denotes element-wise multiplication of matrices. The value of γ determines θ(M t The magnitude of γ, when amplified, outputs G and M. t The gap between them will widen, especially when M t =1.

[0083] Furthermore, in one specific embodiment, it is possible to utilize, as Figure 5The calculation structure shown is used to calculate the importance coefficient of the loss function. The group containing R1 has the lowest importance, so the importance coefficient of R1 can be set to 1; the importance coefficient of R2 is θ(M1)+1; the importance coefficient of R3 is (θ(M1)+1)(θ(M2)+1); and the importance coefficient of R4 is (θ(M1)+1)(θ(M2)+1)(θ(M3)+1).

[0084] Step S24: Process the semantic enhancement loss function using the importance coefficient to obtain the target semantic segmentation loss function.

[0085] In this embodiment, after obtaining the importance coefficient, the semantic enhancement loss function is processed using the importance coefficient to obtain the target semantic segmentation loss function. Specifically, according to the specific implementation in the above steps, the importance coefficient of group R1 is 1; the importance coefficient of group R2 is θ(M1)+1; the importance coefficient of group R3 is (θ(M1)+1)(θ(M2)+1); and the importance coefficient of group R4 is (θ(M1)+1)(θ(M2)+1)(θ(M3)+1). Thus, the corresponding calculation formula for the target semantic segmentation loss function can be obtained:

[0086]

[0087] Therefore, this application can group objects according to the importance of semantic categories and perform weighted learning using the loss weights corresponding to the object classification in the respective groups. This can effectively suppress the classification imbalance problem. Then, the target semantic segmentation loss function can be determined using the obtained semantic enhancement loss function and the preset importance matrix. Processing the loss function with the preset importance matrix can improve the sensitivity of the obtained loss function to the loss bias of important objects.

[0088] In the above embodiments, the preset initial semantic loss function can be optimized according to the importance of semantic categories to obtain the target semantic enhancement loss function. This can improve the sensitivity to semantic information in image segmentation tasks and improve the matching degree with the segmentation requirements of different levels of semantic information in segmentation tasks. The following embodiments will describe the steps of panoramic segmentation using the target semantic enhancement loss function. See Figure 6 As shown, this embodiment of the invention discloses a panoramic segmentation method, including:

[0089] Step S31: Divide the semantic categories into several semantic levels according to their importance, and use a preset initial semantic loss function to perform weighted learning on the several semantic levels to obtain a semantic enhancement loss function.

[0090] Step S32: Determine the target semantic segmentation loss function based on the semantic enhancement loss function using a preset importance matrix.

[0091] Step S33: Process the preset instance segmentation loss function and the target semantic segmentation loss function using preset weight coefficients to obtain the target semantic enhancement loss function.

[0092] Step S34: Obtain instance segmentation results in the Mask R-CNN network based on a fully convolutional neural network and a preset instance segmentation loss function.

[0093] It should be noted that existing technologies can add FCN (Fully Convolutional Networks) to the Faster R-CNN network to obtain Mask R-CNN (Instance Segmentation Network), which can then be used to obtain instance segmentation results. Furthermore, the feature pyramid network can be used as a branch network for semantic information segmentation within the Mask R-CNN network. It is understood that the feature pyramid network can provide a lightweight network structure, fast segmentation speed, and accurate segmentation results in semantic segmentation. In this way, a Mask R-CNN network with an added semantic segmentation branch network can be used for panoptic segmentation tasks.

[0094] In this embodiment, before obtaining the instance segmentation result based on the fully convolutional neural network, a target semantic enhancement loss function can be obtained by processing the preset instance segmentation loss function and the target semantic segmentation loss function with preset weight coefficients. Specifically, the preset instance segmentation loss function consists of three parts: classification loss L... class Boundary loss L box and mask loss L mask L class and L box The number of sampled Regions of Interest (RoIs) is normalized, L mask This can be achieved by normalizing the number of foreground RoIs. In this way, the instance segmentation loss function L corresponding to the instance segmentation sub-network... instance The definition of is:

[0095] L instance =L class +L box +L mask

[0096] Then, the instance segmentation loss function and the target semantic segmentation loss function can be processed according to the preset weight coefficients to obtain the target semantic enhancement loss function, which is defined as follows:

[0097] L=λ1L semantic +λ2(L class +L box +L mask )

[0098] Furthermore, such as Figure 7 As shown, this application can utilize the optimized target semantic enhancement loss function to segment images in a Mask R-CNN network. Specifically, as... Figure 8 As shown, after obtaining the target semantic enhancement loss function, the instance segmentation result can be obtained through the instance segmentation subnetwork in the Mask R-CNN network.

[0099] Step S35: Based on the target semantic enhancement loss function, the original image is processed using a feature pyramid network in the Mask R-CNN network to obtain semantic segmentation results.

[0100] In this embodiment, as Figure 9 As shown, the feature pyramid network can be used as a semantic segmentation sub-network to obtain semantic segmentation results. This can include merging feature information at each pyramid level in the feature pyramid network to obtain the semantic segmentation results. Specifically, the top layer of the feature pyramid network has a 1 / 32 resolution ratio, and a 1 / 4 resolution feature map can be obtained using three upsampling operations. Each upsampling operation consists of a 3×3 convolution, population normalization, ReLU (Rectified Linear Unit, activation function), and 2x bilinear upsampling. This operation is then repeated on FPNs with resolution ratios of 1 / 16, 1 / 8, and 1 / 4. The upsampling result of each layer is the same 1 / 4 resolution feature map, which is then summed element-wise. Finally, 4x bilinear upsampling and a 1×1 convolution are added to obtain pixel classification labels with the same resolution as the original image.

[0101] It should be noted that the semantic segmentation subnetwork may include adding a preset label class to the semantic segmentation function branch corresponding to the target semantic enhancement loss function, so as to output additional pixels. Understandably, adding a label class in this way can avoid forcibly predicting the type of filler for each pixel, thus preventing misjudgments.

[0102] In this embodiment, the FPN configuration has 256 output channels per scale, while the semantic segmentation branch reduces the number of channels to 128. For the backbone network preceding the FPN, a ResNet (residual network) model can be pre-trained on ImageNet (a dataset) using Batch Norm (BN). During fine-tuning, a fixed-channel affine transformation can be used instead of BN.

[0103] For more detailed processing procedures regarding steps S31, S32, and S33, please refer to the corresponding content disclosed in the foregoing embodiments, which will not be repeated here.

[0104] Therefore, this application can construct an instance segmentation subnetwork and a semantic segmentation subnetwork for images based on the Mask R-CNN network and the feature pyramid network. Then, by using the optimized target semantic enhancement loss function to perform panoramic segmentation of the image, the accuracy and reliability of segmentation of important targets in the image segmentation task can be improved.

[0105] The following embodiments will illustrate the technical solutions in this application in detail with reference to experiments.

[0106] In this embodiment, to systematically evaluate the segmentation performance of the aforementioned panoramic segmentation network, the COCO dataset was used as the training and testing dataset for panoramic segmentation. The COCO dataset provides 80 semantic categories, basically covering the classification and learning of common objects in daily life; it also includes data under different resolutions, viewpoints, and lighting conditions.

[0107] In experiments with panoptic segmentation networks, six evaluation metrics are typically used: Average Precision (AP), Average Recall (AR), Intersection-over-Union (IoU), Segmentation Quality (SQ), Recognition Quality (RQ), and Panoptic Quality (PQ). Specifically, Average Precision reflects the accuracy of semantic category segmentation, and its corresponding formula is as follows:

[0108]

[0109] Where TP is the true positives value, representing the number of correctly predicted positive samples; FP is the false positives value, representing the number of incorrectly predicted positive samples.

[0110] The average recall rate reflects the proportion of true instances recalled for a semantic category, and the corresponding formula is as follows:

[0111]

[0112] FN represents the false negatives value, indicating the number of incorrect predictions of negative samples.

[0113] The intersection-union ratio refers to the ratio of the intersection to the union of the true value and the predicted region, and the corresponding formula is as follows:

[0114]

[0115] During the detection process, when the IoU is greater than the preset threshold, the detection result can be judged as positive; otherwise, it can be judged as incorrect. It's understandable that there are three main IoU values: IoU = 0.5:0.95, which means taking a value every 0.05 from the threshold of 0.5 to 0.95, for a total of ten calculations, and averaging the results; IoU = 0.5 is the metric adopted by PASCAL VOC (The PASCAL Visual Object Classes, a computer vision challenge); IoU = 0.75 is a fixed threshold metric, which is relatively strict.

[0116] The segmentation quality index is used to evaluate the semantic segmentation network, and the corresponding definition formula is:

[0117]

[0118] in SQ represents the sum of IoU values ​​of the matched segmented regions, while SQ represents the average IoU value of the matched segmented regions.

[0119] The recognition quality is used to evaluate the instance segmentation subnetwork, and the corresponding definition formula is:

[0120]

[0121] The panoramic segmentation quality index is used to evaluate the overall panoramic segmentation network framework by combining segmentation quality parameters and recognition quality parameters. The corresponding definition formula is as follows:

[0122]

[0123] It should be noted that we will first explore the loss coefficient for gradient balancing. The weighting formula for the loss coefficient is as follows:

[0124] L=λ1L semantic +λ2(L class +L b o x +L mask )

[0125] In the experiment, since the semantic segmentation subnetwork is the main point of application for the semantic enhancement loss function, the instance segmentation subnetwork coefficient λ2 can be set to 1. Segmentation prediction is performed by testing different semantic segmentation loss function coefficients λ1, and the test results are shown in Table 1 below:

[0126] Table 1

[0127]

[0128] Among them, PQ th PQ represents the segmentation quality of objects. st This represents the segmentation quality of the stuffing. Experimental results show that if λ1 is too large or too small, the network equilibrium will be disrupted, leading to a decrease in the learning efficiency of both sub-networks. Furthermore, when λ1 is too large, the gradient magnitude fed from the stuffing network to the base network is too large, which reduces the prediction accuracy of the instance segmentation sub-network. Experiments verify that when λ1 = 0.4, the instance and semantic segmentation networks are in optimal balance, achieving a segmentation quality of 43.02%.

[0129] Understandably, after finding the optimal parameters, a comparative experiment is conducted between the optimized semantic enhancement loss function and the preset cross-entropy loss function on the semantic segmentation sub-branch. The optimized semantic enhancement loss function is the target semantic enhancement loss function mentioned in the above embodiments, and the cross-entropy loss function is the preset initial semantic loss function mentioned in the above embodiments. The experimental configuration remains unchanged in both sets of experiments, except for the semantic loss function used in the semantic branch. Then, a panoptic segmentation experiment is performed, and the experimental results are shown in Table 2.

[0130] Table 2

[0131]

[0132] Where mIoU represents the mean IoU, and fwIoU represents the frequency-weighted IoU. Experimental results show that the semantic enhancement loss function improves the IoU metric compared to the cross-entropy loss function, but significantly enhances the segmentation performance of stuff objects (stuff class). Panoptic segmentation quality improves by 0.97%, and overall segmentation quality improves by 4.30%. This is because, during segmentation, the semantic categories represented by stuff objects are generally assigned higher priority, resulting in a greater improvement in stuff object segmentation quality.

[0133] Furthermore, to more intuitively test the role of the semantic enhancement loss function in the panoptic segmentation process, panoptic segmentation experiments can be conducted according to groups. Table 3 below shows the segmentation results for different groups.

[0134] Experimental results show that the semantic enhancement loss function, optimized based on importance, can effectively improve the semantic classification accuracy in the R3 and R4 groups mentioned in the above embodiments, with a maximum improvement of 3.93%. Meanwhile, in the R1 and R2 groups, the segmentation accuracy of the semantic enhancement loss function is basically on par with that of the cross-entropy loss function. In summary, the semantic enhancement loss function can improve the segmentation accuracy of higher-priority semantic classifications, while the semantic classification accuracy of other less important targets experiences a slight decrease, achieving the intended design goal.

[0135] Table 3

[0136]

[0137]

[0138] like Figure 10 The panoramic segmentation results of the semantic enhancement loss function and the cross-entropy loss function under different test environments are shown. It can be clearly seen that the semantic enhancement loss function can effectively improve the segmentation effect of static objects.

[0139] like Figure 11 As shown, this application discloses a panoramic segmentation device, comprising:

[0140] The first function optimization module 11 is used to divide semantic categories into several semantic levels according to their importance, and to use a preset initial semantic loss function to perform weighted learning on the several semantic levels to obtain a semantic enhancement loss function.

[0141] The second function optimization module 12 is used to determine the target semantic segmentation loss function based on the semantic enhancement loss function using a preset importance matrix;

[0142] The objective function determination module 13 is used to process the preset instance segmentation loss function and the target semantic segmentation loss function using preset weight coefficients to obtain the target semantic enhancement loss function;

[0143] Image segmentation module 14 is used to process the original image based on a preset panoramic segmentation network and the target semantic enhancement loss function to obtain instance segmentation results and semantic segmentation results.

[0144] Therefore, this application can classify semantic categories into different semantic levels according to the importance of semantics, and can perform weighted learning on the classification elements in different levels. This can effectively suppress the problem of classification imbalance. Furthermore, optimizing the semantic loss function in this way can enhance the sensitivity to semantic information in image segmentation tasks, thereby improving the segmentation accuracy of specific objects and backgrounds. In addition, this application can obtain instance segmentation results and semantic segmentation results based on a preset panoramic segmentation network using the optimized target semantic segmentation loss function, which can improve the segmentation effect of specific objects.

[0145] In one specific embodiment, the first function optimization module 11 may include:

[0146] A semantic segmentation unit is used to divide the semantic categories into several semantic levels according to their importance.

[0147] The loss weight determination unit is used to record the loss weights corresponding to the object classification in the several semantic levels using a preset vector.

[0148] The first function determination unit is used to perform weighted learning on the classification elements corresponding to the object classification in the several levels based on the preset initial semantic loss function and the IF-IDF principle, so as to obtain the semantic enhancement loss function.

[0149] In one specific embodiment, the second function optimization module 12 may include:

[0150] A coefficient determination unit is used to determine the importance coefficients corresponding to the several semantic levels using the preset importance matrix;

[0151] The second function determination unit is used to process the semantic enhancement loss function using the importance coefficient to obtain the target semantic segmentation loss function.

[0152] In one specific embodiment, the image segmentation module 14 may include:

[0153] The first result acquisition unit is used to obtain the instance segmentation result in the Mask R-CNN based on the fully convolutional neural network and the preset instance segmentation loss function.

[0154] The second result acquisition submodule is used to obtain the semantic segmentation result obtained by processing the original image using the target semantic enhancement loss function in the Mask R-CNN using the feature pyramid network.

[0155] Accordingly, in one specific embodiment, the second result acquisition submodule may include:

[0156] The second result acquisition unit is used to merge the feature information of the pyramid level in the feature pyramid network to obtain the semantic segmentation result.

[0157] Furthermore, in one specific embodiment, the image segmentation module 14 may further include:

[0158] The label class determination unit is used to add a preset label class to the semantic segmentation function branch corresponding to the target semantic enhancement loss function so as to output as additional pixel points.

[0159] In one specific embodiment, the image segmentation module 14 may further include:

[0160] The first judgment unit is used to determine whether there is overlap between different instances in the instance segmentation result;

[0161] The first elimination unit is used to eliminate instances with low confidence from the instance segmentation results when there is overlap between different instances in the instance segmentation results, so as to obtain the processed instance segmentation results.

[0162] The second judgment unit is used to determine whether there is any overlap between the processed instance segmentation result and the semantic segmentation result;

[0163] The second elimination unit is used to eliminate the semantic segmentation results in the overlapping segmentation results when there are overlapping segmentation results, so as to obtain the processed semantic segmentation results.

[0164] The third judgment unit is used to determine whether there is a preset tag class in the processed instance segmentation result and the processed semantic segmentation result;

[0165] The third elimination unit is used to eliminate the preset marker class when the processed instance segmentation result and the processed semantic segmentation result contain the preset marker class, so as to output the eliminated instance segmentation result and semantic segmentation result.

[0166] The result output unit is used to output the processed instance segmentation result and the processed semantic segmentation result when the preset tag class is not present in the processed instance segmentation result and the processed semantic segmentation result.

[0167] Furthermore, embodiments of this application also disclose an electronic device, Figure 12 This is a structural diagram of an electronic device 20 according to an exemplary embodiment. The content of the diagram should not be construed as limiting the scope of this application.

[0168] Figure 12This is a schematic diagram of the structure of an electronic device 20 provided in an embodiment of this application. Specifically, the electronic device 20 may include: at least one processor 21, at least one memory 22, a power supply 23, a communication interface 24, an input / output interface 25, and a communication bus 26. The memory 22 stores a computer program, which is loaded and executed by the processor 21 to implement the relevant steps in the panoramic segmentation method disclosed in any of the foregoing embodiments. Furthermore, the electronic device 20 in this embodiment may specifically be an electronic computer.

[0169] In this embodiment, the power supply 23 is used to provide operating voltage for each hardware device on the electronic device 20; the communication interface 24 can create a data transmission channel between the electronic device 20 and external devices, and the communication protocol it follows can be any communication protocol applicable to the technical solution of this application, and is not specifically limited here; the input / output interface 25 is used to acquire external input data or output data to the outside world, and its specific interface type can be selected according to specific application needs, and is not specifically limited here.

[0170] In addition, the memory 22, as a carrier for resource storage, can be a read-only memory, random access memory, disk or optical disk, etc. The resources stored thereon can include operating system 221, computer program 222, etc., and the storage method can be temporary storage or permanent storage.

[0171] The operating system 221 is used to manage and control the various hardware devices on the electronic device 20 and the computer program 222, which may be Windows Server, Netware, Unix, Linux, etc. In addition to including a computer program capable of performing the panoramic segmentation method executed by the electronic device 20 as disclosed in any of the foregoing embodiments, the computer program 222 may further include a computer program capable of performing other specific tasks.

[0172] Furthermore, this application also discloses a computer-readable storage medium for storing a computer program; wherein, when the computer program is executed by a processor, it implements the aforementioned panoramic segmentation method. Specific steps of this method can be found in the corresponding content disclosed in the foregoing embodiments, and will not be repeated here.

[0173] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to in the method section.

[0174] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0175] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein can be implemented directly by hardware, a software module executed by a processor, or a combination of both. The software module can be located in random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art.

[0176] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0177] The technical solutions provided in this application have been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the methods and core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A panorama segmentation method, characterized in that, include: The semantic categories are divided into several semantic levels according to their importance, and the semantic levels are weighted and learned using a preset initial semantic loss function to obtain a semantic enhancement loss function. The target semantic segmentation loss function is determined based on the semantic enhancement loss function using a preset importance matrix; The target semantic enhancement loss function is obtained by processing the preset instance segmentation loss function and the target semantic segmentation loss function with preset weight coefficients; Based on the preset panoramic segmentation network, the target semantic enhancement loss function is used to process the original image to obtain instance segmentation results and semantic segmentation results; The process of classifying semantic categories into several semantic levels according to their importance, and then using a preset initial semantic loss function to perform weighted learning on the several semantic levels to obtain a semantic enhancement loss function, includes: The semantic categories are divided into several semantic levels according to their importance, and the loss weights corresponding to the object classification in the several semantic levels are recorded using a preset vector. Based on a preset initial semantic loss function, the IF-IDF principle is used to perform weighted learning on the classification elements corresponding to the object classification in the several semantic levels to obtain the semantic enhancement loss function; The step of determining the target semantic segmentation loss function based on the semantic enhancement loss function using a preset importance matrix includes: The importance coefficients corresponding to the several semantic levels are determined using the preset importance matrix; The semantic enhancement loss function is processed using the importance coefficient to obtain the target semantic segmentation loss function.

2. The panorama segmentation method of claim 1, wherein, The method based on the preset panoramic segmentation network uses the target semantic enhancement loss function to process the original image to obtain instance segmentation results and semantic segmentation results, including: The instance segmentation result is obtained by processing the original image in the Mask R-CNN network based on a fully convolutional neural network and a preset instance segmentation loss function. Based on the target semantic enhancement loss function, the semantic segmentation result is obtained by processing the original image using a feature pyramid network in the Mask R-CNN network.

3. The panorama segmentation method of claim 2, wherein, The semantic segmentation result obtained by processing the original image using a feature pyramid network within the Mask R-CNN network based on the target semantic enhancement loss function includes: The feature information of the pyramid levels in the feature pyramid network is merged to obtain the semantic segmentation result.

4. The panorama segmentation method according to any one of claims 1 to 3, characterized in that, Before obtaining instance segmentation results and semantic segmentation results by processing the original image using the target semantic enhancement loss function based on the preset panoramic segmentation network, the process further includes: A preset label class is added to the semantic segmentation function branch corresponding to the target semantic enhancement loss function so as to be output as additional pixels.

5. The panorama segmentation method of claim 4, wherein, After obtaining instance segmentation results and semantic segmentation results by processing the original image using the target semantic enhancement loss function based on the preset panoramic segmentation network, the process further includes: Determine whether there is overlap between different instances in the instance segmentation results; If there is overlap between different instances in the instance segmentation results, the instances with lower confidence among the overlapping instances will be removed from the instance segmentation results to obtain the processed instance segmentation results. Determine whether there is any overlap between the processed instance segmentation result and the semantic segmentation result; If the segmentation results overlap, the semantic segmentation results in the overlapping segmentation results are removed to obtain the processed semantic segmentation results. Determine whether a preset tag class exists in the processed instance segmentation result and the processed semantic segmentation result; If the preset tag class exists in the processed instance segmentation result and the processed semantic segmentation result, the preset tag class is removed to output the removed instance segmentation result and semantic segmentation result. If the preset tag class is not present in the processed instance segmentation result and the processed semantic segmentation result, then the processed instance segmentation result and the processed semantic segmentation result are output.

6. A panorama segmentation apparatus characterized by comprising: include: The first function optimization module is used to divide semantic categories into several semantic levels according to their importance, and to use a preset initial semantic loss function to perform weighted learning on the several semantic levels to obtain a semantic enhancement loss function; The second function optimization module is used to determine the target semantic segmentation loss function based on the semantic enhancement loss function using a preset importance matrix; The objective function determination module is used to process the preset instance segmentation loss function and the target semantic segmentation loss function using preset weight coefficients to obtain the target semantic enhancement loss function; The image segmentation module is used to process the original image based on a preset panoramic segmentation network and the target semantic enhancement loss function to obtain instance segmentation results and semantic segmentation results; The first function optimization module includes: A semantic segmentation unit is used to divide the semantic categories into several semantic levels according to their importance. The loss weight determination unit is used to record the loss weights corresponding to the object classification in the several semantic levels using a preset vector. The first function determination unit is used to perform weighted learning on the classification elements corresponding to the object classification in the several semantic levels based on the preset initial semantic loss function and the IF-IDF principle, so as to obtain the semantic enhancement loss function. The second function optimization module includes: A coefficient determination unit is used to determine the importance coefficients corresponding to the several semantic levels using the preset importance matrix; The second function determination unit is used to process the semantic enhancement loss function using the importance coefficient to obtain the target semantic segmentation loss function.

7. An electronic device, comprising: include: Memory, used to store computer programs; A processor for executing the computer program to implement the panoramic segmentation method as described in any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that, Used to store a computer program, which, when executed by a processor, implements the panoramic segmentation method as described in any one of claims 1 to 5.

Images