Panorama segmentation method and related apparatus, electronic device, storage medium

Abstract

The application discloses a panoramic segmentation method and related devices, electronic equipment and storage medium, wherein the panoramic segmentation method comprises the following steps: extracting a first feature map of a to-be-segmented image; based on the first feature map, the category information and the position information of a first pixel point in the first feature map are respectively predicted, and the second feature map is obtained through feature generation based on the first feature map; based on the category information and the position information of the first pixel point in the first feature map, the position information of each image object is extracted; and based on the second feature map and the position information of each image object, panoramic segmentation is performed to obtain a panoramic segmentation image of the to-be-segmented image. The above scheme can realize real-time deployment of panoramic segmentation on an edge device.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, in particular to a panorama segmentation method and related device, electronic equipment and storage medium. BACKGROUND

[0002] Scene perception as a basic task of computer vision research has a large number of technology landing in many practical scenarios. Among them, panorama segmentation as a bottom task of scene perception has always been a high research hotspot. Unlike other scene perception tasks such as object detection, semantic segmentation, and instance segmentation, panorama segmentation can be regarded as a combination of semantic segmentation and instance segmentation, that is, both detecting all target objects and distinguishing different instances in the same category (for example, distinguishing target objects "A", "B", "C" that belong to the "person" category). As can be seen, panorama segmentation is the most complex but the most comprehensive analysis in the computer vision scene perception task.

[0003] At present, the mainstream panorama segmentation scheme mainly includes the following two technical routes: two-stage and post-processing fusion to realize panorama segmentation, and single-stage end-to-end based on query to realize panorama segmentation. The former first obtains different foreground instance categories (Things) and background categories (Stuff) segmentation results based on instance segmentation and semantic segmentation, and then obtains panorama segmentation through post-processing fusion. This way is difficult to deploy in real time on edge devices due to its more technical processes and higher complexity of each atomic ability. While the latter simplifies the panorama segmentation task in the process, but due to the high computational complexity of the attention mechanism in the Transformer and the lack of bottom hardware acceleration optimization, it is still difficult to realize real-time deployment on edge devices. Therefore, how to deploy panorama segmentation in real time on edge devices has become a problem to be solved. SUMMARY

[0004] The technical problem solved by the present application is to provide a panorama segmentation method and related device, electronic equipment and storage medium, which can deploy panorama segmentation in real time on edge devices.

[0005] To solve the above technical problems, the first aspect of the present application provides a panoramic segmentation method, comprising: extracting a first feature map of a to-be-segmented image; wherein the to-be-segmented image contains image objects of several categories, and the image objects include at least one of instances and backgrounds; based on the first feature map, class information and position information of a first pixel point in the first feature map are respectively predicted, and feature generation is performed based on the first feature map to obtain a second feature map; based on the class information and the position information of the first pixel point in the first feature map, position information of each image object is extracted; and panoramic segmentation is performed based on the second feature map and the position information of each image object to obtain a panoramic segmentation map of the to-be-segmented image.

[0006] To solve the above technical problems, the second aspect of the present application provides a panoramic segmentation device, comprising: a feature extraction module, an information prediction module, a feature generation module, an information extraction module, and an image segmentation module; the feature extraction module is used to extract a first feature map of a to-be-segmented image; wherein the to-be-segmented image contains image objects of several categories, and the image objects include at least one of instances and backgrounds; the information prediction module is used to respectively predict class information and position information of a first pixel point in the first feature map based on the first feature map; the feature generation module is used to perform feature generation based on the first feature map to obtain a second feature map; the information extraction module is used to extract position information of each image object based on the class information and the position information of the first pixel point in the first feature map; and the image segmentation module is used to perform panoramic segmentation based on the second feature map and the position information of each image object to obtain a panoramic segmentation map of the to-be-segmented image.

[0007] To solve the above technical problems, the third aspect of the present application provides an electronic device, comprising a memory and a processor coupled to each other, the memory stores program instructions, and the processor is used to execute the program instructions to realize the panoramic segmentation method of the first aspect.

[0008] To solve the above technical problems, the fourth aspect of the present application provides a computer readable storage medium, which stores program instructions capable of being executed by a processor, and the program instructions are used to realize the panoramic segmentation method of the first aspect.

[0009] The above scheme extracts a first feature map of a to-be-segmented image, and the to-be-segmented image contains image objects of several categories, and the image objects include at least one of instances and backgrounds. On this basis, based on the first feature map, the category information and the position information of a first pixel point in the first feature map are respectively predicted, and feature generation is performed based on the first feature map to obtain a second feature map, so as to extract the position information of each image object based on the category information and the position information of the first pixel point in the first feature map, and then perform panoramic segmentation based on the second feature map and the position information of each image object to obtain a panoramic segmentation image of the to-be-segmented image. On the one hand, due to the single-process end-to-end technical system, problems caused by the artificial design of the heuristic post-processing process can be effectively avoided, which helps to improve the panoramic segmentation effect. On the other hand, since no high-complexity operator such as a Transformer is needed in the panoramic segmentation process, the consumption of resources such as computing power and storage for deploying panoramic segmentation is greatly reduced. Therefore, the panoramic segmentation can be deployed in real time on an edge device. BRIEF DESCRIPTION OF DRAWINGS

[0010] Figure 1 is a flowchart of an embodiment of the panoramic segmentation method of the present application;

[0011] Figure 2a is a schematic diagram of an embodiment of a to-be-segmented image;

[0012] Figure 2b is a panoramic segmentation Figure 1 schematic diagram of an embodiment;

[0013] Figure 3 is a framework schematic diagram of an embodiment of a panoramic segmentation model;

[0014] Figure 4 is a schematic diagram of an embodiment of a convolution parameter;

[0015] Figure 5 is a flowchart of an embodiment of training a panoramic segmentation model;

[0016] Figure 6 is a framework schematic diagram of an embodiment of the panoramic segmentation device of the present application;

[0017] Figure 7 is a framework schematic diagram of an embodiment of the electronic device of the present application;

[0018] Figure 8 is a framework schematic diagram of an embodiment of the computer-readable storage medium of the present application. DETAILED DESCRIPTION

[0019] The scheme of the embodiments of the present application will be described in detail below with reference to the accompanying drawings of the specification.

[0020] In the following description, for the purposes of explanation, numerous specific details are set forth in order to thoroughly understand the application. It will be apparent, however, to one skilled in the art that the application can be practiced without some or all of these specific details.

[0021] The terms "system" and "network" are often used interchangeably herein. The term "and / or", merely describes an associated relationship between associated objects, which means that there can be three relationships, for example, A and / or B, which can represent the following three cases: A exists alone, A and B exist together, and B exists alone. In addition, the segment " / " herein generally represents that the associated objects before and after are in an "or" relationship. In addition, "multiple" herein means two or more than two.

[0022] The embodiments of the present application can effectively avoid the problems caused by the heuristic post-processing flow designed by artificial design, which helps to improve the panoramic segmentation effect. On the other hand, since no high computational complexity operators such as Transform are needed in the panoramic segmentation process, the consumption of resources such as computing power and storage for deploying panoramic segmentation is greatly reduced. Therefore, panoramic segmentation can be deployed in real time on edge devices. It should be noted that edge devices have the characteristics of low power consumption and low computing power. For example, edge devices can include but are not limited to: sweeping robots, companion robots, teaching robots, vehicle-mounted devices (such as vehicle-mounted automatic driving devices), etc., which are not limited here.

[0023] Please refer to Figure 1 , Figure 1 is a flowchart of an embodiment of the panoramic segmentation method of the present application.

[0024] Specifically, it can include the following steps:

[0025] Step S11: Extracting a first feature map of the image to be segmented.

[0026] In the embodiments of the present disclosure, the image to be segmented contains image objects of several categories, and the image objects include at least one of instances and backgrounds. That is, the image to be segmented can contain instances, can contain backgrounds, and can contain both instances and backgrounds, which are not limited here.

[0027] In an implementation scenario, the image to be segmented can include a pedestrian "A", a pedestrian "B", a pedestrian "C", a vehicle "A", a vehicle "B", a road, and a sky. The image to be segmented includes four categories of image objects, including two categories of instances (i.e., pedestrians and vehicles), and two categories of backgrounds (i.e., roads and skies). In addition, the two categories of instances involve a total of five instances (i.e., pedestrians "A", "B", and "C", and vehicles "A" and "B"). The purpose of panoramic segmentation is to segment the above image objects. Other cases can be similarly deduced and will not be listed one by one here.

[0028] In an implementation scenario, for ease of understanding, please refer to Figure 2a and Figure 2b , Figure 2a is a schematic diagram of an embodiment of the image to be segmented, Figure 2b is a schematic diagram of an embodiment of the panoramic segmentation Figure 1 is more specific, Figure 2b is Figure 2a the panoramic segmentation image obtained after panoramic segmentation. As shown in Figure 2a and 2b , Figure 2b not only different colors are used to mark different image objects, but also the categories to which the respective image objects belong are further marked. Other cases can be similarly deduced and will not be listed one by one here.

[0029] In an implementation scenario, the image to be segmented can be set according to the application scenario. For example, in the smart home scenario, the image to be segmented can be an image captured by a sweeping robot through its built-in camera; or in the autonomous driving scenario, the image to be segmented can be an image captured by a vehicle-mounted device through a vehicle-mounted camera. Other cases can be similarly deduced and will not be listed one by one here. In addition, it should be noted that the camera for capturing the image to be segmented can include but is not limited to a normal camera, a wide-angle camera, a fisheye camera, etc., without limitation.

[0030] In an implementation scenario, in order to improve the efficiency of panoramic segmentation, a panoramic segmentation model can be pre-trained, and the panoramic segmentation model can further include a feature extraction network, which can include but is not limited to a convolutional layer, etc., without limitation. For example, the feature extraction network can be designed as a residual network, etc., without limitation to the network structure of the feature extraction network. On this basis, the feature extraction network can be used to extract features of the image to be segmented to obtain a first feature map.

[0031] In one implementation scenario, different from the foregoing extraction manner, in order to further improve the accuracy of panoramic segmentation, the first feature map can contain feature information of multiple scales of the image to be segmented. Specifically, based on the image to be segmented, third feature maps of multiple scales can be extracted, and based on the third feature maps of multiple scales, the first feature map can be fused. It should be noted that the third feature maps of different scales also have different resolutions. In addition, the larger the resolution of the third feature map is, the more shallow the feature information contained in the third feature map is (such as shallow texture features, etc.), and vice versa, the smaller the resolution of the third feature map is, the more deep the feature information contained in the third feature map is (such as deep semantic features, etc.). The above-mentioned manner extracts third feature maps of multiple scales based on the image to be segmented, and fuses the first feature map based on the third feature maps of multiple scales, so that the first feature map can contain feature information of multiple scales of the image to be segmented, so that shallow features and deep features can be referred to simultaneously in the subsequent class prediction, parameter prediction and feature generation process, which helps to improve the accuracy of panoramic segmentation.

[0032] In one specific implementation scenario, the specific number of multiple scales and the resolution corresponding to each of the multiple scales are not limited here. For example, taking three scales as an example, the resolutions corresponding to the three scales can include one-fourth of the resolution of the image to be segmented, one-eighth of the resolution of the image to be segmented, and one-sixteenth of the resolution of the image to be segmented. Other cases can be similarly extended and will not be repeated here.

[0033] In one specific implementation scenario, as described previously, in order to improve the efficiency of panoramic segmentation, a panoramic segmentation model can be pre-trained, so that the panoramic segmentation image of the image to be segmented can be obtained by performing panoramic segmentation on the image to be segmented based on the panoramic segmentation model. Different from the panoramic segmentation model in the foregoing implementation manner, the panoramic segmentation model in this implementation manner can include a backbone network and a fusion network, the backbone network can include a plurality of feature extraction sub-networks connected in sequence, and the plurality of feature extraction sub-networks are respectively used to extract third feature maps of different scales, and the fusion network is used to fuse the third feature maps of multiple scales to obtain the first feature map. For example, the backbone network can include but is not limited to a convolutional neural network such as ResNet, DenseNet, etc. Taking the case where the backbone network is set to ResNet as an example, the feature extraction sub-networks at this time can be residual blocks in ResNet, and other cases can be similarly extended and will not be repeated here. The model structure with appropriate parameters can be selected according to the actual computing power and memory and other hardware resources of the edge device. Please refer to Figure 3 , Figure 3 is a framework schematic diagram of one embodiment of the panoramic segmentation model. As Figure 3As shown, each feature extraction sub-network of the backbone network can extract third feature maps of different scales, i.e., third feature maps obtained by different down-sampling rates. In addition, the fusion network can include, but is not limited to, FPN (Feature Pyramid Networks), PANet, etc., which are not limited herein. The third feature maps of multiple scales can be fused by the fusion network to obtain a feature map of a unified down-sampling rate, i.e., the first feature map. The specific process of feature extraction by the backbone network can refer to the technical details of ResNet, DenseNet, etc. The specific process of fusing third feature maps of multiple scales by the fusion network can refer to the technical details of FPN, PANet, etc., which are not described herein again.

[0034] Step S12: based on the first feature map, class information and position information of the first pixel point in the first feature map are respectively predicted, and feature generation is performed based on the first feature map to obtain a second feature map.

[0035] In the embodiments of the present disclosure, the class information of the first pixel point in the first feature map represents the likelihood that the image object to which the first pixel point belongs belongs to a plurality of categories. Exemplarily, the class information can be expressed in confidence, i.e., the class information can include the confidence that the image object to which the first pixel point belongs is respectively predicted to belong to a plurality of categories. It should be noted that the confidence specifically represents the degree of credibility, and the numerical range thereof can be positive. The higher the confidence that the image object to which the first pixel point belongs is predicted to belong to a certain category, the greater the degree of credibility that the image object to which the first pixel point belongs belongs to the category. Conversely, the lower the confidence that the image object to which the first pixel point belongs is predicted to belong to a certain category, the smaller the degree of credibility that the image object to which the first pixel point belongs belongs to the category. Alternatively, the class information can also be expressed in a probability value, i.e., the class information can include a probability value that the image object to which the first pixel point belongs is respectively predicted to belong to a plurality of categories. It should be noted that the probability value specifically represents the probability of an event, and the numerical range thereof can be 0 to 1. The greater the probability value that the image object to which the first pixel point belongs is predicted to belong to a certain category, the higher the probability that the event "the image object to which the first pixel point belongs belongs to the category" occurs. Conversely, the smaller the probability value that the image object to which the first pixel point belongs is predicted to belong to a certain category, the lower the probability that the event "the image object to which the first pixel point belongs belongs to the category" occurs. The above two methods are only two possible implementation manners for expressing the class information in actual application, and do not limit the specific manner of expressing the class information. In order to facilitate the description, the probability value is used to express the class information, and the resolution of the first feature map is taken as an example, i.e., S*S. The class information of each first pixel point in the first feature map can be represented as a tensor of S*S*C, wherein C represents the total number of a plurality of categories. Exemplarily, the first feature map can be represented as follows:Figure 2a As shown in the example of the image to be segmented, the several categories include instances of three categories of hanging pictures, potted plants, and doors, and two categories of backgrounds of walls and floors, i.e., a total of five image objects are included, and the category information of each first pixel point in the first feature map can include probability values of the image object to which the first pixel point in the first feature map belongs to the above five image objects, i.e., the category information of each first pixel point in the first feature map can be represented as a tensor of S*S*5. Other cases can be similarly deduced, and examples are not repeated here.

[0036] In one implementation scenario, as described above, in order to improve the efficiency of panoramic segmentation, a panoramic segmentation model can be pre-trained. Please continue to refer to Figure 3 The panoramic segmentation model can include a category prediction network, and the category prediction network can be used to predict category information, i.e., the first feature map can be input into the category prediction network to obtain the category information of the first pixel point in the first feature map. It should be noted that the category prediction network can include but is not limited to convolution layers, etc., and the network structure of the category prediction network is not limited here. Further, in order to facilitate real-time deployment of panoramic segmentation on edge devices, the category prediction network can be composed of several convolution layers, and the number of layers of the convolution layers is not limited here.

[0037] In the embodiments of the present disclosure, the position information of the first pixel point in the first feature map is used to perform image segmentation at the first pixel point to obtain a segmentation image of the image object to which the first pixel point belongs. In order to further facilitate real-time deployment of panoramic segmentation on edge devices, the position information of the first pixel point in the first feature map can specifically include the convolution parameter of the first pixel point. For ease of description, taking the resolution of the first feature map as S*S as an example, the position information of each first pixel point in the first feature map can be represented as a tensor of S*S*D1, where D1 represents the dimension of the convolution parameter, i.e., the position information of each first pixel point can be represented as a D1-dimensional convolution parameter. Exemplarily, D1 can be set to 5, 10, 15, etc., and the specific value of D1 is not limited here. Of course, in actual application, the position information is not limited to being represented by the convolution parameter. Exemplarily, the position information of the first pixel point can specifically include a matrix parameter of the first pixel point, and the position information of each first pixel point can be represented as a matrix parameter of M*M*D2, where M*M represents the size of the matrix parameter, and D2 represents the dimension of the matrix parameter, such as 3*3*5, 5*5*10, etc., which is not limited here. The above two methods are only two possible implementation methods for expressing position information in actual application, and do not limit the specific method actually used to express position information.

[0038] In one implementation scenario, as mentioned above, in order to improve the efficiency of panoramic segmentation, a panoramic segmentation model can be pre-trained. Please continue to refer to Figure 3 The panoramic segmentation model can include a position prediction network, and the position prediction network can be used to predict position information, that is, the first feature map can be input into the position prediction network to obtain the position information. It should be noted that the position prediction network can include but is not limited to a convolution layer, and the network structure of the position prediction network is not limited herein. Further, in order to facilitate real-time deployment of panoramic segmentation on edge devices, the position prediction network can be composed of a plurality of convolution layers, and the number of convolution layers is not limited herein.

[0039] In the embodiments of the present disclosure, the resolution of the second feature map can be the same as the resolution of the image to be segmented. For ease of description, the resolution of the image to be segmented can be denoted as H*W, where H represents the height and W represents the width. Then the second feature map can be denoted as H*W*E, where E represents the feature dimension. For example, E can be set to 5, 10, 15, etc., and the specific value of E is not limited herein. In addition, the second feature map is used for panoramic segmentation in combination with the aforementioned position information to obtain a panoramic segmentation image of the image to be segmented. For details, please refer to the relevant description below, which is not described herein.

[0040] In one implementation scenario, as mentioned above, in order to improve the efficiency of panoramic segmentation, a panoramic segmentation model can be pre-trained. Please continue to refer to Figure 3 The panoramic segmentation model can include a feature generation network, and the feature generation network can be used to perform feature generation, that is, the first feature map can be input into the feature generation network to obtain the second feature map. It should be noted that the feature generation network can include but is not limited to a convolution layer, and the network structure of the feature generation network is not limited herein. Further, in order to facilitate real-time deployment of panoramic segmentation on edge devices, the feature generation network can be composed of a plurality of convolution layers, and the number of convolution layers is not limited herein.

[0041] Step S13: Based on the class information and the position information of the first pixel point in the first feature map, the position information of each image object is extracted.

[0042] In one implementation scenario, based on the category information and the position information of the first pixel point in the first feature map, the image region of each image object in the first feature map can be determined, and based on the barycentric position of the image region of the image object, the first pixel point at the barycentric position in the first feature map can be determined as the target pixel point, and then the position information of the target pixel point can be extracted as the position information of the image object. It should be noted that the calculation method of the image barycenter (i.e. image centroid) can refer to the technical details of the image barycenter (or image centroid), which will not be repeated here. The above method can refer to the position condition of the image object in the subsequent image segmentation process by extracting the position information at the center position as the position information of the image object, which helps to improve the segmentation accuracy of each image object.

[0043] In one specific implementation scenario, non-maximum suppression can be performed based on the category information of the first pixel point in the first feature map to obtain the image region of each image object in the first feature map and the category to which each image object belongs. It should be noted that in the final panoramic segmentation map obtained by the present embodiment, the category to which each image object belongs can be marked. Please refer to the image to be segmented shown in Figure 2a and 2b , Figure 2a The panoramic segmentation map shown in Figure 2b Figure 2b ​The panoramic segmentation map shown is marked with each image object in different color blocks, and the class to which the image object belongs is also marked on the color block of each image object. In addition, in order to improve the accuracy of the image region, the candidate region of each image object in the first feature map can also be further predicted. It should be noted that each image object can correspond to at least one candidate region, and each candidate region corresponds to a prediction confidence, which represents the possibility of the existence of an image object in the candidate region. On this basis, these candidate regions can be screened through non-maximum suppression (NMS), and finally the target region of each image object in the first feature map can be obtained. For the process of region screening using non-maximum suppression, please refer to the technical details of non-maximum suppression, which will not be repeated here. At the same time, for each first pixel point in the first feature map, the class with the highest possibility can be selected as the class to which the image object to which the first pixel point belongs belongs. Based on this, the connected domain formed by the first pixel points belonging to the same class in each target region can be further regarded as the image region of the image object in the first feature map. The above method performs non-maximum suppression based on the class information of the first pixel points in the first feature map to obtain the image region of each image object in the first feature map and the class to which each image object belongs. The class to which each image object belongs is marked in the panoramic segmentation map, so through non-maximum suppression, each image object can be effectively identified, and the accuracy of subsequent extraction of convolution parameters can be improved.

[0044] In one specific implementation scenario, as described above, after obtaining the image region of each image object in the first feature map, the position information of each image object can be determined according to the center of gravity position of the image region of each image object. Please refer to Figure 4 , Figure 4 for details. Figure 4 As shown in the figure, taking an image to be segmented containing two image objects as an example, the image region of one image object has a center of gravity position at (i1, j1), and the image region of the other image object has a center of gravity position at (i2, j2). As described above, the position information of each first pixel point in the first feature map can be represented as a tensor of S*S*D1, so the position information at the position (i1, j1) in the tensor can be taken as the position information of the first image object, and the position information at the position (i2, j2) in the tensor can be taken as the position information of the second image object. Other cases can be similarly extended, which will not be repeated here.

[0045] In another implementation scenario, different from the foregoing implementation of extracting the convolution parameter, taking the example of expressing the category information by using the probability value, in the case where the hardware resources such as the computing power and the memory of the edge device are relatively abundant, the category information of the first pixel point in the first feature map can further include the probability value of the first pixel point belonging to each image object. That is, the category information can include not only the probability value of the image object to which the first pixel point belongs being several categories, but also the probability value of the first pixel point belonging to each image object. In order to distinguish the two probability values, the probability value of the image object to which the first pixel point belongs being several categories can be referred to as the first probability value, and the probability value of the first pixel point belonging to each image object can be referred to as the second probability value. On this basis, for each first pixel point in the first feature map, the image object corresponding to the maximum value of the second probability value of the first pixel point belonging to each image object can be determined as the image object to which the first pixel point belongs, so that the connected domain formed by the first pixel points belonging to the same image object can be taken as the image region of the image object in the first feature map, and at the same time, for each first pixel point in the first feature map, the category corresponding to the maximum value of the first probability value of the image object to which the first pixel point belongs being several categories can be determined as the category to which the image object to which the first pixel point belongs belongs. It should be noted that, as mentioned above, in the panoramic segmentation map finally obtained by the embodiment, the category to which each image object belongs can be marked. Further, for each image object, after obtaining the image region thereof in the first feature map, a plurality of position information respectively corresponding to each first pixel point in the image region can be extracted from the position information (i.e., the foregoing tensor with a size of S*S*D1) of each first pixel point in the first feature map, and the position information is subjected to weighted processing (for example, the foregoing convolution parameter can be subjected to weighted processing, or the foregoing matrix parameter can be subjected to weighted processing), to obtain the final position information of the image object, and the closer the position of the first pixel point to the center position of the image region, the greater the weight of the first pixel point, and vice versa. The foregoing method obtains the position information of the image object by weighting the position information respectively corresponding to each first pixel point in the image region of the image object, and the weight of the first pixel point is negatively correlated with the distance from the first pixel point to the center position, so on one hand, the position information of the image object can simultaneously refer to each first pixel point in the image region, and on the other hand, the first pixel points at different positions can be referred to to different degrees, and therefore the accuracy of the position information of the image object can be improved.

[0046] In a specific implementation scenario, a pixel distance of each first pixel point in the image region to the gravity center position thereof can be obtained, and a maximum pixel distance is filtered out. For each first pixel point in the image region, a ratio of the pixel distance corresponding to the first pixel point to the maximum pixel distance can be obtained, and a value obtained by subtracting the ratio from 1 is taken as an initial weight of the first pixel point. Finally, the initial weights of the first pixel points in the image region can be normalized to obtain the final weights of the first pixel points in the image.

[0047] Step S14: panoramic segmentation is performed based on the second feature map and the position information of the image objects, to obtain a panoramic segmentation map of the image to be segmented.

[0048] In an implementation scenario, the second feature map can be processed based on the position information of the image objects respectively to obtain segmentation images of the image objects, and the segmentation images of the image objects are fused to obtain the panoramic segmentation map of the image to be segmented. The above manner, which first processes the second feature map based on the position information of the image objects to obtain the segmentation images of the image objects, and then fuses the segmentation images of the image objects to obtain the panoramic segmentation map of the image to be segmented, is helpful for image segmentation and image fusion in the dimension of the image objects, and is helpful for refining the granularity of panoramic segmentation and further improving the accuracy of panoramic segmentation.

[0049] In a specific implementation scenario, taking the case of expressing the position information by using convolution parameters as an example, the segmentation image of an image object can be obtained by pixel-by-pixel convolution of the convolution parameters of the image object on each second pixel point in the second feature map. For details of the convolution operation, refer to the technical details of the convolution operation, which will not be described here.

[0050] In a specific implementation scenario, still taking the case of expressing the position information by using convolution parameters and the second feature map being represented as H*W*E as an example, after pixel-by-pixel convolution of the second feature map by the convolution parameters of a certain image object, an initial image of H*W can be obtained, and the pixel value of each pixel point in the initial image represents a probability value that the pixel point belongs to the image object. Based on this, the pixel value of a pixel point whose pixel value in the initial image is lower than a preset threshold (such as 0.5, 0.6, etc.) can be directly set to 0, and the pixel value of a pixel point whose pixel value in the initial image is not lower than the preset threshold can be directly set to 1, to obtain a mask image whose pixel values are represented by 0-1, as the segmentation image of the image object.

[0051] In a specific implementation scenario, taking the case that the image to be segmented contains N image objects as an example, after the above processing, N segmentation images can be finally obtained. In addition, in order to facilitate subsequent processing, each segmentation image can be a mask image represented by 0-1 as described above. For details of the specific processing process, refer to the subsequent description, which will not be described here.

[0052] In one specific implementation scenario, as mentioned above, taking the mask image of each image object as an example, which is represented by a 0-1 mask image, and as shown in Figure 2b , different image objects are marked by different color blocks in the panoramic segmentation image. The aforementioned S*S*C tensor (i.e., the class information of each first pixel point in the first feature map) can be up-sampled (e.g., interpolation operation) in advance to obtain a tensor with the same resolution as the image to be segmented, which can be denoted as W*H*C for ease of description. The value at (i, j, k) in the tensor represents the probability value of the image object to which the pixel point (i, j) in the image to be segmented belongs belonging to the kth class. On this basis, if the pixel point in the image to be segmented has a pixel value of 1 only in the segmentation image of one image object and 0 in the segmentation images of other image objects, the pixel point can be marked with the color corresponding to the image object in the panoramic segmentation image. Conversely, if the pixel point in the image to be segmented has a pixel value of 1 in the segmentation images of multiple image objects, the probability values of the image object to which the pixel point belongs belonging to various classes can be searched in the tensor W*H*C, and the class corresponding to the maximum probability value can be selected as the class to which the image object to which the pixel point belongs belongs. Finally, the class to which the pixel point belongs can be determined according to the class, and the pixel point can be marked with the color corresponding to the image object in the panoramic segmentation image. Please refer to Figure 2b , after the above implementation process, the panoramic segmentation image of the image to be segmented shown in Figure 2a can be obtained. Other cases can be similarly deduced and will not be repeated here.

[0053] In another implementation scenario, unlike the aforementioned image segmentation and image fusion operations in the dimension of image objects, in the case where the accuracy requirement of panoramic segmentation is relatively relaxed, after obtaining the second feature map and the position information of each image object, the position information of one image object can be selected to process the second feature map (e.g., pixel-by-pixel convolution of the second feature map using convolution parameters when the position information is represented by convolution parameters), to obtain the segmentation image of the image object, and based on the segmentation image of the image object, the position area of the image object can be marked on the initial image with the same resolution as the image to be segmented using the image marker (e.g., a certain specific color) corresponding to the image object. Then, the feature information corresponding to the position area is removed from the second feature map, and the position information of the next image object is selected to process the second feature map after removing the aforementioned feature information, to obtain the segmentation image of the first image object, and the position area of the image object is marked on the aforementioned initial image using the image marker corresponding to the image object. In this way, all image objects can be marked until the panoramic segmentation image of the image to be segmented is obtained.

[0054] The above scheme extracts a first feature map of a to-be-segmented image, and the to-be-segmented image contains image objects of several categories, and the image objects include at least one of instances and backgrounds. On this basis, based on the first feature map, the category information and the position information of the first pixel point in the first feature map are respectively predicted, and the feature generation is performed based on the first feature map to obtain a second feature map, so as to extract the position information of each image object based on the category information and the position information of the first pixel point in the first feature map, and then perform panoramic segmentation based on the second feature map and the position information of each image object to obtain a panoramic segmentation image of the to-be-segmented image. On the one hand, due to the single-process end-to-end technical system, problems caused by the artificially designed heuristic post-processing process can be effectively avoided, which helps to improve the panoramic segmentation effect. On the other hand, since no high-complexity operator such as Transformer is needed in the panoramic segmentation process, the consumption of resources such as computing power and storage for deploying panoramic segmentation is greatly reduced. Therefore, the panoramic segmentation can be deployed in real time on an edge device.

[0055] Please refer to Figure 5 , Figure 5 is a flowchart of an embodiment of training a panoramic segmentation model. As described in the foregoing disclosed embodiments, a panoramic segmentation image can be obtained by performing panoramic segmentation on a to-be-segmented image based on a panoramic segmentation model. Further, the panoramic segmentation model is trained based on a sample image. Similarly to the to-be-segmented image, the sample image can contain sample objects of several sample categories, and the sample image is labeled with annotation information, which includes the sample categories to which each sample pixel point in the sample image belongs, and the sample objects include at least one of instances and backgrounds. Further, the panoramic segmentation model is trained based on a category prediction loss and an image segmentation loss. Specifically, it can include the following steps:

[0056] Step S51: Extract a first sample feature map of a sample image, and based on the annotation information, obtain the sample categories to which each first sample pixel point in the first sample feature map actually belongs and the sample segmentation image of each sample object.

[0057] In one implementation scenario, the extraction process of the first sample feature map of the sample image can refer to the related description of "extracting a first feature map of a to-be-segmented image" in the foregoing disclosed embodiments, which will not be repeated here.

[0058] In an implementation scenario, the sample image can be scaled to the same resolution as the first sample feature map, so that for each first sample pixel point in the first sample feature map, a sample pixel point corresponding to the first sample pixel point in the sample image can be determined, and a sample class marked by the corresponding sample pixel point is taken as a sample class to which a sample object to which the first sample pixel point belongs actually belongs. For example, as described above, the sample image is marked with annotation information, and the annotation information includes sample classes to which sample objects to which sample pixel points in the sample image respectively belong. Taking the height and width of the sample image as H1 and W1 respectively, and taking the total number of the plurality of sample classes as C, for ease of description, the resolution of the first sample feature map can be denoted as H2*W2, and the above annotation information can be represented as a tensor of H1*W1*C. It should be noted that in the above tensor H1*W1*C, the element value at (i, j, k) represents a probability value of a sample object to which a sample pixel point (i, j) in the sample image actually belongs to the kth sample class, and based on the annotation information, if the sample pixel point (i, j) belongs to a sample object actually belonging to the kth sample class, the element value at (i, j, k) in the tensor H1*W1*C can be 1, otherwise it can be 0. On this basis, the above tensor can be scaled to the same resolution as the first sample feature map to obtain a new tensor H2*W2*C. It should be noted that in the above tensor H2*W2*C, the element value at (i, j, k) represents a probability value of the first sample pixel point (i, j) in the first sample feature map belonging to the kth sample class, and further, for the first sample pixel point (i, j), the sample class corresponding to the maximum probability value can be taken as a sample class to which a sample object to which the first sample pixel point (i, j) belongs actually belongs.

[0059] In an implementation scenario, the annotation information can further include sample objects to which sample pixel points in the sample image respectively belong. On this basis, for each sample object, if a sample pixel point belongs to the sample object, a pixel value of the sample pixel point on the sample segmentation image of the sample object is set to 1, otherwise it is set to 0. In this way, the sample segmentation image of each sample object can be obtained.

[0060] Step S52: based on the first sample feature map, a sample class to which a sample object to which each first sample pixel point in the first sample feature map respectively belongs is predicted, and prediction position information of each first sample pixel point is obtained, and feature generation is performed based on the first sample feature map to obtain a second sample feature map.

[0061] In an implementation scenario, the specific process of class prediction, parameter prediction and feature generation can be referred to the related description in the foregoing disclosed embodiments, which will not be described herein.

[0062] Step S53: based on the annotation information and the prediction position information of each first sample pixel point, prediction position information of each sample object is extracted.

[0063] Specifically, as described above, the annotation information can further include the sample objects to which each sample pixel point in the sample image respectively belongs, and then the sample objects to which each first sample pixel point in the first sample feature map respectively belongs can be obtained by referring to the implementation process of the aforementioned step “based on the annotation information, obtaining the sample categories to which the sample objects to which each first sample pixel point in the first sample feature map respectively belongs actually belong”. Specifically, the sample image can be scaled to the same resolution as the first sample feature map, so that for each first sample pixel point in the first sample feature map, the corresponding sample pixel point in the sample image corresponding to the first sample pixel point can be determined, and the sample object to which the corresponding sample pixel point belongs is taken as the sample object to which the first sample pixel point belongs. On this basis, the connected domain formed by the first sample pixel points belonging to the same sample object in the first sample feature map is taken as the sample image region of the sample object, and based on the sample barycentric position of the sample image region of the sample object, the first sample pixel point at the sample barycentric position in the first sample feature map is determined as the target sample pixel point, and the prediction position information of the target sample pixel point is taken as the prediction position information of the sample object. For details, please refer to the related description of “extracting the position information of the image object” in the aforementioned disclosed embodiments, which will not be described here.

[0064] Step S54: based on the prediction position information of each sample object, the second sample feature map is processed respectively to obtain the prediction segmentation image of each sample object.

[0065] Specifically, the implementation process of the step “based on the position information of each image object, the second feature map is processed respectively to obtain the segmentation image of each image object” in the aforementioned disclosed embodiments can be referred to. In addition, as described in the aforementioned disclosed embodiments, in the case of using the prediction convolution parameter to express the prediction position information, for each sample object, the prediction segmentation image of the sample object can be obtained by performing pixel-by-pixel convolution on each second sample pixel point in the second sample feature map based on the prediction convolution parameter of the sample object.

[0066] Step S55: based on the difference between the sample category to which the sample object actually belongs and the sample category to which the sample object is predicted to belong, a category prediction loss is obtained, and based on the difference between the sample segmentation image and the prediction segmentation image of the sample object, an image segmentation loss is obtained.

[0067] Specifically, a loss function such as a cross-entropy loss function, a focal loss, etc. can be used to measure the difference between the actual sample class to which the first sample pixel point belongs and the predicted sample class, to obtain a class prediction loss. For details of the specific measurement process, please refer to the technical details of the cross-entropy loss function, the focal loss, etc., which will not be repeated here. In addition, a loss function such as a cross-entropy loss function, a Dice loss, etc. can be used to measure the difference between the sample segmentation image of the sample object and the predicted segmentation image, to obtain an image segmentation loss. For details of the specific measurement process, please refer to the technical details of the cross-entropy loss function, the Dice loss, etc., which will not be repeated here. For ease of description, the class prediction loss can be denoted as L cls The image segmentation loss can be denoted as L mask .

[0068] Step S56: Adjust the network parameters of the panoramic segmentation model based on the class prediction loss and the image segmentation loss.

[0069] Specifically, the class prediction loss and the image segmentation loss can be weighted and summed to obtain a total loss of the panoramic segmentation model, and the network parameters of the panoramic segmentation model can be adjusted based on the total loss. For example, the network parameters of the panoramic segmentation model can be adjusted based on the total loss using an optimization method such as gradient descent. For details of the specific adjustment process, please refer to the technical details of the gradient descent optimization method, which will not be repeated here.

[0070] The above scheme extracts the first sample feature map of the sample image, and based on the annotation information, respectively obtains the sample class to which each first sample pixel point in the first sample feature map respectively belongs, and the sample segmentation image of each sample object. On this basis, based on the first sample feature map, the sample class to which each first sample pixel point respectively belongs is predicted, and the prediction position information of each first sample pixel point is obtained, and the second sample feature map is obtained by generating features based on the first sample feature map. Therefore, based on the annotation information and the prediction position information of each first sample pixel point, the prediction position information of each sample object is extracted, and the second sample feature map is processed based on the prediction position information of each sample object to obtain the prediction segmentation image of each sample object. Based on the difference between the sample class to which the first sample pixel point belongs and the prediction sample class, the class prediction loss is obtained, and based on the difference between the sample segmentation image and the prediction segmentation image, the image segmentation loss is obtained. Further, based on the class prediction loss and the image segmentation loss, the network parameters of the panoramic segmentation model are adjusted, so that through the class prediction loss and the image segmentation loss, on the one hand, the panoramic segmentation model can be constrained to accurately extract features as much as possible, and on the other hand, the panoramic segmentation model can be constrained to accurately perform class prediction as much as possible through the class prediction loss, and the panoramic segmentation model can be constrained to accurately perform parameter prediction and feature generation as much as possible through the image segmentation loss. Therefore, through joint training, the model precision of the panoramic segmentation model can be improved.

[0071] Please refer to Figure 6 , Figure 6 is a schematic diagram of an embodiment of the panoramic segmentation device 60 of the present application. The panoramic segmentation device 60 comprises a feature extraction module 61, an information prediction module 62, a feature generation module 63, an information extraction module 64, and an image segmentation module 65. The feature extraction module 61 is used to extract a first feature map of a to-be-segmented image. The to-be-segmented image contains image objects of several categories, and the image objects include at least one of instances and backgrounds. The information prediction module 62 is used to predict the category information and the position information of the first pixel point in the first feature map based on the first feature map. The feature generation module 63 is used to generate features based on the first feature map to obtain a second feature map. The information extraction module 64 is used to extract the position information of each image object based on the category information and the position information of the first pixel point in the first feature map. The image segmentation module 65 is used to perform panoramic segmentation based on the second feature map and the position information of each image object to obtain a panoramic segmentation image of the to-be-segmented image.

[0072] The scheme can effectively avoid problems caused by artificially designed heuristic post-processing, help improve panoramic segmentation effect, and greatly reduce resource consumption of deploying panoramic segmentation on edge devices.

[0073] In some disclosed embodiments, the information extraction module 64 includes a region determination sub-module configured to determine image regions of each image object in the first feature map based on the category information and the position information of the first pixel point in the first feature map; the information extraction module 64 includes a gravity center determination sub-module configured to determine the first pixel point at the gravity center position in the first feature map as a target pixel point based on the gravity center position of the image region of the image object; and the information extraction module 64 includes an information acquisition sub-module configured to extract the position information of the target pixel point as the position information of the image object.

[0074] In some disclosed embodiments, the region determination sub-module is specifically configured to perform non-maximum suppression based on the category information of the first pixel point in the first feature map to obtain the image regions of each image object in the first feature map and the categories to which each image object belongs; and the categories to which each image object belongs are marked in the panoramic segmentation map.

[0075] In some disclosed embodiments, the image segmentation module 65 includes a feature processing sub-module configured to process the second feature map based on the position information of each image object to obtain a segmented image of each image object, and the image segmentation module 65 includes an image fusion sub-module configured to fuse the segmented images of each image object to obtain a panoramic segmentation map of the image to be segmented.

[0076] In some disclosed embodiments, the position information includes convolution parameters, and the segmented image of the image object is obtained by performing pixel-by-pixel convolution on each second pixel point in the second feature map based on the convolution parameters of the image object.

[0077] In some disclosed embodiments, the panoramic segmentation map is obtained by performing panoramic segmentation on the image to be segmented based on a panoramic segmentation model, the panoramic segmentation model is trained based on a sample image, the sample image contains sample objects of a plurality of sample categories, the sample image is marked with annotation information, the annotation information includes sample categories to which each sample pixel point in the sample image belongs, the sample objects include at least one of instances and backgrounds, and the panoramic segmentation model is trained based on a combination of category prediction loss and image segmentation loss.

[0078] In some disclosed embodiments, the panoramic segmentation device 60 comprises a sample feature extraction module configured to extract a first sample feature map of a sample image, the panoramic segmentation device 60 comprises a sample information obtaining module configured to obtain, based on the annotation information, a sample class to which each first sample pixel in the first sample feature map respectively belongs and a sample segmentation image of each sample object; the panoramic segmentation device 60 comprises a sample information prediction module configured to predict, based on the first sample feature map, a sample class to which each sample object to which each first sample pixel in the first sample feature map respectively belongs is predicted to belong and prediction position information of each first sample pixel; the panoramic segmentation device 60 comprises a sample feature generation module configured to generate features based on the first sample feature map to obtain a second sample feature map; the panoramic segmentation device 60 comprises a prediction information extraction module configured to extract the prediction position information of each sample object based on the annotation information and the prediction position information of each first sample pixel; the panoramic segmentation device 60 comprises a prediction segmentation obtaining module configured to process the second sample feature map based on the prediction position information of each sample object to obtain a prediction segmentation image of each sample object; the panoramic segmentation device 60 comprises a class loss measurement module configured to obtain a class prediction loss based on a difference between the sample class to which each sample object to which each first sample pixel belongs actually belongs and the sample class to which each sample object to which each first sample pixel belongs is predicted to belong; the panoramic segmentation device 60 comprises a segmentation loss measurement module configured to obtain an image segmentation loss based on a difference between the sample segmentation image of each sample object and the prediction segmentation image of each sample object; and the panoramic segmentation device 60 comprises a network parameter adjustment module configured to adjust the network parameters of the panoramic segmentation model based on the class prediction loss and the image segmentation loss.

[0079] In some disclosed embodiments, the panoramic segmentation image is obtained based on panoramic segmentation of the to-be-segmented image by a panoramic segmentation model, the panoramic segmentation model comprises a class prediction network, a position prediction network, and a feature generation network, the class prediction network is configured to predict class information, the position prediction network is configured to predict position information, and the feature generation network is configured to perform feature generation.

[0080] In some disclosed embodiments, the first feature map contains feature information of multiple scales of the to-be-segmented image; the feature extraction module 61 comprises a multi-scale feature extraction submodule configured to extract third feature maps of multiple scales based on the to-be-segmented image; and the feature extraction module 61 comprises a multi-scale feature fusion submodule configured to fuse the first feature map based on the third feature maps of multiple scales.

[0081] In some disclosed embodiments, the panorama segmentation map is obtained by performing panorama segmentation on the image to be segmented based on a panorama segmentation model, the panorama segmentation model comprising a backbone network and a fusion network, the backbone network comprising a plurality of feature extraction sub-networks connected in sequence; wherein the plurality of feature extraction sub-networks are respectively used to extract third feature maps of different scales, and the fusion network is used to fuse the third feature maps of multiple scales to obtain a first feature map.

[0082] Referring to Figure 7 , Figure 7 is a schematic diagram of a frame of an embodiment of an electronic device 70. The electronic device 70 comprises a memory 71 and a processor 72 coupled to each other, the memory 71 stores program instructions, and the processor 72 is configured to execute the program instructions to implement the steps in any of the above panorama segmentation method embodiments. Specifically, the electronic device 70 can be an edge device, such as but not limited to a sweeping robot, a reading robot, and the like, which are not limited herein.

[0083] Specifically, the processor 72 is configured to control itself and the memory 71 to implement the steps in any of the above panorama segmentation method embodiments. The processor 72 can also be referred to as a CPU (Central Processing Unit). The processor 72 can be an integrated circuit chip having a processing capability of signals. The processor 72 can also be a general-purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field-Programmable Gate Array) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor or the like. In addition, the processor 72 can be implemented by an integrated circuit chip together.

[0084] The above scheme, on the one hand, can effectively avoid the problems caused by the artificially designed heuristic post-processing flow, and is helpful to improve the panorama segmentation effect, and on the other hand, since no high-complexity operator such as Transformer is needed in the panorama segmentation process, the deployment of panorama segmentation greatly reduces the consumption of resources such as computing power and storage. Therefore, the panorama segmentation can be deployed in real time on the edge device.

[0085] Referring to Figure 8 , Figure 8is a schematic diagram of a framework of an embodiment of the computer-readable storage medium 80 of the present application. The computer-readable storage medium 80 stores program instructions 81 capable of being executed by a processor, and the program instructions 81 are used to implement the steps in any of the panoramic segmentation method embodiments described above.

[0086] The above scheme, on the one hand, can effectively avoid the problems caused by the artificially designed heuristic post-processing flow, and is helpful to improve the panoramic segmentation effect, and on the other hand, since no high-complexity operator such as a Transformer is needed in the panoramic segmentation process, the consumption of resources such as computing power and storage for deploying panoramic segmentation is greatly reduced. Therefore, the panoramic segmentation can be deployed in real time on an edge device.

[0087] In some embodiments, the apparatus provided by the embodiments of the present disclosure has functions or includes modules that can be used to perform the methods described in the above method embodiments, and the specific implementation can refer to the description of the above method embodiments. For the sake of brevity, it will not be repeated here.

[0088] The above description of various embodiments tends to emphasize the differences between various embodiments, and the same or similar parts can be mutually referred to. For the sake of brevity, it will not be repeated here.

[0089] In several embodiments provided in the present application, it should be understood that the disclosed methods and apparatuses can be implemented in other ways. For example, the above-described apparatus implementation is only schematic, for example, the division of modules or units is only a logical function division, and actual implementation can have another division manner, for example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the shown or discussed mutual ones can be indirect coupling or communication connection through some interfaces, devices or units, and can be electrical, mechanical or other forms.

[0090] The units described as separate components can or can not be physically separate, and the components shown as units can or can not be physical units, that is, they can be located in one place, or can be distributed on a plurality of network units. Part or all of the units can be selected according to actual needs to achieve the purpose of the present embodiment scheme.

[0091] In addition, each functional unit in each embodiment of the present application can be integrated in one processing unit, or each unit can exist physically, or two or more units can be integrated in one unit. The above integrated unit can be realized in the form of hardware or in the form of a software functional unit.

[0092] The integrated unit, if implemented in the form of a software function unit and sold or used as an independent product, can be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the present application, essentially or in part, or all or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium, and includes a plurality of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor to perform all or part of the steps of the methods in the embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), magnetic disk or optical disk, and various other media that can store program codes.

[0093] If the technical solutions of the present application involve personal information, the product applying the technical solutions of the present application has been informed of the personal information processing rules before processing the personal information, and has obtained the personal independent consent. If the technical solutions of the present application involve sensitive personal information, the product applying the technical solutions of the present application has obtained the personal independent consent before processing the sensitive personal information, and at the same time meets the requirement of "explicit consent". For example, at the personal information collection device such as camera, a clear and prominent mark is set to inform that the personal information collection range has been entered, and the personal information will be collected. If the person voluntarily enters the collection range, it is regarded as agreeing to collect the personal information. Or, on the device for processing personal information, the personal information processing rules are informed by using obvious mark / information, and the personal authorization is obtained by means of pop-up information or asking the person to upload the personal information. The personal information processing rules can include personal information processor, personal information processing purpose, processing method, and personal information type, etc.

Claims

1. A panorama segmentation method, characterized in that, The method comprises: extracting a first feature map of a to-be-segmented image; wherein the to-be-segmented image contains image objects of several categories, and the image objects include at least one of instances and backgrounds; based on the first feature map, respectively predicting the category information and the position information of the first pixel points in the first feature map, and generating features based on the first feature map to obtain a second feature map; based on the category information and the position information of the first pixel points, extracting the position information of each image object; based on the second feature map and the position information of each image object, performing panoramic segmentation to obtain a panoramic segmentation map of the to-be-segmented image; wherein the step of extracting the position information of each image object based on the category information and the position information of the first pixel points comprises: determining the image regions of each image object in the first feature map based on the category information of the first pixel points; determining the first pixel points at the center of gravity positions of the image regions of the image objects in the first feature map as target pixel points, and extracting the position information of the target pixel points as the position information of the image objects; or, weighting the position information of each first pixel point in the image region of the image object to obtain the position information of the image object.

2. The method of claim 1, wherein, The step of determining the image regions of each image object in the first feature map based on the category information of the first pixel points comprises: performing non-maximum suppression based on the category information of the first pixel points to obtain the image regions of each image object in the first feature map and the categories to which each image object belongs. The panoramic segmentation map is marked with the categories to which each image object belongs.

3. The method of claim 1, wherein, The step of performing panoramic segmentation based on the second feature map and the position information of each image object to obtain the panoramic segmentation map of the to-be-segmented image comprises: processing the second feature map based on the position information of each image object to obtain a segmented image of each image object; fusing the segmented images of each image object to obtain the panoramic segmentation map of the to-be-segmented image.

4. The method of claim 3, wherein, The position information includes convolution parameters, and the segmented image of the image object is obtained by pixel-by-pixel convolution of each second pixel point in the second feature map based on the convolution parameters of the image object.

5. The method of claim 1, wherein, The panoramic segmentation map is obtained by performing panoramic segmentation on the to-be-segmented image based on a panoramic segmentation model, the panoramic segmentation model is trained based on sample images, the sample images contain sample objects of several sample categories, the sample images are marked with annotation information, the annotation information includes the sample categories to which each sample pixel point in the sample image belongs, the sample objects include at least one of instances and backgrounds, and the panoramic segmentation model is trained based on a combination of category prediction loss and image segmentation loss.

6. The method of claim 5, wherein, The training steps of the panoramic segmentation model comprise: extract a first sample feature map of the sample image, and based on the annotation information, obtain a sample class to which each first sample pixel point in the first sample feature map respectively belongs and a sample segmentation image of each sample object respectively; based on the first sample feature map, respectively predict a sample class to which each sample object to which each first sample pixel point in the first sample feature map respectively belongs is predicted to belong and prediction position information of each first sample pixel point, and perform feature generation based on the first sample feature map to obtain a second sample feature map; based on the annotation information and the prediction position information of each first sample pixel point, extract prediction position information of each sample object; based on the prediction position information of each sample object, process the second sample feature map respectively to obtain a prediction segmentation image of each sample object; based on a difference between the sample class to which each first sample pixel point belongs and a prediction sample class, obtain a class prediction loss, and based on a difference between the sample segmentation image and the prediction segmentation image of each sample object, obtain an image segmentation loss; based on the class prediction loss and the image segmentation loss, adjust network parameters of the panorama segmentation model.

7. The method of claim 1, wherein, The panorama segmentation image is obtained based on panorama segmentation of the to-be-segmented image by a panorama segmentation model, the panorama segmentation model includes a class prediction network, a position prediction network and a feature generation network, the class prediction network is used to predict the class information, the position prediction network is used to predict the position information, and the feature generation network is used to perform the feature generation.

8. The method of claim 1, wherein, The first feature map contains feature information of multiple scales of the to-be-segmented image; the first feature map of the to-be-segmented image is extracted by: based on the to-be-segmented image, a third feature map of the multiple scales is extracted; based on the third feature map of the multiple scales, the first feature map is fused.

9. The method of claim 8, wherein, The panorama segmentation image is obtained based on panorama segmentation of the to-be-segmented image by a panorama segmentation model, the panorama segmentation model includes a backbone network and a fusion network, the backbone network includes a plurality of feature extraction sub-networks connected in sequence; wherein the plurality of feature extraction sub-networks are respectively used to extract third feature maps of different scales, and the fusion network is used to fuse the third feature maps of the multiple scales to obtain the first feature map.

10. A panorama segmentation apparatus characterized by comprising: comprise: a feature extraction module, configured to extract a first feature map of a to-be-segmented image; wherein the to-be-segmented image contains image objects of a plurality of classes, and the image objects include at least one of instances and backgrounds; an information prediction module, configured to based on the first feature map, respectively predict class information and position information of a first pixel point in the first feature map; a feature generation module, configured to perform feature generation based on the first feature map to obtain a second feature map; an information extraction module, configured to based on the class information and the position information of the first pixel point, extract position information of each image object; An image segmentation module is configured to perform panoramic segmentation based on the second feature map and position information of each image object to obtain a panoramic segmentation image of the image to be segmented. The position information of each image object is extracted based on the category information and position information of the first pixel point, and includes the following steps: An image region of each image object in the first feature map is determined based on the category information of the first pixel point. A first pixel point at a gravity center position of the image region of the image object in the first feature map is determined as a target pixel point based on the gravity center position of the image region of the image object, and position information of the target pixel point is extracted as the position information of the image object; or the position information of each first pixel point in the image region of the image object is weighted to obtain the position information of the image object.

11. An electronic device, comprising: The memory and the processor are coupled to each other, the memory stores program instructions, and the processor is configured to execute the program instructions to implement the panoramic segmentation method in any one of claims 1 to 9.

12. A computer-readable storage medium, characterized in that, The memory stores program instructions capable of being executed by the processor, and the program instructions are configured to implement the panoramic segmentation method in any one of claims 1 to 9.

Images