Panorama segmentation method and device, terminal equipment and computer readable storage medium

Abstract

The application provides a panoramic segmentation method and device, terminal equipment and computer readable storage medium, the method comprises: obtaining the pyramid feature map of the initial image; according to the pyramid feature map, the entity object of the initial image and the texture object of the initial image are segmented, and the entity object segmentation map and the texture object segmentation map are obtained; the entity object segmentation map and the texture object segmentation map are spliced to obtain the segmented panoramic segmentation image. The application can extract the image features of the original image, and then obtain the pyramid-shaped feature map. According to the pyramid feature map, the texture object can be segmented, the non-countable objects contained in the image can be effectively segmented, the scene expressed in the image can be segmented, and the panoramic segmentation effect is achieved, and the image recognition and segmentation capability is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of image processing, in particular to a panoramic segmentation method and device, a terminal device and a computer readable storage medium. BACKGROUND

[0002] In the prior art, image recognition technology is usually used to recognize each object in an image to complete segmentation of each object in the image. For example, various objects such as people, trains, and airplanes in the image can be segmented out.

[0003] However, for these objects that can usually be segmented, these objects are usually countable objects. For the sky, leaves, and roads that are connected into a piece, these texture objects (uncountable objects) are usually regarded as the background in the image recognition of the prior art, and the specific uncountable objects are not segmented, so it is not possible to identify where the sky and the road are in the background in the segmented image, and thus the environment in which the countable objects are located in the image cannot be segmented. Therefore, how to simultaneously segment these solid objects and texture objects has become a technical problem to be solved in the field. SUMMARY

[0004] The present application provides a panoramic segmentation method, which can effectively segment each solid object and each texture object, and thus effectively perform panoramic segmentation on an image.

[0005] In a first aspect, the present application provides a panoramic segmentation method, which comprises:

[0006] obtaining a pyramid feature map of an initial image;

[0007] segmenting solid objects of the initial image and texture objects of the initial image according to the pyramid feature map to obtain a solid object segmentation map and a texture object segmentation map;

[0008] stitching the solid object segmentation map and the texture object segmentation map to obtain a segmented panoramic segmentation image.

[0009] In some embodiments of the present application, the step of segmenting the solid objects of the initial image and the texture objects of the initial image according to the pyramid feature map to obtain the solid object segmentation map and the texture object segmentation map comprises:

[0010] calculating a solid object convolution vector of the pyramid feature map;

[0011] segmenting the solid objects of the initial image according to the solid object convolution vector to obtain the solid object segmentation map;

[0012] calculating a texture object convolution vector of the pyramid feature image;

[0013] segmenting a texture object of the initial image according to the texture object convolution vector to obtain a texture object segmentation map.

[0014] In some embodiments of the present application, the calculating the solid object convolution vector of the pyramid feature map comprises:

[0015] calculating a plurality of object center point prediction maps of the pyramid feature map;

[0016] calculating a plurality of convolution weight prediction maps of the pyramid feature map;

[0017] calculating an encoded feature map of the pyramid feature map;

[0018] calculating the solid object convolution vector according to the plurality of object center point prediction maps, the plurality of convolution weight prediction maps and the encoded feature map.

[0019] In some embodiments of the present application, the segmenting the solid object of the initial image according to the solid object convolution vector to obtain the solid object segmentation map comprises:

[0020] calculating a plurality of solid object center points of the plurality of object center point prediction maps;

[0021] respectively taking one solid object center point of the plurality of solid object center points as a target solid object center point, screening a plurality of solid convolution weight vectors corresponding to the target solid object center point in the plurality of convolution weight prediction maps;

[0022] fusing the plurality of solid convolution weight vectors to obtain one solid weight fusion vector, until a plurality of solid weight fusion vectors are obtained;

[0023] performing dynamic convolution on the plurality of solid weight fusion vectors on the encoded feature map to obtain the solid object segmentation map.

[0024] In some embodiments of the present application, the calculating the texture object convolution vector of the pyramid feature image comprises:

[0025] calculating a plurality of convolution weight prediction maps of the pyramid feature map;

[0026] calculating an encoded feature map of the pyramid feature map;

[0027] calculating a semantic segmentation map of the pyramid feature map;

[0028] calculating the texture object convolution vector according to the plurality of convolution weight prediction maps, the semantic segmentation map and the encoded feature map.

[0029] In some embodiments of the present application, the texture object in the initial image is segmented according to the texture object convolution vector to obtain the texture object segmentation map, including:

[0030] The semantic segmentation map is screened on the plurality of convolution weight prediction maps to obtain a plurality of groups of texture convolution weight vectors corresponding to the semantic segmentation map;

[0031] Each group of texture convolution weight vectors is taken as a target group of texture convolution weight vectors, and the target group of texture convolution weight vectors is fused to obtain a texture weight fusion vector, until a plurality of texture weight fusion vectors are obtained.

[0032] The texture object segmentation map is obtained by performing dynamic convolution on the encoding feature map according to the plurality of texture weight fusion vectors.

[0033] In some embodiments of the present application, the encoding feature map of the pyramid feature map is calculated, including:

[0034] The scale feature map of each layer of the pyramid feature map is added with attention information respectively;

[0035] Each layer of the scale feature map added with attention information is fused to obtain a fusion encoding map;

[0036] The attention fusion encoding map is input into a preset encoding neural network to obtain the encoding feature map.

[0037] In some embodiments of the present application, the semantic segmentation map of the pyramid feature map is calculated, including:

[0038] The scale feature map of each layer of the pyramid feature map is added with attention information respectively;

[0039] Each layer of the scale feature map added with attention information is fused to obtain an attention fusion semantic map;

[0040] The attention fusion semantic map is input into a preset semantic segmentation neural network to obtain the semantic segmentation map.

[0041] In some embodiments of the present application, the entity object segmentation map and the texture object segmentation map are spliced to obtain a completed panoramic segmentation image, including:

[0042] The entity segmentation object in the entity object segmentation map is pasted on a preset blank image to obtain an entity object pasted image;

[0043] The texture segmentation object in the texture object segmentation graph is spliced on the entity object pasting image to obtain the segmented panoramic segmentation image.

[0044] In a second aspect, the present application further provides a panoramic segmentation device, the device comprising:

[0045] An acquisition module is configured to acquire a pyramid feature map of an initial image.

[0046] A segmentation module is configured to segment entity objects and texture objects of the initial image according to the pyramid feature map to obtain an entity object segmentation graph and a texture object segmentation graph.

[0047] A splicing module is configured to splice the entity object segmentation graph and the texture object segmentation graph to obtain a segmented panoramic segmentation image.

[0048] In some embodiments of the present application, the segmentation module is specifically configured to:

[0049] An entity object convolution vector of the pyramid feature map is calculated.

[0050] The entity objects of the initial image are segmented according to the entity object convolution vector to obtain the entity object segmentation graph.

[0051] A texture object convolution vector of the pyramid feature map is calculated.

[0052] The texture objects of the initial image are segmented according to the texture object convolution vector to obtain the texture object segmentation graph.

[0053] In some embodiments of the present application, the segmentation module is specifically further configured to:

[0054] A plurality of object center point prediction graphs of the pyramid feature map are calculated.

[0055] A plurality of convolution weight prediction graphs of the pyramid feature map are calculated.

[0056] An encoded feature map of the pyramid feature map is calculated.

[0057] The entity object convolution vector is calculated according to the plurality of object center point prediction graphs, the plurality of convolution weight prediction graphs and the encoded feature map.

[0058] In some embodiments of the present application, the segmentation module is specifically further configured to:

[0059] A plurality of entity object center points of the plurality of object center point prediction graphs are calculated.

[0060] respectively taking one of the plurality of entity object center points as a target entity object center point, screening a plurality of entity convolution weight vectors corresponding to the target entity object center point in the plurality of convolution weight prediction maps;

[0061] fusing the plurality of entity convolution weight vectors to obtain an entity weight fusion vector, until a plurality of entity weight fusion vectors are obtained;

[0062] performing dynamic convolution on the plurality of entity weight fusion vectors on the encoded feature map to obtain the entity object segmentation map.

[0063] In some embodiments of the present application, the segmentation module is specifically further used for:

[0064] calculating a plurality of convolution weight prediction maps of the pyramid feature map;

[0065] calculating an encoded feature map of the pyramid feature map;

[0066] calculating a semantic segmentation map of the pyramid feature map;

[0067] calculating the texture object convolution vector according to the plurality of convolution weight prediction maps, the semantic segmentation map and the encoded feature map.

[0068] In some embodiments of the present application, the segmentation module is specifically further used for:

[0069] screening a plurality of groups of texture convolution weight vectors corresponding to the plurality of convolution weight prediction maps from the semantic segmentation map;

[0070] respectively taking one of the plurality of groups of texture convolution weight vectors as a target group of texture convolution weight vectors, fusing the target group of texture convolution weight vectors to obtain a texture weight fusion vector, until a plurality of texture weight fusion vectors are obtained;

[0071] performing dynamic convolution on the plurality of texture weight fusion vectors on the encoded feature map to obtain the texture object segmentation map.

[0072] In some embodiments of the present application, the segmentation module is specifically further used for:

[0073] adding attention information to each layer of the scale feature map in the pyramid feature map respectively;

[0074] fusing each layer of the scale feature map to which attention information is added respectively to obtain a fusion encoded map;

[0075] inputting the attention fusion encoded map into a preset encoded neural network to obtain the encoded feature map.

[0076] In some embodiments of the present application, the segmentation module is specifically further configured to:

[0077] add attention information to the scale feature map of each layer in the pyramid feature map respectively;

[0078] perform semantic fusion on the scale feature map of each layer to which the attention information is added respectively to obtain an attention fusion semantic map;

[0079] input the attention fusion semantic map into a preset semantic segmentation neural network to obtain the semantic segmentation map.

[0080] In some embodiments of the present application, the splicing module is specifically configured to:

[0081] paste the entity segmented object in the entity object segmentation map on a preset blank image to obtain an entity object pasted image;

[0082] splice the texture segmented object in the texture object segmentation map on the entity object pasted image to obtain the segmented panoramic segmentation image.

[0083] In a third aspect, the present application also provides a terminal device, which comprises a processor, a memory, and a computer program stored in the memory and executable on the processor, and the processor executes the computer program to implement the steps in any panoramic segmentation method.

[0084] In a fourth aspect, the present application also provides a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to implement the steps in any panoramic segmentation method.

[0085] The present application can extract image features from an original image to obtain a pyramid-shaped feature map, and can effectively segment the uncountable objects contained in the image by segmenting the texture object according to the pyramid feature map, so as to segment the scene expressed in the image, thereby achieving a panoramic segmentation effect and improving the image recognition and segmentation capability. BRIEF DESCRIPTION OF DRAWINGS

[0086] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following will briefly introduce the drawings needed in the embodiment description. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can also be obtained by those skilled in the art without creative labor on the basis of these drawings.

[0087] Figure 1 is a scene schematic diagram of the panoramic segmentation system provided in the embodiments of the present application;

[0088] Figure 2 is an embodiment flow diagram of a panoramic segmentation method in the embodiments of the present application.

[0089] Figure 3 is an embodiment functional module diagram of a panoramic segmentation device in the embodiments of the present application.

[0090] Figure 4 is an embodiment structure diagram of a terminal device in the embodiments of the present application. DETAILED DESCRIPTION

[0091] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, any other embodiments obtained by a person skilled in the art without creative work fall within the scope of protection of the present application.

[0092] In the description of the present application, it should be understood that the terms "first", "second" are used only for the purpose of description, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined with "first", "second" can explicitly or implicitly include one or more of the features. In the description of the present application, the meaning of "multiple" is two or more, unless otherwise specifically limited.

[0093] In the present application, the word "exemplary" is used to mean "serving as an example, instance, or illustration." Any implementation described as "exemplary" in the present application is not necessarily to be construed as preferred or advantageous over other implementations. The following description is presented to enable any person skilled in the art to make and use the present application. In the following description, for the purpose of explanation, details are set forth. It is apparent to those skilled in the art that the present application can be practiced without using these specific details. In other instances, well-known structures and processes are not described in detail in order to avoid obscuring the description of the present application. Therefore, the present application is not intended to be limited to the embodiments shown, but is to be accorded the widest scope consistent with the principles and features disclosed herein.

[0094] The present application provides a panoramic segmentation method, device, terminal device and computer readable storage medium, which are described in detail below.

[0095] First, some concepts in the present application are introduced:

[0096] Physical object: In the present application, a physical object refers to a countable object.

[0097] Textured objects: In this application, textured objects refer to uncountable objects.

[0098] Please see Figure 1 , Figure 1 This is a scene diagram of the panoramic segmentation system provided in an embodiment of this application. The panoramic segmentation system may include a terminal device 100 and a storage device 200, and the storage device 200 may transmit data to the terminal device 100. Figure 1 The terminal device 100 can obtain the image data stored in the storage device 200 to execute the panoramic segmentation method in this application.

[0099] In this embodiment of the application, the terminal device 100 may include, but is not limited to, desktop computers, portable computers, network servers, PDAs (personal digital assistants), tablet computers, wireless terminal devices, embedded devices, etc.

[0100] In the embodiments of this application, the terminal device 100 and the storage device 200 can communicate through any communication method, including but not limited to mobile communication based on the 3rd Generation Partnership Project (3GPP), Long Term Evolution (LTE), and Worldwide Interoperability for Microwave Access (WiMAX), or computer network communication based on the TCP / IP Protocol Suite (TCP / IP) and User Datagram Protocol (UDP).

[0101] It should be noted that, Figure 1 The schematic diagram of the panoramic segmentation system shown is merely an example. The panoramic segmentation system and scenario described in this application are intended to more clearly illustrate the technical solutions of this application and do not constitute a limitation on the technical solutions provided in this application. As those skilled in the art will know, with the evolution of panoramic segmentation systems and the emergence of new business scenarios, the technical solutions provided in this application are also applicable to similar technical problems.

[0102] like Figure 2 As shown, Figure 2 This is a schematic flowchart of an embodiment of the panoramic segmentation method in this application. The panoramic segmentation method may include the following steps 201 to 203:

[0103] 201. Obtain the pyramid feature map of the initial image.

[0104] The initial image can be any kind of image, can be a photo, or even a realistic style of drawing image (non-abstract type of painting). The initial image can be a photo acquired in real time by a camera device, or can be image data stored in a storage device, which is not limited here.

[0105] The pyramid feature map is composed of multiple feature images with different resolutions, arranged in order from small to large resolution, so that these images look like a pyramid according to the size of the resolution, hence the name pyramid feature map. The pyramid feature map can be extracted by some image feature extraction model, which is not limited to the feature model here. Among them, the pyramid feature map extracted according to the feature model is a multi-dimensional feature image (multi-scale feature image), {X1, X2, X3, X4, …}, where X i i represents that the resolution of the feature map is 1 / 2 of the original image resolution i Specifically, if the resolution of the initial image is 144x144, the resolution of X1 is (144x1 / 2)x(144x1 / 2), the resolution of X2 is (144x1 / 4)x(144x1 / 4), and so on. This will not be elaborated here.

[0106] 202. According to the pyramid feature map, the entity object and the texture object of the initial image are segmented to obtain the entity object segmentation map and the texture object segmentation map.

[0107] For a picture, the picture usually includes some specific objects, which are the entity objects in this step, and also includes some background objects in the picture, which are the texture objects in this step. For entity objects, they are usually countable objects, such as apples, bicycles, chairs, etc. These objects are usually countable, and for automatic segmentation of images, the equipment or software used for segmentation can usually accurately identify these countable objects.

[0108] However, for the uncountable objects, that is, the texture objects in this step, such as the sky, the road surface, the water surface and other real but uncountable objects, these objects usually appear in the background of the photo, and the information usually displayed by these texture objects is the scene where the entity objects in the picture are located, or the application scene of the entity objects in the image. When the image is segmented by the usual equipment or software, the texture objects in the background information are usually ignored, and the specific ignoring method is usually to cover the texture objects directly with the set background color, so that the segmented image cannot reflect the specific scene information. If the pyramid feature map of the initial image is obtained, the pyramid feature map contains various dimensional feature information, so the feature information of the corresponding texture object can be obtained according to the information of different feature dimensions, so as to segment the texture image. The segmentation method here can be segmented by a pre-set segmentation model, and the specific segmentation model can be a composite model, which can be as follows.

[0109] Therefore, in order to better realize the embodiments of the present application, in an embodiment of the present application, according to the pyramid feature map, the entity objects and the texture objects of the initial image are segmented to obtain the entity object segmentation map and the texture object segmentation map, which can further include the following steps 2021-2024:

[0110] 2021, calculating the entity object convolution vector of the pyramid feature map.

[0111] Among them, the initial image can include multiple entity objects, so if the segmentation image of each entity object is to be obtained, the feature of each entity object needs to be obtained, and after the feature of each entity object is identified, each entity object can be segmented. In order to be able to identify the feature of each entity object, at this time, the pyramid feature map including each scale feature can be used to calculate the entity object convolution vector.

[0112] In order to better realize the embodiments of the present application, in an embodiment of the present application, the method for calculating the entity object convolution vector can include:

[0113] calculating a plurality of object center point prediction maps of the pyramid feature map; calculating a plurality of convolution weight prediction maps of the pyramid feature map; calculating an encoded feature map of the pyramid feature map; and calculating the entity object convolution vector according to the plurality of object center point prediction maps, the plurality of convolution weight prediction maps and the encoded feature map.

[0114] First, the plurality of object center point prediction maps of the pyramid feature map can be obtained by a pre-set center point prediction neural network. If the pyramid feature map is: {X1, X2, X3, X4, …}, at this time, the center point prediction neural network needs to be used to calculate the center point prediction map of each layer (that is, X i, respectively, to predict the object center points. Specifically, taking one layer of the pyramid feature map as an example, the center point prediction neural network includes four convolutional layers and an output layer. The convolution function, convolution kernel and other parameters of the convolutional layer can be set according to the specific situation, and details are not described herein. After one layer after another feature recognition and extraction is performed on the four convolutional layers, and after calculation is performed by the loss function in the output layer, an object center point prediction map can be obtained. After the same operation is performed on each layer of the pyramid feature map, multiple object center point prediction maps can be obtained. The loss function is Focal loss, and the specific formula is:

[0115]

[0116] wherein L center is the loss output value of the object center point prediction map, P i th represents the confidence of each pixel position prediction, Y i th represents the target value, the object center point position is 1, and other positions are 0, and N cen represents the number of object center points. It should be noted that the specific model training method can be achieved by grouping the training samples and the labels corresponding to the training samples into the model, and details are not described herein.

[0117] Secondly, the multiple convolution weight prediction maps of the pyramid feature map can also be obtained by a preset weight prediction neural network. If the pyramid feature map is still {X1, X2, X3, X4, …}, at this time, the feature weight prediction in the image also needs to be performed according to each layer (that is, X i , X2, …, ) of the pyramid feature map. Specifically, taking one layer of the pyramid feature map as an example, the weight prediction neural network also includes four convolutional layers and an output layer. The convolution function, convolution kernel and other parameters of the convolutional layer can be set according to the specific situation, and details are not described herein. After one layer after another feature recognition and extraction is performed on the four convolutional layers, and after calculation is performed by the loss function in the output layer, a convolution weight prediction map can be obtained. After the same operation is performed on each layer of the pyramid feature map, multiple convolution weight prediction maps can be obtained.

[0118] Further, the encoding feature map of the pyramid feature map can be obtained by a preset feature encoding neural network. If the pyramid feature map is still {X1, X2, X3, X4, …}, at this time, the feature images of each scale in the pyramid feature map need to be fused to obtain a fused image with a resolution of 1 / 4 of the initial image, and then the fused image is input into the feature encoding neural network. The feature encoding neural network includes three convolution layers and an output layer. The convolution function, convolution kernel and other parameters of the convolution layer can be set according to specific conditions, and details are not described here.

[0119] When the above three kinds of feature maps, the object center point prediction map, the convolution weight prediction map and the encoding feature map are obtained, the entity object convolution vector can be calculated.

[0120] Wherein, in order to make the error of calculating the entity object convolution vector smaller, attention information can be added to each layer of feature image in the pyramid feature map respectively, so as to exclude the interference of some unnecessary information. Specifically, it can include: adding attention information to the scale feature map of each layer in the pyramid feature map; fusing the scale feature map of each layer to which attention information is added respectively to obtain a fused encoding map; inputting the attention fused encoding map into a preset encoding neural network to obtain an encoding feature map.

[0121] 2022, according to the entity object convolution vector, segmenting the entity object of the initial image to obtain an entity object segmentation map.

[0122] After obtaining the three kinds of features described in the above step 2021, the segmentation of the entity object can be performed by the preset segmentation model in the above step.

[0123] In order to better realize the embodiments of the present application, in one embodiment of the present application, according to the entity object convolution vector, the entity object of the initial image is segmented to obtain an entity object segmentation map, which includes:

[0124] A plurality of entity object center points are calculated for a plurality of object center point prediction maps; one entity object center point in the plurality of entity object center points is taken as a target entity object center point, and a plurality of entity convolution weight vectors corresponding to the target entity object center point in the plurality of convolution weight prediction maps are screened; a plurality of entity convolution weight vectors are fused to obtain an entity weight fusion vector, until a plurality of entity weight fusion vectors are obtained; and the plurality of entity weight fusion vectors are dynamically convolved on the encoding feature map to obtain an entity object segmentation map.

[0125] Since there are multiple object center point prediction maps, there will be different entity object convolution vectors on different object center point prediction maps, so the center points of each entity object can be obtained according to the multiple entity object convolution vectors.

[0126] Meanwhile, since the convolution weight prediction map is also obtained from the pyramid feature map at each layer, the pyramid feature map at each layer corresponds to an object center point prediction map and a convolution weight prediction map, respectively, and the object center point prediction map and the convolution weight prediction map obtained from the same layer pyramid feature map are corresponding images. Assuming that the object center point prediction map A and the convolution weight prediction map B are corresponding images, if the object center point obtained from the center point prediction map A is obtained, the center point in the center point prediction map A can be obtained, and the relevant weight convolution in the convolution weight prediction map B at the same position can be found according to the position.

[0127] When the relevant weight convolution is found, the relevant convolution is fused to obtain an entity weight fusion vector, and when the above method is continuously operated, multiple entity weight fusion vectors are finally obtained, and the number of the multiple entity weight fusion vectors is the number of actual entity objects.

[0128] When the multiple entity weight fusion vectors are obtained, dynamic convolution can be performed in combination with the encoding feature map to obtain multiple entity object segmentation maps.

[0129] 2023, calculating a texture object convolution vector of the feature pyramid image.

[0130] In the initial image, in addition to the multiple entity objects, as described above, multiple texture objects can also be included, for example, an image of a ship sailing on the sea, so the texture objects at least include the sky and the sea surface, and therefore, in order to obtain the segmentation image of each texture object, the features of each texture object need to be obtained, and after identifying the features of each texture object, each texture object can be segmented. In order to identify the features of each texture object, the texture object convolution vector of each texture object can be calculated according to the pyramid feature map including the features of each scale.

[0131] In order to better implement the embodiments of the present application, in an embodiment of the present application, calculating a texture object convolution vector of the feature pyramid image can include:

[0132] calculating multiple convolution weight prediction maps of the pyramid feature map; calculating an encoding feature map of the pyramid feature map; calculating a semantic segmentation map of the pyramid feature map; calculating a texture object convolution vector according to the multiple convolution weight prediction maps, the semantic segmentation map and the encoding feature map.

[0133] Firstly, the manner of calculating the plurality of convolution weight prediction maps is the same as that described in the above embodiments, which will not be repeated here. Meanwhile, the manner of calculating the encoding feature map of the pyramid feature map is also the same as that described in the above embodiments, which will not be repeated here.

[0134] Secondly, the semantic segmentation map can be obtained by a preset semantic extraction neural network. If the pyramid feature map is still {X1, X2, X3, X4, …}, at this time, the feature images of each scale in the pyramid feature map need to be fused to obtain a fused image with a resolution of 1 / 4 of the initial image, and then the fused image is input into the semantic extraction neural network. The semantic extraction neural network can include a convolution layer and an output layer. The parameters of the convolution function and the convolution kernel of the convolution layer can be set according to specific conditions, which will not be repeated here. It should be noted that the cross-entropy loss function can be used in the output layer of the semantic extraction neural network. The formula of the cross-entropy loss function is as follows:

[0135]

[0136] wherein, L stuff_seg is the loss output value of the semantic segmentation map, wherein, P i st represents the predicted score vector of each pixel position, Y i st is the true class label vector of each pixel position. W represents the width of the feature map before entering the loss function, and H represents the height of the feature map before entering the loss function.

[0137] In order to reduce the error of calculating the texture object convolution vector when obtaining the semantic segmentation map, attention information can also be added to each layer of feature images in the pyramid feature map, so as to exclude the interference of some unnecessary semantic information. Specifically, it can include: adding attention information to each scale feature map of each layer in the pyramid feature map; performing semantic fusion on each scale feature map to which attention information is added to obtain an attention fusion semantic map; inputting the attention fusion semantic map into a preset semantic segmentation neural network to obtain a semantic segmentation map.

[0138] After obtaining the convolution weight prediction map, the semantic segmentation map and the encoding feature map, the texture object convolution vector can be calculated.

[0139] 2024, according to the texture object convolution vector, the texture object of the initial image is segmented to obtain a texture object segmentation map.

[0140] After obtaining the three features described in the above step 2023, the segmentation of the texture object can be performed by the preset segmentation model in the above step.

[0141] In order to better realize the embodiments of the present application, in an embodiment of the present application, the texture object in the initial image is segmented according to the texture object convolution vector to obtain a texture object segmentation map, which can include:

[0142] The semantic segmentation map is screened on a plurality of convolution weight prediction maps to obtain a plurality of groups of texture convolution weight vectors corresponding to the semantic segmentation map; one group of texture convolution weight vectors in the plurality of groups of texture convolution weight vectors is taken as a target group of texture convolution weight vectors, and the target group of texture convolution weight vectors is fused to obtain a texture weight fusion vector, until a plurality of texture weight fusion vectors are obtained; and dynamic convolution is performed on the plurality of texture weight fusion vectors on the encoding feature map to obtain the texture object segmentation map.

[0143] Here, the semantic segmentation map is screened on a plurality of convolution weight prediction maps respectively, and a plurality of groups of texture convolution weight vectors corresponding to the semantic segmentation map are screened according to the semantic information in the semantic segmentation map. Assuming that there are 2 semantic information on the semantic segmentation map, for example, including semantic information of the sky and semantic information of the sea, so that the plurality of groups of texture convolution weight vectors corresponding to the plurality of convolution weight prediction maps include a group of sky convolution weight vectors and a group of sea convolution weight vectors. At this time, each vector in the group of sky convolution weight vectors is fused, and each vector in the group of sea convolution weight vectors is fused, so as to obtain two texture weight fusion vectors. Next, dynamic convolution is performed on the encoding feature map.

[0144] Here, the texture weight fusion vector is dynamically convolved on the encoding feature map, and the entity weight fusion vector is dynamically convolved on the encoding feature map in the same way, which can be performed through a corresponding dynamic convolution model. The dynamic convolution model needs to include a Dice loss loss function, and the specific calculation formula is as follows:

[0145]

[0146] Wherein, L seg is the loss output value of the initial segmentation image dynamic convolution of the texture object segmentation image and the entity segmentation image, P j represents the jth predicted initial segmentation image, represents the initial segmentation image of the first prediction map.

[0147] It should be noted that three loss functions, ①, ② and ③, are exemplified in the foregoing, and in actual application scenarios, the training stages of the three are simultaneously performed, therefore, a total loss function for simultaneous training is also provided herein for adjusting the simultaneous training:

[0148] L overall =w centerL center w stuff_seg L stuff_seg w seg L seg ……④

[0149] wherein, L overall is the total loss output value; L center , L stuff_seg and L seg are the same as the formulas in ①, ② and ③ above; w center , w stuff_seg and w seg are loss weights, and the three loss weights are constants which can be manually adjusted according to actual conditions.

[0150] 203, splicing the entity object segmentation graph and the texture object segmentation graph to obtain a completed panoramic segmentation image.

[0151] After obtaining the entity object segmentation graph and the texture object segmentation graph according to the above steps, each entity object and each texture object is in a segmented state, if a complete segmentation image is to be finally presented, the separated entity object segmentation graph and the texture object segmentation graph can be spliced together, and a completed panoramic segmentation image can be finally obtained.

[0152] Since each entity object segmentation graph and each texture object segmentation graph is obtained by convolution calculation of each related object vector, before obtaining the entity object segmentation graph and the texture object segmentation graph, some default settings can be made to control the size (HxW, wherein H represents height and W represents width) of the finally output entity object segmentation graph and the texture object segmentation graph, so that the entity object segmentation image and the texture object segmentation image can be directly aligned and spliced. Finally, the spliced image is the completed panoramic segmentation image.

[0153] Therefore, in order to better realize the embodiments of the present application, in an embodiment of the present application, the entity object segmentation graph and the texture object segmentation graph are spliced to obtain a completed panoramic segmentation image, comprising:

[0154] The entity segmentation object in the entity object segmentation graph is pasted on a preset blank image to obtain an entity object pasted image; the texture segmentation object in the texture object segmentation graph is spliced on the entity object pasted image to obtain a completed panoramic segmentation image.

[0155] Since the "pasting" in the embodiments of the present application is not artificial pasting in the traditional sense, but the device or system pastes each segmented image. The operation of pasting by the device or system is essentially processing of data. Therefore, each position in the blank image in the embodiments of the present application can be binarized, including 0 and 1. That is, the places in the blank image that are not pasted are marked as 0, and the places that are pasted are marked as 1, so that it can be identified which places are pasted or spliced.

[0156] Therefore, the segmented entity object images can be pasted / or spliced on the blank image first, and then the remaining texture object segmentation image can be spliced with the entity object segmentation image after each entity object segmentation is pasted / or spliced. In actual situations, the number of texture images that are usually background information, such as sky, sea surface and the like, is usually small, and the number of entity objects can be large. The purpose of pasting / or splicing the entity object segmentation image first in the embodiments of the present application is that when an error occurs and the texture object cannot be completely spliced seamlessly, the change in the outline of the texture object has little effect, for example, the outline of the sky is slightly improved, which does not affect the identification of the sky. At this time, only the outline of the texture object segmentation image needs to be adjusted to achieve seamless splicing, or the texture object segmentation image can be filled to achieve seamless splicing.

[0157] In order to better implement the panoramic segmentation method in the embodiments of the present application, a panoramic segmentation device is further provided on the basis of the panoramic segmentation method, as shown in Figure 3 The device 300 includes:

[0158] The acquisition module 301 is configured to acquire a pyramid feature map of an initial image.

[0159] The segmentation module 302 is configured to segment entity objects of the initial image and texture objects of the initial image according to the pyramid feature map, to obtain an entity object segmentation image and a texture object segmentation image.

[0160] The splicing module 303 is configured to splice the entity object segmentation image and the texture object segmentation image, to obtain a segmented panoramic segmentation image.

[0161] The present application can extract image features of the original image through the acquisition module 301, and then obtain a pyramid-shaped feature map. The texture objects can be effectively segmented according to the segmentation module 302, so that the scene expressed in the image can be segmented. The segmented images can be spliced together through the splicing module 303, so as to achieve a panoramic segmentation effect, and improve the image recognition and segmentation capability.

[0162] In some embodiments of the present application, the segmentation module 302 is specifically configured to:

[0163] calculate an entity object convolution vector of the pyramid feature map;

[0164] segment an entity object of the initial image according to the entity object convolution vector to obtain an entity object segmentation map;

[0165] calculate a texture object convolution vector of the pyramid feature map;

[0166] segment a texture object of the initial image according to the texture object convolution vector to obtain a texture object segmentation map.

[0167] In some embodiments of the present application, the segmentation module 302 is specifically further configured to:

[0168] calculate a plurality of object center point prediction maps of the pyramid feature map;

[0169] calculate a plurality of convolution weight prediction maps of the pyramid feature map;

[0170] calculate an encoded feature map of the pyramid feature map;

[0171] calculate the entity object convolution vector according to the plurality of object center point prediction maps, the plurality of convolution weight prediction maps and the encoded feature map.

[0172] In some embodiments of the present application, the segmentation module 302 is specifically further configured to:

[0173] calculate a plurality of entity object center points of the plurality of object center point prediction maps;

[0174] respectively take one entity object center point of the plurality of entity object center points as a target entity object center point, and filter a plurality of entity convolution weight vectors corresponding to the target entity object center point in the plurality of convolution weight prediction maps;

[0175] fuse the plurality of entity convolution weight vectors to obtain an entity weight fusion vector, until a plurality of entity weight fusion vectors are obtained;

[0176] perform dynamic convolution on the plurality of entity weight fusion vectors on the encoded feature map to obtain the entity object segmentation map.

[0177] In some embodiments of the present application, the segmentation module 302 is specifically further configured to:

[0178] calculate a plurality of convolution weight prediction maps of the pyramid feature map;

[0179] calculate an encoded feature map of the pyramid feature map;

[0180] calculate a semantic segmentation map of the pyramid feature map;

[0181] According to the multiple convolution weight prediction maps, the semantic segmentation map and the encoded feature map, a texture object convolution vector is calculated.

[0182] In some embodiments of the present application, the segmentation module 302 is specifically further configured to:

[0183] Filtering corresponding multiple groups of texture convolution weight vectors on the multiple convolution weight prediction maps according to the semantic segmentation map;

[0184] Respectively taking one group of texture convolution weight vectors in the multiple groups of texture convolution weight vectors as a target group of texture convolution weight vectors, fusing the target group of texture convolution weight vectors to obtain a texture weight fusion vector, until multiple texture weight fusion vectors are obtained;

[0185] According to the multiple texture weight fusion vectors, performing dynamic convolution on the encoded feature map to obtain a texture object segmentation map.

[0186] In some embodiments of the present application, the segmentation module 302 is specifically further configured to:

[0187] Adding attention information to each layer of the scale feature map in the pyramid feature map respectively;

[0188] Fusing each layer of the scale feature map to which the attention information is added respectively to obtain a fusion encoding map;

[0189] Inputting the attention fusion encoding map into a preset encoding neural network to obtain an encoded feature map.

[0190] In some embodiments of the present application, the segmentation module is specifically further configured to:

[0191] Adding attention information to each layer of the scale feature map in the pyramid feature map respectively;

[0192] Performing semantic fusion on each layer of the scale feature map to which the attention information is added respectively to obtain an attention fusion semantic map;

[0193] Inputting the attention fusion semantic map into a preset semantic segmentation neural network to obtain a semantic segmentation map.

[0194] In some embodiments of the present application, the splicing module 303 is specifically configured to:

[0195] Pasting the entity segmentation object in the entity object segmentation map on a preset blank image to obtain an entity object pasted image;

[0196] Splicing the texture segmentation object in the texture object segmentation map on the entity object pasted image to obtain a completed panoramic segmentation image.

[0197] The embodiment of the present application further provides a terminal device, which comprises a processor, a memory, and a computer program stored in the memory and executable on the processor, and the processor executes the computer program to implement the steps in the panoramic segmentation method of any one of the embodiments of the present application. Wherein, the terminal device integrates any one of the panoramic segmentation methods provided by the embodiments of the present application, as shown in Figure 4 The terminal device provided by the embodiments of the present application can be a mobile phone, a tablet computer, a computer, a wearable device, or any other terminal device.

[0198] The terminal device can comprise a processor 401 with one or more processing cores, a memory 402 with one or more computer readable storage media, a power supply 403, an input unit 404, and the like. Those skilled in the art can understand that the structure of the terminal device shown in the embodiments of the present application does not constitute a limitation on the terminal device, and the terminal device can comprise more or fewer components than those shown in the figure, or some components can be combined, or different components can be arranged. Wherein: Figure 4

[0199] The processor 401 is the control center of the terminal device, and connects all parts of the terminal device through various interfaces and lines. The processor 401 executes the software programs and / or modules stored in the memory 402 and the data stored in the memory 402, processes data, and performs various functions of the terminal device, thereby overall monitoring the terminal device. Optionally, the processor 401 can comprise one or more processing cores. The processor 401 can be a central processing unit (CPU), and can also be other general-purpose processors, digital signal processors (DSPs), application specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, or the like. The general-purpose processor can be a microprocessor, or the processor can also be any conventional processor, and preferably, the processor 401 can integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, the user interface, and the application programs, and the modem processor mainly processes wireless communication. It can be understood that the modem processor can also not be integrated into the processor 401.

[0200] ​The memory 402 can be used to store software programs and modules, and the processor 401 executes various function applications and data processing by running the software programs and modules stored in the memory 402. The memory 402 can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, application programs required by at least one function (such as a sound playing function, an image playing function, etc.), and the like; and the data storage area can store data created according to the use of the terminal device, etc. In addition, the memory 402 can include a high-speed random access memory, and can also include a non-volatile memory, for example, at least one magnetic disk storage device, a flash memory device, or other volatile solid-state memory device. Accordingly, the memory 402 can also include a memory controller to provide access of the processor 401 to the memory 402.

[0201] The terminal device also includes a power supply 403 for supplying power to various components. Preferably, the power supply 403 can be logically connected to the processor 401 through a power management system, so as to realize functions such as management of charging, discharging, and power consumption management through the power management system. The power supply 403 can also include one or more than one direct current or alternating current power supply, a recharging system, a power supply fault detection circuit, a power supply converter or inverter, a power supply state indicator, and the like.

[0202] The terminal device can also include an input unit 404, which can be used to receive input digital or character information, and generate keyboard, mouse, joystick, optical or trackball signal inputs related to user settings and function controls.

[0203] Although not shown, the terminal device can also include a display unit and the like, which will not be described here. Specifically, in the present embodiment, the processor 401 in the terminal device loads executable files corresponding to the processes of one or more than one application program into the memory 402 according to the following instructions, and runs the application programs stored in the memory 402 by the processor 401, so as to realize various functions, for example:

[0204] Obtaining a pyramid feature map of an initial image;

[0205] According to the pyramid feature map, segmenting entity objects and texture objects of the initial image to obtain an entity object segmentation map and a texture object segmentation map;

[0206] Splicing the entity object segmentation map and the texture object segmentation map to obtain a segmented panoramic segmentation image.

[0207] Those skilled in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by instructions, or by controlling relevant hardware by the instructions, which can be stored in a computer readable storage medium and loaded and executed by a processor.

[0208] To this end, the embodiments of the present application provide a computer readable storage medium, which can include a read only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc. A computer program is stored on the storage medium, and the computer program is loaded by a processor to execute the steps in any of the panoramic segmentation methods provided by the embodiments of the present application. For example, the computer program loaded by the processor can execute the following steps:

[0209] obtaining a pyramid feature map of the initial image;

[0210] segmenting the solid object of the initial image and the texture object of the initial image according to the pyramid feature map to obtain a solid object segmentation map and a texture object segmentation map;

[0211] stitching the solid object segmentation map and the texture object segmentation map to obtain a segmented panoramic segmentation image.

[0212] In the above embodiments, the description of each embodiment has its own focus, and the parts not described in detail in a certain embodiment can be referred to the detailed description of other embodiments above, which will not be repeated here.

[0213] In a specific implementation, the above various units or structures can be implemented as independent entities, or can be combined as the same or several entities, and the specific implementation of the above various units or structures can be referred to the method embodiments above, which will not be repeated here.

[0214] The specific implementation of the above various operations can be referred to the embodiments above, which will not be repeated here.

[0215] The above provides a panoramic segmentation method and device, and the principle and implementation manner of the present application are described by specific examples, and the above embodiment is only used to help understand the method and core idea of the present application; meanwhile, for those skilled in the art, the specific implementation manner and application range can be changed according to the idea of the present application, and the above description should not be understood as the limitation of the present application.

Claims

1. A panoramic segmentation method, characterized in that, The method includes: Obtain the pyramid feature map of the initial image; Based on the pyramid feature map, an object center point prediction map, a convolution weight prediction map, and an encoded feature map are generated. At least one entity object center point is determined based on the object center point prediction map. For each entity object center point, multiple corresponding entity convolution weight vectors are selected from the convolution weight prediction map. The multiple entity convolution weight vectors are fused into an entity weight fusion vector. The entity weight fusion vector is used to perform dynamic convolution on the encoded feature map to generate an entity object segmentation map. Calculate the semantic segmentation map of the pyramid feature map, and segment the texture objects of the initial image based on the semantic segmentation map to obtain the texture object segmentation map; The segmentation map of the solid object is stitched together with the segmentation map of the textured object to obtain a panoramic segmentation image that has been segmented.

2. The panoramic segmentation method according to claim 1, characterized in that, For each of the entity object's center points, selecting multiple corresponding entity convolutional weight vectors from the convolutional weight prediction map, and fusing the multiple entity convolutional weight vectors into an entity weight fusion vector, includes: Each of the entity objects is taken as the center point of the target entity object, and multiple entity convolution weight vectors corresponding to the center point of the target entity object in the convolution weight prediction map are selected. By fusing the multiple entity convolution weight vectors, a single entity weight fusion vector corresponding to the center point of the target entity is obtained, until multiple entity weight fusion vectors for the center points of multiple entities are obtained.

3. The panoramic segmentation method according to claim 1, characterized in that, The step of segmenting the texture objects of the initial image based on the semantic segmentation map to obtain a texture object segmentation map includes: The semantic segmentation map is used to filter out multiple sets of corresponding texture convolution weight vectors on the convolution weight prediction map; Each set of texture convolution weight vectors is taken as the target set of texture convolution weight vectors, and the target set of texture convolution weight vectors are fused to obtain a texture weight fusion vector, until multiple texture weight fusion vectors are obtained. The texture object segmentation map is obtained by dynamically convolving the multiple texture weight fusion vectors on the encoded feature map.

4. The panoramic segmentation method according to claim 1, characterized in that, The step of generating an encoded feature map based on the pyramid feature map includes: Add attention information to the scale feature map of each layer in the pyramid feature map; The scale feature maps of each layer, to which attention information has been added, are fused to obtain a fused encoded map. The fused coding map is input into a preset coding neural network to obtain the coding feature map.

5. The panoramic segmentation method according to claim 1, characterized in that, The calculation of the semantic segmentation map of the pyramid feature map includes: Add attention information to the scale feature map of each layer in the pyramid feature map; Semantic fusion is performed on the scale feature maps of each layer, to which attention information has been added, to obtain an attention-fused semantic map. The attention fusion semantic graph is input into a preset semantic segmentation neural network to obtain the semantic segmentation graph.

6. The panoramic segmentation method according to claim 1, characterized in that, The step of stitching the entity object segmentation map with the texture object segmentation map to obtain the segmented panoramic image includes: The entity segmentation objects in the entity object segmentation image are pasted onto a preset blank image to obtain an entity object pasted image; The textured objects in the textured object segmentation image are stitched onto the physical object pasting image to obtain the segmented panoramic image.

7. A panoramic segmentation device, characterized in that, The device includes: The acquisition module is used to acquire the pyramid feature map of the initial image; The segmentation module is used to generate an object center point prediction map, a convolution weight prediction map, and an encoded feature map based on the pyramid feature map; determine at least one entity object center point according to the object center point prediction map; for each entity object center point, select multiple corresponding entity convolution weight vectors from the convolution weight prediction map; fuse the multiple entity convolution weight vectors into an entity weight fusion vector; perform dynamic convolution on the encoded feature map using the entity weight fusion vector to generate an entity object segmentation map; calculate the semantic segmentation map of the pyramid feature map; segment the texture objects of the initial image based on the semantic segmentation map to obtain a texture object segmentation map. The stitching module is used to stitch the segmented image of the physical object with the segmented image of the textured object to obtain a segmented panoramic image.

8. A terminal device, characterized in that, The terminal device includes a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the panoramic segmentation method according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that is executed by a processor to implement the steps of the panoramic segmentation method according to any one of claims 1 to 6.

Images