Panoptic segmentation method, device, electronic device and computer readable medium

Abstract

The embodiments of the present application disclose a panoramic segmentation method, device, electronic device and computer-readable medium. The embodiments of the method include: extracting features from a target image to obtain a multi-scale feature map; inputting the feature map of at least one scale in the multi-scale into a pre-trained positioning network to locate each object in the target image, the object includes an object and a background; inputting the feature map of at least one scale into a pre-trained convolution kernel generation network to obtain convolution kernels corresponding to each object in the target image; fusing the convolution kernels corresponding to the same object to obtain the target convolution kernel corresponding to each object; processing the multi-scale feature map based on the target convolution kernel to obtain a panoramic segmentation result of the target image. This implementation improves the accuracy and processing efficiency of panoramic segmentation.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of computer technology, and specifically to a panoramic segmentation method, device, electronic device, and computer-readable medium. Background Art

[0002] Panoptic Segmentation is an image segmentation technology used to segment regions of different objects in an image (including different objects in the foreground and the background) to obtain a segmented image.

[0003] In the prior art, different network branches are usually used to process foreground objects and backgrounds separately. In the process of processing foreground objects, different objects are separated by means of object detection frames, etc.; in the process of processing the background, another network branch is used for pixel-by-pixel classification. Since the foreground objects and the background are not processed under the same segmentation task, the accuracy of panoramic segmentation is low. In addition, when the foreground objects and the background are processed separately, additional predictions of the detection frame or position offset of the foreground objects are required, so the overall performance of the network is low, resulting in low efficiency of panoramic segmentation. Summary of the invention

[0004] The embodiments of the present application propose a panoramic segmentation method, device, electronic device and computer-readable medium to solve the technical problems of low accuracy and efficiency of panoramic segmentation in the prior art.

[0005] In a first aspect, an embodiment of the present application provides a panoramic segmentation method, the method comprising: extracting features from a target image to obtain a multi-scale feature map; inputting the feature map of at least one scale in the multi-scale into a pre-trained positioning network to locate each object in the target image, wherein the object includes an object and a background; inputting the feature map of at least one scale into a pre-trained convolution kernel generation network to obtain convolution kernels corresponding to each object in the target image; fusing convolution kernels corresponding to the same object to obtain a target convolution kernel corresponding to each object; processing the multi-scale feature map based on the target convolution kernel to obtain a panoramic segmentation result of the target image.

[0006] In a second aspect, an embodiment of the present application provides a panoramic segmentation device, which includes: a feature extraction unit, configured to extract features from a target image to obtain a multi-scale feature map; a positioning unit, configured to input the feature map of at least one scale in the multi-scale into a pre-trained positioning network, and locate each object in the target image, wherein the object includes an object and a background; a convolution kernel generation unit, configured to input the feature map of at least one scale into a pre-trained convolution kernel generation network, and obtain convolution kernels corresponding to each object in the target image; a fusion unit, configured to fuse convolution kernels corresponding to the same object to obtain a target convolution kernel corresponding to each object; and a processing unit, configured to process the multi-scale feature map based on the obtained target convolution kernel to obtain a panoramic segmentation result of the target image.

[0007] In a third aspect, an embodiment of the present application provides an electronic device, comprising: one or more processors; a storage device on which one or more programs are stored, and when the one or more programs are executed by the one or more processors, the one or more processors implement the method described in the first aspect.

[0008] In a fourth aspect, an embodiment of the present application provides a computer-readable medium having a computer program stored thereon, which, when executed by a processor, implements the method described in the first aspect.

[0009] The panoramic segmentation method, device, electronic device and computer-readable medium provided by the embodiments of the present application first extract features of the target image to obtain a multi-scale feature map; then respectively input the feature map of at least one scale into a pre-trained positioning network to locate each object in the target image, wherein the object includes the object and the background; and respectively input the feature map of at least one scale into a pre-trained convolution kernel generation network to obtain convolution kernels corresponding to each object in the target image; then fuse the convolution kernels corresponding to the same object to obtain the target convolution kernel corresponding to each object; and

[0010] Based on the target convolution kernel, the multi-scale feature map is processed to obtain the panoramic segmentation result of the target image. On the one hand, by simultaneously locating each object and background in the target image and obtaining the convolution kernel of each object and background through the convolution kernel generation network, the foreground object and the background can be processed simultaneously under the same segmentation task, which helps to improve the accuracy of panoramic segmentation. On the other hand, there is no need to additionally predict the detection frame or position offset of the foreground object during the panoramic segmentation process, which simplifies the data processing process, improves the network performance, and thus improves the efficiency of panoramic segmentation. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Other features, objects and advantages of the present application will become more apparent by reading the detailed description of non-limiting embodiments made with reference to the following drawings:

[0012] Figure 1 is a flow chart of an embodiment of a panoramic segmentation method according to the present application;

[0013] Figure 2 is a schematic diagram of the connection relationship of each network according to the panoramic segmentation method of the present application;

[0014] Figure 3 is a decomposed flow chart of the steps of generating a panoramic segmentation result in the panoramic segmentation method according to the present application;

[0015] Figure 4 is a structural schematic diagram of an embodiment of a panoramic segmentation device according to the present application;

[0016] Figure 5 It is a schematic diagram of the structure of a computer system for implementing an electronic device in an embodiment of the present application. DETAILED DESCRIPTION

[0017] The present application will be further described in detail below in conjunction with the accompanying drawings and embodiments. It is to be understood that the specific embodiments described herein are only used to explain the relevant invention, rather than to limit the invention. It should also be noted that, for ease of description, only the parts related to the relevant invention are shown in the accompanying drawings.

[0018] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0019] Please refer to Figure 1 , which shows a process 100 of an embodiment of a panoramic segmentation method according to the present application. The execution subject of the panoramic segmentation method can be various electronic devices or processors running in electronic devices. For example, the above-mentioned electronic devices can include but are not limited to servers, smart phones, tablet computers, laptop computers, desktop computers, and wearable devices. The panoramic segmentation method includes the following steps:

[0020] Step 101, extract features from the target image to obtain a multi-scale feature map.

[0021] In this embodiment, the execution subject of the panoramic segmentation method can use any image to be panoramically segmented as a target image, perform feature extraction on the target image, and obtain a multi-scale convolution layer. In practice, a pre-trained feature extraction network can be deployed in the above execution subject. The feature extraction network can be used to extract features from the image and generate a multi-scale feature map. The above execution subject can input the target image into the pre-trained feature extraction network to obtain a multi-scale feature map.

[0022] In some examples, the feature extraction network can be a feature extraction layer in a pre-trained panoptic segmentation network for extracting multi-scale features. The network structure of the feature extraction network can be pre-set as needed, for example, a network structure such as a Feature Pyramid Network (FPN) can be used but is not limited to.

[0023] Taking the feature pyramid network as an example, the feature pyramid network includes two network structures: bottom-up and top-down. Among them, the bottom-up network structure is the traditional convolutional neural network, which is used to extract image features. As the convolution goes deeper, the scale and resolution of the feature map gradually decrease, the detail features gradually decrease, and the semantic features gradually increase, and finally a multi-scale feature map with scales from high to low can be obtained. The top-down network structure is used to fuse the obtained feature maps in order from small to large scales to obtain the final multi-scale feature map. In the process of generating each feature map, the top-down network structure fuses features of different resolutions and different semantic strengths, thereby ensuring that the feature map of each scale has a suitable resolution and strong semantic features.

[0024] Step 102: Input feature maps of at least one scale in the multiple scales into a pre-trained positioning network to locate each object in the target image.

[0025] In this embodiment, a pre-trained positioning network may be deployed in the execution subject. The positioning network may be used to locate objects in the target image based on the feature map. The execution subject may select a feature map of at least one scale from the multi-scale feature maps, and input the selected feature maps of each scale into the positioning network one by one to locate each object in the target image.

[0026] In some examples, the localization network may be a network layer in a pre-trained panoptic segmentation network for object localization. The network structure of the localization network may be pre-set as needed, for example, including multiple (e.g., 3) convolutional layers. Figure 2 The connection relationship diagram of each network is shown. The positioning network can be used as a network branch and connected to the feature extraction network.

[0027] In this embodiment, the objects in the target image may include objects and backgrounds. As an example, if the target image contains a person, two cats (cat A and cat B), a dog, and a background, the person, cat A, cat B, dog, and background in the image can all be considered different objects, that is, there are 5 objects in total. Therefore, after a feature map of a certain scale is input into the positioning network, these 5 objects can be located separately.

[0028] Specifically, Figure 2 As shown in Fig. 1, after a feature map (remembered as Xi) of a certain scale (such as i) is input into the localization network, the localization network can perform convolution and other processing on the input feature map to obtain the localization result (remembered as Li_st) of the object (remembered as Li_st). Figure 2 The positioning results of the object position in the image) and the background (which can be recorded as Li_th) (such as Figure 2 , in Background Position).

[0029] For example, each positioning result can be represented in the form of a feature map. Among them, the positioning result of the object can be represented as a feature map of at least one channel, and the number of channels is the number of objects in the target image. For example, in the above example, the objects include people, cat A, cat B and dogs, then the positioning result of the object can be represented as a feature map of 4 channels. Similarly, the positioning result of the background can also be represented as a feature map of at least one channel, and the number of channels is the number of background types in the target image. For example, if the background in the target image includes grass and sky, then the positioning result of the background can be represented as a feature map with 2 channels. As a result, the feature map in the positioning result corresponds one-to-one to the object in the target image.

[0030] Here, each feature map in the positioning result can represent the probability distribution of an object. Based on the probability distribution, the position of the object can be known, thereby realizing the positioning of the object. It should be noted that the scale of the feature map is not limited here. For example, it can have the same scale as the feature map input to the positioning network, such as 256×256.

[0031] By simultaneously locating each object and background in the target image, it is possible to process foreground objects and background simultaneously in the same segmentation task, which helps to improve the accuracy of panoramic segmentation.

[0032] In some optional implementations of this embodiment, the execution subject may select a small-scale feature map and input it into the positioning network. Specifically, at least one feature map of a scale may be selected in order of scale from low to high. Then, each selected feature map of a scale is input into a pre-trained positioning network to locate each object in the target image.

[0033] As an example, the feature extraction network can output feature maps of 5 scales, which are recorded as P1, P2, P3, P4, and P5 in order from small to large scale. The above execution subject can input P1, P2, P3, and P4 into the positioning network one by one to obtain the positioning result corresponding to P1, the positioning result corresponding to P2, the positioning result corresponding to P3, and the positioning result corresponding to P4. Each time a feature map of a scale is input, each object of the target image is located.

[0034] Compared with using large-scale feature maps, using small-scale feature maps to locate objects and background areas can reduce the amount of data calculation, thereby improving the overall processing efficiency of the network.

[0035] Step 103: Input the feature map of at least one scale into a pre-trained convolution kernel generation network to obtain convolution kernels corresponding to the objects in the target image.

[0036] In this embodiment, a pre-trained convolution kernel generation network may also be deployed in the above execution body. The convolution kernel generation network can be used to generate convolution kernels corresponding to different objects.

[0037] As an example, the convolution kernel generation network can be a network layer used for convolution kernel generation in a pre-trained panoptic segmentation network. The network structure of the convolution kernel generation network can be pre-set as needed, for example, it can include multiple (such as 3) convolution layers. Figure 2 The connection relationship diagram of each network is shown. The convolution kernel generation network can be used as a network branch and connected to the feature extraction network.

[0038] In this embodiment, for a certain scale i selected in step 102, the above-mentioned execution subject can respectively input the feature map of the scale (i.e., Xi) output by the feature extraction network into the convolution kernel generation network to obtain the convolution kernels corresponding to each object in the target image (e.g., Figure 2 The “convolution kernel Wi” in .

[0039] As an example, the feature extraction network can output feature maps of 5 scales, which are recorded as P1, P2, P3, P4, and P5 in order from small to large scales. The above-mentioned execution subject can input P1, P2, P3, and P4 into the convolution kernel generation network one by one to obtain a group of convolution kernels W1 corresponding to P1, a group of convolution kernels W2 corresponding to P2, a group of convolution kernels W3 corresponding to P3, and a group of convolution kernels W4 corresponding to P4. Each group of convolution kernels can include the corresponding convolution kernels of each object in the target image, so there are multiple convolution kernels corresponding to the same object.

[0040] The convolution kernel generation network is used to obtain the convolution kernels of each object and the background at the same time, and the convolution kernel is used to characterize the objects in the target image (each object and the background). This enables the foreground objects and the background to be processed simultaneously under the same segmentation task, which helps to improve the accuracy of panoramic segmentation.

[0041] In some optional implementations of this embodiment, the above execution subject may also combine the position distribution feature to obtain the convolution kernel corresponding to each object in the target image. Specifically, the following steps may be performed:

[0042] In the first step, position distribution features (referred to as first position distribution features for ease of distinction) are extracted for the feature maps of each selected scale (such as P1, P2, P3, and P4 above).

[0043] Here, for the feature map of each selected scale, the first position distribution feature can be directly extracted from the feature map, such as directly extracting the position distribution feature of P1. In addition, the position distribution feature can also be extracted from the corresponding feature map output by the positioning network, such as extracting the position distribution feature from the feature map output after P1 is input into the positioning network. This is not specifically limited here.

[0044] Here, the Coordconv operation can be used to extract the position distribution features. The purpose of the Coordconv operation is to strengthen the network's processing of position information. The principle is to expand two channels based on the original number of channels to store position information. One channel is used to store the x coordinate of the feature map and normalize it to the target interval (such as [-1,1]); the other channel is used to store the y coordinate of the feature map and normalize it to the target interval (such as [-1,1]). In this way, the position information can be integrated into the convolution operation.

[0045] In the second step, the feature map of each scale and the corresponding first position distribution feature are input into the pre-trained convolution kernel generation network to obtain the convolution kernel corresponding to each object in the target image.

[0046] Here, for the feature map of each selected scale, the feature map of the scale can be directly connected (concat) with the first position distribution feature obtained for the feature map of the scale, and the connected feature map is input into the pre-trained convolution kernel generation network to obtain the convolution kernel corresponding to each object in the target image.

[0047] By fusing the feature map of each scale with its corresponding first position distribution feature, the features input to the convolution kernel generation network can contain position information, so that the position information can be fused in the convolution operation during the convolution kernel generation network processing to make the output more accurate.

[0048] Step 104, the convolution kernels corresponding to the same object are fused to obtain the target convolution kernel corresponding to each object.

[0049] In this embodiment, since there are multiple convolution kernels corresponding to the same object, the convolution kernels corresponding to the same object can be fused to obtain the target convolution kernels corresponding to each object. In this way, each object can correspond to a single convolution kernel, avoiding the use of multiple convolution kernels to repeatedly perform subsequent convolution operations, while enhancing the feature expression of a single convolution kernel. Here, a variety of fusion methods can be used to fuse the convolution kernels. For example, the convolution kernel fusion can be performed by averaging and other methods.

[0050] In some optional implementations of this embodiment, the convolution kernels may be fused according to the following steps: First, the convolution kernels corresponding to the same object are divided into the same convolution kernel group. Then, the convolution kernels in the convolution kernel group are clustered to obtain at least one cluster. Finally, the convolution kernels in the same cluster may be fused to obtain a target convolution kernel. For the convolution kernels in the same cluster, the convolution kernels may be fused by averaging or the like.

[0051] For each convolution kernel group, various clustering algorithms can be used to cluster the convolution kernels. For example, the similarity of the convolution kernels in the convolution kernel group can be determined (such as the cosine similarity calculation method can be used to calculate the similarity), and the convolution kernels with similarity greater than a preset threshold are divided into the same cluster, thereby realizing the clustering of the convolution kernel group.

[0052] By clustering the convolution kernels, the convolution kernels corresponding to each object can be further adjusted to make the convolution kernels corresponding to each object more accurate. After clustering, each cluster can correspond to one object, and different clusters correspond to different objects. By fusing the convolution kernels in the same cluster, each object can correspond to a single convolution kernel, avoiding the use of multiple convolution kernels to repeat subsequent convolution operations, while enhancing the feature expression of a single convolution kernel.

[0053] Step 105, processing the multi-scale feature map based on the target convolution kernel to obtain a panoramic segmentation result of the target image.

[0054] In this embodiment, the execution subject can process the multi-scale feature map based on the target convolution kernel (such as convolution processing) to obtain a panoramic segmentation result of the target image. Here, a feature map can be selected from the multi-scale feature map for convolution processing, or two or more scale feature maps can be fused into a feature map of the same scale, and the feature map can be convoluted, which is not specifically limited here.

[0055] In some optional implementations of this embodiment, such as Figure 3 As shown, the above execution subject can obtain the panoptic segmentation result according to the following sub-steps:

[0056] In sub-step S11, the multi-scale feature map is up-sampled to the target scale to obtain a feature map of the target scale (which can be denoted as Fh).

[0057] Here, the target scale can be set as needed, such as the maximum scale of the feature map output by the feature extraction network. That is, if the feature maps output by the feature extraction network are arranged in ascending order of scale as P1, P2, P3, P4, and P5, the target scale can be the scale of the P5 feature map. At this time, P1, P2, P3, and P4 can be upsampled to the scale of P5 to form a feature map of the target scale.

[0058] Since large-scale (i.e., high-resolution) feature maps retain more detailed features, upsampling feature maps of each scale to the maximum scale (i.e., highest resolution) helps to improve the accuracy of the encoding results.

[0059] Sub-step S12, inputting the feature map of the target scale into a pre-trained feature encoding network to obtain a target feature map (which can be denoted as Fe).

[0060] A pre-trained feature encoding network may be deployed in the execution subject. The feature encoding network may be used to encode a feature map and output another feature map. The execution subject may input the second fused feature map into the feature encoding network to obtain a target feature map.

[0061] As an example, the feature encoding network can be a network layer used for feature encoding network in a pre-trained panoptic segmentation network. The network structure of the feature encoding network can be pre-set as needed, for example, it can include multiple (such as 3) convolutional layers. Figure 2 The connection relationship diagram of each network shown in the figure shows that the feature encoding network can be used as a network branch and connected to the feature extraction network. After the feature map Fh of the target scale is input into the feature encoding network, it can output the target feature map Fe.

[0062] Optionally, before inputting into the feature extraction network, the position distribution feature (which may be referred to as the second position distribution feature) may be extracted from the feature map of the target scale. When inputting into the feature extraction network, the feature map of the target scale and the second position distribution feature may be input into the pre-trained feature encoding network to obtain the target feature map. Specifically, the feature map of the target scale and the second position distribution feature may be concatenated and then input. The method for extracting the second position distribution feature is substantially the same as the method for extracting the first position distribution feature, and will not be described in detail here.

[0063] By fusing the feature map of the target scale with the second position distribution feature, the features input into the feature encoding network can contain position information, so that during the feature encoding network processing, the position information can be fused in the convolution operation to make the output more accurate.

[0064] Sub-step S13, performing convolution processing on the target feature map based on the target convolution kernel to obtain a panoramic segmentation result of the target image.

[0065] Here, the target convolution kernel corresponding to each object can be used to perform convolution processing on the target feature map output by the feature encoding network to obtain the panoramic segmentation results corresponding to each object. Among them, the panoramic segmentation result corresponding to each object can be represented by another feature map, and the area of ​​the object in the target image can be mapped through the distribution of the eigenvalue size in the feature map. In this way, the area of ​​each object (including each object and background) in the target image can be obtained, thereby realizing the panoramic segmentation of the target image.

[0066] The method provided by the above-mentioned embodiment of the present application first inputs the target image into a pre-trained feature extraction network to obtain a multi-scale feature map; then inputs the feature map of at least one scale into a pre-trained positioning network to locate each object in the target image, wherein the object includes an object and a background; and inputs the feature map of at least one scale into a pre-trained convolution kernel generation network to obtain convolution kernels corresponding to each object in the target image; the convolution kernels corresponding to the same object are fused to obtain the target convolution kernel corresponding to each object; and the multi-scale feature map is processed based on the target convolution kernel to obtain a panoramic segmentation result of the target image. On the one hand, by simultaneously locating each object and background in the target image, and obtaining the convolution kernels of each object and background at the same time through the convolution kernel generation network, it is possible to simultaneously process foreground objects and backgrounds under the same segmentation task, which helps to improve the accuracy of panoramic segmentation. On the other hand, in the process of panoramic segmentation, there is no need to additionally predict the detection frame of the foreground object or the offset relative to the center of the object, which simplifies the data processing process, improves the network performance, and thus improves the efficiency of panoramic segmentation.

[0067] Further references Figure 4 As an implementation of the methods shown in the above figures, the present application provides an embodiment of a panoramic segmentation device, which is similar to Figure 1 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.

[0068] like Figure 4As shown, the panoramic segmentation device 400 of this embodiment includes: a feature extraction unit 401, configured to extract features from a target image to obtain a multi-scale feature map; a positioning unit 402, configured to input the feature map of at least one scale in the multi-scale into a pre-trained positioning network, and locate each object in the target image, wherein the object includes an object and a background; a convolution kernel generation unit 403, configured to input the feature map of at least one scale into a pre-trained convolution kernel generation network, and obtain convolution kernels corresponding to each object in the target image; a fusion unit 404, configured to fuse convolution kernels corresponding to the same object to obtain a target convolution kernel corresponding to each object; and a processing unit 405, configured to process the multi-scale feature map based on the obtained target convolution kernel to obtain a panoramic segmentation result of the target image.

[0069] In some optional implementations of this embodiment, the above-mentioned positioning unit 402 is further configured to: select a feature map of at least one scale in order from low to high scale; input the selected feature map of each scale into a pre-trained positioning network respectively to locate each object in the above-mentioned target image.

[0070] In some optional implementations of this embodiment, the above-mentioned convolution kernel generation unit 403 is further configured to: extract the first position distribution feature for the feature map of each scale in the above-mentioned at least one scale respectively; input the feature map of each scale and the corresponding first position distribution feature into a pre-trained convolution kernel generation network to obtain the convolution kernel corresponding to each object in the above-mentioned target image.

[0071] In some optional implementations of this embodiment, the fusion unit 404 is further configured to: divide the convolution kernels corresponding to the same object into the same convolution kernel group; cluster the convolution kernels in the convolution kernel group to obtain at least one cluster; and fuse the convolution kernels in the same cluster to obtain the target convolution kernel.

[0072] In some optional implementations of this embodiment, the fusion unit 404 is further configured to: determine the similarity of the convolution kernels in the convolution kernel group, and divide the convolution kernels whose similarity is greater than a preset threshold into the same cluster.

[0073] In some optional implementations of this embodiment, the processing unit 405 is further configured to: upsample the multi-scale feature map to the target scale to obtain a feature map of the target scale; input the feature map of the target scale into a pre-trained feature encoding network to obtain a target feature map; and perform convolution processing on the target feature map based on the target convolution kernel to obtain a panoramic segmentation result of the target image.

[0074] In some optional implementations of this embodiment, the processing unit 405 is further configured to: extract a second position distribution feature from the feature map of the target scale; input the feature map of the target scale and the second position distribution feature into a pre-trained feature encoding network to obtain a target feature map.

[0075] In some optional implementations of this embodiment, the target scale is the maximum scale among the multiple scales; and the feature encoding network includes multiple convolutional layers.

[0076] In some optional implementations of this embodiment, the above-mentioned positioning network and the above-mentioned convolution kernel generation network respectively include multiple convolution layers.

[0077] The device provided by the above embodiment of the present application first inputs the target image into a pre-trained feature extraction network to obtain a multi-scale feature map; then respectively inputs the feature map of at least one scale into a pre-trained positioning network to locate each object in the target image, wherein the object includes an object and a background; then respectively inputs the feature map of at least one scale into a pre-trained convolution kernel generation network to obtain convolution kernels corresponding to each object in the target image; then the convolution kernels corresponding to the same object are fused to obtain the target convolution kernel corresponding to each object; finally, the multi-scale feature map is processed based on the target convolution kernel to obtain a panoramic segmentation result of the target image. On the one hand, by simultaneously locating each object and background in the target image, and simultaneously obtaining the convolution kernels of each object and background through the convolution kernel generation network, it is possible to simultaneously process foreground objects and backgrounds under the same segmentation task, which helps to improve the accuracy of panoramic segmentation. On the other hand, in the process of panoramic segmentation, there is no need to additionally predict the detection frame of the foreground object or the offset relative to the center of the object, which simplifies the data processing process, improves the network performance, and thus improves the efficiency of panoramic segmentation.

[0078] Reference below Figure 5 , which shows a schematic structural diagram of an electronic device for implementing some embodiments of the present application. Figure 5 The electronic device shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0079] like Figure 5As shown, the electronic device 500 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 501, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 502 or a program loaded from a storage device 508 into a random access memory (RAM) 503. In the RAM 503, various programs and data required for the operation of the electronic device 500 are also stored. The processing device 501, the ROM 502, and the RAM 503 are connected to each other via a bus 504. An input / output (I / O) interface 505 is also connected to the bus 504.

[0080] Typically, the following devices may be connected to the I / O interface 505: an input device 506 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 507 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 508 including, for example, a disk, a hard disk, etc.; and a communication device 509. The communication device 509 may allow the electronic device 500 to communicate with other devices wirelessly or by wire to exchange data. Although Figure 5 The electronic device 500 is shown with various devices, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead. Figure 5 Each block shown in the figure may represent one device, or may represent multiple devices as required.

[0081] In particular, according to some embodiments of the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, some embodiments of the present application include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In some such embodiments, the computer program can be downloaded and installed from the network through the communication device 509, or installed from the storage device 508, or installed from the ROM 502. When the computer program is executed by the processing device 501, the above-mentioned functions defined in the method of some embodiments of the present application are executed.

[0082] It should be noted that the computer-readable medium described in some embodiments of the present application may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In some embodiments of the present application, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device. In some embodiments of the present application, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. The computer readable signal medium may also be any computer readable medium other than a computer readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0083] In some embodiments, the client and the server may communicate using any currently known or future developed network protocol such as HTTP (HyperText Transfer Protocol), and may be interconnected with any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), an internet (e.g., the Internet), and a peer-to-peer network (e.g., an ad hoc peer-to-peer network), as well as any currently known or future developed network.

[0084] The computer-readable medium may be included in the electronic device; or it may exist independently without being assembled into the electronic device. The computer-readable medium carries one or more programs. When the one or more programs are executed by the electronic device, the electronic device: extracts features from the target image to obtain a multi-scale feature map; inputs the feature map of at least one scale in the multi-scale into a pre-trained positioning network to locate each object in the target image, including the object and the background; inputs the feature map of at least one scale into a pre-trained convolution kernel generation network to obtain convolution kernels corresponding to each object in the target image; fuses the convolution kernels corresponding to the same object to obtain the target convolution kernel corresponding to each object; processes the multi-scale feature map based on the target convolution kernel to obtain a panoramic segmentation result of the target image.

[0085] Computer program code for performing the operations of some embodiments of the present application may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++; and also conventional procedural programming languages ​​such as "C" language or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, or may be connected to an external computer (e.g., through the Internet using an Internet service provider), including a local area network (LAN) or a wide area network (WAN).

[0086] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present application. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0087] The units described in some embodiments of the present application may be implemented by software or hardware. The units described may also be set in a processor, for example, it may be described as: a processor includes a first determination unit, a second determination unit, a selection unit, and a third determination unit. The names of these units do not, in some cases, constitute limitations on the units themselves.

[0088] The functions described above herein may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.

[0089] The above description is only some preferred embodiments of the present application and an explanation of the technical principles used. Those skilled in the art should understand that the scope of the invention involved in the embodiments of the present application is not limited to the technical solutions formed by the specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above invention concept. For example, the above features are replaced with (but not limited to) the technical features with similar functions disclosed in the embodiments of the present application to form a technical solution.

Claims

1. A panoptic segmentation method, characterized in that: The method comprises: Extract features from the target image to obtain a multi-scale feature map; Inputting the feature map of at least one scale in the multiple scales into a pre-trained positioning network to locate each object in the target image, wherein the object includes an object and a background; Inputting the feature map of at least one scale into a pre-trained convolution kernel generation network to obtain convolution kernels corresponding to the objects in the target image respectively; The convolution kernels corresponding to the same object are fused to obtain the target convolution kernel corresponding to each object; Processing the multi-scale feature map based on the target convolution kernel to obtain a panoramic segmentation result of the target image; The multi-scale feature map is processed based on the target convolution kernel to obtain a panoramic segmentation result of the target image, including: Upsampling the multi-scale feature map to a target scale to obtain a feature map of the target scale; Inputting the target scale feature map into a pre-trained feature encoding network to obtain a target feature map; The target feature map is convolved based on the target convolution kernel to obtain a panoramic segmentation result of the target image.

2. The method according to claim 1, characterized in that The step of inputting the feature map of at least one scale in the multiple scales into a pre-trained positioning network to locate each object in the target image comprises: Select at least one feature map of the scale in order from low to high scales; The selected feature maps of each scale are respectively input into a pre-trained localization network to locate each object in the target image.

3. The method according to claim 1, characterized in that The step of inputting the feature map of at least one scale into a pre-trained convolution kernel generation network to obtain convolution kernels corresponding to the objects in the target image respectively includes: Extracting a first position distribution feature for each feature map of the at least one scale respectively; The feature map of each scale and the corresponding first position distribution feature are input into a pre-trained convolution kernel generation network to obtain a convolution kernel corresponding to each object in the target image.

4. The method according to claim 1, characterized in that The convolution kernels corresponding to the same object are fused to obtain the target convolution kernels corresponding to each object, including: The convolution kernels corresponding to the same object are divided into the same convolution kernel group; Clustering the convolution kernels in the convolution kernel group to obtain at least one cluster; The convolution kernels in the same cluster are fused to obtain the target convolution kernel.

5. The method according to claim 4, characterized in that The clustering of the convolution kernels in the convolution kernel group to obtain at least one cluster includes: The similarity of the convolution kernels in the convolution kernel group is determined, and the convolution kernels whose similarity is greater than a preset threshold are divided into the same cluster.

6. The panoptic segmentation method according to claim 5, characterized in that: The step of inputting the target scale feature map into a pre-trained feature encoding network to obtain a target feature map includes: Extracting a second position distribution feature from the feature map of the target scale; The target scale feature map and the second position distribution feature are input into a pre-trained feature encoding network to obtain a target feature map.

7. The panoptic segmentation method according to claim 6, characterized in that: The target scale is the maximum scale among the multiple scales; and the feature encoding network includes multiple convolutional layers.

8. The panoptic segmentation method according to any one of claims 1 to 7, characterized in that: The positioning network and the convolution kernel generation network respectively include multiple convolution layers.

9. A panoramic segmentation device, characterized in that: The device comprises: A feature extraction unit is configured to extract features from a target image to obtain a multi-scale feature map; A positioning unit, configured to input the feature map of at least one scale in the multiple scales into a pre-trained positioning network, and locate each object in the target image, wherein the object includes an object and a background; A convolution kernel generation unit is configured to input the feature map of at least one scale into a pre-trained convolution kernel generation network to obtain convolution kernels corresponding to the objects in the target image respectively; A fusion unit is configured to fuse the convolution kernels corresponding to the same object to obtain a target convolution kernel corresponding to each object; A processing unit is configured to process the multi-scale feature map based on the obtained target convolution kernel to obtain a panoptic segmentation result of the target image; The multi-scale feature map is processed based on the obtained target convolution kernel to obtain a panoramic segmentation result of the target image, including: Upsampling the multi-scale feature map to a target scale to obtain a feature map of the target scale; Inputting the target scale feature map into a pre-trained feature encoding network to obtain a target feature map; The target feature map is convolved based on the target convolution kernel to obtain a panoramic segmentation result of the target image.

10. An electronic device, characterized in that: include: one or more processors; a storage device having one or more programs stored thereon, When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 8.

11. A computer readable medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 8 is implemented.

Images