Image semantic segmentation method, device, computer equipment and storage medium

Abstract

The present disclosure proposes a method, apparatus, computer equipment and storage medium for image semantic segmentation, the method comprising: acquiring an initial image, processing the initial image, obtaining multiple frames of images to be segmented, wherein the multiple frames of images to be segmented correspond to multiple spatial feature information respectively, determining pixel semantic features of the images to be segmented based on the corresponding spatial feature information, wherein the pixel semantic features are obtained by identifying a semantic segmentation model corresponding to the image to be segmented, fusing multiple pixel semantic features to obtain semantic distribution features, and processing the semantic distribution features according to the initial feature information to obtain image semantic features. Through the present disclosure, it is possible to perform feature extraction processing on an image based on multiple spatial feature information, thereby effectively improving the accuracy of the obtained image semantic features, and using multiple semantic segmentation models to identify and process the image can effectively improve the efficiency of semantic segmentation processing and the effect of image semantic segmentation processing.

Description

Technical Field

[0001] The present disclosure relates to the field of computer vision technology, and in particular to an image semantic segmentation method, apparatus, computer equipment and storage medium. Background Art

[0002] Currently, when performing image dense semantic segmentation tasks, people usually tend to adopt a semantic segmentation quality-driven design approach to build an image semantic segmentation neural network. In related technologies, neural networks with more complex computational structures are usually used to perform semantic segmentation on images.

[0003] In this way, the more complex computing structure will bring more time overhead, resulting in higher semantic reasoning delays and affecting the efficiency of semantic segmentation processing. Summary of the invention

[0004] The present disclosure aims to solve one of the technical problems in the related art at least to some extent.

[0005] To this end, the purpose of the present invention is to propose an image semantic segmentation method, device, computer equipment and storage medium, which can perform feature extraction processing on images based on multiple spatial feature information, thereby effectively improving the accuracy of the obtained image semantic features, and using multiple semantic segmentation models to perform recognition processing on images, which can effectively improve the semantic segmentation processing efficiency and effectively improve the image semantic segmentation processing effect.

[0006] The image semantic segmentation method proposed in the first aspect of the embodiment of the present disclosure includes: obtaining an initial image, wherein the initial image corresponds to initial feature information; processing the initial image to obtain multiple frames of images to be segmented, wherein the multiple frames of images to be segmented respectively correspond to multiple spatial feature information; determining pixel semantic features of the image to be segmented based on the corresponding spatial feature information, wherein the pixel semantic features are obtained by identifying a semantic segmentation model corresponding to the image to be segmented; fusing multiple pixel semantic features to obtain semantic distribution features; and processing the semantic distribution features according to the initial feature information to obtain image semantic features.

[0007] The image semantic segmentation method proposed in the first aspect of the embodiment of the present disclosure obtains an initial image, wherein the initial image corresponds to initial feature information, processes the initial image to obtain multiple frames of images to be segmented, wherein the multiple frames of images to be segmented respectively correspond to multiple spatial feature information, determines the pixel semantic features of the image to be segmented based on the corresponding spatial feature information, wherein the pixel semantic features are obtained by identifying the semantic segmentation model corresponding to the image to be segmented, fuses multiple pixel semantic features to obtain semantic distribution features, and processes the semantic distribution features according to the initial feature information to obtain image semantic features, and can perform feature extraction processing on the image based on multiple spatial feature information, thereby effectively improving the accuracy of the obtained image semantic features, and uses multiple semantic segmentation models to recognize and process the image, which can effectively improve the semantic segmentation processing efficiency and effectively improve the image semantic segmentation processing effect.

[0008] The image semantic segmentation device proposed in the second aspect of the embodiment of the present disclosure includes: an acquisition module, used to acquire an initial image, wherein the initial image corresponds to initial feature information; a first processing module, used to process the initial image to obtain multiple frames of images to be segmented, wherein the multiple frames of images to be segmented respectively correspond to multiple spatial feature information; a first determination module, used to determine the pixel semantic features of the image to be segmented based on the corresponding spatial feature information, wherein the pixel semantic features are obtained by identifying a semantic segmentation model corresponding to the image to be segmented; a fusion module, used to fuse multiple pixel semantic features to obtain semantic distribution features; and a second processing module, used to process the semantic distribution features according to the initial feature information to obtain image semantic features.

[0009] The image semantic segmentation device proposed in the second aspect of the embodiment of the present disclosure obtains an initial image, wherein the initial image corresponds to initial feature information, processes the initial image to obtain multiple frames of images to be segmented, wherein the multiple frames of images to be segmented respectively correspond to multiple spatial feature information, determines the pixel semantic features of the image to be segmented based on the corresponding spatial feature information, wherein the pixel semantic features are obtained by identifying the semantic segmentation model corresponding to the image to be segmented, fuses multiple pixel semantic features to obtain semantic distribution features, and processes the semantic distribution features according to the initial feature information to obtain image semantic features, and can perform feature extraction processing on the image based on multiple spatial feature information, thereby effectively improving the accuracy of the obtained image semantic features, and uses multiple semantic segmentation models to recognize and process the image, which can effectively improve the semantic segmentation processing efficiency and effectively improve the image semantic segmentation processing effect.

[0010] The third aspect embodiment of the present disclosure proposes a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, the image semantic segmentation method proposed in the first aspect embodiment of the present disclosure is implemented.

[0011] The fourth aspect embodiment of the present disclosure proposes a non-temporary computer-readable storage medium, on which a computer program is stored. When the program is executed by a processor, it implements the image semantic segmentation method proposed in the first aspect embodiment of the present disclosure.

[0012] The fifth aspect embodiment of the present disclosure proposes a computer program product. When the instructions in the computer program product are executed by a processor, the image semantic segmentation method proposed in the first aspect embodiment of the present disclosure is performed.

[0013] Additional aspects and advantages of the present disclosure will be given in part in the following description and in part will be obvious from the following description or learned through practice of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] The above and / or additional aspects and advantages of the present disclosure will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:

[0015] Figure 1 is a flowchart of an image semantic segmentation method proposed in an embodiment of the present disclosure;

[0016] Figure 2 is a schematic diagram of multi-scale image processing in an embodiment of the present disclosure;

[0017] Figure 3 is a schematic diagram of the structure of a multi-path semantic segmentation neural network in an embodiment of the present disclosure;

[0018] Figure 4 is a flowchart of an image semantic segmentation method proposed in another embodiment of the present disclosure;

[0019] Figure 5 is a flowchart of an image semantic segmentation method proposed in another embodiment of the present disclosure;

[0020] Figure 6 is a schematic diagram of the performance of a convolution block in an embodiment of the present disclosure;

[0021] Figure 7 is a convolution efficiency curve diagram in an embodiment of the present disclosure;

[0022] Figure 8 is a flowchart of an image semantic segmentation method proposed in another embodiment of the present disclosure;

[0023] Fig. 9is a schematic diagram of available extended parameters in an embodiment of the present disclosure;

[0024] Fig.10 is a parameter diagram of a semantic segmentation neural network in an embodiment of the present disclosure;

[0025] Fig.11 is a flowchart of an image semantic segmentation method proposed in another embodiment of the present disclosure;

[0026] Fig.12 is a schematic diagram of the macroscopic structure of a semantic segmentation neural network in an embodiment of the present disclosure;

[0027] Fig.13 is a schematic diagram of the flow of two-way interaction of information features in an embodiment of the present disclosure;

[0028] Fig.14 is a flowchart of an image semantic segmentation method proposed in another embodiment of the present disclosure;

[0029] Fig.15 is a structural schematic diagram of an image semantic segmentation device proposed in an embodiment of the present disclosure;

[0030] Fig.16 is a structural schematic diagram of an image semantic segmentation device proposed in another embodiment of the present disclosure;

[0031] Fig.17 A block diagram of an exemplary computer device suitable for implementing embodiments of the present disclosure is shown. DETAILED DESCRIPTION

[0032] Embodiments of the present disclosure are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present disclosure, and are not to be construed as limitations of the present disclosure. On the contrary, the embodiments of the present disclosure include all changes, modifications, and equivalents that fall within the spirit and connotation of the appended claims.

[0033] Figure 1 It is a flowchart of an image semantic segmentation method proposed in an embodiment of the present disclosure.

[0034] It should be noted that the executor of the image semantic segmentation method of this embodiment is an image semantic segmentation device, which can be implemented by software and / or hardware. The device can be configured in a computer device, and the computer device may include but is not limited to a terminal, a server, etc.

[0035] like Figure 1 As shown, the image semantic segmentation method includes:

[0036] S101: Acquire an initial image, wherein the initial image corresponds to initial feature information.

[0037] The image to be subjected to semantic segmentation processing may be referred to as an initial image, the initial image may be an image that has not been subjected to scaling processing, and the initial image corresponds to initial feature information.

[0038] The initial feature information refers to feature data information that can describe content data and size data of the initial image.

[0039] In the embodiments of the present disclosure, when acquiring the initial image, a data acquisition device can be pre-configured on the image semantic segmentation device, and the data acquisition device can be used to acquire images in the application scene, and the acquired images are used as the initial images. The size data information and other data information of the initial images are extracted, and the acquired size data information and other data information are used as the initial feature information.

[0040] For example, the application scenario may be an autonomous driving scenario, and the initial image may be an environmental image captured during the autonomous driving process. Alternatively, when the application scenario is to perform semantic segmentation on images in a video, an image acquisition device may be used to take a screenshot of the video, and the screenshot image may be used as the initial image. There is no limitation on this.

[0041] In other embodiments, when acquiring the initial image, a data transmission interface may also be configured on the image semantic segmentation device, and an image transmitted by other electronic devices may be received via the data transmission interface as the initial image, or the initial image may be acquired in any other possible manner, without limitation.

[0042] S102: Processing the initial image to obtain multiple frames of images to be segmented, wherein the multiple frames of images to be segmented correspond to multiple pieces of spatial feature information respectively.

[0043] The image to be segmented may refer to an image obtained by scaling the initial image according to a certain size ratio, and the image to be segmented may be an image for which semantic features are to be extracted. Multiple frames of images to be segmented correspond to multiple spatial feature information respectively.

[0044] The spatial feature information refers to data feature information that can be used to describe feature spatial resolution data of the image to be segmented.

[0045] In the embodiment of the present disclosure, when processing an initial image to obtain multiple frames of images to be segmented, multiple image resolution scale data can be pre-set, and the initial image can be enlarged and reduced according to the multiple image resolution scale data to obtain multiple processed initial images, and the multiple processed images are used as images to be segmented, and the image resolution scale data of the multiple frames of images to be segmented are used as multiple spatial feature information corresponding to the multiple frames of images to be segmented.

[0046] For example, Figure 2 As shown, Figure 2 is a schematic diagram of multi-scale image processing in an embodiment of the present disclosure. When semantic segmentation is performed on the initial image, the image may contain objects of different sizes. Objects of different sizes have different semantic segmentation recognition effects at different image resolutions. The optimal image scales that can be used when segmenting objects of different categories are different. The Mean Intersection over Union (MIoU) indicator can be used to measure the semantic segmentation recognition accuracy. Figure 2 Taking the initial image in as an example, the details of small objects (such as "pillars") can usually have better segmentation and recognition effects at a larger resolution or scale. Correspondingly, objects with large structures (such as "trucks") can be segmented and recognized at a relatively small resolution or scale. The initial image can be scaled at different resolutions and scales. For example, the initial image can be reduced by 0.25 times the resolution to obtain a processed image, and the processed image is used as the image to be segmented. The image resolution scale data of multiple frames of images to be segmented are used as multiple frames of images to be segmented to correspond to multiple spatial feature information respectively, so that the optimal image scale can be used to segment and recognize objects of different categories in the initial image, thereby significantly improving the image semantic segmentation recognition effect and processing performance.

[0047] S103: Determine pixel semantic features of the image to be segmented based on corresponding spatial feature information, wherein the pixel semantic features are identified by a semantic segmentation model corresponding to the image to be segmented.

[0048] The pixel semantic feature refers to a data feature that can describe the probability of the semantic category to which the pixel points in the image to be segmented belong.

[0049] The semantic segmentation model refers to a neural network processing model that can perform semantic segmentation tasks. The semantic segmentation model can perform feature recognition on pixels in the image to be segmented to obtain pixel semantic features corresponding to the pixel points in the image to be segmented.

[0050] After acquiring and processing the initial image to obtain multiple frames of images to be segmented, the disclosed embodiment can use the semantic segmentation model corresponding to the image to be segmented to perform recognition processing on the image to be segmented to determine the pixel semantic features of the image to be segmented based on the corresponding spatial feature information.

[0051] In the disclosed embodiment, when determining the pixel semantic features of the image to be segmented based on the corresponding spatial feature information, multiple semantic segmentation models matching the corresponding multiple spatial feature information can be constructed according to the multiple spatial feature information corresponding to multiple frames of the image to be segmented, and the multiple semantic segmentation models can be constituted into a neural network processing model containing multiple paths. Then, the images to be segmented can be respectively input into the semantic segmentation models corresponding to the images to be segmented, and the semantic segmentation model can be used to perform semantic feature extraction processing on the pixels in the image to be segmented to obtain the probability value of each pixel point output by the semantic segmentation model belonging to each semantic category, and the output result of the semantic segmentation model can be used as the pixel semantic feature of the image to be segmented based on the corresponding spatial feature information.

[0052] For example, Figure 3 As shown, Figure 3 It is a structural schematic diagram of a multi-path semantic segmentation neural network in an embodiment of the present disclosure, which can process an initial image into multiple images to be segmented with different resolutions, construct multiple different semantic segmentation models based on the corresponding spatial feature information of the images to be segmented, and form a neural network model with multiple recognition paths with multiple semantic segmentation models. Then, multiple multi-frame images to be segmented can be input into the semantic segmentation models corresponding to the multiple paths, and the semantic types of pixels in the images to be segmented can be segmented and recognized using the multiple semantic segmentation models to obtain pixel semantic features of the images to be segmented, so as to determine the pixel semantic features of the images to be segmented based on the corresponding spatial feature information.

[0053] S104: Fusing multiple pixel semantic features to obtain semantic distribution features.

[0054] Among them, the semantic distribution feature refers to a data feature that can be used to describe the distribution of semantic types of pixels in an image. The semantic distribution feature can be, for example, a semantic distribution score map, which can be a probability score map of each pixel in the image belonging to each semantic category. By processing the semantic distribution score map, the recognition result of semantic segmentation of the image can be obtained.

[0055] After determining the pixel semantic features of the image to be segmented based on the corresponding spatial feature information, the embodiment of the present disclosure may fuse multiple pixel semantic features to obtain a semantic distribution feature.

[0056] In the disclosed embodiment, when fusing multiple pixel semantic features to obtain semantic distribution features, the multiple pixel semantic features identified from multiple frames of images to be segmented can be subjected to feature fusion processing, the multiple pixel semantic features can be input into a feature fusion model, and the feature fusion model can be used to fuse the multiple pixel semantic features according to corresponding weight information to obtain a score value of the probability that each pixel point in the image belongs to each semantic category, the multiple score values ​​are constructed into a semantic distribution score map, and the obtained semantic distribution score map after fusion processing is used as the semantic distribution feature.

[0057] S105: Processing the semantic distribution features according to the initial feature information to obtain image semantic features.

[0058] Among them, the image semantic feature refers to the data feature of the semantic segmentation result to which the pixel points in the image belong. The image semantic feature can describe the semantic category to which each pixel point in the image belongs, and can be obtained by processing the semantic distribution feature according to the initial feature information.

[0059] In the disclosed embodiments, when the semantic distribution features are processed according to the initial feature information to obtain the image semantic features, feature recovery processing can be performed on the semantic distribution features according to the resolution data information and size data information of the initial image in the initial feature information, and bilinear interpolation processing can be performed on the semantic distribution features according to the initial feature information to obtain a semantic distribution score map of the probability that each pixel in the processed image belongs to each semantic category, and then from the semantic distribution score map, among the multiple semantic category score values ​​to which each pixel belongs, the semantic category with the highest score corresponding to the pixel is selected as the semantic classification result corresponding to the pixel, and the semantic classification result of the pixel in the image is used as the image semantic feature.

[0060] In this embodiment, an initial image is obtained, wherein the initial image corresponds to initial feature information, and the initial image is processed to obtain multiple frames of images to be segmented, wherein the multiple frames of images to be segmented respectively correspond to multiple spatial feature information, and pixel semantic features of the image to be segmented based on the corresponding spatial feature information are determined, wherein the pixel semantic features are obtained by identifying a semantic segmentation model corresponding to the image to be segmented, and multiple pixel semantic features are fused to obtain semantic distribution features, and the semantic distribution features are processed according to the initial feature information to obtain image semantic features. Feature extraction processing can be performed on the image based on multiple spatial feature information, thereby effectively improving the accuracy of the obtained image semantic features, and using multiple semantic segmentation models to recognize and process the image can effectively improve the efficiency of semantic segmentation processing and effectively improve the image semantic segmentation processing effect.

[0061] Figure 4 It is a flowchart of an image semantic segmentation method proposed in an embodiment of the present disclosure.

[0062] like Figure 4 As shown, the image semantic segmentation method includes:

[0063] S401: Acquire an initial image, wherein the initial image corresponds to initial feature information.

[0064] S402: Processing the initial image to obtain multiple frames of images to be segmented, wherein the multiple frames of images to be segmented correspond to multiple pieces of spatial feature information respectively.

[0065] The description of S401 - S402 can be referred to the above embodiment for examples, and will not be repeated here.

[0066] S403: Determine pixel semantic features of the image to be segmented based on corresponding spatial feature information, wherein the pixel semantic features are identified by a semantic segmentation model corresponding to the image to be segmented.

[0067] Optionally, in some embodiments, the semantic segmentation model includes: a plurality of cascaded convolution blocks, some of the cascaded convolution blocks constitute a convolution set, and the convolution block includes: a feature recognition channel, wherein the convolution block in the convolution set is used to perform feature recognition tasks on the image to be segmented belonging to the semantic segmentation model based on the feature recognition channel, and feature recognition is performed on the segmented image through the semantic segmentation model composed of the plurality of cascaded convolution blocks, thereby effectively improving the recognition processing effect of feature recognition on the image to be segmented.

[0068] Among them, the convolution block is the basic computing unit in the convolutional neural network, which can be used to construct a semantic segmentation neural network processing model to perform semantic segmentation and recognition processing on images. Multiple convolution blocks can be cascaded to form a semantic segmentation model, and some cascaded convolution blocks in multiple convolution blocks constitute a convolution set.

[0069] Among them, the feature recognition channel refers to the processing unit in the neural network that can perform the feature recognition task on the image.

[0070] In the disclosed embodiment, multiple convolution blocks can be composed of multiple feature recognition channels, and a semantic segmentation model can be formed by cascading multiple convolution blocks. Some of the cascaded convolution blocks can constitute a convolution set, and the convolution blocks in the convolution set can be used to perform feature recognition tasks on the image to be segmented belonging to the semantic segmentation model based on the feature recognition channels.

[0071] Optionally, in some embodiments, the number of semantic segmentation models is dynamically adjustable, and / or the number of convolution blocks in a convolution set is dynamically adjustable, and / or the number of feature recognition channels is dynamically adjustable, so that the actual needs of image semantic segmentation can be met by dynamically adjusting the semantic segmentation model, the number of convolution blocks in a convolution set, and the number of feature recognition channels, thereby improving the flexibility of image semantic segmentation processing and effectively improving the image semantic segmentation processing performance.

[0072] In the disclosed embodiments, a convolution block can be composed of a certain number of feature recognition channels, a convolution set can be formed by cascading a certain number of convolution blocks, and an image semantic segmentation neural network can be composed of a certain number of semantic segmentation models, which are used to perform image semantic segmentation processing tasks, wherein the number of semantic segmentation models is dynamically adjustable, and / or the number of convolution blocks in the convolution set is dynamically adjustable, and / or the number of feature recognition channels is dynamically adjustable. That is to say, when the semantic segmentation processing task is performed on the image using the semantic segmentation neural network processing model, the number of semantic segmentation models in the semantic segmentation neural network processing model, the number of convolution blocks in the convolution set, and the number of feature recognition channels can be dynamically adjusted, the quantity values ​​of the multiple elements can be adjusted at the same time, or according to the actual needs of image semantic segmentation, any several elements among the number of semantic segmentation models, the number of convolution blocks in the convolution set, and the number of feature recognition channels can be selected to adjust the quantity in any way that meets the needs, and the quantity can also be dynamically adjusted during the image semantic segmentation process, and there is no restriction on this.

[0073] S404: Determine a plurality of reference interaction information corresponding to the plurality of semantic segmentation models, wherein the reference interaction information is used to describe a reference weight of a corresponding pixel semantic feature.

[0074] The reference weight refers to the weight data information corresponding to the pixel semantic features of multiple images to be segmented.

[0075] The reference interaction information refers to data information that can be used to describe the weights corresponding to the pixel semantic features of multiple images to be segmented.

[0076] In the embodiment of the present disclosure, when determining multiple reference interaction information corresponding to multiple semantic segmentation models, multiple images to be segmented corresponding to the multiple semantic segmentation models can be obtained respectively, and weight data information of the semantic segmentation model to which the image to be segmented belongs can be determined according to the spatial feature information corresponding to the image to be segmented, and the acquired weight data information can be used as multiple reference interaction information corresponding to the multiple semantic segmentation models.

[0077] S405: Fusing multiple pixel semantic features according to the reference interaction information to obtain a target semantic feature.

[0078] The target semantic feature refers to a semantic data feature obtained after fusing multiple pixel semantic features according to reference interaction information. The target semantic feature may be a semantic category probability feature of each pixel in the image after the fusion process.

[0079] After determining the pixel semantic features of the image to be segmented based on the corresponding spatial feature information and determining multiple reference interaction information corresponding to multiple semantic segmentation models, the embodiment of the present disclosure can fuse the multiple pixel semantic features according to the reference interaction information to obtain the target semantic features.

[0080] In the embodiment of the present disclosure, when multiple pixel semantic features are fused according to reference interaction information to obtain target semantic features, the multiple pixel semantic features can be input into a feature fusion model, and the feature fusion model can be used to perform weighted fusion on the multiple pixel semantic features according to reference weight information corresponding to the multiple pixel semantic features described in the reference interaction information to obtain fused data features. The fused data features can include semantic category probability features of each pixel in the image, and the fused data features can be used as target semantic features.

[0081] S406: Obtain semantic distribution features according to the target semantic features.

[0082] In the embodiment of the present disclosure, after fusing multiple pixel semantic features according to the reference interaction information to obtain the target semantic feature, the semantic distribution feature is obtained according to the target semantic feature.

[0083] In the disclosed embodiment, when obtaining the semantic distribution feature based on the target semantic feature, the semantic category probability feature of each pixel in the image described in the target semantic feature can be statistically processed to obtain the probability score value that each pixel in the image belongs to each semantic category, and the probability score values ​​are plotted into a semantic distribution score map, and the obtained semantic distribution score map is used as the semantic distribution feature.

[0084] In this embodiment, by determining multiple reference interaction information corresponding to multiple semantic segmentation models, multiple pixel semantic features are fused according to the reference interaction information to obtain target semantic features, and semantic distribution features are obtained based on the target semantic features. This allows weighted fusion processing to be performed on the semantic features identified by multiple semantic segmentation models to ensure the accuracy of the acquired semantic features and effectively ensure the quality of image semantic segmentation processing.

[0085] S407: Perform bilinear upsampling processing on the semantic distribution features according to the initial feature information to obtain image semantic features.

[0086] After obtaining the semantic distribution feature according to the target semantic feature, the embodiment of the present disclosure may process the semantic distribution feature according to the initial feature information to obtain the image semantic feature.

[0087] In an embodiment of the present disclosure, when the semantic distribution features are processed according to the initial feature information to obtain the image semantic features, the semantic distribution features can be bilinearly upsampled according to the initial feature information to obtain the image semantic features, and the semantic distribution features can be bilinearly upsampled according to the resolution data information described in the initial information features to restore the data features identified from the image to be segmented obtained after scaling the initial image to the data features of the original resolution image to obtain the image semantic distribution score map after the bilinear upsampling processing.

[0088] After obtaining the image semantic distribution score map after the bilinear upsampling processing as described above, the disclosed embodiment can select the semantic category with the highest score in the semantic distribution score map of the pixel points in the image as the semantic feature of the pixel point, and use the selected semantic features of each pixel point in the image as the image semantic feature, so as to achieve bilinear upsampling of the semantic distribution features according to the initial feature information to obtain the image semantic feature.

[0089] In this embodiment, an initial image is obtained, wherein the initial image corresponds to initial feature information, the initial image is processed to obtain multiple frames of images to be segmented, wherein the multiple frames of images to be segmented correspond to multiple spatial feature information respectively, and pixel semantic features of the image to be segmented based on the corresponding spatial feature information are determined, wherein the pixel semantic features are obtained by identifying the semantic segmentation model corresponding to the image to be segmented, multiple pixel semantic features are fused to obtain semantic distribution features, and the semantic distribution features are processed according to the initial feature information to obtain image semantic features. Feature extraction processing can be performed on the image based on multiple spatial feature information, thereby effectively improving the accuracy of the obtained image semantic features. Using multiple semantic segmentation models to recognize and process the image can effectively improve the semantic segmentation processing efficiency and the image semantic segmentation processing effect. Multiple reference interaction information corresponding to multiple semantic segmentation models are determined, multiple pixel semantic features are fused according to the reference interaction information to obtain target semantic features, and semantic distribution features are obtained according to the target semantic features. Therefore, the semantic features identified by multiple semantic segmentation models can be weighted fused to ensure the accuracy of the acquired semantic features and effectively ensure the quality of image semantic segmentation processing.

[0090] Figure 5 It is a flowchart of an image semantic segmentation method proposed in an embodiment of the present disclosure.

[0091] like Figure 5 As shown, the image semantic segmentation method includes:

[0092] S501: Acquire an initial image, wherein the initial image corresponds to initial feature information.

[0093] S502: Processing the initial image to obtain multiple frames of images to be segmented, wherein the multiple frames of images to be segmented correspond to multiple pieces of spatial feature information respectively.

[0094] The description of S501 - S502 can be referred to the above embodiment for examples, and will not be repeated here.

[0095] S503: Determine demand segmentation performance.

[0096] The required segmentation performance may include the required segmentation accuracy when performing feature recognition processing on the image to be segmented, and the required inference delay when performing feature recognition processing on the image to be segmented.

[0097] In the embodiment of the present disclosure, when determining the required segmentation performance, the required inference delay when performing feature recognition processing on the segmented image and the required segmentation accuracy when performing feature recognition processing on the segmented image can be obtained respectively, and the inference delay calculation formula can be used Determine the inference delay LT required for the semantic segmentation model, where Ni-1 represents the original semantic segmentation model constructed, and Lat(·) represents the measurement of the inference delay required for semantic segmentation to obtain the required inference delay when performing feature recognition processing on the segmented image, and then the required segmentation accuracy when performing feature recognition processing on the segmented image can be determined, and the required inference delay when performing feature recognition processing on the segmented image and the required segmentation accuracy when performing feature recognition processing on the segmented image are used as the required segmentation performance.

[0098] S504: Determine a first target number of convolution blocks in a convolution set according to required segmentation performance, and determine a second target number of feature recognition channels.

[0099] The first target number refers to the number of convolution blocks in the convolution set of the semantic segmentation model to be constructed.

[0100] The second target number refers to the number of feature recognition channels in the convolution block of the convolution set.

[0101] In an embodiment of the present disclosure, when determining the first target number of convolution blocks in a convolution set based on the required segmentation performance and determining the second target number of feature recognition channels, the number of convolution blocks required in the convolution set can be determined based on the required inference delay when performing feature recognition processing on the segmented image, and the number of convolution blocks can be used as the first target number of convolution blocks in the convolution set, and the number of feature recognition channels in the convolution block can be determined based on the required segmentation accuracy when performing feature recognition processing on the segmented image, and the number of feature recognition channels in the convolution block can be used as the second target number of feature recognition channels.

[0102] In some embodiments, when determining the first target number of convolution blocks in the convolution set, the convolution type of the convolution block may also be determined. For example, Figure 6 As shown, Figure 6 This is a performance diagram of the convolution block in the embodiment of the present disclosure. A convolution block of a suitable convolution type can be selected from a convolution block of a standard convolution type and a convolution block of a depth-separable type. By comparing the performance of the two types of convolution blocks, it is found that the inference delay of the convolution block of the standard convolution type is smaller than that of the convolution block of the depth-separable type. Therefore, the convolution block of the standard convolution type can be selected to build a semantic segmentation model.

[0103] In other embodiments, when determining the second target number of feature recognition channels, the convolution efficiency of the convolution blocks containing different feature recognition channels when performing the recognition task can be analyzed, the inference delay of the convolution blocks of different feature recognition channels when performing the recognition task can be analyzed, and the corresponding convolution efficiency curve graph can be drawn. The number of feature recognition channels included in the convolution block can be selected based on the convolution efficiency curve graph.

[0104] For example, Figure 7 As shown, Figure 7 It is a convolution efficiency curve diagram in the embodiment of the present disclosure, wherein the channel width is the number of feature recognition channels in the convolution block. When the channel width increases from 1 to 17, the floating-point calculation amount increases quadratically, and the inference delay increases only linearly, reaching a plateau when the channel width is around 8 and 16. It can be assumed that the number of feature recognition channels contained in the convolution block is 2N.

[0105] Optionally, in some embodiments, when determining the first target number of convolution blocks in a convolution set and the second target number of feature recognition channels according to required segmentation performance, candidate extension parameters can be determined, the first initial number of convolution blocks in the convolution set and the second initial number of feature recognition channels can be determined, and the first initial number and / or the second initial number can be expanded based on the candidate extension parameters to obtain the first expanded number and / or the second expanded number, and the predicted segmentation performance of the expanded segmentation model is predicted, wherein the expanded segmentation model includes: a convolution set consisting of a first expanded number of convolution blocks, and / or the convolution block includes: a second expanded number of feature recognition channels, if the predicted segmentation performance meets the required segmentation performance, the first expanded number is used as the first target number, and / or the second expanded number is used as the second target number, so that in the process of semantic segmentation and recognition of the image, the semantic segmentation neural network can be expanded to achieve flexible adjustment of the semantic segmentation neural network according to the required performance, achieve diversity of network structure, and effectively improve the image semantic segmentation performance.

[0106] Among them, the candidate expansion parameters refer to the network expansion parameters when the semantic segmentation neural network is expanded, and the candidate expansion parameters may include the convolution block number expansion parameter and the feature recognition channel expansion parameter.

[0107] The first initial number refers to the number of convolution blocks in the convolution set before the semantic segmentation model is expanded.

[0108] The second initial number refers to the number of feature recognition channels in the convolution block of the semantic segmentation model before the semantic segmentation model is expanded.

[0109] The first expansion number refers to the number of convolution blocks in the convolution set of the semantic segmentation model obtained after the first initial number is expanded based on the candidate expansion parameters.

[0110] The second expanded number refers to the number of feature recognition channels in the convolution block in the semantic segmentation model obtained after the second initial number is expanded based on the candidate expansion parameter.

[0111] Among them, the extended segmentation model refers to a semantic segmentation model obtained after expanding the number of convolution blocks in a convolution set and / or the number of feature recognition channels in a convolution block. The extended segmentation model may include: a convolution set consisting of a first expanded number of convolution blocks, and / or a convolution block including: a second expanded number of feature recognition channels.

[0112] Among them, the predicted segmentation performance refers to the performance data obtained after predicting and calculating the segmentation performance of the extended segmentation model. The predicted segmentation performance may include the segmentation accuracy when performing feature recognition processing on the segmented image, and the inference delay when performing feature recognition processing on the segmented image.

[0113] In the embodiment of the present disclosure, when determining the first target number of convolution blocks in the convolution set and determining the second target number of feature recognition channels according to the required segmentation performance, the candidate expansion parameters can be determined, and the step expansion formula can be used. Determine candidate extension parameters, and determine the number of convolution blocks in the initial semantic segmentation model convolution set as the first initial number, determine the number of feature recognition channels in the convolution blocks of the initial semantic segmentation model as the second initial number, and then the first initial number and / or the second initial number can be extended based on the candidate extension parameters, and the number after the extension processing is used as the first extended number and / or the second extended number, and then the extended segmentation model can be constructed based on the first extended number of convolution blocks and / or the second extended number of feature recognition channels.

[0114] After the extended segmentation model is constructed as described above, the embodiment of the present disclosure can predict the predicted segmentation performance of the extended segmentation model.

[0115]

[0116] Yes, the performance prediction formula can be used The predicted segmentation performance of the extended segmentation model is calculated, where Perf(·) represents the semantic segmentation accuracy achieved by the extended segmentation model on the semantic segmentation task, N0 represents the constructed initial semantic segmentation model, and Ni represents the i-th generation extended segmentation model in the iterative training process, so as to obtain the predicted segmentation performance of the predicted extended segmentation model, and then it can be judged whether the predicted segmentation performance meets the required segmentation performance. If the required segmentation performance is met, the first expansion number is used as the first target number, and / or the second expansion number is used as the second target number.

[0117] S505: Form a semantic segmentation model, wherein the semantic segmentation model includes: a convolution set consisting of a first number of convolution blocks, and the convolution blocks include: a second target number of feature recognition channels.

[0118] The embodiment of the present disclosure can form a semantic segmentation model after determining the first target number of convolution blocks in the convolution set and determining the second target number of feature recognition channels according to the required segmentation performance as described above.

[0119] In the disclosed embodiment, when forming a semantic segmentation model, a second target number of feature recognition channels may be used to form a plurality of convolution blocks, and a semantic segmentation model may be formed by a first number of convolution blocks.

[0120] In this embodiment, by determining the required segmentation performance, according to the required segmentation performance, determining the first target number of convolution blocks in the convolution set, and determining the second target number of feature recognition channels, a semantic segmentation model is formed, thereby forming a semantic segmentation model that meets the required segmentation performance. Since the semantic segmentation model can be used to determine the pixel semantic features of the image to be segmented, it can be ensured that the accuracy of the extracted pixel semantic features can meet the required segmentation accuracy, thereby effectively improving the accuracy of image semantic segmentation.

[0121] S506: Determine pixel semantic features of the image to be segmented based on corresponding spatial feature information, wherein the pixel semantic features are identified by a semantic segmentation model corresponding to the image to be segmented.

[0122] S507: Fusing multiple pixel semantic features to obtain semantic distribution features.

[0123] S508: Processing the semantic distribution features according to the initial feature information to obtain image semantic features.

[0124] The description of S505 - S508 can be referred to the above embodiment for examples, and will not be repeated here.

[0125] In this embodiment, an initial image is obtained, wherein the initial image corresponds to initial feature information, the initial image is processed to obtain multiple frames of images to be segmented, wherein the multiple frames of images to be segmented correspond to multiple spatial feature information respectively, and pixel semantic features of the image to be segmented based on the corresponding spatial feature information are determined, wherein the pixel semantic features are obtained by identifying the semantic segmentation model corresponding to the image to be segmented, multiple pixel semantic features are fused to obtain semantic distribution features, and the semantic distribution features are processed according to the initial feature information to obtain image semantic features. Feature extraction processing can be performed on the image based on multiple spatial feature information, thereby effectively improving the accuracy of the obtained image semantic features. Using multiple semantic segmentation models to recognize and process the image can effectively improve the semantic segmentation processing efficiency and the image semantic segmentation processing effect. By determining the required segmentation performance, the first target number of convolution blocks in the convolution set is determined according to the required segmentation performance, and the second target number of feature recognition channels is determined to form a semantic segmentation model, thereby forming a semantic segmentation model that meets the required segmentation performance. Since the semantic segmentation model can be used to determine the pixel semantic features of the image to be segmented, it can be ensured that the accuracy of the extracted pixel semantic features can meet the required segmentation accuracy, thereby effectively improving the accuracy of image semantic segmentation.

[0126] Figure 8 It is a flowchart of an image semantic segmentation method proposed in another embodiment of the present disclosure.

[0127] like Figure 8 As shown, the image semantic segmentation method includes:

[0128] S801: Acquire an initial image, wherein the initial image corresponds to initial feature information.

[0129] S802: Processing the initial image to obtain multiple frames of images to be segmented, wherein the multiple frames of images to be segmented correspond to multiple pieces of spatial feature information respectively.

[0130] S803: Determine demand segmentation performance.

[0131] The description of S801 - S802 can be referred to the above embodiment, and will not be repeated here.

[0132] S804: Determine a depth dimension parameter and a width dimension parameter, wherein the depth dimension parameter is used to determine the number of convolution blocks in the convolution set, and the width dimension parameter is used to determine the number of feature recognition channels.

[0133] The depth dimension parameter refers to a data parameter that can be used to determine the number of convolution blocks in a convolution set.

[0134] The width dimension parameter refers to a data parameter that can be used to determine the number of feature recognition channels in the convolution block.

[0135] In the embodiments of the present disclosure, when determining the depth dimension parameter and the width dimension parameter, the convolution block expansion step number value for expanding the number of convolution blocks in the convolution set can be set in advance according to the required segmentation performance, and the convolution block expansion step number value can be used as the depth dimension parameter. The feature recognition channel expansion step number value for expanding the number of feature recognition channels in the convolution block is determined, and the feature recognition channel expansion step number value is used as the depth dimension parameter.

[0136] S805: Take the depth dimension parameter and the width dimension parameter as candidate extension parameters.

[0137] In the embodiment of the present disclosure, after determining the depth dimension parameter and the width dimension parameter as described above, the depth dimension parameter and the width dimension parameter may be used as candidate extension parameters, and then the first initial quantity and / or the second initial quantity may be extended based on the candidate extension parameters to obtain the first extended quantity and / or the second extended quantity.

[0138] Optionally, in some embodiments, when determining candidate extension parameters, spatial feature parameters can be determined as candidate extension parameters, and multiple spatial feature information can be determined based on the spatial feature parameters. Corresponding multiple semantic segmentation models can be configured based on the multiple spatial feature information, so that the semantic segmentation model can identify the pixel semantic features of the corresponding image to be segmented, wherein the spatial feature information of the corresponding image to be segmented is the same as the spatial feature information configured for the semantic segmentation model, so that the spatial feature parameters can be determined as candidate extension parameters to obtain richer candidate extension parameters. Since the semantic segmentation neural network is expanded from different angles when using multiple candidate extension parameters, flexible and dynamic scaling of the semantic segmentation neural network can be achieved, thereby effectively improving the image semantic segmentation processing performance.

[0139] The spatial feature parameter refers to a data parameter that can be input into a semantic segmentation model to describe the feature spatial resolution of the image to be segmented.

[0140] Among them, spatial feature information refers to the resolution data information of the image to be segmented input into the semantic segmentation model. According to multiple spatial feature information, corresponding multiple semantic segmentation models can be configured. Multiple semantic segmentation models can identify and process the corresponding images to be segmented. The spatial feature information of the corresponding images to be segmented is the same as the spatial feature information configured for the semantic segmentation model.

[0141] In an embodiment of the present disclosure, when determining candidate extension parameters, spatial feature parameters can be determined as candidate extension parameters, and then the resolution data information of the image to be segmented input into the semantic segmentation model can be determined based on the spatial parameter characteristics, and the resolution data information can be used as spatial feature information. Then, the semantic segmentation model can be configured based on the spatial feature information to obtain multiple semantic segmentation models after configuration processing. The multiple semantic segmentation models can perform recognition processing on the corresponding images to be segmented so that the semantic segmentation models can recognize the pixel semantic features of the corresponding images to be segmented.

[0142] In some embodiments, when determining candidate extension parameters, multiple groups of parameter groups consisting of depth dimension parameters, width dimension parameters, and spatial parameter features can be pre-set, and the predicted extension performance of the extended segmentation model constructed after the first initial number and / or the second initial number are expanded according to the multiple parameter groups is compared. The first extension number and the second extension number whose predicted extension performance meets the required segmentation performance are selected as the first target number and the second target number, and the spatial parameter features that meet the required segmentation performance are selected to construct a semantic segmentation model to perform recognition processing on the image to be segmented.

[0143] For example, Fig. 9 As shown, Fig. 9 It is a schematic diagram of the available extended parameters in the embodiment of the present disclosure. The parameters of the initial semantic segmentation model can be set to N0={B0, C0, R0}, where B is a depth dimension parameter, C is a width dimension parameter, and R is a spatial resolution parameter. B0={1, 1, 1, 1, 1}, C0={4, 8, 16, 32, 32}, R0={1 / 2, 0, 0}, and the expandable parameter group can be expressed as ΔB∪ΔC∪ΔR. The extended segmentation performance of multiple parameter groups can be evaluated according to the required semantic segmentation accuracy and the required inference delay, and the parameter group with better performance can be selected to build the semantic segmentation model.

[0144] In some other embodiments, after determining the candidate extension parameters, the first initial number and / or the second initial number can be extended based on the candidate extension parameters to obtain multiple sets of available depth dimension parameters, width dimension parameters, and spatial resolution parameters, and a progressively scalable lightweight semantic segmentation neural network (Lightweight and Progressively-Scalable Networks, LPS-Net) is constructed according to the multiple parameter groups. For example, Fig.10 As shown, Fig.10 It is a parameter diagram of the semantic segmentation neural network in the embodiment of the present disclosure, which includes three typical network structure parameters of LPS-Net. The LPS-Net constructed according to the corresponding network structure parameters can achieve a relatively accurate semantic segmentation effect while ensuring low inference latency.

[0145] S806: Determine a first initial number of convolution blocks in the convolution set, and determine a second initial number of feature recognition channels.

[0146] S807: Perform expansion processing on the first initial number and / or the second initial number based on the candidate expansion parameter to obtain a first expanded number and / or a second expanded number.

[0147] S808: Predicting the predicted segmentation performance of the extended segmentation model, wherein the extended segmentation model includes: a convolution set consisting of a first extended number of convolution blocks, and / or the convolution block includes: a second extended number of feature recognition channels.

[0148] S809: If the predicted segmentation performance meets the required segmentation performance, the first expansion quantity is used as the first target quantity, and / or the second expansion quantity is used as the second target quantity.

[0149] S810: Form a semantic segmentation model, wherein the semantic segmentation model includes: a convolution set consisting of a first number of convolution blocks, and the convolution blocks include: a second target number of feature recognition channels.

[0150] S811: Determine pixel semantic features of the image to be segmented based on corresponding spatial feature information, wherein the pixel semantic features are identified by a semantic segmentation model corresponding to the image to be segmented.

[0151] S812: Fuse multiple pixel semantic features to obtain semantic distribution features.

[0152] S813: Processing the semantic distribution features according to the initial feature information to obtain image semantic features.

[0153] The description of S806-S813 can be referred to the above embodiment for examples, and will not be repeated here.

[0154] In this embodiment, by acquiring an initial image, wherein the initial image corresponds to initial feature information, processing the initial image, obtaining multiple frames of images to be segmented, wherein the multiple frames of images to be segmented correspond to multiple spatial feature information respectively, determining pixel semantic features of the image to be segmented based on the corresponding spatial feature information, wherein the pixel semantic features are obtained by identifying a semantic segmentation model corresponding to the image to be segmented, fusing multiple pixel semantic features to obtain semantic distribution features, and processing the semantic distribution features according to the initial feature information to obtain image semantic features, it is possible to perform feature extraction processing on the image based on multiple spatial feature information, thereby effectively improving the accuracy of the obtained image semantic features, using multiple semantic segmentation models to recognize and process the image, effectively improving the semantic segmentation processing efficiency, and effectively improving the image semantic segmentation processing effect, by determining depth dimension parameters and width dimension parameters, using the depth dimension parameters and width dimension parameters as candidate extension parameters, and determining spatial feature parameters as candidate extension parameters, thereby determining spatial feature parameters as candidate extension parameters, so as to achieve the acquisition of richer candidate extension parameters, since the semantic segmentation neural network is extended from different angles when multiple candidate extension parameters are used, flexible and dynamic scaling of the semantic segmentation neural network can be achieved, effectively improving the image semantic segmentation processing performance.

[0155] Fig.11 It is a flowchart of an image semantic segmentation method proposed in another embodiment of the present disclosure.

[0156] like Fig.11 As shown, the image semantic segmentation method includes:

[0157] S1101: Acquire an initial image, wherein the initial image corresponds to initial feature information.

[0158] S1102: Processing the initial image to obtain multiple frames of images to be segmented, wherein the multiple frames of images to be segmented correspond to multiple pieces of spatial feature information respectively.

[0159] The description of S1101 - S1102 can be referred to the above embodiment for examples, and will not be repeated here.

[0160] S1103: Processing the image to be segmented based on the convolution blocks in the first convolution set to obtain a first semantic feature, wherein the first convolution set includes: a first convolution block in a plurality of cascaded convolution blocks.

[0161] The first convolution set may refer to a convolution set consisting of the first convolution block in a plurality of cascaded convolution blocks, and the first convolution set may be used to process the image to be segmented so as to extract semantic features in the image to be segmented.

[0162] The first semantic feature refers to the semantic feature obtained after performing feature extraction processing on the image to be segmented using the first convolution set.

[0163] In the embodiment of the present disclosure, when processing the image to be segmented based on the convolution block in the first convolution set, the image to be segmented that matches the convolution block in the first convolution set can be selected according to the corresponding spatial feature information, and then feature extraction processing is performed on the corresponding image to be segmented based on the first convolution set to obtain the output result of the convolution block in the first convolution set, and the data result is used as the first semantic feature.

[0164] S1104: Perform a feature recognition task on the first semantic feature based on the convolution blocks in the second convolution set to obtain a pixel semantic feature, wherein the first convolution block in the cascade in the second convolution set is after the last convolution block in the cascade in the first convolution set.

[0165] Among them, the second convolution set can refer to a convolution set located after the first convolution set in the structure of the neural network model, the cascaded first convolution block in the second convolution set is after the cascaded last convolution block in the first convolution set, the number of second convolution sets can be multiple, and the second convolution set can perform feature recognition tasks on the first semantic feature to obtain pixel semantic features.

[0166] In the embodiment of the present disclosure, after the image to be segmented is processed based on the convolution blocks in the first convolution set to obtain the first semantic feature, a feature recognition task can be performed on the first semantic feature based on the convolution blocks in the second convolution set to obtain a pixel semantic feature.

[0167] In the disclosed embodiment, when performing a feature recognition task on the first semantic feature based on the convolution block in the second convolution set to obtain a pixel semantic feature, the first semantic feature output by the last convolution block in the cascade in the first convolution set can be obtained, and then the first semantic feature can be input into the convolution block in the second convolution set, and the convolution block in the second convolution set can be used to perform feature recognition processing on the first semantic feature. Multiple second convolution sets can be used to perform feature recognition processing on the first semantic feature in sequence to obtain the semantic feature output by the last convolution block in the cascade of the last second convolution set, and the semantic feature can be used as the pixel semantic feature of the image to be segmented.

[0168] For example, Fig.12 As shown, Fig.12It is a schematic diagram of the macro structure of the semantic segmentation neural network in the embodiment of the present disclosure. The semantic segmentation neural network adopts the design concept of a multi-path network. The input of the semantic segmentation neural network is a plurality of scales of images to be segmented obtained by performing a plurality of scaling operations on the initial image. Each path of the semantic segmentation neural network is composed of a stacked cascade of convolution blocks that can be divided into five stages, wherein the convolution blocks in the first stage can constitute a second convolution set, and the convolution blocks from the second node to the fifth stage constitute a plurality of second convolution sets. The image to be segmented can be processed based on the convolution blocks in the first convolution set to obtain a first semantic feature, and a feature recognition task can be performed on the first semantic feature based on the convolution blocks in the second convolution set, and the output result of the convolution block in the second convolution set corresponding to the fifth stage is used as the pixel semantic feature, wherein the scaling ratio in the semantic segmentation neural network model The number of convolutional blocks in each stage and the number of channels in each convolutional block Both can be flexibly adjusted according to the required segmentation performance, thereby ensuring the network structure diversity and scalability of the semantic segmentation neural network model.

[0169] Optionally, in some embodiments, there are multiple second convolution sets, and an information exchange model is set between different second convolution sets. The input end of the information exchange model is connected to the last convolution block in the cascade in the previous second convolution set, and the output end of the information exchange model is connected to the first convolution block in the cascade in the next second convolution set. The information exchange model can be used to realize the interaction of feature information between multiple second convolution sets, so as to achieve supplementary processing of feature extraction of multiple second convolution sets, effectively improve the accuracy of feature recognition and extraction, and assist in improving the quality of image semantic segmentation processing.

[0170] Among them, the information exchange model refers to an information processing model that can be used to exchange information between multiple semantic segmentation paths.

[0171] In the disclosed embodiment, a plurality of connected second convolution sets may be set, and an information exchange model may be set between different second convolution sets, the input end of the information exchange model may be connected to the last convolution block in the cascade in the previous second convolution set, and the output end of the information exchange model may be connected to the first convolution block in the cascade in the next second convolution set, and then the information exchange model between the plurality of different second convolution sets may be used to realize the interaction of feature information between the plurality of second convolution sets, thereby assisting in achieving better semantic segmentation quality.

[0172] S1105: Perform bidirectional interactive processing on the first pixel semantic feature and the second pixel semantic feature based on the information exchange model to obtain a fused pixel semantic feature, wherein the first pixel semantic feature is the pixel semantic feature output by the previous second convolution set, and the second pixel semantic feature is the pixel semantic feature output by the corresponding second convolution set in other semantic segmentation models.

[0173] Among them, the first pixel semantic feature refers to the pixel semantic feature output by the previous second convolution set in the multiple connected second convolution sets, and the second pixel semantic feature refers to the pixel semantic feature output by the corresponding second convolution set in other semantic segmentation models.

[0174] After the information exchange model is set between different second convolution sets in the embodiment of the present disclosure, bidirectional interactive processing can be performed on the first pixel semantic feature and the second pixel semantic feature based on the information exchange model to obtain a fused pixel semantic feature.

[0175] In the embodiment of the present disclosure, when bidirectional interactive processing is performed on the first pixel semantic feature and the second pixel semantic feature based on the information exchange model to obtain the fused pixel semantic feature, the information exchange model can be defined as: in, and A pair of features of different scales representing the output and input, respectively, where the input pair consists of pixel semantic features of the high-resolution image to be segmented and the pixel semantic features of the low-resolution image to be segmented The first pixel semantic feature and the second pixel semantic feature can be the pixel semantic feature of the high-resolution image to be segmented and the pixel semantic feature of the low-resolution image to be segmented, so that two-way interactive processing of the first pixel semantic feature and the second pixel semantic feature can be achieved based on the information exchange model, and the semantic feature output by the information exchange model after the two-way interactive processing is used as the fused semantic feature.

[0176] Optionally, in some embodiments, when the first pixel semantic feature and the second pixel semantic feature are bidirectionally interactively processed based on the information exchange model to obtain a fused pixel semantic feature, the first pixel semantic feature is spatially upsampled based on the information exchange model to obtain a first sampling semantic feature, and the second pixel semantic feature is spatially downsampled based on the information exchange model to obtain a second sampling semantic feature, wherein the fineness of the spatial feature information of the image to be segmented to which the first pixel semantic feature belongs is greater than the fineness of the spatial feature information of the image to be segmented to which the second pixel semantic feature belongs, and the first sampling semantic feature and the second sampling semantic feature are fused to obtain a fused pixel semantic feature, thereby realizing bidirectional interaction of feature information between multiple second convolution sets based on the information exchange model to achieve better information interaction effect and assist in improving the quality of image semantic segmentation. Since the bidirectional interaction uses spatial scaling and element summation to interact information, the computational processing amount is small, which effectively saves data computing resources and assists in improving the efficiency of semantic segmentation processing.

[0177] The first sampled semantic feature refers to a semantic feature obtained by performing spatial upsampling processing on the first pixel semantic feature based on the information exchange model.

[0178] Among them, the second sampled semantic feature refers to the semantic feature obtained after spatial downsampling processing is performed on the second pixel semantic feature based on the information exchange model. The fineness of the spatial feature information of the image to be segmented to which the first pixel semantic feature belongs is greater than the fineness of the spatial feature information of the image to be segmented to which the second pixel semantic feature belongs. The fineness of the spatial feature information of the image to be segmented can be the resolution data information of the image to be segmented, that is, the resolution data information of the image to be segmented to which the first pixel semantic feature belongs is greater than the resolution data information of the image to be segmented to which the second pixel semantic feature belongs.

[0179] In the embodiment of the present disclosure, when the first pixel semantic feature and the second pixel semantic feature are bidirectionally interactively processed based on the information exchange model to obtain the fused pixel semantic feature, the first pixel semantic feature can be spatially upsampled based on the information exchange model to obtain the first sampled semantic feature. The information exchange model can be defined as in, Indicates that the first pixel semantic feature is spatially upsampled. It means that the second pixel semantic feature is spatially downsampled, and then the first pixel semantic feature is spatially upsampled based on the information exchange model to obtain the first sampled semantic feature output by the information exchange model, and the second pixel semantic feature is spatially downsampled based on the information exchange model to obtain the second sampled semantic feature output by the information exchange model, and then the first sampled semantic feature and the second sampled semantic feature can be fused to obtain the fused semantic feature, and the fused semantic feature is used as the fused pixel semantic feature.

[0180] For example, Fig.13 As shown, Fig.13 It is a flow chart of the bidirectional interaction of information features in the embodiment of the present disclosure. The first pixel semantic feature and the second semantic feature can be convolution transformed based on the information exchange model, and then the first pixel semantic feature can be spatially upsampled to obtain the first sampling semantic feature, and the second pixel semantic feature can be spatially downsampled to obtain the second sampling semantic feature, so as to realize bidirectional interaction between the first pixel semantic feature and the second pixel semantic feature, and then the first sampling semantic feature and the second semantic feature can be element-wise summed to fuse the first sampling semantic feature and the second semantic feature to obtain a fused pixel semantic feature.

[0181] S1106: Input the fused pixel semantic features into the next second convolution set.

[0182] In the embodiment of the present disclosure, after bidirectional interactive processing is performed on the first pixel semantic feature and the second pixel semantic feature based on the information exchange model to obtain a fused pixel semantic feature, the fused pixel semantic feature can be input into the subsequent second convolution set.

[0183] S1107: Processing and fusing pixel semantic features based on the convolution blocks in the next second convolution set to obtain pixel semantic features output by the next second convolution set.

[0184] After the fused pixel semantic features are input into the subsequent second convolution set as mentioned above, the embodiment of the present disclosure can process the fused pixel semantic features based on the convolution blocks in the subsequent second convolution set, and can perform convolution transformation processing on the fused pixel semantic features based on the convolution blocks in the subsequent second convolution set to obtain the pixel semantic features output by the subsequent second convolution set.

[0185] S1108: Using the next second convolution set as the previous second convolution set until the next second convolution set is the last second convolution set among the multiple second convolution sets.

[0186] In the embodiment of the present disclosure, after the pixel semantic features are fused based on the convolution block processing in the subsequent second convolution set to obtain the pixel semantic features output by the subsequent second convolution set, the subsequent second convolution set is used as the previous second convolution set, the pixel semantic features output by the previous second convolution set are input into the new subsequent second convolution set, and the pixel semantic features are fused based on the convolution block processing in the subsequent second convolution set to implement iterative processing of the fused pixel semantic features, until the subsequent second convolution set is the last second convolution set among multiple second convolution sets, and the convolution processing of the fused pixel semantic features is completed.

[0187] S1109: If it is determined that the latter second convolution set is the last second convolution set, multiple fused pixel semantic features are processed to obtain semantic distribution features.

[0188] In the disclosed embodiment, it is possible to judge whether the subsequent second convolution set is still the last second convolution set. If the cascaded last convolution block of the current subsequent second convolution set has no convolution block connected to it, it is determined that the subsequent second convolution set is the last second convolution set. If it is determined that the subsequent second convolution set is the last second convolution set, multiple fused pixel semantic features output by the last second convolution set can be obtained, and the multiple fused pixel semantic features can be statistically calculated and processed to obtain a semantic distribution score map of the pixel, and the semantic distribution score map of the pixel is used as the semantic distribution feature.

[0189] S1110: Processing the semantic distribution features according to the initial feature information to obtain image semantic features.

[0190] The description of S1110 can be seen from the above-mentioned embodiment, and will not be repeated here.

[0191] In the present embodiment, an initial image is obtained, wherein the initial image corresponds to initial feature information, the initial image is processed to obtain multiple frames of images to be segmented, wherein the multiple frames of images to be segmented respectively correspond to multiple spatial feature information, pixel semantic features of the images to be segmented based on the corresponding spatial feature information are determined, wherein the pixel semantic features are obtained by identifying a semantic segmentation model corresponding to the image to be segmented, multiple pixel semantic features are fused to obtain semantic distribution features, and the semantic distribution features are processed according to the initial feature information to obtain image semantic features, and feature extraction processing can be performed on the image based on multiple spatial feature information, thereby effectively improving the accuracy of the obtained image semantic features, and using multiple semantic segmentation models to recognize and process the image can effectively improve the accuracy of the semantic features. The efficiency of semantic segmentation processing is improved, and the processing effect of image semantic segmentation processing is effectively improved. The first pixel semantic feature is spatially upsampled based on the information exchange model to obtain the first sampling semantic feature, and the second pixel semantic feature is spatially downsampled based on the information exchange model to obtain the second sampling semantic feature, and the first sampling semantic feature and the second sampling semantic feature are fused to obtain the fused pixel semantic feature. Therefore, the two-way interaction of feature information between multiple second convolution sets can be realized based on the information exchange model to achieve better information interaction effect, and assist in improving the quality of image semantic segmentation. Since the two-way interaction uses spatial scaling and element summation to exchange information, the computational processing amount is small, which effectively saves data computing resources and assists in improving the processing efficiency of semantic segmentation.

[0192] Fig.14 It is a flowchart of an image semantic segmentation method proposed in another embodiment of the present disclosure.

[0193] like Fig.14 As shown, the image semantic segmentation method includes:

[0194] S1401: Acquire an initial image, wherein the initial image corresponds to initial feature information.

[0195] S1402: Processing the initial image to obtain multiple frames of images to be segmented, wherein the multiple frames of images to be segmented correspond to multiple pieces of spatial feature information respectively.

[0196] The description of S1401 - S1402 can be referred to the above embodiment for examples, and will not be repeated here.

[0197] S1403: Determine initial pixel features of the image to be segmented.

[0198] The initial pixel feature refers to the pixel data feature obtained after performing feature extraction processing on the pixel points in the image to be segmented. The initial pixel feature may be the data feature before performing dimensionality reduction processing on the pixel feature.

[0199] In the disclosed embodiment, when determining the initial pixel features of the image to be segmented, the image to be segmented can be input into an image processing model for processing, and the image processing model can be used to perform feature extraction processing on the pixel points in the image to be segmented to obtain the initial pixel features of the image to be segmented output by the image processing model.

[0200] S1404: Reduce the dimension of the initial pixel feature based on the first convolution block in the cascade in the first convolution set to obtain a first reduced-dimensional pixel feature.

[0201] The first dimension-reduced pixel feature refers to a pixel feature obtained after dimension reduction processing is performed on the initial pixel feature of the image to be segmented, and the spatial dimension of the first dimension-reduced pixel feature is smaller than the initial pixel feature of the image to be segmented.

[0202] After determining the initial pixel features of the image to be segmented as described above, the disclosed embodiment can reduce the dimension of the initial pixel features based on the first convolution block in the cascade in the first convolution set, input the initial pixel features into the first convolution block in the cascade in the first convolution set, and perform dimension reduction processing on the initial pixel features based on the first convolution block in the cascade in the first convolution set. The spatial dimension of the initial pixel features can be reduced to half of the original dimension, and the pixel features after dimension reduction processing can be used as the first reduced dimension pixel features.

[0203] S1405: Process the first dimensionality reduction pixel feature based on other convolution blocks in the first convolution set to obtain a first semantic feature.

[0204] In the embodiment of the present disclosure, after the initial pixel feature is reduced in dimension based on the first convolution block in the cascade in the first convolution set to obtain the first reduced-dimensionality pixel feature, the first reduced-dimensionality pixel feature can be further input into other convolution blocks in the first convolution set, and the first reduced-dimensionality pixel feature can be convolutionally transformed based on the other convolution blocks in the first convolution set to obtain the semantic feature output by the last convolution block in the cascade in the first convolution set, and the semantic feature is used as the first semantic feature.

[0205] S1406: Reduce the dimension of the first semantic feature based on the cascaded first convolution block in the second convolution set to obtain a second reduced-dimensional pixel feature.

[0206] The second dimension-reduced pixel feature refers to a pixel feature obtained after dimension-reducing processing is performed on the first semantic feature, and the spatial dimension of the second dimension-reduced pixel feature is smaller than that of the first dimension-reduced pixel feature.

[0207] In the disclosed embodiment, after the first reduced-dimensionality pixel feature is processed based on other convolution blocks in the first convolution set to obtain the first semantic feature, the first semantic feature can be input into the convolution block of the second convolution set, and the first semantic feature can be reduced-dimensionality based on the first convolution block in the cascade in the second convolution set to reduce the spatial dimension of the first semantic feature to half of the original dimension, and the semantic feature after the reduced-dimensionality processing is used as the second reduced-dimensionality pixel feature.

[0208] S1407: Process the second dimensionality reduction pixel feature based on other convolution blocks in the second convolution set to obtain a pixel semantic feature.

[0209] In the embodiment of the present disclosure, after the first semantic feature is reduced in dimension based on the first convolution block in the cascade in the second convolution set to obtain the second reduced-dimensionality pixel feature, the second reduced-dimensionality pixel feature can be input into other convolution blocks in the second convolution set, and the second reduced-dimensionality pixel feature is subjected to convolution transformation processing based on the other convolution blocks in the second convolution set to obtain the pixel semantic feature output by the last convolution block in the cascade of the second convolution set, and then the second reduced-dimensionality pixel feature is further subjected to dimensionality reduction and convolution transformation processing to obtain the semantic feature output by the last convolution block in the cascade of the last second convolution set, and the semantic feature is used as the pixel semantic feature.

[0210] In this embodiment, by determining the initial pixel features of the image to be segmented, the initial pixel features are reduced in dimension based on the first convolution block in the cascade in the first convolution set to obtain the first reduced dimension pixel features, the first reduced dimension pixel features are processed based on other convolution blocks in the first convolution set to obtain the first semantic features, the first semantic features are reduced in dimension based on the first convolution block in the cascade in the second convolution set to obtain the second reduced dimension pixel features, the second reduced dimension pixel features are processed based on other convolution blocks in the second convolution set to obtain pixel semantic features, so that the initial image to be segmented can be subjected to multiple dimensionality reduction processing, so as to obtain pixel semantic features that match the image resolution of the initial image, thereby ensuring the accuracy of the acquired semantic features, and effectively improving the image semantic segmentation effect.

[0211] S1408: Fusing multiple pixel semantic features to obtain semantic distribution features.

[0212] S1409: Processing the semantic distribution features according to the initial feature information to obtain image semantic features.

[0213] The description of S1408-S1409 can be referred to the above embodiment for examples, and will not be repeated here.

[0214] In this embodiment, an initial image is obtained, wherein the initial image corresponds to initial feature information, the initial image is processed to obtain multiple frames of images to be segmented, wherein the multiple frames of images to be segmented correspond to multiple spatial feature information respectively, pixel semantic features of the image to be segmented based on the corresponding spatial feature information are determined, wherein the pixel semantic features are obtained by identifying a semantic segmentation model corresponding to the image to be segmented, multiple pixel semantic features are fused to obtain semantic distribution features, and the semantic distribution features are processed according to the initial feature information to obtain image semantic features, and feature extraction processing can be performed on the image based on multiple spatial feature information, thereby effectively improving the accuracy of the obtained image semantic features, and multiple semantic segmentation models are used to recognize and process the image, which can effectively improve the semantic segmentation processing. Efficiency, effectively improving the image semantic segmentation processing effect, by determining the initial pixel features of the image to be segmented, reducing the dimension of the initial pixel features based on the first convolution block in the cascade in the first convolution set, obtaining the first reduced dimension pixel features, processing the first reduced dimension pixel features based on other convolution blocks in the first convolution set, obtaining the first semantic features, reducing the dimension of the first semantic features based on the first convolution block in the cascade in the second convolution set, obtaining the second reduced dimension pixel features, processing the second reduced dimension pixel features based on other convolution blocks in the second convolution set, obtaining pixel semantic features, thereby performing multiple dimensionality reduction processing on the initial image to be segmented, and obtaining pixel semantic features that match the image resolution of the initial image, ensuring the accuracy of the acquired semantic features, and effectively improving the image semantic segmentation effect.

[0215] Fig.15 It is a structural diagram of an image semantic segmentation device proposed in one embodiment of the present disclosure.

[0216] like Fig.15 As shown, the image semantic segmentation device 150 includes:

[0217] An acquisition module 1501 is used to acquire an initial image, wherein the initial image corresponds to initial feature information;

[0218] The first processing module 1502 is used to process the initial image to obtain multiple frames of images to be segmented, wherein the multiple frames of images to be segmented correspond to multiple spatial feature information respectively;

[0219] The first determination module 1503 is used to determine pixel semantic features of the image to be segmented based on corresponding spatial feature information, wherein the pixel semantic features are obtained by identifying a semantic segmentation model corresponding to the image to be segmented;

[0220] A fusion module 1504 is used to fuse multiple pixel semantic features to obtain a semantic distribution feature; and

[0221] The second processing module 1505 is used to process the semantic distribution feature according to the initial feature information to obtain the image semantic feature.

[0222] In some embodiments of the present disclosure, the fusion module 1504 is specifically configured to:

[0223] Determine a plurality of reference interaction information corresponding to the plurality of semantic segmentation models, wherein the reference interaction information is used to describe the reference weight of the corresponding pixel semantic feature;

[0224] fusing multiple pixel semantic features according to the reference interaction information to obtain the target semantic feature; and

[0225] According to the target semantic features, the semantic distribution features are obtained.

[0226] In some embodiments of the present disclosure, the semantic segmentation model includes: a plurality of cascaded convolution blocks, some of the cascaded convolution blocks constitute a convolution set, and the convolution block includes: a feature recognition channel; wherein,

[0227] The convolution block in the convolution set is used to perform feature recognition tasks on the image to be segmented belonging to the semantic segmentation model based on the feature recognition channel.

[0228] In some embodiments of the present disclosure,

[0229] The number of semantic segmentation models is dynamically adjustable; and / or

[0230] The number of convolutional blocks in a convolutional set is dynamically adjustable; and / or

[0231] The number of feature recognition channels is dynamically adjustable.

[0232] In some embodiments of the present disclosure, Fig.16 As shown, Fig.16 is a structural diagram of an image semantic segmentation device proposed in another embodiment of the present disclosure, and further includes:

[0233] The second determination module 1506 is used to determine the demand segmentation performance;

[0234] A third determination module 1507 is used to determine a first target number of convolution blocks in the convolution set and a second target number of feature recognition channels according to the required segmentation performance; and

[0235] The generation module 1508 is used to form a semantic segmentation model, wherein the semantic segmentation model includes: a convolution set consisting of a first number of convolution blocks, and the convolution blocks include: a second target number of feature recognition channels.

[0236] In some embodiments of the present disclosure, the third determining module 1507 is specifically configured to:

[0237] determining candidate expansion parameters;

[0238] Determining a first initial number of convolution blocks in the convolution set, and determining a second initial number of feature recognition channels;

[0239] Performing expansion processing on the first initial number and / or the second initial number based on the candidate expansion parameter to obtain a first expanded number and / or a second expanded number;

[0240] Predicting the predicted segmentation performance of the extended segmentation model, wherein the extended segmentation model includes: a convolution set consisting of a first extended number of convolution blocks, and / or the convolution block includes: a second extended number of feature recognition channels; and

[0241] If the predicted segmentation performance meets the required segmentation performance, the first expansion quantity is used as the first target quantity, and / or the second expansion quantity is used as the second target quantity.

[0242] In some embodiments of the present disclosure, the third determining module 1507 is further configured to:

[0243] Determine a depth dimension parameter and a width dimension parameter, wherein the depth dimension parameter is used to determine the number of convolution blocks in the convolution set, and the width dimension parameter is used to determine the number of feature recognition channels;

[0244] The depth dimension parameter and the width dimension parameter are taken as candidate extension parameters.

[0245] In some embodiments of the present disclosure, the third determining module 1507 is further configured to:

[0246] determining spatial characteristic parameters as candidate expansion parameters;

[0247] Wherein, the third determination module is further used for:

[0248] Determine multiple spatial feature information according to the spatial feature parameters;

[0249] According to the multiple spatial feature information, corresponding multiple semantic segmentation models are configured so that the semantic segmentation models can identify the pixel semantic features of the corresponding image to be segmented, wherein the spatial feature information of the corresponding image to be segmented is the same as the spatial feature information configured for the semantic segmentation model.

[0250] In some embodiments of the present disclosure, the first determining module 1503 is specifically configured to:

[0251] Processing the image to be segmented based on the convolution blocks in the first convolution set to obtain a first semantic feature, wherein the first convolution set includes: a first convolution block in the cascaded plurality of convolution blocks;

[0252] A feature recognition task is performed on the first semantic feature based on the convolution block in the second convolution set to obtain a pixel semantic feature, wherein the first convolution block in the cascade in the second convolution set is after the last convolution block in the cascade in the first convolution set.

[0253] In some embodiments of the present disclosure, the first determining module 1503 is further configured to:

[0254] Determine the initial pixel features of the image to be segmented;

[0255] Reducing the dimension of the initial pixel feature based on the first convolution block in the cascade in the first convolution set to obtain a first reduced-dimensional pixel feature; and

[0256] The first reduced-dimensional pixel feature is processed based on other convolution blocks in the first convolution set to obtain a first semantic feature.

[0257] In some embodiments of the present disclosure, the first determining module 1503 is further configured to:

[0258] Reducing the dimension of the first semantic feature based on the first convolution block in the cascade in the second convolution set to obtain a second reduced-dimensional pixel feature;

[0259] The second dimensionality reduction pixel features are processed based on other convolution blocks in the second convolution set to obtain pixel semantic features.

[0260] In some embodiments of the present disclosure,

[0261] There are multiple second convolution sets, and information exchange models are set between different second convolution sets. The input end of the information exchange model is connected to the last convolution block in the cascade in the previous second convolution set, and the output end of the information exchange model is connected to the first convolution block in the cascade in the next second convolution set.

[0262] In some embodiments of the present disclosure, it further includes:

[0263] The third processing module 1509 is used to perform bidirectional interactive processing on the first pixel semantic feature and the second pixel semantic feature based on the information exchange model to obtain a fused pixel semantic feature, wherein the first pixel semantic feature is the pixel semantic feature output by the previous second convolution set, and the second pixel semantic feature is the pixel semantic feature output by the corresponding second convolution set in other semantic segmentation models.

[0264] In some embodiments of the present disclosure, the third processing module 1509 is specifically configured to:

[0265] Performing spatial upsampling processing on the first pixel semantic feature based on the information exchange model to obtain a first sampled semantic feature;

[0266] Performing spatial downsampling processing on the second pixel semantic feature based on the information exchange model to obtain a second sampled semantic feature, wherein the fineness of the spatial feature information of the image to be segmented to which the first pixel semantic feature belongs is greater than the fineness of the spatial feature information of the image to be segmented to which the second pixel semantic feature belongs; and

[0267] The first sampling semantic feature and the second sampling semantic feature are fused to obtain a fused pixel semantic feature.

[0268] In some embodiments of the present disclosure, it further includes:

[0269] An input module 1510 is used for performing bidirectional interactive processing on the first pixel semantic feature and the second pixel semantic feature based on the information exchange model to obtain a fused pixel semantic feature, and then inputting the fused pixel semantic feature into the latter second convolution set;

[0270] A fourth processing module 1511 is used to process and fuse pixel semantic features based on the convolution blocks in the next second convolution set to obtain pixel semantic features output by the next second convolution set; and

[0271] The fifth processing module 1512 is configured to use the next second convolution set as the previous second convolution set until the next second convolution set is the last second convolution set among the multiple second convolution sets.

[0272] In some embodiments of the present disclosure, the fusion module 1504 is further configured to:

[0273] If it is determined that the latter second convolution set is the last second convolution set, a plurality of fused pixel semantic features are processed to obtain semantic distribution features.

[0274] In some embodiments of the present disclosure, the second processing module 1505 is specifically configured to:

[0275] According to the initial feature information, the semantic distribution features are bilinearly upsampled to obtain the image semantic features.

[0276] With the above Figures 1 to 13 Corresponding to the image semantic segmentation method provided in the embodiment, the present disclosure also provides an image semantic segmentation device. Figures 1 to 13 The image semantic segmentation method provided in the embodiment corresponds to the above, so the implementation of the image semantic segmentation method is also applicable to the image semantic segmentation device provided in the embodiment of the present disclosure, and will not be described in detail in the embodiment of the present disclosure.

[0277] In this embodiment, an initial image is obtained, wherein the initial image corresponds to initial feature information, and the initial image is processed to obtain multiple frames of images to be segmented, wherein the multiple frames of images to be segmented respectively correspond to multiple spatial feature information, and pixel semantic features of the image to be segmented based on the corresponding spatial feature information are determined, wherein the pixel semantic features are obtained by identifying a semantic segmentation model corresponding to the image to be segmented, and multiple pixel semantic features are fused to obtain semantic distribution features, and the semantic distribution features are processed according to the initial feature information to obtain image semantic features. Feature extraction processing can be performed on the image based on multiple spatial feature information, thereby effectively improving the accuracy of the obtained image semantic features, and using multiple semantic segmentation models to recognize and process the image can effectively improve the efficiency of semantic segmentation processing and effectively improve the image semantic segmentation processing effect.

[0278] In order to implement the above embodiments, the present disclosure also proposes a computer device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, the image semantic segmentation method proposed in the above embodiments of the present disclosure is implemented.

[0279] In order to implement the above embodiments, the present disclosure further proposes a non-temporary computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, the image semantic segmentation method proposed in the above embodiments of the present disclosure is implemented.

[0280] In order to implement the above embodiments, the present disclosure further proposes a computer program product. When an instruction processor in the computer program product executes, the image semantic segmentation method proposed in the above embodiments of the present disclosure is executed.

[0281] Fig.17 A block diagram of an exemplary computer device suitable for implementing embodiments of the present disclosure is shown. Fig.17 The computer device 12 shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0282] like Fig.17 As shown, the computer device 12 is in the form of a general-purpose computing device. The components of the computer device 12 may include, but are not limited to: one or more processors or processing units 16, a system memory 28, and a bus 18 that connects various system components (including the system memory 28 and the processing unit 16).

[0283] The bus 18 represents one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, a graphics acceleration port, a processor or a local bus using any of a variety of bus structures. For example, these architectures include but are not limited to Industry Standard Architecture (ISA) bus, Micro Channel Architecture (MAC) bus, Enhanced ISA bus, Video Electronics Standards Association (VESA) local bus and Peripheral Component Interconnection (PCI) bus.

[0284] The computer device 12 typically includes a variety of computer system readable media. These media can be any available media that can be accessed by the computer device 12, including volatile and non-volatile media, removable and non-removable media.

[0285] The memory 28 may include computer system readable media in the form of volatile memory, such as random access memory (RAM) 30 and / or cache memory 32. The computer device 12 may further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, the storage system 34 may be used to read and write non-removable, non-volatile magnetic media ( Fig.17 Not shown, often called a "hard drive").

[0286] although Fig.17 Not shown in the figure, a disk drive for reading and writing a removable non-volatile disk (e.g., a "floppy disk"), and an optical disk drive for reading and writing a removable non-volatile optical disk (e.g., a compact disc read only memory (CD-ROM), a digital versatile disc read only memory (DVD-ROM), or other optical media) may be provided. In these cases, each drive may be connected to the bus 18 via one or more data medium interfaces. The memory 28 may include at least one program product having a set (e.g., at least one) of program modules configured to perform the functions of the various embodiments of the present disclosure.

[0287] A program / utility 40 having a set (at least one) of program modules 42 may be stored, for example, in the memory 28, such program modules 42 including but not limited to an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment. The program modules 42 generally perform the functions and / or methods of the embodiments described in the present disclosure.

[0288] The computer device 12 may also communicate with one or more external devices 14 (e.g., keyboard, pointing device, display 24, etc.), one or more devices that enable a user to interact with the computer device 12, and / or any device that enables the computer device 12 to communicate with one or more other computing devices (e.g., network card, modem, etc.). Such communication may be performed via an input / output (I / O) interface 22. In addition, the computer device 12 may also communicate with one or more networks (e.g., a local area network (LAN), a wide area network (WAN), and / or a public network, such as the Internet) via a network adapter 20. As shown, the network adapter 20 communicates with other modules of the computer device 12 via a bus 18. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with the computer device 12, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0289] The processing unit 16 executes various functional applications and data processing by running the programs stored in the system memory 28, such as implementing the image semantic segmentation method mentioned in the above embodiment.

[0290] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the disclosure disclosed herein. The present disclosure is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The description and examples are to be considered exemplary only, and the true scope and spirit of the present disclosure are indicated by the following claims.

[0291] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

[0292] It should be noted that, in the description of the present disclosure, the terms "first", "second", etc. are only used for descriptive purposes and cannot be understood as indicating or implying relative importance. In addition, in the description of the present disclosure, unless otherwise specified, the meaning of "plurality" is two or more.

[0293] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, segment or portion of code that includes one or more executable instructions for implementing the steps of a specific logical function or process, and the scope of the preferred embodiments of the present disclosure includes alternative implementations in which functions may not be performed in the order shown or discussed, including performing functions in a substantially simultaneous manner or in the reverse order depending on the functions involved, which should be understood by those skilled in the art to which the embodiments of the present disclosure belong.

[0294] It should be understood that the various parts of the present disclosure can be implemented in hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, multiple steps or methods can be implemented in software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0295] A person skilled in the art may understand that all or part of the steps in the method for implementing the above-mentioned embodiment may be completed by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, which, when executed, includes one or a combination of the steps of the method embodiment.

[0296] In addition, each functional unit in each embodiment of the present disclosure may be integrated into a processing module, or each unit may exist physically separately, or two or more units may be integrated into one module. The above-mentioned integrated module may be implemented in the form of hardware or in the form of a software functional module. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it may also be stored in a computer-readable storage medium.

[0297] The storage medium mentioned above can be a read-only memory, a magnetic disk or an optical disk, etc.

[0298] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present disclosure. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.

[0299] Although the embodiments of the present disclosure have been shown and described above, it is to be understood that the above embodiments are exemplary and are not to be construed as limitations of the present disclosure. A person skilled in the art may change, modify, replace and vary the above embodiments within the scope of the present disclosure.

Claims

1. A method for image semantic segmentation, characterized in that: include: Acquire an initial image, wherein the initial image corresponds to initial feature information; Processing the initial image to obtain multiple frames of images to be segmented, wherein the multiple frames of images to be segmented correspond to multiple spatial feature information respectively; Determine the pixel semantic features of the image to be segmented based on the corresponding spatial feature information, wherein the pixel semantic features are obtained by identifying the semantic segmentation model corresponding to the image to be segmented; the semantic segmentation model includes: a plurality of cascaded convolution blocks, some of the cascaded convolution blocks constitute a convolution set, and the convolution block includes: a feature recognition channel; wherein the convolution block in the convolution set is used to perform a feature recognition task on the image to be segmented belonging to the semantic segmentation model based on the feature recognition channel; fusing a plurality of the pixel semantic features to obtain a semantic distribution feature; and Processing the semantic distribution feature according to the initial feature information to obtain an image semantic feature; Wherein, the number of the semantic segmentation models is dynamically adjustable; and / or The number of convolution blocks in the convolution set is dynamically adjustable; and / or The number of the feature recognition channels is dynamically adjustable.

2. The method according to claim 1, characterized in that The fusing of the plurality of pixel semantic features to obtain a semantic distribution feature includes: Determine a plurality of reference interaction information corresponding to the plurality of semantic segmentation models respectively, wherein the reference interaction information is used to describe the reference weight of the corresponding pixel semantic feature; fusing the plurality of pixel semantic features according to the reference interaction information to obtain a target semantic feature; and The semantic distribution feature is obtained according to the target semantic feature.

3. The method according to claim 1, characterized in that Also includes: Determine demand segmentation performance; Determining a first target number of convolution blocks in the convolution set according to the required segmentation performance, and determining a second target number of the feature recognition channels; as well as The semantic segmentation model is formed, wherein the semantic segmentation model includes: a convolution set consisting of the first target number of convolution blocks, and the convolution blocks include: the second target number of feature recognition channels.

4. The method according to claim 3, characterized in that The step of determining a first target number of convolution blocks in the convolution set and determining a second target number of feature recognition channels according to the required segmentation performance includes: determining candidate expansion parameters; Determining a first initial number of convolution blocks in the convolution set, and determining a second initial number of feature recognition channels; Performing expansion processing on the first initial number and / or the second initial number based on the candidate expansion parameter to obtain a first expanded number and / or a second expanded number; Predicting a predicted segmentation performance of the extended segmentation model, wherein the extended segmentation model comprises: a convolution set consisting of the first extended number of convolution blocks, and / or the convolution block comprises: the second extended number of feature recognition channels; and If the predicted segmentation performance meets the required segmentation performance, the first expansion quantity is used as the first target quantity, and / or the second expansion quantity is used as the second target quantity.

5. The method according to claim 4, characterized in that The determining of the candidate extension parameters includes: Determine a depth dimension parameter and a width dimension parameter, wherein the depth dimension parameter is used to determine the number of convolution blocks in the convolution set, and the width dimension parameter is used to determine the number of feature recognition channels; The depth dimension parameter and the width dimension parameter are used as the candidate extension parameters.

6. The method according to claim 5, characterized in that The determining of the candidate extension parameters further includes: Determining a spatial feature parameter as the candidate extension parameter; Wherein, the method further comprises: Determining multiple spatial feature information according to the spatial feature parameters; According to the multiple spatial feature information, corresponding multiple semantic segmentation models are configured respectively, so that the semantic segmentation model can identify the pixel semantic features of the corresponding image to be segmented, wherein the spatial feature information of the corresponding image to be segmented is the same as the spatial feature information configured for the semantic segmentation model.

7. The method according to claim 1, characterized in that The determining of the pixel semantic features of the image to be segmented based on the corresponding spatial feature information includes: Processing the image to be segmented based on the convolution blocks in the first convolution set to obtain a first semantic feature, wherein the first convolution set includes: a first convolution block in the cascaded multiple convolution blocks; The feature recognition task is performed on the first semantic feature based on the convolution block in the second convolution set to obtain the pixel semantic feature, wherein the convolution block at the first position of the cascade in the second convolution set is after the convolution block at the last position of the cascade in the first convolution set.

8. The method according to claim 7, characterized in that The step of processing the image to be segmented based on the convolution block in the first convolution set to obtain the first semantic feature includes: Determining initial pixel features of the image to be segmented; Reducing the dimension of the initial pixel feature based on the first convolution block in the cascade in the first convolution set to obtain a first reduced-dimensional pixel feature; and The first reduced-dimensional pixel feature is processed based on other convolution blocks in the first convolution set to obtain the first semantic feature.

9. The method according to claim 7, characterized in that The performing the feature recognition task on the first semantic feature based on the convolution block in the second convolution set to obtain the pixel semantic feature includes: Based on the first convolution block in the cascade in the second convolution set, the first semantic feature is reduced in dimension to obtain a second reduced-dimensional pixel feature; The second reduced-dimensionality pixel feature is processed based on other convolution blocks in the second convolution set to obtain the pixel semantic feature.

10. The method according to claim 7, characterized in that in, There are multiple second convolution sets, and information exchange models are set between different second convolution sets. The input end of the information exchange model is connected to the last convolution block in the cascade in the previous second convolution set, and the output end of the information exchange model is connected to the first convolution block in the cascade in the next second convolution set.

11. The method according to claim 10, characterized in that Also includes: Based on the information exchange model, the first pixel semantic feature and the second pixel semantic feature are bidirectionally interactively processed to obtain a fused pixel semantic feature, wherein the first pixel semantic feature is the pixel semantic feature output by the previous second convolution set, and the second pixel semantic feature is the pixel semantic feature output by the corresponding second convolution set in other semantic segmentation models.

12. The method according to claim 11, characterized in that The performing bidirectional interactive processing on the first pixel semantic feature and the second pixel semantic feature based on the information exchange model to obtain a fused pixel semantic feature includes: Performing spatial upsampling processing on the first pixel semantic feature based on the information exchange model to obtain a first sampled semantic feature; Performing spatial downsampling processing on the second pixel semantic feature based on the information exchange model to obtain a second sampled semantic feature, wherein the fineness of the spatial feature information of the image to be segmented to which the first pixel semantic feature belongs is greater than the fineness of the spatial feature information of the image to be segmented to which the second pixel semantic feature belongs; and The first sampled semantic feature and the second sampled semantic feature are fused to obtain the fused pixel semantic feature.

13. The method according to claim 11, characterized in that After the first pixel semantic feature and the second pixel semantic feature are subjected to bidirectional interactive processing based on the information exchange model to obtain a fused pixel semantic feature, the method further includes: Inputting the fused pixel semantic features into the latter second convolution set; Processing the fused pixel semantic features based on the convolution blocks in the latter second convolution set to obtain pixel semantic features output by the latter second convolution set; and The latter second convolution set is used as the former second convolution set until the latter second convolution set is the last second convolution set among the multiple second convolution sets.

14. The method according to claim 13, characterized in that The fusing of the plurality of pixel semantic features to obtain a semantic distribution feature includes: If it is determined that the latter second convolution set is the last second convolution set, a plurality of the fused pixel semantic features are processed to obtain the semantic distribution feature.

15. The method according to claim 1, wherein: The processing of the semantic distribution feature according to the initial feature information to obtain the image semantic feature includes: The semantic distribution feature is subjected to bilinear upsampling processing according to the initial feature information to obtain the image semantic feature.

16. An image semantic segmentation device, characterized in that: The device comprises: An acquisition module, used for acquiring an initial image, wherein the initial image corresponds to initial feature information; A first processing module is used to process the initial image to obtain multiple frames of images to be segmented, wherein the multiple frames of images to be segmented correspond to multiple spatial feature information respectively; A first determination module is used to determine the pixel semantic features of the image to be segmented based on the corresponding spatial feature information, wherein the pixel semantic features are obtained by identifying the semantic segmentation model corresponding to the image to be segmented; the semantic segmentation model includes: a plurality of cascaded convolution blocks, some of the cascaded convolution blocks constitute a convolution set, and the convolution block includes: a feature recognition channel; wherein the convolution block in the convolution set is used to perform a feature recognition task on the image to be segmented belonging to the semantic segmentation model based on the feature recognition channel; A fusion module, used for fusing the plurality of pixel semantic features to obtain a semantic distribution feature; and A second processing module, used for processing the semantic distribution feature according to the initial feature information to obtain an image semantic feature; in, The number of the semantic segmentation models is dynamically adjustable; and / or The number of convolution blocks in the convolution set is dynamically adjustable; and / or The number of the feature recognition channels is dynamically adjustable.

17. The device according to claim 16, characterized in that The fusion module is specifically used for: Determine a plurality of reference interaction information corresponding to the plurality of semantic segmentation models respectively, wherein the reference interaction information is used to describe the reference weight of the corresponding pixel semantic feature; fusing the plurality of pixel semantic features according to the reference interaction information to obtain a target semantic feature; and The semantic distribution feature is obtained according to the target semantic feature.

18. The device according to claim 16, characterized in that Also includes: A second determination module is used to determine demand segmentation performance; A third determination module is used to determine a first target number of convolution blocks in the convolution set and a second target number of feature recognition channels according to the required segmentation performance; as well as A generation module is used to form the semantic segmentation model, wherein the semantic segmentation model includes: a convolution set consisting of the first target number of convolution blocks, and the convolution block includes: the second target number of feature recognition channels.

19. The device according to claim 18, characterized in that The third determination module is specifically used to: determining candidate expansion parameters; Determining a first initial number of convolution blocks in the convolution set, and determining a second initial number of feature recognition channels; Performing expansion processing on the first initial number and / or the second initial number based on the candidate expansion parameter to obtain a first expanded number and / or a second expanded number; Predicting a predicted segmentation performance of the extended segmentation model, wherein the extended segmentation model comprises: a convolution set consisting of the first extended number of convolution blocks, and / or the convolution block comprises: the second extended number of feature recognition channels; and If the predicted segmentation performance meets the required segmentation performance, the first expansion quantity is used as the first target quantity, and / or the second expansion quantity is used as the second target quantity.

20. The device according to claim 19, characterized in that The third determining module is further used for: Determine a depth dimension parameter and a width dimension parameter, wherein the depth dimension parameter is used to determine the number of convolution blocks in the convolution set, and the width dimension parameter is used to determine the number of feature recognition channels; The depth dimension parameter and the width dimension parameter are used as the candidate extension parameters.

21. The device according to claim 20, characterized in that The third determining module is further used for: Determining a spatial feature parameter as the candidate extension parameter; Wherein, the third determination module is further used for: Determining multiple spatial feature information according to the spatial feature parameters; According to the multiple spatial feature information, corresponding multiple semantic segmentation models are configured respectively, so that the semantic segmentation model can identify the pixel semantic features of the corresponding image to be segmented, wherein the spatial feature information of the corresponding image to be segmented is the same as the spatial feature information configured for the semantic segmentation model.

22. The device according to claim 16, characterized in that The first determining module is specifically configured to: Processing the image to be segmented based on the convolution blocks in the first convolution set to obtain a first semantic feature, wherein the first convolution set includes: a first convolution block in the cascaded multiple convolution blocks; The feature recognition task is performed on the first semantic feature based on the convolution block in the second convolution set to obtain the pixel semantic feature, wherein the convolution block at the first position of the cascade in the second convolution set is after the convolution block at the last position of the cascade in the first convolution set.

23. The device according to claim 22, characterized in that The first determining module is further used for: Determining initial pixel features of the image to be segmented; Reducing the dimension of the initial pixel feature based on the first convolution block in the cascade in the first convolution set to obtain a first reduced-dimensional pixel feature; and The first reduced-dimensional pixel feature is processed based on other convolution blocks in the first convolution set to obtain the first semantic feature.

24. The device according to claim 22, characterized in that The first determining module is further used for: Based on the first convolution block in the cascade in the second convolution set, the first semantic feature is reduced in dimension to obtain a second reduced-dimensional pixel feature; The second reduced-dimensionality pixel feature is processed based on other convolution blocks in the second convolution set to obtain the pixel semantic feature.

25. The device according to claim 22, characterized in that in, There are multiple second convolution sets, and information exchange models are set between different second convolution sets. The input end of the information exchange model is connected to the last convolution block in the cascade in the previous second convolution set, and the output end of the information exchange model is connected to the first convolution block in the cascade in the next second convolution set.

26. The device according to claim 25, characterized in that Also includes: The third processing module is used to perform bidirectional interactive processing on the first pixel semantic feature and the second pixel semantic feature based on the information exchange model to obtain a fused pixel semantic feature, wherein the first pixel semantic feature is the pixel semantic feature output by the previous second convolution set, and the second pixel semantic feature is the pixel semantic feature output by the corresponding second convolution set in other semantic segmentation models.

27. The device according to claim 26, characterized in that The third processing module is specifically used for: Performing spatial upsampling processing on the first pixel semantic feature based on the information exchange model to obtain a first sampled semantic feature; Performing spatial downsampling processing on the second pixel semantic feature based on the information exchange model to obtain a second sampled semantic feature, wherein the fineness of the spatial feature information of the image to be segmented to which the first pixel semantic feature belongs is greater than the fineness of the spatial feature information of the image to be segmented to which the second pixel semantic feature belongs; and The first sampled semantic feature and the second sampled semantic feature are fused to obtain the fused pixel semantic feature.

28. The device according to claim 26, characterized in that Also includes: An input module, configured to, after bidirectionally interactively processing the first pixel semantic feature and the second pixel semantic feature based on the information exchange model to obtain a fused pixel semantic feature, input the fused pixel semantic feature into the latter second convolution set; A fourth processing module, configured to process the fused pixel semantic features based on the convolution blocks in the latter second convolution set to obtain the pixel semantic features output by the latter second convolution set; as well as The fifth processing module is used to use the latter second convolution set as the former second convolution set until the latter second convolution set is the last second convolution set among the multiple second convolution sets.

29. The device according to claim 28, characterized in that The fusion module is further used for: If it is determined that the latter second convolution set is the last second convolution set, a plurality of the fused pixel semantic features are processed to obtain the semantic distribution feature.

30. The device according to claim 16, characterized in that The second processing module is specifically used for: The semantic distribution feature is subjected to bilinear upsampling processing according to the initial feature information to obtain the image semantic feature.

31. A computer device, characterized in that: include: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the image semantic segmentation method described in any one of claims 1-15.

32. A non-transitory computer-readable storage medium storing computer instructions, characterized in that: in, The computer instructions are used to enable the computer to execute the image semantic segmentation method according to any one of claims 1 to 15.

Images