Model construction method, image semantic segmentation method, device, equipment and medium

Abstract

The embodiment of the present invention discloses a model construction method, image semantic segmentation method, device, equipment and medium. The method includes: obtaining a basic model structure of a model to be constructed, the basic model structure including a backbone neural network, a convolution layer, a feature splicing layer and a segmentation network, and the segmentation network including a pixel reconstruction module; determining the number of convolution layer channels and the number of completion channels according to the number of data channels of the output feature data of the pixel reconstruction module; selecting an intermediate layer from the backbone neural network as a feature completion layer according to the number of completion channels, connecting the feature completion layer and the convolution layer to the feature splicing layer respectively, and connecting the feature splicing layer to the segmentation network to obtain an image semantic segmentation model. This method realizes that before the network model is trained, the constructed model meets the requirements of pixel reconstruction features while reducing the number of convolution layer channels, fully reuses the intermediate data results of the neural network model to reduce the amount of calculation, and achieves the purpose of improving speed.

Description

Technical Field

[0001] The present invention relates to the field of image segmentation technology, and more particularly to a model construction method, an image semantic segmentation method, an apparatus, a device, and a medium. Background Art

[0002] Super-resolution is a common technique in computer vision. In image semantic segmentation tasks, pixel shuffling is a common super-resolution operation. Pixel shuffle requires the previous layer to output a large number of channels. In deep learning models, convolutional layers are generally used to increase the number of channels and simultaneously obtain specific values ​​for the channels and feature maps. Existing methods for optimizing the speed of the aforementioned pixel shuffle include the following three methods: 1. Reducing the number of output channels of the convolutional layer; 2. Using smaller feature maps; and 3. First increasing the feature map size using upsampling before performing the pixel shuffle. However, reducing the number of output channels of the convolutional layer and using smaller feature maps changes the data size of the preceding Pixel Shuffle operation (the convolution operation), resulting in a lower-than-expected resolution of the image generated by the pixel shuffle. Increasing the feature map size using upsampling before performing the pixel shuffle requires more rule-based post-processing to achieve the required image resolution, resulting in lower performance than using the pixel shuffle directly.

[0003] It can be seen that in the implementation process of the relevant technology, there are at least the following technical problems: it is impossible to simultaneously take into account the output image size and the output image effect when improving the pixel reconstruction speed. Summary of the Invention

[0004] The embodiments of the present invention provide a model construction method, image semantic segmentation method, device, equipment and medium to ensure image effect without affecting the image size output by the pixel reconstruction module.

[0005] In a first aspect, an embodiment of the present invention provides a method for constructing an image semantic segmentation model, comprising:

[0006] Obtain the basic model structure of the model to be built. The basic model structure includes a backbone neural network, a convolutional layer, a feature splicing layer, and a segmentation network. The segmentation network includes a pixel reconstruction module.

[0007] Determine the number of convolutional layer channels and the number of completion channels based on the number of data channels of the output feature data of the pixel reconstruction module;

[0008] According to the number of completion channels, the middle layer is selected from the backbone neural network as the feature completion layer. The feature completion layer and the convolution layer are respectively connected to the feature splicing layer, and the feature splicing layer is connected to the segmentation network to obtain the image semantic segmentation model.

[0009] Optionally, the number of convolutional layer channels and the number of completion channels are determined according to the number of data channels of the feature data of the pixel reconstruction module, including:

[0010] Any positive integer less than the number of data channels is used as the number of convolutional layer channels;

[0011] The difference between the number of data channels and the number of convolutional layer channels is used as the number of complementary channels.

[0012] Optionally, an intermediate layer is selected from the backbone neural network as a feature completion layer based on the number of completion channels, including:

[0013] At least one intermediate layer in the backbone neural network is selected as a feature completion layer, wherein the total number of channels of the at least one intermediate layer is equal to the number of completion channels.

[0014] Optionally, the resolution of the output feature map of the feature completion layer is the same as the resolution of the output feature map of the pixel reconstruction module.

[0015] Optionally, the method further includes:

[0016] Obtain training sample data;

[0017] The image semantic segmentation model is iteratively trained based on the training sample data, and when the training end condition is met, a trained image semantic segmentation model is obtained.

[0018] In a second aspect, an embodiment of the present invention further provides an image semantic segmentation method, characterized by comprising:

[0019] Get the image to be semantically segmented;

[0020] The image to be semantically segmented is input into a pre-trained image semantic segmentation model to obtain a semantic segmentation result of the image semantic segmentation model, wherein the image semantic segmentation model is trained based on the image semantic segmentation model constructed according to the image semantic segmentation model construction method provided in the first aspect of the embodiment of the present invention.

[0021] In a third aspect, an embodiment of the present invention further provides a device for constructing an image semantic segmentation model, comprising:

[0022] The basic structure acquisition module is used to obtain the basic model structure of the model to be built. The basic model structure includes a backbone neural network, a convolutional layer, a feature splicing layer, and a segmentation network. The segmentation network includes a pixel reconstruction module.

[0023] A channel number determination module is used to determine the number of convolutional layer channels and the number of completion channels according to the number of data channels of the output feature data of the pixel reconstruction module;

[0024] The semantic segmentation model construction module is used to select the middle layer from the backbone neural network as the feature completion layer according to the number of completion channels, connect the feature completion layer and the convolution layer to the feature splicing layer respectively, and connect the feature splicing layer to the segmentation network to obtain the image semantic segmentation model.

[0025] In a fourth aspect, an embodiment of the present invention further provides an image semantic segmentation apparatus, comprising:

[0026] A module for acquiring images to be semantically segmented, used for acquiring images to be semantically segmented;

[0027] An image semantic segmentation module is used to input the image to be semantically segmented into a pre-trained image semantic segmentation model to obtain the semantic segmentation result of the image semantic segmentation model, wherein the image semantic segmentation model is trained based on the image semantic segmentation model constructed according to the image semantic segmentation model construction method provided in the first aspect of the embodiment of the present invention.

[0028] In a fifth aspect, an embodiment of the present invention further provides a computer device, comprising:

[0029] one or more processors;

[0030] a storage device for storing one or more programs;

[0031] When one or more programs are executed by one or more processors, the one or more processors implement the method for constructing an image semantic segmentation model as provided in any embodiment of the present invention, and / or implement the image semantic segmentation method as provided in any embodiment of the present invention.

[0032] In a sixth aspect, an embodiment of the present invention further provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements a method for constructing an image semantic segmentation model as provided in any embodiment of the present invention, and / or implements an image semantic segmentation method as provided in any embodiment of the present invention.

[0033] The method for constructing an image semantic segmentation model provided by an embodiment of the present invention obtains the basic model structure of the model to be constructed, the basic model structure including a backbone neural network, a convolution layer, a feature splicing layer and a segmentation network, and the segmentation network includes a pixel reconstruction module; the number of convolution layer channels and the number of completion channels are determined according to the number of data channels of the output feature data of the pixel reconstruction module; the intermediate layer is selected from the backbone neural network as the feature completion layer according to the number of completion channels, the feature completion layer and the convolution layer are respectively connected to the feature splicing layer, and the feature splicing layer is connected to the segmentation network to obtain an image semantic segmentation model. This method realizes that before the network model is trained, the constructed image semantic segmentation model meets the requirements of pixel reconstruction features while reducing the number of convolution layer channels, fully reuses the intermediate data results of the neural network model to reduce the amount of calculation, and achieves the purpose of improving speed. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 This is a flow chart of a method for constructing an image semantic segmentation model provided by an embodiment of the present invention;

[0035] Figure 2 1 is a flow chart of an image semantic segmentation method provided by an embodiment of the present invention;

[0036] Figure 3 Schematic diagram of a device for constructing an image semantic segmentation model provided by an embodiment of the present invention;

[0037] Figure 4 1 is a structural diagram of an image semantic segmentation device provided by an embodiment of the present invention;

[0038] Figure 5 It is a structural diagram of a computer device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0039] The present invention will be further described in detail below with reference to the accompanying drawings and examples. It will be understood that the specific embodiments described herein are intended only to illustrate the present invention and are not intended to limit the present invention. It should also be noted that, for ease of description, the accompanying drawings only illustrate portions relevant to the present invention, not all structures.

[0040] Figure 1 This is a flow chart of a method for constructing an image semantic segmentation model provided by an embodiment of the present invention. This embodiment is applicable to situations when constructing an image semantic segmentation model. The method can be executed by a device for constructing an image semantic segmentation model, and the device for constructing an image semantic segmentation model can be implemented in software and / or hardware. For example, the device for constructing an image semantic segmentation model can be configured in a computer device. Figure 1 As shown, the method includes:

[0041] S110. Obtain a basic model structure of a model to be constructed, where the basic model structure includes a backbone neural network, a convolutional layer, a feature splicing layer, and a segmentation network, and the segmentation network includes a pixel reconstruction module.

[0042] In this embodiment, the model to be constructed is an image semantic segmentation model. Generally speaking, the basic model structure of the image semantic segmentation model includes a backbone neural network, a convolutional layer and a segmentation network, and the segmentation network includes a pixel reconstruction module.

[0043] In order to solve the technical problem of irregular imaging size or poor imaging effect caused by reducing the amount of calculation in the prior art. In this embodiment, a feature splicing layer is added between the convolution layer and the segmentation network. In order to reduce the amount of calculation, the number of channels of the convolution layer can be reduced. At the same time, in order to ensure the effect of the image, it is necessary to use the output features of the intermediate layer to complement the output features of the convolution layer, so that the number of channels of the input image features of the pixel reconstruction module meets the number of channels required by the pixel reconstruction module. It can be understood that the feature splicing layer is used to splice the features output by the convolution layer and other layers, so that the spliced ​​features meet the data requirements of the pixel reconstruction module.

[0044] The backbone neural network can be selected according to actual needs. For example, the backbone neural network can be a residual neural network, a convolutional neural network (such as ResNet18, ResNet50, ResNet101, Inception, MobileNet), etc., which are not limited here. The segmentation network can be selected according to actual needs. For example, the segmentation network can be a BiSeNet v1, BiSeNet v2, PSPnet, etc., where the last layer of the segmentation network is the pixel reorganization module Pixel Shuffle.

[0045] S120. Determine the number of convolutional layer channels and the number of completion channels according to the number of data channels of the output feature data of the pixel reconstruction module.

[0046] Optionally, the number of convolutional layer channels and the number of channels to be supplemented can be set based on the number of data channels of the output features of the pixel reconstruction module. Specifically, the number of convolutional layer channels can be determined based on the number of data channels of the output features of the pixel reconstruction module and the available computational load. The number of supplemented channels can then be determined based on the number of data channels of the output features of the pixel reconstruction module and the number of convolutional layer channels.

[0047] In one embodiment, the number of convolutional layer channels and the number of complementary channels are determined based on the number of data channels of the feature data of the pixel reconstruction module, including: taking any positive integer less than the number of data channels as the number of convolutional layer channels; taking the difference between the number of data channels and the number of convolutional layer channels as the number of complementary channels. Optionally, any positive integer less than the number of data channels can be set as the number of convolutional layer channels. In order to be able to complement the number of missing channels of the convolution layer, the difference between the number of data channels and the number of convolutional layer channels is used as the number of complementary channels. Exemplarily, assuming that the size of the feature data of the pixel reconstruction module is C_out×H_out×W_out, and the number of data channels of the feature data is C_out, then any positive integer less than C_out can be selected as the number of convolutional layer channels. Assuming that the number of convolutional layer channels is C_conv, the number of complementary channels is C_out-C_conv.

[0048] S130. Select an intermediate layer from the backbone neural network as a feature completion layer according to the number of completion channels, connect the feature completion layer and the convolution layer to the feature splicing layer respectively, and connect the feature splicing layer to the segmentation network to obtain an image semantic segmentation model.

[0049] In this embodiment, after determining the number of channels in the convolutional layer, the structure of the convolutional layer can be determined. Other parameters of the convolutional layer, such as the convolution kernel size, can be set according to actual needs.

[0050] After determining the number of completion channels, an intermediate layer with the same number of channels as the completion channels is selected from the intermediate layers of the backbone neural network as the feature completion layer. The convolutional layer and the feature completion layer are connected to the feature splicing layer, and the feature splicing layer is connected to the pixel reconstruction module to obtain the image semantic segmentation model.

[0051] It should be noted that, in this embodiment, one intermediate layer can be selected from the backbone neural network as the feature completion layer, or multiple intermediate layers can be selected from the backbone neural network as the feature completion layer, as long as the total number of channels of the feature completion layer is equal to the number of completion channels. That is to say, selecting an intermediate layer from the backbone neural network as the feature completion layer according to the number of completion channels includes: selecting at least one intermediate layer in the backbone neural network as the feature completion layer, wherein the total number of channels of at least one intermediate layer is equal to the number of completion channels. Exemplarily, assuming that an intermediate layer A is selected from the backbone neural network as the feature completion layer, the number of channels of intermediate layer A is equal to the number of completion channels. Assuming that intermediate layers B, C and D are selected from the backbone neural network as the feature completion layers, the sum of the number of channels of intermediate layer B, the number of channels of intermediate layer C and the number of channels of intermediate layer D is equal to the number of completion channels.

[0052] Generally speaking, processing features from multiple layers can avoid the limitations of single-layer output features. Therefore, it is preferable to select at least two intermediate layers from the backbone neural network as feature completion layers, and the total number of channels in at least two intermediate layers should be equal to the number of completion channels.

[0053] On the basis of the above scheme, in order to reduce the amount of calculation and ensure the image effect, the feature size that needs to be completed is the same as the feature size of the pixel reconstruction module. Based on this, the method for constructing the image semantic segmentation model provided by this embodiment also includes: the resolution of the output feature map of the feature completion layer is the same as the resolution of the output feature map of the pixel reconstruction module. That is to say, assuming that the feature data size of the pixel reconstruction module is C_out×H_out×W_out, the number of data channels of the feature data is C_out, and the feature map resolution is H_out×W_out; then it is necessary to select at least one intermediate layer with a feature map resolution of H_out×W_out and a total number of channels equal to the number of completion channels as the feature completion layer. Assuming that the number of completion channels is Cbq, the intermediate layer C1 and the intermediate layer C2 are selected as the feature completion layers, then the output feature size of the intermediate layer C1 is C1×H_out×W_out, the output feature size of the intermediate layer C2 is C2×H_out×W_out, and C1+C2=Cbq.

[0054] After determining the number of convolutional layer channels and completing the feature layer and connecting the layers, the constructed image semantic segmentation model can be obtained.

[0055] The method for constructing an image semantic segmentation model provided by an embodiment of the present invention obtains the basic model structure of the model to be constructed, the basic model structure including a backbone neural network, a convolution layer, a feature splicing layer and a segmentation network, and the segmentation network includes a pixel reconstruction module; the number of convolution layer channels and the number of completion channels are determined according to the number of data channels of the output feature data of the pixel reconstruction module; the intermediate layer is selected from the backbone neural network as the feature completion layer according to the number of completion channels, the feature completion layer and the convolution layer are respectively connected to the feature splicing layer, and the feature splicing layer is connected to the segmentation network to obtain an image semantic segmentation model. This method realizes that before the network model is trained, the constructed image semantic segmentation model meets the requirements of pixel reconstruction features while reducing the number of convolution layer channels, fully reuses the intermediate data results of the neural network model to reduce the amount of calculation, and achieves the purpose of improving speed.

[0056] Based on the above scheme, the method also includes: obtaining training sample data; iteratively training the image semantic segmentation model based on the training sample data, and obtaining a trained image semantic segmentation model when the training end conditions are met.

[0057] Among them, the training sample data can be determined based on the application scenario of the image semantic segmentation model. For example, if the application scenario of the image semantic segmentation model is an autonomous driving scenario, the sample data can be image sample data in the driving scenario.

[0058] In one embodiment, the training sample data may include training data and supervisory data corresponding to the training data. Accordingly, supervised training is performed on the network model based on the training data and the corresponding supervisory data. The network model is subjected to supervised iterative training based on the sample data until a training termination condition is satisfied. The training termination condition may be any one of the following: the network model has been trained a preset number of times, the network model training process has reached a convergence state, or the network model training accuracy has reached a preset accuracy threshold.

[0059] In one embodiment, the sample data may include first sample data provided with supervised data and second training data not provided with supervised data. Accordingly, the network model is semi-supervisedly trained based on the first and second sample data, simplifying the process of determining supervised data in the sample data. Specifically, the sample data is divided into multiple sets of training data, each set of sample data is input into the network model multiple times, and multiple prediction results are obtained. Based on the prediction consistency principle of the network model, the network model is iteratively trained until the training end condition is met, thereby obtaining a trained image semantic segmentation model.

[0060] The image semantic segmentation model constructed based on the image semantic segmentation model construction method provided by this embodiment has a small amount of calculation and accelerates the model training speed.

[0061] Figure 2 This is a flow chart of an image semantic segmentation method provided by an embodiment of the present invention. This embodiment is applicable to situations where semantic segmentation of an image is performed. The method can be executed by an image semantic segmentation device, which can be implemented in software and / or hardware. For example, the image semantic segmentation device can be configured in a computer device. Figure 2 As shown, the method includes:

[0062] S210: Obtain an image to be semantically segmented.

[0063] In this embodiment, the image to be semantically segmented can be determined according to the application scenario. For example, assuming that the application scenario is an autonomous driving scenario, the image to be semantically segmented can be an image taken by an autonomous driving vehicle.

[0064] S220: Input the image to be semantically segmented into a pre-trained image semantic segmentation model to obtain a semantic segmentation result of the image semantic segmentation model.

[0065] The image semantic segmentation model is trained based on an image semantic segmentation model constructed using the image semantic segmentation model construction method provided in any embodiment of the present invention. The image semantic segmentation model constructed using the image semantic segmentation model construction method provided in an embodiment of the present invention is used to process images to be semantically segmented, thereby increasing the processing speed of the images to be semantically segmented while ensuring image processing quality.

[0066] Taking the autonomous driving scenario as an example, the image semantic segmentation model provided by the embodiments of the present invention can be set in the "perception" module of the autonomous vehicle's front-end perception system. In the entire autonomous driving software system, the image semantic segmentation model is used to understand the semantics of objects in the semantically segmented image, giving the autonomous driving software system "eyes" to see objects such as vehicles, obstacles, buildings, isolation fences, pedestrians, etc. in front of it, thereby providing the "navigation planning" module with vehicle trajectory planning and reasonable obstacle avoidance, acceleration and deceleration planning. Improving the processing speed of the semantically segmented image can increase the speed of path planning in navigation, allowing timely adjustments to the vehicle's operating status during autonomous driving, making vehicle operation more controllable.

[0067] The image semantic segmentation method provided by an embodiment of the present invention obtains an image to be semantically segmented; inputs the image to be semantically segmented into a pre-trained image semantic segmentation model, and obtains a semantic segmentation result of the image semantic segmentation model, wherein the image semantic segmentation model is trained based on an image semantic segmentation model constructed using the method for constructing an image semantic segmentation model provided by any embodiment of the present invention. By reducing the number of convolutional layer channels in the image semantic segmentation model, the speed of image semantic segmentation based on the image semantic segmentation model is accelerated, and feature completion is performed through a feature completion layer to ensure the accuracy of image semantic segmentation.

[0068] Figure 3 This is a schematic diagram of the structure of a device for constructing an image semantic segmentation model provided by an embodiment of the present invention. The device for constructing the image semantic segmentation model can be implemented in software and / or hardware. For example, the device for constructing the image semantic segmentation model can be configured in a computer device. In this embodiment, the explanations of terms that are the same as or corresponding to the above embodiments are not repeated here. Figure 3 As shown, the apparatus includes a basic structure acquisition module 310, a channel number determination module 320 and a semantic segmentation model construction module 330, wherein:

[0069] A basic structure acquisition module 310 is used to obtain the basic model structure of the model to be constructed, wherein the basic model structure includes a backbone neural network, a convolution layer, a feature splicing layer, and a segmentation network, and the segmentation network includes a pixel reconstruction module;

[0070] A channel number determination module 320 is configured to determine the number of convolutional layer channels and the number of complement channels according to the number of data channels of the output feature data of the pixel reconstruction module;

[0071] The semantic segmentation model construction module 330 is used to select an intermediate layer from the backbone neural network as a feature completion layer according to the number of completion channels, connect the feature completion layer and the convolution layer to the feature splicing layer respectively, and connect the feature splicing layer to the segmentation network to obtain an image semantic segmentation model.

[0072] The embodiment of the present invention obtains the basic model structure of the model to be constructed through the basic structure acquisition module. The basic model structure includes a backbone neural network, a convolution layer, a feature splicing layer and a segmentation network. The segmentation network includes a pixel reconstruction module. The channel number determination module determines the number of convolution layer channels and the number of completion channels according to the number of data channels of the output feature data of the pixel reconstruction module. The semantic segmentation model construction module selects an intermediate layer from the backbone neural network as a feature completion layer according to the number of completion channels, connects the feature completion layer and the convolution layer to the feature splicing layer respectively, and connects the feature splicing layer to the segmentation network to obtain an image semantic segmentation model. This achieves that before the network model is trained, the constructed image semantic segmentation model meets the requirements of pixel reconstruction features while reducing the number of convolution layer channels, fully reuses the intermediate data results of the neural network model to reduce the amount of calculation, and achieves the purpose of improving speed.

[0073] Optionally, based on the above solution, the channel quantity determination module 320 is specifically configured to:

[0074] Any positive integer less than the number of data channels is used as the number of convolutional layer channels;

[0075] The difference between the number of data channels and the number of convolutional layer channels is used as the number of complementary channels.

[0076] Optionally, based on the above solution, the semantic segmentation model construction module 330 is specifically configured to:

[0077] At least one intermediate layer in the backbone neural network is selected as a feature completion layer, wherein the total number of channels of the at least one intermediate layer is equal to the number of completion channels.

[0078] Optionally, based on the above solution, the resolution of the output feature map of the feature completion layer is the same as the resolution of the output feature map of the pixel reconstruction module.

[0079] Optionally, based on the above solution, the device further includes a model training module for:

[0080] Obtain training sample data;

[0081] The image semantic segmentation model is iteratively trained based on the training sample data, and when the training end condition is met, a trained image semantic segmentation model is obtained.

[0082] The device for constructing an image semantic segmentation model provided in an embodiment of the present invention can execute the method for constructing an image semantic segmentation model provided in any embodiment of the present invention, and has functional modules and beneficial effects corresponding to the execution method.

[0083] Figure 4 : is a structural diagram of an image semantic segmentation device provided by an embodiment of the present invention. The image semantic segmentation device can be implemented in software and / or hardware. For example, the image semantic segmentation device can be configured in a computer device. In this embodiment, the explanations of the terms that are the same or corresponding to the above embodiments are not repeated here. Figure 4 As shown, the apparatus includes an image acquisition module 410 for semantic segmentation and an image semantic segmentation module 420, wherein:

[0084] The image to be semantically segmented is acquired by a module 410, which is used to acquire the image to be semantically segmented;

[0085] The image semantic segmentation module 420 is used to input the image to be semantically segmented into a pre-trained image semantic segmentation model to obtain the semantic segmentation result of the image semantic segmentation model, wherein the image semantic segmentation model is trained based on the image semantic segmentation model constructed by the image semantic segmentation model construction method provided in any embodiment of the present invention.

[0086] In an embodiment of the present invention, an image to be semantically segmented is acquired through an image acquisition module for semantic segmentation. The image semantic segmentation module inputs the image to be semantically segmented into a pre-trained image semantic segmentation model to obtain a semantic segmentation result of the image semantic segmentation model, wherein the image semantic segmentation model is trained based on an image semantic segmentation model constructed using the method for constructing an image semantic segmentation model provided in any embodiment of the present invention. By reducing the number of convolutional layer channels in the image semantic segmentation model, the speed of image semantic segmentation based on the image semantic segmentation model is accelerated, and feature completion is performed through a feature completion layer to ensure the accuracy of image semantic segmentation.

[0087] The image semantic segmentation device provided by the embodiment of the present invention can execute the image semantic segmentation method provided by any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0088] Figure 5 It is a structural diagram of a computer device provided by an embodiment of the present invention. Figure 5 A block diagram of an exemplary computer device 512 suitable for implementing embodiments of the present invention is shown. Figure 5The computer device 512 shown is only an example and should not limit the functionality and scope of use of the embodiments of the present invention.

[0089] like Figure 5 As shown, computer device 512 is implemented as a general-purpose computing device. Components of computer device 512 may include, but are not limited to, one or more processors 516, a system memory 528, and a bus 518 that connects various system components (including system memory 528 and processor 516).

[0090] Bus 518 represents one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, a local bus to processor 516, or a bus using any of a variety of bus architectures. Examples of these architectures include, but are not limited to, an Industry Standard Architecture (ISA) bus, a Micro Channel Architecture (MAC) bus, an Enhanced ISA bus, a Video Electronics Standards Association (VESA) local bus, and a Peripheral Component Interconnect (PCI) bus.

[0091] The computer device 512 typically includes a variety of computer system readable media. These media can be any available media that can be accessed by the computer device 512, including volatile and non-volatile media, removable and non-removable media.

[0092] System memory 528 may include computer system readable media in the form of volatile memory, such as random access memory (RAM) 550 and / or cache memory 552. Computer device 512 may further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, storage device 554 may be configured to read and write to non-removable, non-volatile magnetic media ( Figure 5 Not shown, often called a "hard drive"). Although Figure 5 Not shown, a disk drive for reading and writing to a removable non-volatile disk (e.g., a "floppy disk"), and an optical drive for reading and writing to a removable non-volatile optical disk (e.g., a CD-ROM, DVD-ROM, or other optical media) may be provided. In these cases, each drive may be connected to bus 518 via one or more data media interfaces. Memory 528 may include at least one program product having a set (e.g., at least one) of program modules configured to perform the functions of various embodiments of the present invention.

[0093] A program / utility 540 having a set (at least one) of program modules 542 may be stored, for example, in memory 528. Such program modules 542 include, but are not limited to, an operating system, one or more application programs, other program modules, and program data, each of which, or some combination thereof, may include an implementation of a network environment. Program modules 542 generally implement the functions and / or methodologies of the embodiments described herein.

[0094] The computer device 512 can also communicate with one or more external devices 514 (e.g., a keyboard, pointing device, display 524, etc.), one or more devices that enable a user to interact with the computer device 512, and / or any device that enables the computer device 512 to communicate with one or more other computing devices (e.g., a network card, a modem, etc.). Such communication can occur via an input / output (I / O) interface 522. Furthermore, the computer device 512 can communicate with one or more networks (e.g., a local area network (LAN), a wide area network (WAN), and / or a public network such as the Internet) via a network adapter 520. As shown, the network adapter 520 communicates with the other modules of the computer device 512 via a bus 518. It should be understood that, although not shown, other hardware and / or software modules can be used in conjunction with the computer device 512, including but not limited to microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0095] The processor 516 executes various functional applications and data processing by running programs stored in the system memory 528, such as implementing the method for constructing an image semantic segmentation model provided in an embodiment of the present invention, which includes:

[0096] Obtain the basic model structure of the model to be built. The basic model structure includes a backbone neural network, a convolutional layer, a feature splicing layer, and a segmentation network. The segmentation network includes a pixel reconstruction module.

[0097] Determine the number of convolutional layer channels and the number of completion channels based on the number of data channels of the output feature data of the pixel reconstruction module;

[0098] According to the number of completion channels, the middle layer is selected from the backbone neural network as the feature completion layer. The feature completion layer and the convolution layer are connected to the feature splicing layer respectively. The feature splicing layer is connected to the segmentation network to obtain the image semantic segmentation model.

[0099] And / or, implementing the image semantic segmentation method provided by an embodiment of the present invention, the method comprising:

[0100] Get the image to be semantically segmented;

[0101] The image to be semantically segmented is input into a pre-trained image semantic segmentation model to obtain a semantic segmentation result of the image semantic segmentation model, wherein the image semantic segmentation model is trained based on the image semantic segmentation model constructed by the image semantic segmentation model construction method provided in any embodiment of the present invention.

[0102] Of course, those skilled in the art will understand that the processor can also implement the method for constructing the image semantic segmentation model provided by any embodiment of the present invention, and / or the technical solution of the image semantic segmentation method.

[0103] An embodiment of the present invention further provides a computer-readable storage medium having a computer program stored thereon. When the program is executed by a processor, the method for constructing an image semantic segmentation model provided by an embodiment of the present invention is implemented. The method includes:

[0104] Obtain the basic model structure of the model to be built. The basic model structure includes a backbone neural network, a convolutional layer, a feature splicing layer, and a segmentation network. The segmentation network includes a pixel reconstruction module.

[0105] Determine the number of convolutional layer channels and the number of completion channels based on the number of data channels of the output feature data of the pixel reconstruction module;

[0106] According to the number of completion channels, the middle layer is selected from the backbone neural network as the feature completion layer. The feature completion layer and the convolution layer are connected to the feature splicing layer respectively. The feature splicing layer is connected to the segmentation network to obtain the image semantic segmentation model.

[0107] And / or, implementing the image semantic segmentation method provided by an embodiment of the present invention, the method comprising:

[0108] Get the image to be semantically segmented;

[0109] The image to be semantically segmented is input into a pre-trained image semantic segmentation model to obtain a semantic segmentation result of the image semantic segmentation model, wherein the image semantic segmentation model is trained based on the image semantic segmentation model constructed by the image semantic segmentation model construction method provided in any embodiment of the present invention.

[0110] Of course, the computer-readable storage medium provided by an embodiment of the present invention, on which the computer program stored is not limited to the above method operations, can also execute the method for constructing the image semantic segmentation model provided by any embodiment of the present invention, and / or related operations of the image semantic segmentation method.

[0111] The computer storage medium of the embodiment of the present invention may adopt any combination of one or more computer-readable media. The computer-readable medium may be a computer-readable signal medium or a computer-readable storage medium. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or component, or any combination thereof. More specific examples (a non-exhaustive list) of computer-readable storage media include: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In this document, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device.

[0112] A computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, which carries computer-readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium that can transmit, propagate, or transport a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0113] Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

[0114] Computer program code for carrying out the operations of the present invention may be written in one or more programming languages, or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on the remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., through the Internet using an Internet service provider).

[0115] Note that the above are only preferred embodiments of the present invention and the technical principles employed. Those skilled in the art will appreciate that the present invention is not limited to the specific embodiments herein, and that various obvious changes, readjustments, and substitutions are possible for those skilled in the art without departing from the scope of protection of the present invention. Therefore, although the present invention has been described in detail through the above embodiments, the present invention is not limited to the above embodiments and may include many other equivalent embodiments without departing from the scope of the present invention. The scope of the present invention is determined by the scope of the appended claims.

Claims

1. A method for constructing an image semantic segmentation model, characterized in that: include: Obtaining a basic model structure of a model to be constructed, wherein the basic model structure includes a backbone neural network, a convolutional layer, a feature splicing layer, and a segmentation network, wherein the segmentation network includes a pixel reconstruction module; Determine the number of convolutional layer channels and the number of completion channels based on the number of data channels of the output feature data of the pixel reconstruction module; According to the number of completion channels, an intermediate layer is selected from the backbone neural network as a feature completion layer, the feature completion layer and the convolution layer are respectively connected to the feature splicing layer, and the feature splicing layer is connected to the segmentation network to obtain the image semantic segmentation model.

2. The method according to claim 1, characterized in that The determining the number of convolutional layer channels and the number of complement channels according to the number of data channels of the feature data of the pixel reconstruction module includes: Any positive integer smaller than the number of data channels is used as the number of channels of the convolutional layer; The difference between the number of data channels and the number of convolutional layer channels is used as the number of complementary channels.

3. The method according to claim 1, characterized in that The selecting an intermediate layer as a feature completion layer from the backbone neural network according to the number of completion channels includes: At least one intermediate layer in the backbone neural network is selected as the feature completion layer, wherein the total number of channels of the at least one intermediate layer is equal to the number of completion channels.

4. The method according to claim 3, characterized in that The resolution of the output feature map of the feature completion layer is the same as the resolution of the output feature map of the pixel reconstruction module.

5. The method according to claim 1, wherein The method further comprises: Obtain training sample data; The image semantic segmentation model is iteratively trained based on the training sample data, and a trained image semantic segmentation model is obtained when a training end condition is met.

6. A method for image semantic segmentation, characterized in that: include: Get the image to be semantically segmented; The image to be semantically segmented is input into a pre-trained image semantic segmentation model to obtain a semantic segmentation result of the image semantic segmentation model, wherein the image semantic segmentation model is trained based on an image semantic segmentation model constructed according to the method for constructing an image semantic segmentation model according to any one of claims 1 to 5.

7. A device for constructing an image semantic segmentation model, characterized in that: include: A basic structure acquisition module is used to obtain the basic model structure of the model to be constructed, wherein the basic model structure includes a backbone neural network, a convolution layer, a feature splicing layer and a segmentation network, and the segmentation network includes a pixel reconstruction module; A channel number determination module is used to determine the number of convolutional layer channels and the number of completion channels according to the number of data channels of the output feature data of the pixel reconstruction module; A semantic segmentation model construction module is used to select an intermediate layer from the backbone neural network as a feature completion layer according to the number of completion channels, connect the feature completion layer and the convolution layer to the feature splicing layer respectively, and connect the feature splicing layer to the segmentation network to obtain the image semantic segmentation model.

8. An image semantic segmentation device, characterized in that: include: A module for acquiring images to be semantically segmented, used for acquiring images to be semantically segmented; An image semantic segmentation module is used to input the image to be semantically segmented into a pre-trained image semantic segmentation model to obtain a semantic segmentation result of the image semantic segmentation model, wherein the image semantic segmentation model is trained based on an image semantic segmentation model constructed according to the method for constructing an image semantic segmentation model according to any one of claims 1 to 5.

9. A computer device, characterized in that: The device comprises: one or more processors; a storage device for storing one or more programs; When one or more programs are executed by one or more processors, the one or more processors implement the method for constructing an image semantic segmentation model as described in any one of claims 1 to 5, and / or implement the image semantic segmentation method as described in claim 6.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, it implements the method for constructing an image semantic segmentation model as described in any one of claims 1 to 5, and / or implements the image semantic segmentation method as described in claim 6.

Images