Road scene real-time semantic segmentation method and device based on improved double-branch network
By improving the dual-branch network and combining high- and low-resolution feature extraction and fusion, the problem of limited computing resources in deep learning models is solved, achieving efficient semantic segmentation results that are suitable for autonomous driving environmental perception.
Patent Information
- Application Number
- CN202511039317.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-07-28
- Publication Date
- 2025-11-18
AI Technical Summary
Existing deep learning segmentation models have too many parameters in autonomous driving environment perception, which leads to a shortage of computing resources in the vehicle computing unit and makes it difficult to achieve accurate perception in complex and ever-changing road scenarios.
An improved dual-branch network is adopted, combining high-resolution and low-resolution branches. Through an adaptive feature enhancement module and a pyramid pooling module, feature extraction and fusion are optimized, reducing computational complexity and improving segmentation accuracy.
With low computational complexity and parameter count, it achieves an average intersection-union ratio of 78.41% and an inference speed of 79.26 FPS, significantly improving the efficiency and accuracy of semantic segmentation, and is suitable for autonomous driving environmental perception.
Smart Images
Figure CN120976538A_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The present application relates to the field of artificial intelligence and computer vision technology, and in particular to a road scene real-time semantic segmentation method and device based on an improved double-branch network, a storage medium and an electronic device. BACKGROUND
[0002] Automatic driving technology is a cutting-edge research direction in the field of intelligent transportation, aiming to enable vehicles to drive autonomously and safely without human intervention. Environmental perception is a crucial part of automatic driving technology. Through various sensors, vehicles can dynamically perceive and understand the driving environment, providing digital information about the known driving environment and inputting to the decision-making module. In the field of environmental perception, semantic segmentation technology plays a key role. By assigning semantic categories to each pixel in an image, semantic segmentation technology can achieve a comprehensive understanding of the image scene, thus playing an important role in road scene perception.
[0003] Currently, the implementation of automatic driving technology faces many challenges, especially in the field of environmental perception. How to achieve accurate perception of complex and variable road scenes has always been a cutting-edge research direction. The system needs to complete the fusion processing and decision output of multi-sensor data within milliseconds in dynamic traffic scenarios, which puts strict requirements on the efficiency of semantic segmentation models. Although current mainstream deep learning segmentation models perform well in accuracy, their parameter quantity generally exceeds ten million, causing a huge pressure on the computing resources of on-board computing units. SUMMARY
[0004] The embodiments of the present application provide a road scene real-time semantic segmentation method and device based on an improved double-branch network, a storage medium and an electronic device, which can improve the extraction capability of detail features and multi-scale features, optimize the overall performance, and improve the segmentation accuracy.
[0005] The embodiments of the present application provide a road scene real-time semantic segmentation method based on an improved double-branch network, which includes: obtaining a road scene picture; inputting the road scene picture into an improved double-branch network model to obtain a semantic segmentation result; the improved double-branch network includes a high-resolution branch, a low-resolution branch and a segmentation head, and the high-resolution branch includes an adaptive feature enhancement module; The processing flow of the improved double-branch network model includes: The road scene picture is input into the improved double-branch network model, high-resolution features are extracted through the high-resolution branch, low-resolution features are extracted through the low-resolution branch, the high-resolution features and the low-resolution features are fused through the adaptive feature enhancement module to obtain a fused feature map, and the fused feature map is input into the segmentation head to obtain a semantic segmentation result.
[0006] Further, the road scene real-time semantic segmentation method based on the improved double-branch network, wherein the high-resolution branch comprises a plurality of convolution modules and a plurality of adaptive feature enhancement modules, and the low-resolution branch comprises a pyramid pooling module and a plurality of residual modules. The processing procedures of the high-resolution branch and the low-resolution branch comprise: The road scene picture is input into the plurality of convolution modules for down-sampling to obtain a first high-resolution feature map. The high-resolution feature map is input into a first adaptive feature enhancement module to obtain a first multi-scale feature map. The first multi-scale feature map is input into a residual module to obtain a first low-resolution feature map. The first multi-scale feature map and the first low-resolution feature map are input into the residual module to obtain a second low-resolution feature map. The first multi-scale feature map and the first low-resolution feature map are input into a second adaptive feature module to obtain a second multi-scale feature map. The second multi-scale feature map and the second low-resolution feature map are input into the residual module to obtain a third low-resolution feature map. The second multi-scale feature map and the second low-resolution feature map are input into a third adaptive feature module to obtain a third multi-scale feature map. The third low-resolution feature map is input into the pyramid pooling module and fused with the third multi-scale feature map to obtain a fused feature map.
[0007] Further, the road scene real-time semantic segmentation method based on the improved double-branch network, wherein the processing procedure of the adaptive feature enhancement module is represented by the following formula:
[0008] wherein, is an adaptive feature enhancement module, represents a ReLU activation function, and are high-resolution and low-resolution residual modules, respectively, and respectively represent a low-to-high resolution and a high-to-low resolution transformation, , is a multi-scale feature map output by the last adaptive enhancement module and the current adaptive enhancement module, , is a low-resolution feature map output by the last residual module and the current residual module in the low-resolution branch.
[0009] Further, the road scene real-time semantic segmentation method based on the improved double-branch network, wherein the adaptive feature enhancement module comprises a channel enhancement module and a multi-scale feature enhancement module in parallel; The processing process of the adaptive feature enhancement module comprises: inputting the input feature map into the channel enhancement module for channel weighting to obtain a channel enhanced feature map; inputting the input feature map into the multi-scale feature enhancement module for convolution processing and channel weighting to obtain a feature enhanced feature map; point-by-point summing the channel enhanced feature map and the feature enhanced feature map to obtain a multi-scale feature map.
[0010] Further, the road scene real-time semantic segmentation method based on the improved double-branch network, wherein the channel enhancement module comprises two fully connected layers and an activation function; the processing process of the channel enhancement module comprises: performing dimensionality reduction operation and dimensionality increase operation on the input feature map through the two fully connected layers respectively; calculating the importance weight of each channel through the activation function, and performing dot product operation on the importance weight and the input feature to obtain a channel enhanced feature map.
[0011] Further, the road scene real-time semantic segmentation method based on the improved double-branch network, wherein the multi-scale feature enhancement module comprises a convolution layer; the processing process of the multi-scale feature enhancement module comprises: dividing the input feature map into several parts along the channel, and processing each part using a convolution module of different scales to obtain feature maps of different scales; performing dimensionality reduction operation and dimensionality increase operation on the feature maps of different scales respectively; calculating the importance weight of each channel through the activation function, and performing dot product operation on the importance weight and the feature map of the corresponding scale, and splicing the dot product operation results along the channel direction to obtain a feature enhanced feature map.
[0012] Further, the road scene real-time semantic segmentation method based on the improved double-branch network, wherein the processing process of the pyramid pooling module comprises: up-sampling the input feature map and adjusting the number of channels.
[0013] The embodiment of the present application also provides a road scene real-time semantic segmentation device based on an improved double-branch network, comprising: An acquisition module is configured to acquire a road scene picture. A semantic segmentation module is configured to input the road scene picture into an improved double-branch network model to obtain a semantic segmentation result; the improved double-branch network comprises a high-resolution branch, a low-resolution branch and a segmentation head, and the high-resolution branch comprises an adaptive feature enhancement module. The processing procedure of the improved double-branch network model comprises: The road scene picture is input into the improved double-branch network model, high-resolution features are extracted through the high-resolution branch, low-resolution features are extracted through the low-resolution branch, the high-resolution features and the low-resolution features are fused through the adaptive feature enhancement module to obtain a fused feature map, and the fused feature map is input into the segmentation head to obtain a semantic segmentation result.
[0014] The embodiment of the present application also provides a computer readable storage medium, wherein a plurality of instructions are stored in the computer readable storage medium, and the instructions are suitable for being loaded by a processor to execute any one of the road scene real-time semantic segmentation methods based on the improved double-branch network.
[0015] The embodiment of the present application also provides an electronic device comprising a processor and a memory, wherein the processor is electrically connected with the memory, the memory is configured to store instructions and data, and the processor is configured to execute the steps in any one of the road scene real-time semantic segmentation methods based on the improved double-branch network.
[0016] The road scene real-time semantic segmentation method, device, storage medium and electronic device based on the improved double-branch network provided by the present application, through the improved double-branch design and the introduction of the AFEM module, the average intersection over union (mIoU) on the Cityscapes dataset reaches 78.41% under the conditions of lower computational complexity (88.7 GFLOPs) and less parameter amount (21.13M), the inference speed reaches 79.26 FPS, and excellent performance and efficiency balance are exhibited. Compared with the traditional design, the present application can effectively improve the detail feature extraction capability, guarantee the efficiency and realize higher segmentation precision, and provides an innovative solution and practical value for the semantic segmentation technology. BRIEF DESCRIPTION OF DRAWINGS
[0017] The technical scheme and other beneficial effects of the present application will become apparent through the following detailed description of the specific embodiments of the present application in combination with the accompanying drawings.
[0018] Figure 1A flowchart of a road scene real-time semantic segmentation method based on an improved double-branch network is provided for the embodiments of the present application.
[0019] Figure 2 A structural schematic diagram of an adaptive feature enhancement module is provided for the embodiments of the present application.
[0020] Figure 3 A structural schematic diagram of a multi-scale feature enhancement module is provided for the embodiments of the present application.
[0021] Figure 4 A comparison diagram of segmentation results obtained by the method provided in the present application and other methods is provided for the embodiments of the present application.
[0022] Figure 5 A structural schematic diagram of a road scene real-time semantic segmentation device based on an improved double-branch network is provided for the embodiments of the present application.
[0023] Figure 6 A structural schematic diagram of an electronic device is provided for the embodiments of the present application. DETAILED DESCRIPTION
[0024] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, but not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work fall within the scope of protection of the present application.
[0025] The embodiments of the present application provide a road scene real-time semantic segmentation method, device, storage medium and electronic device based on an improved double-branch network. The road scene real-time semantic segmentation device based on an improved double-branch network provided in the embodiments of the present application can be integrated in an electronic device, which can be a terminal, a server or the like, wherein the terminal can include a tablet computer, a notebook computer, a personal computer (PC), a micro processing box or other devices, etc.
[0026] Please refer to Figure 1 , Figure 1 A flowchart of a road scene real-time semantic segmentation method based on an improved double-branch network is provided for the embodiments of the present application, which is applied in an electronic device. The road scene real-time semantic segmentation method based on an improved double-branch network includes the following steps: S1, obtaining a road scene picture.
[0027] S2, input the road scene picture into the improved double-branch network model to obtain a semantic segmentation result; the improved double-branch network includes a high-resolution branch, a low-resolution branch and a segmentation head; the high-resolution branch includes an adaptive feature enhancement module.
[0028] The processing procedure of the improved double-branch network model includes: The road scene picture is input into the improved double-branch network model, high-resolution features are extracted through the high-resolution branch, low-resolution features are extracted through the low-resolution branch, the high-resolution features and the low-resolution features are fused through the adaptive feature enhancement module to obtain a fused feature map, and the fused feature map is input into the segmentation head to obtain a semantic segmentation result.
[0029] In an embodiment, the high-resolution branch includes a plurality of convolution modules and a plurality of adaptive feature enhancement modules, and the low-resolution branch includes a pyramid pooling module and a plurality of residual modules. The processing procedure of the high-resolution branch and the low-resolution branch includes: S21, input the road scene picture into the plurality of convolution modules to perform down-sampling to obtain a first high-resolution feature map; S22, input the high-resolution feature map into the first adaptive feature enhancement module to obtain a first multi-scale feature map; S23, input the first multi-scale feature map into the residual module to obtain a first low-resolution feature map; S24, input the first multi-scale feature map and the first low-resolution feature map into the residual module to obtain a second low-resolution feature map; S25, input the first multi-scale feature map and the first low-resolution feature map into the second adaptive feature module to obtain a second multi-scale feature map; S26, input the second multi-scale feature map and the second low-resolution feature map into the residual module to obtain a third low-resolution feature map; S27, input the second multi-scale feature map and the second low-resolution feature map into the third adaptive feature module to obtain a third multi-scale feature map; S28, input the third low-resolution feature map into the pyramid pooling module and fuse the third multi-scale feature map to obtain a fused feature map.
[0030] Please refer to Figure 1Specifically, in stages 0~2: the input road scene picture is preprocessed and feature extracted through three convolution modules, the image size is down-sampled to 1 / 8 of the original size, and the number of channels is increased to 256. In stages 3~4: a high-low resolution dual fusion mechanism is introduced to fuse high-resolution and low-resolution features. In stage 5, the fused feature maps of the high-low resolution branches are integrated. The feature map of the low-resolution branch is up-sampled from 1 / 64 size to 1 / 8 size through a deep aggregation pyramid pooling module (DAPPM), and the number of channels is adjusted to be consistent with that of the high-resolution branch, and then the point-wise summation is performed with the fused feature map of the high-resolution branch. The final feature map is input into the segmentation head to generate the segmentation result.
[0031] In an embodiment, the processing process of the adaptive feature enhancement module is represented by the following formula:
[0032] wherein, is the adaptive feature enhancement module, represents a ReLU activation function, and are high-resolution and low-resolution residual modules respectively, and represent low-to-high resolution and high-to-low transformation respectively, , are multi-scale feature maps output by the previous adaptive enhancement module and the current adaptive enhancement module, , are low-resolution feature maps output by the previous residual module and the current residual module in the low-resolution branch.
[0033] Figure 2 is a structural schematic diagram of the adaptive feature enhancement module provided by the embodiment of the present application, Figure 3 is a structural schematic diagram of the multi-scale feature enhancement module provided by the embodiment of the present application, as shown in Figure 2 and Figure 3 , the adaptive feature enhancement module includes a parallel channel enhancement module and a multi-scale feature enhancement module; The processing process of the adaptive feature enhancement module includes: S31, input the input feature map into the channel enhancement module for channel weighting to obtain a channel enhanced feature map.
[0034] S32, input the input feature map into the multi-scale feature enhancement module for convolution processing and channel weighting to obtain a feature enhanced feature map.
[0035] S33, point-wise sum the channel enhanced feature map and the feature enhanced feature map to obtain a multi-scale feature map.
[0036] The channel enhancement module includes two fully connected layers and an activation function; the processing procedure of the channel enhancement module includes: S311, performing dimension reduction operation and dimension increase operation on the input feature map through the two fully connected layers respectively; S312, calculating the importance weight of each channel through the activation function, and performing dot product operation on the importance weight and the input feature to obtain a channel enhanced feature map.
[0037] The multi-scale feature enhancement module includes a convolution layer; the processing procedure of the multi-scale feature enhancement module includes: S321, dividing the input feature map into several parts along the channel, and processing each part using a convolution module of different scales to obtain feature maps of different scales; S322, performing dimension reduction operation and dimension increase operation on the feature maps of different scales respectively; S323, calculating the importance weight of each channel through the activation function, performing dot product operation on the importance weight and the feature map of the corresponding scale, and splicing the dot product operation results along the channel direction to obtain a feature enhanced feature map.
[0038] Specifically, the main function of the channel enhancement module is to enhance the channel features of the image. The principle is to perform dimension increase and dimension reduction operation on the channel dimension of the feature map through two fully connected layers, and then calculate the importance weight of each channel through the Sigmoid function, and perform dot product operation on the feature map to adjust the weight of the channel feature, thereby improving the feature extraction capability of the network. The main function of the multi-scale feature enhancement module is to enhance multi-scale features. Specifically, the input feature map is divided into several parts along the channel direction, and each part is processed using a convolution module of different scales to obtain feature maps of different scales. Then, through similar dimension increase and dimension reduction operation as the module, the importance weight of each channel is calculated, and dot product operation is performed on the feature map of the corresponding scale. Finally, the feature maps of each scale are spliced along the channel direction to obtain a multi-scale feature map. After the two modules process the feature map, the enhanced feature maps are summed point by point to fully fuse the features enhanced by the two modules, so as to fully exert the respective functions of each module. This module can optimize the extraction of detail features and multi-scale features, thereby significantly improving the feature expression capability and minimizing the additional computational load.
[0039] In an embodiment, the processing procedure of the pyramid pooling module includes: up-sampling the input feature map and adjusting the number of channels.
[0040] Specifically, the feature map of the low-resolution branch is upsampled from 1 / 64 size to 1 / 8 size by a deep aggregation pyramid pooling module (DAPPM), and the number of channels is adjusted to be consistent with that of the high-resolution branch, and then the point-wise summation is performed with the fusion feature map of the high-resolution branch. The final feature map is input into the segmentation head to generate the segmentation result.
[0041] In an embodiment, the segmentation head includes the following two parts: Auxiliary segmentation head: used for additional supervision of high-resolution features during the training process; Main segmentation head: used for integrating high and low resolution features and generating the final semantic segmentation result.
[0042] The segmentation head is used to calculate the loss by combining the high and low resolution fusion feature maps, and generate the segmentation result. The loss function calculation formula is:
[0043] wherein, represents the final calculated loss, represents the loss calculated by the normal segmentation head, represents the loss calculated by the auxiliary segmentation head, is the weight of the auxiliary loss, which is set to 0.4.
[0044] Experimental results verify that the network model of the embodiment of the present application is verified on the Cityscapes dataset, Figure 4 The comparison diagram of the segmentation results obtained by the method provided in the embodiment of the present application and other methods, Figure 4 The first column is the image data of the Cityscapes dataset, the second column is the segmentation result of the DDRNet method, and the third column is the segmentation result of the method of the present application. Table 1 is the evaluation data table of the segmentation results obtained by the method provided in the present application and other methods.
[0045] Table 1 Evaluation data table of segmentation results obtained by the method provided in the present application and other methods
[0046] The results show that the average intersection over union (mIoU) of the method provided in the present application reaches 78.41%, which is the highest precision at present; the inference speed reaches 79.26FPS, which is flat with the performance of the latest real-time semantic segmentation network; the computational complexity (88.7 GFLOPs) and the parameter amount (21.13M) are significantly lower than those of the mainstream large-scale model. Through the above experiments, it is verified that the network designed in the present application performs excellently in the balance between performance and efficiency, and can effectively improve the feature extraction quality in the semantic segmentation task while maintaining high running efficiency.
[0047] According to the method described in the above embodiment, this embodiment will be further described from the perspective of a road scene real-time semantic segmentation device based on an improved double-branch network. The road scene real-time semantic segmentation device based on the improved double-branch network can be implemented as an independent entity, or can be integrated in an electronic device, which can be a terminal, a server, or the like. The terminal can include a tablet computer, a notebook computer, a personal computer (PC), a micro processing box, or other devices, and the like.
[0048] Please refer to Figure 5 , Figure 5 The road scene real-time semantic segmentation device based on the improved double-branch network provided by the embodiments of the present application is specifically described, which is applied to an electronic device. The road scene real-time semantic segmentation device based on the improved double-branch network can include: The acquisition module is configured to acquire a road scene picture. The semantic segmentation module is configured to input the road scene picture into an improved double-branch network model to obtain a semantic segmentation result. The improved double-branch network includes a high-resolution branch, a low-resolution branch, and a segmentation head. The high-resolution branch includes an adaptive feature enhancement module. The processing flow of the improved double-branch network model includes: The road scene picture is input into the improved double-branch network model. High-resolution features are extracted through the high-resolution branch, low-resolution features are extracted through the low-resolution branch, the high-resolution features and the low-resolution features are fused through the adaptive feature enhancement module to obtain a fused feature map, and the fused feature map is input into the segmentation head to obtain a semantic segmentation result.
[0049] In specific implementation, each of the above modules and / or units can be implemented as an independent entity, or can be combined as the same or several entities. The specific implementation of each of the above modules and / or units can be referred to the method embodiments above, and the beneficial effects that can be achieved are also referred to the beneficial effects of the method embodiments above, which will not be described here again.
[0050] In addition, the electronic device provided by the embodiments of the present application can be a computer, a tablet computer, or the like. The electronic device can implement the steps in any of the road scene real-time semantic segmentation methods based on the improved double-branch network provided by the embodiments of the present application. Therefore, the electronic device can achieve the beneficial effects of any of the road scene real-time semantic segmentation methods based on the improved double-branch network provided by the embodiments of the present application. Details are described in the above embodiments, which will not be described here again.
[0051] Figure 6A specific structure block diagram of an electronic device provided by an embodiment of the present application is shown, which can be used to implement the road scene real-time semantic segmentation method based on the improved double-branch network provided in the above embodiments. The electronic device 500 can be a terminal, a server, or the like, wherein the terminal can include a tablet computer, a notebook computer, a personal computer (PC), a micro processing box, or other devices, etc.
[0052] The RF circuit 510 is configured to receive and send electromagnetic waves, and to convert the electromagnetic waves and electrical signals to each other, so as to communicate with a communication network or other devices. The RF circuit 510 can include various existing circuit elements for performing these functions, such as an antenna, a radio frequency transceiver, a digital signal processor, an encryption / decryption chip, a subscriber identity module (SIM) card, a memory, and the like. The RF circuit 510 can communicate with various networks such as the Internet, an intranet, a wireless network, or other devices through the wireless network. The wireless network can include a cellular telephone network, a wireless local area network or metropolitan area network. The wireless network can use various communication standards, protocols and technologies, including but not limited to Global System for Mobile Communication (GSM), Enhanced Data GSM Environment (EDGE), Wideband Code Division Multiple Access (WCDMA), Code Division Access (CDMA), Time Division Multiple Access (TDMA), Wireless Fidelity (Wi-Fi) (such as Institute of Electrical and Electronics Engineers (IEEE) 802.11a, 802.11b, 802.11g and / or 802.11n), Voice over Internet Protocol (VoIP), Worldwide Interoperability for Microwave Access (Wi-Max), other protocols for email, instant messaging and short message service, and any other suitable communication protocol, even including those not yet developed.
[0053] The memory 520 can be used to store software programs and modules, such as the corresponding program instructions / modules in the above-described embodiments, and the processor 580 can execute various functions and data processing by running the software programs and modules stored in the memory 520, i.e., realize functions such as front camera shooting, processing of the shot image, and switching of display colors of the display content on the display screen. The memory 520 can include a high-speed random access memory, and can further include a non-volatile memory such as one or more magnetic storage devices, flash memories, or other non-volatile solid-state memories. In some examples, the memory 520 can further include memories disposed remotely with respect to the processor 580, which can be connected to the electronic device 500 through a network. Examples of the network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and a combination thereof.
[0054] The input unit 530 can be used to receive inputted digital or character information, and generate a keyboard, a mouse, and the like related to user settings and function control. The display unit 540 can be used to display information inputted by the user or provided to the user, and various graphical user interfaces which can be constituted by graphics, texts, icons, videos, and any combination thereof. The display unit 540 can include a display panel 541, which can be configured in the form of an LCD (Liquid Crystal Display), an OLED (Organic Light-Emitting Diode), or the like.
[0055] The audio circuit 560, the speaker 561, and the microphone 562 can provide an audio interface between the user and the electronic device 500. The audio circuit 560 can convert received audio data into an electrical signal, transmit the electrical signal to the speaker 561, and convert the electrical signal into a sound signal output by the speaker 561; on the other hand, the microphone 562 can convert a sound signal collected into an electrical signal, and the audio circuit 560 can convert the electrical signal into audio data, output the audio data to the processor 580 for processing, and then transmit the audio data to another terminal through the RF circuit 510, or output the audio data to the memory 520 for further processing. The audio circuit 560 can further include an earphone jack to provide communication between an external earphone and the electronic device 500.
[0056] The electronic device 500 can help the user to receive requests, send information, and the like through the transmission module 570 (e.g., a Wi-Fi module), which provides the user with wireless broadband Internet access. Although the transmission module 570 is shown, it can be understood that it does not belong to the essential components of the electronic device 500, and can be omitted as needed without changing the essence of the application.
[0057] The processor 580 is a control center of the electronic device 500 that connects various parts of the entire mobile phone using various interfaces and lines, performs various functions of the electronic device 500 and processes data by running or executing software programs and / or modules stored in the memory 520 and calling data stored in the memory 520, thereby monitoring the entire electronic device. Optionally, the processor 580 can include one or more processing cores; in some embodiments, the processor 580 can integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, user interface, and application programs, and the modem processor mainly processes wireless communication. Understandably, the above-mentioned modem processor can also not be integrated into the processor 580.
[0058] The electronic device 500 further includes a power supply 590 (such as a battery) for supplying power to various components, and in some embodiments, the power supply can be logically connected to the processor 580 through a power management system, so that the power management system can realize functions such as management of charging, discharging, and power consumption management. The power supply 590 can also include one or more direct or alternating power supplies, recharging systems, power failure detection circuits, power converters or inverters, power state indicators, and any other components.
[0059] Although not shown, the electronic device 500 also includes a camera (such as a front camera or a rear camera), a Bluetooth module, and the like, which are not described here in detail. Specifically, in the present embodiment, the display unit of the electronic device is a touch screen display, and the mobile terminal further includes a memory and one or more programs, wherein one or more programs are stored in the memory and configured to be executed by one or more processors, and the one or more programs include instructions for performing the following operations: obtaining a road scene picture; inputting the road scene picture into an improved double-branch network model to obtain a semantic segmentation result; the improved double-branch network includes a high-resolution branch, a low-resolution branch, and a segmentation head, and the high-resolution branch includes an adaptive feature enhancement module; The processing flow of the improved double-branch network model includes: inputting the road scene picture into an improved double-branch network model, extracting high-resolution features through the high-resolution branch, extracting low-resolution features through the low-resolution branch, fusing the high-resolution features and the low-resolution features through the adaptive feature enhancement module to obtain a fused feature map, and inputting the fused feature map into the segmentation head to obtain a semantic segmentation result.
[0060] In practice, the above various modules can be implemented as independent entities, or combined as the same or several entities, and the specific implementation of the above various modules can refer to the method embodiments above, which will not be repeated here.
[0061] Those skilled in the art can understand that all or part of the steps in the above embodiments can be completed by instructions, or by related hardware controlled by the instructions, which can be stored in a computer readable storage medium and loaded and executed by a processor. Therefore, the embodiments of the present application provide a storage medium, which stores a plurality of instructions capable of being loaded by a processor to execute the steps of any embodiment of the road scene real-time semantic segmentation method based on the improved double-branch network provided by the embodiments of the present application.
[0062] The computer readable storage medium can include a read only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.
[0063] Since the instructions stored in the storage medium can execute the steps in any embodiment of the road scene real-time semantic segmentation method based on the improved double-branch network provided by the embodiments of the present application, the beneficial effects of any road scene real-time semantic segmentation method based on the improved double-branch network provided by the embodiments of the present application can be achieved, which will be described in detail in the foregoing embodiments, and will not be repeated here.
[0064] The above provides a detailed description of the road scene real-time semantic segmentation method, device, storage medium and electronic equipment based on the improved double-branch network provided by the embodiments of the present application. The principle and implementation manner of the present application are described by applying specific examples. The above embodiment description is only used to help understand the method and its core idea of the present application. Meanwhile, for those skilled in the art, according to the idea of the present application, the specific implementation manner and application range will be changed. In summary, the content of the specification should not be understood as a limitation of the present application.
Claims
1. A real-time semantic segmentation method for road scenes based on an improved dual-branch network, characterized in that, The method includes: Acquire road scene images; The road scene image is input into an improved dual-branch network model to obtain semantic segmentation results; the improved dual-branch network includes a high-resolution branch, a low-resolution branch, and a segmentation head, and the high-resolution branch includes an adaptive feature enhancement module; The processing flow of the improved dual-branch network model includes: The road scene image is input into the improved dual-branch network model. High-resolution features are extracted through the high-resolution branch, and low-resolution features are extracted through the low-resolution branch. The high-resolution features and the low-resolution features are fused by the adaptive feature enhancement module to obtain a fused feature map. The fused feature map is input into the segmentation head to obtain the semantic segmentation result.
2. The real-time semantic segmentation method for road scenes based on an improved dual-branch network according to claim 1, characterized in that, The high-resolution branch includes multiple convolutional modules and multiple adaptive feature enhancement modules, while the low-resolution branch includes a pyramid pooling module and multiple residual modules. The processing procedures for the high-resolution branch and the low-resolution branch include: The road scene image is input into the multiple convolutional modules for downsampling to obtain a first high-resolution feature map; The high-resolution feature map is input into the first adaptive feature enhancement module to obtain the first multi-scale feature map; The first multi-scale feature map is input into the residual module to obtain the first low-resolution feature map; The second low-resolution feature map is obtained from the residual module of the first multi-scale feature map and the first low-resolution feature map. The first multi-scale feature map and the first low-resolution feature map are input into the second adaptive feature module to obtain the second multi-scale feature map; The third low-resolution feature map is obtained by using the residual module of the second multi-scale feature map and the second low-resolution feature map. The second multi-scale feature map and the second low-resolution feature map are input into the third adaptive feature module to obtain the third multi-scale feature map; The third low-resolution feature map is input into the pyramid pooling module and then fused with the third multi-scale feature map to obtain a fused feature map.
3. The real-time semantic segmentation method for road scenes based on an improved dual-branch network according to claim 2, characterized in that, The processing procedure of the adaptive feature enhancement module is represented by the following formula: in, It is an adaptive feature enhancement module. Represents the ReLU activation function. and These are high-resolution and low-resolution residual modules, respectively. and These represent a transformation from low to high resolution and from high resolution to low resolution, respectively. , The multi-scale feature maps output by the previous adaptive enhancement module and the current adaptive enhancement module. , This is a low-resolution feature map output by the previous residual module and the current residual module in the low-resolution branch.
4. The real-time semantic segmentation method for road scenes based on an improved dual-branch network according to claim 2, characterized in that, The adaptive feature enhancement module includes a parallel channel enhancement module and a multi-scale feature enhancement module; The processing procedure of the adaptive feature enhancement module includes: The input feature map is fed into the channel enhancement module for channel weighting to obtain a channel-enhanced feature map. The input feature map is fed into the multi-scale feature enhancement module for convolution processing and channel weighting to obtain the feature-enhanced feature map. The channel enhancement feature map and the feature enhancement feature map are summed point by point to obtain a multi-scale feature map.
5. The real-time semantic segmentation method for road scenes based on an improved dual-branch network according to claim 4, characterized in that, The channel enhancement module includes two fully connected layers and an activation function; the processing procedure of the channel enhancement module includes: The input feature map is subjected to dimensionality reduction and dimensionality increase operations through two fully connected layers, respectively. The importance weight of each channel is calculated by using an activation function, and the importance weight is then multiplied by the input feature to obtain the channel-enhanced feature map.
6. The real-time semantic segmentation method for road scenes based on an improved dual-branch network according to claim 4, characterized in that, The multi-scale feature enhancement module includes convolutional layers; the processing procedure of the multi-scale feature enhancement module includes: The input feature map is divided into several parts along the channels, and each part is processed using convolutional modules of different scales to obtain feature maps of different scales. Dimensionality reduction and dimension increase operations are performed on feature maps of different scales respectively; The importance weight of each channel is calculated by using an activation function. The importance weight is then multiplied by the feature map at the corresponding scale. The result of the dot product operation is then concatenated along the channel direction to obtain the feature-enhanced feature map.
7. The real-time semantic segmentation method for road scenes based on an improved dual-branch network according to claim 2, characterized in that, The processing steps of the pyramid pooling module include: The input feature map is upsampled, and the number of channels is adjusted.
8. A real-time semantic segmentation device for road scenes based on an improved dual-branch network, characterized in that, include: The acquisition module is used to acquire road scene images; The semantic segmentation module is used to input the road scene image into the improved dual-branch network model to obtain the semantic segmentation result; the improved dual-branch network includes a high-resolution branch, a low-resolution branch and a segmentation head, and the high-resolution branch includes an adaptive feature enhancement module; The processing flow of the improved dual-branch network model includes: The road scene image is input into the improved dual-branch network model. High-resolution features are extracted through the high-resolution branch, and low-resolution features are extracted through the low-resolution branch. The high-resolution features and the low-resolution features are fused by the adaptive feature enhancement module to obtain a fused feature map. The fused feature map is input into the segmentation head to obtain the semantic segmentation result.
9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a plurality of instructions adapted for loading by a processor to execute the real-time semantic segmentation method for road scenes based on an improved dual-branch network as described in any one of claims 1 to 7.
10. An electronic device, characterized in that, The method includes a processor and a memory, the processor being electrically connected to the memory, the memory being used to store instructions and data, and the processor being used to execute the steps in the real-time semantic segmentation method for road scenes based on an improved dual-branch network as described in any one of claims 1 to 7.