A method and apparatus for drivable region segmentation based on large-scale selective attention using splittable convolution.

By introducing a splittable convolutional large-scale selective attention module into the Deeplabv3+ network, the feature vector processing is optimized, which solves the problem of incorrect judgment in complex environments for drivable region segmentation technology in autonomous driving, and achieves higher detection accuracy and edge segmentation accuracy.

CN117876396BActive Publication Date: 2026-11-17XIAMEN UNIV
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
CN202410037810.8
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-01-10
Publication Date
2026-11-17
Estimated Expiration
2044-01-10

AI Technical Summary

Technical Problem

Existing autonomous driving drivable area segmentation technologies are prone to making incorrect judgments when dealing with complex and ever-changing road environments, which affects safety. It is necessary to improve the model's ability to express input data and its segmentation accuracy.

Method used

A drivable region segmentation method based on splittable convolutional large-scale selective attention is adopted. By introducing a splittable convolutional large-scale selective attention module between the backbone network and the ASPP module of the Deeplabv3+ network, including a series of large-scale convolutional kernel sequences, spatial kernel selection units and a first convolutional layer, the feature vector is optimized and the efficiency of the model in utilizing input information is improved.

Benefits of technology

This improved the model's detection and edge segmentation accuracy, reduced input information loss, enhanced the pixel weights for the target region, and improved the segmentation performance of the neural network.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN117876396B_ABST
    Figure CN117876396B_ABST
Patent Text Reader

Abstract

This invention discloses a method and apparatus for drivable region segmentation based on splittable convolutional large-scale selective attention, comprising: acquiring real-time images of roads; constructing and training a drivable region segmentation model based on an improved Deeplabv3+ network to obtain a trained drivable region segmentation model; wherein the improved Deeplabv3+ network-based drivable region segmentation model sets a splittable convolutional large-scale selective attention module between the backbone network and the ASPP module in the encoder of the traditional Deeplabv3+ network, the splittable convolutional large-scale selective attention module including a series of large-scale convolutional kernel sequences, a spatial kernel selection unit, and a first convolutional layer; inputting the real-time image into the trained drivable region segmentation model to obtain the output features of the splittable convolutional large-scale selective attention module and inputting them into the ASPP module, and then decoding them through the decoder to obtain the segmentation result, which can improve the overall detection accuracy and edge segmentation accuracy.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This invention relates to the field of autonomous driving, and specifically to a method and apparatus for segmenting drivable regions based on large-scale selective attention using splittable convolution. Background Technology

[0002] Driving area segmentation technology in autonomous driving refers to the technique of dividing roads and surrounding scenes into drivable and non-drivable areas by perceiving and analyzing the vehicle's surrounding environment. This technology utilizes sensors, cameras, LiDAR, and other devices to acquire real-time road surface information and obstacle locations, and combines this with map data for processing and analysis to determine areas where the vehicle can safely drive. Driving area segmentation technology plays a crucial role in autonomous driving systems, improving the vehicle's perception capabilities, decision-making abilities, and driving safety, providing important support for achieving highly automated driving. Current mainstream methods mainly use deep learning semantic segmentation models to segment road images acquired by sensors to obtain drivable and non-drivable areas. However, these methods have some shortcomings in practical applications. Due to the complexity and variability of road conditions and the significant differences between different environments, the above methods are prone to misjudgments during image processing, which can seriously affect the safety of autonomous driving. Therefore, designing a method that can improve the model's ability to express input data, better learn the complex features of the data, cope with complex road environments, improve model performance, and achieve higher segmentation accuracy is of great significance. Summary of the Invention

[0003] In view of the aforementioned technical problems, the purpose of the embodiments of this application is to propose a method and apparatus for drivable region segmentation based on large-scale selective attention of splittable convolution, so as to solve the technical problems mentioned in the background section above.

[0004] In a first aspect, the present invention provides a method for segmenting drivable regions based on large-scale selective attention using splittable convolution, comprising the following steps:

[0005] Acquire real-time images of the road;

[0006] A drivable region segmentation model based on an improved Deeplabv3+ network is constructed and trained to obtain a trained drivable region segmentation model. The drivable region segmentation model based on the improved Deeplabv3+ network sets a splittable convolutional large-scale selective attention module between the backbone network and the ASPP module in the encoder of the traditional Deeplabv3+ network. The splittable convolutional large-scale selective attention module includes a series of large-scale convolutional kernel sequences, a spatial kernel selection unit, and a first convolutional layer. The series of large-scale convolutional kernel sequences includes 2 to n splittable convolutions.

[0007] Real-time images are input into a trained drivable region segmentation model and a backbone network to obtain feature vectors. These feature vectors are then input into a splittable convolutional large-scale selective attention module. The feature vectors pass through a series of large-scale convolutional kernel sequences to obtain the output features of each splittable convolution in the sequence. The output features of each splittable convolution in the sequence are concatenated to obtain concatenated features. These concatenated features are input into a spatial kernel selection unit to obtain decomposed large-kernel sequence features. These decomposed large-kernel sequence features are multiplied by the output features of each splittable convolution, and the sum of these multiplication results is input into the first convolutional layer to obtain attention features. These attention features are multiplied by the feature vector to obtain the output features of the splittable convolutional large-scale selective attention module. The output features of the splittable convolutional large-scale selective attention module are input into the ASPP module and then decoded by a decoder to obtain the segmentation result.

[0008] Preferably, the nth splittable convolution in a series of large-scale convolution kernel sequences has a kernel size of k, an expansion rate of d, and a receptive field of R, and is defined as follows:

[0009] k n-1 ≤k n ;d1=1,d n-1 <d n ≤R n-1 ;

[0010] R1 = k1, R n =d n (k n -1)+R n-1 .

[0011] As a preferred embodiment, the calculation process for the nth separable convolution in a series of large-scale convolution kernel sequences is as follows:

[0012] Z0 = X,

[0013] Among them, Z n This represents the output feature of the nth splittable convolution. This indicates that the kernel size is k. n The expansion rate is d n The convolutional layer is a splittable convolution of type D, where X represents the feature vector.

[0014] As a preferred choice, the backbone network is ResNet-101, and the spatial kernel selection unit includes an average pooling layer, a max pooling layer, a second convolutional layer, and a Sigmoid activation function layer.

[0015] Preferably, the cascaded features are input into the spatial kernel selection unit to obtain the decomposed large kernel sequence features, specifically including:

[0016] The cascaded features are input into the average pooling layer and the max pooling layer respectively to obtain the average pooling features and the max pooling features, as shown in the following formula:

[0017] Sa avg =P avg (Z), Sa max =P max (Z);

[0018] Among them, Sa avg and Sa max Let P represent the average pooling feature and the max pooling feature, respectively. avg and P max represent the average pooling layer and the max pooling layer, respectively, and Z represents the cascaded feature;

[0019] The average pooling feature and the max pooling feature are concatenated to obtain the pooling feature. The pooling feature is then passed through a second convolutional layer and a sigmoid activation function layer to obtain the decomposed large kernel sequence feature, as shown in the following formula:

[0020]

[0021] Among them, f 2→N This indicates the second convolutional layer, s represents the Sigmoid activation function layer, [Sa avg Sa max ] represents pooling characteristics. This indicates the characteristics of the large nuclear sequence after decomposition.

[0022] Preferably, the splittable convolution includes several basis sets and a split attention module, which includes a global average pooling layer, a third convolutional layer, a batch normalization layer, a ReLU activation function layer, and an r-Softmax function layer.

[0023] Preferably, let the total number of basis sets be Q. Input the input tensor of the separable convolution into each basis set to obtain the output of each basis set. The output of the a-th basis set is shown in the following formula:

[0024] T = f a (X');

[0025] Where a∈{1, 2, ..., Q}, X' represents the input tensor that can be split convolutionally, f a This indicates that the kernel size is k. a The expansion rate is d a Convolution operations;

[0026] The outputs of each basis set are combined by summing across elements to obtain the basis set combination, as shown in the following equation:

[0027]

[0028] in, Represents basis set combinations;

[0029] The base set is combined and input into the global average pooling layer to obtain the global background information, as shown in the following formula:

[0030]

[0031] Where H represents global background information, H and W represent the height and width of the basis set combination, and i and j represent the coordinates of the basis set combination along the x-axis and y-axis directions;

[0032] The global background information is passed sequentially through the third convolutional layer, the batch normalization layer, and the ReLU activation function layer to obtain the feature tensor. After the feature tensor is passed through the r-Softmax function layer, it is multiplied by the output of each basis set at the pixel level, and then added at the pixel level to obtain the output features of the splittable convolution.

[0033] Secondly, the present invention provides a drivable region segmentation device based on large-scale selective attention of splittable convolution, comprising:

[0034] The image acquisition module is configured to acquire real-time images of the road;

[0035] The model building module is configured to build and train a drivable region segmentation model based on the improved Deeplabv3+ network to obtain a trained drivable region segmentation model. The drivable region segmentation model based on the improved Deeplabv3+ network sets a splittable convolutional large-scale selective attention module between the backbone network and the ASPP module in the encoder of the traditional Deeplabv3+ network. The splittable convolutional large-scale selective attention module includes a series of large-scale convolutional kernel sequences, a spatial kernel selection unit, and a first convolutional layer. The series of large-scale convolutional kernel sequences includes 2 to n splittable convolutions.

[0036] The segmentation module is configured to input real-time images into a trained drivable region segmentation model. The real-time images are input into the backbone network to obtain feature vectors. These feature vectors are then input into a splittable convolutional large-scale selective attention module. The feature vectors pass through a series of large-scale convolutional kernel sequences to obtain the output features of each splittable convolution in the series of large-scale convolutional kernel sequences. The output features of each splittable convolution in the series of large-scale convolutional kernel sequences are concatenated to obtain concatenated features. These concatenated features are input into a spatial kernel selection unit to obtain decomposed large kernel sequence features. The decomposed large kernel sequence features are multiplied by the output features of each splittable convolution to obtain the result of each multiplication. The sum of each multiplication result is then input into the first convolutional layer to obtain attention features. The attention features are multiplied by the feature vector to obtain the output features of the splittable convolutional large-scale selective attention module. The output features of the splittable convolutional large-scale selective attention module are input into the ASPP module and then decoded by the decoder to obtain the segmentation result.

[0037] Thirdly, the present invention provides an electronic device including one or more processors; and a storage device for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in any implementation of the first aspect.

[0038] Fourthly, the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method as described in any of the implementations of the first aspect.

[0039] Compared with the prior art, the present invention has the following beneficial effects:

[0040] (1) The drivable region segmentation method based on splittable convolutional large-scale selective attention proposed in this invention uses the Deeplabv3+ network as a benchmark. A splittable convolutional large-scale selective attention module is added after the backbone network to optimize and improve the input feature vector, thereby enhancing the network model's ability to express input data. It can select appropriate parameter values ​​according to different scenarios and use splittable convolution to improve the representation of the target region of the feature tensor, thus achieving efficient utilization of input information and improving overall detection accuracy and edge segmentation accuracy. Finally, the decoder restores the optimized feature vector to an image with the same size as the input and outputs it to obtain the detection result.

[0041] (2) The drivable region segmentation model in the drivable region segmentation method based on splittable convolutional large-scale selective attention proposed in this invention includes a series of large-scale convolutional kernel sequences, a spatial kernel selection unit and a first convolutional layer. The large-scale kernel convolutional sequence effectively improves the receptive field of the neural network model, enabling the model to obtain usable contextual information from different ranges, reducing input information loss and improving utilization efficiency. The spatial kernel selection unit can select the most suitable convolutional kernel according to the actual scene it faces, thereby improving the segmentation accuracy of the current scene.

[0042] (3) The drivable region segmentation method based on large-scale selective attention of splittable convolution proposed in this invention contains a series of large-scale convolution kernel sequences including 2 to n splittable convolutions. The splittable convolution establishes the input tensor into several basis sets, performs cross-element fusion on them, and then multiplies them with the output of each basis set to enhance the weight of the corresponding pixel in the target region, thereby improving the edge segmentation accuracy of the neural network model. Attached Figure Description

[0043] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0044] Figure 1 This is an exemplary device architecture diagram in which an embodiment of this application can be applied;

[0045] Figure 2 This is a flowchart illustrating a drivable region segmentation method based on large-scale selective attention using splittable convolution, as an embodiment of this application.

[0046] Figure 3 This is a schematic diagram of the drivable region segmentation model based on the drivable region segmentation method with large-scale selective attention based on splittable convolution, as an embodiment of this application.

[0047] Figure 4 This is a schematic diagram of the structure of the splittable convolutional large-scale selective attention module of the drivable region segmentation method based on splittable convolutional large-scale selective attention, which is an embodiment of this application.

[0048] Figure 5 This is a schematic diagram of the splittable convolution in the drivable region segmentation method based on large-scale selective attention, which is an embodiment of this application.

[0049] Figure 6This is a schematic diagram of a drivable region segmentation device based on large-scale selective attention using splittable convolution, as an embodiment of this application.

[0050] Figure 7 This is a schematic diagram of the structure of a computer device suitable for implementing the electronic device of the present application. Detailed Implementation

[0051] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this invention, and not all of them. Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this invention.

[0052] Figure 1 An exemplary device architecture 100 is shown, which can be applied to the drivable region segmentation method based on splittable convolutional large-scale selective attention or the drivable region segmentation apparatus based on splittable convolutional large-scale selective attention according to the embodiments of this application.

[0053] like Figure 1 As shown, the device architecture 100 may include terminal device 101, terminal device 2 102, terminal device 3 103, network 104, and server 105. Network 104 serves as the medium for providing communication links between terminal device 101, terminal device 2 102, terminal device 3 103, and server 105. Network 104 may include various connection types, such as wired or wireless communication links or fiber optic cables, etc.

[0054] Users can use terminal device 101, terminal device 2 102, and terminal device 3 103 to interact with server 105 via network 104 to receive or send messages, etc. Various applications, such as data processing applications and file processing applications, can be installed on terminal device 101, terminal device 2 102, and terminal device 3 103.

[0055] Terminal devices 101, 102, and 103 can be either hardware or software. When terminal devices 101, 102, and 103 are hardware, they can be various electronic devices, including but not limited to smartphones, tablets, laptops, and desktop computers. When terminal devices 101, 102, and 103 are software, they can be installed in the aforementioned electronic devices. They can be implemented as multiple software programs or software modules (e.g., software or software modules used to provide distributed services) or as a single software program or software module. No specific limitations are imposed here.

[0056] Server 105 can be a server that provides various services, such as a background data processing server that processes files or data uploaded by terminal device 101, terminal device 202, and terminal device 303. The background data processing server can process the acquired files or data and generate processing results.

[0057] It should be noted that the drivable region segmentation method based on splittable convolutional large-scale selective attention provided in this application embodiment can be executed by server 105, or by terminal device 101, terminal device 202, or terminal device 303. Correspondingly, the drivable region segmentation device based on splittable convolutional large-scale selective attention can be set in server 105, or in terminal device 101, terminal device 202, or terminal device 303.

[0058] It should be understood that Figure 1 The number of terminal devices, networks, and servers shown is merely illustrative. Any number of terminal devices, networks, and servers can be included depending on implementation needs. If the data being processed does not need to be retrieved remotely, the above architecture may not include a network, requiring only servers or terminal devices.

[0059] Figure 2 This application illustrates an embodiment of a drivable region segmentation method based on large-scale selective attention using splittable convolution, comprising the following steps:

[0060] S1, acquire real-time images of the road.

[0061] Specifically, real-time images of the road can be obtained by a camera mounted on the vehicle. This camera should be mounted in a fixed position on the vehicle to ensure it can capture all necessary information ahead of the vehicle. Furthermore, the camera should be stable after mounting and should not tilt or fall off due to vehicle vibrations. In a preferred embodiment, the camera acquires images at 40 FPS / s.

[0062] S2, Construct and train a drivable region segmentation model based on the improved Deeplabv3+ network to obtain the trained drivable region segmentation model. The drivable region segmentation model based on the improved Deeplabv3+ network sets a splittable convolutional large-scale selective attention module between the backbone network and the ASPP module in the encoder of the traditional Deeplabv3+ network. The splittable convolutional large-scale selective attention module includes a series of large-scale convolutional kernel sequences, a spatial kernel selection unit, and a first convolutional layer. The series of large-scale convolutional kernel sequences includes 2 to n splittable convolutions.

[0063] In a specific embodiment, the backbone network is ResNet-101, and the spatial kernel selection unit includes an average pooling layer, a max pooling layer, a second convolutional layer, and a sigmoid activation function layer.

[0064] In a specific embodiment, the splittable convolution includes several basis sets and a split attention module. The split attention module includes a global average pooling layer, a third convolutional layer, a batch normalization layer, a ReLU activation function layer, and an r-Softmax function layer.

[0065] For details, please refer to Figure 3 The drivable region segmentation model proposed in the embodiments of this application uses the Deeplabv3+ network as a benchmark. Improvements are made to the Deeplabv3+ network; therefore, the improved Deeplabv3+ network includes an encoder and a decoder. The encoder comprises a backbone network, a splittable convolutional large-scale selective attention module, and an ASPP module connected in sequence. The backbone network is ResNet-101, which has five stages, with the corresponding channel numbers for each stage being 64, 256, 512, 1024, and 2048, respectively. The splittable convolutional large-scale selective attention module consists of a series of large-scale convolutional kernel sequences, a spatial kernel selection unit, and a first convolutional layer. The large-scale convolutional kernel sequence can have 2 to n splittable convolutions.

[0066] S3 inputs real-time images into the trained drivable region segmentation model. The real-time images are input into the backbone network to obtain feature vectors. These feature vectors are then input into the splittable convolutional large-scale selective attention module. The feature vectors pass through a series of large-scale convolutional kernel sequences to obtain the output features of each splittable convolution in the series of large-scale convolutional kernel sequences. The output features of each splittable convolution in the series of large-scale convolutional kernel sequences are concatenated to obtain concatenated features. These concatenated features are input into the spatial kernel selection unit to obtain the decomposed large kernel sequence features. The decomposed large kernel sequence features are multiplied by the output features of each splittable convolution to obtain the result of each multiplication. The sum of each multiplication result is then input into the first convolutional layer to obtain attention features. The attention features are multiplied by the feature vector to obtain the output features of the splittable convolutional large-scale selective attention module. The output features of the splittable convolutional large-scale selective attention module are input into the ASPP module and then decoded by the decoder to obtain the segmentation result.

[0067] In a specific embodiment, the nth splittable convolution in a series of large-scale convolution kernel sequences has a kernel size of k, an expansion rate of d, and a receptive field of R. It is then defined as follows:

[0068] k n-1 ≤k n ;d1=1,d n-1 <dn ≤R n-1 ;

[0069] R1 = k1, R n =d n (k n -1)+R n-1 .

[0070] In a specific embodiment, the calculation process of the nth separable convolution in a series of large-scale convolution kernel sequences is as follows:

[0071] Z0 = X,

[0072] Among them, Z n This represents the output feature of the nth splittable convolution. This indicates that the kernel size is k. n The expansion rate is d n The convolutional layer is a splittable convolution of type D, where X represents the feature vector.

[0073] Specifically, the real-time image acquired in step S1 is input into the trained drivable region segmentation model. First, a feature vector is extracted through the backbone network. This feature vector is then input into a splittable convolutional large-scale selective attention module, where the weights corresponding to each pixel are redistributed, allowing the trained drivable region segmentation model to focus more on the drivable region. To obtain features with rich contextual information from different ranges of the input feature vector X, a series of splittable convolutions with different receptive fields are used. The kernel size and dilation rate of each splittable convolution in the series of large-scale convolutional kernels gradually increase, ensuring that the receptive field expands sufficiently quickly. The large-scale convolutional kernel sequence explicitly generates multiple features with various large receptive fields, making subsequent kernel selection easier. Furthermore, sequential decomposition is more efficient than simply applying a single large convolutional kernel, significantly reducing the number of computational parameters.

[0074] In a specific embodiment, the cascaded features are input into the spatial kernel selection unit to obtain the decomposed large kernel sequence features, specifically including:

[0075] The cascaded features are input into the average pooling layer and the max pooling layer respectively to obtain the average pooling features and the max pooling features, as shown in the following formula:

[0076] Sa avg =P avg (Z), Sa max =P max (Z);

[0077] Among them, Sa avg and Sa maxLet P represent the average pooling feature and the max pooling feature, respectively. avg and P max represent the average pooling layer and the max pooling layer, respectively, and Z represents the cascaded feature;

[0078] The average pooling feature and the max pooling feature are concatenated to obtain the pooling feature. The pooling feature is then passed through a second convolutional layer and a sigmoid activation function layer to obtain the decomposed large kernel sequence feature, as shown in the following formula:

[0079]

[0080] Among them, f 2→N This indicates the second convolutional layer, s represents the Sigmoid activation function layer, [Sa avg Sa max ] represents pooling characteristics. This indicates the characteristics of the large nuclear sequence after decomposition.

[0081] For details, please refer to Figure 4 First, the output features of different separable convolutions are concatenated to obtain concatenated features, as shown in the following expression:

[0082] Z = [Z1, ..., Z] n ];

[0083] Among them, Z n This represents the output feature of the nth splittable convolution.

[0084] Cascaded features are input into the spatial kernel selection unit to enhance the network's ability to detect contextual region objects in the most relevant spatial space. First, channel-based average pooling and max pooling layers are applied to effectively extract spatial relationships, yielding average pooling and max pooling features. To allow information exchange between different spatial descriptors, the obtained average pooling and max pooling features are concatenated to obtain pooling features, which are then processed by the second convolutional layer f. 2→N The pooling features are transformed, and a Sigmoid activation function layer is applied to obtain the decomposed large kernel sequence features.

[0085] The decomposed large kernel sequence features are weighted by the output features of their corresponding splittable convolutions and passed through a third convolutional layer f to obtain the attention feature S, as shown in the following expression:

[0086]

[0087] Multiplying the attention feature S by the input feature vector X yields the output feature of the splittable convolutional large-scale selective attention module.

[0088] In a specific embodiment, let the total number of basis sets be Q. The input tensor of the input splittable convolution is input into each basis set to obtain the output of each basis set. The output of the a-th basis set is shown in the following formula:

[0089] T = f a (X');

[0090] Where a∈{1, 2, ..., Q}, X' represents the input tensor that can be split convolutionally, f a This indicates that the kernel size is k. a The expansion rate is d a Convolution operations;

[0091] The outputs of each basis set are combined by summing across elements to obtain the basis set combination, as shown in the following equation:

[0092]

[0093] in, Represents basis set combinations;

[0094] The base set is combined and input into the global average pooling layer to obtain the global background information, as shown in the following formula:

[0095]

[0096] Where H represents global background information, H and W represent the height and width of the basis set combination, and i and j represent the coordinates of the basis set combination along the x-axis and y-axis directions;

[0097] The global background information is passed sequentially through the third convolutional layer, the batch normalization layer, and the ReLU activation function layer to obtain the feature tensor. After the feature tensor is passed through the r-Softmax function layer, it is multiplied by the output of each basis set at the pixel level, and then added at the pixel level to obtain the output features of the splittable convolution.

[0098] For details, please refer to Figure 5 The splittable convolution consists of a basis set and a split attention module. It uses a weighted combination of splitting values ​​to generate the final result for each feature map channel. The split attention module comprises a global average pooling layer, a convolutional layer, a batch normalization layer, a ReLU activation function layer, and an r-Softmax function layer. Global contextual information, embedded from the basis set combinations and channel statistics, is collected through a cross-dimensional global average pooling layer to obtain global background information H. This global background information H is then input to the third convolutional layer, batch normalization layer, and ReLU activation function layer to obtain a feature tensor. After passing through the r-Softmax function layer, the feature tensor is multiplied pixel-wise by the output of the corresponding basis set, and then summed pixel-wise to obtain the output features of the splittable convolution.

[0099] The output features of the splittable convolutional large-scale selective attention module are input into the ASPP module. The purpose is to process these features using dilated convolutions with different receptive field sizes. By setting different dilation rates, the sampling density between the convolutional kernel and the input can be changed, thus achieving receptive fields of varying sizes. The feature maps after each dilated convolution are concatenated with the global average pooling result to obtain a concatenated feature map, capturing global contextual information. The concatenated feature map is then passed through a fourth convolutional layer with a 1×1 kernel for dimensionality reduction and fusion, resulting in dimensionality-reduced features that reduce the number of channels while retaining important features. Next, the dimensionality-reduced features undergo an upsampling operation to obtain an upsampled feature vector. This upsampled feature vector is input into the decoder and fused with the low-level feature vector obtained from the backbone network's feature vector after passing through a fifth convolutional layer with a 1×1 kernel. This fused feature combines low-level and high-level feature information. Finally, the fused features are passed through a sixth convolutional layer with a kernel size of 3×3 and upsampled to obtain the segmentation result.

[0100] The steps S1-S3 above do not necessarily represent the order of the steps, but are represented by step symbols. The order of the steps can be adjusted.

[0101] Further reference Figure 6 As an implementation of the methods shown in the above figures, this application provides an embodiment of a drivable region segmentation device based on large-scale selective attention of splittable convolution. This device embodiment is similar to... Figure 2 Corresponding to the method embodiments shown, this device can be specifically applied to various electronic devices.

[0102] This application provides a drivable region segmentation device based on large-scale selective attention using splittable convolution, comprising:

[0103] Image acquisition module 1 is configured to acquire real-time images of the road;

[0104] Model building module 2 is configured to build and train a drivable region segmentation model based on the improved Deeplabv3+ network to obtain a trained drivable region segmentation model. The drivable region segmentation model based on the improved Deeplabv3+ network sets a splittable convolutional large-scale selective attention module between the backbone network and the ASPP module in the encoder of the traditional Deeplabv3+ network. The splittable convolutional large-scale selective attention module includes a series of large-scale convolutional kernel sequences, a spatial kernel selection unit, and a first convolutional layer. The series of large-scale convolutional kernel sequences includes 2 to n splittable convolutions.

[0105] Segmentation module 3 is configured to input real-time images into a trained drivable region segmentation model. The real-time images are input into the backbone network to obtain feature vectors. These feature vectors are then input into a splittable convolutional large-scale selective attention module. The feature vectors pass through a series of large-scale convolutional kernel sequences to obtain the output features of each splittable convolution in the series of large-scale convolutional kernel sequences. The output features of each splittable convolution in the series of large-scale convolutional kernel sequences are concatenated to obtain concatenated features. These concatenated features are input into a spatial kernel selection unit to obtain decomposed large kernel sequence features. The decomposed large kernel sequence features are multiplied by the output features of each splittable convolution to obtain the result of each multiplication. The sum of each multiplication result is then input into the first convolutional layer to obtain attention features. The attention features are multiplied by the feature vector to obtain the output features of the splittable convolutional large-scale selective attention module. The output features of the splittable convolutional large-scale selective attention module are input into the ASPP module and then decoded by the decoder to obtain the segmentation result.

[0106] The following is for reference. Figure 7 It illustrates an electronic device suitable for implementing embodiments of this application (e.g., Figure 1 The diagram shows the structure of a computer device 700 (a server or terminal device). Figure 7 The electronic device shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of this application.

[0107] like Figure 7 As shown, the computer device 700 includes a central processing unit (CPU) 701 and a graphics processing unit (GPU) 702, which can perform various appropriate actions and processes according to programs stored in read-only memory (ROM) 703 or programs loaded from storage section 709 into random access memory (RAM) 704. The RAM 704 also stores various programs and data required for the operation of the computer device 700. The CPU 701, GPU 702, ROM 703, and RAM 704 are interconnected via a bus 705. An input / output (I / O) interface 706 is also connected to the bus 705.

[0108] The following components are connected to I / O interface 706: an input section 707 including a keyboard, mouse, etc.; an output section 708 including an LCD, speakers, etc.; a storage section 709 including a hard disk, etc.; and a communication section 710 including a network interface card, such as a LAN card or modem. The communication section 710 performs communication processing via a network such as the Internet. A drive 711 may also be connected to I / O interface 706 as needed. A removable medium 712, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on drive 711 as needed so that computer programs read from it can be installed into storage section 709 as needed.

[0109] In particular, according to embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication section 710, and / or installed from removable medium 712. When the computer program is executed by central processing unit (CPU) 701 and graphics processing unit (GPU) 702, the functions defined in the methods of this application are performed.

[0110] It should be noted that the computer-readable medium described in this application can be a computer-readable signal medium, a computer-readable medium, or any combination thereof. A computer-readable medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor device, or any combination thereof. More specific examples of a computer-readable medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this application, a computer-readable medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution device, apparatus, or device. In this application, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. Computer-readable signal media can also be any computer-readable medium other than a computer-readable medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution device, apparatus, or apparatus. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wireless, wire, optical fiber, RF, etc., or any suitable combination thereof.

[0111] Computer program code for performing the operations of this application can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or it can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0112] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of apparatus, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using dedicated hardware-based means to perform the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0113] The modules described in the embodiments of this application can be implemented in software or hardware. These modules can also be located within a processor.

[0114] In another aspect, this application also provides a computer-readable medium, which may be included in the electronic device described in the above embodiments; or it may exist independently and not assembled into the electronic device. The computer-readable medium carries one or more programs, which, when executed by the electronic device, cause the electronic device to: acquire real-time images of the road; construct and train a drivable region segmentation model based on an improved Deeplabv3+ network to obtain a trained drivable region segmentation model; wherein, based on the improved Deeplabv3+ network, a splittable convolutional large-scale selective attention module is set between the backbone network and the ASPP module in the encoder of the traditional Deeplabv3+ network; the splittable convolutional large-scale selective attention module includes a series of large-scale convolutional kernel sequences, a spatial kernel selection unit, and a first convolutional layer; the series of large-scale convolutional kernel sequences includes 2 to n splittable convolutions; input real-time images into the trained drivable region segmentation model; and input real-time images into the backbone network. The feature vector is obtained and input into the splittable convolutional large-scale selective attention module. The feature vector passes through a series of large-scale convolutional kernel sequences to obtain the output features of each splittable convolution in the series of large-scale convolutional kernel sequences. The output features of each splittable convolution in the series of large-scale convolutional kernel sequences are concatenated to obtain concatenated features. The concatenated features are input into the spatial kernel selection unit to obtain the decomposed large kernel sequence features. The decomposed large kernel sequence features are multiplied by the output features of each splittable convolution to obtain the result of each multiplication. The sum of the results of each multiplication is input into the first convolutional layer to obtain the attention features. The attention features are multiplied by the feature vector to obtain the output features of the splittable convolutional large-scale selective attention module. The output features of the splittable convolutional large-scale selective attention module are input into the ASPP module and then decoded by the decoder to obtain the segmentation result.

[0115] The above description is merely a preferred embodiment of this application and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of the invention involved in this application is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described inventive concept. For example, technical solutions formed by substituting the above features with (but not limited to) technical features with similar functions disclosed in this application.

Claims

1. A method for segmenting drivable regions based on large-scale selective attention using splittable convolution, characterized in that, Includes the following steps: Acquire real-time images of the road; A drivable region segmentation model based on an improved Deeplabv3+ network is constructed and trained to obtain a trained drivable region segmentation model. The drivable region segmentation model based on the improved Deeplabv3+ network sets a splittable convolutional large-scale selective attention module between the backbone network and the ASPP module in the encoder of the traditional Deeplabv3+ network. The splittable convolutional large-scale selective attention module includes a series of large-scale convolutional kernel sequences, a spatial kernel selection unit, and a first convolutional layer. The series of large-scale convolutional kernel sequences includes 2 to n splittable convolutions. The real-time image is input into the trained drivable region segmentation model and the backbone network to obtain the feature vector. The feature vector is then input into the splittable convolutional large-scale selective attention module. The feature vector is processed through a series of large-scale convolutional kernel sequences to obtain the output feature of each splittable convolution in the series of large-scale convolutional kernel sequences. The output features of each splittable convolution in the series of large-scale convolutional kernel sequences are concatenated to obtain concatenated features. The concatenated features are input into the spatial kernel selection unit to obtain decomposed large kernel sequence features. The decomposed large kernel sequence features are multiplied by the output features of each splittable convolution to obtain the result of each multiplication. The sum of the results of each multiplication is input into the first convolutional layer to obtain attention features. The attention features are multiplied by the feature vector to obtain the output features of the splittable convolutional large-scale selective attention module. The output features of the splittable convolutional large-scale selective attention module are input into the ASPP module and then decoded by the decoder to obtain the segmentation result.

2. The drivable region segmentation method based on large-scale selective attention using splittable convolution as described in claim 1, characterized in that, If the nth splittable convolution in the series of large-scale convolution kernel sequences has a kernel size of k, an expansion rate of d, and a receptive field of R, then it has the following definition: k n-1 ≤k n ;d1=1,d n-1 <d n ≤R n-1 ; R1=k1,R n =d n (k n -1)+R n-1 。 3. The drivable region segmentation method based on large-scale selective attention using splittable convolution as described in claim 1, characterized in that, The calculation process for the nth splittable convolution in the series of large-scale convolution kernel sequences is as follows: Among them, Z n This represents the output feature of the nth splittable convolution. This indicates that the kernel size is k. n The expansion rate is d n The convolutional layer is a splittable convolution of type D, and X represents the feature vector.

4. The drivable region segmentation method based on large-scale selective attention using splittable convolution as described in claim 1, characterized in that, The backbone network is ResNet-101, and the spatial kernel selection unit includes an average pooling layer, a max pooling layer, a second convolutional layer, and a Sigmoid activation function layer.

5. The drivable region segmentation method based on large-scale selective attention using splittable convolution as described in claim 4, characterized in that, The step of inputting the cascaded features into the spatial kernel selection unit to obtain the decomposed large kernel sequence features specifically includes: The cascaded features are input into the average pooling layer and the max pooling layer respectively to obtain the average pooling features and the max pooling features, as shown in the following formula: With avg =P avg (Z),Sa max =P max (Z); Among them, Sa avg and Sa max Let P represent the average pooling feature and the max pooling feature, respectively. avg and P max represent the average pooling layer and the max pooling layer, respectively, and Z represents the cascaded feature; The average pooling feature and the max pooling feature are concatenated to obtain the pooling feature. The pooling feature is then passed sequentially through the second convolutional layer and the Sigmoid activation function layer to obtain the decomposed large kernel sequence feature, as shown in the following formula: Among them, f 2→N This indicates the second convolutional layer, s represents the Sigmoid activation function layer, [Sa avg Sa max ] represents pooling characteristics. This indicates the characteristics of the large nuclear sequence after decomposition.

6. The drivable region segmentation method based on large-scale selective attention using splittable convolution as described in claim 1, characterized in that, The splittable convolution includes several basis sets and a split attention module, which includes a global average pooling layer, a third convolutional layer, a batch normalization layer, a ReLU activation function layer, and an r-Softmax function layer.

7. The drivable region segmentation method based on large-scale selective attention using splittable convolution as described in claim 6, characterized in that, Let the total number of basis sets be Q. Input the input tensor of the splittable convolution into each basis set to obtain the output of each basis set. The output of the a-th basis set is shown in the following formula: T=f a (X’); Where a∈{1, 2, ..., Q}, X' represents the input tensor of the splittable convolution, f a This indicates that the kernel size is k. a The expansion rate is d a Convolution operations; The outputs of each basis set are combined by summing across elements to obtain the basis set combination, as shown in the following equation: in, Represents basis set combinations; The base set is combined and input into the global average pooling layer to obtain global background information, as shown in the following formula: Where H represents global background information, H and W represent the height and width of the basis set combination, and i and j represent the coordinates of the basis set combination along the x-axis and y-axis directions; The global background information is sequentially passed through the third convolutional layer, the batch normalization layer, and the ReLU activation function layer to obtain a feature tensor. The feature tensor is then passed through the r-Softmax function layer, and then multiplied at the pixel level with the output of each basis set. Finally, the pixels are summed to obtain the output features of the splittable convolution.

8. A drivable region segmentation device based on large-scale selective attention of splittable convolution, characterized in that, include: The image acquisition module is configured to acquire real-time images of the road; The model building module is configured to build and train a drivable region segmentation model based on the improved Deeplabv3+ network to obtain a trained drivable region segmentation model. The drivable region segmentation model based on the improved Deeplabv3+ network sets a splittable convolutional large-scale selective attention module between the backbone network and the ASPP module in the encoder of the traditional Deeplabv3+ network. The splittable convolutional large-scale selective attention module includes a series of large-scale convolutional kernel sequences, a spatial kernel selection unit, and a first convolutional layer. The series of large-scale convolutional kernel sequences includes 2 to n splittable convolutions. The segmentation module is configured to input the real-time image into the trained drivable region segmentation model. The real-time image is input into the backbone network to obtain the feature vector. The feature vector is input into the splittable convolutional large-scale selective attention module. The feature vector is processed through a series of large-scale convolutional kernel sequences to obtain the output feature of each splittable convolution in the series of large-scale convolutional kernel sequences. The output features of each splittable convolution in the series of large-scale convolutional kernel sequences are concatenated to obtain concatenated features. The concatenated features are input into the spatial kernel selection unit to obtain decomposed large kernel sequence features. The decomposed large kernel sequence features are multiplied by the output features of each splittable convolution to obtain the result of each multiplication. The sum of the results of each multiplication is input into the first convolutional layer to obtain attention features. The attention features are multiplied by the feature vector to obtain the output features of the splittable convolutional large-scale selective attention module. The output features of the splittable convolutional large-scale selective attention module are input into the ASPP module and then decoded by the decoder to obtain the segmentation result.

9. An electronic device, comprising: One or more processors; Storage device for storing one or more programs. When the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in any one of claims 1-7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the method as described in any one of claims 1-7.

Citation Information

Patent Citations

  • Multi-scale lightweight smoke image segmentation method and device

    CN116503726A

  • Multi-task traffic scene detection algorithm based on attention mechanism

    CN116958910A