Lightweight neural network sensing and positioning method based on vision

Abstract

The invention requests to protect a vision-based lightweight neural network sensing and positioning method. A network constructed by the method is divided into three layers, namely an input layer, a processing layer and an output layer. The input layer obtains image sequence, positioning, acceleration and angular velocity information through a camera, a global navigation satellite system (GNSS) and an inertial measurement unit (IMU). And the processing layer performs feature extraction on the information acquired by the input layer based on a coding network, performs target detection decoding, semantic segmentation decoding and motion estimation decoding by using a decoding network, and performs fusion to obtain a semantic segmentation result and a pose result. And the output layer outputs target detection, semantic segmentation and vehicle positioning results. According to the method, target detection, semantic segmentation and positioning functions are provided at the same time, dependence on high-precision positioning equipment is avoided, the calculation efficiency and precision of intelligent vehicle sensing and positioning are effectively improved, and meanwhile the calculation cost is reduced through lightweight deployment of the model.

Description

Technical Field

[0001] The present invention relates to the fields of deep learning, computer vision and intelligent vehicle environment perception and positioning, and in particular to a vision-based lightweight neural network perception and positioning method. Background Art

[0002] With the development of deep learning, vision-based perception technology has become the key to autonomous driving. Object detection identifies environmental targets, and semantic segmentation is used to identify lane lines and drivable areas. Although GNSS and IMU combined positioning can provide high-precision positioning, it is limited by cost issues. Traditional visual odometers rely on high-cost computing resources and are difficult to apply in practice. At present, most vehicle perception and positioning methods run separately, making it difficult to achieve multi-task collaboration in the same scene, and consuming a large amount of on-board chip computing resources, which is inefficient and costly.

[0003] Chinese patent application: Lightweight neural network road scene detection method integrating multi-dimensional information pooling (application number: CN202211550617.1) discloses a lightweight multi-task neural network method integrating multi-dimensional information pooling, which uses multiple task heads to share a backbone extraction architecture to complete target detection and lane line detection. This method does not support vehicle positioning.

[0004] Chinese patent application: A traffic scene segmentation and visual odometer method based on multi-task neural network (application number: CN202311680517.5) discloses a segmentation and visual odometer method in traffic scenes, which uses a long short-term memory network to obtain positioning results. This method does not involve lightweight models and does not integrate multiple information.

[0005] To sum up, the problems with the existing technology are: visual perception and positioning in autonomous driving use different solutions, which consume a lot of resources and have low computing efficiency. At the same time, they also require support from high-precision positioning equipment, which is costly. Summary of the invention

[0006] The present invention aims to solve the above problems of the prior art. A lightweight neural network perception and positioning method based on vision is proposed, which can simultaneously perform three tasks: target detection, semantic segmentation, and pose estimation. This method plays the role of high-precision positioning and vehicle perception without relying on high-precision positioning equipment, solves the coordination problem between traditional multi-tasks, and realizes a concise and efficient lightweight neural network based on the GELAN module of deep separable convolution, thereby improving computational efficiency and performance.

[0007] The technical solution of the present invention is as follows:

[0008] A lightweight neural network perception and positioning method based on vision, comprising the following steps:

[0009] Image sequence, positioning, acceleration and angular velocity information are obtained through camera, GNSS and IMU respectively;

[0010] The acquired image sequence, positioning, acceleration and angular velocity information are subjected to feature extraction based on the encoding network, and then the decoding network is used to perform target detection decoding, semantic segmentation decoding and motion estimation decoding. The semantic segmentation result is obtained by fusing the target detection information with the semantic segmentation information, and the pose result is obtained by fusing the semantic segmentation information with the pose estimation information.

[0011] Output target detection, drivable area and vehicle positioning results.

[0012] Furthermore, the encoding network structure is composed of a lightweight multi-task feature extraction encoding network, and the decoding network is composed of multiple task decoding networks, specifically including:

[0013] Feature extraction encoding network: Receives multi-frame serialized image data, IMU data and positioning data as input, extracts image features through the GELAN lightweight module based on deep separable convolution and the spatiotemporal self-attention module, and enhances the multi-scale characteristics of the network through the spatial pyramid structure; at the same time, the positioning data extracts key motion features through the long short-term memory LSTM module;

[0014] Decoding network: It consists of three parts: target detection network, semantic segmentation network and pose estimation decoding network. The target detection decoding network adopts a bottom-up path aggregation (PAN) network structure, uses multi-scale fusion feature maps to detect targets on the encoded features, and fuses the detection information into the semantic segmentation decoding network. The semantic segmentation decoding network achieves fine pixel-level classification through upsampling convolutional neural networks. The pose estimation decoding network combines LSTM with a fully connected network, and fuses semantic segmentation information to obtain the vehicle's own pose prediction.

[0015] Furthermore, the GELAN lightweight module based on depthwise separable convolution specifically refers to:

[0016] This module uses the depthwise separable convolution technology to achieve the lightweight design of the GELAN network. Based on the GELAN network structure, the original ordinary convolution is replaced by the depthwise separable convolution. Specifically, the standard convolution is decomposed into two independent steps: depthwise convolution and pointwise convolution. This operation significantly reduces the number of parameters and the amount of calculation, thereby reducing the computational complexity and memory usage of the network, and pays more attention to the lightweight design of the network while maintaining the expressiveness and performance of the model as much as possible.

[0017] Furthermore, the spatiotemporal self-attention module specifically refers to:

[0018] This module consists of a temporal attention model, a fully connected neural network, and a spatial attention model, and adopts a separated spatiotemporal self-attention architecture. This design allows the attention mechanism to be calculated along the temporal axis and the spatial axis respectively, thereby reducing the computational complexity. Specifically, the model first calculates the self-attention between patches in different frames with the same spatial position on the temporal axis to capture dynamic change information in the temporal dimension. Then, the model performs attention calculations on different patches in the same frame on the spatial axis to extract spatial features. This time-space separation of attention calculations can effectively extract spatiotemporal features and improve the performance of the model.

[0019] Furthermore, the training method of the encoding and decoding neural network specifically includes:

[0020] Sample construction: The synchronously collected image data, GNSS data, and IMU data are made into labels: labels for target detection, mask format labels for semantic segmentation, and translation and rotation transformation labels between serialized images for pose estimation;

[0021] Loss function definition: total loss function L total for:

[0022] L total =α1L det +α2L seg +α3L vo #(1)

[0023] Among them, α1, α2 and α3 are used to balance the balance coefficients of the three task losses, and the target detection loss L det It is composed of classification loss, target loss and bounding box loss; semantic segmentation loss L seg It is composed of classification loss and focal loss; pose estimation loss L vo It is composed of the loss of each pose prediction;

[0024] Network training: Based on domain adaptation technology, the distribution difference between training data (source domain) and test data (target domain) is reduced, and the feature space alignment method is used to map the feature space of the source domain and the target domain to a common feature space to reduce the difference between the two domains.

[0025] Furthermore, the fusion method specifically includes fusion at the feature map level and fusion at the result level, wherein:

[0026] Fusion at the feature map level: By introducing the spatiotemporal self-attention module, the weights of the feature maps in the spatiotemporal dimension are dynamically adjusted to capture the dependencies between different tasks. The adjusted feature maps are then fused to enhance the accuracy of the feature expression of the network in multi-task processing.

[0027] Result-level fusion: including fusion perception and fusion positioning, specifically:

[0028] (1) Combine the target detection results with the semantic segmentation information and use the information between consecutive image frames to identify moving targets. By calculating the motion changes between image frames, the position and speed of the vehicle or object in front relative to itself can be determined, and these areas can be marked as "drivable areas" or "non-drivable areas" in the segmented image.

[0029] (2) The fusion of semantic segmentation and pose estimation is based on conditional semantic pose estimation. The results generated by semantic segmentation are used as conditional input and directly integrated into the pose estimation network, so that the network can adaptively adjust the estimation strategy according to these conditions.

[0030] The advantages and beneficial effects of the present invention are as follows:

[0031] The present invention proposes a lightweight neural network perception and positioning method based on vision. The specific innovative steps include: first, a multi-task neural network architecture is designed, which can simultaneously complete target detection, semantic segmentation and pose estimation, and combines image, IMU and GNSS information to perform efficient feature extraction and fusion, so as to play a role in high-precision positioning and vehicle perception without relying on high-precision positioning equipment. Secondly, an innovative feature fusion strategy is adopted to significantly improve the segmentation accuracy and positioning accuracy in semantic segmentation and pose estimation, and combined with actual maps and data sets, accurate positioning is achieved in scenarios with limited positioning signals. Finally, the GELAN aggregation strategy based on deep separable convolution is used to achieve lightweight deployment of the model, reduce the amount of inference calculations, and improve the inference speed of the network. These innovative points cooperate with each other, so that the present invention can reduce the implementation cost and improve the overall performance of the system while achieving high-precision positioning and vehicle perception. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 It is a flow chart of a perception and positioning method applied to vision according to a preferred embodiment of the present invention.

[0033] Figure 2 It is a schematic diagram of the multi-task neural network structure provided by an embodiment of the present invention.

[0034] Figure 3 It is a schematic diagram of the structure of the spatiotemporal self-attention module provided in an embodiment of the present invention.

[0035] Figure 4 It is a schematic diagram of the application results of a lightweight neural network perception and positioning method based on vision provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0036] The following will describe the technical solutions in the embodiments of the present invention in detail in conjunction with the accompanying drawings in the embodiments of the present invention. The described embodiments are only part of the embodiments of the present invention.

[0037] The technical solution of the present invention to solve the above technical problems is:

[0038] Figure 1 The figure shows the workflow of the multi-task neural network for perception and positioning provided by the embodiment of the present invention. The network constructed by the method is divided into three levels: input layer, processing layer and output layer.

[0039] The input layer obtains image sequences, positioning, acceleration and angular velocity information through the camera, GNSS and IMU respectively. The processing layer extracts features from the information obtained by the input layer based on the encoding network, and then uses the decoding network to perform target detection decoding, semantic segmentation decoding and motion estimation decoding, and obtains the semantic segmentation result based on the fusion of target detection information and semantic segmentation information, and obtains the pose result based on the fusion of semantic segmentation information and pose estimation information. The output layer outputs the target detection, drivable area and vehicle positioning results.

[0040] Furthermore, the fusion strategy of the neural network of the technical solution specifically includes: the fusion strategy in the network is divided into feature map level fusion and result level fusion.

[0041] Fusion at the feature map level: By introducing the spatiotemporal self-attention module, the weights of the feature maps in the spatiotemporal dimension are dynamically adjusted to capture the dependencies between different tasks. The adjusted feature maps are then fused to enhance the accuracy of the feature expression of the network in multi-task processing.

[0042] Fusion at the result level: including fusion perception and fusion positioning, specifically: combining the target detection results with semantic segmentation information, and using the information between consecutive image frames to identify moving targets. By calculating the motion changes between image frames, the position and speed of the vehicle or object in front relative to itself can be determined, and these areas can be marked as "drivable areas" or "non-drivable areas" in the segmented image. The fusion of semantic segmentation and pose estimation is based on conditional semantic pose estimation. The results generated by semantic segmentation are used as conditional inputs and directly integrated into the pose estimation network, so that the network can adaptively adjust the estimation strategy according to these conditions.

[0043] Figure 2 It is a codec network structure, which consists of a lightweight multi-task feature extraction encoding network and multiple task decoding networks.

[0044] Feature extraction encoding network: Multi-frame serialized image data is first fused on the channel through a feature extraction structure based on deep separable convolution, and then processed by a GELAN module based on deep separable convolution, effectively combining deep and shallow features to extract high-level semantic information and underlying detail information. Subsequently, these features are enhanced by a spatiotemporal attention module to accurately extract key image features. In the final stage of extracting image features, a spatial pyramid structure is applied to adapt the network to multi-scale characteristics and enhance the robustness of prediction. For positioning data, key motion features are extracted through a long short-term memory network (LSTM), and these motion features are fused with image features, semantic segmentation results, and IMU data to guide the network to learn features for pose estimation, while ensuring that the complexity of the entire network is controlled and more semantic information is retained.

[0045] Decoding network: After realizing the target detection encoding, the target detection, semantic segmentation and pose estimation decoding networks are constructed. The target detection decoding network adopts the bottom-up path aggregation network (PAN). By combining the features of PAN and feature pyramid network (FPN), it can fuse semantic information with position features, and then detect on the multi-scale fusion feature map. The semantic segmentation decoding network is constructed by upsampling the feature layer using the nearest neighbor interpolation method. The pose estimation decoding network passes through LSTM and fully connected network, and the output of LSTM is mapped to the visual odometer to accurately predict the vehicle's own pose. At the same time, the result-level fusion strategy is designed in the semantic segmentation and pose estimation networks respectively.

[0046] Furthermore, the neural network training method of the technical solution specifically includes: training data set construction, loss function definition and training techniques.

[0047] (1) Construction of training data set: The on-board sensors are used to collect and generate synchronized images and vehicle motion data, and to generate data sets that meet the requirements of multi-task neural network training. Specifically, the data is collected from the camera, IMU, RTK device, and GNSS device installed on the vehicle, and all devices are synchronized. For example, the acquisition frequency is set to 10 Hz, and the RTK device is used to ensure accurate positioning data as training data. According to the network input, the labels are divided into three categories: labels for target detection, labels in mask format for semantic segmentation, and labels for translation and rotation transformation between serialized images for pose estimation. The labels for target detection are annotated using the LabelImg annotation tool, the labels in mask format for semantic segmentation are annotated using the LabelIme annotation tool, and the labels for pose estimation are preprocessed to obtain the corresponding pose transformation in the image sequence and convert it into six-degree-of-freedom pose data.

[0048] (2) Loss function definition: The total loss function of the multi-task neural network is Ltotal , the specific expression is shown in the above formula 1, L total The target detection loss L det , semantic segmentation loss L seg and pose estimation loss L vo The target detection loss L det It consists of classification loss, target loss and bounding box loss, as shown in Expression 2:

[0049] L det =γ1L class +γ2L obj +γ3L box #(2)

[0050] Among them, γ1, γ2 and γ3 are adjustable parameters, L class and L obj Focal Loss is used to solve the problem of sample imbalance. box Use CIoU Loss.

[0051] Semantic segmentation loss L seg It consists of classification loss and focal loss, as shown in Expression 3:

[0052] L seg =L ce +L IoU #(3)

[0053] Among them, L ce Using Cross Entropy Loss, L IoU Use DIoU Loss.

[0054] Design L vo It is composed of the loss of each pose prediction, as shown in Expression 4:

[0055]

[0056] Among them, L pose (i) represents the loss of the i-th prediction, using the position error loss function.

[0057] (3) Network training: Domain adaptation technology is used to reduce the distribution difference between training data (source domain) and test data (target domain) through the maximum mean difference method. Feature space alignment is used to find a change, and the feature space of the source domain and the target domain is mapped to a common space to reduce the difference between the two domains. At the same time, a distributed training strategy is designed so that the proposed loss function can optimize the network parameters during training. Specifically, the parameters in the pose estimation network are first frozen, and the parameters of the feature extraction encoding network, the parameters of the target detection decoding network, and the parameters of the semantic segmentation decoding are updated during training. Then, the parameters of the feature extraction encoding network, the parameters of the target detection decoding network, and the parameters of the semantic segmentation decoding are frozen, and the parameters of the pose estimation network are updated. Finally, all parameters are unfrozen and the parameters of the entire network are updated through joint training.

[0058] Furthermore, the fusion strategy of the neural network of the technical solution specifically includes: the fusion strategy in the network is divided into feature map level fusion and result level fusion.

[0059] Fusion at the feature map level: By introducing the spatiotemporal self-attention module, the weights of the feature maps in the spatiotemporal dimension are dynamically adjusted to capture the dependencies between different tasks. The adjusted feature maps are then fused to enhance the accuracy of the feature expression of the network in multi-task processing.

[0060] Fusion at the result level: including fusion perception and fusion positioning, specifically: combining the target detection results with semantic segmentation information, and using the information between consecutive image frames to identify moving targets. By calculating the motion changes between image frames, the position and speed of the vehicle or object in front relative to itself can be determined, and these areas can be marked as "drivable areas" or "non-drivable areas" in the segmented image. The fusion of semantic segmentation and pose estimation is based on conditional semantic pose estimation. The results generated by semantic segmentation are used as conditional inputs and directly integrated into the pose estimation network, so that the network can adaptively adjust the estimation strategy according to these conditions.

[0061] Figure 3 It is a spatiotemporal self-attention module, which first calculates the attention of patches from the same spatial position on the time axis, and then calculates the attention between patches from the same frame. This can effectively reduce the computational complexity while capturing the spatiotemporal features in the input data.

[0062] The systems, devices, modules or units described in the above embodiments may be implemented by computer chips or entities, or by products with certain functions.

[0063] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0064] The above embodiments should be understood to be only used to illustrate the present invention and not to limit the protection scope of the present invention. After reading the contents of the present invention, technicians can make various changes or modifications to the present invention, and these equivalent changes and modifications also fall within the scope defined by the claims of the present invention.

Claims

1. A lightweight neural network perception and positioning method based on vision, characterized in that: The following steps are involved: Image sequence, positioning, acceleration and angular velocity information are obtained through camera, GNSS and IMU respectively; The acquired image sequence, positioning, acceleration and angular velocity information are subjected to feature extraction based on the encoding network, and then the decoding network is used to perform target detection decoding, semantic segmentation decoding and motion estimation decoding. The semantic segmentation result is obtained by fusing the target detection information with the semantic segmentation information, and the pose result is obtained by fusing the semantic segmentation information with the pose estimation information. Output target detection, drivable area and vehicle positioning results.

2. The method of lightweight neural network perception and positioning based on vision according to claim 1, characterized in that: The encoding network structure is composed of a lightweight multi-task feature extraction encoding network, and the decoding network is composed of multiple task decoding networks, specifically including: 2).1 Feature extraction encoding network: Receives multi-frame serialized image data, IMU data and positioning data as input, extracts image features through the GELAN lightweight module based on deep separable convolution and the spatiotemporal self-attention module, and enhances the multi-scale characteristics of the network through the spatial pyramid structure; at the same time, the positioning data extracts key motion features through the long short-term memory LSTM module; 2).2 Decoding network: It includes three parts: target detection network, semantic segmentation network and pose estimation decoding network; the target detection decoding network adopts a bottom-up path aggregation network structure, uses multi-scale fusion feature maps to perform target detection on the encoded features in 2.1, and fuses the detection information into the semantic segmentation decoding network; the semantic segmentation decoding network realizes fine pixel-level classification through upsampling method; the pose estimation decoding network combines LSTM with a fully connected network, and fuses semantic segmentation information to obtain the vehicle's own pose prediction.

3. The method of lightweight neural network perception and positioning based on vision according to claim 2, characterized in that: The training method of the encoding and decoding neural network specifically includes: 3).1 Sample construction: The synchronously collected image data, GNSS data and IMU data are made into labels: labels for target detection, mask format labels for semantic segmentation, and translation and rotation transformation labels between serialized images for pose estimation; 3).2 Loss function definition: total loss function L total for: L total =α1L det +α2L seg +α3L vo #(1) Among them, α1, α2 and α3 are used to balance the balance coefficients of the three task losses, and the target detection loss L det It is composed of classification loss, target loss and bounding box loss; semantic segmentation loss L seg It is composed of classification loss and focal loss; pose estimation loss L vo It is composed of the loss of each pose prediction; 3).3 Network training: Based on domain adaptation technology, the distribution difference between training data (source domain) and test data (target domain) is reduced, and the feature space alignment method is used to map the feature space of the source domain and the target domain to a common feature space to reduce the difference between the two domains.

4. The method of lightweight neural network perception and positioning based on vision according to claim 2, characterized in that: The fusion method specifically includes feature map level fusion and result level fusion, wherein: 4)1 Fusion at the feature map level: By introducing the spatiotemporal self-attention module, the weights of the feature maps in the spatiotemporal dimension are dynamically adjusted to capture the dependencies between different tasks; the adjusted feature maps are then fused to enhance the accuracy of the feature expression of the network in multi-task processing; 4)2 Result-level fusion: including fusion perception and fusion positioning, specifically: (1) Combine the target detection results with the semantic segmentation information and use the information between consecutive image frames to identify moving targets. By calculating the motion changes between image frames, the position and speed of the vehicle or object in front relative to itself can be determined, and these areas can be marked as "drivable areas" or "non-drivable areas" in the segmented image. (2) The fusion of semantic segmentation and pose estimation is based on conditional semantic pose estimation. The results generated by semantic segmentation are used as conditional input and directly integrated into the pose estimation network, so that the network can adaptively adjust the estimation strategy according to these conditions.

Images