Road detection method and system based on multi-frame information, and storage medium

Abstract

The invention discloses a road detection method and system based on multi-frame information and a storage medium, and the method comprises the following steps: S1, transmitting continuous multi-frame image data collected by a camera at different moments into an encoder for feature extraction, and obtaining corresponding multi-frame advanced semantic features; s2, sending the multi-frame features obtained in the step S1 into a convolution unit based on long short-term memory (LSTM), so as to carry out recursive fusion and superposition on the multi-frame features in time sequence; s3, the fused features in the step S2 are sent to a decoder for feature map decoding; and S4, obtaining corresponding point coordinates according to the feature map obtained by decoding in the step S3. According to the method, the lane line and road boundary detection and extraction are carried out by combining the features of the multi-frame image, additional hardware does not need to be added, meanwhile, the operation speed can be kept within the actual application range, some complex scenes are solved, and the result is more accurate. The scheme is not only suitable for lane lines, but also can be applied to road boundaries and the like.

Description

Technical Field

[0001] The present invention relates to the field of unmanned driving technology, and in particular to a method, system and storage medium for performing road detection based on multi-frame information. Background Art

[0002] In recent years, unmanned driving technology has achieved rapid development and has been successfully applied to all aspects of people's daily lives. For example, unmanned food delivery vehicles avoid human contact and are safer. In production workshops, unmanned vehicles are used for logistics distribution, which greatly improves production efficiency and reduces the error rate. In addition, in the mining industry, the use of unmanned mobile equipment for collection and transportation not only ensures the personal safety of workers, but also reduces labor costs and brings more benefits to enterprises. Unmanned surround view vehicles clean up garbage on the road along a certain route without the help of sanitation workers.

[0003] Autonomous driving plays a huge role in the application of these scenarios. Perception and planning control are important components of autonomous driving. Perception is equivalent to human eyes, telling the system what is in the environment, while planning control makes decisions based on the information obtained from perception.

[0004] Unmanned vehicles need to obtain all elements in the environment during their movement. This includes road information, such as lane information, road boundaries, etc. All moving objects around, such as other vehicles (cars, trucks, and other motor vehicles), non-motor vehicles, and people, etc. Other obstacles, such as stones, trees, green belts, etc. Accurately analyzing these elements is of great significance in unmanned driving.

[0005] Patent CN115775346A provides a lane line detection model training method and a lane line detection method. This method extracts features based on the input road image through a convolutional neural network. Then, based on the high-level semantic information, the polynomial parameters of each lane line in the image are calculated. The parameters include the starting coordinates of the lane line, the inclination angle of the lane line, and the length of the lane line. Using these calculated parameters, all points on each lane line can be calculated. However, this method uses key parameters for lane line detection, which is only applicable to straight lane lines with very regular shapes. When there are lane lines that are partially straight and partially curved at a bend, or when they are S-shaped, the fitting of these parameters is prone to errors and errors. In the case of multiple shapes crossing, it is difficult to accurately fit the parameters. In addition, when the lane line is partially blocked, errors will also be introduced.

[0006] Patent CN118334607A provides a lane line detection method, which first extracts features from the input image sample. Then, on the feature map, the coordinates of the lane line part are screened and calculated. At the same time, the category to which the lane line belongs is calculated based on the features. The loss is calculated based on the predicted result and the input true value. However, this method performs point detection of lane lines based on the feature map, which is prone to missed detection when there are scenes such as occlusion, shadows, or severe line degradation. Summary of the invention

[0007] In view of the problems in the prior art, the present invention proposes a new real-time lane line or road boundary detection method in combination with practical application scenarios. By combining the features of multiple frames of images, lane line and road boundary detection and extraction are performed without adding additional hardware. At the same time, the computing speed can also be kept within the scope of practical application to solve some complex scenarios and make the results more accurate. This solution is not only applicable to lane lines, but also to road boundaries, etc.

[0008] In order to achieve the above-mentioned purpose, the technical solution of the present invention provides a method for road detection based on multi-frame information, which comprises the following steps: S1: sending multiple frames of image data collected by a camera at different times to an encoder for feature extraction to obtain corresponding multi-frame high-level semantic features; S2: sending the multi-frame features obtained in step S1 to a convolution unit based on long short-term memory (LSTM) to recursively fuse and superimpose the multi-frame features in time sequence; S3: sending the fused features in step S2 to a decoder for feature map decoding; S4: obtaining the corresponding point coordinates according to the feature map obtained by decoding in step S3.

[0009] Furthermore, in step S1, the camera collects m consecutive frames of image data at time 0-(m-1), and the encoder obtains the corresponding m image features.

[0010] In step S2, n LSTM convolution units are used to perform feature fusion on m image features, where in a single LSTM convolution unit,

[0011]

[0012] y t =σ(w'h t ),

[0013]

[0014] Among them, c t-1 Indicates the state of the convolution unit at the previous moment, h t-1 represents the hidden state at the previous moment, x t Represents the image feature input at the current moment, yt The feature output at the current moment, c t Indicates the current state of the convolution unit, h t represents the hidden state at the current moment, w, w i 、w f and w o represents weight, tanh and σ are activation functions, z, z i 、z f and z o is the corresponding intermediate output;

[0015] Among them, when the m image features are fused through n LSTM convolution units, the image feature x at time 0 is 0 Input the 0th LSTM and get the output Then As the image feature input of the first LSTM, the output is obtained and will And the image feature x at the first moment 1 As the input of the 0th LSTM, the output is obtained Then As the image feature input of the first LSTM, and input Get Output And so on, we finally get the feature output of the n-1th LSTM convolution unit. That is, the fusion feature output result, where at the 0th moment, the convolution unit state and hidden state of the previous moment take random values.

[0016] Furthermore, in step S4, the corresponding point coordinates are obtained by normalized index operation:

[0017]

[0018] Among them, z i is the output value of the i-th point on the feature map, z c Yes i The total number of, i ranges from 1 to c, C is the number of layers of output points, the size of the feature map is C*W*H, W is the width of the feature map, and H is the height of the feature map.

[0019] Furthermore, in step S1, the encoder adopts a convolutional neural network and an attention mechanism, and in step S3, the decoder adopts a convolutional neural network.

[0020] The technical solution of the present invention also provides a system for road detection based on multi-frame information, which includes the following modules: an image feature extraction module, which is used to send multiple frames of image data collected by a camera at different times into an encoder for feature extraction to obtain corresponding multi-frame high-level semantic features; a feature fusion module, which is used to send the multi-frame features obtained by the image feature extraction module into a convolution unit based on long short-term memory (LSTM) to recursively fuse and superimpose the multi-frame features in time sequence; a decoding module, which is used to send the features fused by the feature fusion module into a decoder for feature map decoding; a point coordinate acquisition module, which is used to obtain corresponding point coordinates based on the feature map obtained by decoding the decoding module.

[0021] Furthermore, in the image feature extraction module, the camera collects m consecutive frames of image data at time 0-(m-1), and the encoder obtains the corresponding m image features.

[0022] In the feature fusion module, n LSTM convolution units are used to fuse m image features. In a single LSTM convolution unit,

[0023]

[0024] y t =σ(w'h t ),

[0025]

[0026] Among them, c t-1 Indicates the state of the convolution unit at the previous moment, h t-1 represents the hidden state at the previous moment, x t Represents the image feature input at the current moment, y t The feature output at the current moment, c t Indicates the current state of the convolution unit, h t represents the hidden state at the current moment, w, w i 、w f and w o represents weight, tanh and σ are activation functions, z, z i 、z f and z o is the corresponding intermediate output;

[0027] Among them, when the m image features are fused through n LSTM convolution units, the image feature x at time 0 is 0 Input the 0th LSTM and get the output Then As the image feature input of the first LSTM, the output is obtained and will And the image feature x at the first moment 1 As the input of the 0th LSTM, the output is obtained Then As the image feature input of the first LSTM, and input Get Output And so on, we finally get the feature output of the n-1th LSTM convolution unit. That is, the fusion feature output result, where at the 0th moment, the convolution unit state and hidden state of the previous moment take random values.

[0028] Furthermore, in the point coordinate acquisition module, the corresponding point coordinates are obtained through normalized index operation:

[0029]

[0030] Among them, z i is the output value of the i-th point on the feature map, z c Yes i The total number of, i ranges from 1 to c, C is the number of layers of output points, the size of the feature map is C*W*H, W is the width of the feature map, and H is the height of the feature map.

[0031] Furthermore, in the image feature extraction module, the encoder adopts a convolutional neural network and an attention mechanism, and in the decoding module, the decoder adopts a convolutional neural network.

[0032] The technical solution of the present invention also provides a computer-readable storage medium containing a computer program, and when the computer program is executed by one or more processors, the method as described above is implemented. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments are briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without creative work.

[0034] Figure 1 and Figure 2 This is the working principle diagram of the LSTM convolution unit;

[0035] Figure 3 It is a flowchart of the method for performing road detection based on multi-frame information of the present invention. DETAILED DESCRIPTION

[0036] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0037] The present invention proposes a real-time detection method based on continuous multiple frames, such as Figure 3 As shown, the process is as follows:

[0038] (1) The scene images captured by the camera, multiple frames of data at different times, are sent to the encoder for feature extraction. Figure 3 As shown, m consecutive frames (0-(m-1) moments) of image data are collected, and the corresponding m image features are extracted through the encoder. The value of m can be flexibly set according to the application scenario. Generally, the value range of m is 5-10. The encoder does not have a single form, and convolutional neural networks, attention mechanisms, etc. can be used. Specifically, adaptive adjustments are made according to the application scenario. The high-level semantic features of each frame of the image are obtained through the encoder. For example, the basic unit of the attention mechanism is composed of query, key, and value. Among them,

[0039] query---The feature representation of the current input

[0040] key----Representation of important features

[0041] value----characteristic vector, usually appears in pairs with key

[0042] output=Score(query,key)*Value

[0043] Score is a calculation function that can be defined according to the application, and the input is query and key.

[0044] (2) The features of multiple frames are sent to the convolution based on long short-term memory (LSTM) to recursively fuse and superimpose the features of multiple frames in time series.

[0045] Among them, the working principle of an LSTM unit is as follows Figure 1 and Figure 2 As shown, in a single LSTM convolutional unit,

[0046]

[0047] y t =σ(w'ht ),

[0048]

[0049] Among them, c t-1 Indicates the state of the convolution unit at the previous moment, h t-1 represents the hidden state at the previous moment, x t Represents the image feature input at the current moment, y t The feature output at the current moment, c t Indicates the current state of the convolution unit, h t represents the hidden state at the current moment, w, w i 、w f and w o represents weight, tanh and σ are activation functions, z, z i 、z f and z o Corresponding to the intermediate output.

[0050] In specific examples, see, for example, Figure 3 , m image features can be fused and superimposed through n (0-(n-1)) LSTM units. The value of n can be flexibly set according to the application scenario. Generally, the value of n ranges from 3 to 6. Specifically, the image feature x at time 0 is 0 Input the 0th LSTM and get the output Then As the image feature input of the first LSTM, the output is obtained and will And the image feature x at the first moment 1 As the input of the 0th LSTM, the output is obtained Then As the image feature input of the first LSTM, and input Get Output And so on, we finally get the feature output of the n-1th LSTM convolution unit. That is, the fusion feature output result, where at the 0th moment, the convolution unit state and hidden state of the previous moment take random values.

[0051] (3) The fused features are sent to the decoder. Similar to the decoder, there is no specific form and it can be designed according to the actual application. For example, a convolutional network can be used to decode the feature map. The decoder includes convolution + softmax operations. The feature map processed by softmax can directly obtain the lane line or roadside coordinates, while the features before decoding are high-level semantic information.

[0052] (4) According to the decoded feature map, the corresponding point coordinates can be obtained through normalized exponential operation (softmax):

[0053]

[0054] Among them, z i is the output value of the i-th point on the feature map, z c Yes i The total number of, i ranges from 1 to c, C is the number of layers of output points, the size of the feature map is C*W*H, W is the width of the feature map, and H is the height of the feature map.

[0055] In an embodiment of the present invention, a method for road detection based on multi-frame information is provided, which includes the following steps: S1: sending multiple frames of image data collected by a camera at different times to an encoder for feature extraction to obtain corresponding multi-frame high-level semantic features; S2: sending the multi-frame features obtained in step S1 to a convolution unit based on long short-term memory (LSTM) to recursively fuse and superimpose the multi-frame features in time sequence; S3: sending the fused features in step S2 to a decoder for feature map decoding; S4: obtaining the corresponding point coordinates according to the feature map obtained by decoding in step S3.

[0056] Furthermore, in step S1, the camera collects m consecutive frames of image data at time 0-(m-1), and the encoder obtains the corresponding m image features.

[0057] In step S2, n LSTM convolution units are used to perform feature fusion on m image features, where in a single LSTM convolution unit,

[0058]

[0059] y t =σ(w'h t ),

[0060]

[0061] Among them, c t-1 Indicates the state of the convolution unit at the previous moment, h t-1 represents the hidden state at the previous moment, x t Represents the image feature input at the current moment, y t The feature output at the current moment, c t Indicates the current state of the convolution unit, h t represents the hidden state at the current moment, w, w i 、w f and w orepresents weight, tanh and σ are activation functions, z, z i 、z f and z o is the corresponding intermediate output;

[0062] Among them, when the m image features are fused through n LSTM convolution units, the image feature x at time 0 is 0 Input the 0th LSTM and get the output Then As the image feature input of the first LSTM, the output is obtained and will And the image feature x at the first moment 1 As the input of the 0th LSTM, the output is obtained Then As the image feature input of the first LSTM, and input Get Output And so on, we finally get the feature output of the n-1th LSTM convolution unit. That is, the fusion feature output result, where at the 0th moment, the convolution unit state and hidden state of the previous moment take random values.

[0063] Furthermore, in step S4, the corresponding point coordinates are obtained by normalized index operation:

[0064]

[0065] Among them, z i is the output value of the i-th point on the feature map, z c Yes i The total number of, i ranges from 1 to c, C is the number of layers of output points, the size of the feature map is C*W*H, W is the width of the feature map, and H is the height of the feature map.

[0066] Furthermore, in step S1, the encoder adopts a convolutional neural network and an attention mechanism, and in step S3, the decoder adopts a convolutional neural network.

[0067] In another embodiment of the present invention, a system for road detection based on multi-frame information is provided, which includes the following modules: an image feature extraction module, which is used to send multiple frames of image data collected by a camera at different times to an encoder for feature extraction to obtain corresponding multi-frame high-level semantic features; a feature fusion module, which is used to send the multi-frame features obtained by the image feature extraction module to a convolution unit based on long short-term memory (LSTM) to recursively fuse and superimpose the multi-frame features in time series; a decoding module, which is used to send the features fused by the feature fusion module to a decoder for feature map decoding; a point coordinate acquisition module, which is used to obtain corresponding point coordinates based on the feature map obtained by decoding the decoding module.

[0068] Furthermore, in the image feature extraction module, the camera collects m consecutive frames of image data at time 0-(m-1), and the encoder obtains the corresponding m image features.

[0069] In the feature fusion module, n LSTM convolution units are used to fuse m image features. In a single LSTM convolution unit,

[0070]

[0071] y t =σ(w'h t ),

[0072]

[0073] Among them, c t-1 Indicates the state of the convolution unit at the previous moment, h t-1 represents the hidden state at the previous moment, x t Represents the image feature input at the current moment, y t The feature output at the current moment, c t Indicates the current state of the convolution unit, h t represents the hidden state at the current moment, w, w i 、w f and w o represents weight, tanh and σ are activation functions, z, z i 、z f and z o is the corresponding intermediate output;

[0074] Among them, when the m image features are fused through n LSTM convolution units, the image feature x at time 0 is 0 Input the 0th LSTM and get the output Then As the image feature input of the first LSTM, the output is obtained and will And the image feature x at the first moment 1 As the input of the 0th LSTM, the output is obtained Then As the image feature input of the first LSTM, and input Get Output And so on, we finally get the feature output of the n-1th LSTM convolution unit. That is, the fusion feature output result, where at the 0th moment, the convolution unit state and hidden state of the previous moment take random values.

[0075] Furthermore, in the point coordinate acquisition module, the corresponding point coordinates are obtained through normalized index operation:

[0076]

[0077] Among them, z i is the output value of the i-th point on the feature map, z c Yes i The total number of, i ranges from 1 to c, C is the number of layers of output points, the size of the feature map is C*W*H, W is the width of the feature map, and H is the height of the feature map.

[0078] Furthermore, in the image feature extraction module, the encoder adopts a convolutional neural network and an attention mechanism, and in the decoding module, the decoder adopts a convolutional neural network.

[0079] In other embodiments of the present invention, a computer-readable storage medium containing a computer program is provided. When the computer program is executed by one or more processors, the method described above is implemented.

[0080] The beneficial technical effects brought by the technical solution of the present invention are:

[0081] The detection method proposed in the present invention is not only applicable to lane line detection, but also to road boundary detection;

[0082] The present invention proposes a method based on continuous frames, which can effectively solve the influence of occlusion, disconnection, etc. and effectively reduce the error compared with the single-frame solution;

[0083] The present invention combines convolution with time series, and has certain improvements for curves or bends that are difficult to detect;

[0084] The solution proposed by the present invention does not introduce additional hardware overhead and does not increase the cost. At the same time, the feature processing of time series can fuse the semantic information of multiple frames, thereby improving the accuracy of complex scenes. The decoder is added to the network to reduce the error and time consumption caused by additional post-processing. It has been verified by practical applications that compared with the original method, it does not increase more time consumption, improves accuracy, and saves hardware resource overhead, and can be widely promoted and applied.

[0085] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can make equivalent replacements or changes according to the technical scheme and inventive concept of the present invention within the technical scope disclosed by the present invention, which should be covered by the protection scope of the present invention.

Claims

1. A method for road detection based on multi-frame information, characterized in that: The steps include: S1: Send the continuous multi-frame image data collected by the camera at different times to the encoder for feature extraction to obtain the corresponding multi-frame high-level semantic features; S2: Send the multi-frame features obtained in step S1 into the convolution unit based on long short-term memory (LSTM) to recursively fuse and superimpose the multi-frame features in time sequence; S3: Send the features fused in step S2 to the decoder for feature map decoding; S4: Obtain the coordinates of the corresponding points according to the feature map decoded in step S3.

2. The method according to claim 1, characterized in that In step S1, the camera collects m consecutive frames of image data at time 0-(m-1), and the encoder obtains the corresponding m image features, and In step S2, n LSTM convolution units are used to perform feature fusion on m image features, where in a single LSTM convolution unit, y t =σ(w'h t ), Among them, c t-1 Indicates the state of the convolution unit at the previous moment, h t-1 represents the hidden state at the previous moment, x t Represents the image feature input at the current moment, y t The feature output at the current moment, c t Indicates the current state of the convolution unit, h t represents the hidden state at the current moment, w, w i 、w f and w o represents weight, tanh and σ are activation functions, z, z i 、z f and z o is the corresponding intermediate output; Among them, when the m image features are fused through n LSTM convolution units, the image feature x at time 0 is 0 Input the 0th LSTM and get the output Then As the image feature input of the first LSTM, the output is obtained and will And the image feature x at the first moment 1 As the input of the 0th LSTM, the output is obtained Then As the image feature input of the first LSTM, and input Get Output And so on, we finally get the feature output of the n-1th LSTM convolution unit. That is, the fusion feature output result, where at the 0th moment, the convolution unit state and hidden state of the previous moment take random values.

3. The method according to claim 1, characterized in that In step S4, the corresponding point coordinates are obtained by normalized index operation: Among them, z i is the output value of the i-th point on the feature map, z c Yes i The total number of, i ranges from 1 to c, C is the number of layers of output points, the size of the feature map is C*W*H, W is the width of the feature map, and H is the height of the feature map.

4. The method according to claim 1, characterized in that: In step S1, the encoder adopts a convolutional neural network and an attention mechanism, and in step S3, the decoder adopts a convolutional neural network.

5. A system for road detection based on multi-frame information, characterized in that: Includes the following modules: The image feature extraction module is used to send the image data of multiple frames at different times collected by the camera to the encoder for feature extraction to obtain the corresponding multi-frame high-level semantic features; The feature fusion module is used to input the multi-frame features obtained by the image feature extraction module into the convolution unit based on the long short-term memory (LSTM) to perform recursive fusion and superposition of the multi-frame features in time sequence; The decoding module is used to send the features fused by the feature fusion module to the decoder for feature map decoding; The point coordinate acquisition module is used to obtain the corresponding point coordinates according to the feature map decoded by the decoding module.

6. The system according to claim 5, characterized in that In the image feature extraction module, the camera collects m consecutive frames of image data at time 0-(m-1), and the encoder obtains the corresponding m image features, and In the feature fusion module, n LSTM convolution units are used to fuse m image features. In a single LSTM convolution unit, y t =σ(w'h t ), Among them, c t-1 Indicates the state of the convolution unit at the previous moment, h t-1 represents the hidden state at the previous moment, x t Represents the image feature input at the current moment, y t The feature output at the current moment, c t Indicates the current state of the convolution unit, h t represents the hidden state at the current moment, w, w i 、w f and w o represents weight, tanh and σ are activation functions, z, z i 、z f and z o is the corresponding intermediate output; Among them, when the m image features are fused through n LSTM convolution units, the image feature x at time 0 is 0 Input the 0th LSTM and get the output Then As the image feature input of the first LSTM, the output is obtained and will And the image feature x at the first moment 1 As the input of the 0th LSTM, the output is obtained Then As the image feature input of the first LSTM, and input Get Output And so on, we finally get the feature output of the n-1th LSTM convolution unit. That is, the fusion feature output result, where at the 0th moment, the convolution unit state and hidden state of the previous moment take random values.

7. The system according to claim 5, characterized in that In the point coordinate acquisition module, the corresponding point coordinates are obtained through normalized index operation: Among them, z i is the output value of the i-th point on the feature map, z c Yes i The total number of, i ranges from 1 to c, C is the number of layers of output points, the size of the feature map is C*W*H, W is the width of the feature map, and H is the height of the feature map.

8. The system according to claim 5, characterized in that In the image feature extraction module, the encoder adopts convolutional neural network and attention mechanism, and in the decoding module, the decoder adopts convolutional neural network.

9. A computer-readable storage medium containing a computer program, characterized in that: When the computer program is executed by one or more processors, the method according to any one of claims 1 to 4 is implemented.

Images