A visual detection method and system for emergency lanes based on drone perspective

Abstract

A visual detection method for emergency lanes based on the perspective of a drone, comprising: obtaining lane images through a drone, and using the lane image stream as the input of a feature extraction network; adjusting the image size according to preset requirements, and extracting feature maps of different scales; receiving feature maps of different scales, and fusing feature maps of different scales through a multi-scale feature aggregation module to obtain feature values ​​of different layers; adding feature values ​​of different layers, and extracting sampling points according to preset rules to perform feature sampling on the image; classifying according to the features of the sampling points, and performing curve fitting according to the classification results; judging whether the current lane image is an emergency lane according to the fitting curve. The present invention solves the problems that the mainstream lane line detection systems are all based on the driver's perspective, the CNN segmentation network has a large amount of calculation and a slow speed, and there are often complex problems such as lane line size changes, vehicle occlusion, and camera exposure in the perspective of drones.

Description

Technical Field

[0001] The present invention relates to the field of image processing, and in particular to a visual detection method and system for an emergency lane based on the perspective of an unmanned aerial vehicle. Background Art

[0002] Lane detection is a fundamental problem in computer vision and has a wide range of applications in autonomous driving. Since the 1990s, developed countries and regions centered on the United States, Japan, and Europe have conducted relatively concentrated research on lane departure warning technology and have achieved many valuable results. At present, some countries in the world have successfully developed some lane departure warning systems with their own characteristics, such as: Mobileye_AWS system, AutoVue system, RALPH system, AURORA system, etc. These systems provide warning information to the driver in advance to remind the driver to take correct operating measures, so as to prevent such accidents or reduce the degree of damage caused by such accidents.

[0003] Due to economic development, my country started late in this field, but overall the development speed is fast. For example, the THMR series of intelligent vehicles developed by the State Key Laboratory of Intelligent Technology and Systems of Tsinghua University can reach an average speed of 100km / h and a maximum speed of 150km / h when automatically tracking lane lines. The speed is close to the international advanced level, and certain technical breakthroughs have been made in image processing, control algorithms, etc.

[0004] Lane detection technology enables vehicles to correctly locate in lanes and comply with traffic rules stipulated by lanes. Most general lane detection algorithms are based on monocular images. Early lane detection algorithms used traditional methods, usually lane detection methods based on visual information. The main idea is to use visual clues, through image processing such as HIS, color model, and edge extraction algorithm, artificially select features, filter out connected areas according to brightness, width, etc., and use image preprocessing, feature extraction, etc. and combine with algorithms such as Hough transform or Kalman filter to form the final lane after identifying the lane line. Usually, the lane line model based on traditional algorithms has robustness problems due to changes in road scenes. At present, the lane line detection algorithm benefits from the powerful feature extraction ability of convolutional neural networks, and its performance is constantly improving. In recent years, with the development of deep learning, some methods based on deep neural networks have shown superiority in lane detection due to their good representation and learning capabilities. Lane detection algorithms are widely used as a basic component of autonomous driving, which requires it to have a very low computational cost. In this case, it is very important to speed up the lane line detection method. The convolutional neural network (CNN) can be trained through sample images from different perspectives to adaptively extract the key features of lane lines on the road surface, and then classify them through the fully connected layer. The complete detection of lane lines and the segmentation of special areas are the two most critical steps of the system. Since the current mainstream lane line detection systems are all based on the driver's perspective, the CNN segmentation network has a large amount of calculation and a slow speed. In addition, there are often complex problems such as lane line size changes, vehicle occlusion, camera exposure, etc. in the drone's perspective. Therefore, the lane line detection system must work normally under the interference of these factors, which is a great technical challenge. Summary of the invention

[0005] In view of the above problems, the present invention is proposed to provide an emergency lane visual detection method and system based on the perspective of a drone, which overcomes the above problems or at least partially solves the above problems.

[0006] In order to solve the above technical problems, the embodiments of the present application disclose the following technical solutions:

[0007] A visual detection method for emergency lanes based on the perspective of a drone, comprising:

[0008] S100. Acquire a lane image through a drone, and use the lane image stream as an input of a feature extraction network;

[0009] S200. Adjust the image size according to preset requirements and extract feature maps of different scales;

[0010] S300. Determine whether the current image recognition is a training process. When the current image recognition is a training process, receive feature maps of different scales, fuse the feature maps of different scales through a multi-scale feature aggregation module, and obtain feature values ​​of different layers;

[0011] S400. Add the feature values ​​of different layers, extract sampling points according to preset rules, and perform feature sampling on the image;

[0012] S500. Classify according to the characteristics of the sampling points, and perform curve fitting according to the classification results;

[0013] S600. Determine whether the current lane image is an emergency lane based on the fitting curve.

[0014] Furthermore, in S200, feature maps of different scales are extracted through the resnet-34 network structure.

[0015] Furthermore, in S200, the image size is adjusted according to a preset requirement, and the aspect ratio of the image is adjusted to 324×780.

[0016] Furthermore, in S300, the multi-scale feature aggregation module is connected to the outputs of the last three stages of the ResNet-34 network structure, wherein the first two stages of the output are used to predict smaller-scale targets, and the third stage is used to predict larger-scale targets.

[0017] Furthermore, in S400, the preset rule for feature sampling of the image is: the lane is represented as a series of horizontal positions on a predefined row, and grid division is performed, the horizontal position is divided into a plurality of cells, sampling points are extracted at fixed intervals and sent to the fully connected layer.

[0018] Further, in S500, the method for classifying according to the sampling point features includes: setting a classifier for each row, classifying the pixels corresponding to each position in the row, the number of classifiers is the same as the number of row anchors, and each classifier classifies the pixel points at the corresponding position in the original image according to a row of input features, and the number of classifications is one more than the maximum number of lanes when the background is included.

[0019] Furthermore, in S500, the method for classifying according to the characteristics of the sampling points also includes: during the labeling process, the labeling category number of the left line of the emergency lane is set to an even number, and the labeling category number of the right line is set to an odd number, and the left and right sides of the point on the lane line are judged by the parity of the classification result number.

[0020] Furthermore, in S500, when the scene is relatively simple, the lane lines are directly fitted through Hough transform, the coordinates of the same type of detection results in the image are counted and generally fitted, and the straight line with the highest degree of fitting that runs through these points is found.

[0021] Furthermore, in S500, when the scene is more complex, a preset number of detection results are subjected to Hough transform to fit a small straight line, and finally all the fitted small straight lines are connected end to end and merged into a complete curve, and the sampling points on the same lane line are sorted to ensure that all the sampling points on this lane line are traversed in a top-down or bottom-up order when the program traverses the results.

[0022] Furthermore, in S600, the method for determining whether the current lane image is an emergency lane is as follows: after fitting the lane line, the category code of the current lane line can be obtained from the result of network classification, and its parity can indicate whether it is on the left or right side of the emergency lane; all detected lane lines are traversed from left to right, and if the category code has an even-odd combination, it is determined that the part formed by them is the emergency lane, and each row between the two lines is scanned from top to bottom for marking, thereby outputting the final emergency lane detection result map.

[0023] The present invention also discloses an emergency lane visual detection system based on the perspective of a drone, comprising: a lane image acquisition module, a feature map extraction module, a multi-scale feature aggregation module, a sampling point feature classification module, and an emergency lane determination module; wherein: the lane image acquisition module is used to acquire a lane image through a drone, and use the lane image stream as the input of a feature extraction network;

[0024] The feature map extraction module is used to adjust the image size according to the preset requirements and extract feature maps of different scales through the resnet-34 network structure;

[0025] The multi-scale feature aggregation module is used to receive feature maps of different scales, fuse them, and obtain feature values ​​of different layers;

[0026] The sampling point feature classification module is used to add the feature values ​​of different layers, extract sampling points according to preset rules, and perform feature sampling on the image; classify according to the sampling point features, and perform curve fitting according to the classification results;

[0027] The emergency lane determination module is used to determine whether the current lane image is an emergency lane based on the fitting curve.

[0028] The beneficial effects of the above technical solution provided by the embodiment of the present invention include at least:

[0029] The present invention designs many fixed-position sampling rows on the original image, extracts global features through the deep residual network model ResNet, and determines the points that may exist on the lane line through the final fully connected layer classification. The detected points are fitted into curves to segment special areas on the road. The present invention solves the problem in the prior art that the mainstream lane line detection systems are all based on the driver's perspective, the CNN segmentation network has a large amount of calculation and a slow speed, and there are often complex problems such as lane line size changes, vehicle occlusion, camera exposure, etc. in the drone's perspective.

[0030] The technical solution of the present invention is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] The accompanying drawings are used to provide a further understanding of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation of the present invention. In the accompanying drawings:

[0032] Figure 1 This is a flow chart of a method for visually detecting an emergency lane based on a drone perspective in Example 1 of the present invention. DETAILED DESCRIPTION

[0033] The exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although the exemplary embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided to enable a more thorough understanding of the present disclosure and to fully convey the scope of the present disclosure to those skilled in the art.

[0034] In order to solve the problems existing in the prior art, an embodiment of the present invention provides an emergency lane visual detection method based on the perspective of an unmanned aerial vehicle.

[0035] Example 1

[0036] As the scenes faced by lane detection tasks become more and more complex, lane lines may be blurred, affected by lighting, or even completely blocked. Traditional lane detection algorithms can hardly detect lane lines. In this case, lane detection needs to seek higher-level semantic analysis. Deep segmentation methods naturally have stronger semantic representation capabilities than traditional image processing methods and have become mainstream. Although deep segmentation methods are more effective than traditional methods, they still have two key problems: slow speed and local receptive field. Because segmentation is pixel-by-pixel classification, very intensive calculations are required to segment lane lines, resulting in a relatively slow speed. Another problem with deep segmentation-based methods is the receptive field problem, because segmentation is generally obtained by full convolution, and convolution is basically local, so the receptive field of each pixel is limited. In lane detection, since most of the problems we are concerned about are the detection of semantic lines, we need to have a good perception of the global situation to achieve good positioning.

[0037] Based on this, this embodiment discloses a visual detection method for an emergency lane based on a drone perspective, including:

[0038] S100. Acquire lane images through a drone, and use the lane image stream as input to a feature extraction network. The present invention collects lane images from a drone's bird's-eye view, which is very different from the prior art in which lane images are collected from a driver's perspective.

[0039] S200. Adjust the image size according to the preset requirements and extract feature maps of different scales; specifically, in the preprocessing stage of the image input, we first change the input image size to a uniform aspect ratio of 324×780, in order to make the feature size used for classification consistent, and then use ResNet-34 as the backbone network for vital sign extraction to extract the features of the input image. This backbone network has a relatively small number of parameters to improve the detection speed and the feature extraction has a higher accuracy.

[0040] S300. Determine whether the current image recognition is a training process. When the current image recognition is a training process, receive feature maps of different scales, fuse the feature maps of different scales through a multi-scale feature aggregation module, and obtain feature values ​​of different layers;

[0041] Specifically, the simple ResNet-34 network has poor results when facing complex overpass road scenes. The reason is that the lane lines on the roads at different heights of the overpass have different scales when photographed by drones, and it is difficult to distinguish them when the network samples them uniformly. Therefore, we added a multi-scale feature aggregation module during the network training process. In the feature map extraction stage, among the several layers of results output by ResNet-34, the later ones have higher receptive fields and express more important semantics when detecting larger targets. The multi-scale feature aggregation module is connected to the outputs of the last three stages of the ResNet-34 network structure. The first two stages of the output are used to predict smaller-scale targets, that is, farther lane lines; the third stage is used to predict larger-scale targets, that is, closer lane lines.

[0042] The output results of the stages of predicting large and small targets are directly connected in the channel dimension to construct new multi-scale features, and then a new convolution operation is performed to perform complete instance segmentation training on this CNN branch. This method only uses auxiliary segmentation tasks in the training phase and removes them in the test phase. In this way, even if we add additional segmentation tasks, the running speed of our method will not be affected. In the forward process, this is the same as the network without auxiliary segmentation tasks. Before adding the multi-scale feature aggregation module, it will be difficult for training data to converge in complex road scenes, resulting in poor detection results. In addition, for images at different heights from the drone's perspective, the network's adaptability to multiple scales will also be problematic. After adding the module, multi-scale information can be used to generate more constraints on parameter updates, which can effectively improve the network's detection effect on multi-scale images and make convergence more stable during training.

[0043] S400. Add the feature values ​​of different layers, extract sampling points according to preset rules, and perform feature sampling on the image; specifically, in S400, the preset rule for feature sampling on the image is: represent the lane as a series of horizontal positions on a predefined row, and perform grid division, divide the horizontal position into multiple cells, extract sampling points at fixed intervals and send them to the fully connected layer.

[0044] Specifically, the feature map used for sampling is obtained by adding the output results of several stages of ResNet-34 to the corresponding element values. If lane detection is performed based on the idea of ​​instance segmentation, all pixels need to be classified. Therefore, all features need to be input into the fully connected layer for classification. The amount of calculation is too large to ensure real-time performance. Therefore, this paper designs a global feature row sampling method to reduce the number of pixels that need to be classified. Sampling In the usual image segmentation network, the number of classification tasks performed in the classification layer is the number of pixels in the original image, which leads to a slow network speed and is difficult to meet the needs of practical applications. To solve the above problems, we propose to define lane detection as a sampling row selection method based on global image features. In other words, our method uses global features to select the correct lane position on each predefined row. In our scheme, the lane is represented as a series of horizontal positions on a predefined row (i.e., row anchor), so that a fixed number of rows are selected and sent to the fully connected layer.

[0045] S500. Classify according to the sampling point features, and perform curve fitting according to the classification results; specifically, in S500 of this embodiment, the method for classifying according to the sampling point features includes: setting a classifier for each row, classifying the pixels corresponding to each position in the row, the number of classifiers is the same as the number of row anchors, and each classifier classifies the pixel points at the corresponding position in the original image according to a row of input features, and the number of classifications is one more than the maximum number of lanes when the background is included.

[0046] In some preferred embodiments, the method for classification based on sampling point features also includes: during the labeling process, the labeling category number of the left line of the emergency lane is set to an even number, and the labeling category number of the right line is set to an odd number, and the left and right sides of the point on the lane line are judged by the parity of the classification result number.

[0047] S600. Determine whether the current lane image is an emergency lane based on the fitting curve. Specifically, in S600 of this embodiment, the method for determining whether the current lane image is an emergency lane is as follows: after fitting the lane line, the category code of the current lane line can be obtained from the result of network classification, and its parity can indicate whether it is on the left or right side of the emergency lane; traverse all detected lane lines from left to right, and if the category code has an even-odd combination, it is determined that the part composed of them is the emergency lane, and each row between the two lines is scanned from top to bottom for marking, thereby outputting the final emergency lane detection result map.

[0048] This embodiment also discloses an emergency lane visual detection system based on the perspective of a drone, comprising: a lane image acquisition module, a feature map extraction module, a multi-scale feature aggregation module, a sampling point feature classification module, and an emergency lane determination module; wherein: the lane image acquisition module is used to acquire a lane image through a drone, and use the lane image stream as an input of a feature extraction network;

[0049] The feature map extraction module is used to adjust the image size according to the preset requirements and extract feature maps of different scales through the resnet-34 network structure;

[0050] The multi-scale feature aggregation module is used to receive feature maps of different scales, fuse them, and obtain feature values ​​of different layers;

[0051] The sampling point feature classification module is used to add the feature values ​​of different layers, extract sampling points according to preset rules, and perform feature sampling on the image; classify according to the sampling point features, and perform curve fitting according to the classification results;

[0052] The emergency lane determination module is used to determine whether the current lane image is an emergency lane based on the fitting curve.

[0053] The present embodiment discloses a visual detection method and system for emergency lanes based on the perspective of a drone, which obtains lane images through a drone and uses the image stream of the lane as the input of a feature extraction network; adjusts the image size according to preset requirements and extracts feature maps of different scales; determines whether the current image recognition is a training process, and when the current image recognition is a training process, receives feature maps of different scales, fuses feature maps of different scales through a multi-scale feature aggregation module to obtain feature values ​​of different layers; adds feature values ​​of different layers, extracts sampling points according to preset rules, and performs feature sampling on the image; classifies according to the features of the sampling points, and performs curve fitting according to the classification results; and determines whether the current lane image is an emergency lane according to the fitting curve. This solves the problem that the mainstream lane line detection systems in the prior art are all based on the driver's perspective, the CNN segmentation network has a large amount of calculation and a slow speed, and there are often complex problems such as lane line size changes, vehicle occlusion, and camera exposure in the perspective of drones.

[0054] It should be understood that the specific order or hierarchy of steps in the disclosed process is an example of an exemplary method. Based on design preferences, it should be understood that the specific order or hierarchy of steps in the process can be rearranged without departing from the scope of protection of the present disclosure. The attached method claims present the elements of the various steps in an exemplary order and are not intended to be limited to the specific order or hierarchy described.

[0055] In the above detailed description, various features are grouped together in a single embodiment to simplify the disclosure. This method of disclosure should not be interpreted as reflecting an intention that the embodiments of the claimed subject matter require more features than are clearly stated in each claim. On the contrary, as reflected in the appended claims, the invention is in a state of having less than all the features of the disclosed individual embodiments. Therefore, the appended claims are hereby expressly incorporated into the detailed description, with each claim standing on its own as a separate preferred embodiment of the invention.

[0056] Those skilled in the art will also appreciate that the various illustrative logic blocks, modules, circuits, and algorithmic steps described in conjunction with the embodiments herein can all be implemented as electronic hardware, computer software, or a combination thereof. In order to clearly illustrate the interchangeability between hardware and software, various illustrative components, blocks, modules, circuits, and steps are generally described above around their functions. Whether such functions are implemented as hardware or software depends on specific applications and the design constraints imposed on the entire system. A skilled person can implement the described functions in an alternative manner for each specific application, but such implementation decisions should not be interpreted as departing from the scope of protection of the present disclosure.

[0057] The steps of the method or algorithm described in conjunction with the embodiments herein may be directly embodied as hardware, a software module executed by a processor, or a combination thereof. The software module may be located in a RAM memory, a flash memory, a ROM memory, an EPROM memory, an EEPROM memory, a register, a hard disk, a mobile disk, a CD-ROM, or any other form of storage medium well known in the art. An exemplary storage medium is connected to the processor so that the processor can read information from the storage medium and can write information to the storage medium. Of course, the storage medium may also be an integral part of the processor. The processor and the storage medium may be located in an ASIC. The ASIC may be located in a user terminal. Of course, the processor and the storage medium may also be present in a user terminal as discrete components.

[0058] For software implementation, the techniques described in this application can be implemented with modules (e.g., procedures, functions, etc.) that perform the functions described in this application. These software codes can be stored in a memory unit and executed by a processor. The memory unit can be implemented within the processor or outside the processor. In the latter case, it is coupled to the processor in a communication manner via various means, which are well known in the art.

[0059] The above description includes examples of one or more embodiments. Of course, it is impossible to describe all possible combinations of components or methods for the purpose of describing the above embodiments, but it should be recognized by those skilled in the art that the various embodiments may be further combined and arranged. Therefore, the embodiments described herein are intended to cover all such changes, modifications and variations that fall within the scope of protection of the appended claims. In addition, with respect to the term "comprising" used in the specification or claims, the word is covered in a manner similar to the term "including", just as "including," is explained as a transitional word in the claims. In addition, any term "or" used in the specification of the claims is intended to mean "non-exclusive or".

Claims

1. A visual detection method for emergency lanes based on the perspective of an unmanned aerial vehicle, characterized in that: include: S100. Acquire a lane image through a drone, and use the lane image stream as an input of a feature extraction network; S200. Adjust the image size according to preset requirements and extract feature maps of different scales through the resnet-34 network structure; S300. Determine whether the current image recognition is a training process. When the current image recognition is a training process, receive feature maps of different scales, fuse the feature maps of different scales through a multi-scale feature aggregation module, and obtain feature values ​​of different layers; S400. Add the feature values ​​of different layers, extract sampling points according to preset rules, and perform feature sampling on the image; In S400, the preset rule for feature sampling of the image is: the lane is represented as a series of horizontal positions on a predefined row, and grid division is performed, the horizontal position is divided into a plurality of cells, sampling points are extracted at fixed intervals and sent to the fully connected layer; S500. Classify according to the characteristics of the sampling points, and perform curve fitting according to the classification results; S600. Determine whether the current lane image is an emergency lane based on the fitting curve; in S600, the method for determining whether the current lane image is an emergency lane is as follows: after fitting the lane line, the category code of the current lane line can be obtained from the result of network classification, and its parity indicates whether it is on the left or right side of the emergency lane; traverse all detected lane lines from left to right, and if the category code has an even-odd combination, it is determined that the part composed of them is the emergency lane, and each row between the two lines is scanned from top to bottom for marking, thereby outputting the final emergency lane detection result map.

2. The method for visually detecting an emergency lane based on a drone perspective as claimed in claim 1, characterized in that: In S200, the image size is adjusted according to the preset requirements, and the aspect ratio of the image is adjusted to 324×780.

3. The method for visually detecting an emergency lane based on a drone perspective as claimed in claim 2, characterized in that: In S300, the multi-scale feature aggregation module is connected to the outputs of the last three stages of the ResNet-34 network structure, where the first two stages of the output are used to predict smaller-scale targets, and the third stage is used to predict larger-scale targets.

4. The method for visually detecting an emergency lane based on a drone perspective as claimed in claim 1, characterized in that: In S500, the method for classifying according to the sampling point features includes: setting a classifier for each row, classifying the pixels corresponding to each position in the row, the number of classifiers is the same as the number of row anchors, and each classifier classifies the pixel points at the corresponding position in the original image according to the input row feature, and the number of classifications is one more than the maximum number of lanes when the background is included.

5. The method for visually detecting an emergency lane based on a drone perspective as claimed in claim 1, characterized in that: In S500, the method for classifying according to the characteristics of the sampling points also includes: during the labeling process, setting the labeling category number of the left line of the emergency lane to an even number, and setting the labeling category number of the right line to an odd number, and judging the left and right sides of the point on the lane line by the parity of the classification result number.

6. The method for visually detecting an emergency lane based on a drone perspective as claimed in claim 1, characterized in that: In S500, when the scene is relatively simple, the lane lines are directly fitted through Hough transform, the coordinates of the same type of detection results in the image are counted and fitted overall, and the straight line with the highest degree of fit that runs through these points is found.

7. The method for visually detecting an emergency lane based on a drone perspective as claimed in claim 1, characterized in that: In S500, when the scene is more complex, a preset number of detection results are subjected to Hough transform to fit a small straight line. Finally, all the fitted small straight lines are connected end to end and merged into a complete curve. The sampling points on the same lane line are sorted to ensure that all the sampling points on this lane line are traversed in a top-down or bottom-up order when the program traverses the results.

8. An emergency lane visual detection system based on drone perspective, characterized in that: include: Lane image acquisition module, feature map extraction module, multi-scale feature aggregation module, sampling point feature classification module, emergency lane determination module; wherein: the lane image acquisition module is used to acquire lane images through a drone and use the lane image stream as the input of the feature extraction network; The feature map extraction module is used to adjust the image size according to the preset requirements and extract feature maps of different scales through the resnet-34 network structure; The multi-scale feature aggregation module is used to receive feature maps of different scales, fuse them, and obtain feature values ​​of different layers; The sampling point feature classification module is used to add the feature values ​​of different layers, extract sampling points according to preset rules, and perform feature sampling on the image; classify according to the sampling point features, and perform curve fitting according to the classification results; the preset rules for feature sampling of the image are: represent the lane as a series of horizontal positions on a predefined row, and perform grid division, divide the horizontal position into multiple cells, extract sampling points at fixed intervals and send them to the fully connected layer; The emergency lane determination module is used to determine whether the current lane image is an emergency lane based on the fitting curve; in S600, the method for determining whether the current lane image is an emergency lane is as follows: after fitting the lane line, the category code of the current lane line can be obtained from the result of network classification, and its parity indicates whether it is on the left or right side of the emergency lane; all detected lane lines are traversed from left to right, and if the category code appears an even-odd combination, it is determined that the part composed of them is the emergency lane, and each row between the two lines is scanned from top to bottom for marking, thereby outputting the final emergency lane detection result map.

Images