A method, device and medium for positioning detection of a road edge

Abstract

The application discloses a road edge positioning detection method and device and a medium. The method comprises the following steps: collecting a road image, and preprocessing the road image; in the preprocessed road information, the road edge position is labeled in each row of pixels in a unit of a row; the labeled road image is used as a training sample for model training, in the model training, an anchor setting region is used as a feature extraction range for feature extraction, and the extracted shallow features are used as inputs of an ASPP module, so that a neural network model is trained; and the trained neural network model is used for road edge detection. Unlike the pixel-by-pixel detection segmentation method based on image segmentation, the collected image is divided and detected according to a region of interest, the input quantity is the road edge position of each row in the image, the deep network Darknet-53 is selected as a feature extraction network, and the gradient problem caused by the excessively deep network is avoided while the super strong expression of the features is ensured according to the characteristics of the ResNet.

Description

Technical Field

[0001] This application relates to the field of computers, specifically to a method, device, and medium for road edge positioning and detection. Background Technology

[0002] In the current construction of various smart scenarios, intelligent transportation systems have become an active research field due to their potential to improve system efficiency and decision-making.

[0003] Intelligent driving technology needs to be incorporated into the research field of outdoor security robots to enhance the safety performance of robot inspection and other behaviors. Among these, the implementation of road edge detection and tracking models that can assist driving can help robots drive on roads that comply with traffic rules outdoors, which is a fundamental aspect of safe driving for robots. The ability to accurately locate the position of the road edge and update it in real time is key to achieving active safety and assisted driving for vehicles.

[0004] Road edge detection and tracking algorithms are primarily used to accurately locate the current road edge in dynamic images and calculate the robot's safe driving area. This is the fundamental guarantee for intelligent robots to move autonomously on roads. Current road edge (commonly known as "curb") detection schemes can be mainly divided into four categories: feature-based, model-based, learning-based, and other traditional methods. Feature-based methods can be further subdivided into detection methods based on color, edge features, etc.; model-based methods mainly use straight lines, curves, and variable models; and learning-based methods involve deep neural networks. With the continuous improvement of computing power in edge devices, improving the real-time performance and accuracy of detection has become an increasingly important benchmark. Summary of the Invention

[0005] To address the aforementioned problems, this application proposes a road edge localization and detection method, comprising:

[0006] Acquire road images and preprocess the road images;

[0007] In the preprocessed road information, the road edge positions are marked in each row of pixels, on a row-by-row basis.

[0008] The labeled road images are used as training samples for model training. During model training, the anchor-defined region is used as the feature extraction range for feature extraction. The extracted shallow features are used as the input of the ASPP module to train the neural network model.

[0009] The trained neural network model is used for road edge detection.

[0010] On the other hand, this application also proposes a road edge positioning and detection device, comprising:

[0011] At least one processor; and,

[0012] A memory communicatively connected to the at least one processor; wherein,

[0013] The memory stores instructions executable by the at least one processor, which, when executed by the at least one processor, enable the at least one processor to perform actions such as:

[0014] Acquire road images and preprocess the road images;

[0015] In the preprocessed road information, the road edge positions are marked in each row of pixels, on a row-by-row basis.

[0016] The labeled road images are used as training samples for model training. During model training, the anchor-defined region is used as the feature extraction range for feature extraction. The extracted shallow features are used as the input of the ASPP module to train the neural network model.

[0017] The trained neural network model is used for road edge detection.

[0018] On the other hand, this application also proposes a non-volatile computer storage medium storing computer-executable instructions, wherein the computer-executable instructions are configured as follows:

[0019] Acquire road images and preprocess the road images;

[0020] In the preprocessed road information, the road edge positions are marked in each row of pixels, on a row-by-row basis.

[0021] The labeled road images are used as training samples for model training. During model training, the anchor-defined region is used as the feature extraction range for feature extraction. The extracted shallow features are used as the input of the ASPP module to train the neural network model.

[0022] The trained neural network model is used for road edge detection.

[0023] The method proposed in this application can bring the following beneficial effects:

[0024] This paper employs deep learning methods to design a convolutional neural network that is height-adaptive to the learning scene based on camera-acquired images for road edge detection and tracking. Unlike pixel-by-pixel detection and segmentation methods based on image segmentation, this method divides and detects regions of interest in the acquired image by row. The input is the road edge position of each row in the image. The feature extraction network can be a deep network Darknet-53, which combines the characteristics of ResNet to ensure strong feature representation while avoiding gradient problems caused by excessive network depth. Attached Figure Description

[0025] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0026] Figure 1 This is a flowchart illustrating the road edge positioning and detection method in the embodiments of this application;

[0027] Figure 2 This is a schematic diagram of the actual captured images and detection in the embodiments of this application;

[0028] Figure 3 This is a schematic diagram of the road edge positioning and detection device in the embodiments of this application. Detailed Implementation

[0029] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below in conjunction with specific embodiments and corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0030] The technical solutions provided by the various embodiments of this application are described in detail below with reference to the accompanying drawings.

[0031] like Figure 1 As shown in the figure, this application provides a road edge positioning and detection method, including:

[0032] S101: Acquire road images and preprocess the road images.

[0033] Road images can be acquired using cameras, camera robots, etc. Preprocessing may include denoising and enhancing the road images, and equalizing illumination. When annotating static images, overfitting should be prevented. Image enhancement can also be performed on the road images, including rotation enhancement and translation enhancement, and lane line extension and completion annotations should still be performed in blank areas after movement or rotation.

[0034] S102: In the preprocessed road information, the road edge positions are marked in each row of pixels, on a row-by-row basis.

[0035] The method of segmenting and detecting regions of interest (ROIs) in the acquired image by row significantly reduces network training time and enables real-time detection on embedded devices at the edge. The input is the road edge position of each row in the image, using a representation method that trades speed for some accuracy.

[0036] S103: Use the labeled road images as training samples for model training. During model training, use the anchor set region as the feature extraction range for feature extraction. Use the extracted shallow features as the input of the ASPP module to train the neural network model.

[0037] Using Darknet53 as the feature extraction network, the feature map of the last 5x part of the feature extraction network is then used as the final feature input to the auxiliary segmentation part, serving as the input to the ASPP module. This achieves road edge detection based on a larger receptive field of the image, adaptable to various complex scenarios such as highways and industrial parks. It can adaptively select and capture ROI regions based on the size of the input image. This method of road edge detection effectively meets the real-time requirements of intelligent driving assistance scenarios for security robots and greatly improves their adaptability to different scenarios, which is of significant practical importance for ensuring the safety of security robots.

[0038] Specifically, the shallow features of the extracted five-layer feature map are used as the input of the ASPP module. According to the ASPP module, the receptive field of the convolution kernel is increased by using dilated convolution, so that the receptive field is increased without pooling loss information, and each convolution output contains a larger range of information.

[0039] In addition, during the training process of the neural network model, a batch normalization (BN) layer is set after the convolutional layer to reduce overfitting, increase the network convergence speed, and thus accelerate the training speed.

[0040] Furthermore, through the ASPP module, the feature map is processed in a parallel connection manner. Atrous convolutional layers with different dilation rates are used to capture multi-scale information, and the output results are fused. The segmentation result of the image is obtained by using concat and upsampling. During the training process, through the auxiliary segmentation module, horizontal lines are drawn at fixed pixels in the input road image to divide the road image into several rows, and each row is cut into several cell blocks in the vertical direction to enhance visual features.

[0041] In a conventional segmentation task, the size of the final feature map is H*W*C. Classification is along the channel direction, and the channel direction vector represents which category the feature vector at a cell position belongs to. In this embodiment, the size of the final feature map is h*(w + 1)*C. h is the number of rows (row anchor) to be sampled in the vertical direction, h < H, because other background regions in the road area are not used as ROI regions for detection; w is the number of positions (gridcell) of road edge candidate points in the row direction, w < W, because multiple pixels are combined to divide cells. C is the set number of road edges. Classification is along the w direction, that is, for each road edge region, at its preset vertical direction h, calculate the probability of it appearing in each grid cell at the horizontal position.

[0042] During the training process of the neural network model, the total loss function is constructed by building a multi-classification loss function, a segmentation loss function, and a structural loss function for road edges. Among them, the multi-classification loss function uses the cross-entropy loss function, the segmentation loss uses the Dice loss function, and the structural loss function for road edges adds a regularization term by combining the prior of road edges to constrain the predicted road edge shape.

[0043] Structural loss function L stru = L simp + λL shap where L simp is the similarity loss between the label and the prediction result, and the L2 norm is used in two adjacent cells to perform loss constraint in similarity calculation. Since most regions of the road edge are straight lines, L shap uses the second-order difference loss to constrain the road edge shape.

[0044] The number of road edges is labeled during the annotation of the image training set and test set, and is defined autonomously during training. When road edges exist, through the formula Loc i,j = argmaxP i,j,kThe location of the cell containing the road edge is obtained. Here, the Loc function refers to the expectation of the lane point in the j-th row anchor of the i-th lane, and the P function (Prob) refers to the probability that the k-th position in the j-th row anchor of the i-th lane is a lane point. Due to the existence of the road edge, the value of k is in the range of k∈[1,w].

[0045] The ROI region is selected as the area where the road edge is located in the image. It is trained using a static image of a certain resolution as input. The ROI region is calculated and extracted according to different camera shooting height and resolution, covering an area of ​​about 100 pixels from the road edge. The selection of rows in the image is preferably 10-15 pixels, rather than fixing the ROI region completely.

[0046] S104: Use the trained neural network model to perform road edge detection.

[0047] This paper employs deep learning methods to design a convolutional neural network that is height-adaptive to the learning scene based on camera-acquired images for road edge detection and tracking. Unlike pixel-by-pixel detection and segmentation methods based on image segmentation, this method divides and detects regions of interest in the acquired image by row. The input is the road edge position of each row in the image. The feature extraction network can be a deep network Darknet-53, which combines the characteristics of ResNet to ensure strong feature representation while avoiding gradient problems caused by excessive network depth.

[0048] To evaluate performance, experiments were conducted on the publicly available real-world highway traffic dataset, Tusimple. The experimental accuracy reached 95.8%, with an average runtime of 42 frames per second on edge embedded devices. Figure 2 As shown, data was collected and a curl dataset was created in a closed park scenario where security robots are used. The video was processed by capturing 3 frames per second, and the images were captured by the front-facing camera of the security robot car with a resolution of 1280*720.

[0049] like Figure 3 As shown in the illustration, this application also provides a road edge positioning and detection device, comprising:

[0050] At least one processor; and,

[0051] A memory communicatively connected to the at least one processor; wherein,

[0052] The memory stores instructions executable by the at least one processor, which, when executed by the at least one processor, enable the at least one processor to perform actions such as:

[0053] Acquire road images and preprocess the road images;

[0054] In the preprocessed road information, the road edge positions are marked in each row of pixels, on a row-by-row basis.

[0055] The labeled road images are used as training samples for model training. During model training, the anchor-defined region is used as the feature extraction range for feature extraction. The extracted shallow features are used as the input of the ASPP module to train the neural network model.

[0056] The trained neural network model is used for road edge detection.

[0057] This application embodiment also provides a non-volatile computer storage medium storing computer-executable instructions, wherein the computer-executable instructions are configured as follows:

[0058] Acquire road images and preprocess the road images;

[0059] In the preprocessed road information, the road edge positions are marked in each row of pixels, on a row-by-row basis.

[0060] The labeled road images are used as training samples for model training. During model training, the anchor-defined region is used as the feature extraction range for feature extraction. The extracted shallow features are used as the input of the ASPP module to train the neural network model.

[0061] The trained neural network model is used for road edge detection.

[0062] The various embodiments in this application are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the device and medium embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the description of the method embodiments.

[0063] The devices and media provided in this application are one-to-one with the methods. Therefore, the devices and media also have similar beneficial technical effects as their corresponding methods. Since the beneficial technical effects of the methods have been described in detail above, the beneficial technical effects of the devices and media will not be repeated here.

[0064] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0065] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0066] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0067] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0068] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0069] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0070] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0071] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0072] The above description is merely an embodiment of this application and is not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.

Claims

1. A method for locating and detecting road edges, characterized in that, include: Acquire road images and preprocess the road images; In the preprocessed road image, the road edge positions are marked in each row of pixels, on a row-by-row basis. The labeled road images are used as training samples for model training. During model training, the anchor-defined region is used as the feature extraction range for feature extraction. The extracted shallow features are used as the input of the ASPP module to train the neural network model. The trained neural network model is used for road edge detection; wherein the input of the neural network model is a road image, and the output is the position of the cell where the road edge is located. Using Darknet53 as the feature extraction network, we obtained the feature maps of the last five layers of the feature extraction network. The extracted shallow features are used as input to the ASPP module to train a neural network model, which specifically includes: The shallow features of the extracted five-layer feature maps are used as input to the ASPP module; According to the ASPP module, by using dilated convolution to increase the receptive field of the convolution kernel, the receptive field can be increased without pooling and losing information. The method further includes: During the training process of the neural network model, a BN layer is set after the convolutional layer to reduce overfitting, increase the network convergence speed, and thus accelerate the training speed. According to the ASPP module, the receptive field of the convolution kernel is increased by using dilated convolution, specifically including: The ASPP module uses parallel connections of dilated convolutional layers with different dilation rates to capture multi-scale information and then fuses the output results. The image segmentation results are obtained by using concat and upsampling. During the training process, an auxiliary segmentation module is used to draw horizontal lines at fixed intervals in the input road image to divide the road image into several rows, and each row is cut into several cell blocks in the vertical direction.

2. The method according to claim 1, characterized in that, The road image is preprocessed, specifically including: The road image is denoised and enhanced, and its illumination is equalized. Image enhancement is then performed based on the road image, including rotation enhancement and translation enhancement.

3. The method according to claim 1, characterized in that, The method further includes: The size of the feature map is determined to be h*(w+1)*C, where h refers to the number of rows sampled in the vertical direction, w refers to the number of candidate points of road edge in the row direction, and C is the set number of road edges.

4. The method according to claim 1, characterized in that, The method further includes: During the training process of the neural network model, the total loss function is constructed by building a multi-classification loss function, a segmentation loss function, and a road edge structural loss function; The multi-class loss function adopts the cross-entropy loss function, the segmentation loss adopts the Dice loss function, and the road edge structure loss function constrains the predicted road edge shape by adding a regularization term in combination with the prior of the road edge. The structural loss function ,in, It is the similarity loss between the label and the prediction result, and the L2 paradigm is used to constrain the loss calculation in adjacent cells. The road edge shape is constrained by a second-order difference loss.

5. The method according to claim 4, characterized in that, The number of road edges is labeled during image training and testing, and is also defined autonomously during training. Furthermore, when road edges exist, they are determined using a formula. The location of the cell containing the road edge is obtained; where cell refers to the image block obtained by dividing the road image through the auxiliary segmentation module; Here, the Loc function refers to the expected value of the lane point in the j-th row anchor of the i-th lane, and the P function refers to the probability that the k-th position in the j-th row anchor of the i-th lane is a lane point, where k takes values ​​ranging from 0 to 1. .

6. A road edge positioning and detection device, characterized in that, include: At least one processor; as well as, A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform, for example: Acquire road images and preprocess the road images; In the preprocessed road image, the road edge positions are marked in each row of pixels, on a row-by-row basis. The labeled road images are used as training samples for model training. During model training, the anchor-defined region is used as the feature extraction range for feature extraction. The extracted shallow features are used as the input of the ASPP module to train the neural network model. The trained neural network model is used for road edge detection; wherein the input of the neural network model is a road image, and the output is the position of the cell where the road edge is located. Using Darknet53 as the feature extraction network, we obtained the feature maps of the last five layers of the feature extraction network. The extracted shallow features are used as input to the ASPP module to train a neural network model, which specifically includes: The shallow features of the extracted five-layer feature maps are used as input to the ASPP module; According to the ASPP module, by using dilated convolution to increase the receptive field of the convolution kernel, the receptive field can be increased without pooling and losing information. Also includes: During the training process of the neural network model, a BN layer is set after the convolutional layer to reduce overfitting, increase the network convergence speed, and thus accelerate the training speed. According to the ASPP module, the receptive field of the convolution kernel is increased by using dilated convolution, specifically including: The ASPP module uses parallel connections of dilated convolutional layers with different dilation rates to capture multi-scale information and then fuses the output results. The image segmentation results are obtained by using concat and upsampling. During the training process, an auxiliary segmentation module is used to draw horizontal lines at fixed intervals in the input road image to divide the road image into several rows, and each row is cut into several cell blocks in the vertical direction.

7. A non-volatile computer storage medium storing computer-executable instructions, characterized in that, The computer-executable instructions are set as follows: Acquire road images and preprocess the road images; In the preprocessed road image, the road edge positions are marked in each row of pixels, on a row-by-row basis. The labeled road images are used as training samples for model training. During model training, the anchor-defined region is used as the feature extraction range for feature extraction. The extracted shallow features are used as the input of the ASPP module to train the neural network model. The trained neural network model is used for road edge detection; wherein the input of the neural network model is a road image, and the output is the position of the cell where the road edge is located. The extracted shallow features are used as input to the ASPP module to train a neural network model, which specifically includes: The shallow features of the extracted five-layer feature maps are used as input to the ASPP module; According to the ASPP module, by using dilated convolution to increase the receptive field of the convolution kernel, the receptive field can be increased without pooling and losing information. Also includes: During the training process of the neural network model, a BN layer is set after the convolutional layer to reduce overfitting, increase the network convergence speed, and thus accelerate the training speed. According to the ASPP module, the receptive field of the convolution kernel is increased by using dilated convolution, specifically including: The ASPP module uses parallel connections of dilated convolutional layers with different dilation rates to capture multi-scale information and then fuses the output results. The image segmentation results are obtained by using concat and upsampling. During the training process, an auxiliary segmentation module is used to draw horizontal lines at fixed intervals in the input road image to divide the road image into several rows, and each row is cut into several cell blocks in the vertical direction.

Images