Method and device for determining drivable area, storage medium, terminal and computer program product

Abstract

A method and device for determining a drivable area, a storage medium, a terminal, and a computer program product, the method comprising: performing drivable area segmentation on an input image to obtain a plurality of first boundary points; performing drivable area segmentation on a target image area to obtain a plurality of second boundary points, the target image area being a part of the input image; matching the plurality of first boundary points with the plurality of second boundary points to obtain successfully matched boundary point pairs; and replacing the successfully matched first boundary points in the plurality of first boundary points with matched second boundary points, and adopting an area formed by the replaced first boundary points and at least one part of the unsuccessfully matched first boundary points as a target drivable area. According to the scheme, the distance boundary error of the drivable area obtained based on image segmentation can be reduced, and the problem of boundary blur of the drivable area is improved.

Description

Technical Field

[0001] The present invention relates to the field of autonomous driving technology, and in particular to a method and device for determining a drivable area, a storage medium, a terminal, and a computer program product. Background Art

[0002] With the rapid development of autonomous driving, how to conduct safe and reliable autonomous driving has become increasingly important. The first aspect to achieve safety is to perceive the surrounding environment, so accurate detection of drivable areas is very important. The detection of drivable areas can not only be used to determine which areas are drivable and which areas are not drivable, but also for subsequent path planning and decision-making, and to assist in the intelligence and safety of subsequent driving control.

[0003] Due to the diversity of shapes of drivable areas, the existing technology usually adopts the method of segmenting the input image to determine the drivable area. However, the drivable area at the 2D pixel level is prone to blurring and other problems at the boundary of the drivable area, and the farther the boundary point is from the camera, the greater the position error caused by the pixel-level error.

[0004] Therefore, there is an urgent need to provide a method for determining a drivable area that can effectively reduce the distant boundary error of the drivable area obtained based on image segmentation. Summary of the invention

[0005] The technical problem solved by the embodiment of the present invention is how to reduce the error of the distant boundary of the drivable area obtained based on image segmentation and improve the fuzzy boundary problem of the drivable area.

[0006] To solve the above technical problems, an embodiment of the present invention provides a method for determining a drivable area, comprising the following steps: performing drivable area segmentation on an input image to obtain a plurality of first boundary points; performing drivable area segmentation on a target image area to obtain a plurality of second boundary points, wherein the target image area is a part of the input image; matching the plurality of first boundary points with the plurality of second boundary points to obtain successfully matched boundary point pairs; replacing a successfully matched first boundary point among the plurality of first boundary points with a matched second boundary point, and using an area formed by the replaced first boundary points and at least a portion of the unsuccessfully matched first boundary points as the target drivable area.

[0007] Optionally, the target image area is a preset area around a center point of the input image, or a preset area around a vanishing point of a drivable area in the input image.

[0008] Optionally, matching is performed based on the multiple first boundary points and the multiple second boundary points to obtain successfully matched boundary point pairs, including: filtering the multiple first boundary points and the multiple second boundary points respectively to obtain filtered first boundary points and filtered second boundary points; pairing the filtered first boundary points and the filtered second boundary points in pairs to obtain multiple pairs of candidate boundary points; using the Euclidean distance between the multiple pairs of candidate boundary points to generate a cost matrix, wherein multiple elements of the cost matrix correspond one-to-one to the multiple pairs of candidate boundary points, and the element value of each element is the Euclidean distance of the corresponding candidate boundary point pair; based on the cost matrix, determining the successfully matched boundary point pairs among the multiple pairs of candidate boundary points.

[0009] Optionally, one or more of the following is satisfied: the first boundary point after screening is each first boundary point among the multiple first boundary points located in the target image area; the second boundary point after screening is the boundary point remaining after removing the target boundary point from the multiple second boundary points, and the target boundary point is a point on the first edge of the drivable area formed by the multiple second boundary points.

[0010] Optionally, the cost matrix is ​​a two-dimensional matrix containing M rows of elements, where M is a positive integer; based on the cost matrix, determining the successfully matched boundary point pairs among the multiple pairs of candidate boundary points, including: for each row of the cost matrix, subtracting the minimum element of the row from each element of the row, and for each column of the cost matrix, subtracting the minimum element of the column from each element of the column, thereby obtaining a preliminary update matrix; performing one or more rounds of iteration operations based on the preliminary update matrix to obtain a target cost matrix, wherein in each round of iteration, a minimum number of horizontal lines and vertical lines are used to cover all elements whose element values ​​in the update matrix of the current round are the first value, and if the horizontal lines and vertical lines are If the sum of the number of straight lines is less than M, the smallest first element is subtracted from each first element in the current round of update matrix that is not covered by the horizontal line or the vertical line, and the smallest first element is added to each second element covered by both the horizontal line and the vertical line to obtain the update matrix for the next round, and the next round of iteration is continued until the sum of the number of horizontal lines and vertical lines is equal to M, the iteration is stopped, and the update matrix of the last round is used as the target cost matrix, wherein the update matrix of the first round is the preliminary update matrix; the candidate boundary point pairs corresponding to each element whose element value in the target cost matrix is ​​the first value are used as the successfully matched boundary point pairs.

[0011] Optionally, the operation of performing drivable area segmentation on the input image includes a boundary feature acquisition operation and a post-processing operation; wherein the boundary feature acquisition operation determines the boundary features of the drivable area based on the input image; the post-processing operation includes: performing pixel value conversion on the boundary features of the drivable area to obtain a corresponding converted image; performing contour extraction based on the converted image to obtain multiple candidate connected areas; taking each boundary point of the candidate connected area with the largest area among the multiple candidate connected areas as a preliminary boundary point; and performing linear transformation on the coordinates of each preliminary boundary point to obtain the multiple first boundary points.

[0012] Optionally, contour extraction is performed based on the conversion image to obtain multiple candidate connected areas, including: binarizing the conversion image to obtain a grayscale image; performing an opening operation on the grayscale image to obtain a denoised image; and performing contour extraction on the denoised image to obtain the multiple candidate connected areas.

[0013] Optionally, the boundary feature acquisition operation includes: performing multiple feature extraction operations of different scales on the input image to obtain multiple image features, wherein the input data of the first feature extraction operation is the input image, and from the second feature extraction operation onwards, the input data of the current feature extraction operation is the output data of the previous feature extraction operation; performing multiple feature fusion operations of different scales based on the multiple image features to obtain fused features, wherein the multiple feature extraction operations correspond to the multiple feature fusion operations one-to-one, the input data of the first feature fusion operation is the image feature output by the last feature extraction operation, and from the second feature fusion operation onwards, the input data of the current feature fusion operation includes the output data of the corresponding feature extraction operation and the output data of the previous feature fusion operation; selecting a target image feature from the multiple image features, and performing a feature splicing operation on the target image feature and the fused feature to obtain a spliced ​​feature, and then decoding the spliced ​​feature to obtain the drivable area boundary feature.

[0014] Optionally, the operation of performing drivable area segmentation on the input image is implemented using a pre-trained model, which includes a feature extraction network, a feature fusion network and a feature prediction network; the feature extraction network performs multiple feature extraction operations of different scales on the input image to obtain the multiple image features; the feature fusion network performs multiple feature fusion operations of different scales based on the multiple image features to obtain the fused features; the feature prediction network performs a feature splicing operation on the target image features in the multiple image features and the fused features to obtain spliced ​​features, and then decodes the spliced ​​features to obtain the drivable area boundary features.

[0015] Optionally, the pre-trained model is trained in the following manner: determining an initial model, the initial model including a feature extraction network to be trained, a feature fusion network to be trained, a feature prediction network to be trained, and an auxiliary network to be trained, the auxiliary network determining auxiliary boundary features based on output data of multiple feature fusion operations in the feature fusion network; based on a preset first target loss function, inputting a sample image into the initial model for initial training to obtain an optimized feature extraction network, an optimized feature fusion network, an optimized feature prediction network, and an optimized auxiliary network; fixing the parameters of the optimized feature extraction network and the optimized feature fusion network unchanged, based on a preset second target loss function, inputting a sample image into the initial model for retraining to obtain a fine-tuned feature prediction network and a fine-tuned auxiliary network; adopting the optimized feature extraction network, the optimized feature fusion network, and the fine-tuned feature prediction network to construct the pre-trained model.

[0016] Optionally, the first objective loss function and the second objective loss function are obtained by weighted operation based on at least the first sub-loss function and the second sub-loss function; during the training process, the function value of the first sub-loss function is determined according to the difference between multiple predicted boundary points and multiple labeled boundary points, and the function value of the second sub-loss function is determined according to the difference between multiple auxiliary predicted boundary points and multiple labeled boundary points; wherein, the multiple predicted boundary points are obtained by post-processing the sample boundary features output by the feature prediction network, the multiple auxiliary predicted boundary points are obtained by post-processing the auxiliary boundary features output by the auxiliary network, and the multiple labeled boundary points are the boundary points of the drivable area annotated by the sample image.

[0017] Optionally, the weight of the first sub-loss function is greater than the weight of the second sub-loss function.

[0018] An embodiment of the present invention also provides a drivable area determination device, comprising: a first segmentation module, used to perform drivable area segmentation on an input image to obtain multiple first boundary points; a second segmentation module, used to perform drivable area segmentation on a target image area to obtain multiple second boundary points, wherein the target image area is a part of the input image; a matching module, used to match the multiple first boundary points with the multiple second boundary points to obtain successfully matched boundary point pairs; a target drivable area determination module, used to replace successfully matched first boundary points among the multiple first boundary points with matched second boundary points, and use the area formed by the replaced first boundary points and at least a part of the unsuccessfully matched first boundary points as the target drivable area.

[0019] An embodiment of the present invention further provides a storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps of the above-mentioned method for determining a drivable area are executed.

[0020] An embodiment of the present invention further provides a terminal, including a memory and a processor, wherein the memory stores a computer program that can be run on the processor, and the processor executes the steps of the above-mentioned drivable area determination method when running the computer program.

[0021] An embodiment of the present invention further provides a computer program product, including a computer program, which executes the steps of the above-mentioned drivable area determination method when executed by a processor.

[0022] Compared with the prior art, the technical solution of the embodiment of the present invention has the following beneficial effects:

[0023] In the embodiment of the present invention, since the target image area is a part of the input image, the original size of the former is smaller. When the original sizes of the two are scaled to meet the same input size, the size of the boundary of the drivable area in the target image area, especially the boundary of the drivable area far from the camera in the image, is larger than the size of the boundary of the drivable area in the input image. Accordingly, after performing the drivable area segmentation operation, the positions of the plurality of second boundary points obtained are often more accurate than those of the plurality of first boundary points. Therefore, by matching the plurality of first boundary points with the plurality of second boundary points, the second boundary points that best match each first boundary point are found; then the matched first boundary points are replaced with the successfully matched second boundary points; and finally, the target drivable area is obtained based on the replaced first boundary points and at least a part of the unsuccessfully matched first boundary points. Compared with using the area formed by the plurality of first boundary points as the target drivable area, this embodiment can effectively improve the clarity and accuracy of the obtained target drivable area, and can especially improve the blur problem at the boundary of the drivable area far from the camera.

[0024] Furthermore, in the conventional prior art, after obtaining the boundary features of the drivable area of ​​the input image, an upsampling scheme based on a complex network structure is usually used to determine the multiple first boundary points. For example, a deconvolution operation is performed through a deconvolution network to restore the size of the boundary features of the drivable area to the size of the input image and give the classification results of the corresponding pixels to obtain the multiple first boundary points. However, when the network is inferred in hardware, the inference process of the operator corresponding to the upsampling operation is relatively time-consuming and difficult to meet the real-time requirements of autonomous driving.

[0025] Furthermore, after determining the boundary features of the drivable area based on the input image, the embodiment of the present invention adopts a post-processing scheme based on image contour extraction and coordinate linear transformation to obtain the multiple first boundary points. Specifically: first convert the boundary features of the drivable area into image data, perform contour extraction and screening at the image level, and determine multiple preliminary boundary points that form the drivable area; then perform coordinate linear transformation to obtain the multiple first boundary points. Therefore, compared with the prior art scheme of obtaining the multiple first boundary points based on upsampling of a complex network, this implementation scheme can significantly reduce the time consumption of post-processing operations, improve the efficiency of drivable area segmentation, meet the real-time requirements of autonomous driving, and this implementation scheme has relatively lower requirements for computing resources.

[0026] Furthermore, since the linear transformation of the size of the boundary feature of the drivable area to the size of the input image usually causes jagged problems on the boundary of the drivable area, and since noise may also exist in the converted image obtained by pixel value conversion, if the contour is directly extracted from the converted image, it often causes the appearance of abnormal values, thereby causing multiple closed polygons, etc. Therefore, in an embodiment of the present invention, after the pixel value conversion is performed to obtain the converted image, binarization processing, opening operation processing, etc. are sequentially performed to effectively remove noise points and edge abnormal values, increase smoothness, and reduce the contour jagged problems caused by the subsequent linear transformation of coordinate points, thereby obtaining more accurate and effective boundary points of the drivable area.

[0027] Furthermore, in the existing conventional technology, a complex auxiliary network is used as a feature prediction network to obtain the boundary features of the drivable area of ​​the input image. This solution consumes more resources during the model deployment and reasoning process. In comparison, the embodiment of the present invention does not include a complex auxiliary network in the reasoning stage for the pre-trained model for determining the boundary features of the drivable area, but only uses an auxiliary network to assist model training and parameter optimization in the model training stage. In the model reasoning stage, this implementation scheme uses a more lightweight and streamlined feature prediction network to predict the boundary features of the drivable area. In this way, the computing resources and time overhead of model deployment and reasoning can be effectively reduced on the basis of obtaining sufficiently optimized network parameters. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 is a flow chart of a method for determining a drivable area in an embodiment of the present invention;

[0029] Figure 2 yes Figure 1 A partial flow chart of a specific implementation of step S11;

[0030] Figure 3It is a comparison diagram of the effect of directly extracting the contour of the converted image and extracting the contour of the image after binarization and opening operation on the converted image;

[0031] Figure 4 yes Figure 1 A partial flow chart of another specific implementation of step S11;

[0032] Figure 5 It is a partial structural schematic diagram of an initial model used for training to obtain a pre-trained model in an embodiment of the present invention;

[0033] Figure 6 Yes Figure 5 A flowchart of a training method for obtaining a pre-trained model by training the initial model shown;

[0034] Figure 7 yes Figure 1 A flowchart of a specific implementation of step S13;

[0035] Figure 8 This is a comparison diagram of the effect of performing drivable area segmentation on the input image and the target drivable area obtained by using the solution of the embodiment of the present invention;

[0036] Fig. 9 It is a structural schematic diagram of a drivable area determination device in an embodiment of the present invention. DETAILED DESCRIPTION

[0037] In order to make the above-mentioned objects, features and beneficial effects of the present invention more obvious and understandable, specific embodiments of the present invention are described in detail below with reference to the accompanying drawings.

[0038] Reference Figure 1 , Figure 1 This is a flow chart of a method for determining a drivable area in an embodiment of the present invention. The method can be applied to a terminal device with an image processing function, such as, but not limited to, a computer, a mobile phone, a tablet computer, a smart wearable device (such as a smart watch, a smart head-mounted device, a laptop device), an autonomous driving vehicle-mounted terminal, a server, a cloud platform, and a distributed computing environment including any of the above terminals or systems.

[0039] The method for determining a drivable area may include steps S11 to S14:

[0040] Step S11: performing drivable area segmentation on the input image to obtain a plurality of first boundary points;

[0041] Step S12: performing drivable area segmentation on the target image area to obtain a plurality of second boundary points, wherein the target image area is a part of the input image;

[0042] Step S13: matching the plurality of first boundary points with the plurality of second boundary points to obtain successfully matched boundary point pairs;

[0043] Step S14: replacing the successfully matched first boundary points among the plurality of first boundary points with matched second boundary points, and using an area formed by the replaced first boundary points and at least a portion of the unsuccessfully matched first boundary points as a target drivable area.

[0044] In the specific implementation of step S11, the input image may be a scene image captured during vehicle driving, for example, a scene image within a preset area in front of an autonomous driving vehicle. The input image may include a drivable area (for example, a road area). It is understandable that in most autonomous driving scenarios, the extension direction of the drivable area is close to the orientation of the camera lens (also referred to as the shooting direction). In other words, the angle between the extension direction of the drivable area and the orientation of the camera lens is usually in the angle range [0°, 90°).

[0045] In a specific implementation of step S12, the target image area is a part of the input image.

[0046] The multiple first boundary points may specifically refer to multiple sampling boundary points that form the outline (or boundary line) of the drivable area (e.g., road area) in the input image. The multiple second boundary points may specifically refer to multiple sampling boundary points that form the outline of the drivable area in the target image area.

[0047] Specifically, the target image area can be determined in at least the following manner:

[0048] Method 1: A preset area around the center point of the input image (for example, a rectangular area or other regular / irregular area centered on the center point of the input image) is used as the target image area.

[0049] Method 2: taking a preset area around the vanishing point of the drivable area in the input image (for example, a rectangular area or other regular / irregular area centered on the vanishing point of the drivable area in the input image) as the target image area.

[0050] Among them, the method for determining the vanishing point of the drivable area in the input image is as follows: first, the drivable area in the input image is determined by using methods such as segmentation or target recognition, and then one of the pixel points within the drivable area (including the boundary of the drivable area) whose depth value is greater than a preset depth value threshold can be selected as the vanishing point, or the pixel position corresponding to the average of the coordinates of the pixel points whose depth value is greater than the preset depth value threshold can be used as the vanishing point, or other appropriate methods can be used to determine the vanishing point.

[0051] Mode 3: Perform average region division on the input image, and then select one or more appropriate regions as the target image region. For example, the input image is evenly divided into 6 rectangular or square sub-regions, and then the sub-region located in the middle of the upper edge of the input image and / or the sub-region located in the middle of the lower edge of the input image is selected as the target image region.

[0052] In a specific implementation, the execution order of the steps S11 and S12 can be interchanged, or both can be executed simultaneously. When the drivable area segmentation is performed on the input image and the target image area, the input sizes of the two are consistent, and the drivable area segmentation methods used can also be completely consistent.

[0053] More specifically, the operation of performing drivable area segmentation on the input image and the target image area may include a feature extraction operation (for example, implemented by a feature extraction network), and the multiple first boundary points may be determined based on feature data obtained by feature extraction of the input image, and the multiple second boundary points may be determined based on feature data obtained by feature extraction of the target image area; wherein the input size and feature extraction scale (also referred to as "downsampling rate") for feature extraction of the input image are consistent with the input size and feature extraction scale for feature extraction of the target image area.

[0054] In a specific implementation, before performing the feature extraction operation, the input image and the target image region may be scaled to an input size suitable for the feature extraction network. For example, the original size of the input image is 2560×1440, the original size of the target image region is 640×360, and the input size of the feature extraction network is 1280×720; therefore, it is necessary to reduce the size of the input image (e.g., downsample) to obtain the input size of the feature extraction network, and to enlarge the size of the target image region (e.g., upsample) to obtain the input size of the feature extraction network.

[0055] In a specific implementation, the input image used to perform the feature extraction operation can be an image obtained by removing invalid edge regions (e.g., the sky region at the upper edge of the image, the front region of the vehicle at the lower edge, etc.) from the original image. To further improve segmentation efficiency and reduce overhead, after scaling to an input size suitable for the feature extraction network, the input image and the target image region can be subjected to operations such as subtracting the mean, dividing by the variance, and converting to a preset format.

[0056] It is understandable that small targets occupy a small area in the input image, so their feature information is often difficult to fully extract. Due to the limited size of small targets, after feature extraction (usually downsampling using a convolutional neural network), the features of the small targets may be completely lost (for example, when the size of the small target is 15×15 pixels and the scale of feature extraction is 1 / 16, the small target can hardly be represented in the feature map obtained by extraction). For the segmented drivable area boundaries, especially the drivable area boundaries that are far away from the camera in the image, because they are at the pixel level and their size is smaller, feature extraction may lead to missed or false detection of the boundaries.

[0057] Furthermore, for the target image area, when performing feature extraction, in the case of being consistent with the input size of the input image, the area occupied by the boundary of the drivable area in the target image area, especially the boundary of the drivable area farther from the camera in the image, is larger than the area occupied by the boundary of the drivable area in the input image, and after performing feature extraction of the same scale, the former can obtain relatively richer boundary feature information. That is, the positions of the multiple second boundary points are more accurate than those of the multiple first boundary points.

[0058] Reference Figure 2 , Figure 2 yes Figure 1 A partial flow chart of a specific implementation of step S11 in the embodiment; in this implementation, the operation of performing drivable area segmentation on the input image in step S11 may specifically include a boundary feature acquisition operation and a post-processing operation, wherein the boundary feature acquisition operation determines the boundary features of the drivable area based on the input image; the post-processing operation may include the following steps S21 to S24.

[0059] In step S21, pixel value conversion is performed on the boundary features of the drivable area to obtain a corresponding conversion image.

[0060] For example, the drivable area boundary feature may be a feature matrix of n×c×h×w, and the conversion image may be a matrix of n×3×h×w, wherein c represents the number of channels or dimensions of the drivable area boundary feature, 3 represents the number of channels of pixels of the conversion image, h represents the number of rows of the matrix (corresponding to the height of the conversion image), w represents the number of columns of the matrix (corresponding to the width of the conversion image), and n represents the batch size, that is, the number of images processed in the same batch.

[0061] In step S22, contour extraction is performed based on the converted image to obtain a plurality of candidate connected regions.

[0062] Furthermore, contour extraction is performed based on the conversion image to obtain multiple candidate connected areas, including: binarizing the conversion image to obtain a grayscale image; performing an opening operation on the grayscale image to obtain a denoised image; and performing contour extraction on the denoised image to obtain the multiple candidate connected areas.

[0063] The opening operation can remove noise data, boundary abnormal values, etc. in the grayscale image, and can also achieve boundary smoothing to a certain extent.

[0064] In step S23, each boundary point of the candidate connected region with the largest area among the multiple candidate connected regions is used as a preliminary boundary point.

[0065] In step S24, a linear transformation process is performed on the coordinates of each preliminary boundary point to obtain the plurality of first boundary points.

[0066] The multiple of the linear transformation is the ratio between the size of the input image (specifically, the input size when the input image is input into the feature extraction network) and the size of the drivable area boundary feature. For example, if the input size of the input image is 1280×720 and the size of the drivable area boundary feature is 320×180, the multiple of the linear transformation is 4.

[0067] It should be pointed out that in the existing conventional technology, after obtaining the boundary features of the drivable area of ​​the input image, an upsampling scheme based on a complex network structure is usually used to determine the multiple first boundary points. For example, a deconvolution operation is performed through a deconvolution network to restore the size of the boundary features of the drivable area to the size of the input image and give the classification results of the corresponding pixels to obtain the multiple first boundary points. However, when the network is inferred in hardware, the inference process of the operator corresponding to the upsampling operation is relatively time-consuming and cannot meet the real-time requirements of autonomous driving.

[0068] In comparison, the embodiment of the present invention adopts an image-based contour extraction and coordinate linear transformation scheme to replace the above-mentioned network-based upsampling scheme, that is, first convert the boundary features of the drivable area into image data, perform contour extraction and screening at the image level, and determine multiple preliminary boundary points that form the drivable area; then perform coordinate linear transformation to obtain the multiple first boundary points, wherein the coordinate linear transformation can be used to restore the coordinate values ​​of each preliminary boundary point to the coordinate values ​​corresponding to the size of the input image, so that the size of the drivable area formed by the multiple first boundary points is consistent with the size of the input image. In this way, the time consumption of post-processing operations can be greatly reduced, thereby improving the efficiency of drivable area segmentation, meeting the real-time requirements of autonomous driving, and the requirements of computing resources for this implementation are relatively lower.

[0069] Furthermore, in the process of performing coordinate linear transformation processing to restore the coordinate values ​​of each preliminary boundary point to the coordinate values ​​corresponding to the size of the input image, the embodiment of the present invention usually causes the boundary of the drivable area to have jagged edges, and there may also be noise points in the converted image obtained by pixel value conversion. If the contour is directly extracted from the converted image, it often causes the appearance of abnormal values, thereby causing multiple closed polygons, etc. Therefore, in the embodiment of the present invention, after the pixel value conversion is performed to obtain the converted image, binarization processing, opening operation processing, etc. are sequentially performed to effectively remove noise points and edge abnormal values, increase smoothness, and reduce the contour jagged edges caused by the subsequent execution of coordinate linear transformation, thereby obtaining more accurate and effective boundary points of the drivable area.

[0070] Reference Figure 3 , Figure 3 This is a comparison chart of the effects of directly extracting contours from a converted image and extracting contours from an image after binarization and opening operations on the converted image.

[0071] Specifically, Figure 3 The blue boundary line shown above is the contour of the drivable area obtained by directly extracting the contour of the converted image (i.e., the image obtained by converting the pixel values ​​of the boundary features of the drivable area); Figure 3 The blue boundary line shown below is the contour of the drivable area obtained by performing binarization and opening operations on the converted image to obtain a denoised image, and performing contour extraction on the denoised image.

[0072] By comparison, we can see that Figure 3 The outline of the drivable area above has some jagged problems and poor smoothness; in comparison, Figure 3 The aliasing issues of the drivable area outline below have been improved, and the smoothness and clarity have been significantly improved.

[0073] Reference Figure 4 , Figure 4 yes Figure 1 A partial flow chart of another specific implementation of step S11 in the embodiment; in this implementation, the operation of performing drivable area segmentation on the input image in step S11 may specifically include a boundary feature acquisition operation and a post-processing operation, wherein the boundary feature acquisition operation may include the following steps S41 to S43.

[0074] In step S41, multiple feature extraction operations of different scales are performed on the input image to obtain multiple image features, wherein the input data of the first feature extraction operation is the input image, and starting from the second feature extraction operation, the input data of the current feature extraction operation is the output data of the previous feature extraction operation.

[0075] In which, since the first feature extraction operation, the scale of feature extraction is gradually reduced. Without limitation, 5 feature extraction operations of different scales can be performed on the input image, and the scales of the first to fifth feature extractions are 1 / 4, 1 / 8, 1 / 16, 1 / 32, and 1 / 64, respectively.

[0076] In step S42, multiple feature fusion operations of different scales are performed based on the multiple image features to obtain fused features, wherein the multiple feature extraction operations correspond one-to-one to the multiple feature fusion operations, the input data of the first feature fusion operation is the image feature output by the last feature extraction operation, and starting from the second feature fusion operation, the input data of the current feature fusion operation includes the output data of the corresponding feature extraction operation and the output data of the previous feature fusion operation.

[0077] Among them, since the first feature fusion operation, the scale of feature fusion increases successively. Furthermore, the extraction scale of each feature extraction operation is consistent with the fusion scale of the corresponding feature fusion operation. Non-limitingly, 5 feature fusion operations of different scales can be performed, and the scales of the first to fifth feature fusions are 1 / 64, 1 / 32, 1 / 16, 1 / 8, and 1 / 4 respectively.

[0078] In step S43, a target image feature is selected from the multiple image features, and a feature splicing operation is performed on the target image feature and the fusion feature to obtain a spliced ​​feature, and then the spliced ​​feature is decoded to obtain a drivable area boundary feature.

[0079] In a specific implementation, the selected target image feature may be an image feature obtained by the first feature extraction operation (for example, an image feature obtained by performing a 1 / 4 scale feature extraction operation on the input image). The target image feature and the fusion feature have the same size.

[0080] Furthermore, before performing a feature splicing operation on the target image features and the fused features, a convolutional neural network (CNN) may be used to perform a convolution operation on the target image features and / or the fused features to obtain richer and more accurate feature information.

[0081] In the specific implementation, Figure 1 In the step S11, the boundary feature acquisition operation performed on the input image can be implemented by using a pre-trained model. Specifically, the pre-trained model at least includes a feature extraction network, a feature fusion network and a feature prediction network; the feature extraction network performs multiple feature extraction operations of different scales on the input image to obtain the multiple image features; the feature fusion network performs multiple feature fusion operations of different scales based on the multiple image features to obtain the fused features; the feature prediction network performs a feature splicing operation on the target image features in the multiple image features and the fused features to obtain spliced ​​features, and then decodes the spliced ​​features to obtain the boundary features of the drivable area.

[0082] The following combination Figure 5 to Figure 6 The initial model and training method, relevant principles, etc. used to obtain the pre-trained model in the embodiment of the present invention are described in detail.

[0083] Reference Figure 5 , Figure 5 It is a partial structural diagram of an initial model 50 used for training to obtain a pre-trained model in an embodiment of the present invention; Figure 6 Yes Figure 5 The flowchart of a training method for training the initial model 50 to obtain a pre-trained model is shown.

[0084] The training method may specifically include the following steps S61 to S64.

[0085] In step S61, an initial model is determined.

[0086] Specifically, the initial model 50 includes at least a feature extraction network 501 to be trained (corresponding to Figure 5 Backbone network in (Backbone)), feature fusion network 502 to be trained (corresponding to Figure 5 Neck network (Neck) in the training), feature prediction network 503 to be trained (corresponding to Figure 5 The drivable area segmentation head (Freespace_head) in FIG. 5 also includes an auxiliary network 504 to be trained (corresponding to Figure 5The auxiliary network 504 determines auxiliary boundary features based on the output data of each of the multiple feature fusion operations in the feature fusion network 502.

[0087] Among them, the main function of the backbone network is to extract useful feature information from the input image and the target image area, and pass this feature information to the subsequent network layer. The main function of the neck network is to perform multi-scale feature fusion based on the feature information extracted by the backbone network. The structure of the neck network can adopt a feature pyramid network (FPN) structure, which realizes the effective fusion of multi-scale features by constructing a feature pyramid to obtain the fusion feature.

[0088] Further, the auxiliary network 504 may include a plurality of first convolutional networks (corresponding to Figure 5 CNN 1 ), multiple second convolutional networks (corresponding to Figure 5 CNN 2 ) and a third convolutional network (corresponding to Figure 5 CNN 3 ), the multiple CNN 1 The output data of each of the multiple feature fusion operations corresponds one to one, and the multiple CNNs 1 And the multiple CNNs 2 One-to-one correspondence. Among them, each CNN 1 Used to perform convolution processing on the corresponding output data; then perform upsampling operation on the convolution processed data and input it into the corresponding CNN 2 Perform convolution processing; then perform convolution on each CNN 2 The output data of CNN is spliced ​​to obtain preliminary auxiliary boundary features; the preliminary auxiliary boundary features are input into the CNN 3 A convolution process is performed to obtain the auxiliary boundary feature.

[0089] Furthermore, the feature prediction network 503 may include a fourth convolutional network (corresponding to Figure 5 CNN 4 ) and the fifth convolutional network (corresponding to Figure 5 CNN 5 ), CNN 4 Decoding the spliced ​​features obtained by splicing the target image features and the fusion features to obtain preliminary boundary features of the drivable area; CNN 5 Used to perform convolution processing on the preliminary boundary features of the drivable area to obtain the boundary features of the drivable area.

[0090] Among them, CNN 1 ~CNN 5 The convolutional neural network structures used may be the same or different, and may be specifically configured in accordance with the actual application scenario. The embodiments of the present invention do not impose any limitation on this.

[0091] It should be pointed out that, in a specific implementation, the auxiliary network 504 can be used only in the model training process (it may no longer be used in the reasoning process) to assist in optimizing the parameters of each network in the initial model 50 during training. Compared with the existing conventional segmentation model reasoning scheme, which uses a complex network structure similar to the auxiliary network 504 to predict the boundary features of the drivable area, this implementation scheme uses a more lightweight and simplified feature prediction network 503 to predict the boundary features of the drivable area during the model reasoning process, and only uses the complex auxiliary network 504 to optimize the model parameters during the model training process. In this way, the computing resources and time overhead of model reasoning can be effectively reduced on the basis of obtaining sufficiently optimized network parameters.

[0092] In step S62, based on a preset first objective loss function, the sample image is input into the initial model for initial training to obtain an optimized feature extraction network, an optimized feature fusion network, an optimized feature prediction network and an optimized auxiliary network.

[0093] In step S63, the parameters of the optimized feature extraction network and the optimized feature fusion network are fixed unchanged, and based on the preset second objective loss function, the sample image is input into the initial model for retraining to obtain a fine-tuned feature prediction network and a fine-tuned auxiliary network.

[0094] In step S64, the pre-training model is constructed by using the optimized feature extraction network, the optimized feature fusion network, and the fine-tuned feature prediction network.

[0095] Furthermore, the first objective loss function L1 and the second objective loss function L2 are based on at least the first sub-loss function L 11 and the second sub-loss function L 12 The first sub-loss function L is obtained by weighted operation; during the training process, 11 The function value of is determined according to the difference between multiple predicted boundary points and multiple labeled boundary points. The second sub-loss function L 12 The function value of is determined according to the difference between multiple auxiliary predicted boundary points and multiple labeled boundary points.

[0096] Among them, the multiple predicted boundary points are obtained by performing post-processing operations on the sample boundary features output by the feature prediction network 503, the multiple auxiliary predicted boundary points are obtained by performing post-processing operations on the auxiliary boundary features output by the auxiliary network 504, and the multiple annotated boundary points are boundary points of the drivable area annotated in the sample image (for example, the drivable area is annotated in the sample image, and the outline or boundary line of the annotated drivable area is sampled to obtain the multiple annotated boundary points).

[0097] In the specific implementation, polygon annotation can be used to annotate the drivable area. During the training process, the annotated drivable area can be converted into a corresponding mask area for training. Two categories can be annotated in the sample image, namely "background" and "drivable area", among which various obstacles, construction areas, etc. belong to the background category.

[0098] In a specific implementation, the first sub-loss function L 11 And the second sub-loss function L 12 Existing loss functions can be used, including but not limited to: L1 loss function, L2 loss function, cross entropy loss function, mean square error loss function, etc.

[0099] Furthermore, the first sub-loss function L 11 The weight is greater than the second sub-loss function L 12 As a non-limiting example, the first sub-loss function L 11 The weight can be set to 1.0, and the second sub-loss function L 12 The weight can be set to 0.5.

[0100] In the embodiment of the present invention, since the feature prediction network 503 and the auxiliary network 504 in the initial model 50 are networks used to construct the pre-trained model and play a core role in the accuracy of obtaining the boundary points of the drivable area, a pre-trained model with higher prediction accuracy can be obtained by optimizing and fine-tuning the parameters of the feature prediction network 503 and the auxiliary network 504 using a two-round training scheme. Furthermore, compared with the auxiliary network 504, the importance level of the feature prediction network 503 is relatively higher, and accordingly, setting a larger weight for its first sub-loss function during the training process helps to obtain better training results.

[0101] Furthermore, the initial model 50 may be a multi-task model. In a specific implementation, the initial model 50 and the pre-trained model for reasoning may also include other appropriate task heads, such as an object detection task head (corresponding to Figure 5 detect_module_head in the ), lane line detection task head (corresponding to Figure 5lane_module_head in) and the depth estimation task header (corresponding to Figure 5 in the depth_module_head).

[0102] Specifically, the target detection task head performs target detection (including but not limited to: vehicle, pedestrian, traffic sign, building and other target detection) based on the fusion features output by the feature fusion network 502, and outputs target detection feature data; the target detection feature data is post-processed to obtain the target detection result. The lane line detection task head performs lane line detection based on the fusion features output by the feature fusion network 502, and outputs lane line detection feature data; the lane line detection feature data is post-processed to obtain the lane line detection result. The depth estimation task head performs depth estimation based on the fusion features output by the feature fusion network 502, and outputs the depth feature data of the target; the depth feature data is post-processed to obtain the depth data of the target (i.e., the distance between the target and the camera).

[0103] In the case where the initial model 50 also includes the above-mentioned task head networks, corresponding training data can be designed for each task head network during the training of the initial model 50. For example, for the target detection task head, rectangles can be used for annotation in the corresponding sample image; for the lane line detection task head, polygons, lines and points can be used for mixed annotation in the corresponding sample image; for the depth estimation task head, rectangles and points can be used for annotation in the corresponding sample image.

[0104] Furthermore, during the initial training, the first objective loss function L1 may be a function of the first sub-loss function L 11 , the second sub-loss function L 12 , the third sub-loss function L 13 , the fourth sub-loss function L 14 , the fifth sub-loss function L 15 The third sub-loss function L 13 The function value of is determined based on the difference between the target detection result obtained by the target detection task head and the target annotated in the sample image. The fourth sub-loss function L 14 The fifth sub-loss function L is determined based on the difference between the lane line detection result obtained by the lane line detection task head and the lane line marked in the sample image. 15 It is determined based on the difference between the depth data of the target obtained by the depth estimation task head and the depth of the target marked in the sample image.

[0105] Furthermore, after obtaining the target drivable area, target detection results, lane line detection results, target depth value and other data respectively, these data can be fused to obtain a complete multi-task detection result; and then provided to subsequent autonomous driving decision nodes (for example, route planning nodes).

[0106] Regarding the specific implementation scheme of post-processing the feature data output by each task head network, please refer to the scheme for post-processing the boundary features of the drivable area output by the feature prediction network 503 (i.e., the image-based post-processing scheme), which will not be repeated here. In addition, the specific network structure of each task head can be implemented using an existing similar task network structure, which is not limited in the embodiment of the present invention.

[0107] Reference Figure 7 , Figure 7 yes Figure 1 FIG. 4 is a flowchart of a specific implementation of step S13 in FIG. 4 . In this implementation, step S13 may specifically include steps S131 to S134.

[0108] In step S131, the plurality of first boundary points and the plurality of second boundary points are screened respectively to obtain screened first boundary points and screened second boundary points.

[0109] Furthermore, the first boundary point after screening may be each first boundary point located in the target image area among the plurality of first boundary points; the second boundary point after screening may be the boundary point remaining after removing the target boundary point from the plurality of second boundary points, and the target boundary point is a point on the first edge of the drivable area formed by the plurality of second boundary points. The first edge may be the lower edge of the drivable area.

[0110] In an embodiment of the present invention, after obtaining the multiple first boundary points and the multiple second boundary points, by screening and removing a portion of invalid boundary points, the amount of data can be effectively reduced, the number of rows and columns of the subsequently obtained cost matrix can be reduced, the overhead can be reduced, and the matching efficiency can be improved.

[0111] In step S132, the first boundary points after screening and the second boundary points after screening are paired in pairs to obtain multiple pairs of candidate boundary points.

[0112] In step S133, a cost matrix is ​​generated using the Euclidean distances between the multiple pairs of candidate boundary points, wherein multiple elements of the cost matrix correspond one-to-one to the multiple pairs of candidate boundary points, and the element value of each element is the Euclidean distance of the corresponding pair of candidate boundary points.

[0113] In step S134, the successfully matched boundary point pairs are determined from among the multiple pairs of candidate boundary points based on the cost matrix.

[0114] Furthermore, the cost matrix is ​​a two-dimensional matrix containing M rows of elements, where M is a positive integer; the step S134 may specifically include: for each row of the cost matrix, subtracting the minimum element of the row from each element of the row, and for each column of the cost matrix, subtracting the minimum element of the column from each element of the column, thereby obtaining a preliminary update matrix; performing one or more rounds of iteration operations based on the preliminary update matrix to obtain a target cost matrix, wherein in each round of iteration, a minimum number of horizontal lines and vertical lines are used to cover all elements whose element values ​​in the update matrix of the current round are the first value, and if the sum of the number of the horizontal lines and the vertical lines is less than M , then the smallest first element is subtracted from each first element in the current round of update matrix that is not covered by the horizontal line or the vertical line, and the smallest first element is added to each second element covered by both the horizontal line and the vertical line, to obtain the update matrix for the next round, and the next round of iteration is continued until the sum of the number of horizontal lines and vertical lines is equal to M, the iteration is stopped, and the update matrix of the last round is used as the target cost matrix, wherein the update matrix of the first round is the preliminary update matrix; the candidate boundary point pairs corresponding to each element whose element value in the target cost matrix is ​​the first value are used as the successfully matched boundary point pairs.

[0115] As a non-limiting embodiment, the first value may be set to 0, that is, each candidate boundary point pair with a Euclidean distance of 0 is regarded as the successfully matched boundary point pair.

[0116] From the above, the cost matrix is ​​obtained by using the Euclidean distance, and the iterative matching algorithm is used to help find the optimal matching point of each first boundary point in the second boundary point, thereby improving the accuracy of the target drivable area determined subsequently.

[0117] Continue to refer to Figure 1 In the specific implementation of step S14, the successfully matched first boundary points among the multiple first boundary points are replaced with matched second boundary points, and the area formed by the replaced first boundary points and at least a part of the unsuccessfully matched first boundary points is used as the target drivable area.

[0118] The at least part of the first boundary points that are not successfully matched may specifically be the first boundary points that are not successfully matched among the screened first boundary points.

[0119] As mentioned above, for the pixel-level drivable area boundary obtained by segmentation, especially the drivable area boundary far from the camera in the image, the smaller its size is, the more likely it is to have boundary missed detection or false detection problems after feature extraction, thereby causing the fuzzy problem of the drivable area boundary. In an embodiment of the present invention, since the target image area is a part of the input image, the original size of the former is smaller. When the original sizes of the two are scaled to meet the same input size, the size of the drivable area boundary in the target image area, especially the drivable area boundary far from the camera in the image, is larger than the size of the drivable area boundary in the input image. After performing feature extraction of the same scale (i.e., sampling at the same downsampling rate), the former can obtain relatively richer boundary feature information. That is, the positions of the multiple second boundary points will be more accurate than those of the multiple first boundary points.

[0120] Furthermore, by matching the multiple first boundary points with the multiple second boundary points, the second boundary points that best match each first boundary point are found; then the successfully matched second boundary points are used to replace the matched first boundary points; and finally the target drivable area is obtained based on the replaced first boundary points. Compared with using the area formed by the multiple first boundary points as the target drivable area, this embodiment can effectively improve the clarity and accuracy of the obtained target drivable area, and can especially improve the blur problem at the boundary of the drivable area far from the camera.

[0121] Reference Figure 8 , Figure 8 This is a comparison diagram of the target drivable area obtained by performing drivable area segmentation on the input image and the target drivable area obtained by using the solution of the embodiment of the present invention. Figure 8 The two pictures in the figure are magnified several times and show the effect.

[0122] Specifically, Figure 8 The blue boundary line shown above is a portion of the contour of the target drivable area formed by the plurality of first boundary points obtained by directly performing drivable area segmentation on the input image; Figure 8 The red boundary line shown below is a part of the outline of the target drivable area obtained by using the solution provided by the embodiment of the present invention, that is, Figure 8 The following is the effect of the target drivable area obtained by fusing multiple second boundary points obtained by segmenting the drivable area of ​​the target image area.

[0123] By comparison, we can see that Figure 8 The upper drivable area outline is at the position of the car on the left side of the truck in the distance (such as Figure 8 The area outlined by the red dotted circle in the middle) has boundary errors, which are relatively fuzzy and less smooth; Figure 8 The outline of the drivable area below is at the position of the car on the left side of the truck in the distance. There is no aforementioned error, and the boundary clarity and smoothness are improved.

[0124] Reference Fig. 9 , Fig. 9 : is a schematic diagram of a structure of a device for determining a drivable area in an embodiment of the present invention. The device may include:

[0125] A first segmentation module 91 is used to perform drivable area segmentation on the input image to obtain a plurality of first boundary points;

[0126] A second segmentation module 92 is used to perform drivable area segmentation on the target image area to obtain a plurality of second boundary points, wherein the target image area is a part of the input image;

[0127] A matching module 93, configured to match the plurality of first boundary points with the plurality of second boundary points to obtain successfully matched boundary point pairs;

[0128] The target drivable area determination module 94 is used to replace the successfully matched first boundary points among the multiple first boundary points with the matched second boundary points, and use the area formed by the replaced first boundary points and at least a part of the unsuccessfully matched first boundary points as the target drivable area.

[0129] For the principle, specific implementation and beneficial effects of the drivable area determination device, please refer to the previous article and Figures 1 to 8 The related description about the method for determining the drivable area shown is not repeated here.

[0130] The embodiment of the present invention further provides a storage medium, such as a computer-readable storage medium, on which a computer program is stored, and the computer program is executed by a processor. Figures 1 to 8 The steps of the method for determining a drivable area are shown. The computer-readable storage medium may include a non-volatile memory or a non-transitory memory, and may also include an optical disk, a mechanical hard disk, a solid-state hard disk, etc.

[0131] Specifically, in the embodiment of the present invention, the processor may be a central processing unit (CPU), and the processor may also be other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0132] It should also be understood that the memory in the embodiments of the present application may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example but not limitation, many forms of random access memory (RAM) are available, such as static RAM (SRAM), dynamic random access memory (DRAM), synchronous DRAM (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link DRAM (SLDRAM) and direct rambus RAM (DR RAM).

[0133] The embodiment of the present invention further provides a terminal, including a memory and a processor, wherein the memory stores a computer program that can be run on the processor, and the processor executes the above-mentioned Figures 1 to 8The steps of the method for determining a drivable area are shown.

[0134] It should be understood that the term "and / or" in this article is only a description of the association relationship of the associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article indicates that the associated objects before and after are in an "or" relationship.

[0135] The "plurality" appearing in the embodiments of the present application refers to two or more.

[0136] The first, second, etc. descriptions appearing in the embodiments of the present application are only used for illustration and distinction of the description objects. There is no order, nor do they indicate any special limitation on the number of devices in the embodiments of the present application, and cannot constitute any limitation on the embodiments of the present application.

[0137] It should be pointed out that the serial numbers of the steps in this embodiment do not limit the execution order of the steps.

[0138] Although the present invention is disclosed as above, the present invention is not limited thereto. Any person skilled in the art can make various changes and modifications without departing from the spirit and scope of the present invention. Therefore, the protection scope of the present invention shall be subject to the scope defined by the claims.

Claims

1. A method for determining a drivable area, characterized in that: include: Perform drivable area segmentation on the input image to obtain a plurality of first boundary points; Performing drivable area segmentation on a target image area to obtain a plurality of second boundary points, wherein the target image area is a part of the input image; Matching the plurality of first boundary points with the plurality of second boundary points to obtain successfully matched boundary point pairs; The successfully matched first boundary points among the plurality of first boundary points are replaced with matched second boundary points, and an area formed by the replaced first boundary points and at least a portion of the unsuccessfully matched first boundary points is used as a target drivable area.

2. The method according to claim 1, characterized in that The target image area is a preset area around the center point of the input image, or a preset area around the vanishing point of the drivable area in the input image.

3. The method according to claim 1, characterized in that Matching the plurality of first boundary points and the plurality of second boundary points to obtain successfully matched boundary point pairs includes: Screening the plurality of first boundary points and the plurality of second boundary points respectively to obtain screened first boundary points and screened second boundary points; Pairing the first boundary points after screening with the second boundary points after screening in pairs to obtain multiple pairs of candidate boundary points; Using the Euclidean distances between the multiple pairs of candidate boundary points, a cost matrix is ​​generated, wherein multiple elements of the cost matrix correspond to the multiple pairs of candidate boundary points one by one, and the element value of each element is the Euclidean distance of the corresponding pair of candidate boundary points; Based on the cost matrix, the successfully matched boundary point pairs are determined among the multiple pairs of candidate boundary points.

4. The method according to claim 3, characterized in that Meet one or more of the following: The screened first boundary points are first boundary points located in the target image area among the plurality of first boundary points; The screened second boundary points are the boundary points remaining after removing the target boundary points from the multiple second boundary points, and the target boundary points are points on the first edge of the drivable area formed by the multiple second boundary points.

5. The method according to claim 3 or 4, characterized in that: The cost matrix is ​​a two-dimensional matrix containing M rows of elements, where M is a positive integer; Determining the successfully matched boundary point pairs among the multiple pairs of candidate boundary points based on the cost matrix includes: For each row of the cost matrix, subtract the minimum element of the row from each element of the row, and for each column of the cost matrix, subtract the minimum element of the column from each element of the column, so as to obtain a preliminary update matrix; Perform one or more rounds of iterations based on the preliminary update matrix to obtain a target cost matrix, wherein in each round of iteration, a minimum number of horizontal lines and vertical lines are used to cover all elements whose element values ​​in the update matrix of the current round are the first value, and if the sum of the number of the horizontal lines and the vertical lines is less than M, then the smallest first element is subtracted from each first element in the update matrix of the current round that is not covered by the horizontal line or the vertical line, and the smallest first element is added to each second element that is covered by both the horizontal line and the vertical line, to obtain the update matrix of the next round, and continue the next round of iterations until the sum of the number of the horizontal lines and the vertical lines is equal to M, stop the iteration, and use the update matrix of the last round as the target cost matrix, wherein the update matrix of the first round is the preliminary update matrix; The candidate boundary point pairs corresponding to the elements whose element values ​​in the target cost matrix are the first values ​​are used as the successfully matched boundary point pairs.

6. The method according to claim 1, characterized in that The operation of performing drivable area segmentation on the input image includes a boundary feature acquisition operation and a post-processing operation; Wherein, the boundary feature acquisition operation determines the boundary features of the drivable area based on the input image; The post-processing operation comprises: Performing pixel value conversion on the boundary features of the drivable area to obtain a corresponding conversion image; Perform contour extraction based on the converted image to obtain multiple candidate connected regions; taking each boundary point of the candidate connected region with the largest area among the multiple candidate connected regions as a preliminary boundary point; The coordinates of each preliminary boundary point are linearly transformed to obtain the plurality of first boundary points.

7. The method according to claim 6, characterized in that Contour extraction is performed based on the conversion image to obtain multiple candidate connected regions, including: Binarizing the converted image to obtain a grayscale image; Performing an opening operation on the grayscale image to obtain a denoised image; Contour extraction is performed on the denoised image to obtain the multiple candidate connected regions.

8. The method according to claim 6, characterized in that The boundary feature acquisition operation includes: Performing multiple feature extraction operations of different scales on the input image to obtain multiple image features, wherein the input data of the first feature extraction operation is the input image, and starting from the second feature extraction operation, the input data of the current feature extraction operation is the output data of the previous feature extraction operation; Performing multiple feature fusion operations of different scales based on the multiple image features to obtain fused features, wherein the multiple feature extraction operations correspond to the multiple feature fusion operations one by one, the input data of the first feature fusion operation is the image feature output by the last feature extraction operation, and starting from the second feature fusion operation, the input data of the current feature fusion operation includes the output data of the corresponding feature extraction operation and the output data of the last feature fusion operation; A target image feature is selected from the multiple image features, and a feature splicing operation is performed on the target image feature and the fusion feature to obtain a splicing feature, and then the splicing feature is decoded to obtain the drivable area boundary feature.

9. The method according to claim 8, characterized in that The boundary feature acquisition operation performed on the input image is implemented by using a pre-trained model, wherein the pre-trained model includes a feature extraction network, a feature fusion network, and a feature prediction network; The feature extraction network performs multiple feature extraction operations of different scales on the input image to obtain the multiple image features; The feature fusion network performs multiple feature fusion operations of different scales based on the multiple image features to obtain the fused features; The feature prediction network performs a feature splicing operation on the target image feature among the multiple image features and the fusion feature to obtain a spliced ​​feature, and then decodes the spliced ​​feature to obtain the drivable area boundary feature.

10. The method according to claim 9, characterized in that The pre-trained model is trained in the following way: Determine an initial model, wherein the initial model includes a feature extraction network to be trained, a feature fusion network to be trained, a feature prediction network to be trained, and an auxiliary network to be trained, wherein the auxiliary network determines auxiliary boundary features based on output data of each of multiple feature fusion operations in the feature fusion network; Based on a preset first objective loss function, the sample image is input into the initial model for initial training to obtain an optimized feature extraction network, an optimized feature fusion network, an optimized feature prediction network, and an optimized auxiliary network; The parameters of the optimized feature extraction network and the optimized feature fusion network are fixed unchanged, and based on a preset second objective loss function, the sample image is input into the initial model for retraining to obtain a fine-tuned feature prediction network and a fine-tuned auxiliary network; The pre-training model is constructed by using the optimized feature extraction network, the optimized feature fusion network, and the fine-tuned feature prediction network.

11. The method according to claim 10, characterized in that The first objective loss function and the second objective loss function are obtained by performing a weighted operation based on at least the first sub-loss function and the second sub-loss function; During the training process, the function value of the first sub-loss function is determined according to the difference between the plurality of predicted boundary points and the plurality of labeled boundary points, and the function value of the second sub-loss function is determined according to the difference between the plurality of auxiliary predicted boundary points and the plurality of labeled boundary points; Among them, the multiple predicted boundary points are obtained by performing post-processing operations on the sample boundary features output by the feature prediction network, the multiple auxiliary predicted boundary points are obtained by performing post-processing operations on the auxiliary boundary features output by the auxiliary network, and the multiple annotated boundary points are boundary points of the drivable area annotated on the sample image.

12. The method according to claim 11, characterized in that The weight of the first sub-loss function is greater than the weight of the second sub-loss function.

13. A device for determining a drivable area, characterized in that: include: A first segmentation module is used to perform drivable area segmentation on the input image to obtain a plurality of first boundary points; a second segmentation module, configured to perform drivable area segmentation on a target image area to obtain a plurality of second boundary points, wherein the target image area is a part of the input image; A matching module, used for matching the plurality of first boundary points with the plurality of second boundary points to obtain successfully matched boundary point pairs; The target drivable area determination module is used to replace the successfully matched first boundary points among the multiple first boundary points with the matched second boundary points, and adopt the area formed by the replaced first boundary points and at least a part of the unsuccessfully matched first boundary points as the target drivable area.

14. A storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method for determining a drivable area according to any one of claims 1 to 12 are executed.

15. A terminal comprising a memory and a processor, wherein the memory stores a computer program that can be run on the processor, characterized in that: When the processor runs the computer program, the processor performs the steps of the method for determining a drivable area according to any one of claims 1 to 12.

16. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method for determining a drivable area according to any one of claims 1 to 12 are executed.

Images