Obstacle detection and positioning method, detection and positioning device and vehicle

Abstract

The present application provides an obstacle detection and positioning method, a detection and positioning device and a vehicle, the method comprising: using a target training set to train an initial detection network with a target loss function to obtain a target detection network, wherein the target loss function at least includes a distance loss function, and the distance loss function is a formula for calculating the difference between an actual distance and a predicted distance, and the target training set includes a target image set and corresponding target label information, and the target image set includes an image containing a target obstacle; using the target detection network to detect the target image, determining whether the target image contains the target obstacle according to the category information of the target obstacle in the detection result, and in the case that the target image contains the target obstacle, at least displaying the distance information between the target obstacle and the vehicle on the corresponding target image, thereby solving the problem in the prior art that it is difficult to accurately calculate the distance between the vehicle and the obstacle.

Description

Technical Field

[0001] The present application relates to the field of visual perception development for autonomous driving, and more specifically, to an obstacle detection and positioning method, a detection and positioning device, and a vehicle. Background Art

[0002] Due to the existence of the perception module, the autonomous driving vehicle can perceive the surrounding environmental information. Combined with the map module, planning module and control module, the autonomous driving vehicle can travel smoothly on various complex roads. Under normal circumstances, it is far from enough to only perceive the existence of different types of obstacles around the vehicle body. The distance information between the vehicle and the target to be detected is also particularly important. The perception module's prediction of the target distance information allows the vehicle to predict the position of the target to be detected in advance, thereby guiding the planning and control modules to control the vehicle's advance, retreat, turn and stop.

[0003] In recent years, the detection of traffic cones has not received enough attention and importance. In the scenario of autonomous driving, static buildings and landmarks can be easily located by building maps, but since traffic cones are temporarily placed and can be moved at any time, whether the vehicle can avoid or change lanes in time when encountering traffic cones cannot be solved by binding high-precision map information. The vehicle still needs to detect and estimate the location of the traffic cones, so the distance information between the vehicle and the traffic cones is very important.

[0004] The detection method for small target objects such as traffic cones is usually based on visual images or radar point cloud-based perception methods. Compared with the perception method based on visual images, the point cloud information received by the lidar will vary depending on the distance and size of the obstacle. It is difficult for the lidar to receive the point cloud information of traffic cones placed at a long distance, which can easily lead to missed detections.

[0005] Therefore, there is an urgent need for a method that can more accurately calculate the distance between a vehicle and an obstacle.

[0006] The above information disclosed in the background technology section is only used to enhance the understanding of the background technology of the technology described in this article. Therefore, the background technology may contain certain information that does not form the prior art known in this country for those skilled in the art. Summary of the invention

[0007] The main purpose of the present application is to provide a method for detecting and locating an obstacle, a detection and locating device and a vehicle, so as to solve the problem in the prior art that it is difficult to accurately calculate the distance between the vehicle and the obstacle.

[0008] According to one aspect of an embodiment of the present invention, a method for detecting and locating an obstacle is provided. The method is applied in a vehicle. The method comprises: using a target training set to train an initial detection network having a target loss function to obtain a target detection network, wherein the target loss function at least includes a distance loss function, and the distance loss function is a formula for calculating a difference between an actual distance and a predicted distance. The target training set includes a target image set and corresponding target label information. The target image set includes images containing target obstacles. The target label information at least includes an actual distance and category information. The actual distance is the distance between the target obstacle and the vehicle obtained by detection, the predicted distance is the distance between the target obstacle and the vehicle obtained by output of the initial detection network, and the category information is used to characterize the category of the target obstacle. Using the target detection network to detect a target image, determining whether the target image contains the target obstacle according to the category information of the target obstacle in the detection result, and if the target image contains the target obstacle, at least displaying the distance information between the target obstacle and the vehicle on the corresponding target image.

[0009] Optionally, the distance loss function includes a first distance loss function and a second distance loss function. When |r|≤α, the first distance loss function is When |r|>α, the first distance loss function is Where r = l d -f d , l d is the actual distance, f d is the predicted distance, α=1, and the second distance loss function is the ratio of |r| to the actual distance.

[0010] Optionally, the initial detection network includes a feature extraction network and a prediction network, and the target training set is used to train the initial detection network with a target loss function to obtain a target detection network, including: inputting the target training set into the feature extraction network for feature extraction to obtain multiple feature maps; inputting multiple feature maps into the prediction network for prediction to obtain at least multiple predicted distances; calculating a first target loss based on at least multiple predicted distances, multiple actual distances and the distance loss function; calculating a total target loss based on at least the first target loss, and adjusting the initial detection network based on the total target loss to obtain the target detection network.

[0011] Optionally, the label information also includes location information, and the location information includes location information of a first coordinate point and location information of a second coordinate point, the first coordinate point and the second coordinate point can form a minimum rectangular frame including the target obstacle, and at least the distance information between the target obstacle and the vehicle is displayed on the corresponding target image, and also includes: displaying the location information of the target obstacle on the target image on the corresponding target image.

[0012] Optionally, before using the target training set to train the initial detection network with the target loss function to obtain the target detection network, the detection and positioning method also includes: receiving the training set, and using the data enhancement method to perform data enhancement on the training set to obtain a predetermined training set, the training set includes an image set and corresponding label information, the image set includes the image containing the target obstacle and the image containing other obstacles, the data enhancement method includes at least one of the following: translation, flipping, random cropping, the other obstacles are obstacles other than the target obstacles; performing predetermined processing on the predetermined training set to obtain the target training set.

[0013] Optionally, receiving a training set and performing data enhancement on the training set using a data enhancement method to obtain a predetermined training set includes: performing data enhancement on the image containing the target obstacle using the data enhancement method, and updating the label information corresponding to the image containing the target obstacle to obtain a data set after data enhancement; and combining the data set after data enhancement with the image containing the other obstacles and the label information corresponding to the image containing the other obstacles to obtain the predetermined training set.

[0014] Optionally, the predetermined training set is subjected to predetermined processing to obtain the target training set, including: using a letterbox method to resize the images in a predetermined image set in the predetermined training set to obtain a first image set, wherein the predetermined image set is a collection of the images in the predetermined training set; converting the color channels of the first image set from RGB format to BGR format to obtain a second image set; normalizing the second image set to obtain the target image set, and combining the target image set and the corresponding target label information to obtain the target training set.

[0015] Optionally, the initial detection network includes a feature extraction network, the initial detection network is constructed based on a CenterNet network, and the feature extraction network is VarGNet.

[0016] According to another aspect of an embodiment of the present invention, a device for detecting and locating an obstacle is provided, the device being arranged in a vehicle, the device comprising: a training unit, configured to train an initial detection network having a target loss function using a target training set to obtain a target detection network, wherein the target loss function at least comprises a distance loss function, the distance loss function being a formula for calculating a difference between an actual distance and a predicted distance, the target training set comprising a target image set and corresponding target label information, the target image set comprising an image containing a target obstacle, the target label information comprising at least an actual distance and category information, the actual distance being the distance between the target obstacle and the vehicle obtained by detection, the predicted distance being the distance between the target obstacle and the vehicle obtained by output of the initial detection network, and the category information being used to characterize the category of the target obstacle; a detection unit, configured to detect a target image using the target detection network, determine whether the target image contains the target obstacle according to the category information of the target obstacle in the detection result, and if the target image contains the target obstacle, at least display the distance information between the target obstacle and the vehicle on the corresponding target image.

[0017] According to another aspect of an embodiment of the present invention, a vehicle is provided, comprising: an obstacle detection and positioning device, wherein the obstacle detection and positioning device is used to execute any one of the obstacle detection and positioning methods.

[0018] In an embodiment of the present invention, in the obstacle detection and positioning method, an initial detection network is trained by a target training set to obtain a target detection network, wherein the target training set includes a target image set and target label information, the target image set includes images containing target obstacles, and the target label information includes at least actual distance and category information. Since the target loss function includes at least a distance loss function, and the distance loss function is a formula for the difference between the actual distance and the predicted distance, the present solution trains the initial detection network so that the obtained target detection network can more accurately detect the category information of the target obstacle and predict the distance between the vehicle and the target obstacle, and then detects the target image by the target detection network. When the target image contains the target obstacle, at least the distance information between the target obstacle and the vehicle predicted by the target detection network is displayed on the corresponding target image. Compared with the prior art method of using laser radar to predict the distance information between the target obstacle and the vehicle, this solution trains the initial detection network with the target loss function to obtain the target detection network, and then uses the target detection network to detect the target image to obtain the category information of the target obstacle and the distance information between the target obstacle and the vehicle. Since this solution detects the target image, it ensures that the target obstacle information can be obtained more accurately, and the recognition ability of the target obstacle is strong, thereby ensuring that the predicted distance information between the target obstacle and the vehicle is more accurate, thereby solving the problem of the prior art that it is difficult to calculate the distance between the vehicle and the obstacle more accurately. In addition, this solution directly detects the acquired target image, that is, this solution realizes the end-to-end detection process, and there is no need to map the acquired laser point cloud information to obtain the mapped information, and then detect the mapped information, etc., which ensures that the target detection network has a small amount of calculation and a fast convergence speed. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The drawings constituting part of the present application are used to provide a further understanding of the present application. The exemplary embodiments and descriptions of the present application are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0020] Figure 1 A flow chart showing a method for detecting and locating an obstacle according to an embodiment of the present application is shown;

[0021] Figure 2 A schematic diagram of the structure of an obstacle detection and positioning device according to an embodiment of the present application is shown;

[0022] Figure 3 A flow chart of an obstacle detection and positioning device according to a specific embodiment of the present application is shown. DETAILED DESCRIPTION

[0023] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0024] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present application.

[0025] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequential order. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present application described here. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0026] As mentioned in the background technology, it is difficult to accurately calculate the distance between a vehicle and an obstacle in the prior art. In order to solve the above problem, in a typical embodiment of the present application, a method for detecting and locating an obstacle, a detection and locating device, and a vehicle are provided.

[0027] According to an embodiment of the present application, a method for detecting and locating an obstacle is provided.

[0028] Figure 1 FIG. 1 is a flow chart of a method for detecting and locating an obstacle according to an embodiment of the present application. The above detection and locating method is applied in a vehicle, such as Figure 1 As shown, the detection and positioning method includes the following steps:

[0029] Step S101, using a target training set to train an initial detection network with a target loss function to obtain a target detection network, wherein the target loss function at least includes a distance loss function, the distance loss function is a formula for calculating the difference between an actual distance and a predicted distance, the target training set includes a target image set and corresponding target label information, the target image set includes images containing target obstacles, the target label information at least includes an actual distance and category information, the actual distance is the distance between the target obstacle and the vehicle obtained by detection, the predicted distance is the distance between the target obstacle and the vehicle obtained by output of the initial detection network, and the category information is used to characterize the category of the target obstacle;

[0030] Step S102, using the target detection network to detect the target image, and determining whether the target image contains the target obstacle based on the category information of the target obstacle in the detection result. If the target image contains the target obstacle, at least the distance information between the target obstacle and the vehicle is displayed on the corresponding target image.

[0031] In the above-mentioned obstacle detection and positioning method, the initial detection network is trained by the target training set to obtain the target detection network, wherein the above-mentioned target training set includes a target image set and target label information, the above-mentioned target image set includes images containing target obstacles, and the above-mentioned target label information at least includes actual distance and category information. Since the above-mentioned target loss function at least includes a distance loss function, and the above-mentioned distance loss function is a formula for the difference between the actual distance and the predicted distance, this scheme trains the initial detection network so that the obtained target detection network can more accurately detect the category information of the target obstacle and predict the distance between the vehicle and the target obstacle. The target image is then detected by the target detection network. When the above-mentioned target image contains the above-mentioned target obstacle, at least the distance information between the target obstacle and the vehicle predicted by the target detection network is displayed on the corresponding target image. Compared with the prior art method of using laser radar to predict the distance information between the target obstacle and the vehicle, this solution trains the initial detection network with the target loss function to obtain the target detection network, and then uses the target detection network to detect the target image to obtain the category information of the target obstacle and the distance information between the target obstacle and the vehicle. Since this solution detects the target image, it ensures that the target obstacle information can be obtained more accurately, and the recognition ability of the target obstacle is strong, thereby ensuring that the predicted distance information between the target obstacle and the vehicle is more accurate, thereby solving the problem of the prior art that it is difficult to calculate the distance between the vehicle and the obstacle more accurately. In addition, this solution directly detects the acquired target image, that is, this solution realizes the end-to-end detection process, and there is no need to map the acquired laser point cloud information to obtain the mapped information, and then detect the mapped information, etc., which ensures that the target detection network has a small amount of calculation and a fast convergence speed.

[0032] Specifically, the target training set includes a target image set and corresponding target label information. The target image set includes images containing target obstacles. Of course, the target image set may include images containing other obstacles. The other obstacles may be obstacles other than the target obstacles. In the case where the target image set includes images containing target obstacles and images containing other obstacles, the category information in the target label information may specifically be the category of each obstacle. For example, in the case where the target obstacle is a traffic cone, the corresponding category information is TrafficCone, and in the case where the other obstacle is a car, the corresponding category information is Car.

[0033] In addition, the target image set may also include some images that do not contain any obstacles, so that the target image set can be further diversified. Of course, for an image in the target image set, the image may contain only the target obstacle, or only other obstacles, or may contain both the target obstacle and other obstacles. In an image, the image may contain several obstacles (target obstacles and / or other obstacles) corresponding to several sets of target label information (category information and actual distance).

[0034] In a specific embodiment of the present application, the target obstacle is a traffic cone, and the target image may be captured by a vehicle-mounted camera. Of course, the target image is not limited to being captured by a vehicle-mounted camera, but may also be captured by any other existing terminal device that can take pictures, and this is not limited in the present application.

[0035] It should be noted that the steps shown in the flowcharts of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and that, although a logical order is shown in the flowcharts, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0036] In order to further more accurately obtain the distance information between the target obstacle and the vehicle, in one embodiment of the present application, the distance loss function includes a first distance loss function and a second distance loss function. When |r|≤α, the first distance loss function is When |r|>α, the first distance loss function is Where r = l d -f d , l d is the actual distance mentioned above, f d is the predicted distance, α=1, and the second distance loss function is the ratio of |r| to the actual distance.

[0037] Specifically, the distance loss function L d Including the first distance loss function L d1 and the second distance loss function L d2 , that is, the mathematical expression of the above distance loss function is L d =L d1 +L d2 , where the above L d1 The mathematical expression of can be:

[0038]

[0039] The above L d2 The mathematical expression of can be:

[0040] In another embodiment of the present application, the above-mentioned initial detection network includes a feature extraction network and a prediction network, and the target training set is used to train the initial detection network with a target loss function to obtain a target detection network, including: inputting the above-mentioned target training set into the above-mentioned feature extraction network for feature extraction to obtain multiple feature maps; inputting the multiple feature maps into the above-mentioned prediction network for prediction to obtain at least multiple predicted distances; calculating the first target loss at least according to the multiple predicted distances, the multiple actual distances and the distance loss function; calculating the total target loss at least according to the first target loss, and adjusting the above-mentioned initial detection network according to the total target loss to obtain the above-mentioned target detection network. In this embodiment, the initial detection network is adjusted according to the calculated total target loss to obtain the target detection network, which ensures that the total target loss of the obtained target detection network is minimized, that is, the target loss function reaches a convergence state, and ensures that the obtained target detection network is better, which further ensures that the category of the target obstacle in the subsequent detection target image is more accurate, and further ensures that the distance information between the target obstacle and the vehicle obtained later is more accurate.

[0041] In actual application, the target loss function may also include a Heatmap loss function, a center point offset loss function, and a target length and width loss function.

[0042] In the process of training the initial detection network, the above-mentioned Heatmap loss function is a calculation formula for calculating the difference between the predicted center point and the actual center point. Taking an image of the target training set as an example, the above-mentioned predicted center point can be the center point of the obstacle (target obstacle and / or other obstacles) in the image predicted by the initial detection network, that is, the point with the largest calculated value of the Gaussian function. In this solution, the Heatmap loss function L hm The specific form of can be constructed using the Focal Loss function, and its mathematical expression is:

[0043]

[0044] Among them, the above Y xyc is the actual center point (i.e. Ground truth), is the predicted center point, N is the number of predicted center points, a=2, b=4.

[0045] Since the target image set in the target training set is subjected to feature extraction by the above feature extraction network, the output feature map will sample the image in the target image set. That is to say, a point in the output feature map represents a 4×4 area of ​​the image in the target image set, which will cause a large error in the position regression of the center point of the image. Therefore, it is necessary to correct it through the above-mentioned center point offset loss function. The specific mathematical expression of the above-mentioned center point offset loss function is:

[0046]

[0047] Among them, the above p is the coordinate of the center point of the image, Represents the approximate integer coordinates of the center point after scaling, R represents the scaling factor of the Heatmap, is the offset value predicted by the prediction network, and N is the number of predicted center points.

[0048] The specific mathematical expression of the above target length and width loss function is:

[0049]

[0050] Among them, the above is the predicted size, the above S m For true size.

[0051] In addition, when the target tag information also includes position information, the actual center point Y can be calculated by the position information. xyc And the real size S m , take an image in the target image set as an example to solve and explain the above position information. If the image contains only one target obstacle, then there is only one position information in the image, that is, the number of obstacles contained in the above image corresponds to the number of position information. In the actual application process, the above position information includes the position information of the first coordinate point and the position of the second coordinate point. The first coordinate point and the second coordinate point can form a minimum rectangular frame. The minimum rectangular frame is the smallest rectangular frame that can include the target obstacle. Then the first coordinate point and the second coordinate point can be understood as two coordinate points that are on the target diagonal of the above minimum rectangular frame and are corner points. Obviously, the actual center point of the corresponding target obstacle and the actual size S of the minimum rectangular frame can be calculated through the first coordinate point and the second coordinate point. m .

[0052] According to the above content, the mathematical expression of the above objective loss function can be: L = ω1L hm +ω2L offset +ω3L wh +ω4L d , where ω1, ω2, ω3 and ω4 are the weights of the corresponding loss functions, which are used to balance the weights between different loss functions.

[0053] When the above-mentioned target loss function includes a distance loss function, a Heatmap loss function, a center point offset loss function and a target length and width loss function, the second target loss can be calculated according to the above-mentioned Heatmap loss function, the third target loss can be calculated according to the above-mentioned center point offset loss function, and the fourth target loss can be calculated according to the above-mentioned target length and width loss function, then the above-mentioned total target loss can be the sum of the first target loss, the second target loss, the third target loss and the fourth target loss.

[0054] In a specific embodiment of the present application, the above-mentioned label information also includes position information, and the above-mentioned position information includes position information of a first coordinate point and position information of a second coordinate point. The above-mentioned first coordinate point and the above-mentioned second coordinate point can form a minimum rectangular frame including the above-mentioned target obstacle, and at least the distance information between the above-mentioned target obstacle and the above-mentioned vehicle is displayed on the corresponding above-mentioned target image, and also includes: displaying the above-mentioned position information of the above-mentioned target obstacle on the above-mentioned target image on the corresponding above-mentioned target image, and subsequently the above-mentioned vehicle can be further controlled according to the above-mentioned position information, further ensuring that the control of the above-mentioned vehicle is more precise, and further ensuring that the above-mentioned vehicle has higher safety.

[0055] In the above embodiment, since the above label information also includes position information, during the training process of the initial detection network, the above initial detection network has the ability to predict the position information of the target obstacle on the target image, further ensuring that the subsequent position information is more accurate.

[0056] In another embodiment of the present application, before the target training set is used to train the initial detection network with the target loss function to obtain the target detection network, the detection and positioning method further includes: receiving the training set, and using the data enhancement method to perform data enhancement on the training set to obtain a predetermined training set, the training set includes an image set and corresponding label information, the image set includes the image containing the target obstacle and the image containing other obstacles, the data enhancement method includes at least one of the following: translation, flipping, random cropping, the other obstacles are obstacles other than the target obstacles; performing predetermined processing on the predetermined training set to obtain the target training set. In this embodiment, the data enhancement method is used to perform data enhancement on the training set, which ensures that a large number of samples in the predetermined training set are obtained and diversified sample data are obtained, further ensuring that the obtained target detection network has strong generalization ability and good robustness. In addition, since the dimension of the input vector of the feature extraction network is consistent with the number of neurons in the corresponding layer, it is also necessary to perform predetermined processing on the predetermined image set in the predetermined training set to further ensure that the size of each image in the target image input to the feature extraction network is consistent.

[0057] In this solution, one or more of the above-mentioned data enhancement methods can be used to enhance the data of the training set. That is to say, the translation method can be used to enhance the data of the training set, the flipping method can be used to enhance the data of the training set, and the random cropping method can be used to enhance the data of the training set. Of course, the training set can also be enhanced by a combination of translation and flipping, translation and random cropping, flipping and random cropping, or translation, flipping and random cropping.

[0058] Specifically, the above-mentioned data enhancement method is not limited to one or more of the above-mentioned translation, flipping, and random cropping. The above-mentioned training set can also be enhanced by scaling, changing the perspective, and blocking a local area. That is to say, the present application does not limit the above-mentioned data enhancement method, and any one or more of the existing methods can be used to enhance the data of the above-mentioned training set.

[0059] In order to further ensure that the obtained target detection network has a strong generalization ability, in another embodiment of the present application, a training set is received, and a data enhancement method is used to perform data enhancement on the above-mentioned training set to obtain a predetermined training set, including: using the above-mentioned data enhancement method to perform data enhancement on the above-mentioned image containing the above-mentioned target obstacle, and updating the above-mentioned label information corresponding to the above-mentioned image containing the above-mentioned target obstacle to obtain a data set after data enhancement; combining the above-mentioned data set after data enhancement with the above-mentioned image containing the above-mentioned other obstacles and the label information corresponding to the above-mentioned image containing the above-mentioned other obstacles to obtain the above-mentioned predetermined training set.

[0060] In a specific embodiment of the present application, assuming that an image in the image set of the training set contains a target obstacle, assuming that the original size of the image is (W, H), and the position information and distance information of the target obstacle in the image are (X1, Y1; X2, Y2, d), if the image is enhanced by the translation method, the specific process is to translate the image in the horizontal direction by a distance of O x , the translation distance in the vertical direction is O y In order to prevent O x and O y Crossing the line, need to be O x and O y To restrict, that is as well as Among them, d_to_left is the horizontal coordinate (i.e., the horizontal coordinate) of the upper left corner of the smallest rectangular box closest to the upper left corner of the image (based on the upper left corner when a person is facing the screen), and d_to_top-1 is the vertical coordinate (i.e., the vertical coordinate) of the point at this time; d_to_right is the horizontal coordinate (i.e., the horizontal coordinate) of the lower right corner of the smallest rectangular box closest to the lower right corner of the image (based on the upper left corner when a person is facing the screen), and d_to_bottom is the vertical coordinate (i.e., the vertical coordinate) of the point at this time. The size of the image obtained after translation is updated to (W1′, H1′), and the corresponding location information and distance information including the target obstacle are updated to (X 1new1 , Y 1new1 ;X 2new1 , Y 2new1 , d′1), where X 1new =X1+O x , X 2new =X2+O x , Y 1new =Y1+O y and Y 2new =Y2+O y , d′1 is calculated by the pinhole imaging principle, and the category information of the target obstacle does not change.

[0061] In another specific embodiment of the present application, assuming that an image in the image set of the training set contains a target obstacle, assuming that the original size of the image is (W, H), and the position information and distance information of the target obstacle in the image are (X1, Y1; X2, Y2, d), the image is data enhanced by flipping, and the size of the image obtained after flipping is updated to (W2', H2'), and the corresponding position information and distance information of the target obstacle are updated to (X1, Y1; X2, Y2, d). 1new2 , Y 1new2 ;X 2new2 , Y 2new2 , d′2), where X 1new2 =W′-X1,X 2new2 =W′-X2,Y 1new2 =Y1 and Y 2new2 =Y2, d′2 is calculated by the pinhole imaging principle, and the category information of the target obstacle does not change.

[0062] In another specific embodiment of the present application, assuming that an image in the image set of the training set contains a target obstacle, assuming that the original size of the image is (W, H), and the position information and distance information of the target obstacle in the image are (X1, Y1; X2, Y2, d), the image is data enhanced by cropping, and the size of the image obtained after flipping is updated to (W3', H3'), and the corresponding position information and distance information of the target obstacle are updated to (X1, Y1; X2, Y2, d). 1new3 , Y 1new3 ;X 2new3 , Y 2new3 , d′3), where W3′=Crop_X_max-Crop_X_min, H3′=Crop_Y_max-Crop_Y_min, Crop X_max, Crop_X_min, Crop_Y max and Crop_Y_min are user-defined variable names, the value range of Crop_X_max is [W-d_to_right, W], the value range of Crop_Y_max is [H-d_to_bottom, H], the value range of Crop_X_min is [0, d_to_left], the value range of Crop_Y_min is [0, d_to_top], X 1new3 =X1-Crop_X_min,Y 1new3 =Y1-Crop_Y_min,X 2new3 =X2-Crop_X_min,Y 2new3 =Y2-Crop_Y_min, d′3 is calculated by the pinhole imaging principle, and the category information of the target obstacle does not change.

[0063] In one embodiment of the present application, the predetermined training set is subjected to predetermined processing to obtain the target training set, including: using the letterbox method to resize the images in the predetermined image set in the predetermined training set to obtain a first image set, the predetermined image set being a collection of the images in the predetermined training set; converting the color channels of the first image set from RGB to BGR to obtain a second image set; normalizing the second image set to obtain the target image set, and combining the target image set and the corresponding target label information to obtain the target training set. In this embodiment, the predetermined image set in the predetermined training set is subjected to predetermined processing before the target training set is input into the initial detection network, which further ensures that the performance of the target detection network obtained by training is better in all aspects.

[0064] Specifically, the size of each image in the first image set is 512×512×3, where 3 is used to represent the number of color channels of each image in the first image set.

[0065] Since the pixel value of each image in the second image set is in [0, 255], after normalization processing, the pixel value of each image in the target image set is obtained to be in [0, 1].

[0066] In order to ensure that the target detection network obtained subsequently is a lightweight model architecture, and to further ensure that the reasoning speed of the target detection network is fast, in another embodiment of the present application, the initial detection network includes a feature extraction network, the initial detection network is constructed based on the CenterNet network, and the feature extraction network is VarGNet.

[0067] The present application also provides an obstacle detection and positioning device. It should be noted that the obstacle detection and positioning device of the present application can be used to execute the obstacle detection and positioning method provided by the present application. The obstacle detection and positioning device provided by the present application is introduced below.

[0068] Figure 2 Schematic diagram of an obstacle detection and positioning device according to an embodiment of the present application. Figure 2 As shown, the detection and positioning device includes:

[0069] The training unit 10 is used to train an initial detection network having a target loss function using a target training set to obtain a target detection network, wherein the target loss function at least includes a distance loss function, and the distance loss function is a formula for calculating the difference between an actual distance and a predicted distance. The target training set includes a target image set and corresponding target label information, and the target image set includes images containing target obstacles. The target label information at least includes an actual distance and category information. The actual distance is the distance between the target obstacle and the vehicle obtained by detection, and the predicted distance is the distance between the target obstacle and the vehicle obtained by output of the initial detection network, and the category information is used to characterize the category of the target obstacle.

[0070] The detection unit 20 is used to detect the target image using the target detection network, determine whether the target image contains the target obstacle according to the category information of the target obstacle in the detection result, and if the target image contains the target obstacle, at least display the distance information between the target obstacle and the vehicle on the corresponding target image.

[0071] In the above-mentioned obstacle detection and positioning device, the training unit is used to train the initial detection network through the target training set to obtain the target detection network, wherein the above-mentioned target training set includes a target image set and target label information, the above-mentioned target image set includes an image containing a target obstacle, and the above-mentioned target label information at least includes an actual distance and category information. Since the above-mentioned target loss function at least includes a distance loss function, and the above-mentioned distance loss function is a formula for the difference between the actual distance and the predicted distance, this scheme trains the initial detection network so that the obtained target detection network can more accurately detect the category information of the target obstacle and predict the distance between the vehicle and the target obstacle. The detection unit is used to detect the target image through the target detection network. When the above-mentioned target image contains the above-mentioned target obstacle, at least the distance information between the target obstacle and the vehicle predicted by the target detection network is displayed on the corresponding target image. Compared with the prior art method based on laser radar to predict the distance information between the target obstacle and the vehicle, this solution trains the initial detection network with the target loss function to obtain the target detection network, and then uses the target detection network to detect the target image to obtain the category information of the target obstacle and the distance information between the target obstacle and the vehicle. Since this solution detects the target image, it ensures that the target obstacle information can be obtained more accurately, and the recognition ability of the target obstacle is strong, thereby ensuring that the predicted distance information between the target obstacle and the vehicle is more accurate, thereby solving the problem of the prior art that it is difficult to calculate the distance between the vehicle and the obstacle more accurately. In addition, this solution directly detects the acquired target image, that is, this solution realizes the end-to-end detection process, and there is no need to map the acquired laser point cloud information to obtain the mapped information, and then detect the mapped information, etc., which ensures that the target detection network has a small amount of calculation and a fast convergence speed.

[0072] Specifically, the target training set includes a target image set and corresponding target label information. The target image set includes images containing target obstacles. Of course, the target image set may include images containing other obstacles. The other obstacles may be obstacles other than the target obstacles. In the case where the target image set includes images containing target obstacles and images containing other obstacles, the category information in the target label information may specifically be the category of each obstacle. For example, in the case where the target obstacle is a traffic cone, the corresponding category information is TrafficCone, and in the case where the other obstacle is a car, the corresponding category information is Car.

[0073] In addition, the target image set may also include some images that do not contain any obstacles, so that the target image set can be further diversified. Of course, for an image in the target image set, the image may contain only the target obstacle, or only other obstacles, or may contain both the target obstacle and other obstacles. In an image, the image may contain several obstacles (target obstacles and / or other obstacles) corresponding to several sets of target label information (category information and actual distance).

[0074] In a specific embodiment of the present application, the target obstacle is a traffic cone, and the target image may be captured by a vehicle-mounted camera. Of course, the target image is not limited to being captured by a vehicle-mounted camera, but may also be captured by any other existing terminal device that can take pictures, and this is not limited in the present application.

[0075] In order to further more accurately obtain the distance information between the target obstacle and the vehicle, in one embodiment of the present application, the distance loss function includes a first distance loss function and a second distance loss function. When |r|≤α, the first distance loss function is When |r|>α, the first distance loss function is Where r = l d -f d , l d is the actual distance mentioned above, f d is the predicted distance, α=1, and the second distance loss function is the ratio of |r| to the actual distance.

[0076] Specifically, the distance loss function L d Including the first distance loss function L d1 and the second distance loss function L d2 , that is, the mathematical expression of the above distance loss function is L d =L d1 +L d2 , where the above L d1 The mathematical expression of can be:

[0077]

[0078] The above L d2 The mathematical expression of can be:

[0079] In another embodiment of the present application, the initial detection network includes a feature extraction network and a prediction network, and the training unit includes a feature extraction module, a prediction module, a calculation module and a first adjustment module, wherein the feature extraction module is used to input the target training set into the feature extraction network for feature extraction to obtain multiple feature maps; the prediction module is used to input multiple feature maps into the prediction network for prediction to obtain at least multiple predicted distances; the calculation module is used to calculate the first target loss at least based on the multiple predicted distances, the multiple actual distances and the distance loss function; the first adjustment module is used to calculate the total target loss at least based on the first target loss, and adjust the initial detection network according to the total target loss to obtain the target detection network. In this embodiment, the initial detection network is adjusted according to the calculated total target loss to obtain the target detection network, which ensures that the total target loss of the target detection network is minimized, that is, the target loss function reaches a convergence state, and ensures that the target detection network is better, which further ensures that the category of the target obstacle in the subsequent detection target image is more accurate, and further ensures that the distance information between the target obstacle and the vehicle obtained later is more accurate.

[0080] In actual application, the target loss function may also include a Heatmap loss function, a center point offset loss function, and a target length and width loss function.

[0081] In the process of training the initial detection network, the above-mentioned Heatmap loss function is a calculation formula for calculating the difference between the predicted center point and the actual center point. Taking an image of the target training set as an example, the above-mentioned predicted center point can be the center point of the obstacle (target obstacle and / or other obstacles) in the image predicted by the initial detection network, that is, the point with the largest calculated value of the Gaussian function. In this solution, the Heatmap loss function L hm The specific form of can be constructed using the Focal Loss function, and its mathematical expression is:

[0082]

[0083] Among them, the above Y xyc is the actual center point (i.e. Ground truth), is the predicted center point, N is the number of predicted center points, a=2, b=4.

[0084] Since the target image set in the target training set is subjected to feature extraction by the above feature extraction network, the output feature map will sample the image in the target image set. That is to say, a point in the output feature map represents a 4×4 area of ​​the image in the target image set, which will cause a large error in the position regression of the center point of the image. Therefore, it is necessary to correct it through the above-mentioned center point offset loss function. The specific mathematical expression of the above-mentioned center point offset loss function is:

[0085]

[0086] Among them, the above p is the coordinate of the center point of the image, Represents the approximate integer coordinates of the center point after scaling, R represents the scaling factor of the Heatmap, is the offset value predicted by the prediction network, and N is the number of predicted center points.

[0087] The specific mathematical expression of the above target length and width loss function is:

[0088]

[0089] Among them, the above is the predicted size, the above S m For true size.

[0090] In addition, when the target tag information also includes position information, the actual center point Y can be calculated by the position information. xyc And the real size S m , take an image in the target image set as an example to solve and explain the above position information. If the image contains only one target obstacle, then there is only one position information in the image, that is, the number of obstacles contained in the above image corresponds to the number of position information. In the actual application process, the above position information includes the position information of the first coordinate point and the position of the second coordinate point. The first coordinate point and the second coordinate point can form a minimum rectangular frame. The minimum rectangular frame is the smallest rectangular frame that can include the target obstacle. Then the first coordinate point and the second coordinate point can be understood as two coordinate points that are on the target diagonal of the above minimum rectangular frame and are corner points. Obviously, the actual center point of the corresponding target obstacle and the actual size S of the minimum rectangular frame can be calculated through the first coordinate point and the second coordinate point. m .

[0091] According to the above content, the mathematical expression of the above objective loss function can be: L = ω1L hm +ω2L offset +ω3L wh +ω4L d , where ω1, ω2, ω3 and ω4 are the weights of the corresponding loss functions, which are used to balance the weights between different loss functions.

[0092] When the above-mentioned target loss function includes a distance loss function, a Heatmap loss function, a center point offset loss function and a target length and width loss function, the second target loss can be calculated according to the above-mentioned Heatmap loss function, the third target loss can be calculated according to the above-mentioned center point offset loss function, and the fourth target loss can be calculated according to the above-mentioned target length and width loss function, then the above-mentioned total target loss can be the sum of the first target loss, the second target loss, the third target loss and the fourth target loss.

[0093] In a specific embodiment of the present application, the above-mentioned label information also includes position information, and the above-mentioned position information includes position information of a first coordinate point and position information of a second coordinate point. The above-mentioned first coordinate point and the above-mentioned second coordinate point can form a minimum rectangular frame including the above-mentioned target obstacle. The above-mentioned detection unit also includes a display module, which is used to display the above-mentioned position information of the above-mentioned target obstacle on the above-mentioned target image on the corresponding above-mentioned target image. Subsequently, the above-mentioned vehicle can be further controlled according to the above-mentioned position information, further ensuring that the control of the above-mentioned vehicle is more precise, and further ensuring that the above-mentioned vehicle has higher safety.

[0094] In the above embodiment, since the above label information also includes position information, during the training process of the initial detection network, the above initial detection network has the ability to predict the position information of the target obstacle on the target image, further ensuring that the subsequent position information is more accurate.

[0095] In another embodiment of the present application, the above-mentioned detection and positioning device also includes a receiving unit and a predetermined processing unit, wherein the above-mentioned receiving unit is used to receive a training set before using the target training set to train the initial detection network with the target loss function to obtain the target detection network, and use a data enhancement method to perform data enhancement on the above-mentioned training set to obtain a predetermined training set, the above-mentioned training set includes an image set and corresponding label information, the above-mentioned image set includes the above-mentioned images containing the above-mentioned target obstacles and the above-mentioned images containing other obstacles, the above-mentioned data enhancement method includes at least one of the following: translation, flipping, random cropping, and the above-mentioned other obstacles are obstacles other than the above-mentioned target obstacles; the above-mentioned predetermined processing unit is used to perform predetermined processing on the above-mentioned predetermined training set to obtain the above-mentioned target training set. In this embodiment, a data enhancement method is used to enhance the training set, which ensures that a large number of samples in the predetermined training set are obtained and diversified sample data are obtained, further ensuring that the obtained target detection network has strong generalization ability and good robustness. In addition, since the dimension of the input vector of the feature extraction network is consistent with the number of neurons in the corresponding layer, it is also necessary to perform predetermined processing on the predetermined image set in the predetermined training set, further ensuring that the size of each image in the target image input to the feature extraction network is consistent.

[0096] In this solution, one or more of the above-mentioned data enhancement methods can be used to enhance the data of the training set. That is to say, the translation method can be used to enhance the data of the training set, the flipping method can be used to enhance the data of the training set, and the random cropping method can be used to enhance the data of the training set. Of course, the training set can also be enhanced by a combination of translation and flipping, translation and random cropping, flipping and random cropping, or translation, flipping and random cropping.

[0097] Specifically, the above-mentioned data enhancement method is not limited to one or more of the above-mentioned translation, flipping, and random cropping. The above-mentioned training set can also be enhanced by scaling, changing the perspective, and blocking a local area. That is to say, the present application does not limit the above-mentioned data enhancement method, and any one or more of the existing methods can be used to enhance the data of the above-mentioned training set.

[0098] In order to further ensure that the obtained target detection network has a strong generalization ability, in another embodiment of the present application, the above-mentioned receiving unit includes an updating module and a combining module, wherein the above-mentioned updating module is used to adopt the above-mentioned data enhancement method to perform data enhancement on the above-mentioned image containing the above-mentioned target obstacle, and update the above-mentioned label information corresponding to the above-mentioned image containing the above-mentioned target obstacle to obtain the data set after data enhancement; the above-mentioned combining module is used to combine the above-mentioned data set after data enhancement with the above-mentioned image containing the above-mentioned other obstacles and the label information corresponding to the above-mentioned image containing the above-mentioned other obstacles to obtain the above-mentioned predetermined training set.

[0099] In a specific embodiment of the present application, assuming that an image in the image set of the training set contains a target obstacle, assuming that the original size of the image is (W, H), and the position information and distance information of the target obstacle in the image are (X1, Y1; X2, Y2, d), if the image is enhanced by the translation method, the specific process is to translate the image in the horizontal direction by a distance of O x , the translation distance in the vertical direction is O y In order to prevent O x and O y Crossing the line, need to be O x and O y To restrict, that is as well as Among them, d_to_left is the horizontal coordinate (i.e., the horizontal coordinate) of the upper left corner of the smallest rectangular box closest to the upper left corner of the image (based on the upper left corner when a person is facing the screen), and d_to_top-1 is the vertical coordinate (i.e., the vertical coordinate) of the point at this time; d_to_right is the horizontal coordinate (i.e., the horizontal coordinate) of the lower right corner of the smallest rectangular box closest to the lower right corner of the image (based on the upper left corner when a person is facing the screen), and d_to_bottom is the vertical coordinate (i.e., the vertical coordinate) of the point at this time. The size of the image obtained after translation is updated to (W1′, H1′), and the corresponding location information and distance information including the target obstacle are updated to (X 1new1 , Y 1new1 ;X 2new1 , Y 2new1 , d′1), where X 1new =X1+O x , X 2new =X2+O x , Y 1new =Y1+O y and Y 2new =Y2+O y , d′1 is calculated by the pinhole imaging principle, and the category information of the target obstacle does not change.

[0100] In another specific embodiment of the present application, assuming that an image in the image set of the training set contains a target obstacle, assuming that the original size of the image is (W, H), and the position information and distance information of the target obstacle in the image are (X1, Y1; X2, Y2, d), the image is data enhanced by flipping, and the size of the image obtained after flipping is updated to (W2', H2'), and the corresponding position information and distance information of the target obstacle are updated to (X1, Y1; X2, Y2, d). 1new2 , Y 1new2 ;X 2new2 , Y 2new2 , d′2), where X 1new2 =W′-X1,X 2new2 =W′-X2,Y 1new2 =Y1 and Y 2new2 =Y2, d′2 is calculated by the pinhole imaging principle, and the category information of the target obstacle does not change.

[0101] In another specific embodiment of the present application, assuming that an image in the image set of the training set contains a target obstacle, assuming that the original size of the image is (W, H), and the position information and distance information of the target obstacle in the image are (X1, Y1; X2, Y2, d), the image is data enhanced by cropping, and the size of the image obtained after flipping is updated to (W3′, H3′), and the corresponding position information and distance information of the target obstacle are updated to (X1, Y1; X2, Y2, d). 1new3 , Y 1new3 ;X 2new3 , Y 2new3 , d′3), where W3′=Crop_X_max-Crop_X_min, H3′=Crop_Y_max-Crop_Y_min, Crop_X_max, Crop_X_min, Crop_Y_max and Crop_Y_min are user-defined variable names, the value range of Crop_X_max is [W-d_to_right, W], the value range of Crop_Y_max is [H-d_to_bottom, H], the value range of Crop_X_min is [0, d_to_left], the value range of Crop_Y_min is [0, d_to_top], X 1new3 =X1-Crop_X_min,Y 1new3 =Y1-Crop_Y_min,X 2new3 =X2-Crop_X_min,Y 2new3 =Y2-Crop_Y_min, d′3 is calculated by the pinhole imaging principle, and the category information of the target obstacle does not change.

[0102] In one embodiment of the present application, the predetermined processing unit includes a second adjustment module, a conversion module and a normalization processing module, wherein the second adjustment module is used to use the letterbox method to resize the images in the predetermined image set in the predetermined training set to obtain a first image set, and the predetermined image set is a collection of the images in the predetermined training set; the conversion module is used to convert the color channels of the first image set from RGB to BGR to obtain a second image set; the normalization processing module is used to normalize the second image set to obtain the target image set, and combine the target image set and the corresponding target label information to obtain the target training set. In this embodiment, before the target training set is input into the initial detection network, the predetermined image set in the predetermined training set is subjected to predetermined processing, which further ensures that the performance of the target detection network obtained by training is better in all aspects.

[0103] Specifically, the size of each image in the first image set is 512×512×3, where 3 is used to represent the number of color channels of each image in the first image set.

[0104] Since the pixel value of each image in the second image set is in [0, 255], after normalization processing, the pixel value of each image in the target image set is obtained to be in [0, 1].

[0105] In order to ensure that the target detection network obtained subsequently is a lightweight model architecture, and to further ensure that the reasoning speed of the target detection network is fast, in another embodiment of the present application, the initial detection network includes a feature extraction network, the initial detection network is constructed based on the CenterNet network, and the feature extraction network is VarGNet.

[0106] The obstacle detection and positioning device comprises a processor and a memory. The training unit and the detection unit are stored in the memory as program units. The processor executes the program units stored in the memory to realize corresponding functions.

[0107] The processor includes a kernel, which retrieves the corresponding program unit from the memory. One or more kernels can be provided, and the problem of difficulty in accurately calculating the distance between the vehicle and the obstacle in the prior art can be solved by adjusting the kernel parameters.

[0108] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM, and the memory includes at least one memory chip.

[0109] An embodiment of the present invention provides a computer-readable storage medium on which a program is stored. When the program is executed by a processor, the above-mentioned obstacle detection and positioning method is implemented.

[0110] An embodiment of the present invention provides a processor, and the processor is used to run a program, wherein the obstacle detection and positioning method is executed when the program is run.

[0111] In a typical embodiment of the present application, a vehicle is further provided, which includes an obstacle detection and positioning device, and the above-mentioned detection and positioning device is used to execute any one of the above-mentioned detection and positioning methods.

[0112] The above-mentioned vehicle includes the above-mentioned detection and positioning device, and the above-mentioned detection and positioning device is used to execute any one of the above-mentioned detection and positioning methods. In the above-mentioned detection and positioning method, the initial detection network is trained by a target training set to obtain a target detection network, wherein the above-mentioned target training set includes a target image set and target label information, the above-mentioned target image set includes an image containing a target obstacle, and the above-mentioned target label information at least includes an actual distance and category information. Since the above-mentioned target loss function at least includes a distance loss function, and the above-mentioned distance loss function is a formula for the difference between the actual distance and the predicted distance, the present scheme trains the initial detection network so that the obtained target detection network can more accurately detect the category information of the target obstacle and predict the distance between the vehicle and the target obstacle, and then detects the target image through the target detection network. When the above-mentioned target image contains the above-mentioned target obstacle, at least the distance information between the target obstacle and the vehicle predicted by the target detection network is displayed on the corresponding target image. Compared with the prior art method of using laser radar to predict the distance information between the target obstacle and the vehicle, this solution trains the initial detection network with the target loss function to obtain the target detection network, and then uses the target detection network to detect the target image to obtain the category information of the target obstacle and the distance information between the target obstacle and the vehicle. Since this solution detects the target image, it ensures that the target obstacle information can be obtained more accurately, and the recognition ability of the target obstacle is strong, thereby ensuring that the predicted distance information between the target obstacle and the vehicle is more accurate, thereby solving the problem of the prior art that it is difficult to calculate the distance between the vehicle and the obstacle more accurately. In addition, this solution directly detects the acquired target image, that is, this solution realizes the end-to-end detection process, and there is no need to map the acquired laser point cloud information to obtain the mapped information, and then detect the mapped information, etc., which ensures that the target detection network has a small amount of calculation and a fast convergence speed.

[0113] An embodiment of the present invention provides a device, the device including a processor, a memory, and a program stored in the memory and executable on the processor, and when the processor executes the program, at least the following steps are implemented:

[0114] Step S101, using a target training set to train an initial detection network with a target loss function to obtain a target detection network, wherein the target loss function at least includes a distance loss function, the distance loss function is a formula for calculating the difference between an actual distance and a predicted distance, the target training set includes a target image set and corresponding target label information, the target image set includes images containing target obstacles, the target label information at least includes an actual distance and category information, the actual distance is the distance between the target obstacle and the vehicle obtained by detection, the predicted distance is the distance between the target obstacle and the vehicle obtained by output of the initial detection network, and the category information is used to characterize the category of the target obstacle;

[0115] Step S102, using the target detection network to detect the target image, and determining whether the target image contains the target obstacle based on the category information of the target obstacle in the detection result. If the target image contains the target obstacle, at least the distance information between the target obstacle and the vehicle is displayed on the corresponding target image.

[0116] The devices in this article can be servers, PCs, PADs, mobile phones, etc.

[0117] The present application also provides a computer program product, which, when executed on a data processing device, is suitable for executing a program for initializing at least the following method steps:

[0118] Step S101, using a target training set to train an initial detection network with a target loss function to obtain a target detection network, wherein the target loss function at least includes a distance loss function, the distance loss function is a formula for calculating the difference between an actual distance and a predicted distance, the target training set includes a target image set and corresponding target label information, the target image set includes images containing target obstacles, the target label information at least includes an actual distance and category information, the actual distance is the distance between the target obstacle and the vehicle obtained by detection, the predicted distance is the distance between the target obstacle and the vehicle obtained by output of the initial detection network, and the category information is used to characterize the category of the target obstacle;

[0119] Step S102, using the target detection network to detect the target image, and determining whether the target image contains the target obstacle based on the category information of the target obstacle in the detection result. If the target image contains the target obstacle, at least the distance information between the target obstacle and the vehicle is displayed on the corresponding target image.

[0120] In order to enable those skilled in the art to more clearly understand the technical solution of the present application, the technical solution and technical effects of the present application will be explained below in conjunction with specific embodiments.

[0121] Example

[0122] like Figure 3 As shown, before the target training set is input into the initial detection network with the target loss function, the received training set is subjected to data enhancement and predetermined processing to obtain the target training set. The specific training process of the initial detection network is as follows:

[0123] The target training set (target image set and target label information) is input into the feature extraction network (i.e., VarGNet network, also commonly referred to as backbone network) to obtain multiple feature maps, and the multiple feature maps are input into the prediction network to obtain the predicted distance, predicted center point, predicted offset and predicted size. The total target loss is then calculated based on the predicted distance, predicted center point, predicted offset and predicted size, the corresponding label information and the target loss function (i.e., distance loss function, Heatmap loss function, center point offset loss function and target width and height loss function). Finally, the initial detection network is continuously adjusted according to the total target loss to obtain the target detection network.

[0124] In the above embodiments of the present invention, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0125] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only schematic. For example, the division of the above-mentioned units can be a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0126] The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0127] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0128] If the above-mentioned integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server or a network device, etc.) to perform all or part of the steps of the above-mentioned methods of each embodiment of the present invention. The aforementioned storage medium includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, RandomAccess Memory), mobile hard disk, magnetic disk or optical disk and other media that can store program codes.

[0129] From the above description, it can be seen that the above embodiments of the present application achieve the following technical effects:

[0130] 1) In the obstacle detection and positioning method of the present application, the initial detection network is trained by a target training set to obtain a target detection network, wherein the target training set includes a target image set and target label information, the target image set includes an image containing a target obstacle, and the target label information includes at least an actual distance and category information. Since the target loss function includes at least a distance loss function, and the distance loss function is a formula for the difference between the actual distance and the predicted distance, the present solution trains the initial detection network so that the obtained target detection network can more accurately detect the category information of the target obstacle and predict the distance between the vehicle and the target obstacle. The target image is then detected by the target detection network. When the target image contains the target obstacle, at least the distance information between the target obstacle and the vehicle predicted by the target detection network is displayed on the corresponding target image. Compared with the prior art method of using laser radar to predict the distance information between the target obstacle and the vehicle, this solution trains the initial detection network with the target loss function to obtain the target detection network, and then uses the target detection network to detect the target image to obtain the category information of the target obstacle and the distance information between the target obstacle and the vehicle. Since this solution detects the target image, it ensures that the target obstacle information can be obtained more accurately, and the recognition ability of the target obstacle is strong, thereby ensuring that the predicted distance information between the target obstacle and the vehicle is more accurate, thereby solving the problem of the prior art that it is difficult to calculate the distance between the vehicle and the obstacle more accurately. In addition, this solution directly detects the acquired target image, that is, this solution realizes the end-to-end detection process, and there is no need to map the acquired laser point cloud information to obtain the mapped information, and then detect the mapped information, etc., which ensures that the target detection network has a small amount of calculation and a fast convergence speed.

[0131] 2) In the obstacle detection and positioning device of the present application, the training unit is used to train the initial detection network through the target training set to obtain the target detection network, wherein the above-mentioned target training set includes a target image set and target label information, the above-mentioned target image set includes an image containing a target obstacle, and the above-mentioned target label information at least includes an actual distance and category information. Since the above-mentioned target loss function at least includes a distance loss function, and the above-mentioned distance loss function is a formula for the difference between the actual distance and the predicted distance, this scheme trains the initial detection network so that the obtained target detection network can more accurately detect the category information of the target obstacle and predict the distance between the vehicle and the target obstacle. The detection unit is used to detect the target image through the target detection network. When the above-mentioned target image contains the above-mentioned target obstacle, at least the distance information between the target obstacle and the vehicle predicted by the target detection network is displayed on the corresponding target image. Compared with the prior art method based on laser radar to predict the distance information between the target obstacle and the vehicle, this solution trains the initial detection network with the target loss function to obtain the target detection network, and then uses the target detection network to detect the target image to obtain the category information of the target obstacle and the distance information between the target obstacle and the vehicle. Since this solution detects the target image, it ensures that the target obstacle information can be obtained more accurately, and the recognition ability of the target obstacle is strong, thereby ensuring that the predicted distance information between the target obstacle and the vehicle is more accurate, thereby solving the problem of the prior art that it is difficult to calculate the distance between the vehicle and the obstacle more accurately. In addition, this solution directly detects the acquired target image, that is, this solution realizes the end-to-end detection process, and there is no need to map the acquired laser point cloud information to obtain the mapped information, and then detect the mapped information, etc., which ensures that the target detection network has a small amount of calculation and a fast convergence speed.

[0132] 3) The vehicle of the present application includes the above-mentioned detection and positioning device, which is used to execute any one of the above-mentioned detection and positioning methods. In the above-mentioned detection and positioning methods, the initial detection network is trained by a target training set to obtain a target detection network, wherein the above-mentioned target training set includes a target image set and target label information, the above-mentioned target image set includes images containing target obstacles, and the above-mentioned target label information at least includes actual distance and category information. Since the above-mentioned target loss function at least includes a distance loss function, and the above-mentioned distance loss function is a formula for the difference between the actual distance and the predicted distance, the present scheme trains the initial detection network so that the obtained target detection network can more accurately detect the category information of the target obstacle and predict the distance between the vehicle and the target obstacle, and then detects the target image through the target detection network. When the above-mentioned target image contains the above-mentioned target obstacle, at least the distance information between the target obstacle and the vehicle predicted by the target detection network is displayed on the corresponding target image. Compared with the prior art method of using laser radar to predict the distance information between the target obstacle and the vehicle, this solution trains the initial detection network with the target loss function to obtain the target detection network, and then uses the target detection network to detect the target image to obtain the category information of the target obstacle and the distance information between the target obstacle and the vehicle. Since this solution detects the target image, it ensures that the target obstacle information can be obtained more accurately, and the recognition ability of the target obstacle is strong, thereby ensuring that the predicted distance information between the target obstacle and the vehicle is more accurate, thereby solving the problem of the prior art that it is difficult to calculate the distance between the vehicle and the obstacle more accurately. In addition, this solution directly detects the acquired target image, that is, this solution realizes the end-to-end detection process, and there is no need to map the acquired laser point cloud information to obtain the mapped information, and then detect the mapped information, etc., which ensures that the target detection network has a small amount of calculation and a fast convergence speed.

[0133] The above description is only the preferred embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A method for detecting and locating an obstacle, characterized in that: The detection and positioning method is applied in a vehicle, and the detection and positioning method comprises: The target training set is used to train an initial detection network having a target loss function to obtain a target detection network, wherein the target loss function at least includes a distance loss function, and the distance loss function is a formula for calculating the difference between an actual distance and a predicted distance. The target training set includes a target image set and corresponding target label information, and the target image set includes images containing target obstacles. The target label information at least includes an actual distance and category information, and the actual distance is the distance between the target obstacle and the vehicle obtained by detection, and the predicted distance is the distance between the target obstacle and the vehicle obtained by output of the initial detection network. The category information is used to characterize the category of the target obstacle. The target image is detected using the target detection network, and whether the target image contains the target obstacle is determined according to the category information of the target obstacle in the detection result. If the target image contains the target obstacle, at least the distance information between the target obstacle and the vehicle is displayed on the corresponding target image. The distance loss function includes a first distance loss function and a second distance loss function. When |r|≤α, the first distance loss function is When |r|>α, the first distance loss function is Where r = l d -f d , l d is the actual distance, f d is the predicted distance, α=1, and the second distance loss function is the ratio of |r| to the actual distance.

2. The detection and positioning method according to claim 1, characterized in that: The initial detection network includes a feature extraction network and a prediction network. The target training set is used to train the initial detection network with a target loss function to obtain a target detection network, including: Inputting the target training set into the feature extraction network to extract features and obtain multiple feature maps; Inputting the plurality of feature maps into the prediction network for prediction to obtain a plurality of prediction distances; Calculating a first target loss based at least on the plurality of predicted distances, the plurality of actual distances and the distance loss function; At least according to the first target loss, a total target loss is calculated, and according to the total target loss, the initial detection network is adjusted to obtain the target detection network.

3. The detection and positioning method according to any one of claims 1 or 2, characterized in that: The target tag information also includes position information, and the position information includes position information of a first coordinate point and position information of a second coordinate point. The first coordinate point and the second coordinate point can form a minimum rectangular frame including the target obstacle, and at least the distance information between the target obstacle and the vehicle is displayed on the corresponding target image, and further includes: The position information of the target obstacle on the target image is displayed on the corresponding target image.

4. The detection and positioning method according to claim 1, characterized in that: Before the target detection network is obtained by training the initial detection network with the target loss function using the target training set, the detection and positioning method further includes: A training set is received, and a data enhancement method is used to perform data enhancement on the training set to obtain a predetermined training set, wherein the training set includes an image set and corresponding label information, wherein the image set includes the image containing the target obstacle and images containing other obstacles, wherein the data enhancement method includes at least one of the following: translation, flipping, and random cropping, and the other obstacles are obstacles other than the target obstacle; The predetermined training set is processed in a predetermined manner to obtain the target training set.

5. The detection and positioning method according to claim 4, characterized in that: Receiving a training set, and performing data enhancement on the training set using a data enhancement method to obtain a predetermined training set, including: Using the data enhancement method, data enhancement is performed on the image containing the target obstacle, and the target label information corresponding to the image containing the target obstacle is updated to obtain a data set after data enhancement; The data set after data enhancement, the image containing the other obstacles, and the target label information corresponding to the image containing the other obstacles are combined to obtain the predetermined training set.

6. The detection and positioning method according to claim 4, characterized in that: Performing predetermined processing on the predetermined training set to obtain the target training set includes: Using a letterbox method, resizing images in a predetermined image set in the predetermined training set to obtain a first image set, wherein the predetermined image set is a collection of images in the predetermined training set; Convert the color channels of the first image set from RGB to BGR to obtain a second image set; The second image set is normalized to obtain the target image set, and the target image set and the corresponding target label information are combined to obtain the target training set.

7. The detection and positioning method according to claim 1, characterized in that: The initial detection network includes a feature extraction network, the initial detection network is constructed based on a CenterNet network, and the feature extraction network is VarGNet.

8. An obstacle detection and positioning device, characterized in that: The detection and positioning device is arranged in a vehicle, and the detection and positioning device comprises: A training unit, configured to train an initial detection network having a target loss function using a target training set to obtain a target detection network, wherein the target loss function at least includes a distance loss function, and the distance loss function is a formula for calculating a difference between an actual distance and a predicted distance. The target training set includes a target image set and corresponding target label information, and the target image set includes images containing target obstacles. The target label information at least includes an actual distance and category information, and the actual distance is a distance between the target obstacle and the vehicle obtained by detection, and the predicted distance is a distance between the target obstacle and the vehicle obtained by output of the initial detection network, and the category information is used to characterize the category of the target obstacle. a detection unit, configured to detect a target image using the target detection network, determine whether the target image contains the target obstacle according to the category information of the target obstacle in the detection result, and if the target image contains the target obstacle, display at least the distance information between the target obstacle and the vehicle on the corresponding target image, The distance loss function includes a first distance loss function and a second distance loss function. When |r|≤α, the first distance loss function is When |r|>α, the first distance loss function is Where r = l d -f d , l d is the actual distance, f d is the predicted distance, α=1, and the second distance loss function is the ratio of |r| to the actual distance.

9. A vehicle, characterized in that: include: A device for detecting and locating an obstacle, wherein the device is used to execute the detection and locating method according to any one of claims 1 to 7.

Images