Obstacle detection method and device, electronic equipment, storage medium and program product

Abstract

The invention relates to application of electronic equipment technology to the field of vehicles, in particular to an obstacle detection method and device, electronic equipment, a storage medium and a program product. The method comprises the steps of obtaining an environment image of the surrounding environment of the electronic equipment; according to the environment image and a pre-trained detection model, determining position information of an obstacle in the surrounding environment; the detection model is used for obtaining aerial view features according to the environment image and determining position information of obstacles in the surrounding environment according to the aerial view features. And the position information of the obstacle under the coordinates of the electronic equipment can be obtained by inputting the environment image into the detection model. Thus, the workload of coordinate conversion is effectively reduced, the obstacle detection process is simplified, the accuracy of obstacle detection is improved due to the fact that coordinate conversion is not needed, and then the obstacle detection efficiency and detection accuracy are improved.

Description

Technical Field

[0001] The present disclosure relates to the application of electronic equipment technology in the field of vehicles, and in particular to an obstacle detection method, device, electronic equipment, storage medium and program product. Background Art

[0002] With the development of intelligent vehicles, intelligent driving has become an indispensable technical means for intelligent vehicles. Intelligent vehicles are usually equipped with cameras. The images collected by the cameras are used to determine the location of obstacles that may affect the vehicle's movement, thereby achieving precise obstacle avoidance for intelligent vehicles. Summary of the Invention

[0003] To overcome the problems existing in the related art, the present disclosure provides an obstacle detection method, device, electronic device, storage medium and program product.

[0004] According to a first aspect of an embodiment of the present disclosure, there is provided an obstacle detection method, comprising: Acquiring an environmental image of the environment surrounding the electronic device; Determining location information of obstacles in the surrounding environment based on the environmental image and a pre-trained detection model; The detection model is used to obtain bird's-eye view features according to the environment image, and determine the position information of obstacles in the surrounding environment according to the bird's-eye view features.

[0005] In this embodiment, the detection model can derive bird's-eye view features from the environmental image and, based on these features, determine the location information of obstacles in the surrounding environment. Thus, simply inputting the environmental image into the detection model can obtain the location information of obstacles in the electronic device's coordinates. This effectively reduces the workload of coordinate conversion, simplifies the obstacle detection process, and, since no coordinate conversion is required, improves the accuracy of obstacle detection, thereby enhancing both the efficiency and precision of obstacle detection.

[0006] In some possible implementations, the detection model is obtained by: Inputting the sample environment image into the initial detection model to obtain sample position information of the sample obstacle in the sample environment image; Determining a current error of the initial detection model based on the sample position information and the marked position information of the sample obstacles; According to the current error of the initial detection model, the current parameters of the initial detection model are adjusted, and the step of inputting the sample environment image into the initial detection model to obtain the sample position information of the sample obstacle in the sample environment image is returned until the detection model is obtained when the training termination condition is met.

[0007] In this way, during the training process, the error of the initial detection model can be determined based on the sample position information output by the initial detection model and the pre-labeled annotation position information, and the current parameters of the initial detection model can be adjusted based on the error, thereby improving the model training effect.

[0008] In some possible embodiments, the position information of the obstacle includes the position information of a preset number of points on the obstacle; the detection model is obtained by training the network model by using the sample environment image as the model input parameter and the position information of a preset number of points on the sample obstacle included in the sample environment image as the model output parameter.

[0009] In this embodiment, since the position information of a point on an obstacle is detected, it is not restricted by the shape of the obstacle and can detect obstacles of various shapes, thereby improving the versatility and robustness of obstacle detection.

[0010] In some possible implementations, the position information of a preset number of points on the sample obstacle is obtained by discretizing the position information of the sample obstacle.

[0011] In this embodiment, the position information of the marked sample obstacles is discretized to obtain the position information of a preset number of points on the sample obstacles, thereby improving the accuracy of the obtained position information of the preset number of points on the sample obstacles. When the model is subsequently trained based on the position information of the preset number of points on the sample obstacles, the accuracy of the model training is improved.

[0012] In some possible implementations, the position information of the obstacle includes position information of a preset number of points on the obstacle, and the method further includes: Determining the outline information of the obstacle; The obstacle is drawn according to the contour information and the position information of the preset number of points.

[0013] In this implementation, after determining the position information of a preset number of points on the obstacle, the obstacle is drawn based on the outline information and the position information of the preset number of points. This ensures that the drawn obstacle is consistent with the real obstacle, effectively avoiding false detections, improving the accuracy of the drawn obstacle, and enhancing the user experience.

[0014] In some possible implementations, the obstacle includes a wheel chock and / or a speed bump, and the method further includes: Obtaining location information of a boundary line of an area where the obstacle is located; Determining the direction information of the obstacle according to the position information of the boundary line; Drawing the obstacle according to the contour information and the position information of the preset number of points includes: The obstacle is drawn according to the outline information, the position information of the preset number of points, and the direction information.

[0015] In this embodiment, the direction information of the obstacle is determined based on the position information of the boundary line of the area where the obstacle is located, and then the obstacle can be drawn based on the contour information, the position information and the direction information of a preset number of points. In this way, the recall rate of obstacle detection can be improved, the probability of false detection can be avoided, and the accuracy of the drawn obstacles can be further improved.

[0016] In some possible implementations, when the obstacle is a wheel chock set in a storage location, the boundary line of the area where the obstacle is located is the boundary line of the storage location; when the obstacle is a speed bump, the boundary line of the area where the obstacle is located is the lane line of the road where the vehicle is currently located.

[0017] In this embodiment, the direction information of the obstacle can be determined based on the position information of the boundary line of the area where the obstacle is located, which simplifies the operation of determining the direction of the obstacle. Since the position of the boundary line of the area where the obstacle is located is accurate, the accuracy and reliability of the determined direction of the obstacle are improved.

[0018] In some possible implementations, when there are multiple environment images, the detection model includes a bird's-eye view feature generation sub-model and a detection sub-model; The bird's-eye view feature generation sub-model is used to extract features of the environment image for each environment image, fuse the features of the multiple environment images to obtain a fused feature, and obtain a bird's-eye view feature based on the fused feature; The detection sub-model is used to determine the position information of obstacles in the surrounding environment according to the bird's-eye view features.

[0019] In this embodiment, the position information of obstacles in the coordinate system of the electronic device can be obtained through the detection model, and the output result is more direct. The output obstacle position information can be directly applied to downstream businesses without the need for excessive post-processing operations, thereby reducing the complexity of obstacle detection and improving the reliability of obstacle detection.

[0020] In some possible implementations, there are multiple environmental images, and acquiring the environmental image of the environment surrounding the electronic device includes: Multiple environmental images of the environment surrounding the electronic device are acquired by installing multiple image acquisition devices on the electronic device; wherein the multiple image acquisition devices are installed at different positions of the electronic device to acquire environmental images from different perspectives.

[0021] In this embodiment, multiple environmental images are captured by multiple image acquisition devices installed on the electronic device, thereby obtaining a more comprehensive environmental image. This facilitates improving the reliability of the bird's-eye view features generated from these environmental images. Furthermore, the multiple environmental images are from different perspectives, further improving the reliability of the generated bird's-eye view features and enhancing the comprehensiveness of obstacle detection.

[0022] In some possible implementations, the multiple image acquisition devices synchronously acquire the multiple environment images.

[0023] In this way, it can be ensured that the multiple environmental images captured by multiple image acquisition devices are images from different perspectives at the same time. When the bird's-eye view features are subsequently generated based on the multiple environmental images, the reliability of the generated bird's-eye view features is further improved, thereby improving the comprehensiveness of obstacle detection.

[0024] According to a second aspect of an embodiment of the present disclosure, there is provided an obstacle detection device, the device comprising: A first acquisition module is configured to acquire an environmental image of the surrounding environment of the electronic device; A first determination module is configured to determine position information of obstacles in the surrounding environment based on the environment image and a pre-trained detection model; The detection model is used to obtain bird's-eye view features according to the environment image, and determine the position information of obstacles in the surrounding environment according to the bird's-eye view features.

[0025] In some possible implementations, the detection model is obtained by: Inputting the sample environment image into the initial detection model to obtain sample position information of the sample obstacle in the sample environment image; Determining a current error of the initial detection model based on the sample position information and the marked position information of the sample obstacles; According to the current error of the initial detection model, the current parameters of the initial detection model are adjusted, and the step of inputting the sample environment image into the initial detection model to obtain the sample position information of the sample obstacle in the sample environment image is returned until the detection model is obtained when the training termination condition is met.

[0026] In some possible embodiments, the position information of the obstacle includes the position information of a preset number of points on the obstacle; the detection model is obtained by training the network model by using the sample environment image as the model input parameter and the position information of a preset number of points on the sample obstacle included in the sample environment image as the model output parameter.

[0027] According to a third aspect of an embodiment of the present disclosure, there is provided an electronic device, including: processor; a memory for storing processor-executable instructions; The processor is configured to execute the instructions to implement the steps of the obstacle detection method described in the first aspect of the embodiment of the present disclosure.

[0028] According to a fourth aspect of an embodiment of the present disclosure, a computer-readable storage medium is provided, on which computer program instructions are stored. When the program instructions are executed by a processor, the steps of the obstacle detection method described in the first aspect of the embodiment of the present disclosure are implemented.

[0029] According to a fifth aspect of an embodiment of the present disclosure, a computer program product is provided, comprising a computer program, which, when executed by a processor, implements the steps of the obstacle detection method described in the first aspect of the embodiment of the present disclosure.

[0030] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0032] Figure 1 The figure is a flowchart of an obstacle detection method according to an exemplary embodiment.

[0033] Figure 2 The figure is a schematic diagram of an obstacle detection scenario according to an exemplary embodiment.

[0034] Figure 3 FIG. 4 is a schematic diagram of another obstacle detection scenario according to an exemplary embodiment.

[0035] Figure 4 FIG. 4 is a schematic diagram of another obstacle detection scenario according to an exemplary embodiment.

[0036] Figure 5 FIG. 4 is a schematic diagram of another obstacle detection scenario according to an exemplary embodiment.

[0037] Figure 6 is a schematic diagram showing an obstacle detection method according to an exemplary embodiment.

[0038] Figure 7 The figure is a block diagram of an obstacle detection device according to an exemplary embodiment.

[0039] Figure 8 is a block diagram of a vehicle according to an exemplary embodiment.

[0040] Figure 9 It is a block diagram of an electronic device according to an exemplary embodiment. DETAILED DESCRIPTION

[0041] Exemplary embodiments will be described in detail herein, with examples illustrated in the accompanying drawings. In the following description, when referring to the drawings, identical numerals in different figures represent identical or similar elements, unless otherwise indicated. The embodiments described in the following exemplary embodiments are not intended to represent all possible embodiments consistent with the present disclosure. Rather, they are merely examples of apparatus and methods consistent with certain aspects of the present disclosure, as detailed in the appended claims.

[0042] It should be noted that all actions of acquiring signals, information or data in the present disclosure are carried out in compliance with the corresponding data protection laws and policies of the country where they are located and with the authorization given by the owner of the corresponding device.

[0043] However, currently, obstacle detection is mostly performed in two-dimensional image space. After detecting the obstacle's position, it must be further converted to the vehicle coordinate system. On the one hand, the complex coordinate transformation required to convert the obstacle's position from the two-dimensional image to the vehicle coordinate system significantly increases the computational effort. On the other hand, because detection methods in two-dimensional image space typically only provide limited information, it is difficult to determine the obstacle's true location. This results in low obstacle detection efficiency and accuracy.

[0044] In order to improve the efficiency of obstacle detection, the present disclosure provides an obstacle detection method, device, electronic device, storage medium and program product.

[0045] First, the method provided by this disclosure can be implemented by the obstacle detection device provided by this disclosure. This device can be deployed in a vehicle, including, but not limited to, any type of vehicle, such as smart cars and new energy vehicles. Furthermore, this device can also be deployed in robots, such as sweeping robots and delivery robots.

[0046] Secondly, the method provided by this disclosure can be used to detect obstacles of any shape, for example, low obstacles or tall obstacles. Obstacles can include static obstacles or dynamic obstacles. Static obstacles can be, for example, wheel chocks installed in storage areas and / or speed bumps installed on roads.

[0047] Figure 1 FIG. 1 is a flow chart showing an obstacle detection method according to an exemplary embodiment. Figure 1 As shown, the obstacle detection method may include the following steps.

[0048] In step S11 , an environmental image of the surrounding environment of the electronic device is acquired.

[0049] For example, the electronic device may be a vehicle or a robot, etc., which is not limited in the present disclosure.

[0050] In the present disclosure, an image capture device can be used to capture an image of the environment surrounding an electronic device. The image capture device can be integrated with the electronic device or separately installed. For example, if the electronic device is a vehicle, the image capture device can be installed on the vehicle or on other equipment in the area where the vehicle is located, such as roadside equipment (streetlights, traffic lights, etc.).

[0051] In one embodiment, acquiring an environmental image of the surrounding environment of the electronic device may include: acquiring the environmental image of the surrounding environment by an image acquisition device installed on the electronic device.

[0052] An image capturing device, such as a camera, is used to capture the surroundings of the electronic device to obtain an image of the environment including obstacles. The image capturing device may be one or more. Considering the limited viewing angle of a single image capturing device, it is difficult to capture a full 360-degree range of the surrounding environment. Therefore, in another embodiment, the environmental images may be multiple. Acquiring the environmental images of the surroundings of the electronic device may include: acquiring multiple environmental images of the surroundings of the electronic device using multiple image capturing devices installed on the electronic device.

[0053] In this embodiment, multiple environmental images are acquired by multiple image acquisition devices installed on the electronic device, thereby being able to acquire a more comprehensive environmental image, thereby facilitating the subsequent improvement of the reliability of the generated bird's-eye view features when generating bird's-eye view features based on the environmental images.

[0054] In addition, multiple image acquisition devices are installed at different positions of the electronic device to acquire environmental images from different perspectives.

[0055] For example, assuming the image acquisition device is a camera, multiple cameras can be set at different positions of the vehicle to capture the environment from different perspectives. For example, the multiple cameras can face different directions to capture the environment within corresponding perspective ranges in different directions.

[0056] In this way, the multiple environmental images are environmental images from different perspectives. When bird's-eye view features are subsequently generated based on the multiple environmental images, the reliability of the generated bird's-eye view features is further improved, and the comprehensiveness of obstacle detection is improved.

[0057] In addition, multiple image acquisition devices synchronously acquire multiple environment images.

[0058] In this way, it can be ensured that the multiple environmental images captured by multiple image acquisition devices are images from different perspectives at the same time. When the bird's-eye view features are subsequently generated based on the multiple environmental images, the reliability of the generated bird's-eye view features is further improved, thereby improving the comprehensiveness of obstacle detection.

[0059] In step S12, the location information of obstacles in the surrounding environment is determined based on the environment image and the pre-trained detection model.

[0060] The detection model is used to obtain bird's-eye view features based on the environment image, and determine the location information of obstacles in the surrounding environment based on the bird's-eye view features.

[0061] For example, after acquiring the environment image, the environment image is input into a pre-trained detection model. The detection model obtains a bird's-eye view feature according to the input environment image, and determines the location information of obstacles in the surrounding environment based on the bird's-eye view feature.

[0062] Among them, BEV (Bird's Eye View) can represent the image seen when looking down at a scene from the air, which is more realistic than a plane view. In the present disclosure, the bird's eye view can represent the three-dimensional space corresponding to the surrounding environment of the electronic device. The BEV space can include multiple elevation ranges when looking down from above. Therefore, in the present disclosure, the detection model determines the position information of the obstacle based on the characteristics of the bird's eye view, so that the determined position information of the obstacle is based on the position in the coordinates of the electronic device, and there is no need to convert the position information of the obstacle into the position in the coordinates of the electronic device through coordinate conversion.

[0063] In addition, the bird's-eye view perception solution can maintain uniform scales under different perspectives, ensuring more stable perception results.

[0064] Using this technical solution, the detection model can extract bird's-eye view features from the environmental image and use them to determine the location of obstacles in the surrounding environment. By inputting the environmental image into the detection model, the obstacle's location information, expressed in the coordinates of the electronic device, can be obtained. This effectively reduces the workload of coordinate conversion, simplifies the obstacle detection process, and, since coordinate conversion is not required, improves the accuracy of obstacle detection, thereby enhancing both the efficiency and precision of obstacle detection.

[0065] In some embodiments, the obstacles may include wheel chocks and / or speed bumps mounted on the ground.

[0066] In some implementations of this embodiment, the barrier comprises a ground-mounted wheel chock.

[0067] For example, in an automatic parking scenario, the obstacle detection method provided by the present disclosure can be used to detect the position of the wheel chock in the parking space, and then parking can be performed according to the position of the wheel chock, thereby improving the accuracy and standardization of automatic parking of the vehicle and enhancing the user experience.

[0068] Figure 2 FIG. 1 is a schematic diagram of an obstacle detection scenario according to an exemplary embodiment. Figure 2 As shown, two wheel chocks 202 are provided in the parking space 201. During the automatic parking process, the position of the wheel chocks can be detected, thereby achieving accurate and standardized parking.

[0069] For example, when a vehicle is autonomously cruising in a parking lot, there may be wheel chocks on the road. By detecting the location of the wheel chocks on the road using the obstacle detection method provided by this disclosure, the vehicle's direction of travel can be adjusted to avoid the wheel chocks, ensuring smooth driving and avoiding bumps for the user.

[0070] Figure 3 FIG. 1 is a schematic diagram of another obstacle detection scenario according to an exemplary embodiment. Figure 3 As shown, a part of the wheel chock 202 is arranged on the road where the vehicle is traveling. During autonomous cruising, the vehicle can detect the position of the wheel chock 202 on the road, and then adjust the driving direction of the vehicle so that the vehicle avoids the wheel chock, ensuring the smooth travel of the vehicle.

[0071] In other implementations of this embodiment, the obstacle includes a speed bump installed on the ground.

[0072] For example, in a vehicle autonomous cruising scenario, the obstacle detection method provided by the present disclosure can be used to detect the location of speed bumps on the road, and then the vehicle driving can be controlled according to the location of the speed bumps, thereby improving driving safety.

[0073] Figure 4 FIG. 1 is a schematic diagram of another obstacle detection scenario according to an exemplary embodiment. Figure 4 As shown, when the vehicle is traveling on the road, the position of the speed bump 401 on the road can be detected, and the vehicle can then slow down when it reaches the position of the speed bump 401 to ensure smooth driving of the vehicle.

[0074] Figure 5 FIG. 1 is a schematic diagram of another obstacle detection scenario according to an exemplary embodiment. Figure 5 As shown, when the vehicle leaves the storage location 201, since the speed of leaving the storage location 201 is usually relatively low, when it detects that there is a speed bump 401 in front of it, it can accelerate when it reaches the position of the speed bump 401, so that the vehicle can pass through the speed bump 401 smoothly.

[0075] By adopting the above technical solution, the position of the wheel chocks and / or the speed bumps installed on the ground can be detected, thereby improving the accuracy and standardization of the vehicle's automatic parking and enhancing driving safety.

[0076] The detection model is described below.

[0077] In some embodiments, the detection model is obtained by: Inputting the sample environment image into the initial detection model to obtain sample position information of the sample obstacle in the sample environment image; Determining a current error of the initial detection model based on the sample position information and the marked position information of the sample obstacles; According to the current error of the initial detection model, the current parameters of the initial detection model are adjusted, and the step of inputting the sample environment image into the initial detection model to obtain the sample position information of the sample obstacle in the sample environment image is returned until the detection model is obtained when the training termination condition is met.

[0078] The detection model may include a bird's-eye view feature generation sub-model and a detection sub-model. The feature generation sub-model is used to extract features of the environment image and obtain bird's-eye view features based on the features; the detection sub-model is used to determine the location information of obstacles in the surrounding environment based on the bird's-eye view features.

[0079] When there are multiple environmental images, the bird's-eye view feature generation sub-model is used to extract the features of the environmental image for each environmental image, fuse the features of multiple environmental images to obtain fused features, and obtain the bird's-eye view features based on the fused features; the detection sub-model is used to determine the location information of obstacles in the surrounding environment based on the bird's-eye view features.

[0080] For example, the bird's-eye view feature generation sub-model may include an image backbone network module, a feature fusion module, and a perspective-to-bird's-eye view module. Multiple environmental images are input into the image backbone network module, and features of each environmental image are extracted through the image backbone network. The image backbone network may be a two-dimensional (2D) image model, which is not limited in this disclosure.

[0081] The feature fusion module is connected to the image backbone network module and is used to fuse the features of each environment image output by the image backbone network module to obtain richer fusion features. Among them, the fusion can be performed using pyramid feature fusion, splicing fusion, etc., which is not limited by this disclosure.

[0082] The perspective-to-bird's-eye view module is connected to the feature fusion module to obtain the fusion features output by the feature fusion module and obtain the bird's-eye view features based on the fusion feature conversion.

[0083] The detection sub-model may include a detection head network module. The detection head network module is used to obtain the bird's-eye view features output by the perspective-to-bird's-eye view module and determine the location information of the obstacle based on the bird's-eye view features.

[0084] In addition, during the model training stage, the detection model can also include an error detection module. During the training process, the error detection module inputs the sample position information of the sample obstacles output by the detection head network module and the labeled position information of the pre-labeled sample obstacles into the preset loss function, calculates the loss value, and uses the standard gradient descent method based on the loss value to update the current parameters of the model, thereby achieving the effect of training the model and improving the model training effect.

[0085] Figure 6 FIG. 1 is a schematic diagram showing an obstacle detection method according to an exemplary embodiment. Figure 6 As shown, assuming the electronic device is a vehicle, the multiple environmental images include the vehicle's left-view environmental image, the vehicle's front-view environmental image, the vehicle's right-view environmental image, and the vehicle's rear-view environmental image. First, the left-view environmental image, the front-view environmental image, the right-view environmental image, and the rear-view environmental image are each input into the image backbone network module to obtain the features of each environmental image. Next, the features of each environmental image are input into the feature fusion module to obtain fused features. The fused features are then input into the perspective-to-bird's-eye view module to obtain bird's-eye view features. Finally, the bird's-eye view features are input into the detection head network module to obtain obstacle location information.

[0086] By adopting the above technical solution, the position information of obstacles in the coordinate system of the electronic device can be obtained through the detection model. The output result is more direct. The output obstacle position information can be directly applied to downstream businesses without excessive post-processing operations, reducing the complexity of obstacle detection and improving the reliability of obstacle detection.

[0087] Currently, obstacle location information is typically represented using target rectangles or line segments. However, this approach has significant shortcomings when dealing with obstacles with varying shapes. Because it requires specific solutions for different obstacles, its versatility is limited and robustness is difficult to guarantee. Furthermore, this representation method is susceptible to interference when viewing from a poor viewing angle or with significant jitter, compromising detection accuracy and stability.

[0088] Therefore, in some embodiments, to improve the versatility and stability of obstacle detection, the obstacle location information output by the detection model may include the location information of a preset number of points on the obstacle. Accordingly, the detection model is trained by using a sample environment image as a model input parameter and the location information of a preset number of points on a sample obstacle included in the sample environment image as a model output parameter. The preset number can be a custom value, which is not limited by this disclosure.

[0089] During the training phase, the sample environment image is used as the model input parameter, and the position information of a preset number of points on the sample obstacle is used as the model output parameter for training. In this way, after the training is completed, the detection model can determine the position information of a preset number of points on the obstacle based on the environment image.

[0090] With the above technical solution, since the position information of points on the obstacle is detected, it is not restricted by the shape of the obstacle and can detect obstacles of various shapes, thereby improving the versatility and robustness of obstacle detection.

[0091] In some embodiments, the position information of a preset number of points on the sample obstacle is obtained by discretizing the position information of the sample obstacle.

[0092] For example, after the position information of the sample obstacle is marked, the position information of the sample obstacle is discretized to obtain the position information of a preset number of points on the sample obstacle.

[0093] By adopting the above scheme, the position information of the marked sample obstacles is discretized to obtain the position information of a preset number of points on the sample obstacles, thereby improving the accuracy of the obtained position information of the preset number of points on the sample obstacles. When the model is subsequently trained based on the position information of the preset number of points on the sample obstacles, the accuracy of the model training is improved.

[0094] After determining the location of an obstacle in the manner described above, the obstacle's location information can be directly transmitted to downstream services within the electronic device. Furthermore, to further improve the accuracy of the determined obstacle location, the obstacle can be rendered based on prior knowledge of the obstacle. The rendered obstacle can then be displayed as needed.

[0095] In some embodiments, the position information of the obstacle includes position information of a preset number of points on the obstacle, and the method further includes: Determine the outline information of the obstacle; Draw obstacles based on the outline information and the position information of a preset number of points.

[0096] For example, assuming the electronic device is a vehicle, and the vehicle is operating in an autonomous parking or cruising scenario, the obstacle type is determined to be a wheel chock or speed bump, and the obstacle's outline is determined to be a straight line. The position information of a preset number of points is then clustered based on the outline information and fitted into a straight line. The fitted line is used to represent the wheel chock or speed bump.

[0097] As another example, the type of obstacle can also be detected by a neural network model. After determining the type of obstacle, the outline information of the obstacle is determined, and then cluster fitting is performed on the position information of a preset number of points based on the outline information to map the obstacle.

[0098] As another example, the neural network model can be used to directly determine the contour information of the obstacle, and then cluster fitting is performed on the position information of a preset number of points based on the contour information to map out the obstacle.

[0099] After determining the location information of a preset number of points on the obstacle, the obstacle is drawn based on the outline information and the location information of the preset number of points. This ensures that the drawn obstacle is consistent with the real obstacle, effectively avoiding false detections, improving the accuracy of the drawn obstacle, and enhancing the user experience.

[0100] In some implementations, when drawing an obstacle, the direction information of the obstacle on the ground may also be obtained to further improve the accuracy of the drawn obstacle.

[0101] In this embodiment, the obstacle comprises a wheel chock and / or a speed bump, and the method further comprises: Obtain the location information of the boundary line of the area where the obstacle is located; Determining the direction information of the obstacle according to the position information of the boundary line; Drawing the obstacle according to the contour information and the position information of the preset number of points includes: Obstacles are drawn according to the contour information, the position information of the preset number of points, and the direction information.

[0102] By adopting the above technical solution, the direction information of the obstacle is determined based on the position information of the boundary line of the area where the obstacle is located. Then, the obstacle can be drawn based on the contour information, the position information and the direction information of a preset number of points. In this way, the recall rate of obstacle detection can be improved, the probability of false detection can be avoided, and the accuracy of the drawn obstacles can be further improved.

[0103] In some possible embodiments, the obstacle is a wheel chock disposed in the storage location, and the boundary line of the area where the obstacle is located is the boundary line of the storage location. For example, the boundary lines of the storage location may include an entrance line, left and right dividing lines, and a bottom boundary line. Typically, the wheel chock disposed in the storage location is a straight line that is perpendicular to the left and right dividing lines and / or parallel to the entrance line or the bottom boundary line. Therefore, the direction of the entrance line or the bottom boundary line is determined as the direction of the wheel chock, or the direction perpendicular to the left and right dividing lines is determined as the direction of the wheel chock.

[0104] In other possible implementations, the obstacle is a speed bump, and the boundary of the obstacle area is the lane line of the road the vehicle is currently on. Speed ​​bumps are typically located on the road and perpendicular to the lane line. Therefore, in this implementation, the direction perpendicular to the lane line can be determined as the direction of the obstacle.

[0105] By adopting the above technical solution, the direction information of the obstacle can be determined based on the position information of the boundary line of the area where the obstacle is located, which simplifies the operation of determining the direction of the obstacle. Since the position of the boundary line of the area where the obstacle is located is accurate, the accuracy and reliability of the determined direction of the obstacle are improved.

[0106] Based on the same inventive concept, the present disclosure also provides an obstacle detection device, which is used to implement the steps of the obstacle detection method described in the present disclosure. For example, Figure 7 FIG. 1 is a block diagram of an obstacle detection device according to an exemplary embodiment. Figure 7 As shown, the obstacle detection device 700 may include: A first acquisition module 701 is configured to acquire an environmental image of the surrounding environment of the electronic device; A first determination module 702 is configured to determine location information of obstacles in the surrounding environment based on the environment image and a pre-trained detection model; The detection model is used to obtain bird's-eye view features according to the environment image, and determine the position information of obstacles in the surrounding environment according to the bird's-eye view features.

[0107] Optionally, the detection model is obtained by: Inputting the sample environment image into the initial detection model to obtain sample position information of the sample obstacle in the sample environment image; Determining a current error of the initial detection model based on the sample position information and the marked position information of the sample obstacles; According to the current error of the initial detection model, the current parameters of the initial detection model are adjusted, and the step of inputting the sample environment image into the initial detection model to obtain the sample position information of the sample obstacle in the sample environment image is returned until the detection model is obtained when the training termination condition is met.

[0108] Optionally, the location information of the obstacle includes the location information of a preset number of points on the obstacle; the detection model is obtained by training the network model by using the sample environment image as the model input parameter and the location information of a preset number of points on the sample obstacle included in the sample environment image as the model output parameter.

[0109] Optionally, the position information of a preset number of points on the sample obstacles is obtained by discretizing the position information of the sample obstacles.

[0110] Optionally, the position information of the obstacle includes position information of a preset number of points on the obstacle, and the obstacle detection device 700 may further include: A second determining module is configured to determine the outline information of the obstacle; The drawing module is configured to draw the obstacle according to the outline information and the position information of the preset number of points.

[0111] Optionally, the obstacle includes a wheel chock and / or a speed bump, and the obstacle detection device 700 may further include: A second acquisition module is configured to acquire position information of a boundary line of the area where the obstacle is located; a third determining module, configured to determine direction information of the obstacle according to the position information of the boundary line; The drawing module is configured to draw the obstacle according to the outline information, the position information of the preset number of points, and the direction information.

[0112] Optionally, when the obstacle is a wheel chock set in a storage location, the boundary line of the area where the obstacle is located is the boundary line of the storage location; when the obstacle is a speed bump, the boundary line of the area where the obstacle is located is the lane line of the road where the vehicle is currently located.

[0113] Optionally, when there are multiple environment images, the detection model includes a bird's-eye view feature generation sub-model and a detection sub-model; The bird's-eye view feature generation sub-model is used to extract features of the environment image for each environment image, fuse the features of the multiple environment images to obtain a fused feature, and obtain a bird's-eye view feature based on the fused feature; The detection sub-model is used to determine the position information of obstacles in the surrounding environment according to the bird's-eye view features.

[0114] Optionally, there are multiple environmental images, and the first acquisition module 701 is configured to: acquire multiple environmental images of the environment around the electronic device through multiple image acquisition devices installed on the electronic device; wherein the multiple image acquisition devices are installed at different positions of the electronic device to capture environmental images from different perspectives.

[0115] Optionally, the multiple image acquisition devices synchronously acquire the multiple environmental images.

[0116] Regarding the apparatus in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0117] The present disclosure also provides a computer-readable storage medium having computer program instructions stored thereon, which implement the steps of the obstacle detection method provided by the present disclosure when the program instructions are executed by a processor.

[0118] Figure 8 6 is a block diagram illustrating a vehicle according to an exemplary embodiment. For example, vehicle 600 may be a hybrid vehicle, a non-hybrid vehicle, an electric vehicle, a fuel cell vehicle, or another type of vehicle. Vehicle 600 may be an autonomous vehicle, a semi-autonomous vehicle, or a non-autonomous vehicle.

[0119] Reference Figure 8 Vehicle 600 may include various subsystems, such as an infotainment system 610, a perception system 620, a decision control system 630, a drive system 640, and a computing platform 650. Vehicle 600 may also include more or fewer subsystems, and each subsystem may include multiple components. Furthermore, each subsystem and each component of vehicle 600 may be interconnected via wired or wireless means.

[0120] In some embodiments, the infotainment system 610 may include a communication system, an entertainment system, a navigation system, and the like.

[0121] The perception system 620 may include several sensors for sensing information about the environment surrounding the vehicle 600. For example, the perception system 620 may include a global positioning system (which may be a GPS system, a BeiDou system, or another positioning system), an inertial measurement unit (IMU), a laser radar, a millimeter-wave radar, an ultrasonic radar, and a camera.

[0122] The decision control system 630 may include a computing system, a vehicle controller, a steering system, a throttle, and a braking system.

[0123] The drive system 640 may include components that provide power to the vehicle 600. In one embodiment, the drive system 640 may include an engine, an energy source, a transmission system, and wheels. The engine may be an internal combustion engine, an electric motor, an air compression engine, or a combination thereof. The engine is capable of converting energy provided by the energy source into mechanical energy.

[0124] Some or all functions of the vehicle 600 are controlled by a computing platform 650. The computing platform 650 may include at least one processor 651 and a memory 652. The processor 651 may execute instructions 653 stored in the memory 652.

[0125] The processor 651 can be any conventional processor, such as a commercially available CPU. The processor can also include a graphics processor (GPU), a field programmable gate array (FPGA), a system on chip (SOC), an application specific integrated circuit (ASIC), or a combination thereof.

[0126] The memory 652 may be implemented by any type of volatile or non-volatile memory device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk, or optical disk.

[0127] In addition to instructions 653 , memory 652 may also store data, such as road maps, route information, and vehicle location, direction, speed, etc. The data stored in memory 652 may be used by computing platform 650 .

[0128] In the embodiment of the present disclosure, the processor 651 may execute the instruction 653 to complete all or part of the steps of the above-mentioned obstacle detection method.

[0129] Figure 9 8 is a block diagram of an electronic device according to an exemplary embodiment. For example, the device 800 may be a mobile phone, a robot, a computer, a digital broadcast terminal, a messaging device, a game console, a tablet device, a medical device, a fitness device, a personal digital assistant, etc.

[0130] Reference Figure 9 , the apparatus 800 may include one or more of the following components: a processing component 802 , a memory 804 , a power component 806 , a multimedia component 808 , an audio component 810 , an input / output interface 812 , a sensor component 814 , and a communication component 816 .

[0131] Processing component 802 generally controls the overall operation of device 800, such as operations associated with display, phone calls, data communications, camera operation, and recording. Processing component 802 may include one or more processors 820 to execute instructions to perform all or part of the steps of the obstacle detection method described above. Furthermore, processing component 802 may include one or more modules to facilitate interaction between processing component 802 and other components. For example, processing component 802 may include a multimedia module to facilitate interaction between multimedia component 808 and processing component 802.

[0132] The memory 804 is configured to store various types of data to support the operations of the device 800. Examples of such data include instructions for any application or method operating on the device 800, contact data, phone book data, messages, pictures, videos, etc. The memory 804 can be implemented by any type of volatile or non-volatile storage device, or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk, or optical disk.

[0133] The power supply component 806 provides power to the various components of the device 800. The power supply component 806 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the device 800.

[0134] The multimedia component 808 includes a screen that provides an output interface between the device 800 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touches, slides, and gestures on the touch panel. The touch sensors can not only sense the boundaries of a touch or slide action, but also detect the duration and pressure associated with the touch or slide action. In some embodiments, the multimedia component 808 includes a front-facing camera and / or a rear-facing camera. When the device 800 is in an operating mode, such as a capture mode or a video mode, the front-facing camera and / or the rear-facing camera can receive external multimedia data. Each front-facing camera and the rear-facing camera can have a fixed optical lens system or have focal length and optical zoom capabilities.

[0135] The audio component 810 is configured to output and / or input audio signals. For example, the audio component 810 includes a microphone (MIC) that is configured to receive external audio signals when the device 800 is in an operating mode, such as a call mode, a recording mode, and a voice recognition mode. The received audio signals may be further stored in the memory 804 or transmitted via the communication component 816. In some embodiments, the audio component 810 also includes a speaker for outputting audio signals.

[0136] The input / output interface 812 provides an interface between the processing component 802 and peripheral interface modules, such as a keyboard, a click wheel, buttons, etc. These buttons may include but are not limited to: a home button, a volume button, a start button, and a lock button.

[0137] The sensor assembly 814 includes one or more sensors for providing various aspects of the status assessment of the device 800. For example, the sensor assembly 814 can detect the open / closed state of the device 800, the relative positioning of components, such as the display and keypad of the device 800. The sensor assembly 814 can also detect changes in the position of the device 800 or a component of the device 800, the presence or absence of user contact with the device 800, the orientation or acceleration / deceleration of the device 800, and temperature changes of the device 800. The sensor assembly 814 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. The sensor assembly 814 may also include an optical sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, the sensor assembly 814 may also include an accelerometer, a gyroscope, a magnetic sensor, a pressure sensor, or a temperature sensor.

[0138] The communication component 816 is configured to facilitate wired or wireless communication between the apparatus 800 and other devices. The apparatus 800 can access a wireless network based on a communication standard, such as WiFi, 2G or 3G, or a combination thereof. In an exemplary embodiment, the communication component 816 receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 816 also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology, and other technologies.

[0139] In an exemplary embodiment, the apparatus 800 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the above-described obstacle detection method.

[0140] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as memory 804 including instructions. The instructions are executable by processor 820 of apparatus 800 to implement the above-described obstacle detection method. For example, the non-transitory computer-readable storage medium may be a ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, optical data storage device, or the like.

[0141] In another exemplary embodiment, a computer program product is further provided. The computer program product includes a computer program executable by a programmable device, and has a code portion for performing the above obstacle detection method when executed by the programmable device.

[0142] Furthermore, the word "exemplary" is used herein to mean serving as an example, instance, or illustration. Any aspect or design described herein as "exemplary" is not necessarily to be construed as advantageous over other aspects or designs. Rather, the use of the word exemplary is intended to present concepts in a concrete manner. As used herein, the term "or" is intended to mean an inclusive "or" rather than an exclusive "or." That is, unless otherwise specified or clear from the context, "X applies to A or B" is intended to mean any of the natural inclusive permutations. That is, if X applies to A; X applies to B; or X applies to both A and B, then "X applies to A or B" satisfies any of the aforementioned instances. Furthermore, the articles "a" and "an," as used in this application and the appended claims, are generally understood to mean "one or more," unless otherwise specified or clear from the context to refer to the singular form.

[0143] Likewise, although the present disclosure has been shown and described with respect to one or more implementations, equivalent variations and modifications will occur to those skilled in the art upon reading and understanding this specification and the accompanying drawings. The present disclosure includes all such modifications and variations and is limited only by the scope of the claims. With particular regard to the various functions performed by the components described above (e.g., elements, resources, etc.), unless otherwise indicated, terms used to describe such components are intended to correspond to any component (functionally equivalent) that performs the specific function of the described component, even if not structurally equivalent to the disclosed structure. In addition, although particular features of the present disclosure may have been disclosed with respect to only one of several implementations, such features may be combined with one or more other features of other implementations as may be desired and advantageous for any given or particular application. Furthermore, to the extent that the terms "include," "have," "have," "have," or variations thereof are used in the detailed description or claims, such terms are intended to be inclusive in a manner similar to the term "comprising."

[0144] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art not disclosed herein. The description and examples are to be considered as exemplary only, with the true scope and spirit of the present disclosure being indicated by the appended claims.

[0145] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes can be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. An obstacle detection method, characterized in that: include: Acquire an environmental image of the environment surrounding the electronic device; Determining location information of obstacles in the surrounding environment based on the environmental image and a pre-trained detection model; The detection model is used to obtain bird's-eye view features according to the environment image, and determine the position information of obstacles in the surrounding environment according to the bird's-eye view features.

2. The method according to claim 1, characterized in that The detection model is obtained in the following way: Inputting the sample environment image into the initial detection model to obtain sample position information of the sample obstacle in the sample environment image; Determining a current error of the initial detection model based on the sample position information and the marked position information of the sample obstacles; According to the current error of the initial detection model, the current parameters of the initial detection model are adjusted, and the step of inputting the sample environment image into the initial detection model to obtain the sample position information of the sample obstacle in the sample environment image is returned until the detection model is obtained when the training termination condition is met.

3. The method according to claim 1, characterized in that The position information of the obstacle includes the position information of a preset number of points on the obstacle; the detection model is obtained by training the network model by using the sample environment image as the model input parameter and the position information of the preset number of points on the sample obstacle included in the sample environment image as the model output parameter.

4. The method according to claim 3, characterized in that The position information of a preset number of points on the sample obstacles is obtained by discretizing the position information of the sample obstacles.

5. The method according to claim 3, characterized in that The method further comprises: Determining the outline information of the obstacle; The obstacle is drawn according to the contour information and the position information of the preset number of points.

6. The method according to claim 5, characterized in that The obstacle includes a wheel chock and / or a speed bump, and the method further includes: Obtaining location information of a boundary line of an area where the obstacle is located; Determining the direction information of the obstacle according to the position information of the boundary line; Drawing the obstacle according to the contour information and the position information of the preset number of points includes: The obstacle is drawn according to the outline information, the position information of the preset number of points, and the direction information.

7. The method according to claim 6, characterized in that When the obstacle is a wheel chock set in the storage space, the boundary line of the area where the obstacle is located is the boundary line of the storage space; when the obstacle is a speed bump, the boundary line of the area where the obstacle is located is the lane line of the road where the vehicle is currently located.

8. The method according to claim 1, characterized in that When there are multiple environmental images, the detection model includes a bird's-eye view feature generation sub-model and a detection sub-model; The bird's-eye view feature generation sub-model is used to extract features of the environment image for each environment image, fuse the features of the multiple environment images to obtain a fused feature, and obtain a bird's-eye view feature based on the fused feature; The detection sub-model is used to determine the position information of obstacles in the surrounding environment according to the bird's-eye view features.

9. The method according to claim 1, characterized in that There are multiple environmental images, and obtaining the environmental image of the surrounding environment of the electronic device includes: Multiple environmental images of the environment surrounding the electronic device are acquired by installing multiple image acquisition devices on the electronic device; wherein the multiple image acquisition devices are installed at different positions of the electronic device to acquire environmental images from different perspectives.

10. The method according to claim 9, characterized in that The multiple image acquisition devices synchronously acquire the multiple environment images.

11. An obstacle detection device, characterized in that: The device comprises: A first acquisition module is configured to acquire an environmental image of the surrounding environment of the electronic device; A first determination module is configured to determine position information of obstacles in the surrounding environment based on the environment image and a pre-trained detection model; The detection model is used to obtain bird's-eye view features according to the environment image, and determine the position information of obstacles in the surrounding environment according to the bird's-eye view features.

12. The device according to claim 11, characterized in that The detection model is obtained in the following way: Inputting the sample environment image into the initial detection model to obtain sample position information of the sample obstacle in the sample environment image; Determining a current error of the initial detection model based on the sample position information and the marked position information of the sample obstacles; According to the current error of the initial detection model, the current parameters of the initial detection model are adjusted, and the step of inputting the sample environment image into the initial detection model to obtain the sample position information of the sample obstacle in the sample environment image is returned until the detection model is obtained when the training termination condition is met.

13. The device according to claim 11, characterized in that The position information of the obstacle includes the position information of a preset number of points on the obstacle; the detection model is obtained by training the network model by using the sample environment image as the model input parameter and the position information of the preset number of points on the sample obstacle included in the sample environment image as the model output parameter.

14. An electronic device, characterized in that: include: processor; a memory for storing processor-executable instructions; The processor is configured to execute the instructions to implement the steps of the obstacle detection method according to any one of claims 1 to 10.

15. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the obstacle detection method according to any one of claims 1 to 10 are implemented.

16. A computer program product, characterized in that The invention comprises a computer program, which, when executed by a processor, implements the steps of the obstacle detection method according to any one of claims 1 to 10.

Images