Obstacle positioning method and device, equipment, storage medium and program product

Abstract

The embodiment of the invention provides an obstacle positioning method and device, equipment, a storage medium and a program product, and relates to the technical field of image processing. The method comprises the following steps: acquiring a scene image acquired by image acquisition equipment in a vehicle; performing cylindrical projection processing on the scene image to obtain a cylindrical image; performing obstacle detection processing on the cylindrical surface image to obtain an enveloping frame of at least one target obstacle; and determining the position information of the target obstacle according to the enveloping frame of the target obstacle. According to the method provided by the invention, direct visual positioning processing can be performed on the detected obstacle enveloping frame in a low-computing-power device, and compared with an aerial view obstacle visual positioning scheme, the scheme provided by the invention is wide in obstacle visual positioning range, simple and convenient in algorithm and relatively high in visual positioning precision of the three-dimensional obstacle.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to an obstacle positioning method, device, equipment, storage medium and program product. Background Art

[0002] During vehicle driving, based on the vehicle's obstacle avoidance function, it is necessary to accurately locate the position of obstacles in order to perform vehicle obstacle avoidance operations and ensure driving safety.

[0003] In the prior art, a bird's-eye view image is obtained by stitching with perspective projection, and an obstacle positioning mapping table is established based on a single-point mapping method that is strongly related to calibration to locate obstacles.

[0004] However, in the above method, due to the field of view angle limitation of the perspective projection method, it is difficult to establish a mapping relationship for obstacles outside the range of 10m in the captured image. For obstacles with a certain height, it is difficult to accurately determine their actual grounding position based on the spliced ​​images, thereby reducing the obstacle positioning accuracy. Summary of the invention

[0005] The embodiments of the present application provide an obstacle positioning method, device, equipment, storage medium and program product to solve the technical problems that the range of obstacles that can be positioned is small and the position of suspended obstacles cannot be positioned due to the angle limitation of the perspective projection method.

[0006] In a first aspect, an embodiment of the present application provides an obstacle positioning method, including:

[0007] Acquire a scene image acquired by an image acquisition device in a vehicle; and perform cylindrical projection processing on the scene image to obtain a cylindrical image;

[0008] Performing obstacle detection processing on the cylindrical image to obtain an inclusion frame of at least one target obstacle; wherein the inclusion frame represents the outline of the target obstacle in the cylindrical image;

[0009] The position information of the target obstacle is determined according to the encompassing frame of the target obstacle.

[0010] In a possible implementation, determining the position information of the target obstacle according to the encompassing frame of the target obstacle includes:

[0011] Determine a first coordinate and a second coordinate of the target obstacle according to the encompassing frame of the target obstacle; wherein the first coordinate is the coordinate of the target obstacle on a first coordinate axis in a preset coordinate system; and the second coordinate is the coordinate of the target obstacle on a second coordinate axis in the preset coordinate system;

[0012] The position information of the target obstacle is determined according to the first coordinate and the second coordinate.

[0013] In a possible implementation, determining the first coordinate and the second coordinate of the target obstacle according to the encompassing frame of the target obstacle includes:

[0014] Determining calibration parameters of the image acquisition device;

[0015] The first coordinate and the second coordinate of the target obstacle are determined according to the calibration parameters and the encompassing frame of the target obstacle.

[0016] In a possible implementation, determining the first coordinate and the second coordinate of the target obstacle according to the calibration parameters and the encompassing frame of the target obstacle includes:

[0017] Determining an obstacle type of the target obstacle, and determining a first coordinate of the target obstacle according to the calibration parameters and the encompassing frame and obstacle type of the target obstacle;

[0018] The second coordinates of the target obstacle are determined according to the calibration parameters and the first coordinates and the encompassing frame of the target obstacle.

[0019] In a possible implementation, determining the first coordinate of the target obstacle according to the calibration parameters and the encompassing frame and obstacle type of the target obstacle includes:

[0020] If it is determined that the obstacle type of the target obstacle is a single size type, the first coordinate of the target obstacle is determined according to a preset target height, an image height in the encompassing frame of the target obstacle, and a focal length in the calibration parameters; wherein the preset target height is a real height of the target obstacle in the physical world, and the image height is a height of the target obstacle in the cylindrical image;

[0021] If it is determined that the obstacle type of the target obstacle is a multi-size type, the first coordinate is determined according to the corner point coordinates of the target corner point of the encompassing frame and the focal length, installation height and vanishing point coordinates in the calibration parameters; wherein the target corner point represents the pixel point corresponding to the grounding point of the target obstacle in the cylindrical image; the corner point coordinates are the coordinates of the target corner point in the projection coordinate system; and the vanishing point coordinates are the coordinates of the vanishing point of the image acquisition device in the image coordinate system.

[0022] In a possible implementation, the encompassing frame has a target corner point; the target corner point represents a pixel point corresponding to a grounding point of the target obstacle in the cylindrical image;

[0023] Determining the second coordinate of the target obstacle according to the calibration parameters and the first coordinate and the encompassing frame of the target obstacle includes:

[0024] The second coordinate is determined according to the first coordinate of the target obstacle and the corner point coordinates of the target corner point, as well as the focal length and the image principal point coordinates in the calibration parameters; wherein the corner point coordinates are the coordinates of the target corner point in the projection coordinate system; and the image principal point coordinates are the coordinates of the image principal point of the image acquisition device in the image coordinate system.

[0025] In a possible implementation, determining the position information of the target obstacle according to the first coordinate and the second coordinate includes:

[0026] Determining calibration parameters of the image acquisition device; wherein the calibration parameters include an offset parameter, and the offset parameter represents an offset condition of the image acquisition device;

[0027] According to the offset parameter, coordinate conversion processing is performed on the first coordinate and the second coordinate respectively to obtain a converted first coordinate and a converted second coordinate;

[0028] The position information is determined according to the converted first coordinates and the converted second coordinates.

[0029] In a second aspect, an embodiment of the present application provides an obstacle locating device, comprising:

[0030] A projection module is used to obtain a scene image captured by an image acquisition device in the vehicle; and to perform cylindrical projection processing on the scene image to obtain a cylindrical image;

[0031] A detection module, configured to perform obstacle detection processing on the cylindrical image to obtain an inclusion frame of at least one target obstacle; wherein the inclusion frame represents the outline of the target obstacle in the cylindrical image;

[0032] The determination module is used to determine the position information of the target obstacle according to the encompassing frame of the target obstacle.

[0033] In a possible implementation manner, the determining module is specifically configured to:

[0034] Determine a first coordinate and a second coordinate of the target obstacle according to the encompassing frame of the target obstacle; wherein the first coordinate is the coordinate of the target obstacle on a first coordinate axis in a preset coordinate system; and the second coordinate is the coordinate of the target obstacle on a second coordinate axis in the preset coordinate system;

[0035] The position information of the target obstacle is determined according to the first coordinate and the second coordinate.

[0036] In a possible implementation manner, the determining module is specifically configured to:

[0037] Determining calibration parameters of the image acquisition device;

[0038] The first coordinate and the second coordinate of the target obstacle are determined according to the calibration parameters and the encompassing frame of the target obstacle.

[0039] In a possible implementation manner, the determining module is specifically configured to:

[0040] Determining an obstacle type of the target obstacle, and determining a first coordinate of the target obstacle according to the calibration parameters and the encompassing frame and obstacle type of the target obstacle;

[0041] The second coordinates of the target obstacle are determined according to the calibration parameters and the first coordinates and the encompassing frame of the target obstacle.

[0042] In a possible implementation manner, the determining module is specifically configured to:

[0043] If it is determined that the obstacle type of the target obstacle is a single size type, the first coordinate of the target obstacle is determined according to a preset target height, an image height in an encompassing frame of the target obstacle, and a focal length in the calibration parameters; wherein the preset target height is a real height of the target obstacle in the physical world, and the image height is a height of the target obstacle in the cylindrical image.

[0044] In a possible implementation, the encompassing frame has a target corner point; the target corner point represents a pixel point corresponding to a grounding point of the target obstacle in the cylindrical image; and the determination module is further specifically configured to:

[0045] If it is determined that the obstacle type of the target obstacle is a multi-size type, the first coordinate is determined according to the corner point coordinates of the target corner point and the focal length, installation height and vanishing point coordinates in the calibration parameters; wherein the corner point coordinates are the coordinates of the target corner point in the projection coordinate system; and the vanishing point coordinates are the coordinates of the vanishing point of the image acquisition device in the image coordinate system.

[0046] In a possible implementation, the encompassing frame has a target corner point; the target corner point represents a pixel point corresponding to a grounding point of the target obstacle in the cylindrical image; and the determination module is further specifically configured to:

[0047] The second coordinate is determined according to the first coordinate of the target obstacle and the corner point coordinates of the target corner point, as well as the focal length and the image principal point coordinates in the calibration parameters; wherein the corner point coordinates are the coordinates of the target corner point in the projection coordinate system; and the image principal point coordinates are the coordinates of the image principal point of the image acquisition device in the image coordinate system.

[0048] In a possible implementation manner, the determining module is further specifically configured to:

[0049] Determining calibration parameters of the image acquisition device; wherein the calibration parameters include an offset parameter, and the offset parameter represents an offset condition of the image acquisition device;

[0050] According to the offset parameter, coordinate conversion processing is performed on the first coordinate and the second coordinate respectively to obtain a converted first coordinate and a converted second coordinate;

[0051] The position information is determined according to the converted first coordinates and the converted second coordinates.

[0052] In a third aspect, an embodiment of the present application provides an electronic device, including: a memory, a processor;

[0053] The memory stores computer-executable instructions;

[0054] The processor executes the computer-executable instructions stored in the memory, so that the processor executes the above first aspect and / or various possible implementations of the first aspect.

[0055] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, in which computer-executable instructions are stored. When the computer-executable instructions are executed by a processor, they are used to implement the first aspect above and / or various possible implementations of the first aspect.

[0056] In a fifth aspect, an embodiment of the present application provides a computer program product, including a computer program, which, when executed by a processor, implements the above first aspect and / or various possible implementation methods of the first aspect.

[0057] The obstacle positioning method, apparatus, device, storage medium and program product provided in the embodiments of the present application perform cylindrical projection processing on the collected scene image to obtain a cylindrical image, and perform obstacle detection processing on the cylindrical image to obtain the outline of each target obstacle in the cylindrical image. Further processing can determine the position of each target obstacle; further, through cylindrical projection processing, a more accurate outline of a target obstacle with a certain height can be obtained, and then the position of the target obstacle can be accurately determined, thereby improving the accuracy of obstacle positioning. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0059] Figure 1 A schematic diagram of an application scenario provided for this application;

[0060] Figure 2 A schematic diagram of a flow chart of an obstacle locating method provided in an embodiment of the present application;

[0061] Figure 3 A schematic diagram of a flow chart of another obstacle locating method provided in an embodiment of the present application;

[0062] Figure 4 A schematic diagram of a flow chart of another obstacle locating method provided in an embodiment of the present application;

[0063] Figure 5 A schematic diagram of a process of locating an obstacle provided in an embodiment of the present application;

[0064] Figure 6 A schematic diagram of the principle of a similar triangle positioning estimation algorithm provided in an embodiment of the present application;

[0065] Figure 7 A schematic diagram of the structure of an obstacle locating device provided in an embodiment of the present application;

[0066] Figure 8 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application.

[0067] The above drawings have shown clear embodiments of the present application, which will be described in more detail later. These drawings and text descriptions are not intended to limit the scope of the present application in any way, but to illustrate the concept of the present application to those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION

[0068] Here, exemplary embodiments will be described in detail, and examples thereof are shown in the accompanying drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present application. Instead, they are only examples of devices and methods consistent with some aspects of the present application.

[0069] First, the terms involved in this application are explained:

[0070] Fisheye Camera: A fisheye camera is a wide-angle lens camera that is able to capture an extremely wide angle of view, usually at or near 180 degrees.

[0071] Cylindrical plane: A cylinder with the camera optical center as the center, and a plane formed by unfolding the cylinder.

[0072] Landing point distance: the horizontal distance from the projection point of the camera optical axis on the cylindrical plane to the camera installation point.

[0073] Angle of incidence: The angle between the camera's optical axis and the vertical line.

[0074] Image principal point: The projection point of the optical center of the camera lens onto the image sensor.

[0075] Vanishing point: In perspective projection, parallel lines in three-dimensional space will no longer appear parallel on the two-dimensional image plane, but will converge to a point, which is the vanishing point. The vanishing point reflects the perspective and perspective effect of the human eye or camera.

[0076] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use, processing, transmission, provision, disclosure and application of the relevant data comply with relevant laws, regulations and standards, take necessary confidentiality measures, do not violate public order and good morals, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0077] It should be noted that the obstacle positioning method, device, equipment, storage medium and program product of the present application can be used in the field of image processing technology, and can also be used in any field other than image processing technology. The application field of the obstacle positioning method, device, equipment, storage medium and program product of the present application is not limited.

[0078] Figure 1 A schematic diagram of an application scenario provided by this application. Figure 1 As shown, the specific application scenario of the present application includes an electronic device 101 in a vehicle and multiple image acquisition devices 102, such as a front-view fisheye camera, a rear-view fisheye camera, a left-view fisheye camera, and a right-view fisheye camera. During vehicle driving, based on the vehicle obstacle avoidance function, it is necessary to accurately locate the position of the obstacle in order to perform the vehicle obstacle avoidance operation and ensure driving safety.

[0079] In one example, a fisheye camera is often used as a visual perception sensor to balance field of view and hardware cost. In general, in an assisted parking system, the fisheye camera 102 collects raw images, and based on the electronic device 101, the raw images collected by the fisheye camera 102 are processed by perspective projection stitching to obtain a stitched image, and an obstacle positioning mapping table is established based on a single-point mapping method that is strongly related to calibration, so as to obtain the position information of the obstacle in the vehicle body environment according to the stitched image.

[0080] However, in the above method, due to the angle limitation of perspective projection, the visible range of the surround stitching image is basically within the range of 8-10m. For high obstacles such as vehicles and warning columns, it is often difficult to obtain the coordinates of their actual grounding points at the edges of the stitching image due to the stretching of the bird's-eye view, thereby reducing the accuracy of locating such obstacles.

[0081] In response to the above technical problems, the present application proposes the following technical concept: by performing cylindrical projection processing on the collected scene image to obtain a cylindrical image, and performing obstacle detection processing on the cylindrical image to obtain the outline of each target obstacle in the cylindrical image, and further processing can determine the position of each target obstacle. In this way, the problem of reduced obstacle positioning accuracy due to the angle limitation of perspective projection in the prior art can be solved.

[0082] The technical solution of the present application and how the technical solution of the present application solves the above-mentioned technical problems are described in detail below with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present application will be described below in conjunction with the accompanying drawings.

[0083] Figure 2 A flowchart of an obstacle location method provided in an embodiment of the present application is shown in FIG. Figure 2 As shown, the method includes:

[0084] 201. Acquire a scene image captured by an image capture device in a vehicle; and perform cylindrical projection processing on the scene image to obtain a cylindrical image.

[0085] Exemplarily, the execution subject of this embodiment may be an electronic device in a vehicle, hereinafter referred to as a device. The device may be a physical device or a virtual device that executes the obstacle positioning method. When the vehicle is in the process of parking or driving, each image acquisition device in the vehicle may acquire a scene image of the scene in which the vehicle is located. For example, each fisheye camera device rigidly mounted on the vehicle body may be used to obtain fisheye camera visual images, which are front view, rear view, left view, and right view images. The device obtains the captured scene image from the image acquisition device, and based on the cylindrical projection calibration system of the device and according to the calibration parameters of the image acquisition device determined in advance, determines an equidistant projection model. According to the equidistant projection model, the scene image of the image acquisition device is projected onto the cylindrical plane, and the landing point distance is linearly proportional to the incident angle, so that the same angular difference corresponds to the same distance difference, and the projection function is simple, so that the scene image is subjected to cylindrical projection processing to obtain a cylindrical image corresponding to each scene image; furthermore, the generally used perspective projection, due to the tan function, when the incident angle of the light entering the lens approaches 90 degrees, the landing point on the image plane approaches infinity, which will eventually reduce the visible range of the fisheye camera, thereby losing the wide-angle advantage of the fisheye camera, while the equidistant projection model adopted in the embodiment of the present application has small image distortion close to the optical axis position, and can reflect spatial points with an incident angle greater than 90 degrees on a finitely smaller image plane, and can retain more effective information.

[0086] For example, according to the predetermined internal and external parameters of the fisheye camera calibration, a cylindrical projection model is defined, such as setting the radius and center position of the cylinder, and establishing a mapping relationship between the fisheye image pixels and the cylindrical projection image pixels. The cylindrical projection parameters are optimized by minimizing the projection error, so that the scene image captured by the fisheye camera can be converted into a less distorted cylindrical projection image through the cylindrical projection parameters.

[0087] 202. Perform obstacle detection processing on the cylindrical image to obtain an inclusion frame of at least one target obstacle; wherein the inclusion frame represents the outline of the target obstacle in the cylindrical image.

[0088] Exemplarily, after obtaining the cylindrical image, the device can perform obstacle detection processing on the cylindrical image based on a preset neural network to obtain one or more target obstacles in the cylindrical image, and then obtain the outline of each target obstacle, i.e., the inclusion frame, through neural network deep learning post-processing for processing.

[0089] 203. Determine the position information of the target obstacle according to the encompassing frame of the target obstacle.

[0090] Exemplarily, the device can use a preset obstacle depth estimation method, such as a cylindrical projection combined with a similar triangle obstacle depth estimation method, to perform depth estimation calculations on the encompassing frame of each detected target obstacle to obtain the depth information of the target obstacle, that is, the current distance between the obstacle and the vehicle body. Furthermore, the position information of each target obstacle can be accurately determined to complete the location of the obstacle, which can be used to assist the vehicle in parking or performing obstacle avoidance operations while driving to ensure the safety of the vehicle.

[0091] In this embodiment, an obstacle positioning method is provided, which performs cylindrical projection processing on a captured scene image to obtain a cylindrical image, and performs obstacle detection processing on the cylindrical image to obtain the outline of each target obstacle in the cylindrical image, and further processing can determine the position of each target obstacle; further, through the cylindrical projection processing, more visual information can be retained in a complete image, and radial distortion is corrected, thereby reducing the influence of large distortion on subsequent obstacle detection based on an image depth network, so as to obtain a more accurate outline of a target obstacle with a certain height, and further the position of the target obstacle can be accurately determined, thereby improving the accuracy of obstacle positioning.

[0092] Figure 3 A flowchart of another obstacle location method provided in an embodiment of the present application is shown as follows: Figure 3 As shown, the method includes:

[0093] 301. Acquire a scene image acquired by an image acquisition device in a vehicle; and perform cylindrical projection processing on the scene image to obtain a cylindrical image.

[0094] Exemplarily, this step may refer to step 201 and will not be described in detail here.

[0095] 302. Perform obstacle detection processing on the cylindrical image to obtain an inclusion frame of at least one target obstacle; wherein the inclusion frame represents the outline of the target obstacle in the cylindrical image.

[0096] Exemplarily, this step may refer to step 202 and will not be described in detail here.

[0097] 303. Determine a first coordinate and a second coordinate of the target obstacle according to the encompassing frame of the target obstacle; wherein the first coordinate is a coordinate of the target obstacle on a first coordinate axis in a preset coordinate system; and the second coordinate is a coordinate of the target obstacle on a second coordinate axis in the preset coordinate system.

[0098] Exemplarily, the encompassing frame of each detected target obstacle in the cylindrical image includes multiple encompassing frame corner points, that is, all pixel points corresponding to the outline of the target obstacle in the cylindrical image; based on a preset projection calculation formula, the device can perform projection calculation processing on the coordinates of each encompassing frame corner point in each encompassing frame in the projection image coordinate system, and obtain the coordinates of each target obstacle on the first coordinate axis in the preset coordinate system, that is, the first coordinate, and the coordinates of each target obstacle on the second coordinate axis in the preset coordinate system, that is, the second coordinate.

[0099] For example, for each image acquisition device in the vehicle, a corresponding preset coordinate system can be pre-set, such as a 3D camera coordinate system, the origin of which is usually located at the optical center of the camera, and the coordinate axes include X-axis, Y-axis, and Z-axis; the X-axis is a horizontal axis, usually pointing to the right side of the image; the Y-axis is a vertical axis, usually pointing to the bottom of the image; the Z-axis is a depth axis, usually pointing to the front of the camera, that is, extending from the optical center of the camera along the optical axis of the lens. For each detected target obstacle in the cylindrical image, based on the relationship between the projection image coordinate system and the camera coordinate system, the coordinates of each corner point of each frame in the projection image coordinate system can be projected and extrapolated from the 2D projection image coordinate system to the 3D camera coordinate system, and the coordinates of each corner point of each frame in the camera coordinate system can be determined, and then the coordinates of each target obstacle on the Z axis in the camera coordinate system and the coordinates of each target obstacle on the X axis in the camera coordinate system can be determined.

[0100] 304. Determine location information of the target obstacle according to the first coordinate and the second coordinate.

[0101] Exemplarily, the device performs coordinate calculation on the first coordinate and the second coordinate of each target obstacle based on a preset coordinate calculation formula. For example, by performing a three-dimensional coordinate conversion between the camera coordinate system and the vehicle's own body coordinate system, the first coordinate and the second coordinate of each target obstacle are converted and calculated to obtain coordinate information in the body coordinate system, which is consistent with human sensory vision. Then, the coordinate information in the body coordinate system is processed by coordinate calculation to obtain the position information of each target obstacle relative to the vehicle body.

[0102] In this embodiment, based on the above embodiment, the enclosing frame of each target obstacle obtained based on cylindrical image detection is depth estimated through projection calculation combined with a similar triangle depth estimation scheme. The obstacle visual positioning range is wide, the algorithm is simple, and the position information of each target obstacle can be accurately obtained, thereby improving the positioning accuracy of higher three-dimensional obstacles.

[0103] Figure 4A flowchart of another obstacle locating method provided in an embodiment of the present application is shown as follows: Figure 4 As shown, the method includes:

[0104] 401. Acquire a scene image captured by an image capture device in a vehicle; and perform cylindrical projection processing on the scene image to obtain a cylindrical image.

[0105] Exemplarily, this step may refer to step 201 and will not be described in detail here.

[0106] 402. Perform obstacle detection processing on the cylindrical image to obtain an inclusion frame of at least one target obstacle; wherein the inclusion frame represents the outline of the target obstacle in the cylindrical image.

[0107] Exemplarily, this step may refer to step 202 and will not be described in detail here.

[0108] 403. Determine calibration parameters of the image acquisition device.

[0109] Exemplarily, the device obtains calibration parameters of the image acquisition device in advance based on cylindrical projection correction, such as the focal length, principal point, vanishing point, installation height and other parameters of the camera, which are used as obstacle depth estimation calculation parameters in the subsequent depth estimation calculation process.

[0110] For example, Figure 5 A flowchart of an obstacle location process provided in an embodiment of the present application is shown as follows: Figure 5 As shown, the device obtains the scene image captured by each fisheye camera, and based on the cylindrical projection calibration system, uses the camera calibration method to accurately calculate the camera focal length, vanishing point and other internal parameters, and uses the computer vision (Perspective-n-Point, referred to as PnP) algorithm to calculate the camera installation height and other external parameters.

[0111] 404. Determine a first coordinate and a second coordinate of the target obstacle according to the calibration parameters and the encompassing frame of the target obstacle.

[0112] For example, for each image acquisition device in the vehicle, in a preset coordinate system such as a 3D camera coordinate system, combined with calibration parameters, a projection calculation formula of the projected image coordinate system to the camera coordinate system can be determined. Figure 5 Through the input and output interface, the image detection processor of the device detects and processes the cylindrical image to obtain the encompassing frame of each detected target obstacle in the cylindrical image. Based on the projection calculation formula, the encompassing frame corner points in each encompassing frame can be projected and calculated to obtain the coordinates of each target obstacle on the first coordinate axis in the preset coordinate system and the coordinates of each target obstacle on the second coordinate axis in the preset coordinate system.

[0113] In one example, step 404 includes the following steps:

[0114] The first step of step 404 is to determine the obstacle type of the target obstacle, and determine the first coordinates of the target obstacle according to the calibration parameters and the encompassing frame and obstacle type of the target obstacle.

[0115] The second step of step 404 is to determine the second coordinates of the target obstacle according to the calibration parameters and the first coordinates and the encompassing frame of the target obstacle.

[0116] For example, in combination Figure 5 , the image detection processor of the device performs semantic recognition on the obstacles in the input cylindrical image, detects the encompassing frame of each detected target obstacle in the cylindrical image, and at the same time, can distinguish the categories to obtain the obstacle type of each target obstacle, including a single-size type with one height dimension such as a cylindrical railing, and a multi-size type with multiple height dimensions such as a cone roadblock. For different obstacle types, combined with calibration parameters, the projection calculation formula corresponding to each obstacle type is determined to calculate the encompassing frame of each target obstacle to obtain the first coordinate of each target obstacle. After calculating the first coordinate of each target obstacle, the obstacle depth is further estimated, that is, the device can determine the corresponding similar triangle positioning estimation algorithm based on the calibration parameters to estimate and calculate the encompassing frame and the first coordinate of each target obstacle to obtain the depth information of each target obstacle relative to the image acquisition device, that is, the second coordinate.

[0117] In one example, the first step of step 404 includes:

[0118] Step 1: If the obstacle type of the target obstacle is determined to be a single size type, the first coordinate of the target obstacle is determined according to the preset target height, the image height in the encompassing frame of the target obstacle, and the focal length in the calibration parameters; wherein the preset target height is the real height of the target obstacle in the physical world, and the image height is the height of the target obstacle in the cylindrical image.

[0119] Specifically, for each target obstacle, if the device determines that the obstacle type of the target obstacle is a single size type, the universal size of the obstacle can be used as a priori condition, that is, a preset target height. For example, in reality, the height of a car is generally about 1.7 meters. The corner points of the inclusive frame in the inclusive frame obtained by joint perception detection can be used to obtain the height of the target obstacle in the cylindrical image, that is, the image height. Based on the preset formula, the preset target height and the image height of the target obstacle, as well as the focal length in the calibration parameters are calculated and processed to obtain the first coordinate.

[0120] For example, the projection of the cylindrical image of the fisheye camera from 2D to the camera's 3D is calculated based on the preset fisheye image cylindrical correction projection formula (1): Combined with the basic perspective projection formula of the camera, we can get the conversion formula (2): Among them, u is the horizontal coordinate on the cylindrical image plane, v is the vertical coordinate on the cylindrical image plane, ρ is the cylindrical projection distance, is the focal length of the cylindrical projection, f y is the focal length in the vertical direction, u0 and v0 are the center points of the horizontal coordinate and the vertical coordinate on the cylindrical image plane respectively. is the cylindrical projection angle, are the horizontal coordinate, vertical coordinate, and depth coordinate in the virtual coordinate system corresponding to the constructed virtual projection plane, respectively; X, Y, and Z are the horizontal coordinate, vertical coordinate, and depth coordinate in the camera coordinate system; a is a preset parameter. By inverting the above formula, we can get the projection calculation formula (3) corresponding to the relationship between the cylindrical projection coordinate system and the camera coordinate system: at the same time, Figure 6 A schematic diagram of the principle of a similar triangle positioning estimation algorithm provided in an embodiment of the present application, such as Figure 6 As shown, under the condition that the camera focal length in the calibration parameters is known and the image height and real height of the target obstacle can be measured, the distance depth between the camera lens and the real target obstacle is calculated based on the formula (4) depth = real h / pixel h*f, and then the first coordinate can be obtained; wherein the arrow symbol represents an object in the real world and its projection on the image plane, f is the focal length in the calibration parameters, depth is the distance, pixel h is the image height, and real h is the preset target height, that is, the preset height.

[0121] For another example, in a vehicle assisted driving system, for obstacles appearing in a parking scene, for the 3D camera coordinate system coordinates restored from the image, the X and Z coordinates of the target obstacle can be calculated to determine the obstacle position. Therefore, by simplifying the projection calculation formula (3) and combining it with formula (4), the formula Z = real h / pixel h·f can be derived. The preset target height real h and the image height pixel h of each target obstacle, as well as the focal length f in the calibration parameters can be calculated and processed to obtain the first coordinate of each target obstacle. Furthermore, the calculation steps of the fisheye cylindrical projection image to the camera coordinate system can be simplified, which saves the obstacle monocular estimation computing power and has a certain positive significance for improving the efficiency of the assisted driving system.

[0122] In one example, the encompassing frame has a target corner point; the target corner point represents a pixel point corresponding to a grounding point of the target obstacle in the cylindrical image; the first step of step 404 also includes: if it is determined that the obstacle type of the target obstacle is a multi-size type, then according to the corner point coordinates of the target corner point and the focal length, installation height and vanishing point coordinates in the calibration parameters, determine the first coordinate; wherein the corner point coordinates are the coordinates of the target corner point in the projection coordinate system; the vanishing point coordinates are the coordinates of the vanishing point of the image acquisition device in the image coordinate system.

[0123] Specifically, after the cylindrical image is processed based on the deep learning model preset in the device, for each of the detected enclosing frame corner points in the enclosing frame, the enclosing frame corner point corresponding to the grounding point of the target obstacle in the cylindrical image can be determined, and the pixel point corresponding to the grounding point of the target obstacle in the cylindrical image is determined as the target corner point, and the coordinates of the target corner point in the projection coordinate system, i.e., the corner point coordinates, are determined. For each target obstacle, if the device determines that the obstacle type of the target obstacle is a multi-size type, such as when there are many types of detected obstacles or the size is small, resulting in fewer valid pixels in the image, the focal length f, the installation height cam h and the vanishing point coordinates vp in the calibration parameters, and the vertical coordinate pixel y in the corner point coordinates of the target corner point are calculated and processed using a preset formula, such as Z=cam h / (pixel y-vp)·f, to calculate and obtain the coordinates of each target obstacle in the Z direction in the camera coordinate system; wherein the vanishing point coordinates are the coordinates of the vanishing point of the camera in the image coordinate system.

[0124] In one example, the encompassing frame has a target corner point; the target corner point represents a pixel point corresponding to a grounding point of a target obstacle in a cylindrical image; the second step of step 404 includes: determining a second coordinate according to the first coordinate of the target obstacle and the corner point coordinates of the target corner point, as well as the focal length and the image principal point coordinates in the calibration parameters; wherein the corner point coordinates are the coordinates of the target corner point in the projection coordinate system; and the image principal point coordinates are the coordinates of the image principal point of the image acquisition device in the image coordinate system.

[0125] Specifically, for each included frame corner point in each detected included frame, the included frame corner point corresponding to the ground point of the target obstacle in the cylindrical image can be determined, and the pixel point corresponding to the ground point of the target obstacle in the cylindrical image is determined as the target corner point, and the coordinates of the target corner point in the projection coordinate system, i.e., the corner point coordinates, are determined. For each target obstacle, based on a preset formula, such as X=Z·(x-cx) / f, the first coordinate Z of each target obstacle and the horizontal coordinate x in the corner point coordinates of the target corner point, as well as the focal length f and the image principal point coordinate cx in the calibration parameters are calculated and processed to obtain the second coordinate of each target obstacle; wherein the image principal point coordinate cx is the coordinate of the image principal point of the image acquisition device in the image coordinate system.

[0126] 405. Determine calibration parameters of the image acquisition device; wherein the calibration parameters include offset parameters, and the offset parameters represent the offset condition of the image acquisition device.

[0127] Exemplarily, the device obtains calibration parameters of the image acquisition device based on the cylindrical image and based on the cylindrical projection correction method, such as the internal and external parameters of the camera are used as obstacle depth estimation calculation parameters in the subsequent depth estimation calculation process; wherein the calibration parameters include offset parameters that characterize the offset of the image acquisition device, such as the offset angle angle of the camera Z axis and the position offset (tx, ty), wherein the offset angle is the angle of rotation of the camera around its own Z axis (usually the optical axis), which is used to describe the rotation posture of the camera; the position offset is the position change of the camera in three-dimensional space.

[0128] 406. Perform coordinate transformation processing on the first coordinate and the second coordinate respectively according to the offset parameter to obtain a transformed first coordinate and a transformed second coordinate.

[0129] For example, in combination Figure 5 Based on the offset parameter, the device uses the counterclockwise rotation of the coordinate system to simplify the coordinate system conversion from the camera coordinate system to the vehicle body coordinate system, and performs coordinate conversion processing on the first coordinate and the second coordinate of each target obstacle in the preset coordinate system, respectively, to obtain the converted first coordinate and the converted second coordinate, that is, to obtain the coordinates of the target obstacle in the vehicle body coordinate system.

[0130] 407. Determine position information according to the converted first coordinate and the converted second coordinate.

[0131] Exemplarily, the device calculates the converted first coordinate and the converted second coordinate corresponding to each target obstacle based on the obstacle positioning algorithm to obtain the obstacle positioning result, so as to obtain the position information of each target obstacle. By displaying the position information of each target obstacle, the user can be reminded to control the vehicle to perform obstacle avoidance operations.

[0132] In this embodiment, based on the above embodiments, a cylindrical projection combined with a similar triangle obstacle depth estimation algorithm is used to obtain the positioning coordinates of the detected obstacle in its camera coordinate system in one step, and the obstacle coordinates are transferred to the vehicle body coordinate system through the coordinate system conversion between the vehicle body and the camera; thereby, while improving the obstacle positioning accuracy, the computational complexity of the obstacle depth estimation algorithm is greatly reduced, so that the obstacle distance estimation can be realized in low or extremely low computing power, thereby reducing the hardware cost.

[0133] Figure 7 A schematic diagram of the structure of an obstacle locating device provided in an embodiment of the present application is shown in FIG. Figure 7 As shown, the device comprises:

[0134] The projection module 501 is used to obtain the scene image captured by the image acquisition device in the vehicle; and perform cylindrical projection processing on the scene image to obtain a cylindrical image;

[0135] A detection module 502 is used to perform obstacle detection processing on the cylindrical image to obtain an inclusion frame of at least one target obstacle; wherein the inclusion frame represents the outline of the target obstacle in the cylindrical image;

[0136] The determination module 503 is used to determine the position information of the target obstacle according to the encompassing frame of the target obstacle.

[0137] In a possible implementation, the determination module 503 is specifically configured to:

[0138] Determine the first coordinate and the second coordinate of the target obstacle according to the encompassing frame of the target obstacle; wherein the first coordinate is the coordinate of the target obstacle on the first coordinate axis in the preset coordinate system; and the second coordinate is the coordinate of the target obstacle on the second coordinate axis in the preset coordinate system;

[0139] The position information of the target obstacle is determined according to the first coordinate and the second coordinate.

[0140] In a possible implementation, the determination module 503 is specifically configured to:

[0141] Determine calibration parameters of image acquisition equipment;

[0142] The first coordinate and the second coordinate of the target obstacle are determined according to the calibration parameters and the encompassing frame of the target obstacle.

[0143] In a possible implementation, the determination module 503 is specifically configured to:

[0144] Determine the obstacle type of the target obstacle, and determine the first coordinate of the target obstacle according to the calibration parameters and the encompassing frame and obstacle type of the target obstacle;

[0145] The second coordinate of the target obstacle is determined according to the calibration parameters and the first coordinate and the encompassing frame of the target obstacle.

[0146] In a possible implementation, the determination module 503 is specifically configured to:

[0147] If it is determined that the obstacle type of the target obstacle is a single-size type, the first coordinate of the target obstacle is determined according to the preset target height, the image height in the encompassing frame of the target obstacle, and the focal length in the calibration parameters; wherein the preset target height is the real height of the target obstacle in the physical world, and the image height is the height of the target obstacle in the cylindrical image; if it is determined that the obstacle type of the target obstacle is a multi-size type, the first coordinate is determined according to the corner point coordinates of the target corner point of the encompassing frame and the focal length, installation height, and vanishing point coordinates in the calibration parameters; wherein the target corner point represents the pixel point corresponding to the grounding point of the target obstacle in the cylindrical image; the corner point coordinates are the coordinates of the target corner point in the projection coordinate system; and the vanishing point coordinates are the coordinates of the vanishing point of the image acquisition device in the image coordinate system.

[0148] In a possible implementation, the encompassing frame has a target corner point; the target corner point represents a pixel point corresponding to a grounding point of the target obstacle in the cylindrical image; the determination module 503 is further specifically used for:

[0149] The second coordinate is determined according to the first coordinate of the target obstacle and the corner point coordinates of the target corner point, as well as the focal length and the image principal point coordinates in the calibration parameters; wherein the corner point coordinates are the coordinates of the target corner point in the projection coordinate system; and the image principal point coordinates are the coordinates of the image principal point of the image acquisition device in the image coordinate system.

[0150] In a possible implementation manner, the determination module 503 is further specifically configured to:

[0151] Determine calibration parameters of the image acquisition device according to the cylindrical image; wherein the calibration parameters include an offset parameter, and the offset parameter represents an offset condition of the image acquisition device;

[0152] According to the offset parameter, coordinate transformation processing is performed on the first coordinate and the second coordinate respectively to obtain a transformed first coordinate and a transformed second coordinate;

[0153] The position information is determined according to the converted first coordinate and the converted second coordinate.

[0154] The device of this embodiment can execute the technical solution in the above method. Its specific implementation process and technical principles are the same and will not be repeated here.

[0155] It should be noted that it should be understood that the division of the various modules of the above device is only a division of logical functions. In actual implementation, they can be fully or partially integrated into one physical entity, or they can be physically separated. And these modules can all be implemented in the form of software called by processing elements; they can also be all implemented in the form of hardware; some modules can also be implemented in the form of software called by processing elements, and some modules can be implemented in the form of hardware. Each module can be a separately established processing element, or it can be integrated in a chip of the above device. In addition, it can also be stored in the memory of the above device in the form of program code, and called and executed by a processing element of the above device. The functions of each module. In addition, all or part of these modules can be integrated together, or they can be implemented independently. The processing element here can be an integrated circuit with signal processing capabilities. In the implementation process, each step of the above method or each module above can be completed by an integrated logic circuit of hardware in the processor element or instructions in the form of software.

[0156] Figure 8 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application is shown in FIG. Figure 8 As shown, the electronic device includes: a memory 601, a processor 602; the memory 601; and a memory for storing instructions executable by the processor 602.

[0157] The processor 602 is configured to execute the method provided in the above embodiment.

[0158] The electronic device further includes a receiver 603 and a transmitter 604. The receiver 603 is used to receive instructions and data sent by other devices, and the transmitter 604 is used to send instructions and data to external devices.

[0159] The specific implementation process of the processor can be found in the above method embodiment, and its implementation principle and technical effect are similar, so this embodiment will not be repeated here.

[0160] In the above embodiments, it should be understood that the processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), etc. The general-purpose processor may be a microprocessor or any conventional processor. The steps of the method disclosed in the invention may be directly implemented as being executed by a hardware processor, or may be executed by a combination of hardware and software modules in the processor.

[0161] The memory may include a high-speed memory (Random Access Memory, referred to as RAM), and may also include a non-volatile memory (Non-volatile Memory, referred to as NVM), such as at least one disk memory.

[0162] An embodiment of the present application also provides a chip for executing instructions, which is used to execute the technical solution of the processing method in the above embodiment.

[0163] The embodiment of the present application also provides a computer-readable storage medium, in which computer-executable instructions are stored. When the computer-executable instructions are executed on a computer, the computer executes the technical solution of the processing method of the above embodiment.

[0164] The above-mentioned readable storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk or optical disk. The readable storage medium can be any available medium that can be accessed by a general or special-purpose computer.

[0165] An exemplary readable storage medium is coupled to a processor so that the processor can read information from the readable storage medium and write information to the readable storage medium. Of course, the readable storage medium can also be an integral part of the processor. The processor and the readable storage medium can be located in a dedicated integrated circuit. Of course, the processor and the readable storage medium can also exist in a device as discrete components.

[0166] An embodiment of the present application also provides a computer program product, which includes a computer program stored in a computer-readable storage medium. At least one processor can read the computer program from the computer-readable storage medium. When at least one processor executes the computer program, the technical solution of the processing method in the above embodiment can be implemented.

[0167] The division of units is only a logical function division, and there may be other divisions in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interface, device or unit, which can be electrical, mechanical or other forms.

[0168] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0169] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0170] If the function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods of various embodiments of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory, random access memory, disk or optical disk, etc., various media that can store program codes.

[0171] Those skilled in the art will appreciate that all or part of the steps of implementing the above-mentioned method embodiments can be completed by hardware associated with program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, the steps of the above-mentioned method embodiments are executed.

[0172] Finally, it should be noted that those skilled in the art will readily conceive of other embodiments of the present invention after considering the specification and practicing the invention disclosed herein. The present invention is intended to cover any variations, uses or adaptations of the present invention, which follow the general principles of the present invention and include common knowledge or customary technical means in the art not disclosed by the present invention, are not limited to the precise structure described above and shown in the drawings, and may be modified and changed in various ways without departing from the scope thereof. The scope of the present invention is limited only by the appended claims.

Claims

1. An obstacle positioning method, characterized in that: include: Acquire a scene image captured by an image acquisition device in the vehicle; and performing cylindrical projection processing on the scene image to obtain a cylindrical image; Performing obstacle detection processing on the cylindrical image to obtain an inclusion frame of at least one target obstacle; wherein the inclusion frame represents the outline of the target obstacle in the cylindrical image; The position information of the target obstacle is determined according to the encompassing frame of the target obstacle.

2. The method according to claim 1, characterized in that Determining the location information of the target obstacle according to the encompassing frame of the target obstacle includes: Determine a first coordinate and a second coordinate of the target obstacle according to the encompassing frame of the target obstacle; wherein the first coordinate is the coordinate of the target obstacle on a first coordinate axis in a preset coordinate system; and the second coordinate is the coordinate of the target obstacle on a second coordinate axis in the preset coordinate system; The position information of the target obstacle is determined according to the first coordinate and the second coordinate.

3. The method according to claim 2, characterized in that Determining the first coordinate and the second coordinate of the target obstacle according to the encompassing frame of the target obstacle includes: Determining calibration parameters of the image acquisition device; The first coordinate and the second coordinate of the target obstacle are determined according to the calibration parameters and the encompassing frame of the target obstacle.

4. The method according to claim 3, characterized in that: Determining a first coordinate and a second coordinate of the target obstacle according to the calibration parameters and the encompassing frame of the target obstacle includes: Determining an obstacle type of the target obstacle, and determining a first coordinate of the target obstacle according to the calibration parameters and the encompassing frame and obstacle type of the target obstacle; The second coordinates of the target obstacle are determined according to the calibration parameters and the first coordinates and the encompassing frame of the target obstacle.

5. The method according to claim 4, characterized in that Determining a first coordinate of the target obstacle according to the calibration parameters and the encompassing frame and obstacle type of the target obstacle includes: If it is determined that the obstacle type of the target obstacle is a single size type, the first coordinate of the target obstacle is determined according to a preset target height, an image height in the encompassing frame of the target obstacle, and a focal length in the calibration parameters; wherein the preset target height is a real height of the target obstacle in the physical world, and the image height is a height of the target obstacle in the cylindrical image; If it is determined that the obstacle type of the target obstacle is a multi-size type, the first coordinate is determined according to the corner point coordinates of the target corner point of the encompassing frame and the focal length, installation height and vanishing point coordinates in the calibration parameters; wherein the target corner point represents the pixel point corresponding to the grounding point of the target obstacle in the cylindrical image; the corner point coordinates are the coordinates of the target corner point in the projection coordinate system; and the vanishing point coordinates are the coordinates of the vanishing point of the image acquisition device in the image coordinate system.

6. The method according to claim 4, characterized in that The encompassing frame has a target corner point; the target corner point represents a pixel point corresponding to a grounding point of the target obstacle in the cylindrical image; Determining the second coordinate of the target obstacle according to the calibration parameters and the first coordinate and the encompassing frame of the target obstacle includes: The second coordinate is determined according to the first coordinate of the target obstacle and the corner point coordinates of the target corner point, as well as the focal length and the image principal point coordinates in the calibration parameters; wherein the corner point coordinates are the coordinates of the target corner point in the projection coordinate system; and the image principal point coordinates are the coordinates of the image principal point of the image acquisition device in the image coordinate system.

7. The method according to claim 2, characterized in that Determining the position information of the target obstacle according to the first coordinate and the second coordinate includes: Determining calibration parameters of the image acquisition device; wherein the calibration parameters include an offset parameter, and the offset parameter represents an offset condition of the image acquisition device; According to the offset parameter, coordinate conversion processing is performed on the first coordinate and the second coordinate respectively to obtain a converted first coordinate and a converted second coordinate; The position information is determined according to the converted first coordinates and the converted second coordinates.

8. An obstacle locating device, characterized in that: include: A projection module, used to obtain a scene image captured by an image acquisition device in the vehicle; and performing cylindrical projection processing on the scene image to obtain a cylindrical image; A detection module, configured to perform obstacle detection processing on the cylindrical image to obtain an inclusion frame of at least one target obstacle; wherein the inclusion frame represents the outline of the target obstacle in the cylindrical image; The determination module is used to determine the position information of the target obstacle according to the encompassing frame of the target obstacle.

9. An electronic device, characterized in that: include: Memory, processor; The memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory, so that the processor performs the method according to any one of claims 1 to 7.

10. A computer-readable storage medium / computer program product, characterized in that: The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the method according to any one of claims 1 to 7; and / or the computer program product includes a computer program, which, when executed by a processor, implements the method according to any one of claims 1 to 7.

Images