A target detection method, device, and vehicle

Abstract

A target detection method is disclosed, which is applied to a mobile body side and includes: acquiring a current image, performing target detection based on the current image to obtain a target identification frame; selecting any first pixel point in a frame containing bearing surface information for supporting the target in the target identification frame; acquiring visual map information at a current positioning information, wherein the visual map information at least includes first global position information of a map point of the bearing surface; projecting the map point to a pixel coordinate system according to the first global position information of the map point in the acquired visual map information to obtain pixel position information of a projection point; and acquiring second global position information corresponding to the first pixel point by using the pixel position information of the first pixel point and the pixel position information of the projection point and the first global position information of the map point corresponding to the projection point, so as to serve as global position information of the target in the target identification frame. The application improves the adaptability of target detection.

Description

Technical Field

[0001] This invention relates to the field of machine vision, and in particular to a target detection method, apparatus, and vehicle. Background Technology

[0002] In image-based, purely visual target detection, not only are the length, width, and height of the target (size information), but also the target's pose, such as its spatial location information.

[0003] Existing monocular image-based target detection methods have the following problems:

[0004] 1) Monocular imaging leads to the loss of depth information;

[0005] 2) Non-deep learning-based object detection methods are generally based on certain assumptions, which limit the adaptability of object detection. Summary of the Invention

[0006] This invention provides a target detection method to improve the adaptability of monocular image-based target detection without relying on a ranging sensor.

[0007] This invention provides a target detection method, applied to, the method comprising: on the moving body side,

[0008] Acquire the current image from the monocular camera, where the current image is a monocular image;

[0009] Target detection is performed based on the current image to obtain the target bounding box;

[0010] Select any first pixel in the border of the target identification frame that contains the bearing surface information for supporting the target;

[0011] Obtain visual map information at the current location of the mobile body, wherein the visual map information includes at least the first global location information of the map points of the bearing surface;

[0012] Based on the first global position information of the map points in the acquired visual map information, the map points are projected onto the pixel coordinate system to obtain the pixel position information of the projected points.

[0013] Using the pixel position information of the first pixel, the pixel position information of the projection point, and the first global position information of the map point corresponding to the projection point, the second global position information corresponding to the first pixel is obtained, which is used as the global position information of the target in the target identifier box.

[0014] Preferably, the step of performing target detection based on the current image to obtain a target bounding box further includes:

[0015] Based on the current image,

[0016] The targets in the target box are classified by attributes to obtain target classification results used to distinguish target attributes.

[0017] The top-view orientation of the target in the target box is classified by angle to obtain the target top-view orientation classification result.

[0018] Preferably, the method further includes:

[0019] Based on the target classification results, determine the spatial size information of the target.

[0020] Based on the target's top-view orientation classification results, the target's top-view orientation is determined.

[0021] Based on the target's global location information, spatial size information, and top-view orientation, the target's spatial information is determined.

[0022] Preferably, the bearing surface is the ground.

[0023] The visual map information used to obtain the current location information of the mobile entity includes:

[0024] Get current location information.

[0025] Based on the location information, ground points within a first predetermined range of the location located by the location information are extracted from the visual map information.

[0026] The captured ground points are identified as the visual map information.

[0027] Preferably, the step of obtaining the second global location information corresponding to the first pixel using the pixel location information of the first pixel, the pixel location information of the projection point, and the first global location information of the map point corresponding to the projection point includes:

[0028] Determine whether the projection points within the second set range of the first pixel meet the conditions for interpolation calculation. If so, use the projection points that meet the interpolation calculation conditions to perform interpolation calculation.

[0029] otherwise,

[0030] The fitting plane is determined using at least three non-collinear projection points.

[0031] Based on the back-projection table, a ray is determined from the first pixel to the corresponding spatial point. The back-projection table is used to characterize the mapping relationship between the pixel position information of a pixel in the distorted image and the spatial position information of its corresponding real spatial point.

[0032] The intersection point of the ray and the fitted plane is determined to obtain the spatial position information of the intersection point, which is used as the second global position information.

[0033] Preferably, determining whether the projection points within the second predetermined range of the first pixel meet the conditions for interpolation calculation includes:

[0034] Based on the distance between the first pixel and each projection point within the second set range, determine whether the number of projection points whose distance is less than a set distance threshold is greater than a set number threshold. If so, it is determined that the conditions for interpolation calculation are met; otherwise, it is determined that the conditions for interpolation calculation are not met.

[0035] Preferably, the interpolation calculation using projection points that satisfy the interpolation calculation conditions includes:

[0036] If the projection points that meet the interpolation calculation conditions also meet the interpolation conditions, then interpolation is performed using the four projection points closest to the first pixel.

[0037] If the projection points that meet the interpolation calculation conditions also meet the extrapolation conditions, extrapolation is performed using the two projection points that are closest to the first pixel.

[0038] Preferably, the back projection table is determined as follows:

[0039] The spatial point position information in the camera coordinate system is normalized to the spatial point position information in the metaplane.

[0040] The distortion coefficients of the camera model are used to perform distortion transformation on the spatial point position information in the metaplane, resulting in the distorted spatial point positions in the metaplane.

[0041] Based on the camera intrinsic parameters, the positions of spatial points in the metaplane after distortion transformation are converted to pixel coordinates to obtain the projected points in the image.

[0042] By cropping the projection points that are outside the image area, we obtain the cropped image.

[0043] In the metaplane, one or more virtual spatial point position information is defined. The position information of each virtual spatial point is then distorted according to the aforementioned distortion coefficients to obtain the distorted position information of each virtual spatial point in the metaplane.

[0044] Based on the camera intrinsic parameters, the position information of each virtual spatial point in the metaplane after distortion transformation is converted to the pixel coordinate system to obtain the pixel position information of each virtual projection point.

[0045] Interpolation calculations are performed using the virtual projection point location information to ensure that the metaplane is filled with spatial points.

[0046] The back projection table is obtained based on the one-to-one correspondence between the position information of each spatial point in the filled metaplane and the position information of each pixel in the pixel coordinate system. The pixels in the pixel coordinate system include: pixels in the cropped image and virtual projection points.

[0047] Preferably, determining the intersection point of the ray and the fitting plane to obtain the spatial location information of the intersection point further includes:

[0048] The spatial location information of the intersection point is corrected based on the target observation angle.

[0049] The spatial location information of the corrected intersection point is determined as the second global location information.

[0050] Preferably, the method further includes:

[0051] Based on the spatial information of the target, determine the spatial bounding box of the target.

[0052] Based on the statistical values ​​of the target's spatial frame, the rotation angle of the target's spatial frame is corrected.

[0053] Based on the spatial dimension statistics of the target, the spatial dimension information of the target is corrected.

[0054] Preferably, selecting any first pixel in the border of the target identification frame that contains information about the bearing surface for supporting the target includes:

[0055] The center point of the lower border of the target identifier is determined as the first pixel.

[0056] A second aspect of the present invention provides a target detection device applied to a moving body side, the device comprising:

[0057] The image acquisition module is used to acquire the current image through a monocular camera;

[0058] The object detection module is used to detect objects based on the current image and obtain object bounding boxes.

[0059] The map information acquisition module is used to acquire visual map information at the current location, wherein the visual map information includes at least the first global location information of the map points on the carrying surface;

[0060] The target spatial information acquisition module is used to select any first pixel in the border of the target identification frame that contains the bearing surface information for supporting the target, and project the map point into the pixel coordinate system according to the first global position information of the map point in the acquired visual map information to obtain the pixel position information of the projected point; using the pixel position information of the first pixel, the pixel position information of the projected point, and the first global position information of the map point corresponding to the projected point, the module obtains the second global position information corresponding to the first pixel, which is used as the global position information of the target in the target identification frame.

[0061] A third aspect of the present invention provides a drivable vehicle, the vehicle comprising:

[0062] The image acquisition module is used to acquire the current image from the monocular camera;

[0063] The object detection module is used to detect objects based on the current image and obtain object bounding boxes.

[0064] The map information acquisition module is used to acquire visual map information at the current location, wherein the visual map information includes at least the first global location information of the map points on the carrying surface;

[0065] The target spatial information acquisition module is used to select any first pixel in the border of the target identification frame that contains the bearing surface information for supporting the target, and project the map point into the pixel coordinate system according to the first global position information of the map point in the acquired visual map information to obtain the pixel position information of the projected point; using the pixel position information of the first pixel, the pixel position information of the projected point, and the first global position information of the map point corresponding to the projected point, the module obtains the second global position information corresponding to the first pixel, which is used as the global position information of the target in the target identification frame.

[0066] A fourth aspect of the present invention provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of any of the target detection methods described above.

[0067] This invention utilizes the pixel position information of any first pixel in the border containing the bearing surface information for supporting the target, as well as the position information of the projection point of the nearby bearing surface map point, to obtain the global position information of the target. This avoids the impact of bearing surface height changes on the target detection results, thereby improving the adaptability to bearing surface height changes. It reduces complexity and improves the accuracy of target detection results under bearing surface height changes, and increases perception redundancy in the absence of laser sensors. Attached Figure Description

[0068] Figure 1 This is a schematic flowchart of a target detection method according to an embodiment of this application.

[0069] Figure 2 This is a schematic diagram showing the longitudinal distance, lateral distance, and target orientation from above.

[0070] Figure 3 This is another flowchart illustrating the target detection method according to an embodiment of this application.

[0071] Figure 4 This is a schematic diagram of an offline ground point cloud.

[0072] Figure 5 It is a schematic diagram of projecting ground points onto an image.

[0073] Figure 6 This is a schematic diagram illustrating the strong correlation between pixel depth and pixel coordinates.

[0074] Figure 7 This is a schematic diagram showing the transformation of the metaplane before and after distortion.

[0075] Figure 8 This is a schematic diagram of a spatial point in the metaplane projected onto an image.

[0076] Figure 9 This is a schematic diagram of the cropped image and its corresponding spatial points in the metaplane.

[0077] Figure 10 A schematic diagram of an image formed by a back-projection table.

[0078] Figure 11 This is a schematic diagram of the target detection result obtained using the target detection method described in this embodiment.

[0079] Figure 12 This is a schematic diagram of a target detection device according to an embodiment of this application.

[0080] Figure 13 This is a schematic diagram of a target detection device according to an embodiment of this application. Detailed Implementation

[0081] To make the objectives, technical means, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings.

[0082] This application obtains the spatial position information of a pixel by using the projection points near the pixel within the border of the target identification frame, which contains information about the bearing surface used to support the target.

[0083] See Figure 1 As shown, Figure 1This is a schematic flowchart of a target detection method according to an embodiment of this application. The method includes:

[0084] Step 101: Acquire the current image using a monocular camera;

[0085] Step 102: Perform target detection based on the current image to obtain the target bounding box;

[0086] Step 103: Select any first pixel in the border of the target identification frame that contains the bearing surface information for supporting the target;

[0087] Step 104: Obtain the visual map information at the current location, wherein the visual map information includes at least the first global location information of the ground point;

[0088] Step 105: Based on the first global position information of the map points in the acquired visual map information, project the map points onto the image to obtain the pixel position information of the projected points.

[0089] Step 106: Using the pixel position information of the first pixel, the pixel position information of the projection point, and the first global position information of the map point corresponding to the projection point, obtain the second global position information corresponding to the first pixel, so as to serve as the global position information of the target in the target identifier box.

[0090] In the above steps, steps 101-103 and steps 104-105 are not strictly ordered and can be performed in parallel.

[0091] This application embodiment utilizes the characteristic that the first pixel obtained by target detection is a ground point, so that the global position information of the target can overcome the influence of different bearing surface slopes, which is beneficial to improving the accuracy of the global position information of the target and improving the adaptability of the target detection method based on monocular images.

[0092] To facilitate understanding of this application, the following description uses the example of a vehicle in an autonomous driving environment using monocular images for target perception. It should be understood that the embodiments of this application are not limited to the autonomous driving environment of a vehicle. Any other application scenario that requires target detection on the mobile body side can be applied. The mobile body in this application can be an autonomous driving device, such as an autonomous vehicle, a transport robot, an automatic sweeping robot, etc.

[0093] The following explains the terminology used in the embodiments of this disclosure.

[0094] Ground: The supporting surface that can support the target object; any physical supporting surface can be considered as the ground.

[0095] Camera metaplane: The plane at z=1 in the camera coordinate system is the metaplane.

[0096] Longitudinal distance: The distance between the target and the vehicle in the current direction of movement.

[0097] Lateral distance: The distance between the target and the vehicle in a direction perpendicular to the vehicle's current direction of movement and parallel to the ground (support surface).

[0098] Target top-down orientation: The angle between the target and the vehicle's current direction of movement.

[0099] The above longitudinal distance, lateral distance, and target orientation can be found in [reference needed]. Figure 2 As shown.

[0100] Compared to Advanced Driver Assistance Systems (ADAS), the planning and decision-making in autonomous driving systems requires a much higher level of target perception, specifically including:

[0101] 1) It is necessary not only to perceive the dangers and important targets ahead that affect driving behavior, but also to perceive all traffic participants in the drivable area;

[0102] 2) It is necessary not only to know the longitudinal distance of the target, but also to know the lateral distance of the target, the three-dimensional size of the target, and the target's top-view orientation;

[0103] 3) It is suitable not only for structured roads, but also for all unstructured roads.

[0104] In the context of autonomous driving, in order to improve and solve the bottlenecks and defects of target detection based on pure vision in autonomous driving systems, and to increase the perception redundancy of autonomous driving systems in the absence of other laser sensors, this embodiment utilizes the offline visual map carried by the autonomous driving system itself to achieve target detection.

[0105] See Figure 3 As shown, Figure 3 This is a schematic flowchart of a target detection method according to an embodiment of this application. The method includes: on the vehicle side,

[0106] Step 301: Acquire the current image in real time using a monocular camera;

[0107] Step 302: Obtain the current location information of the vehicle.

[0108] As one implementation method, the current positioning information of the vehicle is determined by matching the acquired current image with a visual map, wherein the visual map includes at least the first global position information of ground points, and the visual map can be a high-precision map.

[0109] High-precision maps are thematic maps, also known as high-resolution maps, which are different from ordinary navigation electronic maps. Their absolute positional accuracy is close to several meters, and their relative positional accuracy is at the centimeter level. The data is organized by using different layers to describe information such as water systems, railways, streets, and buildings, and then the layers are overlaid to express the data.

[0110] Visual maps can be stored offline and retrieved online. To reduce storage space usage, visual maps can be loaded and stored by region growth based on a portion of the map.

[0111] As another implementation method, the current location information of the vehicle body is obtained through a positioning system.

[0112] Step 303: Based on the vehicle's current positioning information, obtain ground points within the current first set range from the visual map. These ground points form a set of ground points to perceive the surrounding ground environment.

[0113] As one implementation method, ground points in a defined area are extracted from the vicinity of the currently located position in the visual map. The defined area is a region with a defined range starting from the currently located position and moving along the current direction, thereby cropping ground points from the visual map to perceive surrounding ground information.

[0114] In this embodiment, the visual map can be a point cloud map layer composed of map points in a high-precision map.

[0115] See Figure 4 As shown, Figure 4 This is a schematic diagram of an offline ground point cloud. The numerous dense map points in the diagram represent ground points (gray areas).

[0116] Step 304: Obtain the projection points of the captured ground points in the image.

[0117] As one implementation method, based on the camera's extrinsic parameters and the vehicle's pose (i.e., the camera pose, obtained from the currently acquired image), the first global position information of the captured ground point is projected onto the position information in the camera coordinate system. Then, based on the camera's intrinsic parameters, the position information of the projection point projected into the camera coordinate system is projected onto the pixel coordinate system, thereby obtaining the projection position of the captured ground point in the pixel coordinate system, and thus obtaining the pixel position information of the projection point. For ease of writing, the pixel position information of the projection point is denoted as ground_uv.

[0118] Step 305: Perform target detection based on the acquired current image to obtain the target bounding box, target category, and target top-down orientation;

[0119] As one implementation method, a two-dimensional target detector is used to detect targets in the current image. The detector can be a detector with a machine learning model. The machine learning model can be a supervised learning algorithm, or a semi-supervised, self-supervised, or unsupervised algorithm.

[0120] Given that existing object detectors can output object bounding boxes and object categories (object classification results), but directly detecting the orientation of an object in a top-view image from a 2D image is difficult, the detection of the target's top-view orientation is transformed into angle classification. That is, according to a set number of angle classification categories, the angle classification information of the target's top-view orientation is output (target top-view orientation classification results). For example, 360 degrees can be divided into twelve categories, with a 30-degree interval between adjacent categories. The supervision information for the target's top-view orientation can use the projection results of the bounding boxes in the 3D object detection dataset onto the top view, for example... Figure 11 The label box in the BEV diagram.

[0121] Target categories are used to distinguish target attributes. For example, they can include two basic categories: people and objects. They can also include categories with more attributes, such as people, animals, roadblocks, vehicles, and buildings.

[0122] The purpose of obtaining the target category in this step is to subsequently use the target clustering spatial size information in the training sample set, such as three-dimensional size information, to determine the spatial size information of the target based on the target category.

[0123] Step 306: Based on the detected target identification box, obtain the pixel position information of any first pixel in the border of the target identification box that contains ground information for supporting the target, and determine the spatial size information of the target category according to the target category.

[0124] As one implementation, since the border of the ground information used to support the target is usually located at the bottom border of the target identification box, any first pixel point located in the bottom border can be selected. Preferably, the first pixel point can be the center point located at the center of the bottom border. For ease of writing, the pixel position information of the obtained first pixel point is denoted as box_uv.

[0125] The purpose of obtaining the pixel position information of the first pixel in this step is to subsequently determine the spatial position information of the target, such as three-dimensional position information, by assuming that the spatial point corresponding to the first pixel is located on the ground, and by using interpolation or triangulation of the ground point.

[0126] As an example, a pre-defined correspondence between target categories and target spatial size information can be established, such as the spatial size information of a person, the spatial size information of a road obstacle, etc. In this way, the spatial size information of the target category can be determined based on the target category and the pre-defined correspondence. Alternatively, the spatial size information of the target category can be determined by combining the target cluster size information in the training set.

[0127] Step 307: Calculate the distance between each projection point within the second defined range and the first pixel.

[0128] As one implementation method, projection points within a second predetermined range located in the neighborhood of the first pixel are selected, and the Manhattan distance between each projection point and the first pixel is calculated, or it can be the Euclidean distance.

[0129] Step 308: Determine whether the number of projection points whose calculated distance is less than the set distance threshold is greater than the set number threshold. If so, proceed to step 309 to obtain the spatial location information of the target by interpolation calculation. Otherwise, proceed to step 310 to obtain the spatial location information of the target by plane fitting.

[0130] Among them, the spatial location information is global location information.

[0131] Step 309: Determine the interpolation calculation method based on the interpolation conditions.

[0132] Specifically:

[0133] To determine if the interpolation conditions are met, specifically, to determine if there are at least four uniformly distributed projection points within the first neighborhood of the first pixel. If so, bilinear interpolation is performed using the first global position information of the ground points corresponding to the four nearest projection points to obtain the spatial position information of the first pixel, which is referred to as the second global position information for convenience. Then, step 311 is executed.

[0134] The interpolation condition is that there are at least four projection points distributed in different directions within the first neighborhood of the first pixel, such as projection points distributed in the upper left, lower left, upper right, and lower right directions of the first pixel.

[0135] Otherwise, determine whether the interpolation conditions are met. If they are, use the first global position information of the ground points corresponding to the two closest projection points to perform bilinear interpolation or a Gaussian process to obtain the spatial position information of the first pixel, i.e. the second global position information, and then execute step 311. Otherwise, execute step 310.

[0136] The extrapolation condition is that there are two projection points located in different directions within the second neighborhood of the first pixel. For example, projection points distributed in any two of the four directions of the pixel: upper left, lower left, upper right, and lower right.

[0137] See Figure 5 As shown, Figure 5 It is a schematic diagram of projecting ground points onto an image. Based on the assumption that the center point of the bottom border of the target bounding box is located on the ground, the spatial location information of the center point can be obtained by interpolating ground points near the center point.

[0138] As an example, if a fisheye camera is used and distortion is not corrected, the pixel depth and pixel coordinates of a fisheye camera are strongly correlated, such as... Figure 6 As shown, the spatial location information of the first pixel is simultaneously bilinearly interpolated using both the horizontal pixel coordinate u and the vertical pixel coordinate v in the pixel coordinate system.

[0139] Step 310: Estimate the ground information near the first pixel by fitting the plane, and then obtain the second spatial position information of the first pixel by intersecting the ray from the first pixel to the corresponding spatial point determined by the back projection table with the fitting plane.

[0140] As an example, when there are few projection points near the first pixel in the image, or the distance between the projection points and the first pixel is far, making it difficult to obtain the spatial location information of the first pixel by interpolation, a fitting plane is determined by using at least three non-collinear projection points. The ground information near the first pixel is estimated by using the fitting plane. For example, the RANSAC algorithm is used to obtain the fitting plane or the default plane and estimate the ground point information near the first pixel.

[0141] Since the acquired images are usually distorted, in order to avoid errors in the spatial position information of the first pixel and improve the accuracy of the spatial position information of the first pixel, the spatial position information of the first pixel is calculated by combining the back projection table used to characterize the mapping relationship between each pixel in the distorted image and the real spatial point.

[0142] To make it easier to understand, the following explanation will use the back projection table of a fisheye camera as an example.

[0143] If a fisheye camera is used and the distortion is not corrected, a back projection table needs to be established using the intrinsic parameters of the fisheye camera and the distortion parameters. This back projection table is a mapping relationship between the pixel position information of each pixel in the distorted image and the spatial position information of the real space point. If the camera coordinate system is normalized to the metaplane, the back projection table is a mapping relationship between the pixel position information of each pixel in the distorted image and the position information of the space point in the metaplane.

[0144] The back projection table can be obtained in the following way:

[0145] 1) Normalize the spatial point position information (x, y, z) in the camera coordinate system to the metaplane, and its position information in the metaplane (X, Y, Z) = (x / z, y / z, 1);

[0146] 2) Perform distortion transformation on the spatial point position information (X,Y,Z) in the metaplane. Using the distortion coefficients of the fisheye model, transform the spatial point position information (X,Y,Z) in the metaplane to obtain the distorted spatial point position information (X',Y',Z'), such as... Figure 7 This is a schematic diagram of the metaplane before and after distortion transformation. The gray area in the diagram is the region where spatial points are formed. In the diagram, the left image shows the distribution of spatial points in the metaplane before distortion, and the right image shows the distribution of spatial points in the metaplane after distortion.

[0147] 3) Based on the camera's intrinsic parameters, the spatial point position information on the metaplane after distortion transformation is converted to the pixel coordinate system, thus obtaining the projected points in the image. For example... Figure 8 As shown, Figure 8 This is a schematic diagram of projecting spatial points in the metaplane onto an image. Some projected points have pixel coordinates outside the pixel coordinate system's range. For example, if the image resolution is 1280*720, the horizontal pixel range is [0, 1280] and the vertical pixel range is [0, 720]. If the calculated pixel position after distortion exceeds both the horizontal and vertical pixel ranges, cropping is required. The cropped image is shown below. Figure 9 As shown in the left image, the final cropped image corresponds to a pillow-like area on the metaplane, as shown in the left image. Figure 9 The right image in the text.

[0148] 4) Given that a plane at a depth of 1 from the camera in the real 3D world (i.e., the metaplane) may not necessarily have real spatial points, based on the camera imaging transformation process described above, one or more virtual spatial points can be created on the metaplane. The position information of these virtual spatial points is then distorted using the distortion coefficients of the fisheye model to obtain the distorted virtual spatial point position information in the metaplane. This distorted information is then projected onto the pixel coordinate system based on the camera's intrinsic parameters to obtain the pixel position information of the virtual projection points. Using the obtained pixel position information of the virtual projection points, bilinear interpolation is performed to fill the metaplane with spatial points. By filling the metaplane with spatial points, a one-to-one correspondence can be obtained between the pixel position information of each pixel in the distorted image and the position information of the spatial points on the metaplane, thus forming a back-projection table. The pixels in the pixel coordinate system include: pixels in the cropped image and virtual projection points.

[0149] See Figure 10 As shown, Figure 10This is a schematic diagram of an image formed by a back-projection table. In the diagram, each pixel has 3 channels, and the 3 channels represent the x, y, and z coordinates of the pixel on the metaplane. Since the depth of the metaplane is 1, z = 1.

[0150] By fitting a plane using at least three non-collinear ground points near the first pixel, a fitted plane is obtained. Using a back-projection table, the real space point corresponding to the first pixel in the distorted image can be determined. Based on the pixel position information of the first pixel and the spatial position information of the real space point corresponding to the first pixel, the equation of the line containing the ray from the first pixel to the real space point or the metaplane space point can be determined. By combining the equation of the line with the equation of the fitted plane, the three-dimensional coordinates of the intersection point of the ray and the fitted plane can be obtained. This intersection point is the spatial position information of the first pixel in the world coordinate system.

[0151] Step 311: Correct the spatial position information of the first pixel according to the observation angle to obtain the corrected second global position information. Determine the corrected second global position information as the global position information of the target, thereby obtaining the lateral distance and longitudinal distance between the target and the vehicle body.

[0152] As an example, the spatial position of the first pixel is rotated to the opposite position by using the target observation angle, i.e., the view frustum angle. The offset of the first pixel after rotation relative to the first pixel before rotation is analyzed at the opposite position. The first pixel before rotation is then analyzed based on this offset.

[0153] Step 312: Determine the spatial information of the target based on the preset size corresponding to the target category, the global position information of the target, and the target's top-down orientation.

[0154] Preferably, in order to obtain a visual representation of the spatial information of the target, a spatial bounding box, such as a three-dimensional bounding box, is determined based on the spatial information of the target.

[0155] Step 313: Correct the spatial identification frame and spatial dimension information.

[0156] As an example, statistical methods are used to correct spatial bounding boxes and spatial dimension information. For instance, the rotation angle of the target's spatial bounding box is corrected based on statistical values ​​of the target's spatial bounding box, and the spatial dimension information of the target is corrected based on statistical values ​​of the target's spatial dimensions. The statistical values ​​can be obtained by fitting the spatial bounding box and spatial dimensions using formulas or deep learning models. The spatial dimensions include length, width, and depth.

[0157] As another example, a geometric method is used to correct the spatial marker frame and spatial dimension information. For example, the offset of the depth z is determined based on the rotation angle and depth z of the spatial marker frame, as well as the approximate length and width, and the spatial marker frame and spatial dimension information are corrected based on the offset.

[0158] See Figure 11 As shown, Figure 11 This is a schematic diagram of the target detection result obtained using the target detection method described in this embodiment. In the figure, the 2D image is a monocular image, the proj image is a schematic diagram of the map point cloud projected onto the monocular image, the 3D image is a visualization of the target detection result, and the bev image is a top view, in which the target identification box is shown.

[0159] This embodiment utilizes the offline map and positioning output of the autonomous driving positioning system to complete the pure visual 3D target detection without the need for additional ranging sensors such as LiDAR. It uses offline ground point cloud and positioning information, which has low computational load and relatively accurate and reliable results with strong interpretability. The spatial position information of the target is obtained by interpolating ground points or fitting a plane, which has a strong adaptability range and can overcome the influence of different slopes on the spatial position information of the target during vehicle driving, thus improving the accuracy of target 3D detection.

[0160] See Figure 12 As shown, Figure 12 This is a schematic diagram of a target detection device based on monocular images according to an embodiment of this application. The device includes:

[0161] The image acquisition module is used to acquire the current image, which is a monocular image.

[0162] The object detection module is used to detect objects based on the current image and obtain object bounding boxes.

[0163] The map information acquisition module is used to acquire visual map information of the current location information, wherein the visual map information includes at least the first global location information of the map points of the carrying surface;

[0164] The target spatial information acquisition module is used to select any first pixel in the border of the target identification frame that contains the bearing surface information for supporting the target, and to project the map point according to the first global position information of the map point in the acquired visual map information to obtain the pixel position information of the projected point; using the pixel position information of the first pixel, the pixel position information of the projected point, and the first global position information of the map point corresponding to the projected point, the module obtains the second global position information corresponding to the first pixel, which is used as the global position information of the target in the target identification frame.

[0165] in,

[0166] The target detection module is also used to classify the target attributes based on the current image to obtain target classification results for distinguishing target attributes, and to classify the target's top-view orientation by angle to obtain target top-view orientation classification results;

[0167] The target spatial information acquisition module includes:

[0168] The pixel selection submodule is used to select any first pixel in the border of the target identification frame that contains the bearing surface information for supporting the target.

[0169] The map point projection submodule is used to project map points based on the first global position information of map points in the acquired visual map information to obtain the pixel position information of the projected points.

[0170] The location information acquisition submodule is used to acquire the second global location information corresponding to the first pixel using the pixel location information of the first pixel, the pixel location information of the projection point, and the first global location information of the map point corresponding to the projection point, so as to use the global location information of the target in the target identification box.

[0171] The target spatial information acquisition module also includes:

[0172] The size information acquisition submodule is used to determine the spatial size information of the target based on the target classification result.

[0173] The orientation information acquisition submodule determines the top-view orientation of the target based on the target's top-view orientation classification results.

[0174] See Figure 13 As shown, Figure 13 This is another schematic diagram of a target detection device based on monocular images according to an embodiment of this application, or a drivable vehicle, mobile robot, etc. The device includes an image acquisition unit, a memory, and a processor. The memory stores a computer program, and the processor is configured to execute the computer program to implement the steps of the target detection method based on monocular images.

[0175] The memory may include random access memory (RAM) or non-volatile memory (NVM), such as at least one disk storage device. Optionally, the memory may also be at least one storage device located remotely from the aforementioned processor.

[0176] The processors mentioned above can be general-purpose processors, including central processing units (CPUs), network processors (NPs), etc.; they can also be digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0177] This invention also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of the target detection method based on monocular images.

[0178] For the device / network-side equipment / storage medium embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and relevant parts can be referred to in the description of the method embodiments.

[0179] In this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, without necessarily requiring or implying any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0180] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A target detection method, characterized in that, The method includes: on the moving body side, Acquire the current image using a monocular camera; Target detection is performed based on the current image to obtain the target bounding box; Select any first pixel in the border of the target identification frame that contains the bearing surface information for supporting the target; Obtain visual map information at the current location of the mobile body, wherein the visual map information includes at least the first global location information of the map points of the bearing surface; Based on the first global position information of the map points in the acquired visual map information, the map points are projected onto the pixel coordinate system to obtain the pixel position information of the projected points. Using the pixel position information of the first pixel, the pixel position information of the projection point, and the first global position information of the map point corresponding to the projection point, the second global position information corresponding to the first pixel is obtained, which is used as the global position information of the target in the target identification box; in, The step of obtaining the second global location information corresponding to the first pixel using the pixel location information of the first pixel, the pixel location information of the projection point, and the first global location information of the map point corresponding to the projection point includes: When the projection points within the second set range of the first pixel do not meet the interpolation calculation conditions, The fitting plane is determined using at least three non-collinear projection points. Based on the back-projection table, a ray is determined from the first pixel to the corresponding spatial point. The back-projection table is used to characterize the mapping relationship between the pixel position information of a pixel in the distorted image and the spatial position information of its corresponding real spatial point. The intersection point of the ray and the fitted plane is determined to obtain the spatial position information of the intersection point, which is used as the second global position information.

2. The target detection method as described in claim 1, characterized in that, The step of performing target detection based on the current image to obtain a target bounding box further includes: Based on the current image, The targets in the target identification box are classified by attribute to obtain target classification results used to distinguish target attributes. The top-view orientation of the target in the target identification frame is classified by angle to obtain the target top-view orientation classification result.

3. The target detection method as described in claim 2, characterized in that, The method further includes: Based on the target classification results, determine the spatial size information of the target. Based on the target's top-view orientation classification results, the target's top-view orientation is determined. Based on the target's global location information, spatial size information, and top-view orientation, the target's spatial information is determined.

4. The target detection method as described in claim 1, characterized in that, The bearing surface is the ground. The visual map information used to obtain the current location information includes: Obtain the current location information of the mobile body. Based on the location information, ground points within a first predetermined range of the location located by the location information are extracted from the visual map information. The captured ground points are identified as the visual map information.

5. The target detection method as described in claim 1, characterized in that, The step of obtaining the second global location information corresponding to the first pixel using the pixel location information of the first pixel, the pixel location information of the projection point, and the first global location information of the map point corresponding to the projection point further includes: Determine whether the projection points within the second set range of the first pixel meet the conditions for interpolation calculation. If so, then interpolation calculations are performed using projection points that satisfy the interpolation calculation conditions.

6. The target detection method as described in claim 5, characterized in that, The step of determining whether the projection points within the second set range of the first pixel meet the conditions for interpolation calculation includes: Based on the distance between the first pixel and each projection point within the second set range, determine whether the number of projection points whose distance is less than a set distance threshold is greater than a set number threshold. If so, it is determined that the conditions for interpolation calculation are met; otherwise, it is determined that the conditions for interpolation calculation are not met.

7. The target detection method as described in claim 5, characterized in that, The interpolation calculation using projection points that satisfy the interpolation calculation conditions includes: If the projection points that meet the interpolation calculation conditions also meet the interpolation conditions, then interpolation is performed using the four projection points closest to the first pixel. If the projection points that meet the interpolation calculation conditions also meet the extrapolation conditions, extrapolation is performed using the two projection points that are closest to the first pixel.

8. The target detection method as described in claim 5, characterized in that, The back projection table is determined as follows: The spatial point position information in the camera coordinate system is normalized to the spatial point position information in the metaplane. The distortion coefficients of the camera model are used to perform distortion transformation on the spatial point position information in the metaplane, resulting in the distorted spatial point positions in the metaplane. Based on the camera intrinsic parameters, the positions of spatial points in the metaplane after distortion transformation are converted to pixel coordinates to obtain the projected points in the image. By cropping the projection points that are outside the image area, we obtain the cropped image. In the metaplane, one or more virtual spatial point position information is defined. The position information of each virtual spatial point is then distorted according to the aforementioned distortion coefficients to obtain the distorted position information of each virtual spatial point in the metaplane. Based on the camera intrinsic parameters, the position information of each virtual spatial point in the metaplane after distortion transformation is converted to the pixel coordinate system to obtain the pixel position information of each virtual projection point. Interpolation calculations are performed using the virtual projection point location information to ensure that the metaplane is filled with spatial points. The back projection table is obtained based on the one-to-one correspondence between the position information of each spatial point in the filled metaplane and the position information of each pixel in the pixel coordinate system. The pixels in the pixel coordinate system include: pixels in the cropped image and virtual projection points.

9. The target detection method as described in claim 5, characterized in that, The step of determining the intersection point of the ray and the fitted plane to obtain the spatial location information of the intersection point further includes: The spatial location information of the intersection point is corrected based on the target observation angle. The spatial location information of the corrected intersection point is determined as the second global location information.

10. The target detection method as described in claim 3, characterized in that, The method further includes: Based on the spatial information of the target, determine the spatial bounding box of the target. Based on the statistical values ​​of the target's spatial frame, the rotation angle of the target's spatial frame is corrected. Based on the spatial dimension statistics of the target, the spatial dimension information of the target is corrected.

11. The target detection method as described in claim 1, characterized in that, Selecting any first pixel point in the border of the target identification frame that contains the bearing surface information for supporting the target includes: The center point of the lower border of the target identifier is determined as the first pixel.

12. A target detection device, characterized in that, The device includes: The image acquisition module is used to acquire the current image through a monocular camera; The object detection module is used to detect objects based on the current image and obtain object bounding boxes. The map information acquisition module is used to acquire visual map information at the current location of the mobile body, wherein the visual map information includes at least the first global location information of the map points on the carrying surface; The target spatial information acquisition module is used to select any first pixel in the border of the target identification frame that contains the bearing surface information for supporting the target; according to the first global position information of the map point in the acquired visual map information, the map point is projected into the pixel coordinate system to obtain the pixel position information of the projected point; using the pixel position information of the first pixel, the pixel position information of the projected point, and the first global position information of the map point corresponding to the projected point, the second global position information corresponding to the first pixel is obtained as the global position information of the target in the target identification frame; in, The step of obtaining the second global location information corresponding to the first pixel using the pixel location information of the first pixel, the pixel location information of the projection point, and the first global location information of the map point corresponding to the projection point includes: When the projection points within the second set range of the first pixel do not meet the interpolation calculation conditions, The fitting plane is determined using at least three non-collinear projection points. Based on the back-projection table, a ray is determined from the first pixel to the corresponding spatial point. The back-projection table is used to characterize the mapping relationship between the pixel position information of a pixel in the distorted image and the spatial position information of its corresponding real spatial point. The intersection point of the ray and the fitted plane is determined to obtain the spatial position information of the intersection point, which is used as the second global position information.

13. A vehicle, characterized in that, The vehicle includes a processor that executes a computer program to implement the steps of the target detection method as described in any one of claims 1 to 11.

Images