Target detection method and device, vehicle and computer readable storage medium
By extracting the height of the target and the height in the image in the monocular camera, combining the focal length and geometric depth decomposition method, the problem of the monocular camera lacking depth information is solved, and the accuracy of target detection is improved, especially in night scenes, The demand for intelligent driving.
Patent Information
- Application Number
- CN202410955473.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-16
- Publication Date
- 2024-11-26
AI Technical Summary
The lack of depth information of the monocular camera makes the target depth difficult to estimate, especially in night scenes, resulting in low target detection prediction accuracy.
By acquiring the images captured by the monocular camera, extracting the actual height of the target and the height in the image, combining the focal length of the camera, estimating the depth of the target using geometric depth decomposition method, and combining the multi-scale feature fusion and detection head design, improve the depth of the target estimation accuracy.
It improves the depth estimation accuracy of the monocular camera in object detection, enhances the accuracy of object detection in night scenes, and meets the needs of intelligent driving at night.
Smart Images

Figure CN119027910A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of intelligent driving technology, and more specifically, to a target detection method, device, vehicle and computer-readable storage medium. Background Art
[0002] Intelligent driving technology includes autonomous driving technology and assisted driving technology. Intelligent driving technology uses computer vision and deep learning technology for target detection to perceive obstacles around the vehicle, such as other vehicles besides the vehicle.
[0003] However, current target detection methods include monocular target detection, that is, using a monocular camera to capture an image and performing target detection based on the image. However, the monocular camera lacks depth information, which makes it difficult to estimate the depth of the target, especially in night scenes, where estimating the depth of the target is more challenging, resulting in low prediction accuracy for target detection using a monocular camera. Summary of the invention
[0004] The embodiments of the present application propose a target detection method, device, vehicle and computer-readable storage medium to solve the technical problem that the target depth is difficult to estimate due to the lack of depth information of a monocular camera.
[0005] In a first aspect, an embodiment of the present application provides a target detection method, the method comprising: acquiring an image captured by a monocular camera of a vehicle; extracting a target in the image, acquiring an actual height of the target and the height of the target in the image; determining the depth of the target based on the focal length of the monocular camera, the actual height of the target and the height of the target in the image.
[0006] In a second aspect, an embodiment of the present application provides a target detection device, which includes: an image acquisition module for acquiring an image captured by a monocular camera of a vehicle; a target detection module for extracting a target in the image and acquiring an actual height of the target and the height of the target in the image; and a result generation module for determining the depth of the target based on the focal length of the monocular camera, the actual height of the target and the height of the target in the image.
[0007] In a third aspect, an embodiment of the present application provides a vehicle, comprising: a memory and a processor, wherein an application is stored in the memory, and the application is used to enable the processor to execute the method provided by the embodiment of the present application when called by the processor.
[0008] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium having a program code stored thereon, wherein the program code is used to enable the processor to execute the method provided by the embodiment of the present application when called by the processor.
[0009] The target detection method provided in the embodiment of the present application has the following technical effects: utilizing the characteristic that the visual height of the target in the image is less affected by occlusion or truncation than the length and width of the target and is easier to estimate, the depth of the target is determined according to the actual height of the target in the image and the height in the image, thereby estimating the target depth based on the easily estimated visual height in the image captured by the monocular camera, which can improve the accuracy of target depth estimation and solve the technical problem that the monocular camera lacks depth information, making it difficult to estimate the target depth, thereby improving the target prediction accuracy when using a monocular camera for target detection. BRIEF DESCRIPTION OF THE DRAWINGS
[0010] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following is a brief introduction to the drawings required for the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application, not all embodiments. Based on the embodiments of the present application, all other embodiments and drawings obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.
[0011] Figure 1 A schematic diagram of a process flow of a target detection method provided by an embodiment of the present application is shown;
[0012] Figure 2 A schematic diagram showing a camera imaging principle provided by an exemplary embodiment of the present application is shown;
[0013] Figure 3 A grayscale image of a target detection result in a daytime scene provided by an exemplary embodiment of the present application is shown;
[0014] Figure 4 A grayscale image of a target detection result in a night scene provided by an exemplary embodiment of the present application is shown;
[0015] Figure 5 A schematic diagram of a process flow of a target detection method provided by another embodiment of the present application is shown;
[0016] Figure 6 A schematic diagram showing two-dimensional attribute information of a target provided by an exemplary embodiment of the present application is shown;
[0017] Figure 7 A schematic diagram showing three-dimensional attribute information of a target provided by an exemplary embodiment of the present application is shown;
[0018] Figure 8 A schematic diagram showing the structure of a model framework of a three-dimensional object detection model provided in an embodiment of the present application is shown;
[0019] Fig. 9A schematic diagram showing the framework structure of a detection head in a three-dimensional target detection model provided in an embodiment of the present application is shown;
[0020] Fig.10 A schematic diagram showing the orientation angle and observation angle of a target provided in an embodiment of the present application is shown;
[0021] Fig.11 A grayscale image of a target's three-dimensional features from a front-viewing perspective of a vehicle provided by an exemplary embodiment of the present application is shown;
[0022] Fig.12 A grayscale image of a target's three-dimensional features from a top-down perspective of a vehicle provided by an exemplary embodiment of the present application is shown;
[0023] Fig.13 A schematic diagram of the structure of a target detection device provided in an embodiment of the present application is shown;
[0024] Fig.14 A schematic structural diagram of a vehicle provided in an embodiment of the present application is shown. DETAILED DESCRIPTION
[0025] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application.
[0026] The target detection method in the embodiment of the present application can be applied to a target detection device or a vehicle. The target detection device can be integrated in a vehicle. The vehicle can include but is not limited to a gasoline vehicle, an electric vehicle, and a gasoline-electric hybrid vehicle. At least a monocular camera is installed on the vehicle, and the monocular camera is used to capture the scene in front of the vehicle to obtain an image.
[0027] See also Figure 1 , Figure 1 FIG. 1 is a flow chart of a target detection method provided by an embodiment of the present application. Figure 1 As shown, the target detection method may include steps S110 to S130.
[0028] Step S110: Acquire an image captured by the monocular camera of the vehicle.
[0029] The image may include an image captured in a night scene or a day scene. During the driving of the vehicle, the image may be captured by a monocular camera of the vehicle. As an example, the image resolution of the image may be greater than or equal to 720 pixels, and the acquisition frequency of the monocular camera may be greater than or equal to 30 Hz.
[0030] In some embodiments, when the vehicle is powered on and in the intelligent driving mode, the monocular camera of the vehicle is exposed and kept turned on. During the driving process of the vehicle, the image is continuously captured to obtain a video stream of the road in front of the vehicle. The monocular camera transmits the video stream to the on-board computing unit, and the on-board computing unit is used to execute the target detection method in the embodiment of the present application. For example, the on-board computing unit can be a graphics processing unit (GPU). Among them, the intelligent driving mode can include but is not limited to the automatic driving mode or the assisted driving mode.
[0031] It is understandable that three-dimensional (3D) object detection is one of the key environmental perception technologies in the field of intelligent driving. The goal of 3D object detection is to identify objects in images and give accurate 3D position and posture information of objects. Current 3D object detection methods mainly rely on depth sensors, such as lidar and depth cameras. Depth sensors are expensive and easily affected by the environment. Compared with the use of expensive depth sensors, this application uses a monocular camera for 3D object detection, which has the advantages of low cost, long life, easy installation, and higher image detection efficiency.
[0032] In some embodiments, after obtaining the image, the image may be preprocessed, resized, normalized, etc., so as to improve the image quality and subsequent target detection results. The preprocessing may include but is not limited to de-distortion processing or denoising processing. The length, width, and height of the resized image may be 512×1720.
[0033] Step S120: extracting the target in the image, and obtaining the actual height of the target and the height of the target in the image.
[0034] In the embodiment of the present application, the target may be an object that is desired to be detected from an image, and the target may be pre-set according to actual needs. For example, the target may be a vehicle, a pedestrian, or other obstacles.
[0035] In the present application, the actual height of the target refers to the height of the target in the real world. The height of the target in the image can be called the visual height. Figure 2 , Figure 2 FIG. 1 is a schematic diagram showing a camera imaging principle provided by an exemplary embodiment of the present application. Figure 2 As shown, assuming that the target is a vehicle, the height of the vehicle target in the image (imaging plane) is h (visual height), and the actual height of the vehicle target in the real world (physical world) is H. For a detailed description of how to obtain the actual height of the target and the height of the target in the image, please refer to step S220.
[0036] Step S130: Determine the depth of the target according to the focal length of the monocular camera, the actual height of the target and the height of the target in the image.
[0037] like Figure 2 As shown, according to the camera imaging principle, taking the target as a vehicle as an example, the depth z of the vehicle target refers to the distance from the camera optical center to the vehicle target, the actual height of the vehicle target is H meters, and the visual height of the vehicle target in the image plane is h pixels. As shown in the expression below this paragraph, the relationship between the visual height h and the depth z can be established based on similar triangles. Furthermore, the depth z can be calculated by the focal length f of the monocular camera, the actual height H of the vehicle target, and the visual height h (or its reciprocal h rec ) is expressed. After decomposing the depth z, due to H and the inverse h rec Having a negative correlation can minimize the error of a certain variable and thus obtain accurate results.
[0038]
[0039] Where z represents the depth of the target, f represents the focal length of the monocular camera of the vehicle; H represents the actual height of the target, h represents the visual height of the target in the image, and h rec Indicates the reciprocal of visual height.
[0040] In an embodiment of the present application, a geometry-based depth decomposition estimation method is designed based on the camera imaging principle, which can solve the technical problem that the monocular camera lacks depth information, making it difficult to estimate the target, thereby improving the target detection accuracy when using a monocular camera for target detection.
[0041] Steps S110 to S130 have the following technical effects: utilizing the characteristic that the visual height of the target in the image is less affected by occlusion or truncation than the length and width of the target and is easier to estimate, the depth of the target is determined according to the actual height of the target in the image and the height in the image, thereby estimating the depth of the target based on the easily estimated visual height in the image captured by the monocular camera, which can improve the accuracy of target depth estimation and solve the technical problem that the monocular camera lacks depth information, making it difficult to estimate the target depth, thereby improving the target prediction accuracy when using a monocular camera for target detection.
[0042] See also Figure 3 and 4 , Figure 3 FIG. 1 shows a grayscale image of a target detection result in a daytime scene provided by an exemplary embodiment of the present application. Figure 4 The grayscale image of the target detection result in a night scene provided by an exemplary embodiment of the present application is shown. The inventors found in the long-term research on target detection that the current target detection is mainly used in scenes such as daytime and normal lighting conditions, such as Figure 3 and Figure 4 As shown in the figure, the lighting conditions of night scenes are more complex than those of day scenes. The size and shape of targets in night scenes are uncertain, and there is a large difference in the difficulty of detecting objects of the same scale in day scenes and night scenes. The lighting conditions in night scenes are complex. Due to insufficient light, shadows, blur, etc., it is difficult to accurately extract the visual features of targets (such as other vehicles besides the vehicle) in night scenes, resulting in insufficient accuracy of target detection in night scenes, which is difficult to meet the night driving needs of intelligent driving.
[0043] Based on this, the present application can also improve the target depth estimation accuracy based on steps S110 to S130, and combine the target's bounding box information, position, length, width, height, orientation angle and depth in the image to determine the target's three-dimensional features, thereby improving the target detection accuracy in night scenes and solving the technical problem that the target detection accuracy in night scenes is insufficient and it is difficult to meet the needs of night driving. For details, please refer to Figure 5 , Figure 5 FIG. 2 is a flow chart of a target detection method provided by another embodiment of the present application. Figure 5 As shown, the target detection method may include steps S210 to S230.
[0044] Step S210: Acquire an image captured by the monocular camera of the vehicle.
[0045] Step S220: extract the target in the image, obtain the target's bounding box information, position, length, width, height, orientation angle, actual height, and height of the target in the image.
[0046] In an embodiment of the present application, the bounding box information of the target may include: the bounding box of the target in the image (2D), the category of the bounding box, and the confidence of the bounding box. The confidence refers to the probability that the target in the bounding box is an object (such as a vehicle) corresponding to the category of the bounding box. The greater the confidence, the greater the possibility that the target in the bounding box is an object corresponding to the category of the bounding box.
[0047] As an example, see Figure 6 , Figure 6 FIG. 1 is a schematic diagram showing two-dimensional attribute information of a target provided by an exemplary embodiment of the present application. Figure 6 The vehicle in the figure is the target, then the bounding box information of the vehicle target can be as follows Figure 6 shown.
[0048] In an embodiment of the present application, the position and posture of the target may include the position and posture of the target in the camera coordinate system of the monocular camera. The length, width and height of the target may refer to the length, width and height of the target in the camera coordinate system, that is, the three-dimensional length, width and height of the target in the camera coordinate system. The heading angle of the target refers to the global deflection angle or yaw angle of the target. Taking the target as a vehicle as an example, the heading angle of the target may refer to the angle between the front direction of the vehicle target and the positive direction of the x-axis of the monocular camera of the vehicle.
[0049] As an example, see Figure 7 , Figure 7 A schematic diagram of three-dimensional attribute information of a target provided by an exemplary embodiment of the present application is shown. Figure 7 If the vehicle in is the target, the position and posture of the vehicle target can be expressed in the form of (x, y, z). Figure 7 The width (2D) and height (2D) in are the length and width of the bounding box information of the vehicle target, respectively. Figure 7 The length (3D), width (3D), and height (3D) in the image are the length, width, and height of the vehicle target in the image. The yaw (3D) is the orientation angle of the vehicle target.
[0050] In some embodiments, multiple regions of interest in the image can be extracted, and there are targets in the regions of interest; based on the multiple regions of interest, the bounding box information, posture, length, width, height, orientation angle, actual height and height in the image (or the reciprocal of the height) of the target are detected.
[0051] In some embodiments, multiple feature maps with the same channels and different scales can be generated based on the image; multiple candidate boxes containing the target are extracted based on the multiple feature maps; and the regions of interest are aligned for the multiple candidate boxes to obtain multiple regions of interest.
[0052] In some embodiments, a residual network can be used to downsample the image to obtain multiple feature maps with different resolutions and different numbers of channels; a feature pyramid network can be used to perform multi-scale information fusion on multiple feature maps to obtain multiple feature maps with the same channels and different scales.
[0053] In some embodiments, a region proposal network may be used to generate and modify multiple anchor boxes on multiple feature maps respectively to obtain multiple candidate boxes containing the target.
[0054] After obtaining multiple regions of interest, multiple feature maps containing multiple regions of interest can be respectively input into three detection heads, and the three detection heads are respectively used to detect the bounding box information of the target, the position, length, width, height, orientation angle of the target, and the actual height of the target and the inverse of the height in the image; the bounding box information, position, length, width, height, orientation angle, actual height and height in the image (or the inverse of the height) of the target output by the three detection heads are obtained.
[0055] For example, a 3D target detection model can be deployed in the vehicle in advance. The 3D target detection model is used to extract the target in the image, obtain and output the target's bounding box information, position, length, width, height, orientation angle, actual height, and height in the image (or the height of the height), so that the image is input into the 3D target detection model to obtain the target's bounding box information, position, length, width, height, orientation angle, actual height, and height in the image (or the height of the height). The 3D target detection model can perform inference calculations on the vehicle's GPU.
[0056] See also Figure 8 , Figure 8 The schematic diagram of the structure of the model framework of the 3D object detection model provided in the embodiment of the present application is shown. The 3D object detection model may include an encoder ResNet50-FPN, a region proposal network (RPN for short) and a detection head.
[0057] ResNet50 is a deep residual network consisting of an initial block and four stages. Each stage contains multiple residual blocks (Residual Block), and residual learning is achieved through jump connections. ResNet50 can effectively solve the gradient vanishing problem in deep neural networks and improve model performance. Among them, the residual block is an important component of the residual network. It is a neural network module composed of residual connections, usually composed of two or three groups of convolutional layers plus activation functions, and residual connections are added. The residual block can connect the activation values of the deep and shallow layers of the network through jump connections to solve the problems of gradient vanishing and gradient explosion. That is, each residual block contains multiple convolutional layers, batch normalization layers, activation functions and residual connections.
[0058] Feature Pyramid Network (FPN) is a multi-scale feature fusion network for target detection. It fuses information of different scales by constructing a feature pyramid, where the scale can refer to the size or distance of an object in an image. The core idea of FPN is to combine high-resolution underlying features with top-level features of high semantic information to improve the accuracy of target detection. FPN achieves multi-scale feature fusion by connecting feature maps of different levels sideways. It uses low-level and high-level features for bottom-up and top-down connections, which can improve the accuracy of target detection.
[0059] ResNet50-FPN is a network structure that combines the deep features of ResNet50 and the multi-scale information fusion capability of FPN. ResNet50-FPN downsamples the image (i.e., the input image) through the residual block of ResNet50 to generate multiple feature maps with different resolutions and different numbers of channels. Then, multiple feature maps with the same channels and different scales are generated based on multiple feature maps through FPN. Among them, downsampling is a processing method that reduces the resolution or sampling frequency of an image. The purpose of downsampling is to make the image fit the size of the display area, generate thumbnails, extract features, reduce file size, speed up processing, or reduce noise. The process of downsampling is a process of information loss, which is usually achieved by removing pixels or using convolution layers. The feature map is the response of the convolution kernel to the input image, that is, the output of the convolution. The number of channels of the feature map is equal to the number of convolution kernels, and each channel encodes a different visual pattern. The more channels a feature map has, the more features of the image can be extracted, and a richer feature representation can be obtained, which improves the expression and learning capabilities of the model.
[0060] like Figure 8 As shown, ResNet50 downsamples the image through the residual block to generate 4 sets of feature maps {C 2 ,C 3 , C 4 ,C 5}, the resolutions of the four feature maps correspond to 1 / 4, 1 / 8, 1 / 16 and 1 / 32 of the image, respectively, and the number of channels increases in sequence. The last pooling layer and the fully connected layer of ResNet50 are cancelled, and the four-stage feature maps are input into FPN. FPN fuses the feature maps layer by layer in a top-down and horizontal connection manner, and outputs five feature maps with the same channels but different scales {P 2 ,P 3 ,P 4 ,P 5 ,P 6}, the five feature maps are combined into a pyramid structure. Specifically, as shown in the expression below this paragraph, the feature map C 5 Through two convolutions, we get P 5 , P 5 Perform the maximum pooling operation to obtain P 6 , P 6 The size is C 5 1 / 2 of the top-down direction. 5 At the beginning, the feature map size is doubled by upsampling and combined with the C 4 Fusion. Lateral connection refers to C 4 The number of channels is reduced by convolution operation, and the enlarged P 5 Fusion to obtain P 4 , therefore, P 4 Receive P5 And so on, it is passed forward.
[0061] P 6 =MP(P 5 )
[0062] P 5 =Conv 3×3 (Conv 1×1 (C 5 ))
[0063] P 4 =Conv 1×1 (C 4 )+Up(P 5 )
[0064] P 3 =Conv 1×1 (C 3 )+Up(P 4 )
[0065] P 2 =Conv 1×1 (C 2 )+Up(P 3 )
[0066] Among them, C i is the feature map output by the i-th stage of ResNet50; P i Represents the feature map output by the i-th stage of FPN; Conv(·) 1×1 represents a 1×1 convolution operation, which reduces the number of channels by half; Up(·) represents an upsampling operation, which doubles the size of the feature map; MP(·) represents a maximum pooling operation.
[0067] RPN is a network used to extract candidate boxes. RPN generates an anchor box (AnchorBox) on the feature map through a sliding window. The anchor box is a hypothetical box used to generate candidate regions on the feature map, and the model predicts whether the anchor box contains the target (object). In the target detection task, a series of anchor boxes can be generated on the feature map according to certain rules, and these anchor boxes are regarded as possible candidate regions. The model predicts whether these candidate regions contain the target object. If the target object is contained, it is necessary to further predict the category to which the object belongs. The position of the anchor box is fixed, and it is unlikely that it will coincide with the object bounding box information. Therefore, it is necessary to fine-tune it on the basis of the anchor box to form a predicted box that can accurately describe the position of the object, that is, the candidate box, and the model needs to predict the magnitude of the fine-tuning.
[0068] like Figure 8 As shown, the 5 feature maps generated by FPN {P 2 ,P 3 ,P4 ,P 5 ,P 6}, whose sizes are The five feature maps are input into the RPN, which performs anchor box generation and correction on the feature maps in sequence through a sliding window.
[0069] Anchor frame generation can include: RPN in feature map P 4 The anchor box is generated by sliding window (convolution kernel is 3×3, step size is 1, padding is 1). For each center point of the sliding window, its position corresponding to the original image is calculated, and the anchor box is calculated with this point as the center. The ratios of the anchor box are {1:2, 1:1, 2:1}, and each ratio has 5 areas {32 2 ,64 2 ,128 2 ,256 2 ,512 2}. Therefore, 15 anchor boxes are generated at each pixel position of the sliding window, and all anchor boxes constitute a feature matrix. 6 It is only used to calculate the loss when RPN generates candidate boxes and does not participate in subsequent feature extraction.
[0070] Anchor box correction can include: RPN makes independent predictions on 5 input feature maps, and after the generated feature matrix, two 1×1 convolutions are connected in parallel to predict the anchor box category (i.e., target category) and bounding box regression (BBR) parameters respectively. Each anchor box outputs the category probability class (foreground probability and background probability, i.e., whether it contains the target) and bounding box regression parameters (x, y, w, h). Based on this, the anchor box containing the target is adjusted to the nearest target position, that is, the candidate box is obtained. During the training process, the candidate boxes generated on each feature map are sorted by category, and the top k candidate boxes are selected and sent to the detector. Therefore, 5 feature maps generate a total of 5×k candidate boxes, k=2000 during training and k=1000 during inference. Finally, all 5×k candidate boxes are merged, and the intersection of union (IOU) is calculated with the real bounding box information of the target. Non-maximum suppression (NMS) is performed according to the IOU threshold to remove redundant candidate boxes. The IOU threshold can be set to 0.7.
[0071] BBR is a method for fine-tuning windows. After fine-tuning, the window is closer to the true bounding box information, thereby improving the accuracy of target detection. The window can be represented by a four-dimensional vector (x, y, w, h), which represents the center point coordinates and width and height of the window respectively. The intersection of region proposals or anchor boxes and the true value box is not high, and the position of the target is not detected more accurately. However, BBR can obtain a bounding box information close to the true value box.
[0072] NMS is an image processing technology used to remove overlapping candidate boxes. In target detection, a detector is used to detect possible target objects and give candidate boxes of the location and size of the target objects. However, the same object may be detected multiple times, resulting in multiple candidate boxes. In this case, NMS can be used to remove overlapping candidate boxes and only retain the best candidate box. The basic principle of NMS is to first find all rectangular areas that may contain target objects in the image and arrange them according to their confidence. For each category, the candidate boxes are sorted according to their confidence, and the candidate box with the highest confidence is used as the benchmark.
[0073] This application uses a monocular camera for nighttime target detection. For a monocular camera, objects with the same posture may present different visual effects on the image plane. Even if the orientation angle of the vehicle target relative to the vehicle changes slightly, the appearance features of the vehicle target in the image may be quite different, which will affect the model's accurate recognition of the target. In response to the above-mentioned difficulties in multi-scale target detection of nighttime vehicles, this application uses FPN to construct multi-scale feature maps, and uses RPN to generate candidate boxes on feature maps of different scales, thereby improving the efficiency and accuracy of target detection.
[0074] The detection head is a key part of the 3D object detection model, responsible for generating detection results based on the features in the candidate box extracted by RPN. The detection head can include a region of interest alignment module and three detection heads. The three detection heads can include a 2D attribute detection head, a 3D attribute detection head, and a depth information detection head. The detection head uses a fully connected layer distribution to compress the features and regress the final parameters.
[0075] The ROI alignment module is used to scale the input feature map (i.e., feature matrix) to unify the size of all feature maps to the same size. As an example, see Fig. 9 , Fig. 9 FIG. 1 is a schematic diagram showing the framework structure of the detection head in the three-dimensional target detection model provided in the embodiment of the present application. Fig. 9 As shown, the region of interest alignment module allocates all feature maps {P 2 ,P 3 ,P 4 ,P 5The size of} is unified to 7×7. After the regions of interest are aligned, the feature maps of the same size are input into the three detection heads respectively. Among them, the Flatten layer is a layer that converts multi-dimensional input into one dimension, which is used for the transition from the convolutional layer to the fully connected layer.
[0076] like Fig. 9 As shown, the two-dimensional attribute detection head is used to regress the two-dimensional box (i.e., the above-mentioned bounding box information) information of the target, for example, the pixel coordinates of the target on the image, the bounding box information of the target, the category of the bounding box information, and the confidence of the bounding box information. The confidence is used to characterize the probability that the target in the bounding box information is an object corresponding to the category (e.g., a vehicle). The higher the confidence, the greater the probability that the target in the bounding box information is an object corresponding to the category.
[0077] like Fig. 9 As shown, the three-dimensional attribute detection head is used to regress the target's pose information (i.e., the coordinates of the target in the camera coordinate system of the vehicle's monocular camera), the length, width, and height of the target in the image (i.e., the length, width, and height of the target in the camera coordinate system of the vehicle's monocular camera relative to the vehicle's body coordinate system), the target's orientation angle (i.e., the rotation angle of the target relative to the y-axis in the camera coordinate system of the vehicle's monocular camera, that is, the angle between the front of the target and the x-axis of the camera coordinate system of the vehicle's monocular camera), and other information. The target's orientation angle can also be called a global yaw angle, which can be expressed as an observation angle. In some embodiments, the three-dimensional attribute detection head can also regress the sine and cosine values of the target's observation angle, and constrain the sum of the squares of the two to determine the observation angle.
[0078] See also Fig.10 , Fig.10 The schematic diagram of the orientation angle and observation angle of the target provided by the embodiment of the present application is shown. The observation angle describes the orientation of the target relative to the perspective of the vehicle monocular camera, and the orientation angle of the target refers to the angle between the target vehicle head direction and the positive direction of the camera coordinate system x-axis of the vehicle monocular camera.
[0079] like Fig. 9 As shown, there is the following conversion relationship between the target's observation angle α and the orientation angle yaw:
[0080]
[0081] Among them, x and z are the lateral distance and longitudinal distance of the target in the camera coordinate system of the ego-vehicle monocular camera, respectively.
[0082] like Fig. 9 As shown, the depth information detection head is used to regress the actual height and visual height (or the reciprocal of the height) of the target, and the visual height refers to the height of the target in the image.
[0083] Based on the three-dimensional target detection model, this application can accurately extract the three-dimensional features of vehicle targets from nighttime road images, including information such as the position and size of vehicle targets, so as to achieve accurate vehicle target perception and environmental understanding. The output of the three-dimensional target detection model can include the target's bounding box information, position, length, width, height, orientation angle, actual height, and height in the image (or the inverse of the height). Based on these data, the three-dimensional features of the target can be restored to obtain the three-dimensional target detection result.
[0084] Step S230: Determine the three-dimensional features of the target based on the target's bounding box information, position, length, width, height, orientation angle, actual height, and height in the image.
[0085] The depth of the target can be determined based on the focal length of the monocular camera, the actual height of the target, and the height of the target in the image; the three-dimensional features of the target can be restored based on the bounding box information, posture, length, width, height, orientation angle, and depth of the target. The three-dimensional features of the target may include the posture and size (i.e., length, width, and height) of the target. It can be understood that the target posture extracted in step S220 and the target posture restored in step S230 both refer to the posture of the target relative to the camera coordinate system of the monocular camera of the vehicle.
[0086] As an example, see Fig.11 and Fig.12 , Fig.11 shows a grayscale image of a target's three-dimensional features from the vehicle's front view, provided by an exemplary embodiment of the present application. Fig.12 The grayscale image of the target's three-dimensional features from the vehicle's top-down perspective provided by an exemplary embodiment of the present application is shown. From the vehicle's front-view perspective, the size of the restored part of the target can be seen, and from the vehicle's top-down perspective, the restored target's posture can be seen.
[0087] In some embodiments, the three-dimensional features of the target that have been restored can be sent to the decision system of intelligent driving, so that the decision system can implement relevant driving functions according to the three-dimensional features of the target. The relevant driving functions may include but are not limited to intelligent driving control and assisted driving control. For example, route planning and collision detection are performed according to the three-dimensional features of the target. Implementing relevant driving functions based on the three-dimensional features of the target can improve the accuracy of the relevant driving functions.
[0088] Parts not described in detail in steps S210 to S230 can refer to steps S310 to S340. Steps S210 to S230 have the following technical effects: utilizing the fact that the visual height of the target in the image is less affected by occlusion or truncation than the length and width of the target and is easier to estimate, the depth of the target is determined according to the actual height of the target in the image and the height in the image, thereby estimating the depth of the target based on the visual height that is easy to estimate in the image captured by the monocular camera, which can improve the accuracy of target depth estimation and solve the technical problem that the monocular camera lacks depth information and makes it difficult to estimate the depth of the target, thereby improving the target prediction accuracy when using a monocular camera for target detection. In addition, using FPN to construct a multi-scale feature map and using RPN to generate candidate boxes on different scale feature maps can significantly improve the accuracy and efficiency of multi-scale target detection of vehicles (especially at night) and improve the safety and reliability of night driving.
[0089] See also Fig.13 , Fig.13 The schematic diagram of the structure of the target detection device provided by the embodiment of the present application is shown. The target detection device 100 can be applied to a vehicle. The target detection device 100 can include: an image acquisition module 110, a target detection module 120 and a result generation module 130.
[0090] The image acquisition module 110 is used to acquire images captured by a monocular camera of the vehicle.
[0091] The target detection module 120 is used to extract the target in the image and obtain the actual height of the target and the height of the target in the image.
[0092] The result generation module 130 is used to determine the depth of the target according to the focal length of the monocular camera, the actual height of the target and the height of the target in the image.
[0093] In some embodiments, the target detection module 120 is further used to extract the target in the image, obtain the target's bounding box information, position, length, width, height, orientation angle, actual height, and height of the target in the image. The result generation module 130 is further used to determine the three-dimensional features of the target based on the target's bounding box information, position, length, width, height, orientation angle, and depth of the target.
[0094] In some embodiments, the target detection module 120 is also used to extract multiple regions of interest in the image, where there are targets; based on the multiple regions of interest, the target's bounding box information, posture, length, width, height, orientation angle, actual height and height in the image are detected.
[0095] In some embodiments, the target detection module 120 is also used to input the multiple regions of interest into three detection heads respectively, and the three detection heads are used to detect the bounding box information of the target, detect the position, length, width, height, orientation angle of the target, and detect the actual height of the target and the height in the image; obtain the bounding box information, position, length, width, height, orientation angle, actual height and height in the image of the target output by the three detection heads.
[0096] In some embodiments, the target detection module 120 is also used to generate multiple feature maps with the same channels and different scales based on the image; extract multiple candidate boxes containing the target based on the multiple feature maps; and align the regions of interest of the multiple candidate boxes to obtain multiple regions of interest.
[0097] In some embodiments, the target detection module 120 is also used to downsample the image using a residual network to obtain multiple feature maps with different resolutions and different numbers of channels; and to perform multi-scale information fusion on the multiple feature maps using a feature pyramid network to obtain multiple feature maps with the same channels and different scales.
[0098] In some embodiments, the object detection module 120 is further configured to use a region proposal network to generate and modify multiple anchor boxes on multiple feature maps respectively, to obtain multiple candidate boxes containing the object.
[0099] Those skilled in the art can clearly understand that the above device provided in the embodiment of the present application can implement the method provided in the embodiment of the present application. The specific working process of the above-described device and module can refer to the corresponding process of the method in the embodiment of the present application, which will not be repeated here.
[0100] In the embodiments provided in the present application, the coupling, direct coupling or communication connection between the modules shown or discussed may be indirect coupling or communication coupling through some interfaces, devices or modules, and may be electrical, mechanical or other forms, and the embodiments of the present application do not impose specific limitations on this.
[0101] In addition, each functional module in the embodiment of the present application can be integrated into one processing module, or each module can exist physically separately, or two or more modules can be integrated into one module. The above integrated modules can be implemented in the form of hardware or in the form of software functional modules.
[0102] See also Fig.14 , Fig.14 The schematic diagram of the structure of the vehicle provided by the embodiment of the present application is shown. The vehicle 200 may include a memory 210 and a processor 220, wherein the memory 210 stores an application program, and the application program is configured to execute the method provided by the embodiment of the present application when called by the processor 220.
[0103] The processor 220 may include one or more processing cores. The processor 220 uses various interfaces and lines to connect various parts of the entire vehicle 200, and is used to run or execute instructions, programs, code sets or instruction sets stored in the memory 210, and call to run or execute data stored in the memory 210, perform various functions of the vehicle 200 and process data.
[0104] The processor 220 can be implemented in at least one of the hardware forms of digital signal processing (DSP), field programmable gate array (FPGA), and programmable logic array (PLA). The processor 220 can integrate one or a combination of a central processing unit (CPU), a graphics processing unit (GPU), and a modem. Among them, the CPU mainly processes the operating system, user interface, and application programs; the GPU is responsible for rendering and drawing display content; and the modem is used to process wireless communications. It is understandable that the above-mentioned modem may not be integrated into the processor 220, but may be implemented separately through a communication chip.
[0105] The memory 210 may include a random access memory (RAM) or a read-only memory (ROM). The memory 210 may be used to store instructions, programs, codes, code sets or instruction sets. The memory 210 may include a program storage area and a data storage area. The program storage area may store instructions for implementing an operating system, instructions for implementing at least one function, instructions for implementing the above-mentioned various method embodiments, etc. The data storage area may store data created by the vehicle 200 during use, etc.
[0106] An embodiment of the present application also provides a computer-readable storage medium, on which a program code is stored. The program code is configured to execute the method provided in the embodiment of the present application when called by a processor.
[0107] The computer-readable storage medium may be an electronic memory such as a flash memory, an electrically erasable programmable read-only memory (EEPROM), an erasable programmable read-only memory (EPROM), a hard disk or a ROM.
[0108] In some embodiments, the computer readable storage medium includes a non-volatile computer readable medium (Non-Transitory Computer-Readable Storage Medium, referred to as Non-TCRSM). The computer readable storage medium has a storage space for the program code that executes any method step in the above method. These program codes can be read from or written into one or more computer program products. The program code can be compressed in an appropriate form.
[0109] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present application.
Claims
1. A target detection method, characterized in that: include: Obtain the image captured by the vehicle's monocular camera; Extract the target in the image and obtain the actual height of the target and the height of the target in the image; The depth of the target is determined based on the focal length of the monocular camera, the actual height of the target and the height of the target in the image.
2. The method according to claim 1, characterized in that The step of extracting the target from the image and obtaining the actual height of the target and the height of the target in the image includes: Extract the target in the image and obtain the target's bounding box information, position, length, width, height, orientation angle, actual height, and the height of the target in the image; After determining the depth of the target according to the focal length of the monocular camera, the actual height of the target and the height of the target in the image, the method further comprises: The three-dimensional features of the target are determined based on the target's bounding box information, position, length, width, height, orientation angle, and depth.
3. The method according to claim 2, characterized in that The step of extracting the target in the image and obtaining the target's bounding box information, position, length, width, height, orientation angle, actual height, and height of the target in the image includes: Extract multiple regions of interest in the image, where targets exist; According to the multiple regions of interest, bounding box information, position, length, width, height, orientation angle, actual height, and height in the image of the target are detected.
4. The method according to claim 3, characterized in that The detecting, according to the multiple regions of interest, the bounding box information, the position, length, width, height, orientation angle, actual height, and height in the image of the target includes: Input the multiple regions of interest into three detection heads respectively, and the three detection heads are used to detect the bounding box information of the target, the position, length, width, height, and orientation angle of the target, and the actual height of the target and the height in the image; Get the bounding box information, pose, length, width, height, orientation angle, actual height, and height in the image of the target output by the three detection heads.
5. The method according to claim 3, characterized in that: The extracting of multiple regions of interest in the image, where targets exist in the regions of interest, includes: Based on the image, generate multiple feature maps with the same channels and different scales; Based on multiple feature maps, multiple candidate boxes containing the target are extracted; Align the regions of interest for multiple candidate boxes to obtain multiple regions of interest.
6. The method according to claim 5, characterized in that The method of generating multiple feature maps with the same channels and different scales based on the image includes: The residual network is used to downsample the image to obtain multiple feature maps with different resolutions and different numbers of channels; A feature pyramid network is used to perform multi-scale information fusion on multiple feature maps to obtain multiple feature maps with the same channels but different scales.
7. The method according to claim 5, characterized in that The method of extracting multiple candidate boxes containing the target based on multiple feature maps includes: The region proposal network is used to generate and correct multiple anchor boxes on multiple feature maps to obtain multiple candidate boxes containing the target.
8. A target detection device, characterized in that: include: An image acquisition module, used to acquire images captured by a monocular camera of a vehicle; The target detection module is used to extract the target in the image and obtain the actual height of the target and the height of the target in the image; The result generation module is used to determine the depth of the target according to the focal length of the monocular camera, the actual height of the target and the height of the target in the image.
9. A vehicle, characterized in that: include: A memory and a processor, wherein an application is stored in the memory, and the application is configured to cause the processor to execute the method according to any one of claims 1 to 7 when called by the processor.
10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores program codes, and the program codes are configured to cause the processor to execute the method according to any one of claims 1 to 7 when called by the processor.
