Target detection method, device and intelligent vehicle

Abstract

The embodiment of the present application relates to the field of driving technology, and particularly relates to a target detection method, device and intelligent vehicle. The method comprises: acquiring continuous frame images collected by a camera; acquiring a transformation matrix of the camera pose according to the continuous frame images; correcting a current frame image according to the transformation matrix to obtain a corrected image; outputting initial three-dimensional information of a target object based on a monocular detection algorithm and the corrected image; and determining final three-dimensional information of the target object according to the initial three-dimensional information, the monocular detection algorithm and internal parameters of the camera. The present application can eliminate the influence of the camera pose on the target detection result, enhance the robustness of the monocular detection algorithm to the internal parameters of the camera, and solve the problem that the monocular detection algorithm cannot be decoupled from the internal parameters and position parameters of the camera.

Description

TECHNICAL FIELD

[0001] Embodiments of the present application relate to the field of driving technology, and in particular to a target detection method and device and intelligent vehicle. BACKGROUND

[0002] The monocular 3D detection module is an important module in the intelligent vehicle perception system. The module takes an image obtained by a camera of the intelligent vehicle as input, detects a position of a target object of interest in the image, and calculates a 3D position and orientation of the object relative to the camera.

[0003] The training of the existing monocular 3D detection module generally first collects sufficient camera and lidar synchronous data in a driving scene, manually labels 3D positions, orientations, and positions on images of target objects of interest in the scene using the lidar and camera, and then trains a deep learning monocular detection algorithm using the labeled data. A typical monocular detection algorithm uses a convolutional neural network to extract two-dimensional features of an image, and regresses 3D characteristics such as 3D size, orientation, and 3D position relative to a camera coordinate system from the feature map extracted from the image.

[0004] In the process of implementing the embodiments of the present application, the inventors have found that changes in camera pose or camera model used when collecting training data will affect the 3D prediction results of the current monocular detection algorithm, i.e., the current monocular detection algorithm is susceptible to the internal parameters and position parameters of the camera. SUMMARY

[0005] The embodiments of the present application mainly solve the problem that the monocular detection algorithm cannot be decoupled from the internal parameters and position parameters of the camera.

[0006] In view of the above problems, the embodiments of the present application provide a target detection method, device, and intelligent vehicle, which overcome the above problems or at least partially solve the above problems.

[0007] According to an aspect of the embodiments of the present application, a target detection method is provided, applied to an intelligent vehicle, the method comprising: acquiring a continuous frame image collected by the camera; acquiring a transformation matrix of the camera pose according to the continuous frame image; correcting a current frame image according to the transformation matrix to obtain a corrected image; outputting initial 3D information of a target object based on a monocular detection algorithm and the corrected image; and determining final 3D information of the target object according to the initial 3D information, the monocular detection algorithm, and internal parameters of the camera.

[0008] Optionally, the transformation matrix of the camera pose is obtained according to the continuous frames of images, comprising: obtaining an initial pose of the camera corresponding to a previous frame of image; performing feature extraction on the previous frame of image and the current frame of image respectively to obtain coordinates of feature points in the previous frame of image and coordinates of feature points in the current frame of image; performing feature matching on the feature points in the previous frame of image and the feature points in the current frame of image; and calculating the transformation matrix of the camera pose of the previous frame and the current frame according to the coordinates of the matched feature points in the previous frame of image and the coordinates of the matched feature points in the current frame of image and the intrinsic parameters of the camera.

[0009] Optionally, the current frame of image is corrected according to the transformation matrix, comprising: obtaining a current pose of the camera according to the initial pose and the transformation matrix; and performing inverse transformation on the current frame of image according to the initial pose of the camera, the current pose of the camera and the transformation matrix to correct the current frame of image.

[0010] Optionally, the initial three-dimensional information of the target object is obtained based on the monocular detection algorithm and the corrected image, comprising: extracting a feature map from the corrected image; and obtaining the initial three-dimensional information of the target object according to the feature map.

[0011] Optionally, the final three-dimensional information of the target object is determined according to the initial three-dimensional information, the monocular detection algorithm and the intrinsic parameters of the camera, comprising: determining projection positions of eight corner points of the target object on the image according to the initial three-dimensional information and the intrinsic parameters of the camera; determining a corrected feature map based on the projection positions; and determining the final three-dimensional information of the target object according to the corrected feature map.

[0012] Optionally, the projection positions of the eight corner points of the target object on the image are determined according to the initial three-dimensional information and the intrinsic parameters of the camera, comprising: obtaining an intrinsic parameter matrix P according to the intrinsic parameters of the camera; and calculating the projection position of each of the eight corner points according to the initial three-dimensional information and the intrinsic parameter matrix P.

[0013] According to another aspect of the embodiments of the present application, there is provided a target detection device applied to an intelligent vehicle, the intelligent vehicle being provided with a camera, and the target detection device comprises: an original information acquisition module configured to acquire continuous frame images collected by the camera; a transformation matrix acquisition module configured to acquire a transformation matrix of a camera pose according to the continuous frame images; an image rectification module configured to rectify a current frame image according to the transformation matrix to obtain a rectified image; an initial prediction module configured to output initial three-dimensional information of a target object based on a monocular detection algorithm and the rectified image; and a final prediction module configured to determine final three-dimensional information of the target object according to the initial three-dimensional information, the monocular detection algorithm and intrinsic parameters of the camera.

[0014] Optionally, the transformation matrix acquisition module comprises: an initial pose acquisition unit configured to acquire an initial pose of the camera corresponding to a previous frame image; a coordinate acquisition unit configured to respectively perform feature extraction on the previous frame image and the current frame image to obtain coordinates of feature points in the previous frame image and coordinates of feature points in the current frame image; a feature matching unit configured to perform feature matching on the feature points in the previous frame image and the feature points in the current frame image; and a transformation matrix acquisition unit configured to calculate a transformation matrix of a camera pose of a previous frame and a camera pose of a current frame according to the coordinates of the matched feature points in the previous frame image and the coordinates of the matched feature points in the current frame image and intrinsic parameters of the camera.

[0015] Optionally, the image rectification module comprises: a current pose acquisition unit configured to acquire a current pose of the camera according to the initial pose and the transformation matrix; and a pose compensation unit configured to perform inverse transformation on the current frame image according to the initial pose of the camera, the current pose of the camera and the transformation matrix to rectify the current frame image.

[0016] According to still another aspect of the embodiments of the present application, there is provided an intelligent vehicle, which comprises: a camera; at least one processor; and a memory, wherein the memory and the camera are respectively in communication connection with the processor, and the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method as described above.

[0017] Different from the related art, the embodiment of the present application provides a target detection method, device and intelligent vehicle, by acquiring continuous frame images collected by the camera; a transformation matrix of the camera pose is acquired according to the continuous frame images; a current frame image is corrected according to the transformation matrix to obtain a corrected image; initial three-dimensional information of a target object is output based on a monocular detection algorithm and the corrected image; and final three-dimensional information of the target object is determined according to the initial three-dimensional information, the monocular detection algorithm and the camera internal parameter. The embodiment of the present application can eliminate the influence of the camera pose on the target detection result, enhance the robustness of the monocular detection algorithm to the camera internal parameter, and solve the problem that the monocular detection algorithm cannot be decoupled from the internal parameter and the position parameter of the camera. BRIEF DESCRIPTION OF DRAWINGS

[0018] One or more embodiments are illustrated by way of example in the figures that form a part of this patent document, these example are not to be construed as limiting the embodiments, elements having the same reference numbers in figures denote like elements, unless otherwise specifically noted, the figures in the drawings are not to scale limit.

[0019] Figure 1 is a flowchart of a target detection method provided by the embodiment of the present application;

[0020] Figure 2 is a flowchart of acquiring a transformation matrix of a camera pose provided by the embodiment of the present application;

[0021] Figure 3 is a flowchart of correcting a current frame image provided by the embodiment of the present application;

[0022] Figure 4 is a flowchart of acquiring initial three-dimensional information provided by the embodiment of the present application;

[0023] Figure 5 is a flowchart of acquiring final three-dimensional information provided by the embodiment of the present application;

[0024] Figure 6 is a flowchart of determining a projection position of a corner point on an image provided by the embodiment of the present application;

[0025] Figure 7 is a structural schematic diagram of a target detection device provided by the embodiment of the present application;

[0026] Figure 8 is a structural schematic diagram of a transformation matrix acquisition module provided by the embodiment of the present application;

[0027] Figure 9 is a structural schematic diagram of an image correction module provided by the embodiment of the present application;

[0028] Figure 10 is a structural block diagram of an intelligent vehicle provided by an embodiment of the present application. DETAILED DESCRIPTION

[0029] In order to make the objectives, technical solutions and advantages of the present application clearer, the present application is further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and do not limit the present application.

[0030] It should be noted that the various features in the embodiments of the present application can be combined with each other without conflict, and are all within the protection scope of the present application. In addition, although the division of functional modules is made in the device schematic diagram, and the logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in a different order from the module division in the device schematic diagram or the order in the flowchart.

[0031] Unless otherwise defined, all technical and scientific terms used in the present application have the same meanings as those commonly understood by one skilled in the art to which the present application belongs. The terms used in the specification of the present application are only for the purpose of describing specific embodiments and are not intended to limit the present application.

[0032] A typical deep learning monocular detection algorithm such as SMOKE receives an image collected by a camera and uses a backbone network of a convolutional neural network to extract two-dimensional features of the image, for example, a DLA (Deep Layer Aggregation) network. The convolutional output layer first densely outputs a heat map of the target object, and then regresses the three-dimensional characteristics of the target object such as 3D size, orientation, and three-dimensional position relative to the coordinate system of the camera according to the feature map extracted from the image.

[0033] In the prior art, the detection result of the monocular detection algorithm is easily affected by the pitch angle, which is a parameter in the camera pose. The pitch angle is the included angle between the body axis and the ground plane. The body axis is the center line of the light beam passing through the center point of the camera lens. For example, if the forward direction of the camera is parallel to the ground, the pitch angle of the camera relative to the ground is zero. The monocular detection algorithm assumes that the pitch angle of the camera relative to the ground is zero when performing detection, i.e., the forward direction of the camera is parallel to the ground. However, due to installation errors of the camera and up-and-down jolts during the travel of the intelligent vehicle, it is impossible to guarantee that the pitch angle of the camera relative to the ground is completely zero, which will cause changes in geometric elements in the image, thus resulting in errors in the final target detection result.

[0034] In addition, the camera used for collecting test data and training data must have the same internal and external parameters to correctly estimate the depth of the target. In existing monocular detection algorithms, the generation of heat maps and the regression of three-dimensional characteristics of target objects are trained using collected data. If the camera on the vehicle and the camera model used for collecting training data are changed, even if the pitch angle of the camera is zero, the existing algorithm often cannot correctly regress the correct depth. Therefore, the monocular three-dimensional detection algorithm is strongly coupled with the internal parameters of the camera and the position of the camera. The coupling refers to the interaction between them. It can be understood that the strong coupling means that the degree of interaction is high. The decoupling means that the coupling relationship is removed, reducing the influence of the internal parameters of the camera and the position parameters of the camera on the output target detection result of the monocular detection algorithm.

[0035] The application provides a target detection method composed of a pitch angle estimated visual odometer and a monocular detection module based on star convolution. The visual odometer estimates the motion of the camera according to adjacent frame images, corrects the images, and provides more accurate image data for the monocular detection module based on star convolution. The star convolution has a larger receptive field than ordinary convolution, that is, each convolution output contains a larger range of information, making the output more dense.

[0036] Embodiment one

[0037] The embodiment of the application provides a target detection method, which is applied to an intelligent vehicle. Please refer to Figure 1 , Figure 1 It is a flowchart of a target detection method provided by the embodiment of the application, which comprises the following steps:

[0038] S11, acquiring continuous frame images collected by the camera. The camera is a monocular camera, that is, a camera provided with one camera head. In addition, a binocular camera is a camera provided with two camera heads, and a multiocular camera is a camera provided with multiple camera heads. The camera photographs the surrounding scene during the driving of the intelligent vehicle to obtain video materials. The minimum unit of the video materials is a single picture, that is, a frame image. An image sequence composed of frame images is continuous frame images. The intelligent vehicle can be an unmanned vehicle.

[0039] S12, acquiring a transformation matrix of the camera pose according to the continuous frame images. The transformation matrix represents the relative conversion relationship of feature point coordinates in adjacent frame images. Please refer to Figure 2 , Figure 2 It is a flowchart of acquiring a transformation matrix of a camera pose provided by the embodiment of the application. The transformation matrix of the camera pose is acquired according to the continuous frame images, which comprises the following steps:

[0040] S121, obtain an initial pose of the camera corresponding to a previous frame image. Wherein, the current frame image is a picture collected by the camera currently, and the previous frame image is a picture collected by the camera before the current frame image in the image sequence. The initial pose is a position and an attitude of the camera determined by the IMU measurement camera in a calibration site with flat ground. The IMU measurement camera is an inertial odometer, which is used to obtain acceleration and angular velocity of the camera, and to obtain the position and the attitude of the camera corresponding to the previous frame image by integrating the acceleration and the angular velocity of adjacent frames in the image at corresponding time points. The IMU measurement camera is installed on the vehicle body of the intelligent vehicle, and a specific position is obtained through experiments, which is not limited in the present disclosure. The calibration site is a place for calibrating the external parameters of the camera.

[0041] S122, respectively extract features from the previous frame image and the current frame image to obtain coordinates of feature points in the previous frame image and coordinates of feature points in the current frame image. Wherein, the feature points are points with special outstanding attributes in some aspects. Before obtaining the coordinates of the feature points in the previous frame image and the current frame image, the input image needs to be subjected to a gray scale transformation, and then a FAST corner detection algorithm is used to extract features from the gray scale transformed image to obtain the coordinates of the feature points. The gray scale transformation is an image processing method of image enhancement technology, which is used to convert a color image obtained by the camera into a gray scale image, and can make the image display more details and strengthen the image region of interest. The FAST corner detection algorithm takes a certain pixel point as the center of a circle, and considers that the point is a feature point when a feature difference between the pixel point and other pixel points around the pixel point with a certain radius reaches a preset difference threshold. The preset difference threshold is obtained through experiments, which is not limited in the present disclosure. A frame image is divided into several small squares, and each small square is a pixel point.

[0042] S123, performing feature matching on the feature points in the previous frame image and the feature points in the current frame image. The feature matching refers to matching the coordinates of the feature points in two frames of images, finding the corresponding relationship of the feature points between the two images, and determining the positional relationship of the two images. In the process of feature matching, the next frame of the image is received as the current frame, the feature matching is performed on the feature points in the input current frame image and the previous frame image using the optical flow pyramid algorithm, and the coordinates of the corresponding feature points of the current frame and the previous frame are found. The image pyramid represents scaling the same image to obtain images at different resolutions, and the original image is the bottom layer of the pyramid. The optical flow pyramid algorithm calculates the pyramid optical flow from the highest layer of the image first, and then takes the tracking effect of the previous layer as the initial value of the next layer of optical flow, and so on to calculate the optical flow of the bottom layer, that is, the final optical flow result. In simple terms, the feature matching is performed by tracking the coordinates of the feature points of adjacent frames in the image, and the optical flow is generated by the movement of the camera in the scene. In the feature matching process, the current frame is assigned to the previous frame for the next calculation, the feature points of the current frame are used as the matching points of the next frame, and if the number of feature points is less than a preset number threshold, the feature points before matching the two frames are recalculated. The preset number threshold is obtained according to experiments, and the present disclosure does not make a specific limitation on this.

[0043] S124, calculating a transformation matrix of the camera pose of the previous frame and the current frame according to the coordinates of the feature points in the previous frame image and the coordinates of the feature points in the current frame image and the camera internal parameter. According to the result of the feature matching, the coordinates of the matching points are obtained, the fundamental matrix is obtained according to the coordinates of the matching points, and the transformation matrix is obtained according to the fundamental matrix and the camera internal parameter using the epipolar constraint. The epipolar constraint is a geometric constraint on the plane formed by the optical centers of the current frame and the previous frame and the camera. The fundamental matrix is the relationship of a point in space in the image coordinate systems of different view cameras. The camera internal parameter is a parameter related to the characteristics of the camera, such as the focal length and pixel size of the camera.

[0044] S13, correcting the current frame image according to the transformation matrix to obtain a corrected image. By inversely transforming the image, the pose of the camera can be compensated, and the influence of the pitch angle on 3D target detection can be offset. Please refer to Figure 3 , Figure 3 is a flowchart of correcting the current frame image provided by the present application, and the correction of the current frame image according to the transformation matrix includes:

[0045] S131, obtain the current pose of the camera according to the initialization pose and the transformation matrix. Wherein, the transformation matrix is decomposed to obtain the pose transformation of the camera corresponding to the adjacent frame image, the pose transformation includes a rotation matrix R and a translation matrix T, and the current pose of the camera can be calculated according to the initialization pose and the pose transformation.

[0046] S132, inversely transform the current frame image according to the initialization pose of the camera, the current pose of the camera and the transformation matrix to correct the current frame image. Wherein, the pose relationship of the camera relative to the initialization pose can be known according to the initialization pose and the current pose of the camera, and the pitch angle of the camera can be obtained according to the pose relationship. If the pitch angle of the camera relative to the ground is not zero, the image obtained by the camera will be distorted, and the roll transformation basically does not exist in the driving process of the intelligent vehicle, so the pitch error is the main source of image error. According to the pitch angle and the transformation matrix, the collected image is scaled to change the real size, the image preprocessing is completed, and the corrected image is output, that is, the camera is calculated in real time according to the pose of the intelligent vehicle, so that the image always remains horizontal.

[0047] In the prior art, the image is directly extracted by the backbone network of the convolutional neural network without preprocessing, and if the pitch angle of the camera relative to the ground is not zero, the subsequent target detection result will be affected. Therefore, the image is preprocessed and corrected in the embodiment of the application, so that the image always remains horizontal, the posture of the camera is compensated, and the influence of the pitch angle on the 3D target detection result is offset.

[0048] S14, output the initial three-dimensional information of the target object based on the monocular detection algorithm and the corrected image. Wherein, the initial three-dimensional information is the three-dimensional information output by the Head network for the first time, and the Head network is used to convert the input image into the final prediction result. Figure 4 , Figure 4 is a flowchart for obtaining initial three-dimensional information provided by the embodiment of the application, the initial three-dimensional information of the target object is output based on the monocular detection algorithm and the corrected image, which comprises:

[0049] S141, extract a feature map from the corrected image. Wherein, the two-dimensional feature is extracted by the backbone network of the convolutional neural network, for example: DLA (Deep Layer Aggregation) network, the DLA network is a deep aggregation network and can be used for image feature extraction.

[0050] S142, obtaining initial three-dimensional information of the target object according to the feature map. Wherein, the initial three-dimensional information of the target object is obtained by regression according to the center pixel point in the feature map, the center pixel point refers to the most critical pixel point of the target object in the feature map, which is used to regress the initial three-dimensional information of the target object. Wherein, the most critical pixel point can be the center pixel point in all pixels corresponding to the target object, etc. The target object refers to the object to be detected, and the initial three-dimensional information of the target object includes the three-dimensional coordinates (x 3d , y 3d , z 3d ) of the center point of the target object prediction box relative to the camera, the length, width and height (l, w, h) of the target object, and the orientation angle θ of the target object. Before obtaining the initial three-dimensional information of the target object, the heat map is also output by extracting the feature, and the most critical pixel point can be determined according to the heat map, such as the pixel point corresponding to the brightest point in the heat map is the most critical pixel point. The heat map is used to judge whether there is a target object in the image, and the target object can also be classified according to the heat map, such as, for the heat map, focal loss, i.e. focal loss function is used for supervision, the expression of the focal loss is: FL(p t ) = -(1-p t ) γ log(p t ).

[0051] Wherein, p t is the probability value of the same classification prediction and label data, the label data is the data artificially annotated in the data set, FL(p t ) is the focal loss value, (1-p t ) γ is the cross entropy loss, which punishes the wrong prediction, and γ is the adjustable focusing parameter, which can be 2 specifically, and the disclosure does not make specific limitation. The loss is the difference between the predicted value and the true value of each sample, the focal loss function allocates more weight to difficult samples while reducing the loss contribution of easy learning samples to solve the class balance problem, the easy learning sample is the sample with confident attribute, and the difficult sample is the sample without confident attribute, for example: the confidence of an intelligent vehicle in an image is 0.9, so it is confident that it is an intelligent vehicle, so the sample is identified as an easy learning sample; Similarly, the confidence of being an intelligent vehicle is 0.6, so it is not confident that it is an intelligent vehicle, so the sample is called a difficult sample.

[0052] S15, determine final three-dimensional information of the target object according to the initial three-dimensional information, the monocular detection algorithm and the internal parameter of the camera. Wherein, the monocular detection algorithm is the monocular detection module based on star convolution, and the final three-dimensional information is the three-dimensional information output by the Head network for the second time, please combine Figure 5 , Figure 5 is a flowchart for obtaining final three-dimensional information provided by the embodiment of the application, and the final three-dimensional information of the target object is determined according to the initial three-dimensional information, the monocular detection algorithm and the internal parameter of the camera, which comprises:

[0053] S151, determine the projection position of the eight corner points of the target object on the image according to the initial three-dimensional information and the internal parameter of the camera. Please refer to Figure 6 , Figure 6 is a flowchart for determining the projection position of the corner points on the image provided by the embodiment of the application, and the projection position of the eight corner points of the target object on the image is determined according to the initial three-dimensional information and the internal parameter of the camera, which comprises:

[0054] S1511, obtain the internal parameter matrix P according to the internal parameter of the camera. Wherein, the internal parameter matrix reflects the self attribute of the camera, which is related to the model of the camera, and can be selected according to the demand, and the disclosure does not make specific limitation.

[0055] S1512, calculate the projection position of each corner point in the eight corner points according to the initial three-dimensional information and the internal parameter matrix P. The projection position of one corner point is obtained by the following formula, and the formula is:

[0056]

[0057] Wherein, indicates the first two rows of the internal parameter matrix P, (x 3d , y 3d , z 3d ) is the center point of the target object prediction box in the initial three-dimensional information of the target object relative to the three-dimensional coordinates of the camera, l is the length of the target object, w is the width of the target object, h is the height of the target object, theta is the orientation angle of the target object, (x kp0 , y kp0 ) is the projection position of the corner point. The eight corner points are eight vertices of the target object prediction box, and the calculation formula of the projection position of one corner point is known. The calculation formula of the projection position of the remaining seven corner points can be adjusted according to the addition and subtraction of the length, width and height of the target object prediction box on the basis of the known formula, and the calculation formula of the projection position of the remaining seven corner points is the same as that of the known corner point.

[0058] S152, determining a corrected feature map based on the projection position.

[0059] The corrected feature map is determined according to the following formula:

[0060] X'(x i , y i ) = åw i ·X(x kpi , y kpi )

[0061] wherein X'(x i , y i ) is the corrected feature map, and the center pixel point in the feature map is specifically corrected, w i is a learnable parameter of the convolutional neural network, which is learned together with the optimization of the network parameters as a whole, and X(x kpi , y kpi ) is the feature of the corner point, which is determined based on the projection coordinates of the corner point. Wherein the features on the eight corner points are extracted to the center pixel point according to the projection position, and the same operation is performed on each center pixel point on the feature map, that is, the center pixel point on the feature map is corrected.

[0062] S153, determining the final three-dimensional information of the target object according to the corrected feature map. Wherein the final three-dimensional information of the target object is determined by regression based on the center pixel point in the corrected feature map, and each center pixel point corresponds to a target object. For example, the smoothed L1 Loss, i.e. the smooth L1 loss function, is used to supervise the prediction of the three-dimensional target object, and the expression of the smoothed L1 Loss is:

[0063]

[0064] wherein y pred is the predicted value, y gt is the actual value, L smooth－1 (y pred , y gt ) is the smooth L1 loss value, the smoothed L1 Loss improves the zero point non-smoothness problem, and is a slowly changing loss function, which has stronger robustness to abnormal values. The robustness refers to the ability of the monocular detection algorithm to maintain certain performance under uncertain disturbances.

[0065] In the prior art, all operations in the target detection process are only based on image features, so the cameras used for collecting test data and training data must have the same internal and external parameters to correctly estimate the depth of the target. The monocular detection algorithm based on star convolution proposed in the patent encapsulates the projection geometry of the camera in the learnable parameters, so that the network can adjust the final prediction output according to the actual projection effect of the target object predicted based on image features under the parameters of the current camera. The overall operation process of the network is fully derivable, so that end-to-end training can be realized, and the network can adapt to different camera parameters during training, thereby improving the robustness of the monocular detection algorithm to camera internal parameters.

[0066] In the embodiment of the present application, the pitch angle of the camera is continuously tracked and obtained using a visual odometer, the calculated pitch angle is used to correct the image so that the visual effect of the image becomes a common view angle in the training data, and then the monocular detection module based on star convolution is used to reliably output the target object of interest in the image. The embodiment of the present application can eliminate the influence of the camera pose on the target detection result, enhance the robustness of the monocular detection algorithm to the camera internal parameters, and solve the problem that the monocular detection algorithm cannot be decoupled from the internal parameters and position parameters of the camera.

[0067] Embodiment two

[0068] Please refer to Figure 7 , Figure 7 is a structural schematic diagram of a target detection device provided by an embodiment of the present application. The embodiment of the present application provides a target detection device 200, which comprises:

[0069] An original information acquisition module 21 is configured to acquire continuous frame images collected by the camera.

[0070] A transformation matrix acquisition module 22 is configured to acquire a transformation matrix of the camera pose according to the continuous frame images. Please refer to Figure 8 , Figure 8 is a structural schematic diagram of a transformation matrix acquisition module provided by an embodiment of the present application. The transformation matrix acquisition module 22 comprises: an initial pose acquisition unit 221 configured to acquire an initial pose of the camera corresponding to a previous frame image; a coordinate acquisition unit 222 configured to respectively perform feature extraction on the previous frame image and the current frame image to obtain coordinates of feature points in the previous frame image and coordinates of feature points in the current frame image; a feature matching unit 223 configured to perform feature matching on the feature points in the previous frame image and the feature points in the current frame image; and a transformation matrix acquisition unit 224 configured to calculate a transformation matrix of the camera pose of the previous frame and the current frame according to the coordinates of the matched feature points in the previous frame image and the coordinates of the matched feature points in the current frame image and the internal parameters of the camera.

[0071] The image correction module 23 is configured to correct the current frame image according to the transformation matrix to obtain a corrected image. Please refer to Figure 9 , Figure 9 FIG. 2 is a structural diagram of an image correction module according to an embodiment of the present application. The image correction module 23 comprises a current pose acquisition unit 231 configured to acquire a current pose of the camera according to the initial pose and the transformation matrix; and a pose compensation unit 232 configured to inversely transform the current frame image according to the initial pose of the camera, the current pose of the camera and the transformation matrix to correct the current frame image.

[0072] The initial prediction module 24 is configured to output initial three-dimensional information of the target object based on a monocular detection algorithm and the corrected image.

[0073] The final prediction module 25 is configured to determine final three-dimensional information of the target object according to the initial three-dimensional information, the monocular detection algorithm and the intrinsic parameters of the camera.

[0074] It should be noted that the target detection device can perform the target detection method provided by the embodiments of the present application, and technical details not described in detail in the target detection device embodiments can be referred to the target detection method provided by the embodiments of the present application.

[0075] In the embodiments of the present application, the pose of the camera is continuously tracked and acquired using a visual odometer, the image is corrected according to the pose so that the visual effect of the image becomes a common visual angle in the training data, and then the corrected image is reliably output to the target object of interest in the image using a monocular detection module based on star convolution. The embodiments of the present application can eliminate the influence of the camera pose on the target detection result, enhance the robustness of the monocular detection algorithm to the intrinsic parameters of the camera, and solve the problem that the monocular detection algorithm cannot be decoupled from the internal parameters and position parameters of the camera.

[0076] Embodiment three

[0077] The embodiments of the present application also provide an intelligent vehicle. Please refer to Figure 10 , Figure 10 FIG. 4 is a structural block diagram of an intelligent vehicle according to an embodiment of the present application, which shows the hardware structure of the intelligent vehicle capable of performing the target detection method. Figures 1 to 6

[0078] ​The intelligent vehicle 300 comprises a camera 31, at least one processor 32, and a memory 33, which are in communication connection with the processor 32 respectively, the memory 33 stores instructions executable by the at least one processor 32, and the instructions are executed by the at least one processor 32 to enable the at least one processor 32 to execute the target detection method described in the above embodiments.

[0079] The processor 32 and the memory 33 can be connected through a bus or other means, Figure 10 Taking the bus connection as an example, the memory 33 as a non-volatile computer readable storage medium can be used to store non-volatile software programs, non-volatile computer executable programs and modules. The processor 32 executes various function applications and data processing by running the non-volatile software programs, instructions and modules stored in the memory 33, that is, implements the target detection method described in the above embodiments.

[0080] The memory 33 can include a program storage area and a data storage area, wherein the program storage area can store an operating system and at least one application required by a function; the data storage area can store data created by the intelligent vehicle and the like. In addition, the memory 33 can include a high-speed random access memory, and can also include a non-volatile memory, such as at least one magnetic disk storage device, a flash memory device, or other non-volatile solid-state memory device. In some embodiments, the memory 33 can optionally include a memory 33 remotely arranged with respect to the processor 32. These remote memories 33 can be connected to the intelligent vehicle 300 through a network. Examples of the above network include but are not limited to the Internet, an intranet, a local area network, a mobile communication network, and a combination thereof.

[0081] The one or more modules are stored in the memory 33, and when executed by the one or more processors 32, the target detection method in any of the above embodiments is executed, for example, the method steps in the above Figures 1 to 6 are executed.

[0082] The above product can execute the target detection method provided by the embodiments of the present application, and has the corresponding function modules for executing the target detection method. Technical details not described in detail in the present embodiment can be referred to the target detection method provided by the embodiments of the present application.

[0083] Those skilled in the art can clearly understand the implementation of the embodiments by the description of the above embodiments, and the embodiments can be implemented by means of software plus a general hardware platform, and of course, can also be implemented by hardware. Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by a computer program instructing related hardware, and the program can be stored in a computer readable storage medium. When the program is executed, the program can include the processes of the above-mentioned embodiment methods. The storage medium can be a magnetic disc, an optical disc, a read-only memory (ROM) or a random access memory (RAM) and the like.

[0084] Finally, it should be noted that: the above examples are only used to illustrate the technical solutions of the present application, and not to limit them; under the idea of the present application, the technical features of the above examples or different examples can also be combined, the steps can be implemented in any order, and there are many other changes of different aspects of the present application as described above. In order to be brief, they are not provided in detail; although the present application has been described in detail with reference to the foregoing examples, those skilled in the art should understand that: it can still modify the technical solutions recorded in the foregoing examples, or make equivalent replacement for part of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the present application.

Claims

1. A target detection method applied to an intelligent vehicle, wherein the intelligent vehicle is equipped with a camera, characterized in that, The method comprises: acquiring continuous frame images collected by the camera; acquiring a transformation matrix of the camera pose according to the continuous frame images; correcting a current frame image according to the transformation matrix to obtain a corrected image; outputting initial three-dimensional information of a target object based on a monocular detection algorithm and the corrected image, including extracting a feature map from the corrected image, and obtaining the initial three-dimensional information of the target object according to the feature map; determining final three-dimensional information of the target object according to the initial three-dimensional information, the monocular detection algorithm and the intrinsic parameters of the camera, including determining projection positions of eight corner points of the target object on the image according to the initial three-dimensional information and the intrinsic parameters of the camera, determining a corrected feature map based on the projection positions, and determining the final three-dimensional information of the target object according to the corrected feature map; wherein the corrected feature map is determined according to the following formula: In the formula: is the modified feature map, and the center pixel point in the modified feature map is specifically modified, wi is a learnable parameter of the convolutional neural network, which is learned together with the optimization of the whole network parameters, is the feature of the corner point, and the feature of the corner point is determined based on the projection coordinates of the corner point.

2. The method of claim 1, wherein, the acquiring of the transformation matrix of the camera pose according to the continuous frame images comprises: acquiring an initial pose of the camera corresponding to a previous frame image; extracting features from the previous frame image and the current frame image respectively to obtain coordinates of feature points in the previous frame image and coordinates of feature points in the current frame image; performing feature matching on the feature points in the previous frame image and the feature points in the current frame image; calculating a transformation matrix of the camera pose of the previous frame and the current frame according to the coordinates of the matched feature points in the previous frame image and the coordinates of the matched feature points in the current frame image and the intrinsic parameters of the camera.

3. The method of claim 2, wherein, the correction of the current frame image according to the transformation matrix comprises: acquiring a current pose of the camera according to the initial pose and the transformation matrix; performing inverse transformation on the current frame image according to the initial pose of the camera, the current pose of the camera and the transformation matrix to correct the current frame image.

4. The method according to any one of claims 1 to 3, characterized in that, the determination of the projection positions of the eight corner points of the target object on the image according to the initial three-dimensional information and the intrinsic parameters of the camera comprises: acquiring an intrinsic parameter matrix P according to the intrinsic parameters of the camera; calculating the projection position of each of the eight corner points according to the initial three-dimensional information and the intrinsic parameter matrix P.

5. A target detection device applied to an intelligent vehicle, wherein the intelligent vehicle is provided with a camera, and the target detection device is characterized in that, The target detection device comprises: an original information acquisition module configured to acquire continuous frame images collected by the camera; a transformation matrix acquisition module configured to acquire a transformation matrix of the camera pose according to the continuous frame images; an image correction module configured to correct a current frame image according to the transformation matrix to obtain a corrected image; an initial prediction module configured to output initial three-dimensional information of a target object based on a monocular detection algorithm and the corrected image, including extracting a feature map from the corrected image, and obtaining the initial three-dimensional information of the target object according to the feature map; The final prediction module is configured to determine final three-dimensional information of the target object according to the initial three-dimensional information, the monocular detection algorithm and the intrinsic parameters of the camera, and includes: determining projection positions of eight corner points of the target object on the image according to the initial three-dimensional information and the intrinsic parameters of the camera; determining a corrected feature map based on the projection positions; and determining the final three-dimensional information of the target object according to the corrected feature map. The corrected feature map is determined according to the following formula: In the formula: is the modified feature map, and the center pixel point in the modified feature map is specifically modified, wi is a learnable parameter of the convolutional neural network, which is learned together with the optimization of the whole network parameters, is the feature of the corner point, and the feature of the corner point is determined based on the projection coordinates of the corner point.

6. The apparatus of claim 5, wherein: The transformation matrix acquisition module includes: An initial pose acquisition unit is configured to acquire an initial pose of the camera corresponding to a previous frame of image. A coordinate acquisition unit is configured to perform feature extraction on the previous frame of image and the current frame of image respectively to obtain coordinates of feature points in the previous frame of image and coordinates of feature points in the current frame of image. A feature matching unit is configured to perform feature matching on the feature points in the previous frame of image and the feature points in the current frame of image. A transformation matrix acquisition unit is configured to calculate a transformation matrix of the pose of the camera in the previous frame and the pose of the camera in the current frame according to the coordinates of the matched feature points in the previous frame of image and the coordinates of the matched feature points in the current frame of image and the intrinsic parameters of the camera.

7. The apparatus of claim 6, wherein: The image correction module includes: A current pose acquisition unit is configured to acquire a current pose of the camera according to the initial pose and the transformation matrix. A pose compensation unit is configured to perform inverse transformation on the current frame of image according to the initial pose of the camera, the current pose of the camera and the transformation matrix to correct the current frame of image.

8. An intelligent vehicle, characterized by The intelligent vehicle includes: A camera; At least one processor; and a memory, which is in communication connection with the processor respectively, and stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the method in any one of claims 1-4.

Images