Monocular 3D target detection method and device, electronic equipment and storage medium

Abstract

The application discloses a monocular 3D target detection method and device, electronic equipment and storage medium, comprising: acquiring a to-be-detected image; inputting the to-be-detected image into a target detection model to obtain detection data of a target in the to-be-detected image, wherein the detection data comprises a depth compensation value, a 3D size compensation value, a 3D angle, a 2D detection frame of the target in the to-be-detected image and a category; generating a 3D detection frame of the target according to the detection data of the target, a preset background depth map and a preset 3D average size, so as to serve as a 3D target detection result. The target detection model does not need to directly regress the depth of the target, so that the target detection model is simple to train, high in stability, accurate in predicted data, and capable of predicting the 3D angle of the target, without requirement for the installation angle of the monocular camera, and the monocular camera can be applied to various installation position requirements.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image recognition, and in particular to a monocular 3D target detection method and device, electronic equipment and a storage medium. BACKGROUND

[0002] With the rapid development of automatic driving technology, the demand for visual perception in the field of automatic driving is also increasing. Traditional 3D target detection for automatic driving is mostly visual laser combination, laser perception or binocular target detection.

[0003] However, due to the high price of laser radar, some algorithms are not suitable for large-scale promotion. In order to reduce the cost and with the development of neural network algorithms, many monocular camera depth estimation and 3D detection algorithms have also been gradually developed, but the effect is not ideal and is not suitable for practical application. For example, the current 3D target detection based on monocular camera usually requires the optical center of the monocular camera to be parallel to the ground after installation, and directly regresses the depth value of the target through the neural network, which leads to slow network training and poor target detection effect, and cannot be applied to automatic driving technology. SUMMARY

[0004] The present application provides a monocular 3D target detection method, device, electronic equipment and storage medium to solve the problem that the current monocular 3D target detection has requirements for the installation of monocular cameras, and the network training is slow and the detection effect is poor.

[0005] In a first aspect, the present application provides a monocular 3D target detection method, comprising:

[0006] obtaining a to-be-detected image, the to-be-detected image being an image collected by a monocular camera on a detection area;

[0007] inputting the to-be-detected image into a target detection model to obtain detection data of a target in the to-be-detected image, the detection data including a depth compensation value, a 3D size compensation value, a 3D angle, a 2D detection frame of the target in the to-be-detected image, and a category of the target;

[0008] generating a 3D detection frame of the target according to the detection data of the target, a preset background depth map, and a preset 3D average size, to serve as a 3D target detection result.

[0009] In a second aspect, the present application provides a monocular 3D target detection device, comprising:

[0010] a to-be-detected image acquisition module configured to acquire a to-be-detected image, the to-be-detected image being an image collected by a monocular camera on a detection area;

[0011] The detection data acquisition module is configured to input the to-be-detected image into a target detection model to obtain detection data of a target in the to-be-detected image, wherein the detection data comprises a depth compensation value, a 3D size compensation value, a 3D angle, a 2D detection box of the target in the to-be-detected image, and a category of the target.

[0012] The 3D detection box generation module is configured to generate a 3D detection box of the target according to the detection data of the target, a preset background depth map, and a preset 3D average size, to serve as a 3D target detection result.

[0013] In a third aspect, the present application provides an electronic device, which comprises:

[0014] at least one processor; and

[0015] a memory in communication with the at least one processor; wherein

[0016] the memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor to enable the at least one processor to execute the monocular 3D target detection method according to the first aspect of the present application.

[0017] In a fourth aspect, the present application provides a computer readable storage medium storing computer instructions for enabling a processor to implement the monocular 3D target detection method according to the first aspect of the present application when executed.

[0018] After the to-be-detected image collected by the monocular camera in the detection region is acquired, the to-be-detected image is input into a target detection model to obtain detection data of a target in the to-be-detected image, wherein the detection data comprises a depth compensation value, a 3D size compensation value, a 3D angle, a 2D detection box of the target in the to-be-detected image, and a category of the target. Then, a 3D detection box of the target is generated according to the above detection data of the target, a preset background depth map, and a preset 3D average size, to serve as a 3D target detection result. According to the present application, the depth compensation value, the 3D size compensation value, the 3D angle, the 2D detection box of the target in the to-be-detected image, and the category of the target are predicted by the target detection model, and the 3D detection box of the target is generated by combining the predicted data with the preset background depth map and the preset 3D average size. The target detection model does not need to directly regress the depth of the target, so that the training of the target detection model is simple, the stability of the target detection model is high, the predicted data is accurate, and the 3D angle of the target can be predicted. The present application has no requirement on the installation angle of the monocular camera, and the monocular camera can be applied to various installation position requirements.

[0019] It is to be understood that the description of the background art is not an acknowledgement or admission that the art described is "prior art" with respect to the present application. The background art described is not necessarily known to be the prior art to the present application, and it is not necessarily considered to be relevant to the prior art of the present application. BRIEF DESCRIPTION OF DRAWINGS

[0020] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed to be used in the embodiments will be briefly introduced as follows. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative effort based on these drawings.

[0021] Figure 1 is a flowchart of a monocular 3D target detection method provided by the first embodiment of the present application;

[0022] Figure 2A is a flowchart of a monocular 3D target detection method provided by the second embodiment of the present application;

[0023] Figure 2B is a schematic diagram of a target detection scene of the present application;

[0024] Figure 3 is a structural schematic diagram of a monocular 3D target detection device provided by the third embodiment of the present application;

[0025] Figure 4 is a structural schematic diagram of an electronic device provided by the fourth embodiment of the present application. DETAILED DESCRIPTION

[0026] In order to make the technical personnel in the art better understand the present application scheme, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some embodiments of the present application, not all. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative effort should be within the scope of protection of the present application.

[0027] Embodiment one

[0028] Figure 1 is a flowchart of a monocular 3D target detection method provided by the first embodiment of the present application. The present embodiment can be applicable to the case of detecting a target and outputting 3D data of the target. The method can be executed by a monocular 3D target detection device, which can be realized in the form of hardware and / or software. The monocular 3D target detection device can be configured in an electronic device, such as a background server in communication with an autonomous vehicle. Figure 1As shown, the monocular 3D target detection method comprises:

[0029] S101, acquire a to-be-detected image, the to-be-detected image being an image collected by a monocular camera on a detection area.

[0030] In this embodiment, the monocular camera can be a camera provided with one camera head, which can be an RGB camera head. The monocular camera can be installed on a roadside device, for example, on a support rod of a roadside device of an automatic driving system, to collect images at a position at a certain height from the ground. The detection area can be a region of a road concerned in automatic driving, such as an intersection region.

[0031] After the monocular camera at the roadside collects an image, the image can be transmitted to a background server. The background server can acquire images collected by the monocular camera in real time, and determine the currently received image as a to-be-detected image, to detect 3D data of a target relative to the monocular camera, such as three-dimensional coordinates, an external size, a direction, etc. of the target relative to the monocular camera, where the target can be a pedestrian, a vehicle, etc.

[0032] S102, input the to-be-detected image into a target detection model, to obtain detection data of the target in the to-be-detected image, the detection data comprising a depth compensation value, a 3D size compensation value, a 3D angle, a 2D detection frame of the target in the to-be-detected image, and a category.

[0033] In this embodiment, the target detection model comprises a feature extraction network, a 2D detection frame extraction network, a 3D angle extraction network, a depth compensation value extraction network, and a 3D size compensation value extraction network. The output end of the feature extraction network is connected to the input ends of the 2D detection frame extraction network, the 3D angle extraction network, the depth compensation value extraction network, and the 3D size compensation value extraction network.

[0034] The feature extraction network is configured to extract image features of the to-be-detected image. For example, the feature extraction network can be a convolutional neural network, to perform convolution on the to-be-detected image to obtain image features. The 2D detection frame extraction network is configured to output a detection frame of the target in the to-be-detected image and a category of the target when the image features are input. The category can represent that the target is a pedestrian, a vehicle, etc. The 3D angle extraction network is configured to extract a 3D angle of the detected target when the image features are input. The 3D angle can be an included angle between a moving direction of the target and an x-axis of the monocular camera. The depth compensation value extraction network is configured to predict a depth compensation value of the target according to the input image features. The 3D size compensation value extraction network is configured to predict a compensation value of an external size of the target according to the input image features.

[0035] The above networks can be trained by supervised, unsupervised, or other training methods of existing neural networks, which will not be described in detail herein.

[0036] S103, generating a 3D bounding box of the target according to the detection data of the target, the preset background depth map, and the preset 3D average size, as a 3D target detection result.

[0037] The background depth map can be a depth map representing distances from each point on the ground to the monocular camera when there is no target in the detection area after the monocular camera is fixed, that is, each pixel point in the background depth map represents the distance from the point on the ground to the monocular camera. The 3D average size can be the average size of the targets of different categories, for example, when the target is a vehicle, the categories can include small vehicles, medium vehicles, and large vehicles, and different categories of vehicles have different 3D average sizes.

[0038] In an optional embodiment, after obtaining the depth compensation value, the 3D size compensation value, the 3D angle, the 2D detection box of the target in the image to be detected, and the category of the target, the 3D average size corresponding to the category of the target can be determined, and the sum of the 3D average size and the 3D size compensation value is calculated to obtain the 3D size of the target. Further, the target pixel corresponding to the center of the 2D detection box in the preset background depth map is determined, and the depth value associated with the target pixel is obtained from the background depth map. The sum of the depth value associated with the target pixel and the depth compensation value is calculated to obtain the depth value of the target. Finally, the center position of the 2D detection box is converted into a position in the coordinate system of the monocular camera using the calibration parameters of the monocular camera, to obtain the 3D center position of the target, wherein the calibration parameters are the conversion parameters from the image coordinate system of the image to be detected to the coordinate system of the monocular camera. The 3D detection box of the target is generated according to the 3D center position, the 3D size, the 3D angle, and the depth value of the target. The 3D detection box is a cube, the center of the 3D detection box is the 3D center position calculated above, the size of the 3D detection box is equal to the calculated 3D size, and the angle between the 3D detection box and the x coordinate of the monocular camera is equal to the 3D angle calculated above.

[0039] After generating the 3D detection box, the 3D detection box can be sent to the autonomous vehicle, so that the autonomous vehicle calculates the relative position relationship between each target in the detection area and the autonomous vehicle according to the position of the autonomous vehicle and the position of the monocular camera on the roadside, and makes a control decision.

[0040] This invention acquires the image to be detected by a monocular camera over a detection area, then inputs the image into a target detection model to obtain the detection data of the target in the image. This detection data includes the target's depth compensation value, 3D size compensation value, 3D angle, 2D detection box of the target in the image, and category. Then, using the aforementioned detection data, a preset background depth map, and a preset 3D average size, a 3D detection box of the target is generated as the 3D target detection result. This invention predicts the target's depth compensation value, 3D size compensation value, 3D angle, 2D detection box of the target in the image, and category through the target detection model. The predicted data is combined with the preset background depth map and the preset 3D average size to generate the 3D detection box. The target detection model does not need to directly regress the target's depth, making target detection model training simple, the model highly stable, and the predicted data accurate. It can also predict the target's 3D angle, has no requirements on the installation angle of the monocular camera, and can be applied to various installation positions.

[0041] Example 2

[0042] Figure 2A This is a flowchart of a monocular 3D target detection method provided in Embodiment 2 of the present invention. This embodiment optimizes Embodiment 1 as described above. Figure 2A As shown, this monocular 3D target detection method includes:

[0043] S201. Obtain the image to be detected, which is the image of the detection area captured by a monocular camera.

[0044] In this embodiment, images can be received from a monocular camera installed on the roadside via a communication network, and the currently received image can be identified as the image to be detected.

[0045] like Figure 2B The diagram shown is a schematic of the image to be detected in a scenario of this embodiment. In this scenario, a monocular camera is installed above the road to acquire images of the area where the road is located. The detection area can be the area covered by the monocular camera's field of view. The detection area includes the road and vehicles traveling on the road, and can also include targets such as pedestrians.

[0046] S202. Input the image to be detected into the feature extraction network to obtain image features.

[0047] In this embodiment, the target detection model includes a feature extraction network, a 2D bounding box extraction network, a 3D angle extraction network, a depth compensation value extraction network, and a 3D size compensation value extraction network. The output of the feature extraction network is connected to the input of the 2D bounding box extraction network, the 3D angle extraction network, the depth compensation value extraction network, and the 3D size compensation value extraction network, respectively.

[0048] The feature extraction network is an input layer of the entire target detection model, and can be various neural networks, such as a convolutional neural network. After the to-be-detected image is cropped to a preset size and input to the feature extraction network, image features of the to-be-detected image are extracted.

[0049] S203, input the image features into the 2D detection box extraction network, the 3D angle extraction network, the depth compensation value extraction network, and the 3D size compensation value, respectively, to obtain a 2D detection box and a category of the target, a 3D angle, a depth compensation value, and a 3D size compensation value.

[0050] In this embodiment, the 2D detection box extraction network is used to output a detection box of the target in the to-be-detected image and a category of the target when the image features are input, the category can represent that the target is a pedestrian, a vehicle, etc., the 3D angle extraction network is used to extract a 3D angle of the detected target when the image features are input, the 3D angle can be an included angle between a moving direction of the target and an x-axis of the monocular camera, the depth compensation value extraction network is used to predict a depth compensation value of the target according to the input image features, and the 3D size compensation value extraction network is used to predict a compensation value of an external size of the target according to the input image features.

[0051] The 2D detection box extraction network can be trained by using an image set in which 2D detection boxes and categories of targets are labeled, so that the 2D detection box extraction network can learn the ability to output the category and the 2D detection box of the target after the image features of the input image are learned.

[0052] The 3D angle extraction network can be trained by using an image set in which 3D angles of targets are labeled, so that the 3D angle extraction network can learn the ability to output the 3D angle of the target after the image features of the input image are learned.

[0053] When the depth compensation value extraction network is trained, images containing targets of different categories and positions can be collected, and the depth compensation values of the targets of different categories and positions are labeled, and then the images after the depth compensation values are labeled are further used to train the depth compensation value extraction network, so that the depth compensation value extraction network learns the ability to predict the depth compensation value for the targets of different categories and positions.

[0054] When the 3D size compensation value extraction network is trained, images containing targets of different categories and positions can be collected, and 3D size compensation values of the targets of different categories and positions are labeled, and then the images after the 3D size compensation values are labeled are further used to train the depth compensation value extraction network, so that the 3D size compensation value extraction network learns the ability to predict the 3D size compensation value for the targets of different categories and positions.

[0055] The training method of each neural network described above can refer to the supervised neural network training method of the prior art, and the present embodiment will not be described in detail here.

[0056] As shown in Figure 2B , the 2D detection frame extraction network can determine the 2D detection frame A of the target (vehicle) in the image to be detected, and determine that the category of the target is a small vehicle. The 3D angle of the target can be the included angle C between the moving direction F of the target and the x-axis of the monocular camera. Among them, in Figure 2B , in the coordinate system of the monocular camera, the x-axis is the length direction of the image to be detected, the y-axis is the height direction of the image to be detected, and the z-axis is perpendicular to the image to be detected and points to the direction of the image to be detected.

[0057] S204, determine the 3D average size matched with the category, and calculate the sum value of the 3D average size and the 3D size compensation value to obtain the 3D size of the target.

[0058] The present embodiment can determine the 3D average size of targets of different categories through data statistics, and the 3D average size is the outer size of the target, that is, the minimum circumscribed cube that envelopes the target. After predicting the category of the target through the 2D detection frame extraction network, the 3D average size pre-associated with the category can be obtained, and the sum value of the 3D average size and the 3D size compensation value is calculated to obtain the real 3D size of the target.

[0059] Taking a small vehicle as an example, the 3D average size pre-set for a small vehicle is obtained, and the 3D size compensation value of the small vehicle is predicted through the 3D size compensation value extraction network. The sum value of the 3D average size and the 3D size compensation value is the 3D size of the detected small vehicle.

[0060] S205, determine the target pixel corresponding to the center of the 2D detection frame in the preset background depth map, and obtain the depth value associated with the target pixel from the background depth map, wherein the preset depth image is a depth image obtained by calibrating the monocular camera and collecting images of the detection area, with the road surface of the detection area as the background.

[0061] In the present embodiment, the 2D detection frame is a detection frame extracted in the image to be detected, and the background depth map is a depth map of the distances from each point on the ground in the detection area to the monocular camera after the monocular camera is installed. The size of the background depth map is the same as that of the image to be detected. After the 2D detection frame is extracted, the target pixel corresponding to the 2D detection frame can be determined in the background depth map, and the target pixel is associated with a depth value in the background depth map. Among them, the center of the 2D detection frame can be the intersection of the diagonal connecting lines of the 2D detection frame.

[0062] As shown in Figure 2BAs shown, the center of the 2D detection frame A is point P, the coordinates of point P in the to-be-detected image are (x1, y1), the pixel with coordinates (x1, y1) in the background depth map can be taken as a target pixel, and the depth value associated with the target pixel is read, which is the distance from the point on the ground corresponding to the center of the 2D detection frame A of the target to the monocular camera.

[0063] S206, calculate the sum of the depth value and the depth compensation value to obtain the depth value of the target.

[0064] The depth compensation value can be a value less than 0 or greater than 0, and the depth compensation value can be related to factors such as the category of the target and the position of the target in the to-be-detected image. The sum of the depth value and the depth compensation value can be calculated, i.e., the real depth value of the target, i.e., the distance from the target to the monocular camera, i.e., the z coordinate value of the target in the coordinate system of the monocular camera.

[0065] S207, convert the center position of the 2D detection frame to a position in the camera coordinate system using the calibration parameters of the monocular camera to obtain the 3D center position of the target, wherein the calibration parameters are the conversion parameters from the image coordinate system of the to-be-detected image to the coordinate system of the monocular camera.

[0066] Since the center position point P of the 2D detection frame is in the coordinate system of the to-be-detected image, it represents the row and column pixel positions of the center position point P of the 2D detection frame in the to-be-detected image. The position of point P needs to be converted from the coordinate system of the to-be-detected image to the coordinate system of the monocular camera. Moreover, the depth value of the target is obtained in S206, so that the position of the center of the target in the coordinate system of the monocular camera is obtained, i.e., the 3D center position of the target.

[0067] Specifically, after the monocular camera is installed, the monocular camera can be calibrated to obtain the calibration parameters of the monocular camera, which are the conversion parameters from the image coordinate system of the to-be-detected image to the coordinate system of the monocular camera, such as a translation-rotation matrix from the image coordinate system of the to-be-detected image to the coordinate system of the monocular camera. The product of the center position of the 2D detection frame and the translation-rotation matrix can be calculated to obtain the 3D center position of the target. The calibration of the monocular camera can be performed by Zhang Zhengyou calibration method, or other calibration methods can be used.

[0068] S208, generate a 3D detection frame of the target according to the 3D center position, 3D size, 3D angle, and depth value of the target.

[0069] In an optional embodiment, after the 3D center position, the 3D size, the 3D angle and the depth value of the target are determined through the above steps, the 3D center position can be taken as the center of the 3D detection frame, the direction indicated by the 3D angle can be taken as the direction of the 3D detection frame, and the 3D detection frame of the target can be generated by connecting the corner points according to the 3D size, the center of the 3D detection frame and the corner points of the direction of the 3D detection frame.

[0070] As shown in Figure 2B the coordinate system of the monocular camera, the 3D center position P can be taken as the center of the 3D detection frame B, the direction F indicated by the 3D angle C can be taken as the direction of one side of the 3D detection frame B, the 3D detection frame B can be symmetrically bisected with the 3D center position P as the center to obtain the coordinates of the 8 corner points of the 3D detection frame B, and the 3D detection frame can be obtained by connecting the 8 corner points, and the 3D detection frame is output to represent the three-dimensional coordinates of the center of the detected target, the size and the direction, so as to be used for automatic driving vehicle control decision.

[0071] The target detection model of the present application comprises a feature extraction network, a 2D detection frame extraction network, a 3D angle extraction network, a depth compensation value extraction network and a 3D size compensation value extraction network, the output end of the feature extraction network is connected with the input end of the 2D detection frame extraction network, the 3D angle extraction network, the depth compensation value extraction network and the 3D size compensation value extraction network respectively, the 2D detection frame and the category of the target, the 3D angle, the depth compensation value and the 3D size compensation value of the target are obtained after the target detection model is input with the image to be detected, the 3D average size matching the category is determined, the sum of the 3D average size and the 3D size compensation value is calculated to obtain the 3D size of the target, the depth value associated with the target pixel corresponding to the center of the 2D detection frame in the preset background depth map is determined, the sum of the depth value and the depth compensation value is calculated to obtain the depth value of the target, the center position of the 2D detection frame is converted into the position in the coordinate system of the camera to obtain the 3D center position of the target, and the 3D detection frame of the target is generated according to the 3D center position, the 3D size, the 3D angle and the depth value of the target, so that the target detection model does not need to directly regress the depth of the target, the target detection model training is simple, the target detection model has high stability, the predicted data is accurate, the 3D angle of the target can be predicted, the installation angle of the monocular camera is not required, and the monocular camera can be applied to various installation position requirements.

[0072] Embodiment three

[0073] Figure 3 A structure schematic diagram of a monocular 3D target detection device provided for the embodiment three of the present application is shown in Figure 3 As shown in the figure, the monocular 3D target detection device comprises:

[0074] The image to be detected acquisition module 301 is configured to acquire an image to be detected, wherein the image to be detected is an image collected by a monocular camera on a detection region.

[0075] The detection data acquisition module 302 is configured to input the image to be detected into a target detection model to obtain detection data of a target in the image to be detected, wherein the detection data comprises a depth compensation value, a 3D size compensation value, a 3D angle, a 2D detection frame of the target in the image to be detected, and a category of the target.

[0076] The 3D detection frame generation module 303 is configured to generate a 3D detection frame of the target according to the detection data of the target, a preset background depth map, and a preset 3D average size, to serve as a 3D target detection result.

[0077] In one embodiment, the target detection model comprises a feature extraction network, a 2D detection frame extraction network, a 3D angle extraction network, a depth compensation value extraction network, and a 3D size compensation value extraction network, wherein an output end of the feature extraction network is connected to input ends of the 2D detection frame extraction network, the 3D angle extraction network, the depth compensation value extraction network, and the 3D size compensation value extraction network, respectively.

[0078] In one embodiment, the method further comprises:

[0079] The image cropping module is configured to crop the image to be detected to obtain a cropped image to be detected.

[0080] In one embodiment, the detection data acquisition module 302 comprises:

[0081] The image feature extraction unit is configured to input the image to be detected into the feature extraction network to obtain image features.

[0082] The data extraction unit is configured to input the image features into the 2D detection frame extraction network, the 3D angle extraction network, the depth compensation value extraction network, and the 3D size compensation value extraction network, respectively, to obtain a 2D detection frame and a category of the target, a 3D angle, a depth compensation value, and a 3D size compensation value.

[0083] In one embodiment, the 3D detection frame generation module 303 comprises:

[0084] The target 3D size determination unit is configured to determine a 3D average size matched with the category, and calculate a sum value of the 3D average size and the 3D size compensation value to obtain a 3D size of the target.

[0085] A background depth value determination unit is configured to determine a target pixel corresponding to the center of the 2D detection frame in a preset background depth map, and obtain a depth value associated with the target pixel from the background depth map, wherein the preset depth map image is a depth map image of the detection area obtained by calibrating the monocular camera and capturing an image of the detection area, and the depth map image takes the road surface of the detection area as a background.

[0086] A target depth value calculation unit is configured to calculate a sum of the depth value and the depth compensation value to obtain the depth value of the target.

[0087] A target center position determination unit is configured to convert the center position of the 2D detection frame into a position in a camera coordinate system by using the calibration parameters of the monocular camera to obtain the 3D center position of the target, wherein the calibration parameters are conversion parameters from an image coordinate system of a to-be-detected image to a coordinate system of the monocular camera.

[0088] A target 3D detection frame generation unit is configured to generate a 3D detection frame of the target according to the 3D center position, the 3D size, the 3D angle and the depth value.

[0089] In one embodiment, the target 3D detection frame generation unit comprises:

[0090] A center determination subunit is configured to take the 3D center position as the center of the 3D detection frame.

[0091] A detection frame direction determination subunit is configured to take the direction indicated by the 3D angle as the direction of the 3D detection frame.

[0092] A detection frame corner point determination subunit is configured to determine a corner point of the 3D detection frame according to the 3D size, the center of the 3D detection frame and the direction of the 3D detection frame.

[0093] A 3D detection frame generation subunit is configured to connect the corner point to generate the 3D detection frame of the target.

[0094] In one embodiment, the monocular 3D target detection device further comprises:

[0095] A monocular camera calibration module is configured to calibrate the monocular camera to obtain the calibration parameters of the monocular camera.

[0096] The monocular 3D target detection device provided by the embodiment of the application can execute the monocular 3D target detection method provided by the embodiment one and the embodiment two, and has the corresponding function modules and beneficial effects of the execution method.

[0097] Embodiment four

[0098] Figure 4A structural diagram of an electronic device 40 that can be used to implement embodiments of the present application is shown. The electronic device 40 is intended to represent various forms of digital computers, such as laptops, desktops, servers, blade servers, mainframes, and other appropriate computers. The components shown here, their connections and relationships, and their functions, are meant to be examples only, and are not intended to limit implementations of the present application described and / or claimed in this document.

[0099] As shown in Figure 4 The electronic device 40 includes at least one processor 41, and memory, such as read-only memory (ROM) 42, random access memory (RAM) 43, etc., communicatively connected to the at least one processor 41, where the memory stores computer programs executable by the at least one processor. The processor 41 can perform various appropriate actions and processes according to the computer programs stored in the read-only memory (ROM) 42 or loaded into the random access memory (RAM) 43 from the storage unit 48. Various programs and data required for the operation of the electronic device 40 can also be stored in the RAM 43. The processor 41, the ROM 42, and the RAM 43 are connected to each other through a bus 44. An input / output (I / O) interface 45 is also connected to the bus 44.

[0100] Various components in the electronic device 40 are connected to the I / O interface 45, including an input unit 46, such as a keyboard, a mouse, a sensor, etc.; an output unit 47, such as various types of displays, a speaker, etc.; a storage unit 48, such as a magnetic disk, an optical disk, etc.; and a communication unit 49, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 49 allows the electronic device 40 to exchange information / data with other devices through a computer network, such as the Internet, and / or various telecommunication networks.

[0101] The processor 41 can be various general and / or special-purpose processing components with processing and computing capabilities. Some examples of the processor 41 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The processor 41 performs various methods and processes described above, such as the monocular 3D target detection method.

[0102] In some embodiments, the monocular 3D object detection method can be implemented as a computer program tangibly embodied in a computer readable storage medium, e.g., storage unit 48. In some embodiments, portions or all of the computer program can be loaded and / or installed onto electronic device 40 via, e.g., ROM 42 and / or communication unit 49. When the computer program is loaded onto RAM 43 and executed by processor 41, one or more steps of the monocular 3D object detection method described above can be performed. Alternatively, in other embodiments, processor 41 can be configured to perform the monocular 3D object detection method by other means, e.g., with the aid of firmware.

[0103] Various implementations of the systems and techniques described above can be realized in digital electronic circuitry, integrated circuitry, a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), a system on a chip (SOC), a programmable logic device (PLD), a computer hardware, firmware, software, and / or combinations thereof. These various implementations can include implementation in one or more computer programs that are executable and / or interpretable on a programmable system including at least one programmable processor, which can be special or general purpose, coupled to receive data and instructions from, and to transmit data and instructions to, a storage system, at least one input device, and at least one output device.

[0104] Computer programs used to implement the methods of the present application can be written in any combination of one or more programming languages. These computer programs can be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the computer program, when executed by the processor, implements the functions / acts specified in the flowcharts and / or block diagrams. The computer program can be executed entirely on a machine, partially on a machine, partially on a machine as part of a standalone software package, or entirely on a remote machine or server.

[0105] In the context of the present application, a computer-readable storage medium can be a tangible medium that can contain or store a computer program for use by or in connection with an instruction execution system, apparatus, or device. A computer-readable storage medium can include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium can be a machine-readable signal medium. More specific examples of a machine-readable storage medium will include one or more lines of a program of instructions in a transitory signal, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0106] To provide for interaction with a user, the systems and techniques described here can be implemented on an electronic device having a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user and a keyboard and a pointing device (e.g., a mouse or a trackball) by which the user can provide input to the electronic device. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form, including acoustic, speech, or tactile input.

[0107] The systems and techniques described here can be implemented in a computing system that includes a back end component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front end component (e.g., a user computer having a graphical user interface or a Web browser through which a user can interact with an implementation of the systems and techniques described here), or any combination of such back end, middleware, or front end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0108] The computing system can include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system, to solve the defects of large management difficulty and weak business scalability in traditional physical host and VPS service.

[0109] It should be understood that the various forms of flow shown above can be reordered, added to, or have steps deleted. For example, the steps described in the present application can be performed in parallel, in series, or in a different order, as long as the desired results of the technical solutions of the present application can be achieved, and this is not limited herein.

[0110] The above detailed description does not constitute a limitation on the protection scope of the present application. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent replacements, and improvements made within the spirit and principles of the present application shall be included in the protection scope of the present application.

Claims

1. A monocular 3D object detection method, characterized in that, The method comprises the following steps: acquiring a to-be-detected image, the to-be-detected image being an image collected by a monocular camera on a detection area; inputting the to-be-detected image into a target detection model to obtain detection data of a target in the to-be-detected image, the detection data comprising a depth compensation value, a 3D size compensation value, a 3D angle, a 2D detection frame of the target in the to-be-detected image, and a category of the target; determining a 3D average size matching the category, and calculating a sum value of the 3D average size and the 3D size compensation value to obtain a 3D size of the target; determining a target pixel corresponding to a center of the 2D detection frame in a preset background depth map, and acquiring a depth value associated with the target pixel from the background depth map, wherein the preset depth image is a depth image obtained by calibrating the monocular camera to collect an image of the detection area, and taking a road surface of the detection area as a background; calculating a sum value of the depth value and the depth compensation value to obtain a depth value of the target; converting a center position of the 2D detection frame into a position in a camera coordinate system by using a calibration parameter of the monocular camera to obtain a 3D center position of the target, wherein the calibration parameter is a conversion parameter from an image coordinate system of a to-be-detected image to a coordinate system of the monocular camera; generating a 3D detection frame of the target according to the 3D center position, the 3D size, the 3D angle, and the depth value of the target.

2. The monocular 3D object detection method of claim 1, wherein, The target detection model comprises a feature extraction network, a 2D detection frame extraction network, a 3D angle extraction network, a depth compensation value extraction network, and a 3D size compensation value extraction network, and output ends of the feature extraction network are connected to input ends of the 2D detection frame extraction network, the 3D angle extraction network, the depth compensation value extraction network, and the 3D size compensation value extraction network, respectively.

3. The method of claim 1, wherein, Before the to-be-detected image is input into the target detection model to obtain the detection data of the target in the to-be-detected image, the method further comprises the following steps: cropping the to-be-detected image to obtain a cropped to-be-detected image.

4. The monocular 3D object detection method of claim 2, wherein, The step of inputting the to-be-detected image into the target detection model to obtain the detection data of the target in the to-be-detected image comprises the following steps: inputting the to-be-detected image into the feature extraction network to obtain image features; inputting the image features into the 2D detection frame extraction network, the 3D angle extraction network, the depth compensation value extraction network, and the 3D size compensation value extraction network, respectively, to obtain a 2D detection frame and a category of the target, a 3D angle, a depth compensation value, and a 3D size compensation value.

5. The monocular 3D object detection method of claim 1, wherein, The step of generating the 3D detection frame of the target according to the 3D center position, the 3D size, the 3D angle, and the depth value of the target comprises the following steps: taking the 3D center position as a center of the 3D detection frame; taking a direction indicated by the 3D angle as a direction of the 3D detection frame; determining a corner point of the 3D detection frame according to the 3D size, the center of the 3D detection frame, and the direction of the 3D detection frame; connecting the corner point to generate the 3D detection frame of the target.

6. The monocular 3D object detection method of any one of claims 1-5, wherein, Before generating the 3D bounding box of the target according to the detection data of the target, a preset background depth map, and a preset 3D average size as a 3D target detection result, the method further comprises: Calibrating the monocular camera to obtain calibration parameters of the monocular camera.

7. A monocular 3D object detection apparatus, characterized by comprising: The method comprises: an image to be detected acquisition module configured to acquire an image to be detected, the image to be detected being an image collected by a monocular camera on a detection region; a detection data acquisition module configured to input the image to be detected into a target detection model to obtain detection data of a target in the image to be detected, the detection data comprising a depth compensation value, a 3D size compensation value, a 3D angle, a 2D bounding box of the target in the image to be detected, and a category of the target; a 3D bounding box generation module configured to generate a 3D bounding box of the target according to the detection data of the target, a preset background depth map, and a preset 3D average size as a 3D target detection result; the 3D bounding box generation module comprises: a target 3D size determination unit configured to determine a 3D average size matched with the category, and calculate a sum of the 3D average size and the 3D size compensation value to obtain a 3D size of the target; a background depth value determination unit configured to determine a target pixel corresponding to a center of the 2D bounding box in a preset background depth map, and obtain a depth value associated with the target pixel from the background depth map, wherein the preset depth map is a depth map obtained by collecting an image of the detection region with a road surface of the detection region as a background after calibrating the monocular camera; a target depth value calculation unit configured to calculate a sum of the depth value and the depth compensation value to obtain a depth value of the target; a target center position determination unit configured to convert a center position of the 2D bounding box into a position in a camera coordinate system by using calibration parameters of the monocular camera to obtain a 3D center position of the target, wherein the calibration parameters are conversion parameters from an image coordinate system of the image to be detected to a coordinate system of the monocular camera; a target 3D bounding box generation unit configured to generate a 3D bounding box of the target according to the 3D center position of the target, the 3D size, the 3D angle, and the depth value.

8. An electronic device, comprising: The electronic device comprises: at least one processor; and a memory connected with the at least one processor in communication; wherein the memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor to enable the at least one processor to execute the monocular 3D target detection method in any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, The computer readable storage medium stores computer instructions for enabling the processor to execute the monocular 3D target detection method in any one of claims 1-6 when executed.

Images