Object detection method, device, electronic device and autonomous driving vehicle

Abstract

The present disclosure provides an object detection method, device, electronic device, and autonomous vehicle, relating to the field of autonomous driving. The specific implementation scheme comprises: acquiring a monocular camera image; performing prediction on the monocular camera image to obtain the 2D center point of a target object in the monocular camera image and 3D prediction information of the target object; determining the 3D center point of the target object based on the 2D center point of the target object; and determining the position of the target object based on the 3D center point and the 3D prediction information of the target object.

Description

Technical Field

[0001] The present disclosure relates to the field of autonomous driving, and in particular to an object detection method, device, electronic equipment, and autonomous driving vehicle. Background Art

[0002] Currently, monocular 3D object detection technology refers to the technology that uses a monocular camera sensor to detect 3D obstacles in autonomous driving scenarios. In today's autonomous driving technology, in order to obtain more abundant obstacle information and improve the reliability of autonomous driving systems, vision-based obstacle detection technology plays an increasingly important role. In related technologies, there are generally two solutions for detecting obstacles: (1) using traditional object detection technology to detect obstacles. (2) using obstacle detection technology based on neural networks.

[0003] However, when using the above solution to detect obstacles, only the 2D information of the object can be detected, the processing performance is low, and there will be many missed detections and false detections. Summary of the Invention

[0004] The present disclosure provides an object detection method, device, electronic device, and autonomous driving vehicle.

[0005] According to one aspect of the present disclosure, a method for object detection is provided, comprising: acquiring a monocular camera image; performing prediction on the monocular camera image to obtain a 2D center point of a target object in the monocular camera image and 3D prediction information of the target object; determining a 3D center point of the target object based on the 2D center point of the target object; and determining a position result of the target object based on the 3D center point and the 3D prediction information of the target object.

[0006] Optionally, determining the 3D center point of the target object based on the 2D center point of the target object includes: extracting an offset between the 2D center point and the 3D center point of the target object from 3D prediction information of the target object; and determining the 3D center point of the target object based on the 2D center point and the offset.

[0007] Optionally, determining the 3D center point of the target object based on the 2D center point and the offset includes: determining the 3D center point of the target object in the image coordinate system based on the 2D center point of the target object and the offset; determining the 3D center point of the target object in the camera coordinate system based on the 3D center point of the target object in the image coordinate system and the camera intrinsic parameters of the monocular camera.

[0008] Optionally, determining the position result of the target object based on the 3D center point and the 3D prediction information of the target object includes: when the 3D prediction information of the target object includes the angle of the target object, the scale of the target object and the corner points of the target object, determining the 3D border points of the target object based on the 3D center point, the angle and the scale; using the corner points and the 3D border points of the target object, optimizing the 3D center point to obtain the position result of the target object.

[0009] Optionally, predicting the monocular camera image to obtain the 2D center point of the target object in the monocular camera image and the 3D prediction information of the target object includes: using an object prediction model to predict the monocular camera image to obtain the 2D center point of the target object and the 3D prediction information of the target object, wherein the object prediction model is trained using multiple sets of data, and the multiple sets of data include: a sample monocular camera image, and 3D information obtained after annotating the sample monocular camera image.

[0010] Optionally, the monocular camera image is an image captured by a monocular camera sensor in an autonomous driving scene, and the target object is an obstacle in the autonomous driving scene.

[0011] According to another aspect of the present disclosure, an object detection apparatus is provided, comprising: an acquisition module for acquiring a monocular camera image; a prediction module for predicting the monocular camera image to obtain a 2D center point of a target object in the monocular camera image and 3D prediction information of the target object; a first determination module for determining a 3D center point of the target object based on the 2D center point of the target object; and a second determination module for determining a position result of the target object based on the 3D center point and the 3D prediction information of the target object.

[0012] Optionally, the first determination module includes: a first determination unit, used to extract the offset between the 2D center point and the 3D center point of the target object from the 3D prediction information of the target object; and a second determination unit, used to determine the 3D center point of the target object based on the 2D center point and the offset.

[0013] Optionally, the second determination unit includes: a first determination subunit, used to determine the 3D center point of the target object in the image coordinate system based on the 2D center point of the target object and the offset; a second determination subunit, used to determine the 3D center point of the target object in the camera coordinate system based on the 3D center point of the target object in the image coordinate system and the camera intrinsic parameters of the monocular camera.

[0014] Optionally, the 3D prediction information of the target object includes: the angle of the target object, the scale of the target object and the corner points of the target object, and the second determination module includes: a third determination unit, used to determine the 3D border points of the target object based on the 3D center point, the angle and the scale; a processing unit, used to use the corner points and the 3D border points of the target object to optimize the 3D center point to obtain the position result of the target object.

[0015] Optionally, the prediction module includes: a prediction unit, used to use an object prediction model to predict the monocular camera image to obtain the 2D center point of the target object and the 3D prediction information of the target object, wherein the object prediction model is trained using multiple sets of data, and the multiple sets of data include: a sample monocular camera image, and 3D information obtained after annotating the sample monocular camera image.

[0016] Optionally, the monocular camera image is an image captured by a monocular camera sensor in an autonomous driving scene, and the target object is an obstacle in the autonomous driving scene.

[0017] According to another aspect of the present disclosure, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute any one of the methods described.

[0018] According to another aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to cause the computer to execute any one of the methods described.

[0019] According to another aspect of the present disclosure, a computer program product is provided, comprising a computer program, which implements the method according to any one of the preceding claims when executed by a processor.

[0020] According to another aspect of the present disclosure, an autonomous driving vehicle is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute any one of the methods described.

[0021] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present disclosure, nor are they intended to limit the scope of the present disclosure. Other features of the present disclosure will become readily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] The accompanying drawings are provided to facilitate a better understanding of the present invention and do not constitute a limitation of the present disclosure.

[0023] Figure 1 is a flow chart of an object detection method according to an embodiment of the present disclosure;

[0024] Figure 2 is a flowchart of a method provided in an optional embodiment of the present disclosure;

[0025] Figure 3 is a structural block diagram of an object detection device according to an embodiment of the present disclosure;

[0026] Figure 4 is a block diagram of an electronic device for implementing the object detection method according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0027] The following description of exemplary embodiments of the present disclosure is made in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding. These details should be considered as merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0028] Example 1

[0029] Figure 1 is a flow chart of an object detection method according to an embodiment of the present disclosure. Figure 1 As shown, the method includes the following steps:

[0030] Step S101, acquiring a monocular camera image;

[0031] Step S102, predicting the monocular camera image to obtain the 2D center point of the target object in the monocular camera image and the 3D prediction information of the target object;

[0032] Step S103, determining the 3D center point of the target object based on the 2D center point of the target object;

[0033] Step S104 : determining the position result of the target object based on the 3D center point and the 3D prediction information of the target object.

[0034] Through the above steps, the 2D center point of the target object in the monocular camera image and its 3D prediction information are obtained by predicting the acquired monocular camera image. Based on the 2D center point of the target object, the 3D center point of the target object is determined. The position of the target object is then determined based on the 3D center point and the 3D prediction information. Specifically, the 2D center point and 3D prediction information are obtained by predicting the monocular camera image, and then the position of the target object is determined based on the 2D center point and 3D prediction information. Compared to solutions that predict based on a specific area or directly predict the 3D center point, on the one hand, the 3D prediction information is obtained directly from the image, which can make the position result more accurate compared to subsequent processing in related technologies. On the other hand, because the 3D center point is not directly predicted, parts of the target object (e.g., truncated obstacles) are not missed. The above processing can achieve the goal of accurately detecting the position of the target object, thereby solving the technical problem of low detection accuracy when detecting the position of the target object in related technologies.

[0035] As an optional embodiment, a monocular camera image must first be acquired. This can be obtained through various means, such as using a monocular camera sensor in an autonomous driving scenario, or using an on-board monocular camera. After acquiring the monocular camera image, the image can be preprocessed before prediction is performed. For example, the monocular camera image can be preprocessed to remove noise, correct for uneven lighting, and enhance contrast before being input into the model for prediction. This yields the 2D center point of the target object in the monocular camera image and 3D prediction information of the target object. The ability to directly obtain 3D prediction information from the monocular camera image ensures that the 3D prediction information is based on information contained in the monocular camera image itself, rather than information obtained through post-processing. This results in more accurate 3D prediction information. After obtaining the 2D center point of the target object in the monocular camera image, the 3D center point of the target object is determined based on the 2D center point of the target object in the monocular camera image. The 3D center point is obtained based on the 2D center point of the target object, not directly predicted. Since the prediction of the 2D center point is accurate and no omissions occur, it can ensure that the target object is fully processed and no part of the target object is missed, such as the center point of a truncated obstacle. The position result of the target object is determined based on the 3D center point and the 3D prediction information of the target object. Because the position result of the target object is determined based on the 3D center point and the 3D prediction information of the target object, the number and position results of the target objects obtained can be more accurate.

[0036] It should be noted that in the autonomous driving scenario, the target object can be an obstacle in the autonomous driving scenario, and different target objects can be selected according to different scenarios to meet actual applications and needs. The 3D prediction information of the target object can include multiple types, such as the angle of the target object, the scale of the target object, and the corners of the target object, wherein the 3D prediction information of the target object can be information obtained based on a coordinate system. In the autonomous driving scenario, the 3D prediction information of the target object is the angle of the obstacle, the scale of the obstacle, the corners of the obstacle, and so on. As an optional embodiment, when predicting a monocular camera image and obtaining the 2D center point of the target object in the monocular camera image and the 3D prediction information of the target object, an object prediction model can be used to predict the monocular camera image, wherein the object prediction model is trained using multiple sets of data, and the multiple sets of data include: a sample monocular camera image, and 3D information obtained after annotating the sample monocular camera image. The object prediction model is trained using multiple sets of data. Since the multiple sets of data include sample monocular camera images and 3D information obtained after annotating the sample monocular camera images, problems such as inaccurate predictions and prediction omissions can be effectively avoided, providing a basis for subsequent use of the object prediction model to obtain accurate prediction results. In addition, when the object prediction model provided by this optional embodiment predicts a monocular camera image, the structure of the object prediction model can be roughly divided into two parts. The first part is the backbone network, which is mainly used to extract network features, for example, for detecting various object features involved in the monocular camera image; the second part is the detection head, which is mainly used to detect target objects (such as obstacles) based on the features (images) output by the backbone. Unlike the region-based obstacle detection algorithm, the target object detection algorithm of this object prediction model does not require the use of predefined anchors (i.e., it does not need to predetermine the region where the target object is located). Instead, it uses key points (e.g., the 2D center point of the target object) to represent (the target object, such as an obstacle), and the output is more easily scalable. The detection results output by the core detection network, taking obstacles as the target object, can include the following components: the obstacle's 2D center point, the obstacle's corner points in the image, the offset between the 2D and 3D center points, 3D scale information, 3D position information, and angle information. It should be noted that the obstacle detection algorithm in this object prediction model does not require the use of non-maximum suppression to process the detection results; instead, it simply extracts the point with the highest score in the local area to identify the corresponding obstacle.

[0037] As an optional embodiment, the object prediction model may include multiple models, for example, an object prediction model based on a deep neural network. It should be noted that the object prediction model based on a deep neural network mentioned above is only one example. This model can be selected based on different needs, providing a variety of object prediction models based on different methods, making it more flexible and convenient to use, and greatly improving the applicability of the object prediction model.

[0038] As an optional embodiment, when determining the 3D center point of the target object based on the 2D center point of the target object, the 3D center point of the target object can be determined based on the offset between the 2D center point and the 3D center point, which can include the following steps: extracting the offset between the 2D center point and the 3D center point of the target object from the 3D prediction information of the target object, and determining the 3D center point of the target object based on the 2D center point and the offset. The 3D center point is determined based on the predicted offset between the 2D and 3D and the 2D center point. The offset between the 2D center point and the actual 3D center point can be predicted by the above-mentioned object prediction model, that is, it is obtained based on the original information of the monocular camera. Therefore, the offset between the 2D center point and the actual 3D center point is also accurate, and thus, the 3D center point obtained based on the offset and the 2D center point is also more accurate.

[0039] As an optional embodiment, in the process of determining the 3D center point of the target object based on the 2D center point and the offset, the 3D center point of the target object in the image coordinate system can be first determined based on the 2D center point and the offset of the target object, and the 3D center point of the target object in the camera coordinate system can be determined based on the 3D center point of the target object in the image coordinate system and the camera intrinsic parameters of the monocular camera. By determining the 3D center point of the target object in the image coordinate system and performing a coordinate system transformation in combination with the camera intrinsic parameters of the monocular camera, the 3D center point of the target object in the camera coordinate system is determined. That is, by using a geometric method to adjust and correct the center point of the target object, and combining the camera intrinsic parameters to display the 3D center point of the target object, the center point of the target object in the real 3D scene is obtained, which provides a basis for the accurate position of the target object in the real 3D scene obtained by subsequent calculation.

[0040] As an optional embodiment, when determining the target object's position based on the 3D center point and the target object's 3D prediction information, the following steps may also be included: when the target object's 3D prediction information includes the target object's angle, scale, and corner points, determining the target object's 3D bounding box points based on the 3D center point, angle, and scale, thereby calculating the target object's 3D bounding box points. Generally, a 3D bounding box consists of eight points, which appear as a cube containing the target object in space. Using the target object's corner points and 3D bounding box points, the 3D center point can be optimized to obtain the target object's position. The calculated 3D true position is optimized using the target object's corner points and bounding box points, resulting in a more accurate 3D true position. Therefore, calculating the target object's 3D bounding box points using the 3D angle and scale predicted from the monocular camera image using the object prediction model is both fast and accurate. Furthermore, optimizing the 3D center point obtained above using the 3D bounding box points and the corner points predicted from the monocular camera image using the object prediction model yields a more accurate target object position.

[0041] Based on the above embodiment and optional embodiment, an optional implementation is provided, which is described in detail below. It should be noted that in this optional implementation, obstacle detection in a monocular camera image is used as an example for description.

[0042] As mentioned above, in the related technologies, vision-based obstacle detection technology usually adopts the following two schemes: (1) using traditional object detection technology to detect obstacles, and using image texture, edge and other features to detect obstacles in the image; (2) using neural network-based obstacle detection algorithm, the process is to use a neural network model to detect obstacles in the image captured by a monocular camera, output the category, bounding box, scale and angle information of the obstacle, and then use post-processing to estimate the actual position of the obstacle in the camera coordinate system.

[0043] However, when using the above solutions to solve the problem, the problems and shortcomings brought about are different depending on the usage scenario: Solution (1) requires manual design of feature extraction operators, which has low detection efficiency and is also easily interfered by noise. It does not have good generalization and is difficult to apply in actual scenarios. Solution (2) uses a neural network method to expand the 2D obstacle detection algorithm to 3D. This solution effectively solves the problem of low efficiency and poor effect of solution (1). However, since the 3D distance is not predicted, it can only be obtained through post-processing, which limits the effect of distance prediction and cannot effectively improve the accuracy of 3D obstacle detection. In other words, the solutions in the related art are difficult to accurately predict the location information of obstacles.

[0044] In order to solve the problem of difficulty in accurately predicting the location information of obstacles, it is necessary to consider two aspects: the center point prediction problem and the 3D position information prediction problem. In view of this, an optional embodiment of the present disclosure provides a monocular visual obstacle detection method. By considering these two aspects, the center point prediction problem and the 3D position information prediction problem that arise in the process of designing and implementing obstacle detection are solved:

[0045] Regarding the center point prediction problem, accurately finding the 3D center point of an obstacle is essential for obstacle detection. Some algorithms in related technologies typically use the 2D center point of the obstacle in the image instead of the 3D center point. However, there is often a significant offset between the positions of the two points, especially for close-range obstacles, resulting in a significant deviation between the final detected obstacle and the actual situation. Other algorithms in related technologies directly predict the 3D center point in the image. While this avoids the aforementioned deviation, there are cases where the 3D center point of some obstacles is outside the image, resulting in missed detection of truncated obstacles.

[0046] To address the center point prediction problem, the method in an optional embodiment of the present invention uses a 2D center point combined with 2D and 3D offsets to determine the 3D center point of an obstacle. All obstacles in the image have corresponding 2D center points, ensuring that truncated obstacles are not missed. Furthermore, the presence of offsets allows our algorithm to obtain a relatively accurate 3D center point, minimizing significant offsets when mapping obstacles to 3D coordinates.

[0047] When it comes to predicting 3D position information, related technologies rely solely on monocular camera images for detection. Because monocular images lack depth information, they cannot directly and accurately determine the distance to obstacles. Consequently, these algorithms use the 2D bounding boxes of obstacles to calculate distances, which is time-consuming and dependent on the initial values predicted by the model. This often results in unsatisfactory results.

[0048] To address the problem of 3D position prediction, an optional embodiment of the present invention uses a combination of model depth prediction and geometric post-processing optimization to obtain the 3D position of obstacles. This method directly uses a neural network model to output the distance to the obstacle, and then optimizes the result using the geometric correspondence between 2D and 3D keypoints to obtain the final position.

[0049] The following is a detailed description of the monocular visual obstacle detection method provided in an optional embodiment of the present disclosure:

[0050] Taking the scenario of autonomous driving as an example, the method provided in the optional embodiment of the present disclosure is based on a neural network model and a large amount of labeled data. It expands on the basis of the 2D obstacle detection algorithm to obtain 3D obstacle information, which can effectively solve the problem of low detection accuracy caused by 3D center point offset and inaccurate depth prediction in monocular obstacle detection. Figure 2 is a flow chart of a method provided in an optional embodiment of the present disclosure, such as Figure 2 As shown, the key to the method in the optional embodiment of the present disclosure lies in the obstacle detection network, i.e., model prediction, and geometric constraint optimization, i.e., 3D reconstruction. The two parts are introduced below respectively:

[0051] (1) Model prediction: The target image is detected using a keypoint-based visual obstacle detection model, which can annotate obstacle information in the image, including obstacle category, scale, angle, and position. One obstacle can be identified by a 2D center point, meaning each 2D center point represents one obstacle. If there are multiple obstacles, multiple 2D center points can be identified.

[0052] (2) 3D reconstruction: Based on the information predicted by the key point-based visual obstacle detection model, the obstacle position is adjusted using a geometric optimization algorithm, and the camera intrinsic parameters are used to reconstruct the 3D position of the obstacle.

[0053] Here are the steps:

[0054] S1, calculate the 3D center point in the image coordinate system based on the above 2D center point and offset:

[0055] x c =x t +Δx;

[0056] y c =y t +Δy;

[0057] It should be noted that the x in the formula t 、y t are the 2D center points of the obstacle in the image, Δx and Δy are the offsets of the 2D center point and 3D center point predicted by the model.

[0058] S2, after obtaining the 3D center point in the image coordinate system, the 3D center point in the camera coordinate system can be calculated based on the distance and camera intrinsic parameters, that is, the actual position of the obstacle:

[0059]

[0060] It should be noted that the x in the formula c 、y c is the 3D center point obtained in the previous step, is the camera internal parameter, and x, y, and z represent the actual position of the obstacle.

[0061] S3, after obtaining the center point coordinates, the 3D bounding box of the obstacle can be calculated by combining the angle and scale information of the obstacle:

[0062]

[0063] It should be noted that in the formula, θ is the angle, hwl represents the size of the obstacle, and x, y, and z are the positions calculated in the previous step. Finally, the 8 points of the 3D border are obtained, and the 8 points of the 3D border are represented by B.

[0064] In step S4, the bounding box points obtained in the previous step and the bounding box points directly obtained by model prediction, i.e., corner points, are further optimized to obtain the obstacle position information by calculating the position x, y, and z.

[0065]

[0066] It should be noted that x * ,y * , z * This is the final calculated obstacle position information.

[0067] That is, through the method provided in the optional embodiment of the present invention, by inputting a monocular camera image, the 2D center point of the obstacle, the corner points of the obstacle in the image, the offset between the 2D center point and the 3D center point, 3D scale information, 3D position information and angle information can be obtained.

[0068] Through the above optional implementation, it is possible to fully utilize the depth information in the image and the geometric relationship between the detection results, thereby improving the reliability of visual obstacle detection technology in autonomous driving scenarios.

[0069] It should be noted that the acquisition, storage and application of user personal information involved in the technical solution of the present disclosure are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals. Moreover, for the aforementioned method embodiments, for the sake of simplicity of description, they are all expressed as a series of action combinations, but those skilled in the art should know that the present disclosure is not limited by the order of the actions described, because according to the present disclosure, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also know that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present disclosure.

[0070] Through the description of the above implementation methods, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus the necessary general hardware platform, and of course it can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present disclosure, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for enabling a terminal device (which can be a mobile phone, computer, server, or network device, etc.) to execute the methods of the various embodiments of the present disclosure.

[0071] Example 2

[0072] According to an embodiment of the present disclosure, a device for implementing the above object detection method is also provided. Figure 3 is a structural block diagram of an object detection device according to an embodiment of the present disclosure, such as Figure 3 As shown, the device includes: an acquisition module 301, a prediction module 302, a first determination module 303 and a second determination module 304. The device will be described in detail below.

[0073] An acquisition module 301 is used to acquire a monocular camera image; a prediction module 302 is connected to the acquisition module 301 and is used to predict the monocular camera image to obtain the 2D center point of the target object in the monocular camera image and the 3D prediction information of the target object; a first determination module 303 is connected to the prediction module 302 and is used to determine the 3D center point of the target object based on the 2D center point of the target object; and a second determination module 304 is connected to the first determination module 303 and is used to determine the position result of the target object based on the 3D center point and the 3D prediction information of the target object.

[0074] As an optional embodiment, the first determination module 303 includes: a first determination unit, used to extract the offset between the 2D center point and the 3D center point of the target object from the 3D prediction information of the target object; and a second determination unit, used to determine the 3D center point of the target object based on the 2D center point and the offset.

[0075] As an optional embodiment, the second determination unit includes: a first determination subunit, used to determine the 3D center point of the target object in the image coordinate system based on the 2D center point and offset of the target object; a second determination subunit, used to determine the 3D center point of the target object in the camera coordinate system based on the 3D center point of the target object in the image coordinate system and the camera intrinsic parameters of the monocular camera.

[0076] As an optional embodiment, the 3D prediction information of the target object includes: the angle of the target object, the scale of the target object and the corner points of the target object. The second determination module includes: a third determination unit, used to determine the 3D border point of the target object based on the 3D center point, angle and scale; a processing unit, used to use the corner points and 3D border points of the target object to optimize the 3D center point to obtain the position result of the target object.

[0077] As an optional embodiment, the prediction module 302 includes: a prediction unit, used to use an object prediction model to predict the monocular camera image to obtain the 2D center point of the target object and the 3D prediction information of the target object, wherein the object prediction model is trained using multiple sets of data, and the multiple sets of data include: a sample monocular camera image, and 3D information obtained after annotating the sample monocular camera image.

[0078] As an optional embodiment, the monocular camera image is an image captured by a monocular camera sensor in an autonomous driving scene, and the target object is an obstacle in the autonomous driving scene.

[0079] It should be noted here that the above-mentioned acquisition module 301, prediction module 302, first determination module 303 and second determination module 304 correspond to steps S101 to S104 in implementing the object detection method, and the instances and application scenarios implemented by the multiple modules and corresponding steps are the same, but are not limited to the contents disclosed in the above-mentioned embodiment 1.

[0080] Example 3

[0081] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium, a computer program product, and an autonomous driving vehicle.

[0082] Figure 4 is a block diagram of an electronic device for implementing the object detection method of an embodiment of the present disclosure. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are provided for example only and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0083] like Figure 4As shown, the device 400 includes a computing unit 401, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 402 or a computer program loaded from a storage unit 408 into a random access memory (RAM) 403. Various programs and data required for the operation of the device 400 can also be stored in the RAM 403. The computing unit 401, the ROM 402, and the RAM 403 are connected to each other via a bus 404. An input / output (I / O) interface 405 is also connected to the bus 404.

[0084] Various components in device 400 are connected to I / O interface 405, including an input unit 406, such as a keyboard, mouse, etc.; an output unit 407, such as various types of displays, speakers, etc.; a storage unit 408, such as a magnetic disk, optical disk, etc.; and a communication unit 409, such as a network card, modem, wireless communication transceiver, etc. Communication unit 409 allows device 400 to exchange information / data with other devices via a computer network such as the Internet and / or various telecommunication networks.

[0085] The computing unit 401 can be a variety of general-purpose and / or specialized processing components with processing and computing capabilities. Some examples of the computing unit 401 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units that run machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 401 performs the various methods and processes described above, such as the object detection method. For example, in some embodiments, the object detection method can be implemented as a computer software program that is tangibly contained in a machine-readable medium, such as the storage unit 408. In some embodiments, part or all of the computer program can be loaded and / or installed on the device 400 via the ROM 402 and / or the communication unit 409. When the computer program is loaded into the RAM 403 and executed by the computing unit 401, one or more steps of the object detection method described above can be performed. Alternatively, in other embodiments, the computing unit 401 can be configured to perform the object detection method by any other suitable means (e.g., by means of firmware).

[0086] Various embodiments of the systems and techniques described herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), system-on-chip systems (SOCs), programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include being implemented in one or more computer programs that are executable and / or interpreted on a programmable system that includes at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0087] The program code for implementing the method of the present disclosure can be written in any combination of one or more programming languages. These program codes can be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device so that when the program code is executed by the processor or controller, the functions / operations specified in the flow chart and / or block diagram are implemented. The program code can be executed entirely on the machine, partially on the machine, as a stand-alone software package, partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0088] In the context of the present disclosure, a machine-readable medium can be a tangible medium that can contain or store a program for use by or in conjunction with an instruction execution system, device or equipment. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium can include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0089] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0090] The systems and techniques described herein can be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer having a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), and the Internet.

[0091] A computer system may include a client and a server. The client and server are generally remote from each other and typically interact through a communication network. The client-server relationship arises through computer programs running on the respective computers and having a client-server relationship with each other. The server may be a cloud server, a server in a distributed system, or a server integrated with a blockchain.

[0092] An autonomous driving vehicle comprises: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform any of the above-mentioned object detection methods.

[0093] It should be understood that the various forms of the processes shown above can be used to reorder, add, or delete steps. For example, the steps described in this disclosure can be performed in parallel, sequentially, or in a different order, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved. This is not limited herein.

[0094] The above specific embodiments do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art will appreciate that various modifications, combinations, sub-combinations, and substitutions may be made based on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure shall be included within the scope of protection of this disclosure.

Claims

1. An object detection method, comprising: Get monocular camera images; Predicting the monocular camera image to obtain a 2D center point of a target object in the monocular camera image and 3D prediction information of the target object; Determining a 3D center point of the target object based on the 2D center point of the target object; Determining a position result of the target object based on the 3D center point and the 3D prediction information of the target object; Wherein, determining the position result of the target object based on the 3D center point and the 3D prediction information of the target object includes: determining the 3D bounding box points of the target object based on the 3D center point, the angle, and the scale, when the 3D prediction information of the target object includes the angle of the target object, the scale of the target object, and the corner points of the target object; and optimizing the 3D center point using the corner points and the 3D bounding box points of the target object to obtain the position result of the target object; The step of optimizing the 3D center point using the corner points and the 3D border points of the target object to obtain a position result of the target object includes: determining the position information of the target object that minimizes the distance between the corner points and the 3D border points as the position result; The mathematical expression for determining the 3D border points is as follows: ， in, are the coordinates of the 3D border points, is the angle, For the scale, are the coordinates of the 3D center point.

2. The method according to claim 1, wherein The determining the 3D center point of the target object based on the 2D center point of the target object includes: Extracting an offset between a 2D center point and a 3D center point of the target object from the 3D prediction information of the target object; A 3D center point of the target object is determined based on the 2D center point and the offset.

3. The method according to claim 2, wherein: Determining the 3D center point of the target object based on the 2D center point and the offset includes: Determining a 3D center point of the target object in an image coordinate system based on the 2D center point of the target object and the offset; The 3D center point of the target object in the camera coordinate system is determined based on the 3D center point of the target object in the image coordinate system and the camera intrinsic parameters of the monocular camera.

4. The method according to claim 1, wherein The predicting of the monocular camera image to obtain the 2D center point of the target object in the monocular camera image and the 3D prediction information of the target object includes: An object prediction model is used to predict the monocular camera image to obtain a 2D center point of the target object and 3D prediction information of the target object, wherein the object prediction model is trained using multiple sets of data, and the multiple sets of data include: a sample monocular camera image and 3D information obtained by annotating the sample monocular camera image.

5. The method according to any one of claims 1 to 4, wherein The monocular camera image is an image captured by a monocular camera sensor in an autonomous driving scene, and the target object is an obstacle in the autonomous driving scene.

6. An object detection device comprising: Acquisition module, used to acquire monocular camera images; A prediction module, configured to predict the monocular camera image to obtain a 2D center point of a target object in the monocular camera image and 3D prediction information of the target object; A first determining module is configured to determine a 3D center point of the target object based on the 2D center point of the target object; A second determining module is configured to determine a position result of the target object based on the 3D center point and the 3D prediction information of the target object; The 3D prediction information of the target object includes: the angle of the target object, the scale of the target object, and the corner points of the target object; the second determination module includes: a third determination unit, configured to determine the 3D bounding box points of the target object based on the 3D center point, the angle, and the scale; and a processing unit, configured to optimize the 3D center point using the corner points and the 3D bounding box points of the target object to obtain a position result of the target object; The processing unit is further configured to determine the position information of the target object that minimizes the distance between the corner point and the 3D border point as the position result; The mathematical expression for determining the 3D border points is as follows: ， in, are the coordinates of the 3D border points, is the angle, For the scale, are the coordinates of the 3D center point.

7. The device according to claim 6, wherein The first determining module includes: a first determining unit, configured to extract an offset between a 2D center point and a 3D center point of the target object from the 3D prediction information of the target object; A second determining unit is configured to determine a 3D center point of the target object based on the 2D center point and the offset.

8. The device according to claim 7, wherein The second determining unit includes: a first determining subunit, configured to determine a 3D center point of the target object in an image coordinate system based on the 2D center point of the target object and the offset; The second determining subunit is used to determine the 3D center point of the target object in the camera coordinate system based on the 3D center point of the target object in the image coordinate system and the camera intrinsic parameters of the monocular camera.

9. The device according to claim 6, wherein The prediction module includes: A prediction unit is configured to predict the monocular camera image using an object prediction model to obtain a 2D center point of the target object and 3D prediction information of the target object, wherein the object prediction model is trained using multiple sets of data, the multiple sets of data including: a sample monocular camera image and 3D information obtained by annotating the sample monocular camera image.

10. The device according to any one of claims 6 to 9, wherein The monocular camera image is an image captured by a monocular camera sensor in an autonomous driving scene, and the target object is an obstacle in the autonomous driving scene.

11. An electronic device comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 5.

12. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1 to 5.

13. A computer program product comprising a computer program, which, when executed by a processor, implements the method according to any one of claims 1 to 5.

14. An autonomous driving vehicle comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 5.

Images