3D target detection method and device, storage medium and equipment
Through a monocular camera and depth prediction model, the cone is constructed for image feature processing, which solves the problem of high hardware cost of vehicle three-dimensional object detection and realizes efficient and accurate three-dimensional position detection.
Patent Information
- Application Number
- CN202411728355.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-11-28
- Publication Date
- 2025-03-11
AI Technical Summary
In the prior art, the vehicle detects the three-dimensional position information of target obstacles with high hardware costs.
The visual image of the vehicle's surrounding environment is obtained by a monocular camera, image features are extracted, and these features are processed based on the depth prediction model, and the view cone is constructed to predict the probability value of the image features on each depth plane, thereby obtaining the depth value and three-dimensional position information of the target.
The target three-dimensional position detection can be performed using only a single-eye camera, effectively reducing the hardware cost of environment perception, while improving detection accuracy and efficiency.
Smart Images
Figure CN119600571A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of vehicle perception technology, and more specifically, to a 3D target detection method, apparatus, storage medium, and device. Background Art
[0002] In the field of autonomous driving, perceiving the vehicle's surroundings is a crucial foundation for path planning. Object detection is a crucial component of environmental perception. Determining the location of obstacles around the vehicle plays a crucial role in ensuring safe driving in both assisted and autonomous driving scenarios. Related technologies primarily rely on depth sensors, lidar, or multiple vision sensors to detect the three-dimensional position of obstacles, resulting in high hardware costs. Summary of the Invention
[0003] The purpose of this application is to provide a 3D target detection method, device, storage medium and equipment, aiming to solve the problem of high hardware cost in the related art of the vehicle's detection of the three-dimensional position information of the target obstacle.
[0004] In the first aspect, the present application provides a 3D target detection method, including: obtaining a visual image of the vehicle's surrounding environment through a monocular camera, and extracting image features from the visual image; processing the image features based on a depth prediction model to obtain a depth value of the detection target in the visual image; during the training process, the depth prediction model learns to construct a visual cone, predict the probability values of the input image features on each depth plane of the visual cone, and obtain the depth value of the target based on the probability value; the interior of the visual cone is divided into multiple depth planes by preset intervals; the two-dimensional image coordinates of the 3D center point of the detection target are obtained according to the image features, and the three-dimensional position information of the detection target is determined according to the two-dimensional image coordinates, the depth value and the internal and external parameter data of the monocular camera.
[0005] In the above implementation process, a monocular camera captures a visual image of the vehicle's surroundings, extracts image features from the image, and feeds these features into a depth prediction model. This model constructs a viewing cone, separating multiple depth planes within the cone. The model then predicts the probability of image features on each depth plane, thereby obtaining the depth value of the target being detected in the visual image. The two-dimensional image coordinates of the target's 3D center point are then obtained based on the image features. The three-dimensional position of the target is then determined by combining the depth values with the internal and external parameter data from the monocular camera. In this way, the 3D position of the target is detected using only a monocular camera, through depth classification and prediction of image features, effectively reducing the hardware cost of environmental perception.
[0006] Furthermore, in some examples, the image features are multi-scale image features; and extracting the image features from the visual image includes: extracting features of different scales from the visual image using a ResNet50 network; and fusing the features of different scales using an FPN network to obtain multi-scale image features.
[0007] In the above implementation process, the deep features of ResNet50 and the multi-scale information fusion capability of FPN are combined to extract image features, so that the image features have both high-level semantic information and low-level structural information.
[0008] Furthermore, in some examples, the depth prediction model predicts the probability values of the input image features on each depth plane of the viewing cone through a convolutional layer group, converts the probability values into a probability distribution through an activation function layer, and then calculates the depth value of the target based on the probability distribution and the viewing cone depth plane matrix.
[0009] In the above implementation process, an implementation method of the internal structure of the depth prediction model is provided, that is, the internal structure of the depth prediction model includes a convolutional layer group and an activation function layer, wherein the convolutional layer group is used to predict the probability values of the input image features located on each depth plane separated by the viewing cone, and the activation function layer is used to convert the probability values predicted by the convolutional layer group into a probability distribution on the viewing cone, and matrix calculation is performed based on the probability distribution and all depth planes of the viewing cone to obtain the depth value of the target.
[0010] Furthermore, in some examples, the total depth of the viewing cone is 100 m; the preset interval is 10 m; and the viewing cone depth plane matrix is determined according to the median value of each depth plane of the viewing cone.
[0011] In the above implementation, when constructing the viewing cone, the total depth of the cone is set to 100 meters, divided into depth planes at equal intervals of 10 meters. During matrix calculation, the cone depth plane matrix is determined based on the median value of each depth plane in the cone, and the target depth value is calculated based on this. This reduces computational complexity and improves the accuracy and efficiency of target depth calculation.
[0012] Furthermore, in some examples, the parameters of the depth prediction model are obtained based on depth loss optimization; the depth loss is obtained by compressing the depth value predicted by the depth prediction model within a preset range through a depth scaling factor, and then calculating it using an absolute error loss function after exponential operation.
[0013] In the above implementation process, during the training stage of the depth prediction model, the depth value is scaled and compressed within a preset range through a depth scaling factor. After exponential operation, the depth loss is calculated using the absolute error loss function. In this way, the trained depth prediction model can more accurately predict the depth of the target, accelerate convergence and promote feature learning.
[0014] Furthermore, in some examples, obtaining the two-dimensional image coordinates of the 3D center point of the detection target based on the image features includes: inputting the image features into a target detection model to obtain the two-dimensional image coordinates of the 3D center point of the detection target in the visual image; and during the training process, the target detection model learns the pixel offset between the two-dimensional image coordinates of the 3D center point of the input image and the pixel coordinates of the 2D center point.
[0015] In the above implementation, an object detection model is trained. During training, the model learns the pixel offset between the 2D image coordinates of the 3D center point of the input image and the pixel coordinates of the 2D center point. During inference, the model can use this learned pixel offset, combined with the pixel coordinates of the 2D center point, to calculate the pixel coordinates of the 3D center point on the 2D image. This helps improve the accuracy of 3D object detection.
[0016] Furthermore, in some examples, the internal and external parameter data include a camera internal parameter matrix, a rotation matrix from the camera coordinate system to the vehicle body coordinate system, and a translation matrix; determining the three-dimensional position information of the detection target based on the two-dimensional image coordinates, the depth value, and the internal and external parameter data of the monocular camera includes: multiplying the two-dimensional image coordinates, the inverse matrix of the camera internal parameter matrix, and the rotation matrix, and then adding the multiplication result and the translation matrix to obtain the three-dimensional coordinates of the center point of the detection target.
[0017] In the above implementation process, the inverse matrix of the camera intrinsic parameter matrix is multiplied by the two-dimensional image coordinates of the 3D center point to realize the transformation from the image coordinate system to the camera coordinate system, and then multiplied by the rotation matrix to realize the conversion from the camera coordinate system to the vehicle body coordinate system. Finally, it is added with the translation matrix to finally realize the coordinate transformation from the image coordinate system to the vehicle body coordinate system, thereby quickly and accurately obtaining the three-dimensional coordinates of the center point of the detection target.
[0018] In the second aspect, the present application provides a 3D target detection device, including: an acquisition module, used to acquire a visual image of the vehicle's surrounding environment through a monocular camera, and extract image features from the visual image; a processing module, used to process the image features based on a depth prediction model to obtain the depth value of the detection target in the visual image; during the training process, the depth prediction model learns to construct a visual cone, predict the probability value of the input image features on each depth plane of the visual cone, and obtain the depth value of the target according to the probability value; the interior of the visual cone is divided into multiple depth planes by preset intervals; a determination module, used to obtain the two-dimensional image coordinates of the 3D center point of the detection target according to the image features, and determine the three-dimensional position information of the detection target according to the two-dimensional image coordinates, the depth value and the internal and external parameter data of the monocular camera.
[0019] In a third aspect, the present application provides an electronic device comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the method described in any one of the first aspects when executing the computer program.
[0020] In a fourth aspect, the present application provides a computer-readable storage medium having instructions stored thereon. When the instructions are executed on a computer, the computer executes the method as described in any one of the first aspects.
[0021] In a fifth aspect, the present application provides a computer program product, which, when running on a computer, enables the computer to execute the method as described in any one of the first aspects.
[0022] Other features and advantages disclosed in the present application will be described in the following description, or some features and advantages can be inferred or determined without doubt from the description, or can be learned by implementing the above-mentioned technology disclosed in the present application.
[0023] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, preferred embodiments are given below and described in detail with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS
[0024] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following is a brief introduction to the drawings required for use in the embodiments of the present application. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other relevant drawings can be obtained based on these drawings without creative work.
[0025] Figure 1A flowchart of a 3D object detection method provided in an embodiment of the present application;
[0026] Figure 2 A schematic diagram of an implementation process of a 3D object detection solution with enhanced depth estimation provided in an embodiment of the present application;
[0027] Figure 3 A block diagram of a 3D object detection device provided in an embodiment of the present application;
[0028] Figure 4 This is a structural block diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0029] The technical solutions in the embodiments of the present application will be described below in conjunction with the drawings in the embodiments of the present application.
[0030] It should be noted that similar reference numerals and letters represent similar items in the following drawings. Therefore, once an item is defined in one drawing, it does not need to be further defined or explained in subsequent drawings. At the same time, in the description of this application, the terms "first", "second", etc. are only used to distinguish the description and should not be understood as indicating or implying relative importance.
[0031] As described in the background art, the related art method of detecting the three-dimensional position information of a target obstacle by a vehicle has the problem of high hardware cost. Based on this, the embodiments of the present application provide a 3D target detection solution to solve the above problem.
[0032] Next, the embodiments of the present application are introduced:
[0033] like Figure 1 As shown, Figure 1 This is a flow chart of a 3D object detection method provided in an embodiment of the present application. The method can be applied to a vehicle controller, such as an autonomous driving domain controller. The method includes:
[0034] Step 101: Acquire a visual image of the vehicle's surroundings using a monocular camera, and extract image features from the visual image;
[0035] The monocular camera mentioned in this step, also known as a monocular vision sensor, is a device that uses a single camera to capture image information of the surrounding environment. In this embodiment, the vehicle's surrounding environment is perceived using a monocular camera mounted on the vehicle. After the monocular camera captures a visual image of the vehicle's surroundings, feature extraction is performed on the image to obtain image features. Depth estimation is then enhanced based on depth classification to obtain the target's three-dimensional position information.
[0036] In some embodiments, the image features mentioned in this step may be multi-scale image features; and extracting image features from the visual image mentioned in this step may include: extracting features of different scales from the visual image using a ResNet50 network; and fusing the features of different scales using an FPN network to obtain multi-scale image features. The ResNet50 network is a deep residual learning network that addresses the gradient vanishing and performance saturation problems in deep networks by introducing residual connections. In the ResNet50 network, each residual block contains multiple convolutional layers and a skip connection, allowing the network to maintain good performance while becoming deeper. FPN (Feature Pyramid Networks) is a multi-scale feature fusion network used for object detection. It fuses information of different scales by constructing a feature pyramid. Its core concept is to combine high-resolution bottom-level features with top-level features with high semantic information to improve the accuracy of object detection. Therefore, by combining the deep features of ResNet50 with the multi-scale information fusion capabilities of FPN, the extracted image features contain both high-level semantic information and low-level structural information.
[0037] Step 102: Process the image features based on a depth prediction model to obtain a depth value of the detection target in the visual image; during the training process, the depth prediction model learns to construct a viewing cone, predict the probability values of the input image features on each depth plane of the viewing cone, and obtain the depth value of the target based on the probability values; the interior of the viewing cone is divided into multiple depth planes at preset intervals;
[0038] The detection target mentioned in this step can refer to target obstacles around the vehicle, such as other vehicles, pedestrians, roadblocks, and so on. In this embodiment, after obtaining image features, the image features are input into a depth prediction model. This depth prediction model constructs a viewing cone, which is divided into multiple depth planes at preset intervals within the viewing cone. The probability value of the image features on each depth plane is predicted to obtain the depth value of the target. In this way, replacing the regression calculation of abstract depth values with depth classification prediction of image features is more conducive to neural network learning.
[0039] In some embodiments, the depth prediction model mentioned in this step predicts the probability values of the input image features on each depth plane of the viewing cone through a convolutional layer group, converts the probability values into a probability distribution through an activation function layer, and then calculates the depth value of the target based on the probability distribution and the viewing cone depth plane matrix. That is, the internal structure of the depth prediction model may include a convolutional layer group and an activation function layer, wherein the convolutional layer group may be composed of three convolutional layers, which are used to predict the probability values of the input image features on each depth plane separated by the viewing cone, and the activation function layer may include a Softmax layer, which is used to convert the probability values predicted by the convolutional layer group into a probability distribution on the viewing cone, and perform matrix calculation based on the probability distribution and all depth planes of the viewing cone to obtain the depth value of the target.
[0040] In some embodiments, the total depth of the aforementioned viewing cone can be 100m; the preset interval can be 10m; and the viewing cone depth plane matrix can be determined based on the median value of each depth plane of the viewing cone. That is, in the process of constructing the viewing cone, the total depth of the viewing cone is taken as 100m, and each depth plane is divided into a depth plane at equal intervals of 10m. Then, the depth range corresponding to the first depth plane is 0 to 10m, the depth range corresponding to the second depth plane is 10m to 20m, and so on. The depth range corresponding to the tenth depth plane is 90m to 100m. In the process of calculating the matrix, the depth plane of the viewing cone is calculated using the median value of each depth plane, i.e., 5, 15, 25, ..., 95. In this way, the computational complexity is reduced and the accuracy and efficiency of the target depth value calculation are improved.
[0041] In addition, in some embodiments, the parameters of the depth prediction model mentioned in this step are obtained based on depth loss optimization; the depth loss is obtained by compressing the depth value predicted by the depth prediction model within a preset range through a depth scaling factor, and then calculating it using an absolute error loss function after exponential operation. That is to say, during the training process, the depth prediction model needs to perform a scale transformation on the depth value Depth predicted by the training sample, and a depth scaling factor R is used. scale Compress it within the preset range and obtain it after exponential operation By combining the true value of the target depth value in the training sample (which also needs to undergo scale transformation and exponential operation) and the L1 Loss function, the depth loss can be calculated, and the parameters of the model are optimized based on the depth loss. In this way, the trained depth prediction model can more accurately predict the depth of the target, and due to the characteristics of the exponential operation, the depth is reduced to a smaller value, and the gradient value changes more significantly, thereby accelerating convergence and promoting feature learning. Among them, the preset range can be 0 to 5m. Of course, in other embodiments, the preset range can also be set differently according to the needs of the specific scenario.
[0042] Step 103: Obtain the two-dimensional image coordinates of the 3D center point of the detection target according to the image features, and determine the three-dimensional position information of the detection target according to the two-dimensional image coordinates, the depth value, and the internal and external parameter data of the monocular camera.
[0043] In this embodiment, after obtaining the two-dimensional image coordinates and depth value of the 3D center point of the detection target, the coordinate transformation is performed in combination with the internal and external parameter data of the monocular camera to obtain the position information of the detection target in three-dimensional space. Based on this position information, the vehicle can more accurately understand the position of objects in the surrounding environment, and thus can more accurately plan the driving path to avoid collisions with obstacles.
[0044] In some embodiments, the step of obtaining the two-dimensional image coordinates of the 3D center point of the detection target based on the image features mentioned in this step may include: inputting the image features into a target detection model to obtain the two-dimensional image coordinates of the 3D center point of the detection target in the visual image; during the training process, the target detection model learns the pixel offset between the two-dimensional image coordinates of the 3D center point of the input image and the pixel coordinates of the 2D center point. That is, a target detection model is trained, and the training data of the target detection model is a sample image with annotated 3D information. During the training process, the target detection model learns the pixel offset between the two-dimensional image coordinates of the 3D center point of the sample image and the pixel coordinates of the 2D center point, where the 3D center point is the center point of the target in the sample image in three-dimensional space, and the 2D center point is the center point of the target in the two-dimensional image. During the inference stage, the target detection model can use the learned pixel offset, combined with the pixel coordinates of the 2D center point, to calculate the pixel coordinates of the 3D center point on the two-dimensional image, that is, the two-dimensional image coordinates.
[0045] In addition, in some embodiments, the internal and external parameter data mentioned in this step may include a camera internal parameter matrix, a rotation matrix from the camera coordinate system to the vehicle body coordinate system, and a translation matrix; then, the step of determining the three-dimensional position information of the detection target based on the two-dimensional image coordinates, the depth value, and the internal and external parameter data of the monocular camera may include: multiplying the two-dimensional image coordinates, the inverse matrix of the camera internal parameter matrix, and the rotation matrix, and then adding the multiplication result to the translation matrix to obtain the three-dimensional coordinates of the detection target. That is, during the coordinate transformation, the inverse matrix of the camera internal parameter matrix is multiplied by the two-dimensional image coordinates of the 3D center point to achieve the transformation from the image coordinate system to the camera coordinate system, and then multiplied by the rotation matrix to achieve the conversion from the camera coordinate system to the vehicle body coordinate system, and finally added to the translation matrix to finally achieve the coordinate transformation from the image coordinate system to the vehicle body coordinate system, thereby obtaining the three-dimensional coordinates of the center point of the detection target, which represents the position information of the detection target in three-dimensional space.
[0046] In an embodiment of the present application, a monocular camera is used to capture a visual image of the vehicle's surroundings, image features are extracted from the visual image, and the image features are input into a depth prediction model. This depth prediction model constructs a visual cone, separates multiple depth planes within the visual cone, and predicts the probability values of the image features on each depth plane to obtain the depth value of the detection target in the visual image. Subsequently, the two-dimensional image coordinates of the 3D center point of the detection target are obtained based on the image features. The three-dimensional position information of the detection target is determined by combining the depth value and the internal and external parameter data of the monocular camera. In this way, the three-dimensional position detection of the target is achieved by depth classification prediction of the image features using only a monocular camera, effectively reducing the hardware cost of environmental perception.
[0047] In order to explain the solution of this application in more detail, a specific embodiment is introduced below:
[0048] This embodiment provides a 3D object detection solution with enhanced depth estimation. The implementation process of the solution is as follows: Figure 2 Shown, including:
[0049] A visual image is obtained through a single camera sensor 21 and input into a target perception model. The target perception model includes three main modules: an image feature extraction module 22, a depth prediction module 23, and a target center detection module 24.
[0050] The image feature extraction module 22 uses Resnet50 to extract image features and uses FPN to perform multi-scale feature fusion, so that the image features have both high-level semantic information and low-level structural information;
[0051] The depth prediction module 23 constructs a viewing cone and predicts the probability value of the image feature on each depth plane by separating the depth distance inside the viewing cone. The target depth is obtained by using the matrix calculation of the probability and the depth plane. The depth classification prediction of the image feature replaces the regression calculation of the abstract depth value, which is more conducive to the learning of the neural network. The specific implementation method includes: in the process of constructing the viewing cone, the total depth of the viewing cone is taken as 100m, and a depth plane is divided into 10m intervals, that is, a total of 10 continuous depth values. Assume that the dimension of the image feature is [B, 12, 32, 72], and B represents Batch. Size, that is, the number of images in the batch, 12 represents the number of feature map channels, 32 represents the height of the feature map, and 72 represents the width of the feature map; three convolutional layers are used to predict the probability values [B, 10, 32, 72] of the image features located on these depth planes, and the Softmax layer is used to convert the probability values into probability distributions [B, 10, 32, 72] on the viewing cone. The probability distribution [B, 10, 32, 72] is converted to [B*32*72, 10] and all the depth planes of the viewing cone [10, 1] are matrix-calculated to obtain the depth value [1, B*32, 72] and then converted to [B, 1, 32, 72]. In the matrix calculation process, the depth plane of the viewing cone is calculated using the middle value row of each depth plane. The calculation method is as follows:
[0052]
[0053] The three matrices in this formula are probability distribution, frustum depth plane matrix and depth value from left to right, where n = B*32*72;
[0054] During the training of the depth prediction module 23, the depth value after matrix calculation needs to be scaled and compressed to the range of 0 to 5m by multiplying it by a depth scaling factor depth_scale. The depth value is modeled by the nonlinear relationship between image features and depth through exponential operation, starting from y=e x As can be seen from the curve graph, the exponential function has a smaller and smaller gradient as the depth value increases. Therefore, based on this characteristic, the depth value is scaled to increase the distinction between different depths in loss calculation.
[0055] The target center detection module 24 projects the annotated 3D information onto the image through coordinate transformation to obtain the pixel coordinates of the 3D center point on the image, and calculates the pixel offset between the above pixel coordinates and the center point of the 2D annotation box. This module learns whether there is a target center point on the image, learns the pixel offset of the 3D center point and the length, width and height of the 3D box at the target center point, and calculates the loss through L1 Loss. The pixel coordinates of the 2D center point and the pixel offset of the 3D center point can be calculated using the pixel coordinates of the 3D center point on the image. In the post-processing module 25, the coordinates and the depth value are transformed to obtain the three-dimensional position information of the target;
[0056] Afterwards, the output of the target perception model and the internal and external parameter data of the single camera sensor are input into the post-processing module 25, and the post-processing module 25 calculates the three-dimensional position information of the target. During implementation, the post-processing module 25 determines whether the target center detection module 24 has a value greater than the threshold. If so, the coordinate position is recorded and the length and width, three-dimensional center offset, and three-dimensional length, width, height, and depth values of the corresponding positions of other channels in the detection head are extracted. The internal and external parameter data include the camera internal parameter matrix [B, 3, 3], the rotation matrix from the camera coordinate system to the vehicle body coordinate system [B, 3, 3], and the translation matrix from the camera coordinate system to the vehicle body coordinate system [B, 3,]. First, the inverse matrix of the camera internal parameter matrix is multiplied with the two-dimensional image coordinates of the center point to realize the transformation from the image coordinate system to the camera coordinate system, and then multiplied with the rotation matrix to realize the transformation from the camera coordinate system to the vehicle body coordinate system. Finally, it is added with the translation matrix to finally realize the coordinate transformation from the image coordinate system to the vehicle body coordinate system, and the three-dimensional coordinates of the target center point [B, 32, 72, 3] are obtained. The specific calculation formula is as follows:
[0057]
[0058] In this formula, from left to right are the rotation matrix, the inverse matrix of the camera intrinsic parameter matrix, the depth value, the two-dimensional image coordinates of the three-dimensional center point, the translation matrix, and the three-dimensional coordinates of the target center point.
[0059] This embodiment has at least the following advantages: First, it uses a single-camera visual image to obtain image features, and enhances depth estimation based on the idea of depth classification, thereby obtaining target three-dimensional position detection and reducing the hardware cost of environmental perception; Second, by constructing a cone, the directly predicted depth value is converted into a probability distribution of the predicted depth on the cone depth plane, that is, the regression problem is transformed into a classification problem, which is more friendly to neural network learning; Third, the nonlinear relationship between the depth value and the pixel features in the image is modeled through exponential operation, which helps to capture subtle changes in depth information and can more accurately predict the depth of the target. Due to the characteristics of the exponential operation, the depth is reduced to a smaller value, the gradient value changes more obviously, and can accelerate convergence and promote feature learning; Fourth, the target perception model can avoid the space occupation problem of the Transformer structure, is easier to implement, and has stronger practicality.
[0060] Corresponding to the embodiments of the aforementioned method, the present application also provides embodiments of a 3D object detection device and a terminal using the same:
[0061] like Figure 3 As shown, Figure 3 is a block diagram of a 3D object detection device provided in an embodiment of the present application, the device comprising:
[0062] An acquisition module 31 is configured to acquire a visual image of the vehicle's surroundings through a monocular camera and extract image features from the visual image;
[0063] A processing module 32 is configured to process the image features based on a depth prediction model to obtain a depth value of a detection target in the visual image; during training, the depth prediction model learns to construct a viewing cone, predicts probability values of input image features on various depth planes of the viewing cone, and obtains the depth value of the target based on the probability values; the interior of the viewing cone is divided into a plurality of depth planes at preset intervals;
[0064] The determination module 33 is used to obtain the two-dimensional image coordinates of the 3D center point of the detection target according to the image features, and determine the three-dimensional position information of the detection target according to the two-dimensional image coordinates, the depth value and the internal and external parameter data of the monocular camera.
[0065] The implementation process of the functions and effects of each module in the above-mentioned device is specifically described in the implementation process of the corresponding steps in the above-mentioned method, and will not be repeated here.
[0066] This application also provides an electronic device, see Figure 4 , Figure 4This is a block diagram of the structure of an electronic device provided in an embodiment of the present application. The electronic device may include a processor 410, a communication interface 420, a memory 430, and at least one communication bus 440. The communication bus 440 is used to enable direct communication between these components. The communication interface 420 of the electronic device in this embodiment of the present application is used to communicate signaling or data with other node devices. The processor 410 may be an integrated circuit chip with signal processing capabilities.
[0067] The processor 410 can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic device, a discrete gate or transistor logic device, or a discrete hardware component. It can implement or execute the various methods, steps, and logic block diagrams disclosed in the embodiments of this application. A general-purpose processor can be a microprocessor, or the processor 410 can also be any conventional processor.
[0068] The memory 430 may be, but is not limited to, a random access memory (RAM), a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), etc. The memory 430 stores computer-readable instructions. When the computer-readable instructions are executed by the processor 410, the electronic device can perform the above-mentioned operations. Figure 1 The various steps involved in the method embodiment.
[0069] Optionally, the electronic device may further include a storage controller and an input / output unit.
[0070] The memory 430, storage controller, processor 410, peripheral interface, and input / output units are electrically connected to each other directly or indirectly to enable data transmission or interaction. For example, these components can be electrically connected to each other via one or more communication buses 440. The processor 410 is used to execute executable modules stored in the memory 430, such as software function modules or computer programs included in the electronic device.
[0071] The input and output unit is used to provide users with the ability to create tasks and to create optional time periods or preset execution times for the tasks to enable interaction between the user and the server. The input and output unit can be, but is not limited to, a mouse and a keyboard.
[0072] I understand. Figure 4 The structure shown is only for illustration, and the electronic device may also include Figure 4 More or fewer components than shown, or with Figure 4 Different configurations shown. Figure 4 Each component shown in the figure can be implemented by hardware, software or a combination thereof.
[0073] An embodiment of the present application further provides a storage medium having instructions stored thereon. When the instructions are run on a computer, the computer program is executed by a processor to implement the method described in the method embodiment. To avoid repetition, details are not given here.
[0074] The present application also provides a computer program product, which, when running on a computer, enables the computer to execute the method described in the method embodiment.
[0075] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can also be implemented in other ways. The device embodiments described above are merely illustrative. For example, the flowcharts and block diagrams in the accompanying drawings show the possible architectures, functions and operations of the devices, methods and computer program products according to multiple embodiments of the present application. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a portion of code, and the module, program segment or a portion of code contains one or more executable instructions for implementing the specified logical functions. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in an order different from that marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of boxes in the block diagram and / or flowchart, can be implemented using a dedicated hardware-based system that performs the specified function or action, or can be implemented using a combination of dedicated hardware and computer instructions.
[0076] In addition, the functional modules in each embodiment of the present application can be integrated together to form an independent part, or each module can exist independently, or two or more modules can be integrated to form an independent part.
[0077] If the functions are implemented in the form of software function modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.
[0078] The foregoing is merely an embodiment of the present application and is not intended to limit the scope of protection of the present application. Various modifications and variations are possible for those skilled in the art. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application shall be included within the scope of protection of the present application. It should be noted that similar reference numerals and letters represent similar items in the following figures. Therefore, once an item is defined in one figure, it does not need to be further defined or explained in subsequent figures.
[0079] The above description is merely a specific embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in the present application should be included in the scope of protection of the present application. Therefore, the scope of protection of the present application should be based on the scope of protection of the claims.
[0080] It should be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply the existence of any such actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, method, article, or device comprising the element.
Claims
1. A 3D object detection method, characterized in that: include: Acquire a visual image of the vehicle's surroundings through a monocular camera, and extract image features from the visual image; The image features are processed based on a depth prediction model to obtain a depth value of a detection target in the visual image; during the training process, the depth prediction model learns to construct a visual cone, predicts probability values of input image features on each depth plane of the visual cone, and obtains the depth value of the target according to the probability values; the interior of the visual cone is divided into a plurality of depth planes by a preset interval; The two-dimensional image coordinates of the 3D center point of the detection target are obtained according to the image features, and the three-dimensional position information of the detection target is determined according to the two-dimensional image coordinates, the depth value and the internal and external parameter data of the monocular camera.
2. The method according to claim 1, characterized in that The image features are multi-scale image features; The step of extracting image features from the visual image comprises: Extracting features of different scales from the visual image using a ResNet50 network; The features of different scales are fused using the FPN network to obtain multi-scale image features.
3. The method according to claim 1, characterized in that The depth prediction model predicts the probability values of the input image features on each depth plane of the viewing cone through a convolutional layer group, converts the probability values into probability distributions through an activation function layer, and then calculates the depth value of the target based on the probability distribution and the viewing cone depth plane matrix.
4. The method according to claim 3, characterized in that The total depth of the viewing cone is 100m; the preset interval is 10m; and the viewing cone depth plane matrix is determined according to the middle values of each depth plane of the viewing cone.
5. The method according to claim 3, characterized in that: The parameters of the depth prediction model are obtained based on depth loss optimization; the depth loss is obtained by compressing the depth value predicted by the depth prediction model within a preset range through a depth scaling factor, and then calculating it using an absolute error loss function after exponential operation.
6. The method according to claim 1, characterized in that The acquiring the two-dimensional image coordinates of the 3D center point of the detection target according to the image features includes: The image features are input into a target detection model to obtain the two-dimensional image coordinates of the 3D center point of the detection target in the visual image; during the training process, the target detection model learns the pixel offset between the two-dimensional image coordinates of the 3D center point of the input image and the pixel coordinates of the 2D center point.
7. The method according to claim 1, characterized in that The internal and external parameter data include a camera internal parameter matrix, a rotation matrix from a camera coordinate system to a vehicle body coordinate system, and a translation matrix; determining the three-dimensional position information of the detection target according to the two-dimensional image coordinates, the depth value, and the internal and external parameter data of the monocular camera includes: The two-dimensional image coordinates, the inverse matrix of the camera intrinsic parameter matrix and the rotation matrix are multiplied, and then the multiplication result and the translation matrix are added to obtain the three-dimensional coordinates of the center point of the detection target.
8. A 3D object detection device, characterized in that: include: An acquisition module, used to acquire a visual image of the vehicle's surroundings through a monocular camera, and extract image features from the visual image; A processing module is used to process the image features based on a depth prediction model to obtain a depth value of a detection target in the visual image; during the training process, the depth prediction model learns to construct a visual cone, predicts the probability value of the input image features on each depth plane of the visual cone, and obtains the depth value of the target according to the probability value; the interior of the visual cone is divided into a plurality of depth planes by a preset interval; A determination module is used to obtain the two-dimensional image coordinates of the 3D center point of the detection target according to the image features, and determine the three-dimensional position information of the detection target according to the two-dimensional image coordinates, the depth value and the internal and external parameter data of the monocular camera.
9. A computer-readable storage medium, characterized in that: A computer program is stored thereon, and when the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.
10. An electronic device, characterized in that: The method comprises a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the method according to any one of claims 1 to 7 is implemented.