Three-dimensional target detection method, electronic device, and storage medium

Abstract

The application provides a three-dimensional object detection method, an electronic device and a storage medium. The method comprises the following steps: acquiring a training image; constructing and training a semantic segmentation model to obtain a trained semantic segmentation model; acquiring a detection image and a depth image; inputting the detection image into the trained semantic segmentation model to obtain an object category and an object position; determining an object model from a three-dimensional object model library according to the object category; obtaining point cloud data of the object and a distance from a depth camera to the object model according to the depth image; determining a rotation angle of the object model according to the point cloud data and the object model; and determining a position of the object model in a three-dimensional space according to the distance from the depth camera to the object model, the rotation angle and the position of the object, so as to determine a position of the object in the three-dimensional space. The application can quickly determine the position of the object in the three-dimensional space.

Description

Technical Field

[0001] This application relates to the field of computer vision, and more particularly to a three-dimensional target detection method, electronic device, and storage medium. Background Technology

[0002] In the field of autonomous driving, autonomous driving systems utilize various types of sensors to detect objects in front of or near the vehicle and make corresponding decisions. For example, autonomous driving systems need to quickly and accurately detect the type and 3D position of objects, and then make corresponding decisions to ensure driving safety. However, current methods use 3D object detection algorithms to detect the type and position of objects, and then use regression calculations to obtain the 3D position of the objects; however, regression calculations take a long time to predict. Furthermore, when performing 3D object detection, existing autonomous driving systems use LiDAR or radar to obtain depth information to detect the distance between the vehicle and objects in front, but currently, using LiDAR or radar is expensive and has a relatively small field of view. Summary of the Invention

[0003] In view of the above, it is necessary to provide a three-dimensional target detection method, electronic device and storage medium to solve the problems of slow object detection efficiency and high cost.

[0004] This application provides a three-dimensional object detection method, which includes: acquiring a training image; constructing a semantic segmentation model based on a fully convolutional network; inputting the training image into the semantic segmentation model, performing multiple convolutions and pooling operations using convolutional and pooling layers in the semantic segmentation model to obtain multiple feature maps of different sizes; upsampling the multiple feature maps of different sizes to obtain a first image of the same size as the training image; performing pixel classification on the first image and optimizing the classification loss; outputting the object category and position of the object in the training image to obtain the trained semantic segmentation model; and acquiring a detection image and the detection image. Like a corresponding depth image, wherein the depth image is acquired by a depth camera; the detected image is input into the trained semantic segmentation model to obtain the object category and object position of the object in the detected image; according to the object category, the object model corresponding to the object is determined from the 3D object model library; the point cloud data of the object and the distance from the depth camera to the object model are obtained from the depth image; the rotation angle of the object model is determined according to the point cloud data and the object model; the position of the object model in 3D space is determined according to the distance from the depth camera to the object model, the rotation angle, and the position of the object.

[0005] In one optional implementation, the step of upsampling the plurality of feature maps of different sizes to obtain a first image of the same size as the training image includes: upsampling the plurality of feature maps of different sizes, performing deconvolution operation to obtain the operation result; and performing a summation operation on the operation result to obtain a first image of the same size as the training image.

[0006] In one optional implementation, determining the object model corresponding to the object from the 3D object model library based on the object category includes: searching the 3D object model library based on the object category to determine the object model of the object, wherein the 3D object model library includes object categories and object models corresponding to the object categories.

[0007] In one optional implementation, obtaining the point cloud data of the object and the distance from the depth camera to the object model based on the depth image includes: obtaining the depth value and coordinates of the object based on the depth image; determining the distance from the depth camera to the object model based on the depth value; and obtaining the point cloud data based on the coordinates and the transformation formula of the intrinsic and extrinsic parameter matrix of the depth camera.

[0008] In one optional implementation, determining the rotation angle of the object model based on the point cloud data and the object model includes: obtaining first point cloud data of the object contour based on the point cloud data; converting the object model into second point cloud data; performing point cloud matching between the first point cloud data and the second point cloud data; fitting the points of the object contour in the first point cloud data to a first plane and calculating the curvature of the first plane; fitting the points of the second point cloud data to a second plane and calculating the curvature of the second plane; calculating the difference between the curvature of the first plane and the curvature of the second plane to obtain a curvature deviation value; and determining the rotation angle of the object model based on the curvature deviation value.

[0009] In one optional implementation, converting the object model into second point cloud data includes:

[0010] The object model of the object is processed using multiple functions in the point cloud library to generate the second point cloud data.

[0011] In one optional implementation, determining the position of the object model in three-dimensional space based on the distance from the depth camera to the object model, the rotation angle, and the position of the object includes: determining the orientation of the object model in three-dimensional space based on the rotation angle; and determining the position of the object model in three-dimensional space based on the orientation of the object model in three-dimensional space, the distance from the depth camera to the object model, and the position of the object.

[0012] In an optional implementation, the method further includes: taking the position of the object model of the object in three-dimensional space as the position of the object in three-dimensional space, and outputting the object category and the position of the object in three-dimensional space.

[0013] This application also provides an electronic device, which includes a processor and a memory. The processor is used to execute a computer program stored in the memory to implement the three-dimensional target detection method.

[0014] This application also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the three-dimensional target detection method.

[0015] The technical solution of this application is low in cost and does not require complex calculations, and can quickly obtain the three-dimensional position of an object. Attached Figure Description

[0016] Figure 1 This is a flowchart of a three-dimensional target detection method provided in an embodiment of this application.

[0017] Figure 2 The image provided in this application embodiment is segmented by the semantic segmentation model after training.

[0018] Figure 3 A schematic diagram of the three-dimensional object model library provided in the embodiments of this application.

[0019] Figure 4 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0020] To better understand the above-mentioned objectives, features, and advantages of this application, the application will be described in detail below with reference to the accompanying drawings and specific embodiments. It should be noted that the specific embodiments described herein are merely illustrative and are not intended to limit the scope of this application.

[0021] Numerous specific details are set forth in the following description to provide a thorough understanding of this application. The described embodiments are merely some, not all, of the embodiments described herein. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are within the scope of protection of this application.

[0022] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the application.

[0023] See Figure 1 The diagram shown is a flowchart of a three-dimensional target detection method provided in an embodiment of this application. The method is applied to electronic devices (e.g., Figure 4 In the electronic device 4) shown, the electronic device can be any electronic product that can interact with the user, such as a personal computer, tablet computer, smartphone, personal digital assistant (PDA), game console, interactive network television (Internet Protocol Television, IPTV), smart wearable device, etc.

[0024] The electronic device is a device capable of automatically performing numerical calculations and / or information processing according to pre-set or stored instructions. Its hardware includes, but is not limited to: microprocessors, application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), digital signal processors (DSPs), embedded devices, etc.

[0025] The electronic device may also include network devices and / or user devices. The network devices include, but are not limited to, a single network server, a server group consisting of multiple network servers, or a cloud based on cloud computing consisting of a large number of hosts or network servers.

[0026] The networks in which the electronic devices are located include, but are not limited to, the Internet, wide area networks, metropolitan area networks, local area networks, and virtual private networks (VPNs).

[0027] The method specifically includes the following:

[0028] 101, Obtain the training image.

[0029] In at least one embodiment of this application, the training images include, but are not limited to, scene images of various types of roads, such as those in cities and rural areas, at different time periods. In this embodiment, the training images include images from the Pascal VOC dataset and the Cityscapes dataset.

[0030] In at least one embodiment of this application, the method further includes: performing data augmentation operations on the training images to amplify different training images. The data augmentation operations include, but are not limited to, flipping, rotating, scaling, and cropping images. It should be noted that the data augmentation operations can obtain more vehicle-front images in different scenarios as training samples, enabling the semantic segmentation model to be trained and optimized, thus making the semantic segmentation model more robust.

[0031] 102. Construct a semantic segmentation model and train the semantic segmentation model using training images to obtain a trained semantic segmentation model.

[0032] In at least one embodiment of this application, constructing a semantic segmentation model includes:

[0033] A semantic segmentation model is built based on a fully convolutional network.

[0034] Based on the fully convolutional network, multiple targets in an image input to the network can be segmented and identified. For example, an image containing multiple targets, such as a car, a person, or a dog, can be input into the semantic segmentation model. After segmenting and identifying the multiple targets in the image, the semantic segmentation model outputs the object category and location of the objects in the training image.

[0035] In at least one embodiment of this application, a method for training a semantic segmentation model to obtain a trained semantic segmentation model includes:

[0036] The training image is input into the semantic segmentation model, and multiple convolutions and poolings are performed using the convolutional and pooling layers in the semantic segmentation model to obtain multiple feature maps of different sizes.

[0037] The feature maps of different sizes are upsampled and then deconvolutioned to obtain the result.

[0038] The results of the calculations are summed to obtain a first image of the same size as the training image.

[0039] The first image is classified pixel by pixel, and the classification loss is calculated and optimized. The object category and location of the objects in the training image are output to obtain the trained semantic segmentation model. In this embodiment, the softmax cross-entropy function is used as the loss function to train the semantic segmentation model. For example, as... Figure 2 As shown, Figure 2 The image provided in this application embodiment is segmented by the semantic segmentation model after training.

[0040] 103. Obtain the detection image and the depth image corresponding to the detection image.

[0041] In at least one embodiment of this application, a camera installed inside or outside the vehicle is used to capture images of the front of the vehicle, which are then used as detection images.

[0042] In at least one embodiment of this application, obtaining a depth image corresponding to the detected image includes:

[0043] A depth camera is used to acquire a depth image, specifically an image of the area in front of the vehicle captured by a depth camera mounted on the vehicle. It should be noted that when an image of the area in front of the vehicle is captured by a camera mounted inside or outside the vehicle as a detection image, the depth camera simultaneously captures the same image as a depth image, which corresponds to the detection image. For example, different types of cameras can be used to capture images of the same object in front of the vehicle to obtain both a detection image and a depth image.

[0044] It should be noted that the depth camera is an existing depth camera, and the method further includes: acquiring depth information of the depth image from the depth camera.

[0045] 104. Based on the object category, determine the object model corresponding to the object from the three-dimensional object model library.

[0046] In at least one embodiment of this application, determining the object model corresponding to the object from the three-dimensional object model library according to the object category includes:

[0047] Establish a 3D object model library that corresponds to object categories and object models, wherein the 3D object model library includes object categories and object models corresponding to the object categories;

[0048] The object model is determined by searching the 3D object model library based on the object category. (Reference) Figure 3 The diagram shown is a schematic of the three-dimensional object model library provided in an embodiment of this application. When the object category is a car, the object model of the car is searched based on the three-dimensional object model library; when the object category is a truck, the object model of the truck is searched based on the three-dimensional object model library; when the object category is an electric vehicle, the object model of the electric vehicle is searched based on the three-dimensional object model library; when the object category is a bus, the object model of the bus is searched based on the three-dimensional object model library.

[0049] In this embodiment of the application, the object model of the object includes a three-dimensional model of the object.

[0050] 105. Obtain the point cloud data and the distance from the depth camera to the object model based on the depth image.

[0051] In at least one embodiment of this application, determining the distance from the depth camera to the object model includes: obtaining a depth value of the object based on the depth image; and determining the distance from the depth camera to the object model based on the depth value. In this embodiment, the depth value is obtained from the depth camera. Specifically, when a depth image is obtained using the depth camera, the depth camera displays a depth value, which is the distance from the depth camera to the object. In this embodiment, the distance from the depth camera to the object is used as the distance from the depth camera to the object model of the object.

[0052] In at least one embodiment of this application, the method for obtaining the point cloud data includes:

[0053] The coordinate set of the object is obtained based on the depth image;

[0054] The point cloud data of the object is obtained according to the coordinate set and the intrinsic and extrinsic parameter matrix transformation formula of the depth camera. In this embodiment, the coordinate set of the object is the pixel coordinate set of the object; the point cloud data of the object is the world coordinates corresponding to the coordinates in the coordinate set of the object. The point cloud data of the object is used to represent the contour of the object. Transforming the coordinates in the coordinate set of the object into the corresponding world coordinates requires the use of the intrinsic and extrinsic parameter matrix transformation formula, which is:

[0055]

[0056] Where (x, y, z) are world coordinates, used to represent a point cloud as a single pixel coordinate, f is the focal length, D is the depth value, and (x1, y1) are the pixel coordinates of any pixel in the coordinate set of the object within the two-dimensional bounding box. Using the above formula, all coordinates in the coordinate set are converted one by one into world coordinates to obtain the point cloud data of the object.

[0057] 106. Based on the point cloud data and the object model, determine the rotation angle of the object model.

[0058] In at least one embodiment of this application, determining the rotation angle of the object model based on the point cloud data and the object model includes:

[0059] Based on the point cloud data, the first point cloud data of the object contour is obtained;

[0060] The object model is converted into second point cloud data;

[0061] Point cloud matching is performed between the first point cloud data and the second point cloud data. Points of the object contour in the first point cloud data are fitted to a first plane and the curvature of the first plane is calculated. Points of the second point cloud data are fitted to a second plane and the curvature of the second plane is calculated.

[0062] The curvature deviation value is obtained by calculating the difference between the curvature of the first plane and the curvature of the second plane, and the rotation angle of the object model is determined based on the curvature deviation value.

[0063] In at least one embodiment of this application, converting the object model of the object into second point cloud data includes:

[0064] The object model of the object is processed using multiple functions in the Point Cloud Library (PCL) to generate point cloud data of the object model as the second point cloud data.

[0065] 107. Determine the position of the object in three-dimensional space.

[0066] In at least one embodiment of this application, the orientation of the object model in the three-dimensional space is determined based on the rotation angle, and the position of the object model in the three-dimensional space is determined based on the orientation of the object model in the three-dimensional space, the distance from the depth camera to the object model, and the position of the object. Specifically, the position of the object model in the three-dimensional space is used as the position of the object in the three-dimensional space, and the object category and the position of the object in the three-dimensional space are output. For example, the object category and the position of the object in the three-dimensional space are displayed on a display screen.

[0067] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. For those skilled in the art, improvements can be made without departing from the inventive concept of this application, but these all fall within the scope of protection of this application.

[0068] like Figure 4 As shown, Figure 4 This is a schematic diagram of an electronic device structure provided in an embodiment of this application. The electronic device 4 includes a memory 401, at least one processor 402, a computer program 403 stored in the memory 401 and executable on the at least one processor 402, and at least one communication bus 404.

[0069] Those skilled in the art will understand that Figure 4The schematic diagram shown is merely an example of the electronic device 4 and does not constitute a limitation on the electronic device 4. It may include more or fewer components than shown, or combine certain components, or different components. For example, the electronic device 4 may also include input / output devices, network access devices, etc.

[0070] The at least one processor 402 may be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The at least one processor 402 may be a microprocessor or any conventional processor, etc. The at least one processor 402 is the control center of the electronic device 4, connecting various parts of the entire electronic device 4 via various interfaces and lines.

[0071] The memory 401 can be used to store the computer program 403. The at least one processor 402 implements various functions of the electronic device 4 by running or executing the computer program 403 stored in the memory 401 and calling data stored in the memory 401. The memory 401 may mainly include a program storage area and a data storage area. The program storage area may store the operating system, application programs required for at least one function (such as sound playback function, image playback function, etc.), etc.; the data storage area may store data created according to the use of the electronic device 4 (such as audio data), etc. In addition, the memory 401 may include non-volatile memory, such as hard disk, memory, plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, at least one disk storage device, flash memory device, or other non-volatile solid-state storage device.

[0072] If the modules / units integrated in the electronic device 4 are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include: any entity or device capable of carrying the computer program code, a recording medium, a USB flash drive, a portable hard drive, a magnetic disk, an optical disk, a computer memory, and a read-only memory (ROM).

[0073] It will be apparent to those skilled in the art that this application is not limited to the details of the exemplary embodiments described above, and that this application can be implemented in other specific forms without departing from the spirit or essential characteristics of this application. Therefore, the embodiments should be considered illustrative and non-limiting in all respects, and the scope of this application is defined by the appended claims rather than the foregoing description. Thus, all variations falling within the meaning and scope of equivalents of the claims are intended to be embraced within this application. No appended diagram markings in the claims should be construed as limiting the scope of the claims.

[0074] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application and are not intended to limit it. Although this application has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of this application without departing from the spirit and scope of the technical solutions of this application.

Claims

1. A three-dimensional object detection method, characterized by, The three-dimensional object detection method comprises: acquiring a training image; constructing a semantic segmentation model based on a full convolutional network; inputting the training image into the semantic segmentation model, performing multiple times of convolution and pooling by using convolution layers and pooling layers in the semantic segmentation model, and obtaining multiple feature maps with different sizes; performing upsampling on the multiple feature maps with different sizes to obtain a first image with the same size as the training image, performing pixel classification on the first image, optimizing a classification loss, outputting an object category of an object in the training image and a position of the object, and obtaining a trained semantic segmentation model; acquiring a detection image and a depth image corresponding to the detection image, wherein the depth image is acquired by using a depth camera; inputting the detection image into the trained semantic segmentation model to obtain an object category of an object in the detection image and an object position; determining an object model corresponding to the object from a three-dimensional object model library according to the object category; obtaining point cloud data of the object and a distance from the depth camera to the object model according to the depth image; determining a rotation angle of the object model according to the point cloud data and the object model, comprising: obtaining first point cloud data of an object contour according to the point cloud data, converting the object model into second point cloud data, performing point cloud matching on the first point cloud data and the second point cloud data, fitting points of the first point cloud data into a first plane and calculating a curvature of the first plane, fitting points of the second point cloud data into a second plane and calculating a curvature of the second plane, calculating a difference between the curvatures of the first plane and the second plane to obtain a curvature deviation value, and determining the rotation angle of the object model according to the curvature deviation value; determining a position of the object model in a three-dimensional space according to the distance from the depth camera to the object model, the rotation angle and the object position.

2. The three-dimensional object detection method of claim 1, wherein, The upsampling on the multiple feature maps with different sizes to obtain a first image with the same size as the training image comprises: performing deconvolution operation on the multiple feature maps after upsampling to obtain an operation result; performing addition operation on the operation result to obtain a first image with the same size as the training image.

3. The three-dimensional object detection method of claim 1, wherein, The determining an object model corresponding to the object from the three-dimensional object model library according to the object category comprises: searching the three-dimensional object model library according to the object category to determine the object model, wherein the three-dimensional object model library comprises object categories and object models corresponding to the object categories.

4. The three-dimensional object detection method of claim 1, wherein, The obtaining point cloud data of the object and a distance from the depth camera to the object model according to the depth image comprises: obtaining a depth value and coordinates of the object according to the depth image, determining the distance from the depth camera to the object model according to the depth value, and obtaining the point cloud data according to the coordinates and an internal and external parameter matrix transformation formula of the depth camera. The converting the object model into second point cloud data comprises:

5. The three-dimensional object detection method of claim 1, wherein, ​ The object model of the object is processed by using a plurality of functions in a point cloud library and the second point cloud data is generated.

6. The three-dimensional object detection method of any one of claims 1 to 5, wherein, The determining the position of the object model in the three-dimensional space according to the distance from the depth camera to the object model, the rotation angle and the position of the object comprises: determining the direction of the object model in the three-dimensional space according to the rotation angle; determining the position of the object model in the three-dimensional space according to the direction of the object model in the three-dimensional space, the distance from the depth camera to the object model and the position of the object.

7. The three-dimensional object detection method of claim 6, wherein, The method further comprises: outputting the object category and the position of the object in the three-dimensional space by taking the position of the object model of the object in the three-dimensional space as the position of the object in the three-dimensional space.

8. An electronic device, comprising: The electronic device comprises a processor and a memory, and the processor is configured to execute a computer program stored in the memory to implement the three-dimensional target detection method according to any one of claims 1 to 7.

9. A computer-readable storage medium, characterized in that, The computer readable storage medium stores at least one instruction, and the at least one instruction is executed by the processor to implement the three-dimensional target detection method according to any one of claims 1 to 7.

Images