Motor vehicle three-dimensional target detection method, device and computer readable storage medium

Abstract

Embodiments of the present application provide a motor vehicle three-dimensional target detection method, device and computer readable storage medium, the method comprising: acquiring a first original image and a second original image; using a pre-trained depth information extraction model to extract depth information from the first original image and the second original image, and screening out interference three-dimensional targets in the three-dimensional targets; performing convolution calculation on the depth information of each three-dimensional target, projecting the convolution calculation result into a pre-stored attention mechanism model to obtain a to-be-calculated feature, obtaining a semantic feature of the three-dimensional target, and taking the semantic feature as a feature classification branch; performing convolution processing on the depth information to obtain a to-be-enhanced feature, processing the to-be-enhanced feature to obtain a to-be-extracted feature, and obtaining a feature regression branch; and obtaining a class confidence and three-dimensional data information of the three-dimensional target. The embodiments can effectively improve the three-dimensional target detection efficiency, reduce memory consumption and cost.

Description

TECHNICAL FIELD

[0001] The embodiment of the application relates to the technical field of three-dimensional target detection in a vehicle-mounted environment, in particular to a motor vehicle three-dimensional target detection method, device and computer readable storage medium. BACKGROUND

[0002] At present, a three-dimensional target detection device for detecting three-dimensional information of objects around a vehicle body is usually mounted on a motor vehicle to assist the motor vehicle to realize accurate automatic driving.

[0003] A motor vehicle three-dimensional target detection method usually adopts a laser radar installed on a motor vehicle to scan and obtain three-dimensional data of three-dimensional objects in the environment around the vehicle body, and then processes the three-dimensional data of the three-dimensional objects through a convolutional neural network to determine the category and three-dimensional distance of the three-dimensional objects.

[0004] However, the inventors have found in the implementation that the calculation process of the motor vehicle three-dimensional target detection method is relatively complex, resulting in relatively low three-dimensional target detection efficiency, and a large amount of memory is consumed when the convolutional neural network is used to process the three-dimensional data of the three-dimensional objects, and the cost of the laser radar is also relatively high. SUMMARY

[0005] The technical problem to be solved by the embodiment of the application is to provide a motor vehicle three-dimensional target detection method that can effectively improve the three-dimensional target detection efficiency and reduce memory consumption and cost.

[0006] The further technical problem to be solved by the embodiment of the application is to provide a motor vehicle three-dimensional target detection device that can effectively improve the three-dimensional target detection efficiency and reduce memory consumption and cost.

[0007] The further technical problem to be solved by the embodiment of the application is to provide a computer readable storage medium for storing a computer program that can effectively improve the three-dimensional target detection efficiency and reduce memory consumption and cost.

[0008] To solve the above technical problems, the embodiment of the application first provides the following technical solution: a motor vehicle three-dimensional target detection method, comprising the following steps:

[0009] acquiring a first original image and a second original image from video images respectively collected by vehicle-mounted cameras that simultaneously shoot the same area around the vehicle body from different perspectives;

[0010] extracting depth information corresponding to the same three-dimensional target from the first original image and the second original image by using a pre-trained depth information extraction model, and screening and removing interference targets in the three-dimensional target;

[0011] The depth information of each three-dimensional target is convoluted by two predetermined size convolution kernels to obtain a first to-be-projected feature and a second to-be-projected feature, respectively, the first to-be-projected feature and the second to-be-projected feature are projected into a pre-stored attention mechanism model as a query value and a key value of an attention mechanism respectively to obtain a to-be-calculated feature, the to-be-calculated feature is subjected to a scaled dot product calculation to obtain a semantic feature of the three-dimensional target, and the semantic feature is taken as a feature classification branch;

[0012] The depth information of each three-dimensional target is convoluted by a predetermined backbone convolution network to obtain a to-be-enhanced feature, the to-be-enhanced feature is sequentially input into two predetermined node number fully connected layers, and a pre-stored SEblock model is used to calculate a to-be-extracted feature, the to-be-enhanced feature and the to-be-extracted feature are input into a sigmoid excitation function to obtain a geometric feature of each three-dimensional target, and the geometric feature is taken as a feature regression branch.

[0013] The feature classification branch and the feature regression branch are input into a pre-stored Transformer network model for calculation to obtain a class confidence and three-dimensional data information of each three-dimensional target.

[0014] Further, the depth information extraction model uses a large number of training images of calibrated three-dimensional targets as input into a DenseNet deep learning network, and then obtains the training images through data normalization and data augmentation.

[0015] Further, the predetermined backbone convolution network is a MobileNet network, a RetinaNet network or an EfficientNet network.

[0016] Further, after obtaining the semantic feature of the three-dimensional target, a convolution kernel with a size of 1*1 is used to convolute the semantic feature, and the convoluted semantic feature is taken as the feature classification branch.

[0017] Further, the two predetermined size convolution kernels are both 3*3 convolution kernels.

[0018] Further, the two predetermined node number fully connected layers are both 1024 node fully connected layers.

[0019] On the other hand, in order to solve the above-mentioned further technical problems, the embodiments of the present application further provide the following technical solutions: a motor vehicle three-dimensional target detection device connected with two vehicle-mounted cameras, the motor vehicle three-dimensional target detection device comprising a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, and the processor implements the motor vehicle three-dimensional target detection method according to any one of the above-mentioned embodiments when executing the computer program.

[0020] Further, the two vehicle-mounted cameras are integrated into a vehicle-mounted binocular camera device.

[0021] In another aspect, to solve the above further technical problem, the embodiment of the present application further provides the following technical solution: a computer readable storage medium, the computer readable storage medium comprises a stored computer program, wherein when the computer program runs, the device where the computer readable storage medium is located executes the motor vehicle three-dimensional target detection method of any one of the above.

[0022] After the above technical solution, the embodiment of the present application has at least the following beneficial effects: the embodiment of the present application acquires the first original image and the second original image from the video images collected by the two vehicle-mounted cameras which shoot the same area around the motor vehicle from different angles, uses the differences of the same three-dimensional target in the first original image and the second original image, and extracts the depth information corresponding to the three-dimensional target from the first original image and the second original image by using the pre-trained depth information extraction model, thereby replacing the three-dimensional information of the three-dimensional target obtained by using the laser radar in the traditional technology, and reducing the cost; further, the depth information is processed by using two convolution kernels of predetermined size, a pre-stored attention mechanism model, and scaled dot product calculation in sequence to obtain the semantic features of the three-dimensional target, and the semantic features are used as the feature classification branch; at the same time, the depth information is also processed by using a predetermined backbone convolution network, two fully connected layers of predetermined node number, a pre-stored SEblock model, and a sigmoid excitation function to obtain the geometric features of each three-dimensional target, and the geometric features are used as the feature regression branch, finally, the feature classification branch and the feature regression branch are input into the Transformer network model to calculate and obtain the category confidence and the three-dimensional data information of the three-dimensional target, the whole process combines the Transformer attention mechanism and the convolution network, and simultaneously fuses the global information and the semantic information, which can effectively improve the detection efficiency of the three-dimensional target and reduce the consumption of the memory. BRIEF DESCRIPTION OF DRAWINGS

[0023] Figure 1 The step flow chart of an optional embodiment of the motor vehicle three-dimensional target detection method of the present application.

[0024] Figure 2 The principle block diagram of an optional embodiment of the motor vehicle three-dimensional target detection device of the present application.

[0025] Figure 3 The functional module diagram of an optional embodiment of the motor vehicle three-dimensional target detection device of the present application. DETAILED DESCRIPTION

[0026] The application will be described in further detail below with reference to the drawings and specific embodiments. It should be understood that the following illustrative embodiments and descriptions are only intended to explain the present application and are not intended to limit the present application, and the embodiments in the present application and the features in the embodiments can be combined with each other without conflict.

[0027] As shown in Figure 1 An optional embodiment of the present application provides a motor vehicle three-dimensional target detection method, comprising the following steps:

[0028] S1: obtaining a first original image and a second original image from video images respectively collected by vehicle-mounted cameras 1 which simultaneously shoot the same region around the body of a motor vehicle from different angles;

[0029] S2: extracting depth information corresponding to the same three-dimensional target from the first original image and the second original image respectively by using a pre-trained depth information extraction model, and screening and removing interference targets in the three-dimensional target;

[0030] S3: performing convolution calculation on the depth information of each three-dimensional target by using two predetermined size convolution kernels to respectively obtain a first to-be-projected feature and a second to-be-projected feature, projecting the first to-be-projected feature and the second to-be-projected feature as query values (query) and key values (keys) of an attention mechanism into a pre-stored attention mechanism model to obtain a to-be-calculated feature, performing scaling dot product calculation on the to-be-calculated feature to obtain a semantic feature of the three-dimensional target, and taking the semantic feature as a feature classification branch;

[0031] S4: performing convolution processing on the depth information of each three-dimensional target by using a predetermined backbone convolution network to obtain a to-be-enhanced feature, sequentially passing the to-be-enhanced feature through two predetermined node number fully connected layers, and then calculating a to-be-extracted feature by using a pre-stored SEblock model, inputting the to-be-enhanced feature and the to-be-extracted feature into a sigmoid excitation function to obtain a geometric feature of each three-dimensional target, and taking the geometric feature as a feature regression branch; and

[0032] S5: inputting the feature classification branch and the feature regression branch into a pre-stored Transformer network model for calculation to obtain a class confidence and three-dimensional data information of each three-dimensional target.

[0033] The embodiment of the application acquires a first original image and a second original image from the video images collected by two vehicle-mounted cameras 1 shooting the same area around the motor vehicle from different perspectives, uses the differences of the same three-dimensional target in the first original image and the second original image, extracts the depth information corresponding to the three-dimensional target from the first original image and the second original image by using a pre-trained depth information extraction model, replaces the three-dimensional information of the three-dimensional target obtained by using a laser radar in the traditional technology, thereby reducing the cost; further, the depth information is processed by using two convolution kernels of a predetermined size, a pre-stored attention mechanism model and scaled dot product calculation in sequence to obtain the semantic features of the three-dimensional target, and the semantic features are used as a feature classification branch; at the same time, the depth information is also processed by using a predetermined backbone convolution network, two fully connected layers of a predetermined node number, a pre-stored SEblock model and a sigmoid excitation function to obtain the geometric features of each three-dimensional target, and the geometric features are used as a feature regression branch, finally, the feature classification branch and the feature regression branch are input into a Transformer network model to calculate and obtain the category confidence and the three-dimensional data information of the three-dimensional target, the overall process combines the Transformer attention mechanism and the convolution network, simultaneously fuses the global information and the semantic information, can effectively improve the detection efficiency of the three-dimensional target, and reduces the consumption of the memory.

[0034] In the specific implementation, after the feature classification branch and the feature regression branch are input into the pre-stored Transformer network model, the Transformer network model processes the feature sequence in the form of text, and finally outputs the category of the three-dimensional target, the confidence corresponding to the category and the three-dimensional data (for example, the length, width, height of the three-dimensional target and the three-dimensional distance from the motor vehicle) of the three-dimensional target; in addition, the interference target in the three-dimensional target is screened and removed, wherein the interference target usually refers to the background area in the image.

[0035] In an optional embodiment of the application, the depth information extraction model is obtained by inputting a large number of training images of calibrated three-dimensional targets into a DenseNet deep learning network, data normalization and data augmentation. In this embodiment, the depth information extraction model based on deep learning is obtained by using the above training process, which covers relatively wide image information, and the generated depth information extraction model can effectively extract the depth information of various three-dimensional targets.

[0036] In the specific process, the specific training process is as follows:

[0037] Firstly, the training images of the pre-three-dimensional target are respectively input into a DenseNet deep learning network, and the gray information of the same three-dimensional target is obtained by comparison calculation, the gray information of the three-dimensional target is taken as the depth information of the three-dimensional target, and the prior condition is taken as the prior condition, different gray values correspond to different depth information, and different three-dimensional targets correspond to different depth information; then, the depth information of all three-dimensional targets is normalized to the interval range of [-1, 1], finally, the normalized depth information is flipped around the X axis and the Y axis of the world coordinate and rotated around the Z axis to realize data augmentation, and finally the depth information extraction model is obtained.

[0038] In an optional embodiment of the present application, the predetermined backbone convolutional network is a MobileNet network, a RetinaNet network or an EfficientNet network. In this embodiment, the above-mentioned backbone convolutional network can adopt a MobileNet network, a RetinaNet network or an EfficientNet network to perform convolutional processing on the depth information of the three-dimensional target, and the corresponding backbone convolutional network can effectively strengthen the features of the three-dimensional target and improve the accuracy of detection and recognition.

[0039] In an optional embodiment of the present application, after obtaining the semantic features of the three-dimensional target, a convolution kernel with a size of 1*1 is used to perform convolutional processing on the semantic features, and the convolutional processing semantic features are taken as the feature classification branch. In this embodiment, after obtaining the semantic features, a convolution kernel with a size of 1*1 is used to perform convolutional processing on the semantic features, which can adapt the input requirements of the subsequent Transformer network model, increase the nonlinear capability of the network, and improve the expression capability of the network.

[0040] In an optional embodiment of the present application, the two predetermined size convolution kernels are both 3*3 convolution kernels. In this embodiment, the size of the convolution kernel is only 3*3, and the calculation amount is relatively small when performing convolution calculation, which reduces the memory consumption, and also effectively improves the expression capability of the network to a certain extent.

[0041] In an optional embodiment of the present application, the two predetermined node number full connection layers are both 1024 node full connection layers. In this embodiment, the above-mentioned node full connection layer is used, and the main function of the full connection layer is to integrate the feature representation into a value, which can effectively reduce the influence of the feature position on the classification result and improve the robustness of the entire network.

[0042] On the other hand, as Figure 2As shown, the embodiment of the present application further provides a motor vehicle three-dimensional target detection device connected with the two vehicle cameras 1. The motor vehicle three-dimensional target detection device 3 comprises a processor 30, a memory 32, and a computer program stored in the memory 32 and configured to be executed by the processor 30. When the processor 30 executes the computer program, the motor vehicle three-dimensional target detection method according to any one of the above embodiments is implemented.

[0043] For example, the computer program can be divided into one or more modules / units, which are stored in the memory 32 and executed by the processor to complete the present application. The one or more modules / units can be a series of computer program instruction segments capable of completing a specific function, which are used to describe the execution process of the computer program in the motor vehicle three-dimensional target detection device 3. For example, the computer program can be divided into Figure 3 The motor vehicle three-dimensional target detection device 3 comprises an image acquisition module 41, a depth information extraction module 42, a feature classification module 43, a feature regression module 44, and a Transformer module 45, which correspond to steps S1-S5 respectively.

[0044] The motor vehicle three-dimensional target detection device 3 can be a desktop computer, a notebook computer, a palm computer, a cloud server, or the like. The motor vehicle three-dimensional target detection device 3 can include, but is not limited to, a processor 30 and a memory 32. Those skilled in the art can understand that the schematic diagram is only an example of the motor vehicle three-dimensional target detection device 3, and does not constitute a limitation on the motor vehicle three-dimensional target detection device 3, which can include more or fewer components than the diagram, or combine certain components, or different components, for example, the motor vehicle three-dimensional target detection device 3 can further include an input / output device, a network access device, a bus, and the like.

[0045] The processor 30 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor. The processor 30 is the control center of the vehicle 3D target detection device 3, connecting all parts of the device via various interfaces and lines.

[0046] The memory 32 can be used to store the computer program and / or modules. The processor 30 implements various functions of the three-dimensional target detection device 3 for motor vehicles by running or executing the computer program and / or modules stored in the memory 32 and calling the data stored in the memory 32. The memory 32 may mainly include a program storage area and a data storage area. The program storage area may store the operating system, application programs required for at least one function (such as image recognition function, image overlay function, etc.), etc.; the data storage area may store data created according to the use of the calibration device (such as image data, etc.). In addition, the memory 32 may include high-speed random access memory, and may also include non-volatile memory, such as hard disk, memory, plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, at least one disk storage device, flash memory device, or other volatile solid-state storage device.

[0047] If the functions described in the embodiments of the present invention are implemented in the form of software functional modules or units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the embodiments of the present invention can implement all or part of the processes in the methods described above, or they can be accomplished by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by the processor 30, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include: any entity or device capable of carrying the computer program code, recording media, USB flash drives, portable hard drives, magnetic disks, optical disks, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the content included in the computer-readable medium can be appropriately added or removed according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media do not include electrical carrier signals and telecommunication signals.

[0048] In an optional embodiment of the present invention, the two vehicle-mounted cameras 1 are integrated into a single vehicle-mounted binocular camera device. In this embodiment, the two vehicle-mounted cameras 1 are integrated into a single vehicle-mounted binocular camera device, resulting in a relatively simple overall installation structure, and both can simultaneously capture images of the same area around the vehicle body from different perspectives.

[0049] In another aspect, embodiments of the present invention provide a computer-readable storage medium, the computer-readable storage medium including a stored computer program, wherein, when the computer program is executed, it controls the device where the computer-readable storage medium is located to execute the motor vehicle three-dimensional target detection method described above.

[0050] The various embodiments in this specification are described in a progressive manner. Each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.

[0051] The embodiments of the present invention have been described above with reference to the accompanying drawings. However, the present invention is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of the present invention without departing from the spirit and scope of the claims. All of these forms are within the scope of protection of the present invention.

Claims

1. A method for detecting three-dimensional targets of motor vehicles, characterized in that, The method includes the following steps: The first and second original images are obtained from video images captured by two vehicle-mounted cameras that simultaneously capture the same area around the vehicle from different perspectives. A pre-trained depth information extraction model is used to extract the depth information corresponding to the same three-dimensional target from the first original image and the second original image respectively, and interference targets in the three-dimensional target are filtered out. Two convolutional kernels of predetermined size are used to perform convolutional calculations on the depth information of each three-dimensional target to obtain a first feature to be projected and a second feature to be projected, respectively. The first feature to be projected and the second feature to be projected are used as the query value and key value of the attention mechanism, respectively, and projected into the pre-stored attention mechanism model to obtain the feature to be calculated. The feature to be calculated is scaled and dot-producted to obtain the semantic features of the three-dimensional target. The semantic features are used as the feature classification branch. A predetermined backbone convolutional network is used to convolve the depth information of each 3D target to obtain features to be enhanced. These features are then passed sequentially through two fully connected layers with a predetermined number of nodes, and a pre-stored SEblock model is used to calculate features to be extracted. The enhanced and extracted features are then input into a sigmoid activation function to obtain the geometric features of each 3D target, and these geometric features are used as a feature regression branch. The feature classification branch and the feature regression branch are input into a pre-stored Transformer network model for calculation to obtain the category confidence and 3D data information of each 3D target.

2. The method for detecting three-dimensional targets of motor vehicles as described in claim 1, characterized in that, The depth information extraction model is obtained by inputting a large number of calibrated 3D target training images into a DenseNet deep learning network, followed by data normalization and data augmentation.

3. The method for detecting three-dimensional targets of motor vehicles as described in claim 1, characterized in that, The predetermined backbone convolutional network is a MobileNet network, a RetinaNet network, or an EfficientNet network.

4. The method for detecting three-dimensional targets of motor vehicles as described in claim 1, characterized in that, After obtaining the semantic features of the three-dimensional target, the semantic features are convolved using a 1*1 kernel, and the convolved semantic features are used as the feature classification branch.

5. The method for detecting three-dimensional targets of motor vehicles as described in claim 1, characterized in that, Both of the predetermined-size convolution kernels are 3*3 convolution kernels.

6. The method for detecting three-dimensional targets of motor vehicles as described in claim 1, characterized in that, Both of the predetermined number of nodes in the fully connected layers are 1024-node fully connected layers.

7. A three-dimensional target detection device for motor vehicles, connected to two vehicle-mounted cameras, characterized in that, The vehicle three-dimensional target detection device includes a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor. When the processor executes the computer program, it implements the vehicle three-dimensional target detection method as described in any one of claims 1 to 6.

8. The three-dimensional target detection device for motor vehicles as described in claim 7, characterized in that, The two vehicle-mounted cameras are integrated into a single vehicle-mounted binocular camera device.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a stored computer program, wherein, when the computer program is executed, it controls the device containing the computer-readable storage medium to perform the motor vehicle three-dimensional target detection method as described in any one of claims 1 to 6.

Images