A target detection method and device

Abstract

The present application provides a target detection method and device, which relates to the field of unmanned driving technology and can improve the accuracy of target detection. Specific solution package: obtain visual images and point cloud data of the area to be tested, the point cloud data includes one or more radar reflection points; associate one or more image features extracted based on the visual image with the point cloud data to obtain one or more radar reflection points associated with each image feature; determine the radar feature corresponding to each image feature based on the one or more radar reflection points associated with each image feature; detect the spatial state of the target object in the area to be tested based on each image feature and the radar feature corresponding to each image feature, the target object includes a human body and / or an object.

Description

Technical Field

[0001] The present application relates to the field of unmanned driving technology, and in particular to a target detection method and device. Background Art

[0002] Self-driving automobiles, also known as automatic driving cars, computer-driven cars, or wheeled mobile robots, are intelligent cars that are unmanned by computer systems. Among them, for self-driving cars, target detection of the driving road environment is a key technical link in realizing the perception of vehicle automatic driving. This ensures that the vehicle's driving perception system determines the surrounding environment of the vehicle and avoids collisions with other vehicles, pedestrians and other targets, thereby realizing safe automatic driving of the vehicle on the lane.

[0003] The above-mentioned target detection technology refers to the technology that the driverless car uses the sensors it carries to identify other vehicles, pedestrians and other targets in the surrounding environment. Driverless cars can carry one or more sensors to work. In a complex working environment, the information obtained by a single sensor is limited. Relying solely on a single sensor cannot guarantee the correct detection of the target under various conditions, which will cause the driverless car to have incorrect environmental perception. Therefore, multi-sensor fusion is usually used to make up for the defects of each sensor, so as to detect targets more accurately and quickly and perceive the environment around the vehicle. Among them, the fusion technology based on camera and millimeter-wave radar has become one of the important technologies for autonomous driving.

[0004] At present, most of the solutions based on the fusion of camera and millimeter wave radar use the projection of millimeter wave radar on the camera image to generate the detection area, and then perform two-dimensional target detection in the image coordinate system. However, the process of converting the 3D point cloud information of the object received by the millimeter wave radar into a 2D plane image consistent with vision has large errors, which will affect the accuracy of target detection. Summary of the invention

[0005] The present application provides a target detection method and device, which can be used to improve the accuracy of target detection.

[0006] In order to achieve the above technical objectives, this application adopts the following technical solutions:

[0007] In a first aspect, an embodiment of the present application provides a target detection method, the method comprising: acquiring a visual image and point cloud data of an area to be measured, the point cloud data comprising one or more radar reflection points; associating one or more image features extracted based on the visual image with the point cloud data to obtain one or more radar reflection points associated with each image feature; determining a radar feature corresponding to each image feature based on the one or more radar reflection points associated with each image feature; and detecting a spatial state of a target object in the area to be measured based on each image feature and the radar feature corresponding to each image feature, the target object comprising a human body and / or an object.

[0008] Based on the above method, this method associates the visual image with the point cloud data, and then extracts the features of the point cloud data according to the association results to generate radar features corresponding to the image features. Finally, the two features are fused to detect the spatial state of the target object in the area to be detected. In this way, on the one hand, in the process of processing the visual image and the point cloud data, the method does not lose the three-dimensional spatial information reflected by the radar point cloud data. The image features can be associated with one or more radar reflection points, which can reduce the impact of isolated outliers on the association results; thereby, the accuracy of the cross-modal sensor data association between the visual image and the point cloud data can be improved. On the other hand, while detecting the target object in the area to be detected, the three-dimensional spatial state of the target object can also be estimated.

[0009] In a possible implementation, the above-mentioned associating one or more image features extracted based on the visual image with the point cloud data to obtain one or more radar reflection points associated with each image feature includes: for a first image feature among the one or more image features, determining the association probability between the first image feature and each radar reflection point according to the first image feature and the point cloud data; and determining the radar reflection point with an association probability greater than or equal to a preset probability as the radar reflection point associated with the first image feature. It can be understood that in this implementation, the association probability between the image feature and each radar reflection point can be determined based on the spatial position information between the image feature and the point cloud data, and then judging whether the image feature is associated with the radar reflection point based on the association probability, which can improve the accuracy of the judgment process.

[0010] In another possible implementation, the above-mentioned determining the association probability between the first image feature and each radar reflection point based on the first image feature and the point cloud data, determining one or more radar reflection points associated with any image feature based on the image feature and the point cloud data, includes: determining the relative position relationship between the first arbitrary image feature and each radar reflection point in the point cloud data; determining the depth information of the first arbitrary image feature and the depth information of each radar reflection point in the point cloud data; determining the association probability of the first arbitrary image feature and each radar reflection point based on the relative position relationship, the depth information of the first arbitrary image feature and the depth information of each radar reflection point in the point cloud data; and determining the radar reflection point whose association probability is greater than or equal to the preset probability as the radar reflection point associated with any image feature.

[0011] In another possible implementation, the association probability satisfies the following relationship:

[0012]

[0013] Where d is the depth information of the radar reflection point, μ d is the depth mean in the depth information of the first image feature, σ d is the depth standard deviation in the depth information of the first image feature, and the viewing cone is used to indicate a spatial area determined based on the positions of the target image frame and the image acquisition device corresponding to the first image feature.

[0014] In another possible implementation, the detecting the spatial state of the target object in the measured area according to each image feature and the radar feature corresponding to each image feature includes: determining, for a first image feature among the one or more image features, a joint feature of the first image feature and the radar feature corresponding to the first image feature; and determining the spatial state of the target object in the measured area according to the first image feature and the joint feature.

[0015] It can be understood that in this implementation, the fusion features of the image features and the radar features are first determined, and then target detection is performed based on the fusion features. In this way, the correlation between the visual image and the point cloud data in spatial position can be fully utilized to improve the accuracy of target detection.

[0016] In another possible implementation, the above-mentioned determining the spatial state of the target object in the target area to be measured based on the first image feature and the joint feature includes: determining one or more of the target category, orientation angle, size, and center offset of the target object in the target area to be measured based on the first image feature; determining the speed information and direction information of the target object in the target area to be measured based on the joint feature; determining the depth information of the target object in the target area to be measured based on the first image feature and the joint feature; determining the spatial state of the target object based on one or more of the target category, orientation angle, size, center offset, speed information, direction information and depth information of the target object in the target area to be measured.

[0017] It can be understood that in this implementation, the target detection device can group the prediction information of the target object to be predicted. For example, the image features extracted based on the visual image can be used to predict the target category, orientation angle, size, center offset and other information of the target object. Based on the joint features of the image features and the radar data, the speed information and direction information of the target object can be predicted. In this way, the respective advantages of the collected data of the image acquisition device and the collected data of the radar can be fully utilized, and the accuracy of the prediction can be improved.

[0018] In another possible implementation, one or more of the target category, orientation angle, size, and center offset of the target object in the test area are determined based on the first image feature, including: inputting the first image feature into a first prediction module to obtain one or more of the target category, orientation angle, size, and center offset of the target object in the test area; wherein the first prediction module includes one or more first sub-modules, a first sub-module includes a 3×3 convolution kernel, a linear rectification unit, and a 1×1 convolution kernel, and a first sub-module is used to output the target category, orientation angle, size, or center offset of the target object in the test area.

[0019] In another possible implementation, the above-mentioned determining the speed information and direction information of the target object in the measured area based on the joint features includes: inputting the joint features into a second prediction module to obtain the speed information and direction information of the target object in the measured area; wherein the second prediction module includes a 1×1 convolution kernel, a linear rectification unit, a 1×1 convolution kernel, a linear rectification unit and a 1×1 convolution kernel.

[0020] In another possible implementation, the above-mentioned determining the depth information of the target object in the area to be measured based on the first image feature and the joint feature includes: inputting the first image feature and the joint feature into a third prediction module to obtain the depth information of the target object in the area to be measured; wherein the third prediction module includes one or more first sub-modules and second sub-modules, the first sub-module includes a 3×3 convolution kernel, a linear rectification unit and a 1×1 convolution kernel, and the second sub-module includes a 1×1 convolution kernel, a linear rectification unit, a 1×1 convolution kernel, a linear rectification unit and a 1×1 convolution kernel.

[0021] In a second aspect, an embodiment of the present application provides a target detection device, which includes an acquisition module, a feature association module, a feature extraction module and a target detection module, wherein the acquisition module is used to acquire a visual image and point cloud data of the area to be measured; the feature association module is used to associate one or more image features extracted based on the visual image with the point cloud data to obtain one or more radar reflection points associated with each image feature; the feature extraction module is used to determine the radar feature corresponding to each image feature based on the one or more radar reflection points associated with each image feature; the target detection module is used to detect the spatial state of the target object in the area to be measured based on each image feature and the radar feature corresponding to each image feature, the target object including the human body and / or the object.

[0022] In a possible implementation, the feature association module is specifically used to: for a first image feature among one or more image features, determine the association probability between the first image feature and each radar reflection point based on the first image feature and the point cloud data; and determine the radar reflection point whose association probability is greater than or equal to a preset probability as the radar reflection point associated with the first image feature.

[0023] In another possible implementation, the feature association module is specifically used to: determine the relative position relationship between the first image feature and each radar reflection point in the point cloud data; determine the depth information of the first image feature and the depth information of each radar reflection point in the point cloud data; determine the association probability of the first image feature and each radar reflection point based on the relative position relationship, the depth information of the first image feature and the depth information of each radar reflection point in the point cloud data.

[0024] In another possible implementation, the association probability satisfies the following relationship:

[0025]

[0026] Where d is the depth information of the radar reflection point, μ d is the depth mean in the depth information of the first image feature, σ dis the depth standard deviation in the depth information of the first image feature, and the viewing cone is used to indicate a spatial area determined based on the positions of the target image frame and the image acquisition device corresponding to the first image feature.

[0027] In another possible implementation, the target detection module is specifically used to: determine, for a first image feature among one or more image features, a joint feature of the first image feature and a radar feature corresponding to the first image feature, and determine the spatial state of the target object in the area to be measured based on the first image feature and the joint feature.

[0028] In another possible implementation, the target detection module is specifically used to: determine one or more of the target category, orientation angle, size, and center offset of the target object in the measured area based on the first image feature; determine the speed information and direction information of the target object in the measured area based on the joint feature; determine the depth information of the target object in the measured area based on the first image feature and the joint feature; determine the spatial state of the target object based on one or more of the target category, orientation angle, size, center offset, speed information, direction information, and depth information of the target object in the measured area.

[0029] In another possible implementation, the above-mentioned target detection module is specifically used to: input the first image feature into the first prediction module to obtain one or more of the target category, orientation angle, size, and center offset of the target object in the target area to be measured; wherein the first prediction module includes one or more first sub-modules, a first sub-module includes a 3×3 convolution kernel, a linear rectification unit and a 1×1 convolution kernel, and a first sub-module is used to output the target category, orientation angle, size or center offset of the target object in the target area to be measured.

[0030] In another possible implementation, the above-mentioned target detection module is also specifically used to: input the joint features into the second prediction module to obtain the speed information and direction information of the target object in the measured area; wherein the second prediction module includes a 1×1 convolution kernel, a linear rectification unit, a 1×1 convolution kernel, a linear rectification unit and a 1×1 convolution kernel.

[0031] In another possible implementation, the above-mentioned target detection module is further specifically used to: input the first image feature and the joint feature into a third prediction module to obtain depth information of the target object in the detected area; wherein the third prediction module includes one or more first sub-modules and second sub-modules, the first sub-module includes a 3×3 convolution kernel, a linear rectification unit and a 1×1 convolution kernel, and the second sub-module includes a 1×1 convolution kernel, a linear rectification unit, a 1×1 convolution kernel, a linear rectification unit and a 1×1 convolution kernel.

[0032] In a third aspect, the present application provides an electronic device, the electronic device comprising a memory and a processor. The memory and the processor are coupled. The memory is used to store a computer program code, the computer program code comprising a computer instruction. When the processor executes the computer instruction, the electronic device performs the target detection method as described in the first aspect and any possible design thereof.

[0033] In a fourth aspect, the present application provides a chip system, which is applied to a target detection device; the chip system includes one or more interface circuits and one or more processors. The interface circuit and the processor are interconnected through a line; the interface circuit is used to receive a signal from a memory of the target detection device and send a signal to the processor, the signal including a computer instruction stored in the memory. When the processor executes the computer instruction, the electronic device executes the target detection method as described in the first aspect and any possible design thereof.

[0034] In a fifth aspect, the present application provides a computer-readable storage medium storing computer instructions. When the computer instructions are executed on an electronic device, the electronic device executes the target detection method as described in the first aspect and any possible design thereof.

[0035] In a sixth aspect, the present application provides a computer program product, which includes computer instructions. When the computer instructions are executed on an electronic device, the electronic device executes the target detection method as described in the first aspect and any possible design thereof.

[0036] For the specific description of the second to sixth aspects and their various implementations in the present application, reference may be made to the detailed description in the first aspect and its various implementations; and for the beneficial effects of the second to sixth aspects and their various implementations, reference may be made to the beneficial effects analysis in the first aspect and its various implementations, which will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 A schematic diagram of the composition of a target detection system provided in an embodiment of the present application;

[0038] Figure 2 A schematic diagram of a target detection scenario provided in an embodiment of the present application;

[0039] Figure 3 A schematic diagram of the structure of a target detection device provided in an embodiment of the present application;

[0040] Figure 4 A flow chart of a target detection method provided in an embodiment of the present application;

[0041] Figure 5A flowchart of another target detection method provided in an embodiment of the present application;

[0042] Figure 6 A schematic diagram of a residual network provided in an embodiment of the present application;

[0043] Figure 7 A schematic diagram of the composition of a joint module of image features and point cloud data provided in an embodiment of the present application;

[0044] Figure 8 A schematic diagram of a viewing cone provided in an embodiment of the present application;

[0045] Fig. 9 A flowchart of another target detection method provided in an embodiment of the present application;

[0046] Fig.10 A schematic diagram of the composition of a target detection module provided in an embodiment of the present application;

[0047] Fig.11 A schematic diagram of the composition of a prediction module provided in an embodiment of the present application;

[0048] Fig.12 A schematic diagram of the structure of a target detection device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0049] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0050] In the description of this application, unless otherwise specified, " / " means "or", for example, A / B can mean A or B. "And / or" in this article is merely a description of the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. In addition, "at least one" means one or more, and "plurality" means two or more. The words "first", "second", etc. do not limit the quantity and execution order, and the words "first", "second", etc. do not limit them to be different.

[0051] It should be noted that, in this application, words such as "exemplary" or "for example" are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "exemplary" or "for example" in this application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a specific way.

[0052] To facilitate understanding, some basic concepts of terms or technologies involved in the embodiments of the present invention are first briefly introduced and explained.

[0053] 1. Object Detection

[0054] Target detection refers to the process of finding a target from a scene (e.g., an image) and determining the location of the target. Target detection has a wide range of applications in many areas of life, such as autonomous driving, driver assistance and warning. In the process of target detection and recognition, multi-sensor fusion is usually required. For example, the data collected by LiDAR, millimeter-wave radar, visual sensor, infrared sensor, etc. are fused to obtain information about the vehicle's surrounding environment, that is, to detect target objects in the vehicle's surrounding environment. Exemplarily, the target object can be any object in the vehicle's surrounding environment, such as a vehicle, a person, a tree, a building, etc.

[0055] 2. Convolutional neural networks (CNN)

[0056] Convolutional neural networks are a type of feedforward neural networks that include convolutional calculations and have a deep structure. They are one of the representative algorithms of deep learning. Convolutional neural networks can be applied to computer vision such as image classification, object recognition, action recognition, pose estimation, neural style transfer, and can also be applied to natural language processing (NLP).

[0057] Generally speaking, a convolutional neural network consists of an input layer, a hidden layer, and an output layer.

[0058] Among them, the input layer of the convolutional neural network can process multi-dimensional data. Taking image processing as an example, the input layer can receive the pixel values ​​(three-dimensional array) of the image, that is, the two-dimensional pixel points on the plane and the values ​​of the RGB channels.

[0059] The hidden layer of a convolutional neural network includes one or more convolutional layers, one or more pooling layers, and one or more fully-connected layers. The function of the convolutional layer is to extract features from the input data. The pooling layer is usually connected after the convolutional layer, so that after the convolutional layer extracts features, the output data is passed to the pooling layer for selection and information filtering. Each node of the fully-connected layer is connected to all nodes of the previous layer to integrate the acquired features. The fully-connected layer plays the role of a "classifier" in the entire convolutional neural network.

[0060] The output layer of the convolutional neural network has the same structure and working principle as the output of the traditional feedforward neural network. For example, for the convolutional neural network for image classification, the output layer uses a logistic function or a normalized exponential function (softmax function) to output classification labels, such as: people, scenery, objects, etc.

[0061] 3. Radar

[0062] Radar is a device that can detect targets using electromagnetic waves. The radar's transmitting antenna converts the high-frequency current signal generated by the transmitter circuit or the guided wave on the transmission line into an electromagnetic wave with a certain polarization mode that can be transmitted in space and is transmitted along a preset direction. When the electromagnetic wave encounters an obstacle in the forward direction, part of the electromagnetic wave will be reflected back in the opposite direction of the transmission direction. At this time, the radar's receiving antenna can receive the reflected electromagnetic wave and convert it into a high-frequency current signal or a guided wave on the transmission line. By subsequently processing the obtained echo signal, the target's distance, speed, angle and other status information can be extracted.

[0063] Exemplarily, a radar may be composed of a radar transmitter, a radar receiver and an antenna.

[0064] Radar transmitter is a radio device that provides high-power radio frequency signals for radar. It can generate high-power radio frequency signals with modulated carrier waves, i.e. electromagnetic waves. According to the modulation method, transmitters can be divided into two categories: continuous wave transmitters and pulse transmitters. The transmitter consists of a first-level radio frequency oscillator and a pulse modulator.

[0065] Radar receiver is a device that performs frequency conversion, filtering, amplification and demodulation in radar. Through appropriate filtering, the weak high-frequency signal received by the antenna is selected from the accompanying noise and interference, and after amplification and detection, it is used for target detection, display or other radar signal processing.

[0066] Antenna is a device used in radar equipment to transmit or receive electromagnetic waves and determine the detection direction. When transmitting, it radiates energy to the direction that needs to be illuminated; when receiving, it receives the echo in the detection direction and distinguishes the azimuth and / or elevation of the target.

[0067] 4. Point cloud data

[0068] Point cloud data is a collection of massive points that express the spatial distribution and surface characteristics of the target in the same spatial reference system. After obtaining the spatial position of each reflection point in the area to be measured or on the surface of the object to be measured, a collection of points is obtained, which is called a "point cloud". Among them, the spatial reference system can be called a radar coordinate system.

[0069] The above is an introduction to the technical terms involved in the embodiments of the present disclosure, which will not be repeated below.

[0070] As described in the background technology, the fusion of camera and millimeter wave radar to detect the surrounding environment has become one of the common methods of autonomous driving. Among them, the millimeter wave radar has a long detection distance and high detection accuracy, and can reflect the spatial three-dimensional information of the environment, but it lacks in detail description. Although the visual sensor (such as a camera) cannot reflect the spatial three-dimensional information of the environment, it has outstanding effects in detail and color description, which makes up for the disadvantage of millimeter wave radar in identifying objects. At present, the error of the fusion technology based on camera and millimeter wave radar is large, resulting in a low accuracy rate of target detection.

[0071] In view of this, the present application provides a target detection method, which includes: first obtaining a visual image and point cloud data of the area to be measured; performing feature extraction on the obtained visual image to obtain one or more image features; determining the radar feature corresponding to each image feature based on the one or more image features and the point cloud data; and detecting the spatial state of the target object in the area to be measured based on each image feature and the radar feature corresponding to each image feature, the target object including the human body and / or an object. In this way, in the process of processing the visual image and the point cloud data, the three-dimensional spatial information embodied by the radar point cloud data is not lost, thereby improving the accuracy of the cross-modal sensor data association between the visual image and the point cloud data.

[0072] The method provided in the embodiment of the present application The target detection method of the embodiment of the present application can be applied to drone navigation systems, unmanned driving systems, or driving assistance warning systems, for example, including pedestrian collision warning (PCW), forward collision warning (FCW), lane departure warning (LDW), vehicle distance detection and warning (HMW), etc.

[0073] like Figure 1As shown, an embodiment of the present application provides a schematic diagram of a target detection system. The target detection system 100 includes: an image acquisition device 10, a radar 20 and a control device 30. The image acquisition device 10 and the radar 20 are connected to the control device 30 respectively.

[0074] It should be understood that the above-mentioned connection method can be a wireless connection, such as a Bluetooth connection, a Wi-Fi connection, etc.; or, the above-mentioned connection method can also be a wired connection, such as a fiber optic connection, etc., without limitation.

[0075] The image acquisition device 10 is any one of a camera, a video camera, a camera, a scanner, or other devices with a camera function (for example, a mobile phone, a tablet computer, etc.).

[0076] Optionally, the image acquisition device 10 may include a lens (LENS) and an image sensor. Objects in the field of view of the image acquisition device 10 are projected onto the image sensor through an optical image generated by the lens (LENS), and the image sensor converts it into an electrical signal, and then obtains a visual image of the area to be measured after processing such as analog-to-digital conversion (A / D conversion).

[0077] The radar 20 may be any one or more combinations of a laser radar sensor, a millimeter wave radar sensor, etc. The radar 20 may detect targets in the area to be detected using electromagnetic waves to obtain radar point cloud data.

[0078] For example, Figure 2 As shown, the radar installed on vehicle A can transmit detection electromagnetic waves to obstacle B, and receive the echo signal generated by obstacle B reflecting the detection electromagnetic waves, that is, the above-mentioned radar point cloud data.

[0079] In some embodiments of the present application, a millimeter wave radar with strong anti-interference capability, strong resolution capability and high measurement accuracy is used.

[0080] Among them, millimeter wave radar is a radar that works in the millimeter wave band, which can emit signals with a wavelength of 1-10mm and a frequency of 30GHZ-300GHZ. In the electromagnetic spectrum, this wavelength is considered a short wavelength, and a short wavelength means high accuracy. For example, a millimeter wave system with an operating frequency of 76-81GHz (corresponding to a wavelength of about 4mm) will be able to detect movements as small as a few tenths of a millimeter.

[0081] The control device 30 is used to process the visual image and point cloud data of the area to be detected by the target detection method of the embodiment of the present application to detect the spatial information of the target object in the area to be detected, and the spatial information can be used to reflect the position and / or category attributes of the target. The target detection method of the embodiment of the present application can improve the accuracy of target detection, and its specific implementation method can refer to the explanation of the following method embodiment.

[0082] In a possible implementation, the image acquisition device 10, radar 20 and control device 30 may be arranged on the same device, such as a vehicle, a drone, etc.

[0083] In another possible implementation, the image acquisition device 10 and radar 20 may be arranged on the same device, such as a vehicle, a drone, etc. The control device 30 may be a terminal device respectively connected to the image acquisition device 10 and the radar 20. The control device 30 may communicate with the device in which the image acquisition device 10 and the radar 20 are arranged to obtain visual images and point cloud data of the area to be measured.

[0084] Among them, the terminal device in the embodiment of the present application can be, for example, a mobile phone, a tablet computer, a desktop, a laptop, a handheld computer, a notebook computer, an ultra-mobile personal computer (UMPC), a netbook, and a cellular phone, a personal digital assistant (PDA), an augmented reality (AR)\virtual reality (VR) device, an intelligent remote controller, etc. It can interact with the user through one or more methods such as a keyboard, a touchpad, a touch screen, a remote controller, voice interaction or a handwriting device. In addition, in the embodiment of the present application, the terminal device may have an interface for communicating with a cellular network and / or a wireless local area network (WLAN).

[0085] The embodiment of the present application also provides a target detection device, which is the execution subject of the above-mentioned target detection control method. The target detection device is an electronic device with data processing capabilities. For example, the target detection device can be the control device 30 in the above-mentioned target detection system 100, or the target detection device can be a functional module in the control device 30, or the target detection device can be any computing device connected to the control device 30, etc. The embodiment of the present application does not limit this.

[0086] Combine the following Figure 3 A hardware structure of the target detection device 200 is introduced.

[0087] like Figure 3 As shown, the target detection device 200 includes a processor 210 , a communication line 220 and a communication interface 230 .

[0088] Optionally, the target detection device 200 may further include a memory 240 . The processor 210 , the memory 240 and the communication interface 230 may be connected via a communication line 220 .

[0089] The processor 210 may be a central processing unit (CPU), a general-purpose processor, a network processor (NP), a digital signal processor (DSP), a microprocessor, a microcontroller, a programmable logic device (PLD), or any combination thereof. The processor 210 may also be any other device having a processing function, such as a circuit, a device, or a software module, without limitation.

[0090] In one example, the processor 210 may include one or more CPUs, such as Figure 3 CPU0 and CPU1 in.

[0091] As an optional implementation, the object detection device 200 includes multiple processors, for example, in addition to the processor 210, it may also include a processor 270. The communication line 220 is used to transmit information between the components included in the object detection device 200.

[0092] The communication interface 230 is used to communicate with other devices or other communication networks. The other communication networks may be Ethernet, radio access network (RAN), wireless local area network (WLAN), etc. The communication interface 230 may be a module, a circuit, a transceiver or any device capable of achieving communication.

[0093] The memory 240 is used to store instructions, where the instructions may be computer programs.

[0094] Among them, the memory 240 can be a read-only memory (ROM) or other types of static storage devices that can store static information and / or instructions, or a random access memory (RAM) or other types of dynamic storage devices that can store information and / or instructions, or an electrically erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compressed optical disc, laser disc, optical disc, digital versatile disc, Blu-ray disc, etc.), magnetic disk storage medium or other magnetic storage devices, etc., without limitation.

[0095] It should be noted that the memory 240 may exist independently of the processor 210 or may be integrated with the processor 210. The memory 240 may be used to store instructions or program codes or some data, etc. The memory 240 may be located inside the target detection device 200 or outside the target detection device 200, without limitation.

[0096] The processor 210 is used to execute the instructions stored in the memory 240 to implement the communication method provided in the following embodiments of the present application. For example, when the target detection device 200 is a terminal or a chip or a system on a chip in the terminal, the processor 210 can execute the instructions stored in the memory 240 to implement the target detection device provided in the present application.

[0097] As an optional implementation, the target detection device 200 further includes an output device 250 and an input device 260. The output device 250 may be a device such as a display screen, a speaker, etc. that can output data of the target detection device 200 to a user. The input device 260 may be a device such as a keyboard, a mouse, a microphone, or a joystick that can input data to the target detection device 200.

[0098] It should be pointed out that Figure 3 The structure shown in the figure does not constitute a limitation on the control device, except Figure 3 In addition to the components shown, the control device may include more or fewer components than shown, or combine certain components, or arrange the components differently.

[0099] The target detection method provided in this application is described in detail below in conjunction with the accompanying drawings.

[0100] like Figure 4 As shown, the embodiment of the present application provides a target detection method, optionally, the method comprises Figure 2 The target detection device shown is executed, and the method includes the following steps:

[0101] S101, the target detection device obtains the visual image and point cloud data of the area to be detected.

[0102] The above-mentioned area to be tested refers to Figure 1 The area that the target detection system can detect is shown in FIG. This area is within the area where the image acquisition device 10 can capture pictures, and similarly, this area is also within the area that the radar 20 can detect.

[0103] For example, for the target detection system of an unmanned vehicle, the target detection system in the vehicle will continuously detect obstacles around the vehicle to assist the vehicle in sensing the surrounding environment and safely driving automatically on the lane. In the target detection system, the area to be detected refers to the area in the surrounding environment of the vehicle where target detection is being performed. In addition, the visual image can be acquired by an image acquisition device installed on the vehicle to acquire an image of the area to be detected, and the image acquisition device can send the acquired visual image to the target detection device of this embodiment. The point cloud data is the data acquired by the radar installed on the vehicle to detect the area to be detected, and the radar can also send the acquired point cloud data to the target detection device of this embodiment.

[0104] In actual use, radar usually emits electromagnetic waves periodically. When the electromagnetic waves encounter obstacles in the forward direction, part of the electromagnetic waves will be reflected back in the opposite direction of the emission direction. The electromagnetic waves reflected back by the target object in the detection area are the echo signals received by the radar. After signal processing, the required radar point cloud data can be obtained. Optionally, the signal processing of the echo signal includes fast Fourier transform (FFT), constant false alarm rate (CFAR), etc.

[0105] Among them, the target object refers to any target such as vehicles, people, trees, etc. in the area to be measured. Each frame of point cloud data acquired by the radar includes multiple radar reflection point data. In addition, a radar reflection point data may include the three-dimensional coordinate information of the radar reflection point. Alternatively, a radar reflection point data may include the two-dimensional coordinate information (i.e., plane coordinate information) of the radar reflection point. Optionally, a radar reflection point data may also include speed information, reflection cross-sectional area, signal-to-noise ratio of the reflection signal, time information, reflection intensity information, etc. Before fusion processing, for each radar reflection point three-dimensional spatial coordinate, it is necessary to use the external parameters of the calibrated radar device and the image acquisition device to convert it to the image acquisition device coordinate system, expressed as (x, y, z), where the z-axis direction is also called depth d.

[0106] Optionally, the point cloud data and the visual image of the area to be inspected may be data acquired synchronously.

[0107] It should be noted that the above-mentioned synchronous acquisition can be understood as the radar device 20 and the image acquisition device 10 in the above-mentioned target detection system 100 collecting data at the same time, or it can be understood as the deviation of the frame rate of the radar device 20 and the image acquisition device 10 collecting data is within a preset range. For example, the radar periodically collects point cloud data at a first frame rate, and the image acquisition device periodically collects visual images at a second frame rate. If the difference between the first frame rate and the second frame rate is less than the preset frame rate threshold, it can be considered that the above-mentioned point cloud data and visual images are synchronously acquired data.

[0108] S102: The target detection device associates one or more image features extracted based on the visual image with the point cloud data to obtain one or more radar reflection points associated with each image feature.

[0109] In some embodiments, the target detection device may first extract visual features to obtain one or more image features, and then associate each image feature with the point cloud data to obtain one or more radar reflection points associated with each image feature.

[0110] Optional, such as Figure 5 As shown, the above step S102 may specifically include S1021 and S1022:

[0111] S1021. The target detection device extracts visual features to obtain one or more image features.

[0112] Feature extraction refers to a method and process of extracting characteristic information from an image. For example, a target detection device can extract characteristic information from the visual image by analyzing and transforming the visual image.

[0113] The extracted characteristic information may include low-level information such as the contour and color of the target object, and may also include high-level characteristic information of the complete information of the target object.

[0114] As a possible implementation manner, the target detection device may input the visual image into a pre-trained feature extraction model for feature extraction to obtain one or more image features of the visual image.

[0115] Optionally, the feature extraction model may use a convolutional neural network to extract features from the acquired visual image. The feature extraction model may use an encoder-decoder structure, where the encoder in the first half uses a downsampling operation to perform preliminary feature extraction on the visual image through multiple feature extraction layers to obtain feature information at different levels. The decoder in the second half uses an upsampling operation to concatenate and fuse feature information at different levels to obtain fused feature information.

[0116] Optionally, the first half encoder of the feature extraction model can be implemented using a residual neural network (ResNet).

[0117] Residual network is a type of neural network. Residual network contains skip connection or shortcut connection, that is, adding a direct connection channel to the neural network to directly transmit the input original image information to the subsequent network layer. In this way, the subsequent network layer does not need to learn the entire image information output by the previous network layer, but directly learns the residual output by the previous network layer. In this way, the introduction of residual network connection can avoid the gradient disappearance phenomenon in the network and accelerate training.

[0118] Figure 6 A schematic diagram of a residual network is shown, such as Figure 6 As described above, the residual network includes five convolution modules. The first convolution module is Figure 6In the convolution module 1, 64 convolution filters with a step size of 2 and a 7x7 convolution kernel are used to convolve the input visual image, thereby obtaining an image feature with a size of 1 / 2 of the original visual image and a number of channels of 64. The second convolution module, i.e., convolution module 2, performs pooling on the feature map output by convolution module 1 to obtain an image feature with a size of 1 / 4 of the original feature image and a number of channels of 64, and then successively inputs three consecutive identical residual blocks to obtain an image feature with a size of 1 / 4 of the original visual image and a number of channels of 256. Among them, the structure of the residual block consists of 64 1x1 convolution kernels, 64 3x3 convolution kernels, and 256 1x1 convolution kernels. After three convolutions, the output of the final residual block is obtained. The third convolution module, i.e., convolution module 3, adopts the same structure as convolution module 2, which contains four residual blocks with three groups of convolutions of 128, 128, and 512, respectively, and finally obtains a feature map with a size of 1 / 8 of the original feature image and a number of channels of 512. The fourth convolution module, namely convolution module 4, adopts the same structure as convolution module 2, in which the three groups of convolution numbers contained in the residual block are 256, 256, and 1024 respectively, with a total of 23 residual blocks and 69 convolution layers. Convolution module 4 convolves on the feature map output by convolution module 3, and finally obtains image features with a size of 1 / 16 of the original visual image and a channel number of 1024. The fifth convolution module, namely convolution module 5, adopts the same structure as convolution module 2, in which the three groups of convolution numbers contained in the residual block are 512, 512, and 2048 respectively, with a total of 3 residual blocks and 9 convolution layers. Convolution module 5 convolves on the feature map output by convolution module 4, and finally obtains image features with a size of 1 / 32 of the original visual image and a channel number of 2048.

[0119] Furthermore, the decoder of the second half of the feature extraction model may include three deconvolution layers to upsample the acquired feature map with a size of 1 / 32 of the original feature image and a channel number of 2048. Optionally, the upsampling step size may be 2, that is, the magnification factor used for upsampling is 2. Thus, the image features with a size of 1 / 4 of the original feature image and a channel number of 256 obtained after three upsamplings can be expressed as Among them, W f =W img / 4,H f =H img / 4,W img is the width of the original visual image, H img is the height of the original visual image.

[0120] Optionally, the three deconvolution layers of the above decoder may be a combination of a 1×1 convolution kernel, a 3×3 convolution kernel, and a 1×1 convolution kernel.

[0121] S1022: The target detection device determines one or more radar reflection point data in the point cloud data associated with each image feature.

[0122] It should be noted that, according to the description of the visual image and point cloud data of the area to be detected in the above step S101, the target object in the area to be detected reflects the electromagnetic waves emitted by the radar, thereby receiving the point cloud data of the radar. The visual image of the area to be detected also includes the image of the target object in the area to be detected, and the image features obtained by extracting features from the visual image can also reflect the visual features of the target object in the area to be detected. Thus, an image feature can indicate the same position of the same target object as one or more radar reflection point data in the point cloud data. It should be understood that the above-mentioned image feature can be associated with one or more radar reflection point data in the point cloud data.

[0123] Therefore, in order to determine the radar feature corresponding to each image feature, the target detection device may first determine one or more radar reflection point data in the point cloud data associated with each image feature.

[0124] In some embodiments, for each extracted image feature, the target detection device determines one or more radar reflection points associated with the image feature in the point cloud data based on the image feature and the point cloud data.

[0125] Optionally, the target detection device may perform the following steps S11 to S14 to determine one or more radar reflection points associated with each image feature.

[0126] S11. The target detection device determines the relative position relationship between the image feature and each radar reflection point in the point cloud data.

[0127] Optional, such as Figure 7 As shown in (a) in FIG. 1 , the target detection device may first perform preliminary processing on the image feature through the first processing module 71 to generate a target image frame corresponding to the image feature.

[0128] The target image frame corresponding to each image feature may include a 4-dimensional vector, which are the horizontal coordinate, the vertical coordinate, and the width and height of the upper left point of the rectangular frame corresponding to the image feature in the visual image coordinate system.

[0129] Optional, such as Figure 7 As shown in (b) in FIG. 8 , the structure of the first processing module 71 may be a combination of a 3×3 convolution kernel, a rectified linear unit (ReLU) unit, and a 1×1 convolution kernel.

[0130] Among them, the ReLU unit is also called the rectified linear unit. The ReLU function used by this unit is an activation function commonly used in artificial neural networks. It usually refers to a nonlinear function represented by a ramp function and its variants. It is a nonlinear activation function, which can simulate a more accurate activation model of brain neurons receiving signals from a biological perspective.

[0131] Furthermore, based on the target image frame corresponding to the image feature, the position of the image acquisition device and the position of the radar reflection point, the target detection device can sequentially determine whether multiple radar reflection points in the point cloud data are within the visual cone.

[0132] It should be noted that, based on the imaging principle of the image acquisition device, since light propagates in a straight line, when an object is imaged, an inverted image will be formed on an image plane that is a distance f (that is, focal length) from the center point (that is, optical center) of the image acquisition device. Furthermore, a virtual image plane of forward imaging can be obtained at a position symmetrical to the image plane about the center of the camera. It can be understood that the pixels corresponding to the object in the virtual image plane can be determined based on the intersection of the line connecting the object and the optical center and the phase plane. It should be understood that the above-mentioned parameters such as optical center and focal length can be determined based on the preset parameters when the image acquisition device acquires the image.

[0133] Thus, the above-mentioned viewing cone can be Figure 8 As shown, the viewing cone includes 6 planes, namely, far plane, near plane, upper plane, lower plane, left plane and right plane. The above-mentioned viewing cone can be obtained by taking the plane where the target image frame corresponding to the image feature is located as the above-mentioned virtual image plane, connecting the optical center and the four corner points of the target image frame, and setting two sections at different distances from the optical center as the far and near planes of the viewing cone. Exemplarily, a section at a distance of 0 meters from the optical center can be set as the near plane, and a section at a distance of 200 meters from the optical center can be set as the far plane.

[0134] In one example, the target detection device can determine the target image frame corresponding to the image feature. Figure 8 The visual cone shown in the figure then determines the position of the radar reflection point according to the data of each radar reflection point, and then determines whether the radar reflection point is within the visual cone.

[0135] In another example, the target detection device does not generate Figure 8 The frustum shown can project the radar reflection point onto the virtual image plane where the target image frame is located based on the data of each radar reflection point. If the projection point corresponding to the radar reflection point is within the target image frame, it is determined that the radar reflection point is within the frustum. Otherwise, it is determined that the radar reflection point is not within the frustum.

[0136] S12. The target detection device determines the depth information of the image features and the depth information of each radar reflection point in the point cloud data.

[0137] Based on the relevant description of the point cloud data in step S101, it can be known that a radar reflection point data may include the depth information of the radar reflection point.

[0138] Optional, such as Figure 7 As shown in (a), the target detection device can process the image features through the second processing module 72 to generate depth information corresponding to the image features.

[0139] The depth information corresponding to each image feature may include a two-dimensional vector, which is respectively the mean and standard deviation of the depths of multiple pixel points corresponding to the image feature in three-dimensional space.

[0140] Optionally, the structure of the second processing module may be the same as the structure of the first processing module.

[0141] S13. The target detection device determines the association probability between the image feature and each radar reflection point according to the acquired relative position relationship, the depth information of the image feature, and the depth information of each radar reflection point in the point cloud data.

[0142] Exemplarily, the target detection device may determine the above association probability p according to the following formula (1).

[0143]

[0144] Where d is the depth information of the radar reflection point, μ d is the depth mean of the depth information of the above image features, σ d is the depth standard deviation in the depth information of the above image features.

[0145] S14. The target detection device determines the radar reflection point whose association probability is greater than or equal to the preset probability as the radar reflection point associated with the image feature.

[0146] It should be noted that, for an image feature, all radar reflection points with an association probability greater than or equal to a preset probability can be determined as radar reflection points associated with the image feature based on the association probability between the image feature and each radar reflection point. It should be understood that the number of radar reflection points associated with an image feature can be one, multiple, or zero. Furthermore, the present application does not limit the value of the preset probability.

[0147] Optionally, the target detection device can be Figure 7 The association module 73 shown in (a) executes the above steps S13 to S14 and outputs an association result, which is a radar reflection point associated with the image feature.

[0148] S103: The target detection device determines a radar feature corresponding to each image feature according to one or more radar reflection points associated with each image feature.

[0149] Optionally, the target detection device can perform feature extraction on a set of one or more radar reflection points associated with an image feature based on a preset PointNet network model. The input of the PointNet network model can be an N×7 dimensional matrix, where N is the number of radar reflection points associated with the image feature, and each radar reflection point includes a 7-dimensional vector (x, y, z, v x , v y , v z , p), (x, y, z) represents the position coordinates of the radar reflection point, v x ,v y and v z represents the radar speed and p represents the association probability.

[0150] Among them, the PointNet network model is a classic network model in point cloud research. The PointNet network model is a widely used research method in this field and has excellent performance in classification, semantic segmentation (point-by-point classification) and other problems.

[0151] It should be understood that when an image feature has no associated radar reflection point, the radar feature vector corresponding to the image feature can be 0. In addition, after point cloud feature extraction by the PointNet network model, each image feature has a corresponding 1024-dimensional radar feature vector. Finally, all radar feature vectors are sorted according to the image feature f img The corresponding radar features can be obtained by arranging them in order.

[0152] S104. The target detection device detects the spatial state of the target object in the detection area according to each image feature and the radar feature corresponding to each image feature. The target object includes a human body and / or an object.

[0153] The spatial state of the target object may include information such as the spatial position coordinates, rotation angle, and spatial movement speed of the target object.

[0154] The rotation angle may include a pitch angle, a yaw angle, and a roll angle. The pitch angle refers to the angle between the moving direction of the target object and the horizontal plane, the yaw angle refers to the angle between the projection direction of the moving direction of the target object on the horizontal plane and a predetermined direction on the horizontal plane, the predetermined direction may be set to the road direction, and the roll angle is used to indicate the lateral inclination angle.

[0155] It should be understood that the above-mentioned pitch angle, yaw angle and roll angle are three rotation angles used to identify the movement of the target in the navigation system.

[0156] Optional, such as Fig. 9 As shown, the above step S104 can be specifically implemented as the following steps:

[0157] S1041. For each image feature, the target detection device determines a joint feature of the image feature and a radar feature corresponding to the image feature.

[0158] Optionally, the target detection device may perform a splicing operation on the image feature and the radar feature corresponding to the image feature to obtain the above-mentioned joint feature.

[0159] Specifically, the target detection device can splice the image features extracted from the feature extraction model and the radar features extracted from the PointNet network model along the channel dimension to obtain the fused joint information.

[0160] S1042: The target detection device determines the spatial state of the target object in the area to be detected based on the image features and the joint features.

[0161] It should be noted that when determining the spatial state of the target object in the area to be measured, it may be necessary to predict the target object's target category, orientation angle, size, center offset, speed information, direction information, depth information and other prediction information. Since visual images usually have dense and rich semantic information, they have great advantages in object classification, size measurement and other aspects, while radar point cloud data usually lacks rich semantic information, but can be applied to spatial positioning and speed measurement and other aspects. Therefore, the target detection device can group the prediction information of the target object that needs to be predicted. For example, the image features extracted based on the visual image can be used to predict the target object's target category, orientation angle, size, center offset and other information. Based on the joint features of the image features and the radar data, the speed information and direction information of the target object can be predicted. In this way, the respective advantages of the acquisition data of the image acquisition device and the acquisition data of the radar can be fully utilized, and the accuracy of the prediction can be improved.

[0162] Optionally, the target detection device may perform the following steps S21 to S24 to determine the spatial state of the target object in the area to be detected based on the image features and the joint features.

[0163] S21. The target detection device determines one or more of the target category, orientation angle, size, and center offset of the target object in the detection area according to the image features.

[0164] Among them, target categories may include vehicles, people, trees, and buildings.

[0165] The heading angle refers to the angle formed by rotating the due north or due south direction as the starting direction to the target direction line of the target object with the position of the target object as the center. The target direction line can point to the moving direction of the target object.

[0166] like Fig.10 As shown, the target detection device can predict the information of the acquired image features through the first prediction module 101 to obtain one or more of the target category, orientation angle, size, and center offset of the target object. The first prediction module 101 may include four first submodules with the same structure, which respectively output the above-mentioned target category, orientation angle, size, or center offset.

[0167] like Fig.11 As shown in (a) in FIG. 1 , a first submodule of the first prediction module 101 may have a structure that is a combination of a 3×3 convolution kernel, a linear rectification unit, and a 1×1 convolution kernel.

[0168] S22. The target detection device determines the speed information and direction information of the target object in the detection area according to the joint features.

[0169] like Fig.10 As shown, the target detection device can perform information prediction on the acquired joint features through the second prediction module 102 to obtain the speed information and direction information of the target object. Fig.11 As shown in (b) in FIG. 1 , the structure adopted by the second prediction module 102 may be a combination of a 1×1 convolution kernel, a linear rectification unit, a 1×1 convolution kernel, a linear rectification unit, and a 1×1 convolution kernel.

[0170] S23. The target detection device determines the depth information of the target object in the area to be detected according to the image features and the joint features.

[0171] The depth information of the target object includes the distance between the target object and the target detection device.

[0172] like Fig.10 As shown, the target detection device can perform information prediction on the acquired joint features through the third prediction module 103 to obtain the speed information and direction information of the target object. Fig.11As shown in (c) in the figure, the structure adopted by the third prediction module 103 can be a combination of the first prediction module 101 and the second prediction module 102. The third prediction module 103 can input the image features into one or more first submodules based on the structure of the first prediction module 101, and input the joint features into the second submodule based on the structure of the second prediction module 101, so that the third prediction module 103 can further combine the data output by each first submodule with the data output by the second submodule based on the preset custom weights of various data to obtain the speed information and direction information of the target object. In addition, the combination processing may include a calculation method of adding one by one, multiplying one by one, or combining one by one with one by one.

[0173] S24. The target detection device determines the spatial state of the target object in the detection area according to one or more of the target category, orientation angle, size, center offset, speed information, direction information and depth information of the target object.

[0174] like Fig.10 As shown, the target detection device can determine the spatial state of the target object by analyzing one or more of the target category, orientation angle, size, center offset, speed information, direction information and depth information through the three-dimensional target detection module 104.

[0175] Optionally, the target detection device can perform detection based on each image feature and the radar feature corresponding to each image feature, and project the target object into a three-dimensional spatial expression of the area to be detected based on the spatial state obtained by the detection, so as to display the position and motion state of the target object, that is, the three-dimensional spatial state of the target object.

[0176] Based on the above embodiment, the method performs feature association between the visual image and the point cloud data, and then extracts features from the point cloud data according to the association results to generate radar features corresponding to the image features. Finally, the two features are fused to detect the spatial state of the target object in the area to be detected. In this way, on the one hand, the method does not lose the three-dimensional spatial information embodied by the radar point cloud data during the processing of the visual image and the point cloud data, thereby improving the accuracy of the cross-modal sensor data association between the visual image and the point cloud data. On the other hand, while detecting the target object in the area to be detected, the three-dimensional spatial state of the target object can also be estimated.

[0177] The above mainly introduces the solution provided by the embodiment of the present application from the perspective of the method. In order to achieve the above functions, it includes hardware structures and / or software modules corresponding to the execution of each function. It should be easy to realize that the technical goals in this field are combined with the units and algorithm steps of each example described in the embodiments disclosed in this article, and the present application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Professional and technical goals can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0178] like Fig.12 , which is a schematic diagram of the structure of a target detection device 300 provided in an embodiment of the present application. The target detection device 300 includes an acquisition module 301 , a feature association module 302 , a feature extraction module 303 and a target detection module 304 .

[0179] The acquisition module 301 is used to acquire the visual image and point cloud data of the area to be measured.

[0180] The feature association module 302 is used to associate one or more image features extracted based on the visual image with the point cloud data to obtain one or more radar reflection points associated with each image feature.

[0181] The feature extraction module 303 is used to determine the radar feature corresponding to each image feature according to one or more radar reflection points associated with each image feature.

[0182] The target detection module 304 is used to detect the spatial state of the target object in the detection area according to each image feature and the radar feature corresponding to each image feature. The target object includes a human body and / or an object.

[0183] In a possible implementation, the feature association module 302 is specifically used to: for a first image feature among one or more image features, determine, based on the first image feature and point cloud data, an association probability between the first image feature and each radar reflection point; and determine a radar reflection point whose association probability is greater than or equal to a preset probability as a radar reflection point associated with the first image feature.

[0184] In another possible implementation, the feature association module 302 is specifically used to: determine the relative position relationship between the first image feature and each radar reflection point in the point cloud data; determine the depth information of the first image feature and the depth information of each radar reflection point in the point cloud data; and determine the association probability of the first image feature and each radar reflection point based on the relative position relationship, the depth information of the first image feature and the depth information of each radar reflection point in the point cloud data.

[0185] In another possible implementation, the association probability satisfies the following relationship:

[0186]

[0187] Where d is the depth information of the radar reflection point, μ d is the depth mean in the depth information of the first image feature, σ d is the depth standard deviation in the depth information of the first image feature, and the viewing cone is used to indicate a spatial area determined based on the positions of the target image frame and the image acquisition device corresponding to the first image feature.

[0188] In another possible implementation, the target detection module 304 is specifically used to: determine, for a first image feature among one or more image features, a joint feature of the first image feature and a radar feature corresponding to the first image feature, and determine a spatial state of a target object in the area to be measured based on the first image feature and the joint feature.

[0189] In another possible implementation, the target detection module 304 is specifically used to: determine one or more of the target category, orientation angle, size, and center offset of the target object in the measured area based on the first image feature; determine the speed information and direction information of the target object in the measured area based on the joint feature; determine the depth information of the target object in the measured area based on the first image feature and the joint feature; determine the spatial state of the target object based on one or more of the target category, orientation angle, size, center offset, speed information, direction information, and depth information of the target object in the measured area.

[0190] In another possible implementation, the target detection module 304 is specifically used to: input the first image feature into the first prediction module to obtain one or more of the target category, orientation angle, size, and center offset of the target object in the test area; wherein the first prediction module includes one or more first sub-modules, a first sub-module includes a 3×3 convolution kernel, a linear rectification unit, and a 1×1 convolution kernel, and a first sub-module is used to output the target category, orientation angle, size, or center offset of the target object in the test area.

[0191] In another possible implementation, the target detection module 304 is further specifically used to: input the joint features into a second prediction module to obtain speed information and direction information of the target object in the measured area; wherein the second prediction module includes a 1×1 convolution kernel, a linear rectification unit, a 1×1 convolution kernel, a linear rectification unit and a 1×1 convolution kernel.

[0192] In another possible implementation, the target detection module 304 is further specifically used to: input the first image feature and the joint feature into a third prediction module to obtain depth information of the target object in the detection area; wherein the third prediction module includes one or more first submodules and second submodules, the first submodule includes a 3×3 convolution kernel, a linear rectification unit and a 1×1 convolution kernel, and the second submodule includes a 1×1 convolution kernel, a linear rectification unit, a 1×1 convolution kernel, a linear rectification unit and a 1×1 convolution kernel. State.

[0193] For the specific description of the above optional methods, please refer to the above method embodiments, which will not be repeated here. In addition, the explanation of any object detection device 300 provided above and the description of the beneficial effects can refer to the above corresponding method embodiments, which will not be repeated here.

[0194] As an example, combining Figure 3 The functions implemented by the acquisition module 301 of the target detection device 300 can be Figure 3 The communication line 220 in is implemented, but is certainly not limited to this.

[0195] Those skilled in the art should easily realize that, in combination with the units and algorithm steps of each example described in the embodiments disclosed herein, the present application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0196] It should be noted that Fig.12 The division of modules in the example is schematic and is only a logical function division. There may be other division methods in actual implementation. For example, two or more functions may be integrated into one processing module. The above integrated modules may be implemented in the form of hardware or software function modules.

[0197] The present application also provides a computer-readable storage medium including computer-executable instructions, which, when executed on a computer, enable the computer to execute any one of the methods provided in the above embodiments. For example, Figure 4One or more features of S101 to S104 may be undertaken by one or more computer-executable instructions stored in the computer-readable storage medium.

[0198] The embodiment of the present application also provides a computer program product including computer-executable instructions, which, when executed on a computer, enables the computer to execute any one of the methods provided in the above embodiments.

[0199] An embodiment of the present application also provides a chip, including: a processor and an interface, wherein the processor is coupled to a memory via the interface, and when the processor executes a computer program or a computer execution instruction in the memory, any one of the methods provided in the above embodiments is executed.

[0200] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using a software program, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When loading and executing computer execution instructions on a computer, the process or function according to the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. Computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, computer instructions can be transmitted from a website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (digital subscriber line, DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, server or data center. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server, data center, etc. that contains one or more servers that can be integrated with a medium. The available medium may be a magnetic medium (eg, a floppy disk, a hard disk, a magnetic tape), an optical medium (eg, a DVD), or a semiconductor medium (eg, a solid state disk (SSD)).

[0201] Although the present application has been described in conjunction with specific features and embodiments thereof, it is obvious that various modifications and combinations may be made thereto without departing from the spirit and scope of the present application. Accordingly, this specification and the drawings are merely exemplary illustrations of the present application as defined by the appended claims, and are deemed to have covered any and all modifications, variations, combinations or equivalents within the scope of the present application. Obviously, those skilled in the art may make various modifications and variations to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

[0202] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions within the technical scope disclosed in the present application should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A target detection method, characterized in that: The method comprises: Acquire a visual image and point cloud data of the area to be measured, wherein the point cloud data includes one or more radar reflection points; For a first image feature among the one or more image features, determining, according to the first image feature and the point cloud data, an association probability between the first image feature and each radar reflection point; Determining the radar reflection point whose association probability is greater than or equal to a preset probability as the radar reflection point associated with the first image feature; Determining a radar feature corresponding to each image feature according to the one or more radar reflection points associated with each image feature; Detecting a spatial state of a target object in the detection area according to each image feature and a radar feature corresponding to each image feature, wherein the target object includes a human body and / or an object; The association probability satisfies the following relationship: Wherein, d is the depth information of the radar reflection point, μ d is the depth mean in the depth information of the first image feature, σ d is the depth standard deviation in the depth information of the first image feature, and the viewing cone is used to indicate the spatial area determined based on the position of the target image frame corresponding to the first image feature and the image acquisition device.

2. The method according to claim 1, characterized in that: The determining, according to the first image feature and the point cloud data, the association probability between the first image feature and each radar reflection point includes: Determining a relative positional relationship between the first image feature and each radar reflection point in the point cloud data; Determining depth information of the first image feature and depth information of each radar reflection point in the point cloud data; The association probability between the first image feature and each radar reflection point is determined according to the relative position relationship, the depth information of the first image feature, and the depth information of each radar reflection point in the point cloud data.

3. The method according to claim 1 or 2, characterized in that: The detecting the spatial state of the target object in the detection area according to each image feature and the radar feature corresponding to each image feature comprises: For a first image feature among the one or more image features, determining a joint feature of the first image feature and a radar feature corresponding to the first image feature; The spatial state of the target object in the area to be detected is determined according to the first image feature and the joint feature.

4. The method according to claim 3, characterized in that The determining the spatial state of the target object in the area to be detected according to the first image feature and the joint feature includes: Determine one or more of a target category, an orientation angle, a size, and a center offset of a target object in the area to be detected according to the first image feature; Determine the speed and direction information of the target object in the detection area according to the joint features; Determine depth information of a target object in the area to be measured according to the first image feature and the joint feature; The spatial state of the target object is determined according to one or more of the target category, orientation angle, size, center offset, speed information, direction information and depth information of the target object in the area to be detected.

5. The method according to claim 4, characterized in that Determining one or more of a target category, a direction angle, a size, and a center offset of the target object in the area to be detected according to the first image feature includes: Inputting the first image feature into a first prediction module to obtain one or more of a target category, a direction angle, a size, and a center offset of a target object in the area to be detected; Among them, the first prediction module includes one or more first sub-modules, a first sub-module includes a 3×3 convolution kernel, a linear rectification unit and a 1×1 convolution kernel, and the first sub-module is used to output the target category, orientation angle, size or center offset of the target object in the test area.

6. The method according to claim 4, characterized in that Determining the speed information and direction information of the target object in the area to be detected according to the joint feature includes: Inputting the joint feature into a second prediction module to obtain speed information and direction information of the target object in the detection area; The second prediction module includes a 1×1 convolution kernel, a linear rectification unit, a 1×1 convolution kernel, a linear rectification unit and a 1×1 convolution kernel.

7. The method according to claim 4, characterized in that The determining the depth information of the target object in the area to be detected according to the first image feature and the joint feature includes: Inputting the first image feature and the joint feature into a third prediction module to obtain depth information of the target object in the area to be detected; Among them, the third prediction module includes one or more first sub-modules and second sub-modules, the first sub-module includes a 3×3 convolution kernel, a linear rectification unit and a 1×1 convolution kernel, and the second sub-module includes a 1×1 convolution kernel, a linear rectification unit, a 1×1 convolution kernel, a linear rectification unit and a 1×1 convolution kernel.

8. A target detection device, characterized in that: The device comprises: An acquisition module is used to acquire visual images and point cloud data of the area to be tested; a feature association module, configured to determine, for a first image feature among the one or more image features, an association probability between the first image feature and each radar reflection point according to the first image feature and the point cloud data, and determine a radar reflection point having an association probability greater than or equal to a preset probability as a radar reflection point associated with the first image feature; a feature extraction module, configured to determine a radar feature corresponding to each image feature according to one or more radar reflection points associated with each image feature; A target detection module, used to detect the spatial state of a target object in the detection area according to each image feature and a radar feature corresponding to each image feature, wherein the target object includes a human body and / or an object; The association probability satisfies the following relationship: Wherein, d is the depth information of the radar reflection point, μ d is the depth mean in the depth information of the first image feature, σ d is the depth standard deviation in the depth information of the first image feature, and the viewing cone is used to indicate the spatial area determined based on the position of the target image frame corresponding to the first image feature and the image acquisition device.

9. The device according to claim 8, characterized in that The feature association module is specifically used for: Determining a relative positional relationship between the first image feature and each radar reflection point in the point cloud data; Determining depth information of the first image feature and depth information of each radar reflection point in the point cloud data; Determining an association probability between the first image feature and each radar reflection point according to the relative position relationship, the depth information of the first image feature, and the depth information of each radar reflection point in the point cloud data; The target detection module is specifically used for: For a first image feature among the one or more image features, determining a joint feature of the first image feature and a radar feature corresponding to the first image feature; Determine the spatial state of the target object in the area to be detected according to the first image feature and the joint feature; The target detection module is specifically used for: Determine one or more of a target category, an orientation angle, a size, and a center offset of a target object in the area to be detected according to the first image feature; Determine the speed and direction information of the target object in the detection area according to the joint features; Determine depth information of a target object in the area to be measured according to the first image feature and the joint feature; Determine the spatial state of the target object according to one or more of the target category, orientation angle, size, center offset, speed information, direction information, and depth information of the target object in the detection area. The target detection module is specifically used for: Inputting the first image feature into a first prediction module to obtain one or more of a target category, a direction angle, a size, and a center offset of a target object in the area to be detected; The first prediction module includes one or more first submodules, one first submodule includes a 3×3 convolution kernel, a linear rectifier unit and a 1×1 convolution kernel, and the one first submodule is used to output the target category, orientation angle, size or center offset of the target object in the area to be detected; The target detection module is further specifically used for: Inputting the joint feature into a second prediction module to obtain speed information and direction information of the target object in the detection area; The second prediction module includes a 1×1 convolution kernel, a linear rectification unit, a 1×1 convolution kernel, a linear rectification unit, and a 1×1 convolution kernel; The target detection module is further specifically used for: Inputting the first image feature and the joint feature into a third prediction module to obtain depth information of the target object in the area to be detected; Among them, the third prediction module includes one or more first sub-modules and second sub-modules, the first sub-module includes a 3×3 convolution kernel, a linear rectification unit and a 1×1 convolution kernel, and the second sub-module includes a 1×1 convolution kernel, a linear rectification unit, a 1×1 convolution kernel, a linear rectification unit and a 1×1 convolution kernel.

10. An electronic device, characterized in that: The electronic device comprises a memory and a processor; the memory and the processor are coupled; the memory is used to store computer program code, and the computer program code comprises computer instructions; When the processor executes the computer instructions, the electronic device executes the target detection method as described in any one of claims 1 to 7.

11. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and when the computer instructions are executed on an electronic device, the electronic device executes the target detection method according to any one of claims 1 to 7.

Images