A target detection method and system based on pixelated millimeter wave radar point cloud

Abstract

The application discloses a target detection method and system based on pixelated millimeter wave radar point clouds, aligns and projects a single frame of millimeter wave radar point clouds on an image plane to obtain a point cloud projection graph of the millimeter wave radar point clouds on the image plane, combines multiple frames of the point cloud projection graphs received within a time t on the same picture, expands the point cloud projection graph along a transverse direction according to a radar angle resolution of a point to obtain a pixelated point cloud image, and uses a Darknet23 network to perform target detection on the pixelated point cloud image. The application breaks through the limitation of target detection in a complex scene and provides a reliable solution for realizing reliable and accurate human target detection.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of target detection, and particularly relates to a target detection method and system based on pixelized millimeter wave radar point cloud. BACKGROUND

[0002] The rapid development of deep learning algorithms has promoted the vigorous development of the field of target detection. Human target detection is crucial for various real-world applications, such as surveillance systems and autonomous driving systems. Human target detection systems are required to accurately and quickly detect pedestrian targets in complex scenes. However, traditional vision-based methods have some limitations, such as failure in low-light and occlusion conditions and privacy security issues. Therefore, radar sensors represented by millimeter wave radars provide a solution to break through the above limitations.

[0003] Millimeter wave radars are radars that work in the millimeter wave band, have all-weather and all-day working ability, and have certain obstacle penetration perception ability. Millimeter wave radars not only can perceive point cloud data of the three-dimensional world, but also can detect the speed and signal strength information of the corresponding points, and have rich information dimensions. Existing millimeter wave radar target detection methods mainly assist image detection methods, and the perception ability of radar point cloud is not deep enough. Therefore, it is worth considering to use mature image detection networks after expanding radar point cloud to an image. SUMMARY

[0004] The technical problem to be solved by the application is to provide a target detection method and system based on pixelized millimeter wave radar point cloud to solve the technical problem of human target detection in low-light, occlusion and other conditions by traditional vision methods, enhance the representation ability of radar point cloud for target detection tasks, and further enhance the perception and recognition ability of millimeter wave radar nodes.

[0005] The application adopts the following technical solutions:

[0006] A target detection method based on pixelized millimeter wave radar point cloud comprises the following steps:

[0007] S1, aligning and projecting a single frame of millimeter wave radar point cloud on an image plane to obtain a point cloud projection image of the millimeter wave radar point cloud on the image plane;

[0008] S2, merging multiple frames of point cloud projection images received within a time t from step S1 onto the same image;

[0009] S3, expanding the point cloud projection image obtained in step S2 along the horizontal direction according to the radar angle resolution of the point to obtain a pixelized point cloud image;

[0010] S4, using the Darknet23 network to perform target detection on the pixelated image of the point cloud obtained in step S3.

[0011] Specifically, in step S1, the projection point P of any point P in the millimeter wave radar point cloud on the image is P uv .

[0012]

[0013] wherein u, v are points in the pixel coordinate system of the image, K and Γ are the internal and external parameter matrices of the camera, and x, y, z are the spatial three-dimensional coordinates of the point P.

[0014] Specifically, in step S2, t is 0.1-0.2s.

[0015] Specifically, step S3 is specifically:

[0016] S301, according to the principle of millimeter wave radar angle resolution, the range of the image point cloud points is expanded point by point according to the incident angle, and the figure silhouette profile is restored;

[0017] S302, according to the range of the image point cloud points expanded in step S301, R, G, and B three-channel color assignment is performed on all image pixel points in the range.

[0018] Further, in step S301, the received millimeter wave point is a point S(x0, y0, z0) in the camera coordinate system, and the point S is projected and aligned to a point S uv in the image pixel coordinate system, while the millimeter wave point cloud is received, the point at angle θ has an angle error θ res , according to the pixel error range of each point, the expansion is carried out point by point in the pixel U direction, and the compensation of 1 / 16 in the pixel V direction is carried out at the same time.

[0019] Further, the angle error θ res is:

[0020]

[0021] wherein λ is the wavelength of the millimeter wave radar, N is the number of virtual antenna arrays, and l is the spacing of adjacent antennas.

[0022] The point S(x, y, z) in the camera coordinate system and the projected image coordinate system point satisfy:

[0023]

[0024] wherein u, v are points in the pixel coordinate system of the image, f x x is the focal length in the x-axis direction, f y y is the focal length in the y-axis direction, and z is the depth of the target point cloud. cx is the x-coordinate of the principal point of the camera on the image plane, c y is the y-coordinate of the principal point of the camera on the image plane.

[0025] Further, in step S302, the distance and speed information of the extended image point cloud points are mapped to 0-255 image color channel values according to the maximum value set by the system, as R and G color channels; the density information of the point cloud points is taken as the third channel B dimension to extend the image point cloud points.

[0026] Further, if multiple points in the frame are at the same pixel position, the distance and speed information of the point with the maximum distance is placed in the R and G channels, and the B channel is the density of the point corresponding to the pixel position.

[0027] Specifically, in step S4, the number of channels of the Darknet23 network is 1 / 8 of the Darknet53 network, and the number of feature extraction layers is (1, 2, 2, 2, 1); in each training iteration, the loss function includes: the deviation of the prediction results of the two networks from the true values; and the mean square error between the prediction results of the two networks.

[0028] In a second aspect, an embodiment of the present application provides a target detection system based on pixelated millimeter wave radar point cloud, comprising:

[0029] The collection module aligns and projects a single frame of millimeter wave radar point cloud on an image plane to obtain a point cloud projection image of the millimeter wave radar point cloud on the image plane.

[0030] The projection module combines multiple frames of point cloud projection images received within a time t from the collection module on the same picture.

[0031] The extension module extends the point cloud projection image obtained by the projection module along the horizontal direction according to the radar angle resolution of the point to obtain a pixelated point cloud image.

[0032] The detection module uses a Darknet23 network to perform target detection on the pixelated point cloud image obtained by the extension module.

[0033] Compared with the prior art, the present application has at least the following beneficial effects:

[0034] A target detection method based on pixelated millimeter wave radar point cloud, by millimeter wave point cloud and image coordinate alignment technology, millimeter wave point cloud frame merging technology, the characteristics of millimeter wave radar points are comprehensively utilized in space and time information, so that the millimeter wave radar points can be integrated into the existing mature image target detection method. At the same time, the image point cloud point expansion technology adopted by the present application expands and utilizes the rich feature dimension of the millimeter wave radar point, so that the millimeter wave radar point cloud point can simulate the human form collected in the image, and fit the image human silhouette; The pixelated data set of the collected millimeter wave point cloud can be directly trained using the existing network architecture of the image target detection method, such as the YOLO series target detection network, to realize better perception detection of the human in the low light and shielding scene. At the same time, unlike image perception technology, the perception method using millimeter wave radar is also a more effective privacy protection scheme, which has wide application scenarios in mobile computing, smart city and other aspects.

[0035] Further, step S3 aims to further improve the reliability and accuracy of the target detection method based on pixelated point cloud image, by expanding the range of image point cloud points according to the angle resolution of the millimeter wave radar based on the point cloud projection image, the restoration of the human silhouette contour is realized. At the same time, the rich multi-dimensional information (speed, distance, density) of the point cloud is utilized, and three-channel color assignment is performed, which better identifies the position of the human and distinguishes different humans.

[0036] Further, step S301 considers the angle error of the point cloud data caused by the position, and reflects it in the pixelated image, thereby improving the expression accuracy of the human target in the image; Specifically, the target in front of the radar has good angle resolution, so the expansion is less; the farther the target deviates from the front of the radar, the worse the angle resolution, and the more the expansion. Through this method, the distribution form of the target in the image is more realistically reproduced, laying a solid foundation for subsequent target detection.

[0037] Further, step S302 aims to further enrich the information of the point cloud pixelated image and provide more recognizable visual features; by mapping the distance and speed dimension information as R, G two color channel values, and the density information as B channel value, the multi-dimensional attributes of the point cloud are presented. Through the color marking method, the point cloud pixelated image is more hierarchical; this visual display method makes the target point cloud with different speeds and positions in the image present different colors, shows the distribution characteristics of the target point cloud density in space, provides more accurate position information, and provides more rich features for subsequent target detection.

[0038] Further, the target detection method based on the Darknet23 network is introduced in step S4, aiming to efficiently complete the target detection task under a relatively lightweight network structure. By reducing the channel number and feature extraction layer number of the network, the Darknet23 network reduces the consumption of computing resources while maintaining the detection performance. The design of the loss function further improves the robustness of the network, and the deviation of the prediction results of the two networks from the true value and the mean square error between the prediction results of the two networks are considered, which helps to improve the stability and accuracy of target detection. The design of this step makes the whole system more efficient and reliable in the target detection process.

[0039] It can be understood that the beneficial effects of the above-mentioned second aspect can be referred to the related description in the above-mentioned first aspect, which will not be repeated here.

[0040] In summary, the present application based on pixelated millimeter wave radar point cloud solves the problem of human target detection under complex conditions such as low light and occlusion by traditional vision methods. Through point cloud projection, merging, expansion and deep learning detection, efficient and accurate detection of human targets in various environments is achieved. This technology has wide application prospects in intelligent security, autonomous driving, emergency rescue and other fields, breaking through the limitations of target detection in complex scenes, and providing a reliable solution for reliable and accurate human target detection.

[0041] The technical solutions of the present application will be further described in detail below with the help of the accompanying drawings and embodiments. BRIEF DESCRIPTION OF DRAWINGS

[0042] Figure 1 It is a schematic diagram for the projection alignment of millimeter wave radar point cloud to the image coordinate system of the camera view;

[0043] Figure 2 It is a schematic diagram for the merging of millimeter wave point cloud frames;

[0044] Figure 3 It is a schematic diagram for the expansion of image point cloud points;

[0045] Figure 4 It is an image point cloud point expansion, wherein (a) is the original frame merging result, (b) is the result after image point cloud point expansion, and (c) is the corresponding camera view of the person;

[0046] Figure 5 It is an effect diagram of human target detection and recognition in different scenes;

[0047] Figure 6 It is a block diagram of a chip provided by the present application according to an embodiment. DETAILED DESCRIPTION

[0048] Clearly, the described embodiments are only a part of the embodiments of the present application, but not all the embodiments. Based on the embodiments of the present application, all the other embodiments obtained by a person of ordinary skill in the art without creative effort are within the protection scope of the present application.

[0049] In the description of the present application, it should be understood that the terms "comprising" and "including" indicate the presence of the described features, integers, steps, operations, elements, and / or components, but do not exclude one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0050] It should also be understood that the terms used in the present application specification are only for the purpose of describing particular embodiments and are not intended to limit the present application. As used in the present application specification and the appended claims, the singular forms "a", "an" and "the" are intended to include the plural forms unless the context clearly indicates otherwise.

[0051] It should be further understood that the term "and / or" used in the present application specification and the appended claims means one or more of the associated listed items as well as all possible combinations thereof and includes these combinations, for example, A and / or B can mean A alone, A and B together, and B alone. In addition, the character " / " in the present application generally represents an "or" relationship between the front and rear associated objects.

[0052] It should be understood that although the terms first, second, third, etc. can be used in the embodiments of the present application to describe preset ranges, etc., these preset ranges should not be limited to these terms. These terms are only used to distinguish the preset ranges from each other. For example, the first preset range can also be referred to as the second preset range, and similarly, the second preset range can also be referred to as the first preset range without departing from the scope of the embodiments of the present application.

[0053] Depending on the context, the word "if" as used herein can be interpreted to mean "when" or "while" or "in response to determining" or "in response to detecting". Similarly, the phrase "if determined" or "if detecting (a stated condition or event)" can be interpreted to mean "when determined" or "in response to determining" or "when detecting (a stated condition or event)" or "in response to detecting (a stated condition or event)", depending on the context.

[0054] Various structural diagrams according to the disclosed embodiments of the present application are shown in the drawings. These diagrams are not drawn to scale, in which certain details are exaggerated for clarity of presentation and may omit certain details. The shapes of various regions, layers shown in the drawings and their relative sizes and positional relationships may deviate in actuality due to manufacturing tolerances or technical limitations, and regions / layers with different shapes, sizes, relative positions can be additionally designed according to actual needs by those skilled in the art.

[0055] The present application provides a target detection method based on pixelated millimeter wave radar point cloud, which comprehensively utilizes the characteristics of millimeter wave radar points in space and time information through millimeter wave point cloud and image coordinate alignment technology, millimeter wave point cloud frame merging technology, so that the millimeter wave radar points can be integrated into the existing mature image target detection method. At the same time, the image point cloud point expansion technology used in the present application expands and utilizes the rich feature dimension of millimeter wave radar points, so that the millimeter wave radar point cloud points can simulate the human form collected in the image and fit the image human silhouette. The collected millimeter wave point cloud pixelated dataset can be directly trained using the existing network architecture of image target detection method, such as YOLO series target detection series network, to achieve better perception detection of human in low light and occlusion scene. At the same time, unlike image perception technology, the perception method using millimeter wave radar is a more effective privacy protection scheme, which has wide application scenarios in mobile computing, smart city, etc.

[0056] The present application provides a target detection method based on pixelated millimeter wave radar point cloud, which comprehensively utilizes the characteristics of millimeter wave radar points in space and time information through millimeter wave point cloud and image coordinate alignment technology, millimeter wave point cloud frame merging technology, so that the millimeter wave radar points can be integrated into the existing mature image target detection method. At the same time, the image point cloud point expansion technology used in the present application expands and utilizes the rich feature dimension of millimeter wave radar points, so that the millimeter wave radar point cloud points can simulate the human form collected in the image and fit the image human silhouette. The collected millimeter wave point cloud pixelated dataset can be directly trained using the existing network architecture of image target detection method, such as YOLO series target detection series network, to achieve better perception detection of human in low light and occlusion scene. At the same time, unlike image perception technology, the perception method using millimeter wave radar is a more effective privacy protection scheme, which has wide application scenarios in mobile computing, smart city, etc.

[0057] S1, aligning millimeter wave point cloud and image coordinates;

[0058] On the collected millimeter wave radar point cloud frame, the millimeter wave radar point cloud is projected onto the coordinates of the camera image with the same view angle by using the camera internal and external parameter matrix, image length and width parameters, etc. The alignment technology of millimeter wave radar space points and image pixel points facilitates the generation of human target frame by the existing image detector to provide real values for radar point recognition.

[0059] Please refer to Figure 1 The collected millimeter wave point cloud is first projected and aligned into the image coordinate system of the camera view angle.

[0060] For any point P(x, y, z) in the millimeter wave radar point cloud, x, y, z are its three-dimensional space coordinates, and the projection point of point P on the image is P uv (u, v), uv is the pixel coordinate system point in the image, and has:

[0061]

[0062]

[0063]

[0064] Where K and Γ are the intrinsic and extrinsic parameter matrices of the camera.

[0065] The original point cloud was mapped and aligned to the camera's image coordinate system.

[0066] In this invention, the camera and millimeter-wave radar are placed facing the same direction, with a distance of 5 cm between them. After aligning the radar and camera viewpoints, the resulting image point cloud is used as the basis for subsequent processing.

[0067] S2, millimeter-wave point cloud frame merging;

[0068] Several point cloud frames captured within a short period (within 0.2 seconds) are merged together and placed in the same spatial coordinate system. The merged millimeter-wave point cloud frames can largely avoid the inaccuracy of target detection results caused by random point cloud reflections; at the same time, not merging too many frames is to consider that the detected human target has a certain speed, and the same human figure will have a non-negligible positional difference between many frames.

[0069] Please see Figure 2 Multiple frames of millimeter-wave radar point cloud data were merged into a single frame. The merged millimeter-wave point cloud data was then used... Figure 1 The image is characterized by the projection alignment. Within a short time frame of frame merging, the displacement of the target object relative to the radar viewpoint is small. Frame merging can reduce the detection error caused by the randomness of the radar point cloud.

[0070] S3, Image point cloud point augmentation;

[0071] S301, Expanding the point range of the image point cloud;

[0072] Based on the principle of millimeter-wave radar angular resolution, the image point cloud is expanded point by point according to the incident angle to restore the silhouette outline of the person. Radar point cloud points with different incident angles have different expansion ranges. That is, the point cloud points that are further away from the front have a larger expansion range, which can better solve the difference in the representation of the person caused by the resolution error of the point cloud at different angles.

[0073] Specifically, the augmentation of image point cloud points is mainly applied to the horizontal direction of the image, with less compensation used in the vertical direction.

[0074] The received millimeter-wave point is located at point S(x0, y0, z0) in the camera coordinate system, and point S is projected and aligned to point S in the image pixel coordinate system. uv However, when receiving millimeter-wave point clouds, points located at angle θ exhibit an angular error θ.res , satisfy:

[0075]

[0076] where λ is the wavelength of the millimeter wave radar, N is the number of virtual antenna arrays, and l is the spacing of adjacent antennas.

[0077] Therefore, for the point S in the figure, in the limit case, S l from the left (negative direction of the x-axis) or S r from the right (positive direction of the x-axis). Considering that S l and S r are located at different angles, different spatial distance errors and will be caused and will be reflected in the image coordinate system of the projection result and , causing pixel errors and

[0078] At the same time, the point S (x, y, z) in the camera coordinate system and the projected image coordinate system point satisfy:

[0079]

[0080] There are:

[0081]

[0082]

[0083] It can be seen that for any point in the image point cloud point, there is a measurable pixel error and in the image.

[0084] S302, image point cloud point color expansion.

[0085] According to the range of the image point cloud point expanded in step S301, R, G, and B three-channel color assignment is performed on all image pixel points in the range.

[0086] Specifically, in the expanded image point cloud points, the distance and speed two-dimensional information is contained, and by mapping the two-dimensional information to the image color channel value of 0-255 according to the maximum value set by the system, R and G two color channels are obtained; considering that the person point cloud points at different positions have different densities of reflection points, for example, the point cloud point density of the head and feet is small, and the point cloud point density of the body trunk is large; finally, the density information of the point cloud points is taken as the third channel B dimension to expand the image point cloud points.

[0087] Compared with the equal expansion method (without considering the angle resolution), the method of the application expands more in the part with larger angle. This is because the part with smaller angle is closer to the front of the radar, has more accurate angle estimation, and needs to expand a smaller range to compensate. At the same time, as long as the radar receiving point is not located at the front of the radar, the positive and negative ranges of the expansion are not equal.

[0088] Please refer to Figure 4 Compared with the unexpanded graph (a), the graph (b) has three advantages:

[0089] 1) The expanded area can better cover the contour of the person than the original points, simulating the silhouette of the person.

[0090] 2) The expanded image has color, which can more easily distinguish targets from different distances and speeds, for example, the direction and depth of the person walking in the picture are very different, resulting in different R and G channel colors.

[0091] 3) Each human body itself has different reflection signal strengths, resulting in different reflection point densities of different parts of the body, thereby causing different B channel colors.

[0092] The above three advantages are more conducive to the detection of human targets by the image detector. In particular, if multiple points in the frame are at the same pixel position, the distance and speed information of the point with the largest distance is placed in the R and G channels, and the B channel is the density of the point at the pixel position.

[0093] S4, target detection four processes.

[0094] Based on the mutual learning strategy, a Darknet23 network is used to detect the pixelated radar image. The Darknet23 backbone network is a Darknet53 network with reduced channels and feature extraction layers.

[0095] Compared with the original Darknet53 network, the channel number of the Darknet23 network is reduced to 1 / 8 of the original, and the feature extraction layer number is reduced from (1, 2, 8, 8, 4) of the original to (1, 2, 2, 2, 1). After the double reduction of the channel and the feature extraction layer, the Darknet23 is more lightweight, more suitable for coarse-grained pixelated radar images, and effectively avoids the overfitting of network training. The mutual learning method, that is, using two Darknet23 networks for synchronous training.

[0096] In each training iteration, the loss function includes:

[0097] 1) the deviation of the prediction results of the two networks from the true value;

[0098] 2) the mean square error between the prediction results of the two networks.

[0099] The two networks learn from each other, improve the adaptability of the pixelated radar point cloud in different environments, and effectively improve the generalization ability of the network.

[0100] In another embodiment of the present application, a target detection system based on pixelated millimeter wave radar point cloud is provided, which can be used to implement the target detection method based on pixelated millimeter wave radar point cloud described above. Specifically, the target detection system based on pixelated millimeter wave radar point cloud includes a collection module, a projection module, an expansion module and a detection module.

[0101] The collection module projects a single frame of millimeter wave radar point cloud on the image plane to obtain a point cloud projection image of the millimeter wave radar point cloud on the image plane.

[0102] The projection module combines multiple frames of point cloud projection images received from the collection module within a time t into the same picture.

[0103] The expansion module expands the point cloud projection image obtained by the projection module along the horizontal direction according to the radar angle resolution of the point to obtain a pixelated point cloud image.

[0104] The detection module uses a Darknet23 network to perform target detection on the pixelated point cloud image obtained by the expansion module.

[0105] In another embodiment of the present application, a terminal device is provided, which comprises a processor and a memory, the memory being configured to store a computer program, the computer program comprising program instructions, and the processor being configured to execute the program instructions stored in the computer storage medium. The processor can be a central processing unit (CPU), and can also be other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gates or transistor logic, discrete hardware components, etc., which are the computing core and control core of the terminal, and are suitable for implementing one or more instructions, and are particularly suitable for loading and executing one or more instructions to implement a corresponding method flow or a corresponding function; the processor in the embodiments of the present application can be used for the operation of the target detection method based on the pixelized millimeter wave radar point cloud, comprising:

[0106] aligning and projecting a single-frame millimeter wave radar point cloud on an image plane to obtain a point cloud projection graph of the millimeter wave radar point cloud on the image plane; merging multiple frames of point cloud projection graphs received within a time t onto the same picture; expanding the point cloud projection graph along the horizontal direction according to the radar angle resolution of the point to obtain a pixelized point cloud image; and using a Darknet23 network to perform target detection on the pixelized point cloud image.

[0107] Please refer to Figure 6 , the terminal device is a chip, and the chip 600 of the embodiment comprises a processor 622, the number of which can be one or more, and a memory 632 configured to store a computer program executable by the processor 622. The computer program stored in the memory 632 can comprise one or more modules each corresponding to a set of instructions. In addition, the processor 622 can be configured to execute the computer program to perform the target detection method based on the pixelized millimeter wave radar point cloud described above.

[0108] In addition, the chip 600 can further comprise a power supply component 626 and a communication component 650, the power supply component 626 can be configured to perform power management of the chip 600, and the communication component 650 can be configured to implement communication of the chip 600, such as wired or wireless communication. In addition, the chip 600 can further comprise an input / output (I / O) interface 658. The chip 600 can operate based on an operating system stored in the memory 632.

[0109] In another embodiment of the present application, the present application also provides a storage medium, specifically a computer readable storage medium (Memory), which is a memory device in a terminal device, used for storing programs and data. It can be understood that the computer readable storage medium herein can include a built-in storage medium in the terminal device, and of course can also include an extended storage medium supported by the terminal device. The computer readable storage medium provides a storage space, which stores an operating system of the terminal. In addition, one or more instructions suitable for being loaded and executed by the processor are also stored in the storage space, and these instructions can be one or more computer programs (including program codes). It should be noted that the computer readable storage medium herein can be a high-speed RAM memory or a non-volatile memory such as at least one disk memory.

[0110] The one or more instructions stored in the computer readable storage medium can be loaded and executed by the processor to implement the corresponding steps of the target detection method based on the pixelated millimeter wave radar point cloud in the above embodiments; the one or more instructions in the computer readable storage medium are loaded and executed by the processor as follows:

[0111] Align and project the single-frame millimeter wave radar point cloud on the image plane to obtain a point cloud projection image of the millimeter wave radar point cloud on the image plane; merge the multiple frames of point cloud projection images received within the time t into the same image; expand the point cloud projection image along the horizontal direction according to the radar angle resolution of the point to obtain a pixelated point cloud image; and use the Darknet23 network to perform target detection on the pixelated point cloud image.

[0112] To make the objectives, technical solutions, and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments. The components of the embodiments of the present application described and shown in the drawings herein can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the claimed present application, but only represents selected embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by a person of ordinary skill in the art without creative labor fall within the scope of protection of the present application.

[0113] Please refer to Figure 1 In the projection process, if the millimeter wave point cloud falls outside the camera view range, it will be deleted, for example, the points in the red circle part in the figure.

[0114] Please refer toFigure 2 In a short time (three frames in the figure), the image point cloud points of the image points are recognized by the system. It can be found that compared with a frame of image point cloud points, the three frames of image point cloud points can better represent the details of the contour of the person to be detected.

[0115] Referring to Figure 3 , the received millimeter wave point is located at point S (x0, y0, z0) in the camera coordinate system. The point can be projected and aligned to point S uv in the image pixel coordinate system. While the millimeter wave point cloud is received, the point at angle θ has an angle error θ res . For the point S in the figure, in the limit case, S l may come from the left side (negative direction of the x-axis), or S r may come from the right side (positive direction of the x-axis). Considering that S l and S r are located at different angles, different spatial distance errors and will be caused and will be reflected in the image coordinates of the projection result and , causing pixel errors and The expansion range of the final image point cloud point is determined according to the pixel errors and .

[0116] Referring to Figure 4 , Figure 4 (a) is the result of original frame merging, Figure 4 (b) is the result of image point cloud point expansion, Figure 4 (c) is the person corresponding to the camera view, and the green frame is the true value of the target detection result.

[0117] Referring to Figure 5 , the first line is the working detection result of the present application, the second line is the RGB image detection result, and the third line is the depth image detection result. It can be found from the figure that compared with the method of the RGB camera and the depth camera, the millimeter wave radar method proposed by the present application can work in a dark and occluded scene, and is more universal.

[0118] From Figure 5It is found that the point cloud image method of the application can well represent the character in the case of outdoor, strong light, darkness and shielding, and the recognition result and accuracy are better than the result of RGB image and depth image. Among them, the sensor of the data set is Microsoft Azure Kinect camera and AWR1443BOOST millimeter wave radar of Texas Instruments. The backbone network of the trained network model is lightweight Darknet-53. The data set includes 25 different characters, a total of 13048 millimeter wave images and camera frames processed by alignment, 9172 frames are used for training, and 3876 frames are used for testing. The application is not limited to using the above-mentioned sensors.

[0119] In summary, the target detection method and system based on pixelized millimeter wave radar point cloud of the application can realize efficient and accurate human target detection in various environments by overcoming the limitations of traditional visual methods in low light, shielding and other complex conditions. The point cloud data generated by the millimeter wave radar signal is expanded according to the angle resolution of the point cloud, providing a reliable solution for target detection. Not only has wide application prospects in intelligent security, autonomous driving, emergency rescue and other fields, but also provides a new breakthrough for human target detection in low light, shielding and other complex scenes.

[0120] Those skilled in the art can clearly understand that, for the convenience and brevity of description, only the division of the above functional units and modules is exemplified, and in actual application, the above functions can be completed by different functional units and modules according to needs, that is, the internal structure of the device is divided into different functional units or modules to complete all or part of the functions described above. Each functional unit and module in the embodiment can be integrated in one processing unit, or each unit can exist physically, or two or more units can be integrated in one unit. The above integrated unit can be realized in the form of hardware or software. In addition, the specific name of each functional unit and module is only for easy distinction, and does not limit the protection scope of the application. The specific working process of the units and modules in the above system can refer to the corresponding process in the foregoing method embodiments, which will not be repeated here.

[0121] In the above embodiments, the description of each embodiment has its own emphasis, and the parts not described or recorded in a certain embodiment can be referred to the related description of other embodiments.

[0122] Those skilled in the art can understand that the units and algorithm steps of each example described in combination with the embodiments disclosed in the present application can be realized in electronic hardware or in combination of computer software and electronic hardware. Whether the functions are realized in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to realize the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0123] In the embodiments provided by the present application, it should be understood that the disclosed apparatus / terminal and method can be implemented in other ways. For example, the apparatus / terminal embodiments described above are merely schematic. The division of the modules or units is merely a logical function division, and there can be another division manner in actual implementation. For example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the displayed or discussed coupling or direct coupling or communication connection between the units can be indirect coupling or communication connection through some interface, device or unit, and can be electrical, mechanical or in other forms.

[0124] The units described as separate components can or can not be physically separate, and the components shown as units can or can not be physical units, i.e. they can be located in one place, or can be distributed on a plurality of network units. Part or all of the units can be selected according to actual needs to achieve the purpose of the embodiments.

[0125] In addition, each functional unit in each embodiment of the present application can be integrated into a processing unit, or each unit can exist physically independently, or two or more units can be integrated into one unit. The integrated unit can be realized in the form of hardware or in the form of a software functional unit.

[0126] The integrated module / unit, if realized in the form of a software function unit and sold or used as an independent product, can be stored in a computer-readable storage medium. Based on such understanding, all or part of the processes in the above-mentioned embodiment methods can also be completed by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of each method embodiment when executed by a processor. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or some intermediate forms, etc. The computer-readable medium can include any entity or device capable of carrying the computer program code, recording medium, U disk, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signal, telecommunication signal, and software distribution medium, etc. It should be noted that the computer-readable medium can include or exclude contents according to the requirements of legislation and patent practice in the jurisdiction, for example, in some jurisdictions, according to legislation and patent practice, the computer-readable medium does not include electrical carrier signals and telecommunication signals.

[0127] The present application is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present application. It should be understood that each flow and / or block in the flowcharts and / or block diagrams, and the combination of flows and / or blocks in the flowcharts and / or block diagrams can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing apparatus to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing apparatus generate a means for implementing the functions specified in the flowcharts and / or block diagrams. Figure 1 The functions specified in one or more flows and / or blocks Figure 1 The functions specified in one or more flows and / or blocks

[0128] These computer program instructions can also be stored in a computer-readable memory capable of guiding a computer or other programmable data processing apparatus to work in a specific manner, so that the instructions stored in the computer-readable memory produce a product including instruction means, which implements the functions specified in the flowcharts and / or block diagrams. Figure 1 The functions specified in one or more flows and / or blocks Figure 1 The functions specified in one or more flows and / or blocks

[0129] These computer program instructions can also be loaded into a computer or other programmable data processing devices, so that a series of operational steps are performed on the computer or other programmable data processing devices to generate a computer implemented process, so that the instructions executed on the computer or other programmable data processing devices provide a process for implementing the flowchart Figure 1 one flow or a plurality of flows and / or the functions specified in the block Figure 1 one block or a plurality of blocks.

[0130] The above is only to illustrate the technical idea of the present application, and cannot limit the protection scope of the present application. Any modification made according to the technical idea of the present application on the basis of the technical scheme falls within the protection scope of the claims of the present application.

Claims

1. A target detection method based on pixelated millimeter wave radar point cloud, characterized in that, The method comprises the following steps: S1, aligning and projecting a single-frame millimeter wave radar point cloud on an image plane to obtain a point cloud projection graph of the millimeter wave radar point cloud on the image plane; S2, merging multiple frames of the point cloud projection graph from step S1 received within a time t onto the same picture; S3, expanding the point cloud projection graph obtained in step S2 along the transverse direction according to the radar angle resolution of the point to obtain a point cloud pixelized image, specifically: S301、According to the principle of millimeter wave radar angle resolution, the range of the image point cloud point is expanded point by point according to the incident angle, and the figure silhouette profile is restored. The received millimeter wave point is located in the point S (x0, y0, z0) in the camera coordinate system, and the point S is projected and aligned to the point S in the image pixel coordinate system uv The millimeter wave point cloud is received, and the point with an angle exists an angle error . According to the pixel error range of each point, the expansion is carried out in the pixel U direction point by point, and the same U direction is compensated by 1 / 16 in the pixel V direction. S302, performing R, G and B three-channel color assignment on all image pixel points in the range of the expanded image point cloud points, the expanded image point cloud points containing distance and speed two-dimensional information, the distance and speed dimensional information being mapped to 0-255 image color channel values as the R and G two color channels according to the maximum value set by the system; and the density information of the point cloud points being used as the third channel B dimension to expand the image point cloud points; S4, using a Darknet23 network to perform target detection on the point cloud pixelized image obtained in step S3.

2. The pixelated millimeter wave radar point cloud based target detection method of claim 1, wherein, In step S1, the projection point P of any point P in the millimeter wave radar point cloud on the image uv is: where u, v are the pixel coordinates of the point in the image, and are the intrinsic and extrinsic matrix of the camera, x, y, z are the spatial three-dimensional coordinates of the point P.

3. The pixelated mm-wave radar point cloud based target detection method of claim 1, wherein, In step S2, t is 0.1-0.2 s.

4. The pixelated mm-wave radar point cloud based target detection method of claim 1, wherein, angle error is: wherein, is the millimeter wave radar wavelength, N is the number of virtual antenna arrays, and l is the spacing of adjacent antennas. The point S(x, y, z) in the camera coordinate system and the projected image coordinate system point satisfy: where u, v are the coordinates of a point in the pixel coordinate system of the image, is the focal length in the x-axis direction, is the focal length in the y-axis direction, is the depth of the target point cloud, is the x-coordinate of the principal point of the camera on the image plane, is the y-coordinate of the principal point of the camera on the image plane.

5. The pixelated mm-wave radar point cloud based object detection method of claim 1, wherein, If multiple points in the frame are at the same pixel position, the distance and speed information of the point with the largest distance is placed in the R and G two channels, and the B channel is the density of the point corresponding to the pixel position.

6. The pixelated mmWave radar point cloud based target detection method of claim 1, wherein, In step S4, the number of channels of the Darknet23 network is 1 / 8 of that of the Darknet53 network, and the number of feature extraction layers is (1, 2, 2, 2, 1). In each training iteration, the loss function includes: The mean square error between the prediction results of the two networks.

7. A pixelated millimeter wave radar point cloud based target detection system, characterized in that, It comprises: A collection module, which aligns and projects a single-frame millimeter wave radar point cloud on an image plane to obtain a point cloud projection graph of the millimeter wave radar point cloud on the image plane; A projection module, which merges multiple frames of the point cloud projection graph from the collection module received within a time t onto the same picture; An expansion module, which expands the point cloud projection graph obtained by the projection module along the transverse direction according to the radar angle resolution of the point to obtain a point cloud pixelized image, specifically: According to the principle of millimeter wave radar angle resolution, the range of the image point cloud points is expanded point by point according to the incident angle, and the figure silhouette profile is restored. The received millimeter wave point is located in the point S (x0, y0, z0) in the camera coordinate system, and the point S is projected and aligned to the point S in the image pixel coordinate system uv When the millimeter wave point cloud is received, the point exists angle error According to the pixel error range of each point, the expansion is carried out in the pixel U direction point by point, and the same U direction is compensated by 1 / 16 in the pixel V direction.​ According to the range of the expanded image point cloud points, R, G and B three-channel color assignment is performed on all image pixel points in the range, the expanded image point cloud points containing distance and speed two-dimensional information, the distance and speed dimensional information being mapped to 0-255 image color channel values as the R and G two color channels according to the maximum value set by the system; and the density information of the point cloud points being used as the third channel B dimension to expand the image point cloud points; A detection module, which uses a Darknet23 network to perform target detection on the point cloud pixelized image obtained by the expansion module.

Images