Radar point cloud processing method and device for moving target detection

Abstract

The embodiment of the invention provides a radar point cloud processing method and device for moving target detection. The method comprises the following steps: respectively carrying out point cloud preprocessing on multiple frames of radar point clouds by taking a radar frame as a unit; carrying out frame merging processing on the plurality of radar frames subjected to point cloud preprocessing; and performing point cloud post-processing on the radar point clouds on the plurality of radar frames after frame combination processing. According to the embodiment of the invention, motion target detection can be realized without limiting motion types, and the detection precision is improved.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of radar detection technology. Background Art

[0002] The human key point detection method based on radar point cloud has been widely used because it does not infringe on privacy, can work in dark or partially occluded scenes, and can obtain the real spatial position of the human key points.

[0003] Figure 1 This is a schematic diagram of the application of the human key point detection method based on radar point cloud. Figure 1 As shown, when a human body moves, the corresponding point cloud can be obtained from the radar, and the key point information of the human body can be detected through the point cloud. The key point information of the human body can be used for motion detection, behavior analysis, and fall detection, etc.

[0004] In motion detection and behavior analysis, the relative positions and statistical information of key points can be analyzed to determine the motion and behavior. In fall detection, the height information of the key points of the human body can be used to determine whether the target has fallen.

[0005] It should be noted that the above introduction to the technical background is only for the convenience of providing a clear and complete description of the technical solutions of the present application and for the convenience of understanding by those skilled in the art. It cannot be considered that the above technical solutions are well known to those skilled in the art simply because these solutions are described in the background technology part of the present application. Summary of the invention

[0006] The inventors found that in some cases, the radar can only obtain partial reflection information of the human body at a time, and the point cloud distribution varies in space and time for different actions. In addition, the quality of the radar point cloud input into the key point detection model directly affects the detection accuracy of the key points.

[0007] In response to at least one of the above technical problems or other similar problems, an embodiment of the present application provides a method and device for processing radar point clouds for moving target detection to improve detection accuracy.

[0008] According to one aspect of an embodiment of the present application, a processing device for radar point cloud for moving target detection is provided, comprising:

[0009] A preprocessing unit, which performs point cloud preprocessing on multiple frames of radar point clouds in units of radar frames;

[0010] A merging processing unit, which performs frame merging processing on a plurality of radar frames that have undergone point cloud preprocessing;

[0011] A post-processing unit performs point cloud post-processing on the radar point clouds on the multiple radar frames after the frame merging process.

[0012] According to another aspect of an embodiment of the present application, a moving object detection device is provided, the device comprising:

[0013] The processing device described in the aforementioned embodiment performs point cloud preprocessing on multiple frames of radar point clouds to obtain a point cloud with a combined total number of points;

[0014] A moving target detection model performs moving target detection on the point cloud of the combined total number of points to obtain a moving target.

[0015] According to another aspect of an embodiment of the present application, an electronic device is provided, comprising a memory and a processor, wherein the memory stores a computer program, and the processor is configured to execute the computer program to implement the following method:

[0016] Taking radar frame as the unit, perform point cloud preprocessing on multiple frames of radar point cloud respectively;

[0017] Perform frame merging processing on multiple radar frames that have undergone point cloud preprocessing;

[0018] Performing point cloud post-processing on the radar point clouds on the multiple radar frames after the frame merging process to obtain a point cloud with a merged total number of points;

[0019] The moving target detection model is used to perform moving target detection on the point cloud of the combined total number of points to obtain the moving target.

[0020] One of the beneficial effects of the embodiments of the present application is that: by preprocessing the radar point clouds of multiple frames separately in units of radar frames, determining the radar frames that need to be merged according to the point cloud merging strategy, and selecting the radar point cloud as the input of the motion target detection model according to the predetermined strategy from the determined radar frames that need to be merged, the motion target detection can be achieved without limiting the action category, thereby improving the detection accuracy.

[0021] With reference to the following description and accompanying drawings, the specific implementation of the embodiment of the present application is disclosed in detail, indicating the way in which the principle of the embodiment of the present application can be adopted. It should be understood that the implementation of the present application is not limited in scope. Within the scope of the spirit and clauses of the attached claims, the implementation of the present application includes many changes, modifications and equivalents. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] The included drawings are used to provide a further understanding of the embodiments of the present application, which constitute a part of the specification, are used to illustrate the implementation methods of the present application, and together with the text description, explain the principles of the present application. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other implementation methods can be obtained based on these drawings without creative work. In the drawings:

[0023] Figure 1 It is a schematic diagram of the application of the human body key point detection method based on radar point cloud;

[0024] Figure 2 is a frame of radar point cloud set P t Schematic diagram of

[0025] Figure 3 It is N frames of continuous radar point cloud Schematic diagram of

[0026] Figure 4 It is a schematic diagram of the key points of the human body;

[0027] Figure 5 It is a schematic diagram of the existing point cloud preprocessing;

[0028] Figure 6 This is a comparison diagram of the point cloud merging results with a fixed number of frames;

[0029] Figure 7 is a schematic diagram of a radar point cloud processing method for moving target detection according to an embodiment of the present application;

[0030] Figure 8 is a schematic diagram of an example of a method for processing radar point cloud according to an embodiment of the present application;

[0031] Fig. 9 is a schematic diagram of a radar point cloud processing method according to an embodiment of the present application;

[0032] Fig.10 is another schematic diagram of the radar point cloud processing method according to an embodiment of the present application;

[0033] Fig.11 is a schematic diagram of a moving target detection method according to an embodiment of the present application;

[0034] Fig.12 It is a schematic diagram of the key point detection model;

[0035] Fig.13 It is a schematic diagram of the post-processing model;

[0036] Fig.14 is a schematic diagram of a radar point cloud processing device according to an embodiment of the present application;

[0037] Fig.15 is a schematic diagram of a moving target detection device according to an embodiment of the present application;

[0038] Fig.16 It is a schematic diagram of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0039] With reference to the accompanying drawings, the foregoing and other features of the embodiments of the present application will become apparent through the following description. In the description and the accompanying drawings, specific embodiments of the present application are specifically disclosed, which show some embodiments in which the principles of the embodiments of the present application can be adopted. It should be understood that the present application is not limited to the described embodiments. On the contrary, the embodiments of the present application include all modifications, variations and equivalents that fall within the scope of the attached claims.

[0040] In the embodiments of the present application, the terms "first", "second", etc. are used to distinguish different elements in terms of title, but do not indicate the spatial arrangement or temporal order of these elements, etc., and these elements should not be limited by these terms. The term "and / or" includes any one and all combinations of one or more of the associated listed terms. The terms "comprising", "including", "having", etc. refer to the presence of the stated features, elements, components or components, but do not exclude the presence or addition of one or more other features, elements, components or components.

[0041] In the embodiments of the present application, the singular forms "a", "the", etc. include plural forms and should be broadly understood as "a kind" or "a type" rather than being limited to the meaning of "one"; in addition, the term "said" should be understood to include both singular and plural forms, unless the context clearly indicates otherwise. In addition, the term "according to" should be understood as "at least in part according to...", and the term "based on" should be understood as "at least in part based on...", unless the context clearly indicates otherwise.

[0042] Features described and / or shown for one embodiment may be used in one or more other embodiments in the same or similar manner, combined with features in other embodiments, or replace features in other embodiments. The term "includes / comprising" when used herein refers to the presence of a feature, an entirety, a step, or a component, but does not exclude the presence or addition of one or more other features, entireties, steps, or components.

[0043] In the embodiment of the present application, the radar may be a millimeter wave (mmWave) radar, but the present application is not limited thereto. The radar transmits electromagnetic waves through a transmitting antenna, and receives corresponding reflected waves (which may be referred to as radar echo information) after being reflected by different objects. By analyzing the radar echo information, information such as the position of the object from the radar, radial motion speed, etc. can be effectively extracted, and this information can meet the needs of many application scenarios.

[0044] In the embodiment of the present application, the detection target can be people of various ages, for example, it can be an elderly person, it can also be a child, it can also be an elderly person and / or a caregiver, a child and / or a guardian. The present application is not limited thereto, and the detection target can also be an animal with life characteristics, or a robot without life characteristics, etc. The following is described by taking the human body as an example.

[0045] In an embodiment of the present application, a single radar point cloud is represented by p=(s, v, I, x, y, z), where s is the frame sequence number, v is the Doppler velocity relative to the radar, I is the signal strength, and x, y, z are spatial coordinates.

[0046] In the embodiment of the present application, a radar point cloud set is represented by P = {p j ,1≤j≤n}, where n is the number of radar point clouds in the current frame. Figure 2 is a frame of radar point cloud set P t Schematic diagram of .

[0047] In the embodiment of the present application, N frames of continuous radar point cloud collection are used To represent, where N is the number of radar point cloud frames, i is the frame sequence number, representing time, and the frame sequence number is sorted by time. The larger the frame sequence number, the later the corresponding point cloud data appears. Figure 3 It is N frames of continuous radar point cloud Schematic diagram of .

[0048] In the embodiment of the present application, the key point of the human body refers to the spatial position of the joints or organs of the human body, which can be expressed as H = {(x k ,y k ,z k ),1≤k≤m}, where (x k ,y k ,z k ) represents the spatial position of the kth key point, and m is the number of key points. Figure 4 It is a schematic diagram of the key points of the human body.

[0049] In the existing human key point detection methods based on radar point clouds, the quality of the input radar point clouds directly affects the detection accuracy of the key points. Existing work processes radar point clouds through trial and error and lacks a unified optimization strategy.

[0050] Figure 5 This is a schematic diagram of the existing point cloud preprocessing, such as Figure 5 As shown in FIG. 1 , the existing point cloud preprocessing method selects a fixed number of frames, that is, radar frames within a fixed time period, and a fixed number of radar points are selected for each frame. In addition, for different targets and / or different actions, a unified preprocessing method is used, which cannot accurately describe the key point information.

[0051] Figure 6 This is a comparison diagram of the point cloud merging results with a fixed number of frames, such as Figure 6 As shown in the right figure of (a), using the existing point cloud preprocessing method, some redundant interference information (N=10) will be merged in the case of fast movement; Figure 6 As shown in the left figure of (b), when the movement is slow, information loss will occur (N=5).

[0052] In view of the above problems, the present application is proposed, and the embodiments of the present application are described below with reference to the accompanying drawings.

[0053] Embodiments of the first aspect

[0054] An embodiment of the present application provides a method for processing radar point clouds for moving target detection. Figure 7 is a schematic diagram of a method for processing radar point clouds for moving target detection according to an embodiment of the present application, such as Figure 7 As shown, the method includes:

[0055] 701, taking radar frames as units, respectively performing point cloud preprocessing on radar point clouds of multiple frames;

[0056] 702, performing frame merging processing on a plurality of radar frames that have undergone point cloud preprocessing; and

[0057] 703 , performing point cloud post-processing on the radar point clouds on the multiple radar frames after the frame merging process.

[0058] It is worth noting that the above Figure 7 The embodiments of the present application are only schematically described, but the present application is not limited thereto. For example, the execution order of the various operations can be appropriately adjusted, and other operations can be added or some operations can be reduced. Those skilled in the art can make appropriate modifications based on the above content, and are not limited to the above appended Figure 7 Records of.

[0059] According to the above embodiment, by performing point cloud preprocessing on multiple frames of radar point clouds in units of radar frames, determining the radar frames that need to be merged according to the point cloud merging strategy, and selecting radar point clouds as input of the moving target detection model according to a predetermined strategy from the determined radar frames that need to be merged, moving target detection can be achieved without limiting the action category, thereby improving the detection accuracy.

[0060] Figure 8 FIG. 1 is a schematic diagram of an example of a method for processing radar point clouds according to an embodiment of the present application. Figure 8 As shown, in an embodiment of the present application, different radar frames are preprocessed separately, for example, single frame preprocessing-0 is performed on the i-th frame, single frame preprocessing-1 is performed on the i-1-th frame, single frame preprocessing-N is performed on the iN-1-th frame, and so on.

[0061] In the above embodiment, the point cloud preprocessing operations performed on different radar frames can be the same or different. For example, the single frame preprocessing-0, single frame preprocessing-1, ..., single frame preprocessing-N can be the same or different, or partially the same. Therefore, there is no need to limit the action category in the process of point cloud preprocessing, which is more flexible.

[0062] In the above embodiment, after the radar frame is pre-processed with the point cloud, the point cloud frame merging processing can be performed, the radar frame to be merged can be determined, and then the merged point cloud post-processing can be performed, that is, the radar point cloud is selected in the determined merged radar frame as the input of the motion target detection model (such as the key point detection model).

[0063] In some embodiments, in operation 701, performing point cloud preprocessing on a single frame of radar point cloud may include the following operations:

[0064] S11: Filter the point cloud on the current radar frame;

[0065] S12: Determine the point cloud that needs to be retained on the radar frame according to a predetermined strategy.

[0066] N frames of continuous radar point cloud collection For example, a single frame radar point cloud is represented as P t ={p j ,1≤j≤n}, where n is the number of point clouds in a single frame and t is the frame sequence number.

[0067] In operation S11, filtering the point cloud can be expressed as follows:

[0068] P t ′=filter(P t )

[0069] In the above embodiments, any filtering method can be used to filter the point cloud, such as a spatial filtering method, a speed filtering method, a cluster filtering method, etc. Among them, a spatial filtering method is used to filter the point cloud, for example, filtering out point clouds outside a predetermined range; a speed filtering method is used to filter the point cloud, for example, filtering out point clouds with too fast and / or too slow speeds; a cluster filtering method is used to filter the point cloud, for example, after clustering the point cloud, filtering out point clouds that cannot be clustered into clusters, or point clouds with too few clusters, etc. The specific filtering method to be used can be determined as needed, and this application does not impose any restrictions on this. In addition, for the specific implementation process of the above-mentioned spatial filtering method, speed filtering method, cluster filtering method, etc., reference can be made to the relevant technology, and the description is omitted here.

[0070] In operation S12 , it may be determined which point clouds need to be retained on the radar frame according to a predetermined strategy.

[0071] In some embodiments, the predetermined strategy is that the number of point clouds on the radar frame does not exceed a first threshold.

[0072] In the above embodiment, if the number of point clouds filtered on the radar frame does not exceed the first threshold, the filtered point clouds are used as point clouds that need to be retained on the radar frame; if the number of point clouds filtered on the radar frame exceeds the first threshold, the filtered point clouds are sampled, such as farthest point sampling or down sampling, to obtain point clouds that need to be retained on the radar frame, wherein the number of point clouds that need to be retained on the radar frame does not exceed the first threshold.

[0073] In other embodiments, the predetermined strategy is that the number of point clouds on the radar frame does not exceed a predetermined proportion of the total number of merged points.

[0074] In the above embodiment, if the number of filtered point clouds on the radar frame does not exceed a predetermined proportion of the total number of points merged, the filtered point cloud is used as the point cloud that needs to be retained on the radar frame; if the number of filtered point clouds on the radar frame exceeds the predetermined proportion of the total number of points merged, the filtered point cloud is sampled, for example, farthest point sampling or down sampling is performed to obtain the point cloud that needs to be retained on the radar frame, wherein the number of point clouds that need to be retained on the radar frame does not exceed the above-mentioned predetermined proportion of the total number of points merged.

[0075] In the embodiment of operation S12, the predetermined strategies (such as the first threshold or the predetermined ratio) corresponding to radar frames in different time sequences may be the same or different. For example, the first threshold / predetermined ratio used for the radar point cloud on the i-1th frame is different from the first threshold / predetermined ratio used for the radar point cloud on the i-th frame.

[0076] In the above embodiment, a predetermined strategy (such as a first threshold or a predetermined ratio) can be determined according to the time sequence of the single-frame radar point cloud, and then the point cloud P to be retained in the frame can be determined according to the predetermined strategy. t ″, expressed as follows:

[0077] P″ t =select(P t ,t,M0)

[0078] Among them, M0 is the total number of point clouds that need to be merged in the end.

[0079] For example, the predetermined strategy is that the number of radar point clouds on the current frame does not exceed m1, and when it exceeds m1, m1 points are obtained by sampling the farthest point. For another example, the predetermined strategy is that the number of radar point clouds on the current frame does not exceed q(t)% of the total number of points to be merged, and when it exceeds q(t)%, points less than or equal to q(t)% of the total number of points to be merged are obtained by sampling the farthest point.

[0080] In some embodiments, in operation 702, performing frame merging processing on multiple radar frames that have undergone point cloud preprocessing may include the following operations:

[0081] S21: judging the validity of the data of the current radar frame;

[0082] S22: If the data of the radar frame is valid, starting from the radar frame, within a fixed number of frames, the point clouds on each radar frame are merged sequentially until the number of point clouds is at least greater than or equal to the total number of merged points.

[0083] In operation S21, it is determined whether to merge the radar data of the frame. For example, if the total number of point clouds of the current radar frame and a predetermined number of radar frames before the radar frame is less than a threshold (referred to as the second threshold), it is determined that the data of the current radar frame is invalid and the data of the current radar frame is not merged. Otherwise, it is determined that the data of the current radar frame is valid and the data of the current radar frame is merged.

[0084] For example, if the latest N1 frames (including the current radar frame P N If the total number of point clouds of the current radar frame (and N1-1 radar frames before the current radar frame) is less than M1 (the second threshold), the data of the current radar frame is determined to be invalid, and the radar data of the frame is not merged. It can be expressed as follows:

[0085]

[0086] Among them, M(P N ,…,P N-N1-1 ) is the total number of points in the radar point cloud of the latest N1 frame.

[0087] Taking N1=3 and M1=30 as an example, if the sum of the number of point clouds of the latest three frames (including the current radar frame and the two radar frames before the current radar frame) is less than 30, the data of the current radar frame is considered invalid and the current radar frame is not merged.

[0088] In operation S22, in the valid radar frames, starting from the latest radar frame (current radar frame), the point clouds on each radar frame are merged in sequence within a fixed number of frames. For example, N frames of radar point cloud data are merged in descending order of frame numbers, and the process stops when the sum of the number of merged point clouds is greater than or equal to the total number of merged points M (M≥M0), and the merged radar frame number tn is obtained, and then the point cloud set of the tn frames is obtained. Among them, tn≤N.

[0089] In the above embodiment, N frames are used as the maximum limit for frame merging, thereby limiting the frames to be merged to a certain time period, thereby avoiding using too long historical information to affect the detection result.

[0090] The above is only an exemplary description of frame merging, and the present application is not limited to this. Other strategies can also be used for frame merging. For example, instead of judging the validity of the data, a predetermined number of frames are directly merged from the current radar frame to the front to obtain the total number of merged points as the convergence condition, and so on.

[0091] In some embodiments, in operation 703, a point cloud having a combined total number of points may be selected from the plurality of radar frames after the merging process according to the spatial distribution characteristics and / or velocity distribution characteristics of the point clouds.

[0092] For example, all point clouds on the merged multiple radar frames are sampled, such as sampling the farthest point, to obtain a point cloud with a total number of merged points.

[0093] For another example, first sample the point cloud on the merged multiple radar frames except the latest frame in time (which may be the current radar frame), for example, perform farthest point sampling, and determine the number of point clouds on the frame, recorded as MP. Then, merge the point clouds on the remaining frames and the point cloud on the frame to obtain a point cloud with a merged total number of points.

[0094] Still taking the above as an example, according to the above embodiment, in the determined tn frame, M0 point clouds are selected, and sampling can be performed according to the spatial distribution, velocity distribution and other characteristics of the point clouds. For example, in all radar point clouds of the tn frame, M0 points are obtained by sampling the farthest point; or, only the last frame (the latest frame, such as the current frame) is sampled so that the number of point clouds on the frame is MP, and then the remaining M0-MP point clouds are merged from the remaining tn-1 frames in turn to obtain M0 points.

[0095] The above is just an example, and other post-processing methods or other sampling methods can also be used to process the radar point cloud of tn frames to obtain the required number of points.

[0096] The following two examples illustrate the radar point cloud processing method of the embodiment of the present application.

[0097] Fig. 9 FIG. 1 is a schematic diagram of a radar point cloud processing method according to an embodiment of the present application. Fig. 9 In the example, a spatial filtering method is used to filter a single frame point cloud, and m1=50, N=10, M=M0=200. Fig. 9 As shown, the method includes:

[0098] 901: single frame preprocessing, for example, spatial filtering of a single frame point cloud to filter out radar points with a height above 2.3 meters. The number of radar points in the current radar frame cannot exceed 50.

[0099] 902: point cloud frame merging processing, for example, merging within 10 frames, stopping when the total number of points is greater than 200, and obtaining the number of frames tn for point cloud merging;

[0100] 903: Merge point cloud post-processing, for example, retain all point clouds of the first tn-1 frames, sample the farthest point of the last frame, so that the total number of final point clouds is 200.

[0101] Afterwards, the obtained point cloud can be input into a moving object detection model (such as a key point detection model) for subsequent processing.

[0102] Fig.10 FIG. 1 is another schematic diagram of a radar point cloud processing method according to an embodiment of the present application. Fig.10 In the example of , a spatial filtering method is used to filter a single frame point cloud, and N = 10, M = 300, and M0 = 200. Fig.10 As shown, the method includes:

[0103] 1001: single frame preprocessing, for example, performing spatial filtering on the single frame point cloud to filter out radar points with a height above 2.3 meters;

[0104] 1002: point cloud frame merging processing, for example, merging within 10 frames, stopping when the total number of points is greater than 300, and obtaining the number of frames tn for point cloud merging;

[0105] 1003: Merge the point clouds for post-processing, for example, use the farthest point sampling to select 200 point clouds through the farthest point sampling.

[0106] Afterwards, the obtained point cloud can be input into a moving object detection model (such as a key point detection model) for subsequent processing.

[0107] The above embodiments are merely exemplary of the embodiments of the present application, but the present application is not limited thereto, and appropriate modifications may be made based on the above embodiments. For example, the above embodiments may be used alone, or one or more of the above embodiments may be combined.

[0108] It can be seen from the above embodiments that by performing point cloud preprocessing on multiple frames of radar point clouds in units of radar frames, determining the radar frames that need to be merged according to the point cloud merging strategy, and selecting radar point clouds as input of the moving target detection model according to a predetermined strategy from the determined radar frames that need to be merged, moving target detection can be achieved without limiting the action category, thereby improving the detection accuracy.

[0109] Embodiments of the second aspect

[0110] An embodiment of the present application provides a moving target detection method. Fig.11 is a schematic diagram of a moving target detection method according to an embodiment of the present application. Fig.11 As shown, the method includes:

[0111] 1101: performing point cloud preprocessing on multiple frames of radar point clouds to obtain a point cloud with a combined total number of points;

[0112] 1102: In a moving target detection model, performing moving target detection on the point cloud of the combined total number of points to obtain a detection result of the moving target.

[0113] In the above embodiment, operation 1101 may be implemented by the method of the embodiment of the first aspect. Since the method has been described in the embodiment of the first aspect, its content is incorporated herein and will not be repeated here.

[0114] In the above embodiment, the radar point cloud is preprocessed according to the method of the embodiment of the first aspect before entering the detection model, so that moving target detection can be achieved without limiting the action category, thereby improving the detection accuracy.

[0115] In some embodiments, the moving object detection model is a key point detection model. In the above embodiments, the key point detection model can be used to perform key point detection on the point cloud with the total number of points merged to obtain the key points of the human body.

[0116] Fig.12 It is a schematic diagram of the key point detection model. Fig.12 As shown, after the input point cloud enters the key point detection model, the key point detection is performed on the input point cloud in the key point detection model to obtain the key points of the human body.

[0117] In the above embodiments, a key point detection model based on modules such as conv1d, EdgeConv, Adaptive graph convolution, spatialattention+channel attention, and full connection is taken as an example, but the present application is not limited to this. The key point detection model may also include other structures (layers), or reduce some structures (layers). For details, please refer to the relevant technology.

[0118] In the above embodiment, taking the prediction of the three-dimensional coordinates (x, y, z) of 17 key points of the human body as an example, the key point detection model may be implemented with other parameters depending on the number of predicted key points.

[0119] In some embodiments, the moving target detection model is a behavior analysis model. In the above embodiments, the behavior analysis model can be used to perform behavior analysis on the point cloud of the combined total points to obtain output action categories, such as walking, sitting on a chair, standing, lying on the ground, lying on a bed, falling, etc.

[0120] In the above embodiment, there is no limitation on the structure of the behavior analysis model, and reference may be made to related technologies.

[0121] In some embodiments, Fig.11 As shown, the method also includes:

[0122] 1103: Post-process the detection result of the moving target to obtain a processed moving target.

[0123] In some embodiments, post-processing the moving target includes:

[0124] S31: extracting the moving target detection results of the current frame and the previous n consecutive frames;

[0125] S32: In the post-processing model, the moving target detection results of the current frame and the previous n consecutive frames are temporally connected using 1d convolution;

[0126] S33: In the post-processing model, full connection is used to connect the moving objects to obtain a corrected moving object of the current frame.

[0127] In the above embodiment, the detection accuracy is further improved by post-processing the detected moving target detection results (such as the key point detection structure of the human body).

[0128] Fig.13 is a schematic diagram of the post-processing model. Fig.13As shown in the figure, after the detection results of the moving targets of the input multiple frames (human key point detection results) enter the post-processing model, in the post-processing model, the moving target detection results of these frames are temporally connected using 1d convolution, and then the moving targets are connected using full connection to obtain the corrected moving target on the current frame.

[0129] This application does not restrict the structure of the post-processing model. Fig.13 Modifications can be made on the basis of, for example, adding some layers of structure, or reducing some layers of structure, for details, please refer to the relevant technology.

[0130] It is worth noting that the above Fig.11 The embodiments of the present application are only schematically described, but the present application is not limited thereto. For example, the execution order of the various operations can be appropriately adjusted, and other operations can be added or some operations can be reduced. Those skilled in the art can make appropriate modifications based on the above content, and are not limited to the above appended Fig.11 Records of.

[0131] Table 1 is an example of key point detection results after preprocessing the radar point cloud using the radar point cloud processing method of an embodiment of the present application.

[0132] Table 1:

[0133]

[0134] As shown in Table 1, for person 117 (walking), Fig. 9 The error of the method is only 0.0629, and the detection accuracy is improved by 23.7%. Fig.10 The error of the method is only 0.0628, and the detection accuracy is improved by 23.8%. For person 128 (falling), the method Fig. 9 The error of the method is only 0.0499, and the detection accuracy is improved by 38.4%. Fig.10 The error of the method is only 0.0499, and the detection accuracy is improved by 38.4%.

[0135] Table 2 is an example of test results after post-processing the key point detection results using the post-processing method of an embodiment of the present application.

[0136]

[0137] As shown in Table 2, for person 117 (walking), after post-processing the key point detection results of 10 frames, the error is only 0.0613, and the accuracy is improved by 8.17%; for person 128 (falling), after post-processing the key point detection results of 10 frames, the error is only 0.0488, and the accuracy is improved by 7.22%.

[0138] According to the above embodiment, by performing point cloud preprocessing on multiple frames of radar point clouds in units of radar frames, determining the radar frames that need to be merged according to the point cloud merging strategy, and selecting radar point clouds as input of the moving target detection model according to a predetermined strategy from the determined radar frames that need to be merged, moving target detection can be achieved without limiting the action category, thereby improving the detection accuracy.

[0139] Embodiments of the third aspect

[0140] The embodiment of the present application provides a processing device for radar point cloud for moving target detection, and the principle of solving the problem is similar to that of the embodiment of the first aspect, and the same contents are not repeated here.

[0141] Fig.14 is a schematic diagram of a radar point cloud processing device according to an embodiment of the present application. Fig.14 As shown, the radar point cloud processing device 1400 of the embodiment of the present application includes:

[0142] A preprocessing unit 1401 performs point cloud preprocessing on multiple frames of radar point clouds in units of radar frames;

[0143] A merging processing unit 1402 performs frame merging processing on a plurality of radar frames that have undergone point cloud preprocessing;

[0144] The post-processing unit 1403 performs point cloud post-processing on the radar point clouds on the multiple radar frames after the frame merging process.

[0145] In some embodiments, the preprocessing unit 1401 performs point cloud preprocessing on a single frame of radar point cloud, including:

[0146] Filter the point cloud on the current radar frame;

[0147] The point cloud that needs to be retained on the radar frame is determined according to a predetermined strategy.

[0148] In some embodiments, the predetermined strategy is:

[0149] The number of point clouds on the radar frame does not exceed a first threshold.

[0150] In the above embodiment, if the number of the point clouds after filtering does not exceed the first threshold, the preprocessing unit uses the filtered point clouds as the point clouds that need to be retained on the radar frame; if the number of the point clouds after filtering exceeds the first threshold, the preprocessing unit samples the filtered point clouds to obtain the point clouds that need to be retained on the radar frame, wherein the number of point clouds that need to be retained on the radar frame does not exceed the first threshold.

[0151] In some embodiments, the predetermined strategy is:

[0152] The number of point clouds on the radar frame does not exceed a predetermined proportion of the total number of merged points.

[0153] In the above embodiment, if the number of the filtered point cloud does not exceed the predetermined proportion of the total number of merged points, the preprocessing unit uses the filtered point cloud as the point cloud that needs to be retained on the radar frame; if the number of the filtered point cloud exceeds the predetermined proportion of the total number of merged points, the preprocessing unit samples the filtered point cloud to obtain the point cloud that needs to be retained on the radar frame, wherein the number of the point cloud that needs to be retained on the radar frame does not exceed the predetermined proportion of the total number of merged points.

[0154] In some embodiments, the merging processing unit 1402 performs the following processing:

[0155] Determine the validity of the data of the current radar frame;

[0156] If the data of the radar frame is valid, starting from the radar frame, within a fixed number of frames, the point clouds on each radar frame are merged sequentially until the number of point clouds is at least greater than or equal to the total number of merged points.

[0157] In the above embodiment, if the total number of point clouds of the radar frame and a predetermined number of radar frames before the radar frame is less than the second threshold, the merging processing unit 1402 determines that the data of the radar frame is invalid, otherwise it determines that the data of the radar frame is valid.

[0158] In some embodiments, the post-processing unit 1403 selects a point cloud having a combined total number of points from the multiple radar frames after the merging process according to the spatial distribution characteristics and / or velocity distribution characteristics of the point clouds.

[0159] In the above embodiment, the post-processing unit 1403 may sample all point clouds on the multiple radar frames after the merging process to obtain the point cloud with the merged total number of points; or, the post-processing unit 1403 retains the point clouds on other radar frames except the latest frame in time among the multiple radar frames after the merging process; and samples the point cloud on the latest frame to obtain the point cloud with the merged total number of points.

[0160] The embodiment of the present application also provides a moving target detection device, the principle of solving the problem is similar to that of the embodiment of the first aspect, and the same contents are not repeated here.

[0161] Fig.15 is a schematic diagram of a moving target detection device according to an embodiment of the present application. Fig.15 As shown, the moving object detection device 1500 of the embodiment of the present application includes:

[0162] A processing device 1501 performs point cloud preprocessing on multiple frames of radar point clouds using the method of the embodiment of the first aspect to obtain a point cloud with a combined total number of points; and

[0163] The detection device 1502 uses a moving target detection model to perform moving target detection on the point cloud of the combined total number of points to obtain a moving target.

[0164] In the above embodiment, the processing device 1501 may be implemented by the radar point cloud processing device 1400 of the aforementioned embodiment, the content of which is incorporated herein and will not be repeated here.

[0165] In the above embodiment, the detection device 1502 can use Fig.12 The key point detection model shown performs key point detection on the input radar point cloud to obtain key point detection results. The present application is not limited thereto, and the detection device 1502 may also use other similar key point detection models or behavior analysis models to detect or analyze the input radar point cloud to obtain detection results or analysis results.

[0166] In some embodiments, Fig.15 As shown, the device 1500 may further include:

[0167] The post-processing device 1503 uses a post-processing model to post-process the moving target detection results (such as key point detection results) of multiple frames to obtain the final moving target (key point) detection results.

[0168] In the above embodiment, the post-processing device 1503 can use Fig.13 The post-processing model shown performs post-processing on the motion target detection results (key point detection results) of multiple frames. The present application is not limited to this. The post-processing device 1503 can also use other similar post-processing models to post-process the motion target detection results (key point detection results) of multiple frames.

[0169] It is worth noting that the above only describes the components or modules related to the present application, but the present application is not limited thereto. The above devices 1400 and 1500 of the embodiments of the present application may also include other components or modules, and the specific contents of these components or modules may refer to the relevant technology.

[0170] To keep it simple, Fig.14 and Fig.15 The connection relationship or signal direction between various components or modules is only exemplified, but it should be clear to those skilled in the art that various related technologies such as bus connection can be used. The above-mentioned components or modules can be implemented by hardware facilities such as processors and memories; the embodiments of the present application are not limited to this.

[0171] The above embodiments are merely exemplary of the embodiments of the present application, but the present application is not limited thereto, and appropriate modifications may be made based on the above embodiments. For example, the above embodiments may be used alone, or one or more of the above embodiments may be combined.

[0172] It can be seen from the above embodiments that by performing point cloud preprocessing on multiple frames of radar point clouds in units of radar frames, determining the radar frames that need to be merged according to the point cloud merging strategy, and selecting radar point clouds as input of the moving target detection model according to a predetermined strategy from the determined radar frames that need to be merged, moving target detection can be achieved without limiting the action category, thereby improving the detection accuracy.

[0173] Embodiments of the fourth aspect

[0174] The embodiment of the present application provides an electronic device, including the apparatus 1400 or 1500 as described in the embodiment of the third aspect, the contents of which are incorporated herein. The electronic device may be, for example, a computer, a server, a workstation, a laptop computer, a smart phone, etc., but the embodiment of the present application is not limited thereto.

[0175] Fig.16 Schematic diagram of an electronic device according to an embodiment of the present application. Fig.16 As shown, the electronic device 1600 may include: a processor (e.g., a central processing unit CPU) 1610 and a memory 1620; the memory 1620 is coupled to the central processing unit 1610. The memory 1620 may store various data; in addition, it may store a program 1621 for information processing, and execute the program 1621 under the control of the processor 1610.

[0176] In some embodiments, the functions of the apparatus 1400 or 1500 described in the embodiment of the third aspect are integrated into the processor 1610 for implementation. The processor 1610 is configured to implement the method described in the embodiment of the first aspect.

[0177] In some embodiments, the device 1400 or 1500 described in the embodiment of the third aspect is configured separately from the processor 1610. For example, the device 1400 or 1500 described in the embodiment of the third aspect can be configured as a chip connected to the processor 1610, and the functions of the device 1400 or 1500 described in the embodiment of the third aspect are implemented through the control of the processor 1610.

[0178] In addition, if Fig.16 As shown, the electronic device 1600 may also include: an input / output (I / O) device 1630 and a display 1640, etc.; wherein the functions of the above components are similar to those of the prior art and are not described in detail here. It is worth noting that the electronic device 1600 does not necessarily have to include Fig.16 In addition, the electronic device 1600 may also include Fig.16 For components not shown in the figure, reference may be made to the related art.

[0179] An embodiment of the present application also provides a computer-readable program, wherein when the program is executed in an electronic device, the program enables a computer to execute the method as described in the embodiment of the first aspect in the electronic device.

[0180] An embodiment of the present application further provides a storage medium storing a computer-readable program, wherein the computer-readable program enables a computer to execute the method as described in the embodiment of the first aspect in an electronic device.

[0181] The above devices and methods of the present application can be implemented by hardware, or by hardware combined with software. The present application relates to such a computer-readable program, which, when executed by a logic component, enables the logic component to implement the above-mentioned devices or components, or enables the logic component to implement the various methods or steps described above. The present application also relates to a storage medium for storing the above program, such as a hard disk, a magnetic disk, an optical disk, a DVD, a flash memory, etc.

[0182] The method / device described in conjunction with the embodiments of the present application may be directly embodied as hardware, a software module executed by a processor, or a combination of the two. For example, one or more of the functional block diagrams shown in the figure and / or one or more combinations of the functional block diagrams may correspond to various software modules of the computer program flow or to various hardware modules. These software modules may correspond to the various steps shown in the figure, respectively. These hardware modules may be implemented by solidifying these software modules, for example, using a field programmable gate array (FPGA).

[0183] The software module may be located in a RAM memory, a flash memory, a ROM memory, an EPROM memory, an EEPROM memory, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art. A storage medium may be coupled to a processor so that the processor can read information from the storage medium and write information to the storage medium; or the storage medium may be an integral part of the processor. The processor and the storage medium may be located in an ASIC. The software module may be stored in a memory of a mobile terminal or in a memory card that can be inserted into the mobile terminal. For example, if a device (such as a mobile terminal) uses a large-capacity MEGA-SIM card or a large-capacity flash memory device, the software module may be stored in the MEGA-SIM card or the large-capacity flash memory device.

[0184] For one or more of the functional blocks described in the drawings and / or one or more combinations of functional blocks, it can be implemented as a general-purpose processor, digital signal processor (DSP), application-specific integrated circuit (ASIC), field programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic device, discrete hardware component or any appropriate combination thereof for performing the functions described in the present application. For one or more of the functional blocks described in the drawings and / or one or more combinations of functional blocks, it can also be implemented as a combination of computing devices, such as a combination of a DSP and a microprocessor, multiple microprocessors, one or more microprocessors in communication with a DSP, or any other such configuration.

[0185] The present application is described above in conjunction with specific implementation methods, but it should be clear to those skilled in the art that these descriptions are exemplary and are not intended to limit the scope of protection of the present application. Those skilled in the art can make various modifications and variations to the present application based on the principles of the present application, and these modifications and variations are also within the scope of the present application.

[0186] Regarding the implementation methods including the above embodiments, the following additional notes are also disclosed:

[0187] 1. A method for processing radar point clouds for moving target detection, the method comprising:

[0188] S1, taking radar frame as unit, performing point cloud preprocessing on multiple frames of radar point cloud respectively;

[0189] S2, performing frame merging processing on multiple radar frames that have undergone point cloud preprocessing;

[0190] S3, performing point cloud post-processing on the radar point clouds on the multiple radar frames after the frame merging process.

[0191] 2. The method according to Note 1, wherein in S1, performing point cloud preprocessing on a single frame of radar point cloud comprises:

[0192] S11, filtering the point cloud on the current radar frame;

[0193] S12: Determine the point cloud that needs to be retained on the radar frame according to a predetermined strategy.

[0194] 3. According to the method described in Supplement 2, in S11,

[0195] The filtering includes at least one of the following: space filtering, speed filtering and cluster filtering.

[0196] 4. According to the method described in Supplement 2, in S12, the predetermined strategy is:

[0197] The number of point clouds on the radar frame does not exceed a first threshold.

[0198] 5. The method according to Note 4, wherein S12 comprises:

[0199] If the number of the filtered point cloud does not exceed the first threshold, the filtered point cloud is used as the point cloud that needs to be retained on the radar frame;

[0200] If the number of the filtered point cloud exceeds the first threshold, the filtered point cloud is sampled to obtain the point cloud that needs to be retained on the radar frame, wherein the number of the point cloud that needs to be retained on the radar frame does not exceed the first threshold.

[0201] 6. According to the method described in Supplement 2, in S12, the predetermined strategy is:

[0202] The number of point clouds on the radar frame does not exceed a predetermined proportion of the total number of merged points.

[0203] 7. The method according to Supplementary Note 6, wherein S12 comprises:

[0204] If the number of the filtered point clouds does not exceed the predetermined proportion of the total number of merged points, the filtered point clouds are used as the point clouds that need to be retained on the radar frame;

[0205] If the number of the filtered point cloud exceeds the predetermined proportion of the total number of merged points, the filtered point cloud is sampled to obtain the point cloud that needs to be retained on the radar frame, wherein the number of the point cloud that needs to be retained on the radar frame does not exceed the predetermined proportion of the total number of merged points.

[0206] 8. The method according to Note 1, wherein S2 comprises:

[0207] S21, judging the validity of the data of the current radar frame;

[0208] S22, if the data of the radar frame is valid, starting from the radar frame, within a fixed number of frames, the point clouds on each radar frame are merged sequentially until the number of point clouds is at least greater than or equal to the total number of merged points.

[0209] 9. The method according to Supplementary Note 8, wherein S21 comprises:

[0210] If the total number of point clouds of the radar frame and a predetermined number of radar frames before the radar frame is less than a second threshold, it is determined that the data of the radar frame is invalid; otherwise, it is determined that the data of the radar frame is valid.

[0211] 10. The method according to Note 1, wherein S3 comprises:

[0212] Among the multiple radar frames after the merging process, a point cloud having a combined total number of points is selected according to the spatial distribution characteristics and / or velocity distribution characteristics of the point cloud.

[0213] 11. The method according to Note 10, wherein S3 comprises:

[0214] All point clouds on the merged multiple radar frames are sampled to obtain a point cloud with the total number of merged points.

[0215] 12. The method according to Note 10, wherein S3 comprises:

[0216] retaining point clouds on other radar frames except the latest frame in time among the multiple radar frames after the merging process;

[0217] The point cloud on the latest frame is sampled to obtain the point cloud with the combined total number of points.

[0218] 13. A moving target detection method, wherein the method comprises:

[0219] Use any method in Notes 1-12 to pre-process the radar point clouds of multiple frames to obtain a point cloud with a combined total number of points;

[0220] In the moving target detection model, moving target detection is performed on the point cloud of the combined total number of points to obtain key points of the human body.

[0221] 14. The method according to Note 13, wherein the method further comprises:

[0222] Extract the moving target detection results of the current frame and the previous n consecutive frames;

[0223] In the post-processing model, the moving target detection results of the current frame and the previous n consecutive frames are temporally connected using 1D convolution;

[0224] In the post-processing model, full connection is used to connect the moving objects to obtain the corrected moving object of the current frame.

[0225] 15. An electronic device, comprising a memory and a processor, wherein the memory stores a computer program, wherein the processor is configured to execute the computer program to implement the method according to any one of claims 1 to 14.

[0226] 16. A storage medium storing a computer-readable program, wherein the computer-readable program enables a computer to execute the method as described in any one of Notes 1 to 14 in an electronic device.

Claims

1. A processing device for radar point cloud for moving target detection, the device comprising: A preprocessing unit, which performs point cloud preprocessing on multiple frames of radar point clouds in units of radar frames; A merging processing unit, which performs frame merging processing on a plurality of radar frames that have undergone point cloud preprocessing; A post-processing unit performs point cloud post-processing on the radar point clouds on the multiple radar frames after the frame merging process.

2. The device according to claim 1, wherein: The preprocessing unit performs point cloud preprocessing on a single frame of radar point cloud, including: Filter the point cloud on the current radar frame; The point cloud that needs to be retained on the radar frame is determined according to a predetermined strategy.

3. The device according to claim 2, wherein: The predetermined strategy is: The number of point clouds on the radar frame does not exceed a first threshold; If the number of the filtered point cloud does not exceed the first threshold, the preprocessing unit uses the filtered point cloud as the point cloud that needs to be retained on the radar frame; If the number of the filtered point clouds exceeds the first threshold, the preprocessing unit samples the filtered point clouds to obtain point clouds that need to be retained on the radar frame, wherein the number of point clouds that need to be retained on the radar frame does not exceed the first threshold.

4. The device according to claim 2, wherein: The predetermined strategy is: The number of point clouds on the radar frame does not exceed a predetermined proportion of the total number of merged points; If the number of the filtered point cloud does not exceed the predetermined proportion of the total number of merged points, the preprocessing unit uses the filtered point cloud as the point cloud that needs to be retained on the radar frame; If the number of the filtered point cloud exceeds the predetermined proportion of the total number of points to be merged, the preprocessing unit samples the filtered point cloud to obtain the point cloud that needs to be retained on the radar frame, wherein the number of the point cloud that needs to be retained on the radar frame does not exceed the predetermined proportion of the total number of points to be merged.

5. The device according to claim 1, wherein: The merging processing unit performs the following processing: Determine the validity of the data of the current radar frame; If the data of the radar frame is valid, starting from the radar frame, within a fixed number of frames, the point clouds on each radar frame are merged sequentially until the number of point clouds is at least greater than or equal to the total number of merged points.

6. The device according to claim 5, wherein: If the total number of point clouds of the radar frame and a predetermined number of radar frames before the radar frame is less than a second threshold, the merging processing unit determines that the data of the radar frame is invalid; otherwise, it determines that the data of the radar frame is valid.

7. The device according to claim 1, wherein: The post-processing unit selects a point cloud having a combined total number of points from the multiple radar frames after the merging process according to the spatial distribution characteristics and / or velocity distribution characteristics of the point clouds.

8. The device according to claim 7, wherein: The post-processing unit samples all point clouds on the merged multiple radar frames to obtain the point cloud with the merged total number of points; or The post-processing unit retains point clouds on other radar frames except the latest frame in time among the multiple radar frames after the merging process; and samples the point cloud on the latest frame to obtain the point cloud with the total number of merged points.

9. A moving target detection device, wherein: The device comprises: The processing device according to any one of claims 1 to 8, which performs point cloud preprocessing on multiple frames of radar point clouds to obtain a point cloud with a combined total number of points; A detection device uses a moving target detection model to perform moving target detection on the point cloud of the combined total number of points to obtain a moving target.

10. An electronic device comprising a memory and a processor, wherein the memory stores a computer program, wherein: The processor is configured to execute the computer program to implement the following method: Taking radar frame as the unit, perform point cloud preprocessing on multiple frames of radar point cloud respectively; Perform frame merging processing on multiple radar frames that have undergone point cloud preprocessing; Performing point cloud post-processing on the radar point clouds on the multiple radar frames after the frame merging process to obtain a point cloud with a merged total number of points; The moving target detection model is used to perform moving target detection on the point cloud of the combined total number of points to obtain the moving target.

Images