Object motion trend detection method based on millimeter wave radar and related device

Abstract

The invention discloses an object motion trend detection method based on a millimeter wave radar and a related device, and relates to the field of automatic driving, and the method comprises the steps: obtaining a plurality of objects corresponding to multi-frame point cloud data; clustering the plurality of objects corresponding to each frame of point cloud data to obtain a plurality of clusters; and determining the cluster containing the most objects as the cluster where the objects which are relatively static to the ground are located. And determining the objects except the cluster as moving objects. Determining a moving track of the moving object based on the multi-frame point cloud data; and inputting the moving track of the moving object into a pre-constructed track prediction model, and obtaining a predicted track of the moving object in future time through the track prediction model. Therefore, the purpose of obtaining the motion trend of the moving object is achieved.

Description

Technical Field

[0001] The present application relates to the field of autonomous driving technology, and in particular to a method and related device for detecting object motion trends based on millimeter-wave radar. Background Art

[0002] During the automatic train driving process, it is necessary to detect the movement trend of objects in front of the train in real time to avoid collision between the train and the objects.

[0003] Based on this, how to detect the movement trend of the object ahead of the train is a problem that continues to be solved in this field. Summary of the invention

[0004] In view of the above problems, the present application provides a method and a related device for detecting the motion trend of an object based on a millimeter wave radar, so as to achieve the purpose of detecting the motion trend of an object in front of a train. The specific scheme is as follows:

[0005] The first aspect of the present application provides a method for detecting object motion trend based on millimeter wave radar, comprising:

[0006] Obtain multiple objects corresponding to multiple frames of point cloud data;

[0007] For each of the point cloud data, obtaining relative speeds respectively corresponding to a plurality of objects in the point cloud data, where the relative speed of the object is a relative speed between the object and the millimeter wave radar;

[0008] For each of the point cloud data, based on the relative speeds respectively corresponding to the multiple objects of the point cloud data, the multiple objects of the point cloud data are divided into different clusters;

[0009] For each of the point cloud data, determining an object outside the cluster containing the most objects as a moving object corresponding to the point cloud data;

[0010] Determining a moving trajectory of the moving object based on the multiple frames of point cloud data;

[0011] The moving trajectory of the moving object is input into a pre-built trajectory prediction model, and the predicted trajectory of the moving object in the future time is obtained through the trajectory prediction model.

[0012] In a possible implementation, the step of dividing the multiple objects of the point cloud data into different clusters based on the relative speeds respectively corresponding to the multiple objects of the point cloud data includes:

[0013] Determine an untraversed object among the multiple objects corresponding to the point cloud data as a search object;

[0014] If the search object has been classified into a target cluster, a target object whose distance from the search object is less than or equal to a neighbor distance threshold among the multiple objects corresponding to the point cloud data is classified into the target cluster;

[0015] If the search object is not classified into any cluster, the number of target objects whose distance from the search object is less than or equal to the neighbor distance threshold is obtained from the multiple objects corresponding to the point cloud data; the distance between the search object and the target object is the absolute value of the difference between the relative speed of the search object and the relative speed of the target object;

[0016] If the target number is greater than or equal to a preset minimum number of points, the search object and the target object are divided into the same cluster;

[0017] If the target number is less than the preset minimum number of points, determining that the search object is a noise object;

[0018] The returning step determines an untraversed object among the multiple objects corresponding to the point cloud data as a search object until all the multiple objects corresponding to the point cloud data are determined as the search object.

[0019] In a possible implementation, the step of acquiring multiple objects corresponding to multiple frames of point cloud data includes:

[0020] Acquire multiple initial objects respectively corresponding to multiple frames of point cloud data collected by the millimeter wave radar;

[0021] Determine the area to be inspected;

[0022] For each of the point cloud data, determining that an initial object located in the to-be-detected area among a plurality of initial objects corresponding to the point cloud data is a plurality of candidate objects corresponding to the point cloud data;

[0023] For each of the point cloud data, a false alarm object among a plurality of candidate objects corresponding to the point cloud data is deleted to obtain a plurality of objects corresponding to the point cloud data.

[0024] In a possible implementation, the step of deleting false alarm objects from a plurality of candidate objects corresponding to the point cloud data includes:

[0025] Determining the candidate object whose relative speed is zero as the false alarm object;

[0026] If the absolute value of the difference between the relative speeds of the candidate object in adjacent multi-frame point cloud data is greater than a preset threshold, the candidate object is determined to be the false alarm object;

[0027] Determining the candidate object whose dynamic attribute is a stationary state as the false alarm object;

[0028] Determine as the false alarm object the candidate object whose relative speed in adjacent multiple frames of point cloud data is not zero and whose coordinate information in adjacent multiple frames of point cloud data is the same;

[0029] Determine that the candidate object whose continuous existence period is less than or equal to the first duration and whose loss duration period is greater than or equal to the second duration is the false alarm object;

[0030] The false alarm object among the multiple candidate objects corresponding to the point cloud data is deleted.

[0031] In a possible implementation, the millimeter-wave radar is installed on the front of the train, and the detection area includes: X-axis direction [0, 200 meters], Y-axis direction [-5 meters, 5 meters], the X-axis is the direction of the train's travel, the Y-axis is the direction perpendicular to the X-axis, and the intersection of the X-axis and the Y-axis is located at the front of the train.

[0032] In a possible implementation, the front of the train corresponds to a central axis, and the distance between the central axis and the leftmost side of the front of the train is the same as the distance between the central axis and the rightmost side of the front of the train; the left and right position areas for installing the millimeter-wave radar are within 600mm on both sides of the central axis; taking the road surface as the reference point, the upper and lower position areas for installing the millimeter-wave radar are [road surface + 295mm, road surface + 800mm].

[0033] In a possible implementation, the millimeter wave radar, image acquisition device, laser radar and inertial measurement unit are arranged on the same vertical line perpendicular to the ground.

[0034] In a possible implementation, the method further includes:

[0035] If the reflection cross-sectional area of ​​the mobile object belongs to the first preset area range and the moving speed of the mobile object belongs to the first preset speed range, determining that the target category of the mobile object is a person;

[0036] If the reflection cross-sectional area of ​​the mobile object belongs to the second preset area range and the moving speed of the mobile object belongs to the second preset speed range, it is determined that the target category of the mobile object is a vehicle.

[0037] A second aspect of the present application provides an object motion trend detection device based on millimeter wave radar, comprising:

[0038] A first acquisition module is used to acquire multiple objects corresponding to multiple frames of point cloud data;

[0039] A second acquisition module is used to acquire, for each of the point cloud data, relative speeds corresponding to a plurality of objects in the point cloud data, where the relative speed of the object is a relative speed between the object and the millimeter wave radar;

[0040] A clustering module, for dividing the multiple objects of the point cloud data into different clusters based on the relative speeds respectively corresponding to the multiple objects of the point cloud data for each of the point cloud data;

[0041] A first determination module is used to determine, for each point cloud data, an object outside the cluster containing the most objects as a moving object corresponding to the point cloud data;

[0042] A second determination module, used to determine the moving trajectory of the moving object based on multiple frames of point cloud data;

[0043] The third acquisition module is used to input the moving trajectory of the moving object into a pre-built trajectory prediction model, and obtain the predicted trajectory of the moving object in the future through the trajectory prediction model.

[0044] The third aspect of the present application provides a computer program product, including computer-readable instructions. When the computer-readable instructions are executed on an electronic device, the electronic device implements the object motion trend detection method based on millimeter-wave radar according to the above-mentioned first aspect or any implementation method of the first aspect.

[0045] A fourth aspect of the present application provides an electronic device, comprising at least one processor and a memory connected to the processor, wherein:

[0046] The memory is used to store computer programs;

[0047] The processor is used to execute the computer program so that the electronic device can implement the object motion trend detection method based on millimeter wave radar according to the above-mentioned first aspect or any implementation method of the first aspect.

[0048] In a fifth aspect, the present application provides a computer storage medium, which carries one or more computer programs. When the one or more computer programs are executed by an electronic device, the electronic device can implement the object motion trend detection method based on millimeter wave radar according to the above-mentioned first aspect or any implementation of the first aspect.

[0049] By means of the above technical scheme, the present application provides a method for detecting the motion trend of an object based on a millimeter-wave radar, and obtains multiple objects corresponding to multiple frames of point cloud data; there are usually two scenarios in which the millimeter-wave radar works. The first scenario is a stationary train scenario, i.e., a stationary millimeter-wave radar scenario, and the second scenario is a moving train scenario, i.e., a scene in which the millimeter-wave radar moves with the train. In the first scenario, the millimeter-wave radar will not collect objects that are relatively stationary with the ground, such as trees; in the second scenario, objects that are originally stationary with the ground will also produce a relative speed relative to the laser radar, so the collected objects include both objects that are originally relatively stationary with the ground and objects that are actually moving. Since the speed of the millimeter-wave radar moving with the train cannot be obtained, the absolute speed of the object cannot be calculated. Since the operating environment of trains such as high-speed railways and motor vehicles is generally in the wild, the number of moving objects in motion is small, and the number of objects that are relatively stationary with the ground is large. Based on this, the embodiment of the present application clusters multiple objects corresponding to each frame of point cloud data to obtain multiple clusters; the cluster containing the most objects is determined as the cluster where the objects that are relatively stationary with the ground are located. Objects other than this cluster are determined to be moving objects. Based on multi-frame point cloud data, the moving trajectory of the moving object is determined; the moving trajectory of the moving object is input into the pre-built trajectory prediction model, and the predicted trajectory of the moving object in the future time is obtained through the trajectory prediction model. Thus, the purpose of obtaining the movement trend of the moving object is achieved. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. Throughout the accompanying drawings, the same or similar reference numerals represent the same or similar elements. It should be understood that the drawings are schematic and the originals and elements are not necessarily drawn to scale.

[0051] Figure 1 A schematic diagram of a system architecture provided for this application;

[0052] Figure 2 A schematic diagram of an optional hardware structure of a vehicle-mounted controller 100 provided in the present application;

[0053] Figure 3 A schematic diagram of the structure of a server 200 provided in this application;

[0054] Figure 4 A schematic diagram of the locations of multiple sensors provided in an embodiment of the present application deployed on a train;

[0055] Figure 5 In the embodiment of the present application, the millimeter wave radar Figure 4 Schematic diagram of the installation position in the coordinate system shown;

[0056] Figure 6 A schematic diagram of a flow chart of a method for detecting object motion trend based on millimeter wave radar provided in an embodiment of the present application;

[0057] Figure 7 A schematic diagram of a detection area provided in an embodiment of the present application;

[0058] Figure 8 A schematic diagram of a dangerous moving object provided in an embodiment of the present application;

[0059] Fig. 9 A schematic diagram of the structure of a device for detecting object motion trend based on millimeter wave radar provided in an embodiment of the present application;

[0060] Fig.10 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0061] The following describes the embodiments of the present application in conjunction with the drawings in the embodiments of the present application. The terms used in the implementation method section of the present application are only used to explain the specific embodiments of the present application, and are not intended to limit the present application.

[0062] The embodiments of the present application are described below in conjunction with the accompanying drawings. Those skilled in the art will appreciate that, with the development of technology and the emergence of new scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0063] The terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and need not be used to describe a specific order or sequential order. It should be understood that the terms used in this way can be interchangeable under appropriate circumstances, which is only to describe the distinction mode adopted by the objects of the same attributes when describing in the embodiments of the present application. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, so that the process, method, system, product or equipment comprising a series of units need not be limited to those units, but may include other units that are not clearly listed or inherent to these processes, methods, products or equipment.

[0064] There are many methods for detecting the motion trend of objects based on millimeter-wave radar in the related art. One method is to collect the Doppler spectrum of radar data based on the Doppler effect of the millimeter-wave radar, obtain the attribute information of the object corresponding to the radar data in the Doppler spectrum, and then use the constant false alarm rate to perform peak detection on the spectrum information to complete the motion trend detection of the object. This method requires the use of Fourier transform and other methods to collect the radar Doppler spectrum, and the steps are relatively complicated; another method is to use the graph convolution network in deep learning to convert the millimeter-wave radar point cloud data into a graph structure, and then apply convolution operations on the graph structure to extract features, thereby realizing the detection of the motion trend of the object. This method can efficiently process the sparse data of the millimeter-wave radar, but this method has high computational complexity, strong dependence on data, and requires the establishment of a high-quality data structure, and has poor adaptability.

[0065] Millimeter-wave radar is an important sensor for sensing obstacles and other objects during automatic train driving. Due to its strong penetrating properties, millimeter-wave radar can work normally in extreme conditions such as rain, snow and fog. However, due to the multipath effect of millimeter-wave radar, there are a large number of false alarm objects such as empty targets and false targets in the objects corresponding to the collected point cloud data; the resolution of millimeter-wave radar is low, which makes it difficult to develop objects that rely solely on millimeter-wave radar detection.

[0066] Based on this, the present application provides a method for detecting the motion trend of an object based on a millimeter-wave radar. First, a millimeter-wave radar is used to obtain multiple objects corresponding to multiple frames of point cloud data. Then, through heuristic filtering operations such as ROI (Region of Interest) region (i.e., the area to be detected in this application) filtering, motion consistency filtering, and target life cycle filtering, the multiple objects corresponding to the acquired multiple frames of point cloud data are processed to filter out false alarm objects such as empty targets and false targets, and retain the real objects in the scene in front of the train to the greatest extent, that is, retain the real objects in the multiple objects corresponding to the multiple frames of point cloud data to the greatest extent. Since the vehicle is in motion when the millimeter-wave radar collects point cloud data, the real objects corresponding to the obtained multiple frames of point cloud data include both stationary objects that are stationary with the ground and moving objects that are actually moving. Therefore, the DBSCAN (Density-Based Spatial Clustering of Applications with Noise) algorithm is combined to screen the moving objects that are actually moving. Then, the real-time prediction of the motion trajectory of the moving object is completed through the trajectory prediction model, thereby realizing the determination of dangerous moving objects. For example, finally, the reflection characteristics and motion characteristics of the moving object are combined to complete the target category classification and realize the object type detection of the millimeter wave radar. This method has low hardware requirements and low computational complexity. At the same time, the method is simple and easy to implement. It does not require the analysis and research of the underlying Doppler signal of the millimeter wave radar, and overcomes the problems of cumbersome steps, strong dependence on underlying data, complex calculations, and low adaptability in related technologies.

[0067] The present application is described below in conjunction with system architecture, method and device.

[0068] See also Figure 1 , Figure 1 A schematic diagram of a system architecture is shown. The system may include a vehicle controller 100, a server 200 and a plurality of sensors 300. The server 200 may include one or more servers ( Figure 1 In the example, a server is included, and the server 200 can provide the method provided in the embodiment of the present application for one or more terminals.

[0069] The onboard controller 100 and multiple sensors 300 are deployed in the same train.

[0070] Exemplarily, the multiple sensors 300 include, but are not limited to: one or more millimeter wave radars, one or more image acquisition devices, one or more laser radars, and an IMU (Inertial Measuring Unit).

[0071] Exemplarily, the image acquisition device may be a camera or a webcam.

[0072] Exemplarily, a Global Positioning System (GPS) is also deployed on the train.

[0073] In some optional implementations, the vehicle controller 100 may send information collected by multiple sensors to the server 200 , and the server 200 sends the processing results to the vehicle controller 100 .

[0074] In some optional implementations, the vehicle controller 100 can obtain processing results based on information collected by multiple sensors without the need for server cooperation, which is not limited in the embodiments of the present application.

[0075] Exemplarily, the onboard controller 100 may be located in a driver's cab of a train.

[0076] It is understandable that the locations where the multiple sensors 300 are deployed on the train can be determined based on actual conditions, and the embodiments of the present application do not limit this.

[0077] Next describe Figure 1 The product form of the vehicle-mounted controller 100;

[0078] The vehicle-mounted controller 100 in the embodiment of the present application can be a mobile phone, a tablet computer, a wearable device, a vehicle-mounted device, an augmented reality (AR) / virtual reality (VR) device, a laptop computer, an ultra-mobile personal computer (UMPC), a netbook, a personal digital assistant (PDA), etc., and the embodiment of the present application does not impose any restrictions on this.

[0079] Figure 2 An optional hardware structure diagram of the vehicle-mounted controller 100 is shown.

[0080] refer to Figure 2 As shown, the vehicle controller 100 may include a radio frequency unit 110, a memory 120, an input unit 130, a display unit 140, a camera 150 (optional), an audio circuit 160 (optional), a speaker 161 (optional), a microphone 162 (optional), an earphone jack 163 (optional), a processor 170, an external interface 180, a power supply 190 and other components. Those skilled in the art will appreciate that Figure 2The foregoing is merely an example of an on-board controller and does not constitute a limitation on the on-board controller. The foregoing may include more or fewer components than those shown in the figure, or a combination of certain components, or different components.

[0081] The input unit 130 can be used to receive input digital or character information, and generate key signal input related to the user settings and function control of the vehicle-mounted controller. Specifically, the input unit 130 may include a touch screen 131 (optional) and / or other input devices 132. The touch screen 131 can collect the user's touch operations on or near it (such as the user's operation on or near the touch screen using any suitable object such as fingers, joints, stylus, etc.), and drive the corresponding connection device according to a pre-set program. The touch screen can detect the user's touch action on the touch screen, convert the touch action into a touch signal and send it to the processor 170, and can receive and execute the command sent by the processor 170; the touch signal at least includes contact point coordinate information. The touch screen 131 can provide an input interface and an output interface between the vehicle-mounted controller 100 and the user. In addition, the touch screen can be implemented using multiple types such as resistive, capacitive, infrared, and surface acoustic waves. In addition to the touch screen 131, the input unit 130 can also include other input devices. Specifically, other input devices 132 may include, but are not limited to, one or more of a physical keyboard, function keys (such as a volume control key, a switch key, etc.), a trackball, a mouse, a joystick, and the like.

[0082] Among them, the input device 132 can receive input data and the like.

[0083] The display unit 140 can be used to display information input by the user or information provided to the user, various menus of the vehicle controller 100, interactive interfaces, file display and / or playback of any multimedia file. In the embodiment of the present application, the display unit 140 can be used to display interfaces, processing results, etc.

[0084] The memory 120 can be used to store instructions and data. The memory 120 can mainly include an instruction storage area and a data storage area. The data storage area can store various data, such as multimedia files, texts, etc.; the instruction storage area can store software units such as operating systems, applications, instructions required for at least one function, or their subsets and extensions. It can also include a non-volatile random access memory; provide the processor 170 with hardware, software and data resources including management of computing and processing equipment, and support control software and applications. It is also used for the storage of multimedia files, and the storage of running programs and applications.

[0085] The processor 170 is the control center of the vehicle controller 100. It uses various interfaces and lines to connect various parts of the entire vehicle controller 100. By running or executing instructions stored in the memory 120 and calling data stored in the memory 120, it executes various functions of the vehicle controller 100 and processes data, thereby controlling the vehicle controller device as a whole. Optionally, the processor 170 may include one or more processing units; preferably, the processor 170 may integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, user interface and application program, and the modem processor mainly processes wireless communication. It is understandable that the above-mentioned modem processor may not be integrated into the processor 170. In some embodiments, the processor, memory, can be implemented on a single chip, and in some embodiments, they can also be implemented separately on independent chips. The processor 170 can also be used to generate corresponding operation control signals, send them to the corresponding components of the computing and processing equipment, read and process data in the software, especially read and process data and programs in the memory 120, so that each functional module therein performs corresponding functions, thereby controlling the corresponding components to act according to the requirements of the instructions.

[0086] Among them, the memory 120 can be used to store software codes related to the object motion trend detection method based on millimeter wave radar, the processor 170 can execute the steps of the object motion trend detection method based on millimeter wave radar, and can also schedule other units (such as the above-mentioned input unit 130 and the display unit 140) to realize corresponding functions.

[0087] The radio frequency unit 110 (optional) can be used for receiving and sending information or receiving and sending signals during a call, for example, after receiving the downlink information of the base station, it is sent to the processor 170 for processing; in addition, the designed uplink data is sent to the base station. Generally, the RF circuit includes but is not limited to an antenna, at least one amplifier, a transceiver, a coupler, a low noise amplifier (Low Noise Amplifier, LNA), a duplexer, etc. In addition, the radio frequency unit 110 can also communicate with network devices and other devices through wireless communication. The wireless communication can use any communication standard or protocol, including but not limited to Global System of Mobile communication (Global System of Mobile communication, GSM), General Packet Radio Service (General Packet Radio Service, GPRS), Code Division Multiple Access (Code Division Multiple Access, CDMA), Wideband Code Division Multiple Access (Wideband Code Division Multiple Access, WCDMA), Long Term Evolution (Long Term Evolution, LTE), email, Short Messaging Service (SMS), etc.

[0088] In this embodiment of the present application, the RF unit 110 can send data to the server 200 and receive processing results sent by the server 200.

[0089] It should be understood that the radio frequency unit 110 is optional and can be replaced by other communication interfaces, such as a network port.

[0090] The vehicle controller 100 also includes a power source 190 (such as a battery) for supplying power to various components. Preferably, the power source can be logically connected to the processor 170 through a power management system, so that the power management system can manage functions such as charging, discharging, and power consumption.

[0091] The vehicle controller 100 also includes an external interface 180 , which may be a standard Micro USB interface or a multi-pin connector, and may be used to connect the vehicle controller 100 to communicate with other devices, or may be used to connect a charger to charge the vehicle controller 100 .

[0092] Although not shown, the vehicle controller 100 may also include a flashlight, a wireless fidelity (WiFi) module, a Bluetooth module, sensors with different functions, etc., which will not be described in detail here. Some or all of the methods described below may be applied to Figure 2 In the vehicle controller 100 shown.

[0093] Next describe Figure 1 The product form of the server 200;

[0094] Figure 3 A schematic diagram of the structure of a server 200 is provided, such as Figure 3 As shown, the server 200 includes a bus 201, a processor 202, a communication interface 203, and a memory 204. The processor 202, the memory 204, and the communication interface 203 communicate with each other via the bus 201.

[0095] The bus 201 may be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 3 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0096] The processor 202 may be any one or more of a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP).

[0097] The memory 204 may include a volatile memory, such as a random access memory (RAM). The memory 204 may also include a non-volatile memory, such as a read-only memory (ROM), a flash memory, a hard disk drive (HDD), or a solid state drive (SSD).

[0098] Among them, the memory 204 can be used to store software codes related to the object motion trend detection method based on millimeter wave radar, the processor 202 can execute the steps of the object motion trend detection method based on millimeter wave radar of the chip, and can also schedule other units to implement corresponding functions.

[0099] It should be understood that the above-mentioned vehicle controller 100 and server 200 can be centralized or distributed devices, and the processors in the above-mentioned vehicle controller 100 and server 200 (such as processor 170 and processor 202) can be hardware circuits (such as application specific integrated circuit (ASIC), field-programmable gate array (FPGA), general-purpose processor, digital signal processor (DSP), microprocessor or microcontroller, etc.), or a combination of these hardware circuits. For example, the processor can be a hardware system with the function of executing instructions, such as CPU, DSP, etc., or a hardware system without the function of executing instructions, such as ASIC, FPGA, etc., or a combination of the above-mentioned hardware systems without the function of executing instructions and hardware systems with the function of executing instructions.

[0100] It is understandable that the positions where the multiple sensors 300 are deployed on the train can be determined according to actual conditions, and the embodiments of the present application do not limit this. The following example illustrates the positions where the multiple sensors 300 are deployed on the train.

[0101] like Figure 4 As shown, it is a schematic diagram of the locations of multiple sensors provided in an embodiment of the present application deployed on a train.

[0102] like Figure 4 The figure shows the appearance of the train's driver's cab, i.e., the front of the train. Figure 4 The appearance of the driver's cab shown in the figure is only an example, and this application does not limit the appearance of the driver's cab.

[0103] Multiple sensors 300 can be installed on the driver's cab. Exemplarily, multiple sensors such as millimeter wave radars, image acquisition devices, laser radars and inertial measurement units can be installed on the same vertical line perpendicular to the ground. Exemplarily, the vertical line can correspond to the central axis of the front of the train. Exemplarily, the vertical line may not be the central axis of the front of the train. The distance between the central axis (the thick solid line with an arrow represents the central axis) and the leftmost side of the front of the train (the leftmost side is represented by a dotted line) is the same as the distance between the central axis and the rightmost side of the front of the train (the rightmost side is represented by a dotted line).

[0104] It is understandable that the present application deploys multiple sensors such as millimeter-wave radar, lidar, image acquisition device, IMU, etc. on the same vertical line. The advantage of this layout is that the external parameter calibration is more convenient during the multi-sensor fusion process. Since the 3D millimeter-wave radar does not have height information, when it is on the same vertical line with other sensors, it is only necessary to measure the longitudinal distance between the millimeter-wave radar and other sensors to calibrate the translation matrix without calibrating the rotation matrix. The final external parameter matrix includes the unit rotation matrix and the translation matrix, which provides a solid foundation for subsequent multi-sensor fusion detection.

[0105] Exemplarily, multiple sensors such as millimeter wave radars, image acquisition devices, lidars, and inertial measurement units can be installed on different multiple vertical lines.

[0106] Exemplarily, the laser radar includes, but is not limited to: solid-state laser radar, rotating laser radar.

[0107] For example, Figure 4 The central axis (y-axis) and the thin solid line with an arrow (x-axis) in the figure form the coordinate system. Figure 5 As shown, the millimeter wave radar in the embodiment of the present application Figure 4 Schematic diagram of the installation position in the coordinate system shown.

[0108] It is understandable that the millimeter-wave radar is installed along the central axis and strictly follows the requirements of the horizontal position deviation limit and the vertical position deviation limit of the millimeter-wave radar. That is, when installing the millimeter-wave radar, it is necessary to ensure that its position in the horizontal and vertical directions is very accurate and does not exceed the allowable error range. This is to ensure that the millimeter-wave radar can correctly perceive the surrounding environment without generating perception errors or blind spots due to installation position deviations. In this way, the millimeter-wave radar can maintain the best perception performance.

[0109] like Figure 5 As shown in the figure, according to the perception performance of millimeter-wave radar, the deployment area of ​​millimeter-wave radar is divided into three areas: full operation area, performance-limited area and avoid installation area; the safe operation area is the middle position area of ​​the train area, the performance-limited area is the local area around the middle position area, and the avoid installation area is other areas except the middle position area and the performance-limited area. The following takes the train as an example to explain the full operation area, performance-limited area and avoid installation area.

[0110] like Figure 5As shown, the full operation area (rectangular area filled with squares) includes: the left and right position areas are within 600mm on both sides of the central axis and the upper and lower position areas are [road surface + 295mm, road surface + 800mm]. The performance-limited area (long direction area filled with dots) includes: the left and right position areas are within 600mm on the left side of the central axis (600mm, 900mm) and the right side of the central axis (600mm, 900mm), and the upper and lower position areas are [road surface + 800mm, road surface + 1000mm]. Other areas are areas to avoid installation.

[0111] It can be understood that, for the same millimeter-wave radar, the perception performance of the millimeter-wave radar installed in the fully operating area is greater than the perception performance of the millimeter-wave radar installed in the performance-limited area and greater than the perception performance of the millimeter-wave radar installed in the avoidance installation area.

[0112] Reference Figure 6 , Figure 6 A flow chart of a method for detecting object motion trend based on millimeter wave radar provided in an embodiment of the present application is shown in FIG. Figure 6 As shown, a method for detecting object motion trend based on millimeter wave radar provided in an embodiment of the present application may include steps S601 to S606, and these steps are described in detail below.

[0113] Step S601: Acquire multiple objects corresponding to multiple frames of point cloud data.

[0114] Exemplarily, multiple objects in each frame of point cloud data can be obtained based on multiple frames of point cloud data collected by the millimeter wave radar.

[0115] Exemplarily, the millimeter wave radar has a cluster mode and an object mode, wherein in the object mode, the millimeter wave radar can output the objects in each frame of point cloud data, that is, the object mode clusters the point cloud data, thereby reducing preprocessing and reducing computational complexity. In the present application, the millimeter wave radar can be set in the object mode, so that the millimeter wave radar can output multiple objects corresponding to multiple frames of point cloud data.

[0116] Exemplarily, the multiple objects corresponding to each frame of point cloud data output by the millimeter wave radar may be msg messages, and the msg message includes a message header (Header) and a data part (Data).

[0117] Exemplarily, the millimeter wave radar collects echo signals of an object through its sensor module and processes these signals to extract key parameters of the object, such as distance, speed, angle, and echo intensity.

[0118] It is understandable that there are usually two scenarios in which millimeter-wave radar works. The first scenario is a stationary train scenario, that is, a stationary millimeter-wave radar scenario, and the second scenario is a moving train scenario, that is, a scenario in which the millimeter-wave radar moves with the train. Based on the characteristics of millimeter-wave radar, millimeter-wave radar can collect moving objects. In the first scenario, the millimeter-wave radar will not collect objects that are relatively stationary with the ground, such as trees; in the second scenario, objects that are originally stationary with the ground will also produce a relative speed relative to the laser radar, so the collected objects include both objects that were originally stationary with the ground and objects that were actually moving. Therefore, in order to accurately detect the moving objects that are actually moving, it is necessary to screen the moving objects among the multiple objects in step S601.

[0119] Step S602: for each point cloud data, obtaining relative speeds corresponding to a plurality of objects in the point cloud data, where the relative speed of the object is the relative speed between the object and the millimeter wave radar.

[0120] Step S603: for each of the point cloud data, based on the relative speeds respectively corresponding to the multiple objects of the point cloud data, divide the multiple objects of the point cloud data into different clusters.

[0121] Exemplarily, the absolute value of the difference in relative speeds of objects in the same cluster is less than or equal to a preset difference.

[0122] Step S604: for each point cloud data, determine that an object outside the cluster containing the most objects is a moving object corresponding to the point cloud data.

[0123] Since the speed of the millimeter-wave radar moving with the train cannot be obtained, the absolute speed of the object cannot be calculated. Since the operating environment of trains such as high-speed railways and motor vehicles is generally in the wild, the number of moving objects in motion is relatively small, and the number of stationary objects that are relatively stationary with respect to the ground is relatively large. Based on this, the embodiment of the present application clusters multiple objects corresponding to each frame of point cloud data to obtain multiple clusters; the cluster containing the most objects is determined as the cluster where the objects that are relatively stationary with respect to the ground are located. Objects other than this cluster are determined to be moving objects.

[0124] Step S605: Determine the moving trajectory of the moving object based on the multiple frames of point cloud data.

[0125] It can be understood that multi-frame point cloud data is obtained in real time and is continuous in time; multi-frame point cloud data may contain the same moving object, and the coordinate information of the same moving object in different point cloud data is sorted from early to late according to the acquisition time of the point cloud data to obtain the sorting result, which is the moving trajectory of the moving object.

[0126] The following describes a process of how to obtain the same moving object from multiple frames of point cloud data. The process includes the following steps A1 to A3.

[0127] Step A1: For each frame of point cloud data, obtain the coordinate information of multiple moving objects corresponding to the point cloud data, and add timestamps to the multiple moving objects respectively (the timestamp is the acquisition time of the point cloud data) to obtain multiple groups of {moving object identification, moving object coordinate information, moving object timestamp}.

[0128] Step A2: For each mobile object determined to be tracked, the point cloud data of the previous and next frames are associated through the nearest neighbor algorithm, and the distance measurement between the newly scanned mobile object and the tracked mobile object is calculated. If the distance between the new mobile object and the tracked mobile object is the smallest and within the set heuristic threshold range, they are associated as the same mobile object.

[0129] The following example is used to illustrate. It is assumed that multiple frames of point cloud data are sorted from early to late according to the acquisition time: first point cloud data, second point cloud data, third point cloud data, and fourth point cloud data; it is assumed that the moving object A corresponding to the first point cloud data is determined to be the tracked moving object, then the nearest neighbor algorithm can be used to obtain the moving object B closest to the moving object A in the second point cloud data. If the distance between the moving object B and the moving object A is within the set heuristic threshold range, it is determined that the moving object A and the moving object B belong to the same moving object.

[0130] Assuming that the coordinate information of moving object A is (x1, y1, z1), and the coordinate information of moving object B is (x2, y2, z2), the distance d between moving object A and moving object B is: .

[0131] Step S606: inputting the moving trajectory of the moving object into a pre-built trajectory prediction model, and obtaining the predicted trajectory of the moving object in the future through the trajectory prediction model.

[0132] Exemplarily, the trajectory prediction model is obtained by training using the movement trajectory of the sample moving object as input and the labeled trajectory of the sample moving object as a training target.

[0133] The process of training the trajectory prediction model involves at least one of the techniques in machine learning, such as artificial neural network, belief network, reinforcement learning, transfer learning, inductive learning, and method-based learning.

[0134] Exemplarily, the trajectory prediction model may be any one of a neural network model, a logistic regression model, a linear regression model, a support vector machine (SVM), an Adaboost, an XGboost, a Transformer-Encoder model, or a large language model.

[0135] Exemplarily, the neural network model can be any one of a recurrent neural network-based model, a convolutional neural network-based model, and a Transformer-encoder-based classification model.

[0136] Exemplarily, the trajectory prediction model may be a deep hybrid model of a recurrent neural network-based model, a convolutional neural network-based model, and a Transformer-encoder-based classification model.

[0137] Exemplarily, the trajectory prediction model can be any one of an attention-based deep model, a memory network-based deep model, and a short text classification model based on deep learning.

[0138] The short text classification model based on deep learning is a recurrent neural network (RNN) or a convolutional neural network (CNN) or a variant based on a recurrent neural network or a convolutional neural network.

[0139] For example, some simple domain adaptation transformation can be performed on the pre-trained model to obtain a trajectory prediction model.

[0140] Exemplarily, “simple domain adaptation” includes but is not limited to re-pre-training an already pre-trained model using large-scale unsupervised domain corpus, and / or compressing an already pre-trained model by model distillation.

[0141] Exemplarily, the trajectory prediction model can be subjected to supervised learning or semi-supervised learning. Semi-supervised learning is a learning method that combines supervised learning with unsupervised learning. Semi-supervised learning uses a large amount of unlabeled data and labeled data to perform pattern recognition.

[0142] Exemplarily, the trajectory prediction model may be a Brownian motion model. Exemplarily, the key parameters are estimated by the Brownian motion model, the future position of the moving object is inferred based on the Brownian motion model and the parameters, and the prediction is adjusted considering the influence of rain, snow and fog.

[0143] The embodiment of the present application provides a method for detecting the motion trend of an object based on a millimeter-wave radar, and obtains multiple objects corresponding to multiple frames of point cloud data; there are usually two scenarios in which the millimeter-wave radar works. The first scenario is a stationary train scenario, that is, a stationary millimeter-wave radar scenario, and the second scenario is a moving train scenario, that is, a scene in which the millimeter-wave radar moves with the train. In the first scenario, the millimeter-wave radar will not collect objects that are relatively stationary with the ground, such as trees; in the second scenario, objects that are originally stationary with the ground will also produce relative speeds relative to the laser radar, so the collected objects include both objects that are originally relatively stationary with the ground and objects that are actually moving. Since the speed of the millimeter-wave radar moving with the train cannot be obtained, the absolute speed of the object cannot be calculated. Since the operating environment of trains such as high-speed railways and motor vehicles is generally in the wild, the number of moving objects in motion is small, and the number of objects that are relatively stationary with the ground is large. Based on this, the embodiment of the present application clusters the multiple objects corresponding to each frame of point cloud data to obtain multiple clusters; the cluster containing the most objects is determined as the cluster where the objects that are relatively stationary with the ground are located. Objects other than this cluster are determined to be moving objects. Based on multi-frame point cloud data, the moving trajectory of the moving object is determined; the moving trajectory of the moving object is input into the pre-built trajectory prediction model, and the predicted trajectory of the moving object in the future time is obtained through the trajectory prediction model. Thus, the purpose of obtaining the movement trend of the moving object is achieved.

[0144] It is understandable that the millimeter wave radar is affected by the Doppler effect and can only collect objects in relative motion. When the millimeter wave radar moves with the train, the collected objects include two types: objects that were originally stationary with the ground and objects that actually move with the ground. At this time, these two types of objects have a relative speed for the millimeter wave radar, and it is impossible to directly detect which objects are stationary objects that are actually stationary with the ground and which objects are moving objects that actually move with the ground. Therefore, this application also needs to detect moving objects that actually move with the ground. Since the running speed of the train is not easy to obtain when the millimeter wave radar moves with the train, it is impossible to directly determine whether it is a moving object by calculating the absolute speed of the object. Considering that stationary objects that are stationary with the ground have similar relative speeds, the relative speeds of moving objects and stationary objects have a relatively obvious difference. This application adopts the DBSCAN algorithm, takes relative speed as the feature dimension, and takes Euclidean distance as the distance threshold. Through multi-layer iterations of the algorithm, stationary objects with similar relative speeds can be clustered into the largest cluster (i.e., the cluster containing the most objects), and the remaining objects are considered to be moving objects. It is understandable that there are multiple ways to implement step S603. The embodiment of the present application provides but is not limited to the following method. For each frame of point cloud data, the method includes the following steps B1 to B5 during implementation.

[0145] Step B1: determining an untraversed object among a plurality of objects corresponding to the point cloud data as a search object.

[0146] Exemplarily, a label may be assigned to each object; different objects may have different labels. Initially, the labels of different objects all have the same value, such as -1, indicating that the object is not classified.

[0147] Exemplarily, if the object has been traversed, a traversal identifier may be assigned to the object; an object that has not been traversed is not assigned a traversal identifier, so that it can be determined which object among the multiple objects has not been traversed.

[0148] Exemplarily, the search object may be a noise object.

[0149] Step B2: If the search object has been classified into a target cluster, a target object whose distance to the search object is less than or equal to a neighbor distance threshold among the multiple objects corresponding to the point cloud data is classified into the target cluster.

[0150] It can be understood that the target cluster is any cluster.

[0151] The distance between the search object and the target object is the absolute value of the difference between the relative speed of the search object and the relative speed of the target object.

[0152] Step B3: If the search object is not classified into any cluster, the number of target objects whose distance to the search object is less than or equal to the neighbor distance threshold is obtained from the multiple objects corresponding to the point cloud data. If the number of targets is greater than or equal to the preset minimum number of points, step B4 is executed; if the number of targets is less than the preset minimum number of points, step B5 is executed.

[0153] Exemplarily, the neighbor distance threshold (eps) may be determined based on actual conditions, and the present application does not impose any limitation thereto. For example, the neighbor distance threshold is 0.08.

[0154] Step B4: Divide the search object and the target object into the same cluster.

[0155] Exemplarily, the preset minimum number of points (min_pts) may be determined based on actual conditions, and the present application does not impose any limitation thereto. For example, the preset minimum number of points is 5.0.

[0156] Exemplarily, objects belonging to the same cluster have the same label.

[0157] Step B5: Determine that the search object is a noise object.

[0158] Exemplarily, the label of the noise object is -2.

[0159] Step B6: Return to step B1 until the multiple objects corresponding to the point cloud data are all determined as search objects.

[0160] After multiple objects have been determined as search objects, it can be determined that all objects except the object in the largest cluster are moving objects, namely "move_object", and the objects in the largest cluster are stationary objects, namely "geo_stationary_object".

[0161] It is understandable that the clustering of the millimeter-wave radar in object mode has certain errors. Since the millimeter-wave radar relies on Doppler reflection to collect point cloud data, there are a large number of false alarm objects such as empty targets and false targets in the multiple objects corresponding to the collected point cloud data. Therefore, it is necessary to set a corresponding heuristic filtering algorithm to remove false alarm objects and retain real objects.

[0162] Among the multiple objects corresponding to the point cloud data collected by the millimeter-wave radar, the existence of false alarm objects is mainly due to the following reasons:

[0163] Environmental interference: When working, millimeter-wave radar will be interfered by the surrounding environment, such as metal objects, road reflections, etc. These interferences may cause the millimeter-wave radar to mistakenly identify non-existent objects.

[0164] Limitations of millimeter-wave radar characteristics: Due to the relatively low angular resolution of millimeter-wave radar, it cannot return the width information of the object, resulting in long, continuous obstacles being identified as multiple echo points.

[0165] Impact of train structure: Metal particles in the metallic paint on the front bumper of the train refract and reflect the electromagnetic waves emitted and received by the millimeter-wave radar, causing the millimeter-wave radar to generate echoes in multiple directions for the same object, which is mistakenly judged as multiple objects.

[0166] Train attitude change: The pitch angle of the train attitude changes during the bumpy process, causing the forward millimeter-wave radar to output the echo from the road surface as an object.

[0167] Exemplarily, the method for removing false alarm objects includes the following steps C1 to C4.

[0168] Step C1: Acquire multiple initial objects corresponding to multiple frames of point cloud data collected by the millimeter wave radar.

[0169] This application refers to the multiple objects output by the millimeter wave radar as initial objects.

[0170] Step C2: Determine the area to be detected.

[0171] It can be understood that the area to be detected is the ROI area.

[0172] The above method uses the ROI area filtering method: by setting a region of interest (ROI), objects outside the region are excluded to reduce false alarms caused by environmental interference.

[0173] The following is a description of the drawings. Figure 7 The figure shows a schematic diagram of the detection area provided in an embodiment of the present application. Figure 7 In the figure, the X-axis is the traveling direction of the train.

[0174] Assume that the millimeter wave radar used in this application is a 77GHz long-range radar. The FOV (Field of View) of the millimeter wave radar within the short-range beam range is as high as plus or minus 60 degrees, and the FOV within the long-range beam range is plus or minus 9 degrees. Figure 7 The fan-shaped area filled with square grids is the field of view of the millimeter-wave radar. However, in actual use scenarios, the purpose of detecting moving objects is to detect objects in front of the millimeter-wave radar that affect the progress of the train. Therefore, under the premise of meeting the detection requirements, this application uses ROI area segmentation to segment the redundant areas and only retain the areas required for detection, that is, only retain the objects contained in the area to be detected as the input for subsequent detection. Specifically, the ROI area segmentation algorithm uses the collected object distance latitude (lateral) and object distance longitude (longitudinal) as heuristic filtering conditions, and then sets a reasonable area threshold to accurately define the area of ​​interest based on the spatial distribution characteristics of the object, so as to ensure that the area to be detected can cover all objects of interest. Finally, the data collected by the millimeter-wave radar is screened and segmented using the set threshold to extract all objects within the area of ​​interest. This application sets the area to be detected to include: X-axis direction [0, 200 meters], Y-axis direction [-5 meters, 5 meters]. Because the trains installed with the millimeter-wave radar only run in this area to be detected, the objects outside the remaining area are regarded as objects that do not affect the operation of the train. All unnecessary redundant points are discarded to reduce the complexity of subsequent detection.

[0175] Exemplarily, the millimeter-wave radar is installed on the front of the train, and the area to be detected includes: X-axis direction [0, 200 meters], Y-axis direction [-5 meters, 5 meters], the X-axis is the direction of travel of the train, the Y-axis is the direction perpendicular to the X-axis, and the intersection of the X-axis and the Y-axis is located at the front of the train.

[0176] The following is an explanation of the format of the data output by the millimeter wave radar. The millimeter wave radar includes: Object0Status (0x60A) message, Object1General (0x60B) message, Object2Quality (0x60C) message, Object3Extended (0x60D), Object4Warning (0x60E).

[0177] The Object0Status (0x60A) message includes the number of objects currently detected and the measurement cycle counter.

[0178] The Object1General (0x60B) message provides the position and speed of the tracked object and is sent repeatedly to all tracked objects. It includes object ID, distance (vertical x, horizontal y), relative speed (v x , v y ), dynamic attributes (0x0 moving indicates that the object is moving, 0x1 stationary indicates that the object is stationary, 0x2 oncoming indicates that the object is approaching the sensor, 0x3 stationary candidate indicates that the object is in the stationary candidate state, 0x4 unknown indicates that the object state is unknown, 0x5 crossing stationary indicates that the object is crossing and is stationary, 0x6 crossing moving indicates that the object is crossing and is moving, 0x7 stopped indicates that the object has stopped) and radar cross section (RCS) to facilitate real-time monitoring and analysis of object status.

[0179] The Object2Quality (0x60C) message is sent when the signal RadarCfg_SendQuality (0x200) is activated to provide quality information of the object and is also sent repeatedly. The Object Quality Information (0x60C) message provides quality indicators of the object, including the object ID, the RMS value of the distance, velocity and acceleration, and the measurement status of the object. The status code indicates the current status of the object, such as deleted, new, measured, etc.

[0180] The Object3Extended (0x60D) message provides additional attributes, including object ID, relative acceleration, category, relative position, orientation angle, length, and width.

[0181] The Object4Warning (0x60E) message provides the collision detection warning status and is sent only when the collision detection in the message CollDetCfg (0x400) is activated. The sending method is the same as Object1General, and the content includes the object ID and the collision detection area field.

[0182] Step C3: for each of the point cloud data, determining that an initial object located in the to-be-detected area among a plurality of initial objects corresponding to the point cloud data is a plurality of candidate objects corresponding to the point cloud data.

[0183] Step C4: for each of the point cloud data, delete the false alarm objects among the multiple candidate objects corresponding to the point cloud data, so as to obtain multiple objects corresponding to the point cloud data.

[0184] Exemplarily, there are multiple ways to implement step C4, and the embodiments of the present application provide but are not limited to the following five methods.

[0185] The first method for implementing step C4 includes the following steps: determining the candidate object with a relative speed of zero as the false alarm object.

[0186] It is understandable that the millimeter wave radar cannot detect an object with a relative speed of zero to itself. If it detects an object, it must be a false alarm object. The reason is as follows: due to the characteristics of the millimeter wave radar, it can only detect objects that move relative to the millimeter wave radar. Therefore, the objects collected by the millimeter wave radar should have a relative speed relative to the millimeter wave radar. For some false alarm objects such as empty targets and false targets, although other attributes exist, the relative speed is still 0. Therefore, this method can filter out these false alarm objects.

[0187] The second method for implementing step C4 includes the following steps: if the absolute value of the difference between the relative speeds of the candidate object in adjacent multi-frame point cloud data is greater than a preset threshold, determining that the candidate object is the false alarm object.

[0188] The second method of implementing step C4 is to use motion consistency filtering: using the motion characteristics of the object, analyzing the motion trajectory of the object, and excluding those objects that do not conform to the laws of physical motion, such as objects that suddenly appear or disappear.

[0189] If the absolute value of the object's speed (lateral speed and longitudinal speed) is greater than 0 and the speed jumps between the previous and next frames of point cloud data are small, then the object should not be a false object. The reason is as follows: Since the acquisition time interval between adjacent multi-frame point cloud data is very short, the relative speed jump of the same object will not be very large. If the relative speed jump of the same object in adjacent multi-frame point cloud data is large, then the object should be a false object.

[0190] The third method for implementing step C4 comprises the following steps: determining the candidate object whose dynamic attribute is a stationary state as the false alarm object.

[0191] The dynamic attributes of the millimeter-wave radar, namely the dynamic attributes contained in the Object1General (0x60B) message, can be used for judgment. The dynamic attributes include 0x0 moving, which means the object is moving; 0x1 stationary, which means the object is stationary; 0x2 oncoming, which means the object is approaching the sensor; 0x3 stationarycandidate, which means the object is in a stationary candidate state; 0x4 unknown, which means the object state is unknown; 0x5 crossing stationary, which means the object is crossing and stationary; 0x6 crossing moving, which means the object is crossing and moving; and 0x7 stopped, which means the object has stopped. The crossing mentioned above means that the moving direction of the object in the millimeter-wave radar field of view intersects with the moving direction of the train. For example, when pedestrians or vehicles cross the road, this situation is "crossing moving" or "crossing stationary". In simple terms, the path of the object intersects with the monitoring direction of the millimeter-wave radar.

[0192] Since millimeter-wave radar cannot detect relatively stationary objects, when the dynamic attribute of the collected object is stationary, it means that the object is a false alarm object. That is, when the dynamic attribute of the object is 0x1 stationary, 0x3 stationary candidate, 0x5 crossing stationary or 0x7 stopped, the dynamic attribute of the object is stationary.

[0193] The implementation method of the fourth step C4 includes the following steps: determining that the candidate object whose relative speed in adjacent multiple frames of point cloud data is not zero and whose coordinate information in adjacent multiple frames of point cloud data is the same is the false alarm object.

[0194] It can be understood that if the relative speed of an object in the previous and next frames of point cloud data is greater than 0, the coordinate information of the object in the previous and next frames of point cloud data will definitely change; if the relative speed of an object in the previous and next frames of point cloud data is greater than 0, and the coordinate information of the object in the previous and next frames of point cloud data does not change, it means that the object is a false object.

[0195] The fifth implementation of step C4 includes the following steps: determining the candidate object whose continuous existence period is less than or equal to the first duration and whose loss duration period is greater than or equal to the second duration as the false alarm object.

[0196] The fifth step C4 is implemented by object life cycle filtering, that is, tracking the life cycle of the object and excluding those objects that appear only in a short time, which are often caused by environmental interference or millimeter wave radar errors.

[0197] The fifth method is mainly to filter out false alarm objects caused by millimeter wave radar vibration and interference from metal, etc. These objects have the characteristics of short duration and short continuous appearance time. Therefore, in view of this characteristic, this application tracks each object and counts the continuous appearance time of each object. Specifically, first use the multi-target tracking algorithm based on Kalman filtering to track the object, and count the survival time of the object during the tracking process. Set the continuous existence period S L and loss duration S D During the tracking process, if an object is detected in a frame of point cloud data, the number of continuous existence cycles of the object is increased by one, otherwise the number of lost duration cycles is increased by one, and then the first duration and the second duration are set. When the condition is met: the continuous existence cycle < the first duration & the lost duration cycle > the second duration, it means that the duration of the object is short and the continuous appearance time is short, then the object is judged as a false alarm object and filtered out.

[0198] Exemplarily, in the present invention, the first duration is set to 2 and the second duration is set to 3.

[0199] Exemplarily, after the continuous existence period is interrupted, if the multi-frame point cloud data is collected in order from earliest to latest, the first point cloud data, the second point cloud data, the third point cloud data, the fourth point cloud data, and the fifth point cloud data. For object A, if the first point cloud data includes object A, the continuous existence period is updated to 1, if the second point cloud data includes object A, the continuous existence period is updated to 2, if the third point cloud data does not include object A, the loss continuous period is updated to 1; if the fourth point cloud data does not include object A, the loss continuous period is updated to 2; if the fifth point cloud data does not include object A, the loss continuous period is updated to 3.

[0200] Exemplarily, after the continuous existence cycle is interrupted, if the multi-frame point cloud data is collected in order from earliest to latest, the first point cloud data, the second point cloud data, the third point cloud data, the fourth point cloud data, and the fifth point cloud data. For object A, if the first point cloud data includes object A, the continuous existence cycle is updated to 1, if the second point cloud data does not include object A, the loss duration cycle is updated to 1, if the third point cloud data includes object A, the continuous existence cycle is updated to 2; if the fourth point cloud data does not include object A, the loss duration cycle is updated to 2; if the fifth point cloud data does not include object A, the loss duration cycle is updated to 3.

[0201] It can be understood that, for each frame of point cloud data, the false alarm object among the multiple candidate objects corresponding to the point cloud data is deleted.

[0202] In an optional implementation, the following steps D1 to D2 are also included.

[0203] Step D1: If the reflection cross-sectional area of ​​the moving object belongs to a first preset area range and the moving speed of the moving object belongs to a first preset speed range, it is determined that the target category of the moving object is a person.

[0204] Step D2: If the reflection cross-sectional area of ​​the mobile object belongs to a second preset area range and the moving speed of the mobile object belongs to a second preset speed range, determine that the target category of the mobile object is a vehicle.

[0205] It is understandable that trains usually have a larger reflection area and a larger movement speed, while pedestrians have a smaller reflection area and a smaller speed range, so by setting the reflection cross-sectional area RCS and the movement speed, pedestrians, vehicles, etc. can be classified.

[0206] It is understandable that if it is determined that the moving object may collide with the train based on the movement trend of the moving object, then the moving object is determined to be dangerous and can be displayed on the display screen. Figure 8 , which is a schematic diagram of a dangerous moving object provided in an embodiment of the present application.

[0207] Figure 8 In the figure, a square framed by a box represents a dangerous moving object, and a cube not framed by a box represents a non-dangerous moving object.

[0208] A method for detecting an object motion trend based on a millimeter-wave radar provided in an embodiment of the present application is introduced above. A device for executing the method for detecting an object motion trend based on a millimeter-wave radar is introduced below.

[0209] See also Fig. 9 , Fig. 9 The structure diagram of a device for detecting object motion trend based on millimeter wave radar provided in an embodiment of the present application is shown in FIG. Fig. 9 As shown, the object motion trend detection device based on millimeter wave radar includes:

[0210] A first acquisition module 901 is used to acquire multiple objects corresponding to multiple frames of point cloud data;

[0211] A second acquisition module 902 is used to acquire, for each of the point cloud data, relative speeds corresponding to a plurality of objects in the point cloud data, where the relative speed of the object is a relative speed between the object and the millimeter wave radar;

[0212] A clustering module 903 is used for dividing the multiple objects of the point cloud data into different clusters based on the relative speeds respectively corresponding to the multiple objects of the point cloud data for each of the point cloud data;

[0213] A first determining module 904 is used to determine, for each of the point cloud data, an object outside the cluster containing the most objects as a moving object corresponding to the point cloud data;

[0214] A second determination module 905 is used to determine the moving trajectory of the moving object based on the multiple frames of point cloud data;

[0215] The third acquisition module 906 is used to input the moving trajectory of the moving object into a pre-built trajectory prediction model, and obtain the predicted trajectory of the moving object in the future through the trajectory prediction model.

[0216] In an optional implementation, the clustering module includes:

[0217] A first determining unit, configured to determine an untraversed object among a plurality of objects corresponding to the point cloud data as a search object;

[0218] A first division unit is configured to, if the search object has been divided into a target cluster, divide a target object whose distance from the search object is less than or equal to a neighbor distance threshold among a plurality of objects corresponding to the point cloud data into the target cluster;

[0219] A first acquisition unit is used for acquiring, if the search object is not classified into any cluster, the number of target objects whose distance from the search object is less than or equal to a neighbor distance threshold from the multiple objects corresponding to the point cloud data; the distance between the search object and the target object is the absolute value of the difference between the relative speed of the search object and the relative speed of the target object;

[0220] A second division unit, configured to divide the search object and the target object into the same cluster if the target number is greater than or equal to a preset minimum number of points;

[0221] A second determining unit, configured to determine that the search object is a noise object if the target number is less than the preset minimum number of points;

[0222] A triggering unit is used to trigger the first determining unit until the multiple objects corresponding to the point cloud data are determined as search objects.

[0223] In an optional implementation, the first acquisition module includes:

[0224] A second acquisition unit is used to acquire a plurality of initial objects respectively corresponding to the multi-frame point cloud data collected by the millimeter wave radar;

[0225] A third determining unit, used to determine the area to be detected;

[0226] A fourth determining unit is used to determine, for each of the point cloud data, an initial object located in the to-be-detected area among the multiple initial objects corresponding to the point cloud data as the multiple candidate objects corresponding to the point cloud data;

[0227] The deleting unit is used to delete, for each of the point cloud data, a false alarm object from a plurality of candidate objects corresponding to the point cloud data, so as to obtain a plurality of objects corresponding to the point cloud data.

[0228] In an optional implementation, the deleting unit includes:

[0229] A first determining subunit, configured to determine that the candidate object with a relative speed of zero is the false alarm object;

[0230] A second determination subunit is configured to determine that the candidate object is the false alarm object if the absolute value of the difference between the relative speeds of the candidate object in adjacent multiple frames of point cloud data is greater than a preset threshold;

[0231] A third determination subunit, configured to determine the candidate object whose dynamic attribute is a stationary state as the false alarm object;

[0232] A fourth determination subunit is used to determine that the candidate object whose relative speed in adjacent multiple frames of point cloud data is not zero and whose coordinate information in adjacent multiple frames of point cloud data is the same is the false alarm object;

[0233] A fifth determination subunit, configured to determine that the candidate object whose continuous existence period is less than or equal to the first duration and whose loss duration period is greater than or equal to the second duration is the false alarm object;

[0234] The deleting subunit is used to delete the false alarm object from the multiple candidate objects corresponding to the point cloud data.

[0235] In an optional implementation, the millimeter-wave radar is installed on the front of the train, and the detection area includes: X-axis direction [0, 200 meters], Y-axis direction [-5 meters, 5 meters], the X-axis is the direction of the train's travel, the Y-axis is the direction perpendicular to the X-axis, and the intersection of the X-axis and the Y-axis is located at the front of the train.

[0236] In an optional implementation, the front of the train corresponds to a central axis, and the distance between the central axis and the leftmost side of the front of the train is the same as the distance between the central axis and the rightmost side of the front of the train; the left and right position areas for installing the millimeter-wave radar are within 600mm on both sides of the central axis; taking the road surface as the reference point, the upper and lower position areas for installing the millimeter-wave radar are [road surface + 295mm, road surface + 800mm].

[0237] In an optional implementation, the millimeter wave radar, image acquisition device, laser radar and inertial measurement unit are arranged on the same vertical line perpendicular to the ground.

[0238] In an optional implementation, the method further includes:

[0239] A third determination module, configured to determine that the target category of the moving object is a person if the reflection cross-sectional area of ​​the moving object belongs to a first preset area range and the moving speed of the moving object belongs to a first preset speed range;

[0240] The fourth determination module is used to determine that the target category of the moving object is a vehicle if the reflection cross-sectional area of ​​the moving object belongs to a second preset area range and the moving speed of the moving object belongs to a second preset speed range.

[0241] The present application also provides an electronic device in an embodiment. Fig.10 As shown, it shows a schematic diagram of the structure of an electronic device suitable for implementing the embodiment of the present application. The electronic device in the embodiment of the present application may include but is not limited to fixed terminals such as mobile phones, laptops, PDAs (personal digital assistants), PADs (tablet computers), desktop computers, etc. Fig.10 The electronic device shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0242] like Fig.10 As shown, the electronic device may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 1001, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1002 or a program loaded from a storage device 1008 to a random access memory (RAM) 1003. When the electronic device is powered on, various programs and data required for the operation of the electronic device are also stored in the RAM 1003. The processing device 1001, the ROM 1002, and the RAM 1003 are connected to each other via a bus 1004. An input / output (I / O) interface 1005 is also connected to the bus 1004.

[0243] Typically, the following devices may be connected to the I / O interface 1005: an input device 1006 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 1007 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 1008 including, for example, a memory card, a hard disk, etc.; and a communication device 1009. The communication device 1009 may allow the electronic device to communicate with other devices wirelessly or by wire to exchange data. Although Fig.10 An electronic device having various devices is shown, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead.

[0244] An embodiment of the present application also provides a computer program product including computer-readable instructions. When the computer-readable instructions are executed on an electronic device, the electronic device implements any of the millimeter-wave radar-based object motion trend detection methods provided in the embodiments of the present application.

[0245] A computer-readable storage medium is also provided in an embodiment of the present application. The storage medium carries one or more computer programs. When the one or more computer programs are executed by an electronic device, the electronic device can implement any of the millimeter-wave radar-based object motion trend detection methods provided in the embodiments of the present application.

[0246] It should also be noted that the device embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed over multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. In addition, in the drawings of the device embodiments provided by the present application, the connection relationship between the modules indicates that there is a communication connection between them, which may be specifically implemented as one or more communication buses or signal lines.

[0247] Through the description of the above implementation mode, the technicians in the field can clearly understand that the present application can be implemented by means of software plus necessary general hardware, and of course, it can also be implemented by special hardware including special integrated circuits, special CPUs, special memories, special components, etc. In general, all functions completed by computer programs can be easily implemented by corresponding hardware, and the specific hardware structure used to implement the same function can also be various, such as analog circuits, digital circuits or special circuits. However, for the present application, software program implementation is a better implementation mode in more cases. Based on such an understanding, the technical solution of the present application is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a readable storage medium, such as a computer floppy disk, a U disk, a mobile hard disk, a ROM, a RAM, a disk or an optical disk, etc., including a number of instructions to enable a computer device (which can be a personal computer, a training device, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0248] In the above embodiments, all or part of the embodiments may be implemented by software, hardware, firmware or any combination thereof. When implemented by software, all or part of the embodiments may be implemented in the form of a computer program product.

[0249] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from a website site, a computer, a training device, or a data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, training device, or data center. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a training device, a data center, etc. that includes one or more available media integrations. The available medium may be a magnetic medium, (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive (SSD)), etc.

Claims

1. A method for detecting object motion trend based on millimeter wave radar, characterized in that: include: Obtain multiple objects corresponding to multiple frames of point cloud data; For each of the point cloud data, obtaining relative speeds respectively corresponding to a plurality of objects in the point cloud data, where the relative speed of the object is a relative speed between the object and the millimeter wave radar; For each of the point cloud data, based on the relative speeds respectively corresponding to the multiple objects of the point cloud data, the multiple objects of the point cloud data are divided into different clusters; For each of the point cloud data, determining an object outside the cluster containing the most objects as a moving object corresponding to the point cloud data; Determining a moving trajectory of the moving object based on the multiple frames of point cloud data; The moving trajectory of the moving object is input into a pre-built trajectory prediction model, and the predicted trajectory of the moving object in the future time is obtained through the trajectory prediction model.

2. The object motion trend detection method based on millimeter wave radar according to claim 1, characterized in that: The step of dividing the multiple objects of the point cloud data into different clusters based on the relative speeds respectively corresponding to the multiple objects of the point cloud data comprises: Determine an untraversed object among the multiple objects corresponding to the point cloud data as a search object; If the search object has been classified into the target cluster, the target object whose distance from the search object is less than or equal to the neighbor distance threshold among the multiple objects corresponding to the point cloud data is classified into the target cluster; the distance between the search object and the target object is the absolute value of the difference between the relative speed of the search object and the relative speed of the target object; If the search object is not classified into any cluster, obtaining the number of target objects whose distances to the search object are less than or equal to a neighbor distance threshold from the multiple objects corresponding to the point cloud data; If the target number is greater than or equal to a preset minimum number of points, the search object and the target object are divided into the same cluster; If the target number is less than the preset minimum number of points, determining that the search object is a noise object; The returning step determines an untraversed object among the multiple objects corresponding to the point cloud data as a search object until all the multiple objects corresponding to the point cloud data are determined as the search object.

3. The object motion trend detection method based on millimeter wave radar according to claim 1, characterized in that: The step of obtaining multiple objects corresponding to multiple frames of point cloud data includes: Acquire multiple initial objects respectively corresponding to multiple frames of point cloud data collected by the millimeter wave radar; Determine the area to be inspected; For each of the point cloud data, determining that an initial object located in the to-be-detected area among a plurality of initial objects corresponding to the point cloud data is a plurality of candidate objects corresponding to the point cloud data; For each of the point cloud data, a false alarm object among a plurality of candidate objects corresponding to the point cloud data is deleted to obtain a plurality of objects corresponding to the point cloud data.

4. The object motion trend detection method based on millimeter wave radar according to claim 3 is characterized in that: The step of deleting false alarm objects from a plurality of candidate objects corresponding to the point cloud data comprises: Determining the candidate object whose relative speed is zero as the false alarm object; If the absolute value of the difference between the relative speeds of the candidate object in adjacent multi-frame point cloud data is greater than a preset threshold, the candidate object is determined to be the false alarm object; Determining the candidate object whose dynamic attribute is a stationary state as the false alarm object; Determine as the false alarm object the candidate object whose relative speed in adjacent multiple frames of point cloud data is not zero and whose coordinate information in adjacent multiple frames of point cloud data is the same; Determine that the candidate object whose continuous existence period is less than or equal to the first duration and whose loss duration period is greater than or equal to the second duration is the false alarm object; The false alarm object among the multiple candidate objects corresponding to the point cloud data is deleted.

5. The object motion trend detection method based on millimeter wave radar according to claim 3 is characterized in that: The millimeter-wave radar is installed on the front of the train, and the detection area includes: X-axis direction [0, 200 meters], Y-axis direction [-5 meters, 5 meters], the X-axis is the direction of the train's travel, the Y-axis is the direction perpendicular to the X-axis, and the intersection of the X-axis and the Y-axis is located at the front of the train.

6. The object motion trend detection method based on millimeter wave radar according to claim 5 is characterized in that: The front of the train corresponds to a central axis, and the distance between the central axis and the leftmost side of the front of the train is the same as the distance between the central axis and the rightmost side of the front of the train; the left and right position areas for installing the millimeter-wave radar are within 600mm on both sides of the central axis; taking the road surface as the reference point, the upper and lower position areas for installing the millimeter-wave radar are [road surface + 295mm, road surface + 800mm].

7. The object motion trend detection method based on millimeter wave radar according to claim 6 is characterized in that: The millimeter wave radar, image acquisition device, laser radar and inertial measurement unit are arranged on the same vertical line perpendicular to the ground.

8. The method for detecting object motion trend based on millimeter wave radar according to any one of claims 1 to 8, characterized in that: Also includes: If the reflection cross-sectional area of ​​the mobile object belongs to the first preset area range and the moving speed of the mobile object belongs to the first preset speed range, determining that the target category of the mobile object is a person; If the reflection cross-sectional area of ​​the mobile object belongs to the second preset area range and the moving speed of the mobile object belongs to the second preset speed range, it is determined that the target category of the mobile object is a vehicle.

9. A device for detecting object motion trend based on millimeter wave radar, characterized in that: include: A first acquisition module is used to acquire multiple objects corresponding to multiple frames of point cloud data; A second acquisition module is used to acquire, for each of the point cloud data, relative speeds corresponding to a plurality of objects in the point cloud data, where the relative speed of the object is a relative speed between the object and the millimeter wave radar; A clustering module, for dividing the multiple objects of the point cloud data into different clusters based on the relative speeds respectively corresponding to the multiple objects of the point cloud data for each of the point cloud data; A first determination module is used to determine, for each point cloud data, an object outside the cluster containing the most objects as a moving object corresponding to the point cloud data; A second determination module, used to determine the moving trajectory of the moving object based on multiple frames of point cloud data; The third acquisition module is used to input the moving trajectory of the moving object into a pre-built trajectory prediction model, and obtain the predicted trajectory of the moving object in the future through the trajectory prediction model.

10. An electronic device, characterized in that: The method comprises at least one processor and a memory connected to the processor, wherein: The memory is used to store computer programs; The processor is used to execute the computer program so that the electronic device can implement the object motion trend detection method based on millimeter wave radar as described in any one of claims 1 to 8.

Images